Google is good at its job. Sometimes, a little too good – indexing pages it was never meant to find. Undesirable URLs that are not contained in your sitemap. Orphaned URLs not linked on your website.

Google’s John Mueller recently “liked” a tweet by search marketing consultant Barry Adams (of Polemic Digital) that concisely stated the purpose of the robots.txt exclusion protocol. He freshened up ...