The last two weeks were relatively calm in the search department, but certain things are going away and many new features were rolled out in Google Ads and Google My Business. Also, the search giant has started several new sites that are supposed to make small business owners' and creative makers' lives easier. So, keep reading to see all the updates, and please share any insights you have in the comments below.
GOOGLE SEARCH NEWS Google Says Goodbye to Unsupported Rules in Robots.txt Google announced on Twitter yesterday that they are saying goodbye to "undocumented and unsupported rules in robots.txt." Today we're saying goodbye to undocumented and unsupported rules in robots.txt ? If you were relying on these rules, learn about your options in our blog post. https://t.co/Go39kmFPLT — Google Webmasters (@googlewmc) July 2, 2019 In their blog post, they state, "While open-sourcing our parser library, we analyzed the usage of robots.txt rules. In particular, we focused on rules unsupported by the internet draft, such as crawl-delay, nofollow, and noindex. Since these rules were never documented by Google, naturally, their usage in relation to Googlebot is very low." Google's Gary Illyes chimed in and explained sites are hurting themselves, so this part of the reason the decision was made: As promised a few weeks ago, i ran the analysis about noindex in robotstxt. The number of sites that were hurting themselves very high. I honestly believe that this is for the better for the Ecosystem & those who used it correctly will find better ways to achieve the same thing. https://t.co/LvdhsN2pIE — Gary "鯨理" Illyes (@methode) July 2, 2019 What you should know is if you are using crawl-delay, nofollow, and noindex in your robots.txt file, they will be ignored by Google starting September 1, 2019. If you have relied on this method, Google has provided the following alternative options: Noindex in robots meta tags: Supported both in the HTTP response headers and in HTML, the noindex directive is the most effective way to remove URLs from the index when crawling is allowed. 404 and 410 HTTP status codes: Both status codes mean that the page does not exist, which will drop such URLs from Google's index once they're crawled and processed. Password protection: Unless markup is used to indicate subscription or paywalled content, hiding a page behind a login will generally remove it from Google's index. Disallow in robots.txt: Search engines can only index pages that they know about, so blocking the page from being crawled usually means its content won’t be indexed. While the search engine may also index a URL based on links from other pages, without seeing the content itself, we aim to make such pages less visible in the future. Search Console Remove URL tool: The tool is a quick and easy method to remove a URL temporarily from Google's search results.
Comments