Gary Illyes, Google Webmaster Trends Analyst, recently warned website owners on LinkedIn about the potential consequences of their server returning a 500 or 503 HTTP status code for an extended period of time for their robots.txt file.
He wrote on LinkedIn:
“A robots.txt file that returns a 500/503 HTTP status code for an extended period of time will remove your site from search results, even if the rest of the site is accessible to Googlebot. Same goes for network timeouts.”
According to Illyes, if this occurs, Google may remove the website from its search results entirely, even if the rest of the site is accessible and not returning any error codes.
This is because the robots.txt file is used by Google to determine which pages on a website should be crawled and indexed, and if it is inaccessible or returns an error code, the website may not be properly indexed.
Illyes also noted that network timeouts can cause similar problems, and may also result in a website being removed from search results.
To avoid these issues, it is important to ensure that the robots.txt file is always accessible and does not return any error codes.
If a website needs to be taken offline temporarily, it is recommended to use a 503 server status code to indicate that the site is unavailable, allowing Google’s crawlers to continue to crawl and index the rest of the website.
It is also a good idea to have a static version of the website available during extended downtime, as this can be indexed by Google instead of the live site.
This can help to maintain the website’s presence in search results, even if the website is unavailable for an extended period of time.
In short, it is crucial for website owners to be aware of the potential consequences of their server returning error codes, and to take steps to avoid these issues in order to maintain their ranking on Google Search.