Gary Illyes from Google stated on LinkedIn that in case your server returns a 500/503 HTTP standing code for an prolonged time period in your robots.txt file, then Google might take away your web site fully from Google Search.
That is even when the remainder of your web site is accessible and never returning a 500 or 503 standing code.
It isn’t only a 500/503 HTTP standing code that it’s essential fear about, it is usually a difficulty in case your web site does these community timeout points.
Once more, it needs to be for an “prolonged time period,” which was not outlined, however I assume it’s greater than only a day or two.
Gary wrote, “A robots.txt file that returns a 500/503 HTTP standing code for an prolonged time period will take away your web site from search outcomes, even when the remainder of the positioning is accessible to Googlebot.” “Identical goes for community timeouts,” Gary added.
Gary referred to the HTTP docs and added, “if we won’t decide what’s within the robotstxt file and the server would not inform us a robotstxt file would not exist, it will be far more hurtful to crawl as if the whole lot was allowed (eg. we’d index martin’s awkward hat photos by chance).”
We all know that Google recommends utilizing a 503 server standing code for when your web site goes offline or down quickly for lower than a day (hours not a number of days). If it goes offline for longer, than attempt to add a static model of your web site as a replacement.
Simply watch out with lengthy down time – not that you simply probably have a alternative.
Discussion board dialogue at LinkedIn.
