Right here is one other PSA from Gary Illyes of Google. In brief, when you serve a 4xx standing code along with your robots.txt file, then Google will ignore the foundations you’ve gotten laid out in that file.
Why? Properly, 4xx standing codes means the doc just isn’t obtainable, so Google will not verify it as a result of the server says it isn’t obtainable. Gary mentioned this as a result of he obtained a criticism or two about Google not respecting the robots.txt guidelines.
Gary wrote on LinkedIn, “PSA from my inbox: when you serve your robotstxt with a 403 HTTP standing code, all guidelines within the file can be ignored by Googlebot. Consumer errors (4xx, besides 429) imply unavailable robotstxt, as in, a 404 and a 403 are equal on this case.”
In brief, be sure your robots.txt file serves a 200 standing code and Google can entry it.
Discussion board dialogue at LinkedIn.
