Gary Illyes from Google posted a brand new PSA on LinkedIn saying that the commonest cause a web site unexpectedly blocks Googlebot from crawling is because of a misconfiguration of a firewall or CDN.
Gary wrote, “test what site visitors your firewalls and CDN are blocking.” “By far the commonest difficulty in my inbox is said to firewalls or CDNs blocking googlebot site visitors. If I attain out to the blocking web site, within the overwhelming majority of the instances the blockage is unintended.”
So what are you able to do? Gary stated, “I’ve stated this earlier than, however need to emphasize it once more: make a behavior of checking your block guidelines. We publish our IP ranges so it ought to be very simple to run an automation that checks the block guidelines towards the googlebot subnets.”
Gary linked to this assist doc for extra particulars.
Briefly, do what you may to check to see in case your web site is accessible to Googlebot. You should utilize the URL inspection instrument in Google Search Console, as one methodology. Additionally, verify along with your CDN or firewall firm that they’re permitting Googlebot and ask them to show it.
Discussion board dialogue at on LinkedIn.
