Google’s Gary Illyes stated on the final Search Off the Document podcast that Google can crawl sure sections of your website extra often and in addition infer the standard of sure sections of your website in a different way.
This got here up on the 9:09 minute mark into the podcast however Glenn Gabe summarized it tremendous properly on Twitter. Glenn stated “Google can infer from a website total which areas they may have to crawl extra often. E.g. if there is a weblog subdirectory & there are alerts that it is common/necessary, then Google may wish to crawl there extra.” ” And it isn’t simply replace frequency, it is also about high quality. E.g. if G sees a sure sample is common (folder), & individuals are speaking about it & linking to it, then it is a sign that ppl like that listing,” he added.
Right here is the video embed:
Right here is the transcript of this part on crawl frequency by part of the location:
Yeah. As a result of like we stated, we do not have infinite area, so we wish to index stuff that we think– properly, not we– however our algorithms decide that it may be looked for sooner or later, and if we do not have alerts, for instance, but, a few sure website or a sure URL or no matter, then how would we all know that we have to crawl that for indexing?
And a few issues you possibly can infer from– for instance, in the event you launch a brand new weblog in your fundamental website, for instance, and you’ve got a brand new /weblog subdirectory, for instance, then we will kind of infer, based mostly on the entire website, whether or not we wish to crawl lots from that /weblog or not.
Then right here is the part on high quality:
Nevertheless it’s not simply replace frequency. It is also the standard alerts that the primary website has.
So, for instance, if we see {that a} sure sample could be very common on the web, like a slash product could be very common on the web, and other people on Reddit are speaking about it, different websites are linking to URLs in that sample, then it is a sign for us that individuals like the location on the whole.
Whereas when you’ve got one thing that individuals are not linking to, after which you are attempting to launch a brand new listing, it is like, properly, individuals do not like the location, then why would we crawl this new listing that you simply simply launched?
Discussion board dialogue at Twitter.
