What is a crawl budget?
Google’s goal is to make useful information available to people searching the web. To accomplish that, Google wants to crawl and index content from quality sources.
Crawling the web is costly: Google uses as much energy per year as the entire city of San Francisco, just to crawl websites. In order to crawl as many useful pages as possible, bots must follow planning algorithms that prioritize which pages to crawl and when. Google’s page importance is the idea that there are measurable ways to determine which pages to prioritize.
There’s no index of set values of crawls for each site. Instead, available crawls are distributed based on what Google thinks your server will handle and the interest it believes users will have in your pages.
Your website’s crawl budget is a way of quantifying how much Google spends to crawl it, expressed as an average number of pages per day.
Why optimize your crawl budget?
Thanks to OnCrawl’s data on hundreds of millions of pages, we’ve also learned that there is a strong correlation between how frequently Google crawls a page and the number of impressions it receives: pages that are crawled more often are seen more often in search results.