Crawl budget is the number of pages a search engine will crawl on your site within a given timeframe. It is determined by crawl capacity (how fast your server can handle requests) and crawl demand (how important Google considers your content).
Google defines crawl budget as the intersection of crawl rate limit (the maximum crawl frequency your server can handle without degradation) and crawl demand (how much Google wants to crawl based on popularity, freshness, and other signals). Large sites with millions of pages may not have all pages crawled regularly, making crawl budget optimization essential.
For small sites (under a few thousand pages), crawl budget is rarely an issue — Google crawls everything. For large sites with millions of pages, crawl budget becomes critical. If Google spends its crawl budget on low-value pages (faceted navigation, duplicate filters, expired products), important new or updated content may not be crawled promptly.
Optimize crawl budget by blocking low-value pages in robots.txt, using noindex on thin or duplicate pages, fixing crawl errors that waste crawl budget, improving server response time, updating XML sitemaps to prioritize important pages, and removing or consolidating low-quality content. Monitor crawl stats in Google Search Console.
Want to learn more? Explore our SEO Glossary or check out our expert guides on SEO strategies.