Crawl budget is the number of URLs on a site that Google can and wants to crawl in a given period. Google’s crawl budget guide describes it as the combination of two things: the crawl capacity limit, which is how much crawling your server can handle, and crawl demand, which is how much Google wants to crawl your pages.
Google says most sites don’t need to worry about it. The guide is written for sites with more than a million unique pages that change weekly, or sites where Search Console lists a large share of URLs as “Discovered, currently not indexed.” If your pages get crawled the same day you publish them, Google says keeping your sitemap current and checking indexing regularly is enough. Most B2B sites are nowhere near a million pages. When their deeper pages go uncrawled, the problem is usually that Google can’t find them or doesn’t want them, not that the budget ran out.
Capacity and demand
| Crawl capacity limit | Crawl demand | |
|---|---|---|
| What it is | How much crawling your server can take | How much Google wants to crawl your URLs |
| What raises it | A server that responds quickly over time | Pages that are popular, useful and updated |
| What lowers it | Slow responses and server errors | Duplicate, low-value or stale URLs |
The two don’t add up the way people expect. Google notes that even if your server could handle more, low demand means it crawls less, and that making your site more available won’t on its own increase your budget.
When crawling stops short of your best pages
A specialty paint manufacturer with more than 300 coatings had product pages that Google’s crawler wasn’t reaching. They sat too deep in the site, so they weren’t crawled, weren’t indexed and didn’t rank. The site was small by Google’s standards, so budget wasn’t the constraint. The crawl paths were. After the crawl and indexing fixes, alongside new landing pages, organic traffic grew 24.8% and tracked leads rose 47.7% from December 2023 to March 2024, the slowest season in its industry.
What wastes crawling
- Duplicate URLs. Google lists avoiding crawl time on duplicates as one reason to set canonical URLs.
- Soft 404 pages. Google says these keep getting crawled and waste budget. Return a real 404 or 410 for pages that are gone.
- Endless URL variations from filters, sorting options and tracking parameters.
- Redirect chains, where each hop costs a request.
Google warns that if its crawler spends too much time on URLs it shouldn’t, it may not get to the rest of your site.
How to check crawling
Search Console’s Crawl Stats report shows how many requests Google makes, how fast your server responds, and which responses it gets back. Pair it with the Page indexing report. A growing list of “Discovered, currently not indexed” pages means Google knows the URLs exist but hasn’t crawled them. On a small site, that points to weak internal links or low-value pages, not a budget limit.
Common mistakes
- Blaming crawl budget for indexing problems on a site of a few hundred pages.
- Burying important pages several clicks deep with few internal links.
- Using robots.txt to remove pages from the index. Google may still index blocked URLs without their content.
- Leaving soft 404s and parameter URLs live for crawlers to find.
Related terms
XML sitemap · Robots.txt · Canonical tag · Internal linking · Redirect chain
