Crawl budget is the most over-discussed idea in technical SEO.
It is real, it matters at scale, and it is almost certainly not your problem. Google itself has been clear that sites with a few thousand URLs are generally crawled efficiently without any intervention.
So the useful version of this page is not "how to optimise crawl budget" — it is "how to tell whether you have a crawl budget problem at all", and what to do only if you do. The wider technical picture sits in our SEO guide and in technical SEO.
How Google decides how much to crawl
Two inputs, and they pull in different directions.
| Input | What it means | What raises it |
|---|---|---|
| Crawl capacity limit | How hard Google can crawl without degrading your site for real users. If your server slows or errors, Google backs off. | Fast, stable server responses |
| Crawl demand | How much Google actually wants your pages. Popular, frequently updated and freshly linked URLs get crawled more. | Links, updates, genuine popularity |
Note what is missing from that list: paying for anything, and any setting you can toggle. Crawl budget is earned through speed and interest, not configured.
Do you actually have a problem?
Run these four checks before you spend an hour on this.
- Count your indexable URLs. Under about 10,000 and healthy, this is not your bottleneck. Move on to something that matters.
- Open Search Console's Crawl Stats report. Look at total crawl requests and average response time. A response time creeping up while requests fall is the clearest real signal.
- Check the Pages report for "Discovered — currently not indexed". A large and growing number here on a large site is the actual symptom of crawl starvation.
- Ask whether new pages take weeks to appear. On a small site that is usually a quality or linking issue, not a crawl one.
What wastes it when it is genuinely tight
- Faceted navigation. Filter combinations multiply into millions of near-identical URLs. This is the single biggest cause on ecommerce sites.
- Internal search result pages. Infinite, low-value, and frequently crawlable by accident.
- Session IDs and tracking parameters creating duplicate URLs for the same page.
- Long redirect chains. Each hop is a request spent on nothing.
- Soft 404s. Pages that return 200 while showing "nothing found" waste crawls and confuse indexing.
- Orphaned and low-value pages that exist, get crawled, and serve no one.
What to do about it
| Action | Effect | Caution |
|---|---|---|
| Block junk URL patterns in robots.txt | Stops the crawl before it happens | Blocked pages can still be indexed if linked. Blocking is not deindexing. |
| Fix redirect chains | Each removed hop is a recovered request | Update the internal links too, not just the redirect |
| Return proper 404 or 410 | Google stops asking | 410 is a stronger signal than 404 for permanent removal |
| Tidy the XML sitemap | Points crawling at what matters | Include only indexable, canonical URLs |
| Improve server response time | Directly raises the capacity limit | The one fix that also helps users |
| Consolidate duplicates with a canonical tag | Reduces crawl demand on near-duplicates | A hint, not a directive — Google can ignore it |
Two things that do not work
The crawl-rate setting. Google retired the crawl rate limiter tool in Search Console. Crawling now adjusts automatically based on your server's responses.
Noindex to save crawl budget. A noindexed page still has to be crawled for Google to see the noindex. It saves index bloat, not crawl. Only robots.txt or removing the URL saves the crawl itself.
Where to go from here
Open Crawl Stats. If total requests are steady and response time is healthy, you are done — genuinely.
If you run a large catalogue, start with faceted navigation. It is almost always the biggest single source of waste, and it is also the source people are most reluctant to touch.
Frequently asked questions
What is crawl budget in SEO?
Crawl budget is the number of URLs a search engine will crawl on your site in a given period. Google sets it from crawl capacity limit — how much your server can handle without slowing down — and crawl demand, which reflects how much Google wants your pages based on popularity and freshness.
Does my site need crawl budget optimisation?
Probably not. Google has said that sites with a few thousand URLs are generally crawled efficiently without intervention. Crawl budget becomes a genuine constraint on large ecommerce catalogues, faceted navigation, news publishers and programmatically generated URLs. If none of those describe your site, your time is better spent elsewhere.
How do I check my crawl budget?
Open the Crawl Stats report in Google Search Console. Look at total crawl requests over time and average server response time. Then check the Pages report for a large or growing count under 'Discovered — currently not indexed', which is the clearest symptom of crawl starvation on a big site.
Does noindex save crawl budget?
No. Google has to crawl a page to see the noindex tag, so the request is spent either way. Noindex reduces index bloat, not crawling. To save the crawl itself you have to block the URL pattern in robots.txt or remove the URLs entirely.
What wastes crawl budget the most?
Faceted navigation on ecommerce sites, by a wide margin — filter combinations multiply into enormous numbers of near-identical URLs. After that: internal search result pages, tracking parameters creating duplicates, long redirect chains, and soft 404s that return a 200 status while showing nothing useful.
We audit the crawl before we touch the content
Log analysis, crawl stats and index coverage — so budget goes where the problem actually is.