Skip to content
MrKeyword
Technical SEO

Crawl budget explained: demand, capacity and practical fixes

Crawl budget is the number of URLs Googlebot can crawl on your site in a given time. It is set by how much Google wants to crawl you (crawl demand) and how much your server lets it crawl (crawl capacity). Here is how both work, how to spot waste in your logs and sitemaps, and what to fix first.

A crawl budget dashboard showing crawl requests, response times and host status
The short version
  • Crawl budget is crawl demand plus crawl capacity. Demand is how much Google wants to crawl you; capacity is how much your server can serve without slowing down.
  • Crawl demand comes from popularity and freshness. Pages that earn links, clicks and frequent updates are crawled more often than pages nobody visits or changes.
  • Crawl capacity is limited by server health. Slow responses, 5xx errors and timeouts make Googlebot fetch fewer URLs in the same time.
  • Server logs show where crawls really go. Filter for Googlebot and compare the URLs it fetches with the URLs you want indexed.
  • Every URL in your sitemap is a crawl request you are asking for. Keep it to canonical, indexable pages and fix or drop the rest.
  • Most crawl-budget fixes are duplicate and error fixes: remove parameter and filter URLs, return real 404s, flatten redirects and keep response times low.

What is crawl budget?

Crawl budget is the amount of crawling Googlebot will do on your site in a given period, and Google describes it as crawl demand plus crawl capacity. Crawl demand is how much Google wants to crawl your URLs. Crawl capacity is how much your site's server and infrastructure can serve without becoming slow or unreliable. Google's crawl budget documentation uses these two terms and says most sites do not need to worry about crawl budget at all.

Crawl budget is not a score you can increase on demand, and it is not a ranking factor. It is a limit on how many URLs Google can reasonably visit while it also serves the rest of the web. On a small, fast site the limit is rarely felt. On a large site, or one that generates many duplicate URLs, it decides how quickly new pages are found and how often existing pages are refreshed.

What is crawl demand?

Crawl demand is how much Google wants to crawl your site, based on how popular, fresh and useful it thinks your pages are. Google says demand is driven mainly by how popular a URL is and how often it changes, plus site-level signals such as quality and how much the site is linked and visited.

  • Popularity. URLs that get links, clicks and shares from across the web tend to be crawled more often. Pages nobody links to or visits are crawled less.
  • Freshness. Pages that change regularly are re-crawled more often than pages that never change. A page that has not changed in a year does not need the same crawl rate as a live news feed.
  • Perceived quality and spam signals. Sites with lots of thin, duplicate or spammy URLs tend to see less efficient crawling, because Google has less reason to spend requests there.

You cannot directly dial crawl demand up. You can earn more of it by making pages that people link to and return to, and by keeping existing pages accurate and updated. That is why content quality and crawl efficiency are connected: the pages you most want crawled should be the pages that deserve it.

What is crawl capacity?

Crawl capacity is how many requests your server can answer without slowing down or failing, and Google reduces crawling when capacity is poor. Google will not keep sending requests at a rate that overloads your hosting. If pages take a long time to respond, time out or return server errors, Googlebot spends more time waiting per request and fetches fewer URLs.

  • Response time. Slow server responses are the most common capacity problem. Cache pages that are expensive to render and move static files to a CDN.
  • Server errors. Repeated 5xx responses tell Google the host is unhealthy, and crawling can be slowed until the errors clear.
  • Timeouts and connection problems. A firewall, rate limit or DNS problem that blocks or drops Googlebot requests lowers effective capacity, even if the site is fast for visitors.
  • Redirects. Each redirect is an extra request. Long chains and redirect loops consume capacity without delivering content.

The Crawl stats report in Search Console shows total crawl requests, average response time and host status over 90 days. A falling request count alongside a rising response time is a capacity signal, not a content signal.

How do server logs show where crawl budget goes?

Your server logs record every Googlebot request, including the URL, status code and response time, so they show exactly which URLs are using your crawl budget. Search Console reports are sampled and grouped; logs are the full record for your host.

  1. Identify Googlebot by verified IP, not just user agent. Spoofed user agents are common. Google publishes its crawler IP ranges, and reverse DNS should resolve to googlebot.com or google.com.
  2. Group requests by URL pattern. Look for parameters, filters, search pages, pagination, calendars and session IDs. These often take a large share of requests while adding no unique indexable content.
  3. Group by status code. Count 2xx, 3xx, 4xx and 5xx. A large share of 3xx, 4xx or 5xx means crawls are not reaching live pages.
  4. Compare with the URLs you want indexed. If important pages are crawled rarely while parameter URLs are crawled daily, you have a crawl-budget problem worth fixing.

Log analysis answers a different question from the Crawl stats report. Use logs to find the exact patterns to block or fix, then use Crawl stats and the Page indexing report to confirm the change over time.

How do sitemaps affect crawl budget?

A sitemap is a list of URLs you are asking search engines to crawl, so every URL in it spends crawl budget if Google accepts it. Sitemaps help discovery, especially for new or weakly linked pages, but they do not force indexing and they do not raise crawl capacity.

  • List only canonical, indexable URLs. Do not include redirects, 404s, noindex pages, duplicate parameter URLs or pages blocked by robots.txt.
  • Keep lastmod accurate. A lastmod that changes on every request teaches Google to ignore it. Only update it when the page content actually changes.
  • Split large sitemaps. A sitemap can hold up to 50,000 URLs and be up to 50 MB uncompressed. Use a sitemap index for larger sites.
  • Submit the sitemap in Search Console and Bing Webmaster Tools. Then check the reported errors and fix them at the source.

For the full format and submission process, see the XML sitemap guide. Use the free sitemap validator to check the XML, count URLs and test a sample of links before you submit.

Practical crawl-budget fixes, in priority order

Fix the waste first, then improve capacity, then earn more demand. Most crawl-budget gains come from removing URLs and errors Google should not be spending requests on.

Crawl-budget problems, what they look like and the fix

ProblemWhat it looks likeFix
Parameter and filter URLsThousands of URLs from sorting, filters, tracking or session IDs, each crawled but not indexed.Block the patterns in robots.txt where they add no content, use canonical tags, and stop linking to them. Keep the parameters that genuinely change content.
Soft 404s and thin pagesEmpty search results, out-of-stock pages or placeholder pages return 200 but have no useful content.Return a real 404 or 410 for removed content, or add genuinely useful content. Do not let empty results look like normal pages.
Redirect chainsMultiple hops before the final page, and redirects that point to other redirects.Point redirects straight at the final URL and update internal links so they no longer hit the old address.
Slow responsesHigh average response time in Crawl stats, especially on templates that query a database or call an API on render.Cache rendered pages, remove live third-party calls from render, and serve static assets from a CDN.
Server errorsRepeated 5xx responses or a red host status in Crawl stats.Fix the server or application error, check the hosting limits, and make sure firewalls allow verified Googlebot IPs.
Duplicate hostnames and protocolshttp and https, www and non-www, or trailing-slash variants all return 200.Choose one canonical host and 301 every other variant to it. Keep the same rule across internal links and the sitemap.
Priorities follow Google's crawl budget documentation and the host status and response checks in the Crawl stats report.

Work through these in order. Removing duplicate and error URLs reduces demand on capacity immediately, and it makes the crawl reports readable, so you can see whether the remaining requests are going to pages that matter.

When does crawl budget actually matter?

Crawl budget matters when your site has far more URLs than Google can crawl in the time you need, or when server problems are limiting crawling. For a small brochure site with a few hundred pages, it is usually not the cause of an indexing problem. Check the basics first: is the page linked, is it canonical, is it noindexed, and does it return 200?

It is worth investigating when you see one or more of these signs:

  • New pages take weeks to be discovered even though they are linked and in the sitemap.
  • Crawl stats show a large share of 3xx, 4xx or 5xx responses.
  • Average response time is high or rising while crawl requests fall.
  • Logs show Googlebot crawling parameter, filter or duplicate URLs more than your real pages.
  • Your site has hundreds of thousands of URLs, or a marketplace, listings or faceted-search structure.

If none of these apply, spend the time on titles, content and internal links instead. Those are more likely to change rankings than chasing a crawl-budget number that is not actually limiting you.

Questions and answers

Crawl budget questions

What is crawl budget in simple terms?

Crawl budget is the amount of crawling Googlebot will do on your site in a given period. Google describes it as crawl demand (how much it wants to crawl your URLs) plus crawl capacity (how much your server can serve without slowing down). It is not a ranking factor.

What is the difference between crawl demand and crawl capacity?

Crawl demand is how much Google wants to crawl your site, driven by popularity, freshness and quality signals. Crawl capacity is how much your server can handle without slowing down or failing, limited by response time, errors, timeouts and host health.

How do I check my crawl budget?

Use the Crawl stats report in Google Search Console for total crawl requests, average response time and host status over 90 days. Then analyse your server logs, filtered to verified Googlebot IPs, to see which URL patterns and status codes actually use the requests.

Do sitemaps increase crawl budget?

No. A sitemap helps Google discover URLs, but it does not increase crawl demand or capacity. Every URL you list is a crawl request you are asking for, so keep the sitemap to canonical, indexable pages that return 200.

Does crawl budget affect small websites?

Usually not. Google says most sites do not need to worry about crawl budget. For a small site, an indexing problem is far more often a robots.txt, noindex, canonical or internal-linking issue than a crawl-budget limit.

How do I stop Googlebot crawling parameter URLs?

Stop linking to parameter URLs, use canonical tags to point variants at the main URL, and block the patterns that add no unique content in robots.txt. Do not block parameters that genuinely change page content.

About this article

Written by the MrKeyword team, who build an SEO workspace on first-party Search Console and Bing data. Outside facts link to their original source below. We update this page when the product or the sources change.

Sources

  1. Crawl budget — Google Search Central: crawl demand, crawl capacity and when crawl budget matters
  2. Crawl stats report — Search Console Help: total crawl requests, response time, host status and request breakdowns
  3. Build and submit a sitemap — Google Search Central: sitemap formats, limits, lastmod and submission
  4. Robots.txt introduction — Google Search Central: how robots.txt rules control crawling
Keep reading

More on technical SEO, plus the feature and free tool for the same job. All technical SEO guides →

Check what is eating your crawl budget.

Run a free site audit to find duplicate URLs, redirect chains, 5xx errors and slow templates, then fix the issues that affect the most pages first. Free plan for one website, no card; paid plans from $10 a month.