Technical SEO

Crawl budget

Crawl budget describes the range of resources that a search engine can and is willing to devote to crawling a given website. It combines download performance constraints with the demand for visiting addresses.

Published: Updated: 3 min read
Diagram: Crawl budget - scanning budget. Opportunity - Demand - Scanning.

What is crawl budget?

Crawl budget describes the range of resources that a search engine can and is willing to devote to crawling a given website. It combines download performance constraints with the demand for visiting addresses. Scanning is not the same as indexing: a retrieved page may be omitted from the index. The concept is most relevant to large or rapidly changing sites, not as a default explanation for every small site problem.

Possibility and need for downloading

The server must respond stably so that the robot does not overload it with many connections. At the same time, the system assesses the need to revisit known addresses. Server errors, lags, and large duplicate spaces can make it difficult to download efficiently. However, there is no simple switch to increase the limit arbitrarily. Removing several addresses does not guarantee that the robot will immediately allocate exactly the same number of requests to other sites.

How to recognize the real problem?

Compare verified bot logs with scan statistics and indexing status in Search Console. Check whether important addresses are being discovered and retrieved, not just whether requests are increasing. Lots of requests to parameters or the calendar can waste resources. Lack of indexation of an already downloaded page also requires quality, duplication and canonical analysis. Speeding up your server alone won't fix unhelpful content.

Organize your URL space

Reduce the generation of unnecessary filter, sort, and session ID variants. Update the sitemap, shorten redirect chains and return the correct status of deleted resources. The choice of robots.txt, canonical and noindex depends on the goal: download blocking is not equivalent to de-indexing. The crawler needs to fetch the page to read the noindex. Don't combine conflicting instructions without understanding which one will even be available.

What should not be optimized apparently?

Don't remove valuable pages and links just to reduce the URL counter. Don't force lastmod date changes frequently without actually updating. A large number of requests is not an end in itself. What is more important is whether the robot efficiently reaches current, important content. In a small site with proper downloads, the priority may be responding to audience needs and good architecture, rather than advanced scanning budget management.

How to evaluate improvement?

Record your starting point: share of requests to important pages, errors, response time, and change discovery latency. After implementation, observe a comparable period and do not attribute every difference to one rule. Fluctuations in robot activity are natural. Effective improvement increases the efficiency of access to needed materials without disrupting navigation and indexing. The report should separate log evidence from hypotheses about the search engine's decisions.

Example of a parameter space problem

A store can have a moderate catalog but generate multiple filter orders, sorts, and session IDs. The robot then visits numerous variants of the same list. Before you increase the size of your server, check whether you can limit redundant addresses and maintain access to valuable combinations. Not every situation like this means a download limit, but order helps both diagnostics and the stability of the structure.

Compare verified robot requests to a map of valid URLs. Pay attention to statuses, time and frequency of changes. The increase in downloads after migration may be due to address reprocessing and the decrease may be due to the reduced need for updates. Don't treat both counter moves as an automatic success or failure. The goal is to efficiently download the content you need, not the highest possible number of requests.

Sources and further reading

Use the primary documentation to check the principles and limitations described in this article.

How we prepare the glossary

These articles combine original editorial work, AI support for editing and translation, and checks against linked primary documentation. Examples are illustrative unless stated otherwise; they do not represent client results. Dates refer to publication and actual content changes.

From knowledge to action

Need help putting this into practice?

Tell us about your website or process. We will define the scope and the next practical step.

Discuss your project