Crawl Budget how much of your site a crawler fetches.

Crawl budget is the set of URLs a search engine can and wants to crawl on your site, limited by how much load your server can take and how much the engine wants your pages. Prefer’s Agent Analytics reads your server logs to show which AI crawlers visit and what they fetch, the AI side of the same question. For most small and mid-size sites, crawl budget is not the bottleneck; access is.

How crawl budget works#

Google’s guide to managing crawl budget defines it as “the set of URLs that Google can and wants to crawl,” made of two parts:

• Crawl capacity limit. How much Googlebot can fetch without overloading your server. Google says that if your site responds quickly and steadily the limit goes up, and if it slows down or returns server errors (5xx) or rate-limiting signals (HTTP 429), the limit goes down.
• Crawl demand. How much Google wants your pages, based on things like site size, how often pages change, page quality and relevance.

Google is explicit about who needs to care. The guide is written for large sites with over a million unique pages that change about weekly, and for sites with over 10,000 pages that change daily.

Why it matters for AI search visibility#

A page that is never fetched cannot be indexed, and a page that is not indexed or retrieved cannot be cited in an AI answer. On a large site, crawlers spending their time on duplicate filter pages or dead URLs means your important pages get fetched less often, so the version an engine has may be stale.

Google’s guide covers Google’s crawlers. AI companies run their own bots, each documented separately. OpenAI, for example, lists its bots: GPTBot collects training data, OAI-SearchBot surfaces sites in ChatGPT search, and ChatGPT-User visits pages on a user’s behalf. Do not assume Google’s crawl budget numbers apply to them. Your server logs show what each AI crawler actually requests.

Common confusions#

Crawl budget is not a ranking factor. Being crawled more often does not make a page rank or get cited more. It only means the engine has a fresher copy.

Blocking is not budgeting. Disallowing a bot in robots.txt removes those pages from what that bot fetches. Block OAI-SearchBot and, per OpenAI, your site will not appear in ChatGPT search answers. Use robots.txt to cut waste, not to cut off the crawlers you want citing you. The should I block AI crawlers answer covers the trade-offs.

Example#

An online store has 5,000 products, but filter combinations (colour, size, price) create 800,000 crawlable URLs. Crawlers spend most of their visits on near-duplicate filter pages, and new product pages wait days to be fetched. Blocking the filter parameters in robots.txt and keeping the sitemap current points crawlers back at the pages that matter.

How to protect your crawl budget#

Fig. 01

Four ways to stop wasting crawls

1Consolidate duplicatesMerge or canonicalise pages that say the same thing so crawlers fetch one version.

2Block low-value URLsUse robots.txt for filter, search and session URLs, never for pages you want cited.

3Return real status codesSend 404 or 410 for removed pages and fix soft 404s, which Google says keep getting crawled.

4Keep sitemaps and servers healthyCurrent sitemaps and fast, error-free responses let crawlers fetch more.

How to check crawl activity#

For Google, the Crawl Stats report in Search Console shows how many requests Googlebot made, how fast your server responded and which status codes it returned. For AI crawlers, read your server logs and filter by user-agent, such as GPTBot, OAI-SearchBot or ClaudeBot. Look for three things: whether the bot visits at all, which pages it fetches, and how many of its requests hit errors or redirects.

How Prefer helps#

Prefer’s Agent Analytics reads your server logs to show which AI crawlers visit and what they fetch, and the free Crawler Access Checker tests whether your robots.txt lets the AI search crawlers in. Then Prefer tracks whether five AI engines cite you.