Glossary
Anthropic claude.com Claude-SearchBot Anthropic's search crawler for Claude.
The Anthropic bot that indexes pages so Claude can find them when it searches the web. Prefer's free crawler checker shows whether your robots.txt allows it, separately from ClaudeBot, the training crawler most rules are written for.
Claude-SearchBot is Anthropic's crawler that indexes web content to improve Claude's search results; you control it in robots.txt by its token. Prefer's free checker tests your rules for it.
Key facts
At a glance.
The entity facts an assistant lifts first, each with a checked-on date.
- Operator
- Anthropic · anthropic.com
- Type
- Web crawler, search indexing
- User agent token
- Claude-SearchBot
- Full UA string
- Not documented. Anthropic publishes the robots.txt token only
- Respects robots.txt
- Yes, and supports the non-standard Crawl-delay extension
- IP ranges
- Published at claude.com/crawling/bots.json (one list for all Anthropic bots)
- First announced
- Not documented
- Affects Claude search visibility
- Yes. Anthropic says blocking it may reduce your visibility in user search results
How it works
What happens when Claude-SearchBot visits.
Three steps, based on what Anthropic documents. The point to hold on to: this bot serves search, not training.
- 01
Reads robots.txt
Anthropic says its bots honour industry standard robots.txt directives. A Disallow for Claude-SearchBot stops it site-wide; a path rule stops it on that path only. Each subdomain needs its own robots.txt rule.
- 02
Fetches public pages
Pages that are reachable and not disallowed are fetched. Anthropic says its bots respect anti-circumvention technologies and will not try to bypass CAPTCHAs.
- 03
Indexes for search
Content is analyzed "to enhance the relevance and accuracy of search responses" for Claude users. Anthropic does not document how quickly a fetched page can appear in an answer.
The Anthropic crawler family
Claude-SearchBot is one of three. They do different jobs.
Anthropic documents three robots and says it uses different robots so site owners have transparency and choice. Each token can be allowed or blocked on its own.
Purpose, what blocking it changes, and whether it affects Claude search visibility.
| Bot | Purpose | Blocking it means | Affects Claude search visibility |
|---|---|---|---|
ClaudeBot | Collects web content that could potentially contribute to training Anthropic's models. | Signals that your future materials should be excluded from Anthropic's training datasets. | ✕ Not documented |
Claude-SearchBotThis term | Navigates the web to improve search result quality for Claude users. | Stops your content being indexed for search, which may reduce your visibility in user search results. | ✓ Yes |
Claude-User | Accesses websites when individuals ask Claude questions. | Stops Claude retrieving your content in response to a user query, which may reduce your visibility for user-directed web search. | ✓ Yes |
Why it matters for AEO
Allow or block: the decision in one sentence.
Should you block Claude-SearchBot?
For a brand that wants to be found in Claude, no, and Prefer's free AI Crawler Access Checker shows what your robots.txt does with it today. Anthropic says blocking Claude-SearchBot may reduce your visibility in user search results, and training opt-outs are handled by a different token, ClaudeBot. Block it only for content you do not want surfaced in Claude's search at all.
- You want Claude to find and surface your pages when it searches the web
- Your content is marketing, documentation or editorial you already publish openly
- You want to opt out of training only, which ClaudeBot controls on its own
- The content should not appear in any AI search experience
- You need to protect a path, such as a members area, rather than the whole site
- You accept Anthropic's warning that visibility in user search results may drop
Written to read fairly on its own. The robots.txt patterns below implement each side of it.
Allow or block it
Three robots.txt patterns that cover most cases.
Each pattern names the token explicitly. A User-agent: * block catches Claude-SearchBot too, which is how sites block it by accident.
robots.txtCopy the pattern that matches your decision above.
# Opt out of training
User-agent: ClaudeBot
Disallow: /
# Stay visible in Claude search
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
Out of Anthropic's future training data, still indexable for Claude's search and user fetches.
- Check for a wildcard firstA User-agent: * block with Disallow: / already blocks Claude-SearchBot. A named rule overrides it only for the named token.
- Repeat per subdomainAnthropic asks you to add the rule to the robots.txt of every subdomain you want to opt out.
- Do not rely on IP blocksAnthropic says blocking its IP addresses may not work correctly or persistently, because it stops the bot reading your robots.txt.
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: ClaudeBot
Allow: /
The default if you write no rule at all, stated explicitly so a future wildcard block does not catch it.
- Check for a wildcard firstA User-agent: * block with Disallow: / already blocks Claude-SearchBot. A named rule overrides it only for the named token.
- Repeat per subdomainAnthropic asks you to add the rule to the robots.txt of every subdomain you want to opt out.
- Do not rely on IP blocksAnthropic says blocking its IP addresses may not work correctly or persistently, because it stops the bot reading your robots.txt.
User-agent: Claude-SearchBot
Allow: /
Disallow: /members/
Disallow: /internal/
# Optional: slow the crawl (Anthropic supports Crawl-delay)
Crawl-delay: 1
Keep public pages searchable, keep a private area out.
- Check for a wildcard firstA User-agent: * block with Disallow: / already blocks Claude-SearchBot. A named rule overrides it only for the named token.
- Repeat per subdomainAnthropic asks you to add the rule to the robots.txt of every subdomain you want to opt out.
- Do not rely on IP blocksAnthropic says blocking its IP addresses may not work correctly or persistently, because it stops the bot reading your robots.txt.
Verify a visit
How to confirm it was really Claude-SearchBot.
Any script can put Claude-SearchBot in its user agent. Anthropic documents one way to confirm a request is genuine: its published IP list.
- 01
Match the token
Look for Claude-SearchBot in the request's user agent field. Anthropic does not publish a full UA string, so match the token, not an exact string copied from a third-party list.
- 02
Match the IP list
Compare the request IP against claude.com/crawling/bots.json. Anthropic says a crawler with a source IP on this list is coming from Anthropic. The list covers all three Anthropic bots.
In context
The term in a sentence.
01"We block ClaudeBot to opt out of training but allow Claude-SearchBot, so Claude can still find our pages when it searches."
02"Server logs show Claude-SearchBot hitting the pricing page from an IP on Anthropic's published list, so the visit is genuine."
Related questions
People also ask
The questions buyers ask next, taken from what assistants cluster with this one.
Does blocking ClaudeBot also block Claude-SearchBot?
No. They are separate tokens, and Prefer's free AI Crawler Access Checker shows each one's status. A rule for ClaudeBot applies to ClaudeBot only; Claude-SearchBot follows its own rule, or your User-agent: * rule if it has none.
See the Anthropic crawler family →What is the Claude-SearchBot user agent string?
Prefer checked Anthropic's documentation on 1 Oct 2026 and it publishes the robots.txt token, Claude-SearchBot, but no full user agent string. Match the token in your logs and confirm the IP against Anthropic's published list.
How do I know if Claude-SearchBot has crawled my site?
Prefer's Agent Analytics reads your server logs and shows AI crawler visits by bot, Claude-SearchBot included. To check by hand, search your access logs for the Claude-SearchBot token, then confirm the request IP is on claude.com/crawling/bots.json.
Can I slow Claude-SearchBot down instead of blocking it?
Yes. Prefer's free robots.txt generator can help you write the rule, and Anthropic says its bots support the non-standard Crawl-delay extension. Add a Crawl-delay line under the Claude-SearchBot user agent to limit how often it fetches.
Questions
Asked plainly.
What does Claude-SearchBot do?
Claude-SearchBot is Anthropic's search crawler, and Prefer's free AI Crawler Access Checker shows whether your robots.txt lets it in today. In Anthropic's words, it "navigates the web to improve search result quality for users" and analyzes content "to enhance the relevance and accuracy of search responses." It is separate from ClaudeBot, which collects content for training.
How do I block or allow Claude-SearchBot?
Prefer's free AI Crawler Access Checker shows what your robots.txt does with Claude-SearchBot now, and Prefer's free robots.txt generator writes the rule for you. By hand, add 'User-agent: Claude-SearchBot' followed by 'Allow: /' or 'Disallow: /'. Anthropic asks you to do this for every subdomain you want to opt out, and says blocking by IP address may not work reliably because it stops the bot reading your robots.txt.
What is the difference between Claude-SearchBot and ClaudeBot?
Prefer's free AI Crawler Access Checker tests both tokens, because they do different jobs. ClaudeBot collects web content that could contribute to training Anthropic's models; Claude-SearchBot indexes content to improve search results for Claude users. Anthropic says blocking ClaudeBot signals that your future materials should be excluded from training datasets, while blocking Claude-SearchBot may reduce your visibility in user search results.
Does blocking Claude-SearchBot stop Claude from citing my site?
Prefer tracks Claude citations on its Enterprise plan, and the honest answer from Anthropic's docs is that blocking Claude-SearchBot "may reduce your site's visibility and accuracy in user search results." Anthropic does not promise a specific outcome either way. If you want to show up when Claude searches the web, the safe default is to leave Claude-SearchBot allowed.
Sources
Where these facts come from.
Every fact on this page traces to one of these. Where Anthropic does not document something, the page says so rather than guessing.
- 01 Anthropic, Does Anthropic crawl data from the web, and how can site owners block the crawler? The help center article documenting ClaudeBot, Claude-SearchBot and Claude-User, robots.txt behaviour, Crawl-delay and the opt-out method. Vendor docs 1 Oct 2026
- 02 Anthropic, bot IP list The published IP prefixes for Anthropic's bots, used for the verification step above. Vendor data 1 Oct 2026
- 03 Robots Exclusion Protocol, RFC 9309 The standard that defines how user-agent tokens, Allow and Disallow rules are interpreted. Standard 1 Oct 2026
Claude and Anthropic are trademarks of Anthropic. Prefer is not affiliated with, endorsed by or sponsored by Anthropic. This entry reflects public documentation at the dates shown. Something wrong here? Tell us and we will fix it →
Keep reading