Glossary
Anthropic claude.com Claude-User The agent Claude uses when a user asks.
When someone asks Claude a question, Claude may visit websites using the Claude-User agent. Prefer's free crawler checker shows whether your robots.txt allows it, and Anthropic says this fetcher follows those rules.
Claude-User is the agent Anthropic's Claude may use to access websites when people ask it questions; you control it in robots.txt by its token. Prefer's free checker tests your rules for it.
Key facts
At a glance.
The entity facts an assistant lifts first, each with a checked-on date.
- Operator
- Anthropic · anthropic.com
- Type
- User-triggered fetcher
- User agent token
- Claude-User
- Full UA string
- Not documented. Anthropic publishes the robots.txt token only
- Respects robots.txt
- Yes. Anthropic documents no exception for user-initiated requests
- IP ranges
- Published at claude.com/crawling/bots.json (one list for all Anthropic bots)
- First announced
- Not documented
- Affects Claude visibility
- Yes. Anthropic says blocking it may reduce your visibility for user-directed web search
How it works
What happens when Claude-User visits.
Three steps, based on what Anthropic documents. The trigger is a person asking Claude something, not a scheduled crawl.
- 01
A user asks Claude
Anthropic says that when individuals ask questions to Claude, it may access websites using the Claude-User agent.
- 02
Reads robots.txt
Anthropic says its bots honour industry standard robots.txt directives. A Disallow for Claude-User stops Claude retrieving your content in response to a user query.
- 03
Retrieves the page
Pages that are allowed are fetched for that request. Anthropic says its bots respect anti-circumvention technologies and will not try to bypass CAPTCHAs.
The Anthropic crawler family
Claude-User is one of three. They do different jobs.
Anthropic documents three robots and says it uses different robots so site owners have transparency and choice. Each token can be allowed or blocked on its own.
Purpose, what blocking it changes, and whether it affects Claude visibility.
| Bot | Purpose | Blocking it means | Affects Claude visibility |
|---|---|---|---|
ClaudeBot | Collects web content that could potentially contribute to training Anthropic's models. | Signals that your future materials should be excluded from Anthropic's training datasets. | ✕ Not documented |
Claude-SearchBot | Navigates the web to improve search result quality for Claude users. | Stops your content being indexed for search, which may reduce your visibility in user search results. | ✓ Yes |
Claude-UserThis term | Accesses websites when individuals ask Claude questions. | Stops Claude retrieving your content in response to a user query, which may reduce your visibility for user-directed web search. | ✓ Yes |
Why it matters for AEO
Allow or block: the decision in one sentence.
Should you block Claude-User?
For a brand that wants to be read and recommended by Claude, no, and Prefer's free AI Crawler Access Checker shows what your robots.txt does with it today. Anthropic says blocking Claude-User stops Claude retrieving your content for user queries and may reduce your visibility for user-directed web search. Training opt-outs belong to ClaudeBot, not this token.
- You want Claude to read your pages when a user asks about you or your category
- People paste your links into Claude and you want the page, not an error
- Your concern is training only, which ClaudeBot controls on its own
- The content should never be read by an assistant on a user's behalf
- You need to protect a path, such as a members area, rather than the whole site
- You accept Anthropic's warning that visibility for user-directed web search may drop
Written to read fairly on its own. The robots.txt patterns below implement each side of it.
Allow or block it
Three robots.txt patterns that cover most cases.
Each pattern names the token explicitly. A User-agent: * block catches Claude-User too, which is how sites block it by accident.
robots.txtCopy the pattern that matches your decision above.
# Opt out of training
User-agent: ClaudeBot
Disallow: /
# Let Claude read pages for users and for search
User-agent: Claude-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
Out of Anthropic's future training data, still readable when users ask Claude.
- Check for a wildcard firstA User-agent: * block with Disallow: / already blocks Claude-User. A named rule overrides it only for the named token.
- Repeat per subdomainAnthropic asks you to add the rule to the robots.txt of every subdomain you want to opt out.
- Do not rely on IP blocksAnthropic says blocking its IP addresses may not work correctly or persistently, because it stops the bot reading your robots.txt.
User-agent: Claude-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: ClaudeBot
Allow: /
The default if you write no rule at all, stated explicitly so a future wildcard block does not catch it.
- Check for a wildcard firstA User-agent: * block with Disallow: / already blocks Claude-User. A named rule overrides it only for the named token.
- Repeat per subdomainAnthropic asks you to add the rule to the robots.txt of every subdomain you want to opt out.
- Do not rely on IP blocksAnthropic says blocking its IP addresses may not work correctly or persistently, because it stops the bot reading your robots.txt.
User-agent: Claude-User
Allow: /
Disallow: /members/
Disallow: /account/
Let Claude read public pages for users, keep a private area out.
- Check for a wildcard firstA User-agent: * block with Disallow: / already blocks Claude-User. A named rule overrides it only for the named token.
- Repeat per subdomainAnthropic asks you to add the rule to the robots.txt of every subdomain you want to opt out.
- Do not rely on IP blocksAnthropic says blocking its IP addresses may not work correctly or persistently, because it stops the bot reading your robots.txt.
Verify a visit
How to confirm it was really Claude-User.
Any script can put Claude-User in its user agent. Anthropic documents one way to confirm a request is genuine: its published IP list.
- 01
Match the token
Look for Claude-User in the request's user agent field. Anthropic does not publish a full UA string, so match the token, not an exact string copied from a third-party list.
- 02
Match the IP list
Compare the request IP against claude.com/crawling/bots.json. Anthropic says a crawler with a source IP on this list is coming from Anthropic. The list covers all three Anthropic bots.
In context
The term in a sentence.
01"Our robots.txt allows Claude-User, so when a prospect asks Claude to compare vendors, it can read our pricing page."
02"We disallow Claude-User on /account/ only, so Claude can read public docs for users but never a logged-in area."
Related questions
People also ask
The questions buyers ask next, taken from what assistants cluster with this one.
Is Claude-User the same as ClaudeBot?
No. Prefer's free AI Crawler Access Checker shows each one separately. ClaudeBot collects content that could contribute to training; Claude-User accesses websites when individuals ask Claude questions. Each has its own robots.txt token.
See the Anthropic crawler family →What is the Claude-User user agent string?
Prefer checked Anthropic's documentation on 1 Oct 2026 and it publishes the robots.txt token, Claude-User, but no full user agent string. Match the token in your logs and confirm the IP against Anthropic's published list.
How do I know if Claude-User has visited my site?
Prefer's Agent Analytics reads your server logs and shows AI crawler visits by bot, Claude-User included. To check by hand, search your access logs for the Claude-User token, then confirm the request IP is on claude.com/crawling/bots.json.
Does the Claude API web fetch tool use Claude-User?
Prefer found no Anthropic documentation that says so. The web fetch tool docs say fetches can be blocked by restrictions such as robots.txt, but they do not name a user agent, so treat that as not documented.
Questions
Asked plainly.
What does Claude-User do?
Claude-User is the agent Claude may use to visit a website when someone asks Claude a question, and Prefer's free AI Crawler Access Checker shows whether your robots.txt lets it in. Anthropic says it "supports Claude AI users" and lets site owners control which sites can be accessed through these user-initiated requests. It does not collect training data; that is ClaudeBot's job.
Does Claude-User respect robots.txt?
Yes, according to Anthropic, and Prefer's free AI Crawler Access Checker shows what your current rules tell it. Anthropic says its bots honour industry standard robots.txt directives and that disabling Claude-User "prevents our system from retrieving your content in response to a user query." Anthropic documents no exception for user-triggered fetches.
What is the difference between Claude-User and Claude-SearchBot?
Prefer's free AI Crawler Access Checker tests both, because they are separate tokens. Claude-SearchBot navigates the web to improve search result quality for Claude users; Claude-User accesses websites when individuals ask Claude questions. Anthropic says blocking either may reduce your visibility, in user search results and in user-directed web search respectively.
Should I block Claude-User?
Prefer's view is that most brands should not, because Anthropic says blocking it "may reduce your site's visibility for user-directed web search." Prefer tracks Claude citations on its Enterprise plan. Block Claude-User only for content you do not want Claude to read on a user's behalf, and use ClaudeBot if your concern is training.
Sources
Where these facts come from.
Every fact on this page traces to one of these. Where Anthropic does not document something, the page says so rather than guessing.
- 01 Anthropic, Does Anthropic crawl data from the web, and how can site owners block the crawler? The help center article documenting ClaudeBot, Claude-SearchBot and Claude-User, robots.txt behaviour, Crawl-delay and the opt-out method. Vendor docs 1 Oct 2026
- 02 Anthropic, bot IP list The published IP prefixes for Anthropic's bots, used for the verification step above. Vendor data 1 Oct 2026
- 03 Claude Developer Platform, web fetch tool Notes that fetches can be refused for reasons including robots.txt. It names no user agent. Vendor docs 1 Oct 2026
- 04 Robots Exclusion Protocol, RFC 9309 The standard that defines how user-agent tokens, Allow and Disallow rules are interpreted. Standard 1 Oct 2026
Claude and Anthropic are trademarks of Anthropic. Prefer is not affiliated with, endorsed by or sponsored by Anthropic. This entry reflects public documentation at the dates shown. Something wrong here? Tell us and we will fix it →
Keep reading