Free tool
AI Crawler Access Checker can the bots behind AI answers reach you?
Paste your robots.txt and see, crawler by crawler, whether the bots behind ChatGPT, Claude, Perplexity, Copilot and Google AI Overviews can read your pages, and which ones only collect training data. Fourteen named crawlers, one clear verdict each. Runs in your browser, no signup.
- All five answer engines can reach this path.
- GPTBot is blocked: OpenAI will not use these pages for training. ChatGPT search still reaches you through OAI-SearchBot.
- Google-Extended is disallowed: you have opted out of Gemini training and grounding. This does not affect Search or AI Overviews.
GPTBotTrainingBlockedDisallow: /OAI-SearchBotSearch indexAllowedAllow: /, via *ChatGPT-UserUser-triggeredAllowedAllow: /, via *ClaudeBotTrainingAllowedAllow: /, via *Claude-SearchBotSearch indexAllowedAllow: /, via *Claude-UserUser-triggeredAllowedAllow: /, via *PerplexityBotSearch indexAllowedAllow: /, via *Perplexity-UserUser-triggeredAllowedAllow: /, via *GooglebotIndex + AI OverviewsAllowedAllow: /, via *Google-ExtendedTraining + groundingBlockedDisallow: /BingbotIndex + CopilotAllowedAllow: /, via *Applebot-ExtendedTraining controlAllowedAllow: /, via *CCBotOpen datasetBlockedDisallow: /BytespiderTrainingAllowedAllow: /, via *Free. Runs in your browser, so nothing you paste is uploaded. Once crawlers can reach you, check whether engines actually cite you with the AI Visibility Checker.
The short answer
Which AI crawlers you should allow.
Which AI crawlers should I allow in robots.txt?
To be found in AI answers, allow the crawlers that build each engine's search results: OAI-SearchBot for ChatGPT, Claude-SearchBot for Claude, PerplexityBot for Perplexity, Bingbot for Copilot, and Googlebot for Google AI Overviews, which use the regular Google index. Blocking one of these removes you from that engine's search answers. GPTBot and ClaudeBot are training crawlers, and Applebot-Extended is a training control, so disallowing them keeps your content out of model training without taking you out of search answers. Google-Extended covers Gemini training and grounding in Gemini Apps; it does not affect Google Search or AI Overviews. This checker reads a robots.txt you paste and returns a plain allowed or blocked verdict for fourteen named crawlers, plus the exact rule behind each one. It is a guide, not a guarantee: a crawler still has to choose to visit, and user-triggered fetchers may not follow robots.txt at all.
crawlers
engines
signup
paste
What it checks
Fourteen crawlers, grouped by the engine they feed.
Each is matched to the most specific rule in your file, with Allow beating Disallow on an equal-length path, the way the crawlers themselves read it.
OAI-SearchBot for ChatGPT search, GPTBot for training, and ChatGPT-User for fetches a user triggers. OpenAI says each setting is independent of the others.
Claude-SearchBot for Claude's search, ClaudeBot for training, and Claude-User for fetches a user triggers. A blanket rule catches all three.
PerplexityBot builds the index and is not used for training. Perplexity-User fetches live and, per Perplexity, generally ignores robots.txt.
Googlebot powers both Search and AI Overviews, so blocking it costs you both. Google-Extended is a separate opt-out for Gemini training and grounding only.
Bingbot powers Bing and Copilot, and its results feed ChatGPT search, so it reaches further than its name suggests.
Applebot-Extended is a training control token, not a crawler. Disallowing it opts you out of training without touching Siri indexing.
CCBot (Common Crawl) and Bytespider (ByteDance) feed models across the field. Blocking them is a reach-versus-control trade-off.
A crawler follows the most specific user-agent group that names it, and only falls back to the wildcard group when nothing names it.
Within a group, the rule with the longest matching path wins, and Allow beats Disallow when two paths tie, wildcards and end-anchors included.
Test the whole site or one path. A page blocked by a deep Disallow can be invisible even when the homepage is open.
How to use it
Paste robots.txt, read the verdicts.
No account, no crawl. Paste the file at yourdomain.com/robots.txt and act on the crawlers you did not mean to block.
- Paste your robots.txt
- Open yourdomain.com/robots.txt, copy all of it, and paste it in. The checker reads what you paste; it does not fetch your domain.
- Set the path
- Leave it as a slash to test the whole site, or enter a specific path to see whether one section is reachable.
- Read the verdicts
- Each of fourteen crawlers is marked allowed or blocked, grouped by engine, with the exact rule that decided it.
- Fix the reds
- Remove the Disallow lines blocking crawlers you want, republish, and re-check. Then confirm the answers follow.
Allowing a crawler lets it read you; it does not force it to visit or to cite you. Access is the floor, not the finish.
What good looks like
Open to the answer engines, deliberate about training.
A healthy setup is not open to everything. It is a clear, intentional choice per crawler.
OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot and Bingbot can all reach your key pages, so every major engine can find and cite you.
Because AI Overviews ride the regular index, a stray Disallow on Googlebot quietly removes you from both Search and AI Overviews.
If GPTBot, ClaudeBot or Applebot-Extended is disallowed, it is a decision you made about training, and you know it does not remove you from search answers. If Google-Extended is, you know it also keeps Gemini Apps from grounding answers in your pages.
A blanket Disallow under User-agent star catches AI crawlers too. Healthy files scope blocks to real paths, not everything.
The pages you want cited are reachable at their own URL, not just the homepage, with no deep Disallow hiding them.
Questions
AI crawlers and robots.txt.
Which AI crawlers should I allow?
Allow the crawlers that build search answers: OAI-SearchBot for ChatGPT, Claude-SearchBot for Claude, PerplexityBot for Perplexity, Bingbot for Copilot, and Googlebot for Google AI Overviews. Blocking any of these removes you from that engine's search answers. GPTBot, ClaudeBot and Applebot-Extended control training, which is a separate choice that does not affect search answers. Google-Extended also controls grounding in Gemini Apps: blocking it keeps Gemini Apps from grounding answers in your pages, but does not take you out of Google Search or AI Overviews.
Does blocking GPTBot remove me from ChatGPT?
No. OpenAI documents GPTBot as its training crawler and says each crawler setting is independent. ChatGPT search uses OAI-SearchBot, so you can disallow GPTBot to keep your pages out of training and still appear in ChatGPT search answers, as long as OAI-SearchBot can reach you.
What is the difference between Googlebot and Google-Extended?
Googlebot is the crawler behind Google Search, and AI Overviews are built from that same index, so blocking Googlebot costs you both. Google-Extended is not a crawler at all. It is a control token that decides whether your content is used for Gemini training and grounding. Google says it does not affect inclusion or ranking in Search.
Will allowing AI crawlers hurt my normal SEO?
No. Allowing GPTBot, ClaudeBot or PerplexityBot has no effect on your Google rankings; they are separate crawlers. The one that matters for both is Googlebot, which you should keep allowed for Search and AI Overviews.
Do user-triggered fetchers follow robots.txt?
Not always. OpenAI says robots.txt rules may not apply to ChatGPT-User because a person started the request, and Perplexity says Perplexity-User generally ignores robots.txt. The checker still shows what your file says for them, but treat those verdicts as your stated preference, not a guarantee.
Why can't I just enter my domain?
A browser cannot read another site's robots.txt for security reasons, and this tool runs entirely in your browser with nothing sent to a server. So you paste the file in. That also lets you test a draft before you publish it.
Is the checker free?
Yes. It is free, needs no signup, and runs entirely in your browser, so nothing you paste is uploaded. Access is only the first gate, though. Prefer's paid product is the work of turning a reachable page into an actual citation.
Keep going
Get your free AI visibility report
in about 10 minutes.
See how answer engines describe your brand today, and where the openings are to outpace the competition.