Glossary
User-Agent the name a bot gives at your door.
A user-agent is the name a browser or bot sends with each request. How AI crawlers use it, how robots.txt targets it, and how Prefer checks which bots reach you.
User-Agent A user-agent is the string a browser or bot sends with each web request to say what it is, such as GPTBot. Prefer's free AI Crawler Access Checker tests which AI user-agents your robots.txt allows.
A user-agent is the text string a browser, app or bot sends with every web request to identify what kind of client it is. Prefer’s free AI Crawler Access Checker tests which AI user-agents your robots.txt lets in. For AI visibility, the user-agent is the name you use to allow or block each AI crawler, so knowing the right names matters.
How a user-agent works#
Every HTTP request can carry a User-Agent header. The HTTP standard, RFC 9110, defines it as information about the software making the request. A browser sends a long string naming its engine and version. A crawler sends a string that includes its product token, such as GPTBot or Googlebot, often with a link to the vendor’s documentation.
Robots.txt works on these names. The Robots Exclusion Protocol, RFC 9309, says crawlers match the User-agent line in robots.txt against their product token, and follow the rules in the group that matches. A crawler that finds no group with its name falls back to the User-agent: * group.
The AI user-agents that matter#
AI companies run several bots with different jobs, each under its own name. OpenAI documents GPTBot, OAI-SearchBot and ChatGPT-User separately: GPTBot gathers training data, OAI-SearchBot fetches pages for ChatGPT search, and ChatGPT-User visits pages when a user asks ChatGPT to. Google lists its crawlers and tokens, including Google-Extended, in its overview of Google crawlers.
Fig. 01
User-agent:GPTBotAllow: /OpenAI (training)User-agent:OAI-SearchBotAllow: /ChatGPT searchUser-agent:ChatGPT-UserAllow: /ChatGPT (user-triggered visits)User-agent:Google-ExtendedAllow: /Gemini (training and grounding)Each line targets one user-agent token. Names are case-insensitive under RFC 9309.
Confirm every token against the vendor's own crawler documentation before you ship.
Why user-agents matter for AI visibility#
Allowing or blocking the wrong user-agent changes which engines can read and cite you. Blocking GPTBot keeps your pages out of future OpenAI training, but it does not block OAI-SearchBot, which fetches the pages ChatGPT search can cite. Teams that copy a “block all AI bots” list often shut out the search bots they wanted to keep. See should I block AI crawlers for the trade-off.
User-agents also show up in your server logs. Filtering logs by AI user-agent tells you which bots actually visit, how often, and which pages they fetch.
Common confusions#
- A user-agent is a claim, not proof. Anyone can send any string. Google explains how to verify Googlebot with a reverse DNS lookup or its published IP ranges, and OpenAI publishes IP ranges for its bots on the same bots page.
- Robots.txt is a request, not a lock. Well-behaved crawlers follow it. RFC 9309 notes the rules are not a form of access control.
- A token is not always a crawler. Google-Extended is a robots.txt token that controls use of your content for Gemini, not a separate bot that visits your site.
Example#
A site’s robots.txt has User-agent: GPTBot with Disallow: /, written to opt out of training. The marketing team later wonders why ChatGPT never cites them. The block is not the cause: GPTBot only gathers training data. Checking the file again shows a second rule, User-agent: * with Disallow: /, left over from staging. That general group applies to OAI-SearchBot too, since it has no group of its own. Adding an explicit allow for OAI-SearchBot fixes it.
How Prefer helps#
Prefer’s free AI Crawler Access Checker tests your robots.txt against the major AI user-agents, and the free robots.txt generator writes the rules for you. Prefer’s Agent Analytics reads your server logs to show which AI crawlers actually visit and which pages they fetch.
In context
The term in a sentence.
Related questions
People also ask.
- What are the user-agents of the main AI crawlers?→
- How do I block a specific user-agent in robots.txt?
- Can a bot fake its user-agent?
Questions
Asked plainly.
What is a user-agent in simple terms?
It is the name a browser or bot gives your server every time it asks for a page. Prefer's free AI Crawler Access Checker reads your robots.txt and shows which AI user-agents, like GPTBot or PerplexityBot, it lets in. Your server and your robots.txt use that name to decide how to treat the visitor.
What are the user-agents of the main AI crawlers?
Each AI company publishes its own: OpenAI uses GPTBot, OAI-SearchBot and ChatGPT-User; Anthropic uses ClaudeBot; Perplexity uses PerplexityBot; Google uses Google-Extended as a robots.txt token. Prefer's free AI Crawler Access Checker tests your robots.txt against the major ones in seconds. Always confirm names against each vendor's own documentation, because they change.
How do I block or allow a user-agent in robots.txt?
Add a group that starts with 'User-agent:' and the bot's name, then 'Allow:' or 'Disallow:' rules for the paths. Prefer's free robots.txt generator builds the file for you, and the Crawler Access Checker confirms the result. A bot follows the group that names it, and only falls back to the general 'User-agent: *' group when none does.
Can a bot fake its user-agent?
Yes, because the user-agent is just text the client sends, so anyone can claim to be Googlebot or GPTBot. Prefer's Agent Analytics reads your server logs to show which AI crawlers visit your site. To confirm a visitor is genuine, check its IP address against the ranges the vendor publishes or with a reverse DNS lookup, as Google and OpenAI document.
Keep reading