> ## Documentation Index > Fetch the complete documentation index at: https://tryprefer.com/docs/llms.txt > Use this file to discover all available pages before exploring further. # Does Prefer track AI crawlers on my site? > Yes. Prefer's Agent Analytics reads your server or CDN logs to show which AI crawlers visit, which pages they miss, and how a page moves from crawl to citation. Yes. Prefer's Agent Analytics module reads your server or CDN logs, with no code on your site, and shows which AI crawlers fetch your pages, the status codes they get, which pages they never reach, and how a new page moves from published to cited. *Last updated 3 October 2026* Yes. Prefer tracks AI crawlers through its Agent Analytics module. Prefer reads the logs your server or CDN already keeps and shows which AI bots fetch your pages, what they get back, and which pages they never reach. A page an engine cannot fetch cannot be cited, so this is often the first thing to check. ## How it works: logs, not a script **Agent Analytics is log file analysis, so it sees bot traffic that never runs JavaScript, which is most AI crawler traffic.** There is no tag to install. That is the key difference from web analytics: a GA4 tag only fires in a browser, and crawlers like GPTBot and PerplexityBot do not behave like browsers. Prefer currently maps 8 crawlers, each to the assistant it serves: | Crawler | Type | | - | - | | OAI-SearchBot | Answer (ChatGPT search) | | ChatGPT-User | Answer (ChatGPT, on a user request) | | Claude-User | Answer (Claude, on a user request) | | PerplexityBot | Answer (Perplexity) | | Googlebot | Answer (Google search and its AI features) | | GPTBot | Training (OpenAI) | | ClaudeBot | Training (Anthropic) | | Google-Extended | Training (Google) | New crawlers are added as vendors ship them. The full profiles live in our [AI crawler directory](https://tryprefer.com/learn/ai-crawlers/). ## Answer crawlers and training crawlers are different **Only answer crawlers move today's AI answers; training crawlers shape the next model.** Answer crawlers fetch a page while an assistant is writing a reply, so a spike often comes right before a citation. Training crawlers harvest pages for future models. Agent Analytics lets you filter the whole module by crawler type, because one blended "bot traffic" number hides the half that matters for citations. ## What Agent Analytics shows * **Who crawls you:** volume, share, last-seen time and whether your robots.txt lets each bot in. * **Coverage:** every URL with the bots that fetched it, the status code they got, and whether your last edit has been re-crawled. * **The citation funnel:** pages move through 4 stages (published, discovered by any bot, fetched by an answer crawler, cited), and you can see where pages drop out. * **Issues:** blocked bots, 404s served to bots, and edits that no bot has re-read yet. Each issue routes to the Action Center as a fix. Technical work there covers crawler access and discovery files, such as allowing AI bots in robots.txt or publishing an llms.txt file. ## A quick check you can run today 1. Open your robots.txt and look for rules that name OAI-SearchBot, PerplexityBot or ChatGPT-User. 2. Test 3 key URLs with the free [crawler access checker](https://tryprefer.com/tools/crawler-access-checker/). 3. Fix any rule that blocks an answer crawler from a page you want cited. 4. Confirm the next fetch returns a 200 status in your logs. **Plan availability is not listed as its own row on /pricing, so confirm which plan includes Agent Analytics with the Prefer team before you rely on it.** The product details are on the [Agent Analytics](https://tryprefer.com/platform/agent-analytics/) page. To see whether AI engines already cite you, [run a free AI visibility audit](https://app.tryprefer.com/audit). ## Frequently asked questions No. Prefer's Agent Analytics reads your server or CDN logs directly. That matters because most AI crawlers do not run JavaScript, so a script-based tracker would miss them. Not directly, and Prefer reports the two separately. GPTBot and ClaudeBot are training crawlers that gather pages for future models. Answer crawlers such as OAI-SearchBot, ChatGPT-User and PerplexityBot fetch a page while an assistant writes an answer, so they arrive just before a citation. Not the answer crawlers. Prefer's guidance is that blocking them removes you from AI answers. Training crawlers are a separate choice, and Agent Analytics shows the traffic from each type so you can decide with numbers in front of you. ## Also asked as * Can Prefer show if GPTBot is crawling my site? * Does Prefer do AI bot log file analysis? * How do I see which AI bots visit my website in Prefer? ## Sources * [Prefer Agent Analytics](https://tryprefer.com/platform/agent-analytics/) * [Prefer Agent Analytics FAQ](https://tryprefer.com/platform/agent-analytics/) ## Related * [How do I check if ChatGPT is blocked from crawling my site?](https://tryprefer.com/learn/answers/is-chatgpt-blocked-from-my-site/) * [Should I block AI crawlers from my website?](https://tryprefer.com/learn/answers/should-i-block-ai-crawlers/) * [How often does ChatGPT recrawl my website?](https://tryprefer.com/learn/answers/how-often-does-chatgpt-recrawl-my-site/) * [Can AI engines read JavaScript-rendered content?](https://tryprefer.com/learn/answers/do-ai-engines-read-javascript/) * [Does Prefer connect to Google Search Console and GA4?](/docs/help/does-prefer-connect-to-google-search-console) * [Agent Analytics](https://tryprefer.com/platform/agent-analytics/) * [AI crawler directory](https://tryprefer.com/learn/ai-crawlers/) * [Free crawler access checker](https://tryprefer.com/tools/crawler-access-checker/) This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.