Skip to main content
Yes. Prefer’s Agent Analytics module reads your server or CDN logs, with no code on your site, and shows which AI crawlers fetch your pages, the status codes they get, which pages they never reach, and how a new page moves from published to cited. Last updated 3 October 2026 Yes. Prefer tracks AI crawlers through its Agent Analytics module. Prefer reads the logs your server or CDN already keeps and shows which AI bots fetch your pages, what they get back, and which pages they never reach. A page an engine cannot fetch cannot be cited, so this is often the first thing to check.

​
How it works: logs, not a script

Agent Analytics is log file analysis, so it sees bot traffic that never runs JavaScript, which is most AI crawler traffic. There is no tag to install. That is the key difference from web analytics: a GA4 tag only fires in a browser, and crawlers like GPTBot and PerplexityBot do not behave like browsers. Prefer currently maps 8 crawlers, each to the assistant it serves: New crawlers are added as vendors ship them. The full profiles live in our AI crawler directory.

​
Answer crawlers and training crawlers are different

Only answer crawlers move today’s AI answers; training crawlers shape the next model. Answer crawlers fetch a page while an assistant is writing a reply, so a spike often comes right before a citation. Training crawlers harvest pages for future models. Agent Analytics lets you filter the whole module by crawler type, because one blended “bot traffic” number hides the half that matters for citations.

​
What Agent Analytics shows

  • Who crawls you: volume, share, last-seen time and whether your robots.txt lets each bot in.
  • Coverage: every URL with the bots that fetched it, the status code they got, and whether your last edit has been re-crawled.
  • The citation funnel: pages move through 4 stages (published, discovered by any bot, fetched by an answer crawler, cited), and you can see where pages drop out.
  • Issues: blocked bots, 404s served to bots, and edits that no bot has re-read yet.
Each issue routes to the Action Center as a fix. Technical work there covers crawler access and discovery files, such as allowing AI bots in robots.txt or publishing an llms.txt file.

​
A quick check you can run today

  1. Open your robots.txt and look for rules that name OAI-SearchBot, PerplexityBot or ChatGPT-User.
  2. Test 3 key URLs with the free crawler access checker.
  3. Fix any rule that blocks an answer crawler from a page you want cited.
  4. Confirm the next fetch returns a 200 status in your logs.
Plan availability is not listed as its own row on /pricing, so confirm which plan includes Agent Analytics with the Prefer team before you rely on it. The product details are on the Agent Analytics page. To see whether AI engines already cite you, run a free AI visibility audit.

​
Frequently asked questions

No. Prefer’s Agent Analytics reads your server or CDN logs directly. That matters because most AI crawlers do not run JavaScript, so a script-based tracker would miss them.
Not directly, and Prefer reports the two separately. GPTBot and ClaudeBot are training crawlers that gather pages for future models. Answer crawlers such as OAI-SearchBot, ChatGPT-User and PerplexityBot fetch a page while an assistant writes an answer, so they arrive just before a citation.
Not the answer crawlers. Prefer’s guidance is that blocking them removes you from AI answers. Training crawlers are a separate choice, and Agent Analytics shows the traffic from each type so you can decide with numbers in front of you.

​
Also asked as

  • Can Prefer show if GPTBot is crawling my site?
  • Does Prefer do AI bot log file analysis?
  • How do I see which AI bots visit my website in Prefer?

​
Sources