8 Best AI Crawler Analytics Tools in 2026, Compared
Prefer's Agent Analytics ties every AI crawler visit to a citation. 8 AI crawler analytics tools compared: Cloudflare, Known Agents, Vercel, Profound and more.
Prefer is the best AI crawler analytics tool if you want to know whether a crawl turned into a citation: its Agent Analytics reads your server logs, names each AI crawler, separates answer traffic from training traffic, lists the pages AI has never fetched, and follows each page from published to cited, on plans from $24/mo. Known Agents (formerly Dark Visitors) is the best standalone option, with a free plan, 2,000+ named agents and AI chat referral tracking. If your site is behind Cloudflare, AI Crawl Control is free on every plan. Otterly ($189), PromptWatch ($245), Qwairy ($519) and Profound (custom) sell the same idea as a tier inside a visibility platform.
Our pick (our product)PreferCrawl data tied to citations, plus the fix
Best dedicatedKnown AgentsFree plan, 2,000+ agents, any host
Best on CloudflareCloudflare AI Crawl ControlEvery request at the edge, all plans
DisclosurePrefer publishes this list and ranks itself first. We did not open accounts with the other seven tools. Every competitor fact comes from the vendor's own documentation or pricing page, read on the date given in each entry, and each entry names where that tool beats ours.
In short
The takeaways, up front.
AI crawlers read raw HTML and do not run JavaScript, so Google Analytics and every tag-based tool miss them. Crawler analytics has to come from server logs or from the edge (CDN or host).
The crawlers have different jobs. GPTBot and ClaudeBot collect training data; OAI-SearchBot, Claude-SearchBot and PerplexityBot fetch pages for live answers. A tool that blends them into one bot count hides the half that produces citations.
Cloudflare's managed robots.txt block covers GPTBot, ClaudeBot and Google-Extended but not OAI-SearchBot or PerplexityBot; a plain User-agent: * disallow blocks all of them. Check the rule, not the intention.
Edge tools only see the sites they serve: Cloudflare AI Crawl Control needs a Cloudflare zone and Vercel's bot data needs a Vercel deployment. Log-based tools (Prefer, Known Agents) work on any host.
Inside AI-visibility platforms, crawler analytics is a mid or top tier feature: Otterly at $189, PromptWatch at $245, Qwairy at $519, Profound on custom Enterprise (checked 22 August to 26 September 2026).
How we evaluated
Four criteria. Nothing else counted.
Prefer's Agent Analytics reads your server logs, names each AI crawler, flags blocked pages and ties every fetch to whether the page was cited, on plans from $24/mo. Of the 8 tools here, Known Agents is the best standalone option (free plan, 2,000+ agents), Cloudflare AI Crawl Control is free on any plan, and Otterly, PromptWatch, Qwairy and Profound sell crawler modules inside a platform.
01
It names the crawler, not just the bot
GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot and Google-Extended have different jobs. A useful tool reports each one, with its operator and whether it trains a model or fetches pages for live answers.
02
It shows what was blocked and what errored
A 403 or a 404 served to OAI-SearchBot is a citation you cannot win. We looked for per-crawler status codes or an allowed-versus-blocked view, and for robots.txt violation reporting.
03
It goes down to the page
The pages an AI crawler never fetched are the pages that cannot be cited. We looked for a per-URL view, not only a top-paths table.
04
It connects the crawl to what happened next
Visits referred from ChatGPT or Perplexity, or better, the citation itself. Crawl counts alone do not tell you whether the work landed.
05
It works where your site lives
Edge tools only see the zones or deployments they serve. Log-based tools work on any host. We note which is which, and what each entry tier costs.
The scores
Every tool, the same five numbers.
Each dimension is scored 0 to 5 on the scale below and the overall is the plain average, so you can check every number against the facts on this page. Prefer is our product and is scored on the same scale.
Tool
Named crawlers
Blocked and errors
Page-level view
Crawl to answer
Any host
Free to start
Overall
Known Agents
5.0
5.0
5.0
3.0
5.0
5.0
4.7/5
Prefer (our product)
5.0
5.0
5.0
5.0
5.0
2.0
4.5/5
Cloudflare AI Crawl Control
5.0
5.0
3.0
3.0
1.0
5.0
3.7/5
Vercel Bot Management
3.0
3.0
3.0
1.0
1.0
5.0
2.7/5
How each number is set
Named crawlers: reports each AI crawler by name with its operator or category = 5; groups requests by user agent string or by rule on the entry plan, with named bots only on a paid add-on = 3; one blended bot count = 1.
Blocked and errors: status codes or allowed versus blocked per crawler = 5; blocked traffic shown by rule or in aggregate only = 3; not documented = 1.
Page-level view: every page with the crawlers that fetched it, or a per-page breakdown = 5; a top-paths or grouped-paths table only = 3; not documented = 1.
Crawl to answer: ties a page's crawls to its citations in AI answers = 5; tracks visitors referred from AI answers (on any plan) = 3; neither documented = 1.
Any host: reads logs or connectors from any host = 5; only sites on the vendor's own network = 1.
Free to start: real crawler analytics on a free plan = 5; a free trial only = 2; no free plan or trial = 1.
Scored 26 September 2026 from the facts on this page, which come from each vendor's documentation as dated in its entry. Only the four tools whose crawler features are documented feature by feature are scored: Prefer, Cloudflare AI Crawl Control, Known Agents and Vercel. Profound, Otterly, PromptWatch and Qwairy are not scored, because the pages we read document the module as a plan inclusion and, for Profound, as three feature names, not what it shows per crawler or per page; all four stay in the ranked list on their documented tier and price. Prefer's row is scored on the same scale from its own module page, and its published facts do not document AI referral tracking or a free plan, so it does not get those points.
Three kinds of tool
They are not all the same product.
CDN and host controls
Cloudflare AI Crawl Control, Vercel Bot Management
Built into the network your site already sits behind. They see every request without a tag, name verified bots, and can block a crawler in one click. They only cover sites on that network, and the deeper views (referrals, bot names in queries) sit on paid plans.
Buy when your site is already on Cloudflare or Vercel and you mainly want to see and control bot traffic.
Dedicated agent analytics
Known Agents (formerly Dark Visitors)
A product whose whole job is bot and agent traffic: a directory of 2,000+ agents, per-agent and per-page visits with status codes, spoof checks, spike alerts, AI chat referral tracking and an automatic robots.txt. Host-agnostic through connectors or an API, metered by events.
Buy when you are on any other host, or when you want the widest agent roster and referral data on a free plan.
AI-visibility platforms with a crawler module
Prefer, Profound, Otterly, PromptWatch, Qwairy
The crawler view lives next to prompt tracking and citations, so the crawl can be read against the answer. Prefer ties each page's crawls to its citations; the others document the module as a tier inclusion (Otterly Standard, PromptWatch Professional, Qwairy Business, Profound Enterprise).
Buy when you already track AI answers and want the crawl data in the same place, or when the point is finding out whether the crawl became a citation.
The ranked list
Every tool worth shortlisting.
Prefer first (our product, disclosed). Then tools that see every request (edge controls, a dedicated agent-analytics product), then visibility platforms with a crawler module, by tier price.
01
Prefer
tryprefer.com
Crawl to citation (ours)Plans from $24/mo, 7-day free trial
Agent Analytics is Prefer's crawler module. It is log file analysis, so it needs no code on your site and sees the bot traffic that never runs JavaScript. It tracks eight AI crawlers today (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, PerplexityBot, Googlebot and Google-Extended), each mapped to the assistant it feeds, with volume, share, last-seen time and whether your robots.txt lets it in. It separates answer crawlers (the bots that fetch a page while an assistant writes an answer) from training crawlers, lists every URL with the bots that fetched it and the status code they got, and shows a four-stage pipeline per page: published, discovered, answer-crawled, cited, with the median days between stages. Blocked bots, 404s served to crawlers, edits never re-crawled and pages never fetched each surface as an issue that routes to Action Center as a fix with a rollback point. It is our product.
Strengths
Answer crawlers and training crawlers reported separately, so you see the traffic that precedes a citation instead of one blended bot number
Per-page coverage: every URL with the bots that fetched it, the status code they got, and whether your last edit has been re-crawled
Citation pipeline per page (published, discovered, answer-crawled, cited) with median time to first crawl and first citation
Crawler problems (blocked bot, 404 to a bot, stale edit, page never fetched) become one-click fixes in Action Center
Limitations
Our published plan facts do not state which plan Agent Analytics starts on; ask before you buy on it
No report of visitors referred from AI answers is documented; Known Agents and Cloudflare (paid) have one
Eight crawlers tracked today; Known Agents lists 2,000+ agents and Cloudflare covers every verified bot on your zone
We are the vendor here; weigh this entry accordingly
VerdictOur pick if the question is whether AI can read you and whether that turned into a citation. If you only want to count and block bots, the free tools below do that job.
02
Cloudflare AI Crawl Control
cloudflare.com
Edge control, freeAll Cloudflare plans, including Free
AI Crawl Control (launched as AI Audit in September 2024, renamed July 2025) shows total and allowed requests per AI crawler, which operators are crawling (OpenAI, Anthropic, ByteDance and the rest), a breakdown by category (AI Crawler for training, AI Search, AI Assistant), the most requested paths, status code distribution and data transfer. Each crawler can be allowed or blocked from the Actions column, and a Directives tab lists crawlers that requested paths your robots.txt disallows. It is available on all plans; referral metrics, a custom 403 or 402 block response and Bot Management detection are paid-plan features, and on the Free plan crawlers are identified by user agent string. A managed robots.txt that blocks eight AI crawlers is one setting away, and pay per crawl (HTTP 402 with a price) is in closed beta. Docs read 26 September 2026.
Strengths
Sees every request without a tag or log export, with per-crawler and per-operator counts and allowed versus total requests
Robots.txt violations table names the crawlers that fetched disallowed paths; managed robots.txt and per-crawler block are one click
Available on all plans, including Free
Limitations
Only for sites on Cloudflare; no log import for other hosts
Free plan detection is user-agent based; deeper detection, referral metrics and custom block responses need a paid plan (plan prices not verified)
Measures crawl and referral traffic, not what the engines answer or cite
VerdictIf your DNS already points at Cloudflare, turn it on today. It will not tell you whether a crawl became a citation. Visit site →
Dark Visitors renamed itself Known Agents; darkvisitors.com now redirects there and existing integrations keep working. It started as a single page listing the user agents of the eight AI bots that existed at the time, and now tracks 2,000+ crawlers, scrapers and AI agents across 15+ categories, with a public directory (AI Data Scraper, AI Search Crawler, AI Assistant, AI Agent and more) and a page per agent. Data arrives through connectors for Cloudflare, AWS, Google Cloud, Vercel, Fastly, Akamai, WordPress and Node.js, or from your backend through a REST API. The dashboard breaks visits down by agent, category and page, shows HTTP status codes per agent (200, 403, 500), verifies bots that claim a trusted identity, and alerts on spikes. AI Chat Referral Tracking counts human visitors from ChatGPT, Claude, Gemini and other AI chat platforms, and an automatic robots.txt keeps exclusion lists current. Free is $0 for 1 project and 100K events a month; Startup is $29 (3 projects, 1M events, CSV export, API); Growth is $299 (10 projects, 10M events); Enterprise is custom. Site read 26 September 2026.
Strengths
Works with any host through nine named connectors or a plain API call, so it is not tied to one CDN
Per-agent status codes, spoof verification and spike alerts, with a public directory that explains what each agent is
AI chat referral tracking and automatic robots.txt in the same tool, on the free plan
Limitations
Every visit event and API call counts against a metered allowance (100K a month free)
CSV export, the identification API and team seats start at $29; the Reporting API is Enterprise only
Counts bot visits and AI referrals; it does not measure what the engines say or which pages they cite
VerdictThe best standalone crawler analytics product on this list, and the one to pick if you are not on Cloudflare or Vercel and do not need citation data. Visit site →
04
Vercel Bot Management
vercel.com
Edge control, Vercel$0 Hobby; Observability Plus on Pro ($20/mo + $1.20 per 1M events)
Vercel's AI Bots managed ruleset controls traffic from AI bots that crawl for training, search or user-initiated fetches, with a Vercel-maintained list that includes GPTBot, ClaudeBot, PerplexityBot and Bytespider; it is inactive by default and free on all plans since May 2025. In Log mode it records AI bot traffic without blocking, which the docs call useful for monitoring. The verified bots directory labels gptbot, claudebot and google-extended as ai_crawler and oai-searchbot, perplexitybot, claude-searchbot and chatgpt-user as ai_assistant, noting that OAI-SearchBot does not crawl for training. The Firewall Traffic page groups requests by user agent and request path for up to 24 hours; querying by bot name, category and verified versus spoofed needs Observability Plus, which is $1.20 per 1M events on the $20 Pro plan and not available on Hobby. Docs and pricing read 26 September 2026.
Strengths
AI Bots ruleset with Log or Deny in one toggle, and an auto-updated bot list, on every plan
Verified bots directory separates training crawlers from search and assistant fetchers and marks spoofed traffic
Limitations
Only for sites deployed on Vercel; the Traffic page shows at most 24 hours without Observability Plus
The ruleset is Log or Deny for the whole list; per-bot rules need custom firewall rules, capped at 3 on Hobby
No robots.txt compliance report, no referral attribution from AI answers, no per-crawler requests table documented
VerdictA firewall with an AI bot view, not a crawler analytics product. Fine for seeing and blocking bots on Vercel; pair it with a log-based tool for the rest. Visit site →
05
Profound
tryprofound.com
Enterprise moduleFree trial, then Enterprise (custom)
Agent Analytics is one of Profound's five modules, described on its features page as crawler monitoring, AI traffic attribution and technical optimization (read 23 September 2026). Our August 2026 research recorded server-log bot tracking with IP verification and CDN hooks for Cloudflare, Vercel and Fastly; the pricing page lists Agent Analytics integrations for Akamai, AWS, Cloudflare, Fastly, Google Analytics, GCP, Netlify, Vercel and WordPress. For brands Profound publishes a free trial (50 prompts run daily for 7 days on ChatGPT, Gemini and Google AI Overviews) and a custom Enterprise plan; no brand price is published (rendered 23 September 2026).
Strengths
Crawler tracking with IP verification and nine named integrations, from Akamai to WordPress
AI traffic attribution next to the answer-engine data in the same platform
Limitations
Custom Enterprise pricing only; the free trial is a prompt-tracking trial, not a crawler analytics trial
Per-crawler, per-page and blocked-request views are not documented on the pages we read, so we do not score them
VerdictThe pick if you are already buying Profound. Nobody should buy an Enterprise platform for the crawler module alone. Compare vs Prefer →
06
Otterly
otterly.ai
Cheapest platform moduleAgent Analytics from $189/mo (Standard)
Otterly's Standard plan is $189 a month for 100 prompts and adds API and MCP access and Agent Analytics; Lite at $29 (15 prompts, four engines) does not include it, Premium is $489 for 400 prompts, and Enterprise starts at 1,000 prompts (pricing rendered 23 September 2026). What the module shows per crawler or per page is not documented on the pages we read, so we list the tier and stop there.
Strengths
The lowest published price on this list for a crawler module inside a prompt-tracking platform
Standard also brings API and MCP access, so the crawler data can leave the tool
Limitations
Not on the $29 Lite plan
Which crawlers, pages and status codes it reports is not documented on the pages we read
VerdictBuy Standard for the monitoring and treat Agent Analytics as a bonus until you have seen it in the trial. Compare vs Prefer →
07
PromptWatch
promptwatch.com
Platform module, meteredAgent Analytics from $245/mo (Professional)
PromptWatch's Professional plan is $245 a month for 150 prompts, 5 AEO articles and Agent Analytics; Essential at $95 (50 prompts, 4 models, 500 agent credits) does not include it, and Business is $579 for 350 prompts (pricing rendered 23 September 2026). Its crawler log analysis is metered by rows, 200K to 100M by tier (checked 22 August 2026). Per-crawler and per-page views are not documented on the pages we read.
Strengths
Crawler logs and AEO article generation on the same plan
7-day free trial; MCP and API access from the $95 Essential plan
Limitations
Not on the $95 Essential plan; log rows are capped by tier (200K to 100M)
What the module reports per crawler or page is not documented on the pages we read
VerdictReasonable if you are buying Professional for the articles. Compare the row caps with your real log volume first. Compare vs Prefer →
08
Qwairy
qwairy.co
Platform module, Business tierCrawler Analytics from $519/mo (Business)
Qwairy's Business plan is $519 a month ($432 on annual billing) for 20 workspaces, 10,400 credits and up to 800 prompts, and it is the tier that adds Crawler Analytics and a Looker Studio connector; Starter is $89 and Growth $229 without it (pricing rendered 26 September 2026). Its plan comparison groups AI Crawler Analytics under Measure with Google Search Console, Bing Webmaster Tools, Referrer Analytics, Page Performance and AI Revenue, and every plan covers 10 AI engines. Which plan the referrer analytics start on, and what the crawler view shows, are not documented on the page we read.
Strengths
Crawler analytics sits with Search Console, Bing, referrer and revenue data in the same Measure group
Unlimited team members and 10 engines on every plan
Limitations
Crawler Analytics is a $519 Business feature; Starter ($89) and Growth ($229) do not list it
Module detail not documented on the pricing page we read
VerdictOnly makes sense at 20 workspaces. Below that, a free edge tool plus the Growth plan costs a third as much. Visit site →
Side by side
The whole field, one table.
Capability at the checked-on date, for the four tools whose crawler features are documented feature by feature. Partial means the tool does something adjacent, or only on a paid plan. A dash means the vendor's public pages we read do not document it, and that includes our own module.
Capability comparison across the tools
Tool
Per-crawler requests
Allowed vs blocked
Per-page detail
Referral from AI answers
robots.txt control
From
Prefer
✓
✓
✓
-
✓
$24
Cloudflare AI Crawl Control
✓
✓
~
~
✓
Free
Known Agents
✓
✓
✓
✓
✓
Free
Vercel Bot Management
~
~
~
-
-
$0
FullPartialNot offered
What changed in 2026
Four shifts that reshaped the category.
This list looked different a year ago. These are the changes that moved it.
01
The CDNs built it in
Cloudflare launched AI Audit in September 2024, renamed it AI Crawl Control in July 2025 and took per-crawler analytics to general availability that August, on all plans. Vercel shipped a one-click AI bots ruleset in May 2025, free on all plans. If your site is on either network, basic crawler analytics is already paid for.
02
Tools now label crawlers by job
Cloudflare groups verified bots into AI Crawler (training), AI Search and AI Assistant. Vercel's directory marks gptbot and claudebot as ai_crawler and oai-searchbot, claude-searchbot and perplexitybot as ai_assistant. Known Agents separates AI Data Scraper, AI Search Crawler, AI Assistant and AI Agent. Prefer splits answer crawlers from training crawlers. A bot count with no job label is now behind.
03
Referral tracking arrived next to crawl counts
Cloudflare added referrer analytics in October 2025 (paid plans) and referral trends in February 2026. Known Agents tracks human visitors from ChatGPT, Claude, Gemini and other chat platforms on its free plan. Profound lists AI traffic attribution in its Agent Analytics module. The crawl is the input; the visit or the citation is the output.
04
Dark Visitors became Known Agents
The best-known standalone tool renamed itself because people heard dark as bad rather than hidden. It began as one page listing eight AI bot user agents; its directory now runs to 2,000+ agents across 40 pages, and its homepage claims half of web traffic is no longer human. Old integrations keep working.
05
Crawling is starting to carry a price
Cloudflare's pay per crawl, in closed beta, answers an AI crawler with HTTP 402 and a price unless it presents payment intent, with Cloudflare as merchant of record. The tooling to charge for crawls and the tooling to invite them are converging in the same dashboards.
Red flags
Things to check before you sign.
Every tool here demos well. These are the questions that separate them once the trial ends.
Blocking search crawlers by accident
Cloudflare's managed robots.txt disallows GPTBot, ClaudeBot, Google-Extended and five other AI crawlers, and leaves OAI-SearchBot and PerplexityBot alone. A plain User-agent: * with Disallow: / blocks all of them, and takes you out of the answers. Run your file through the free crawler access checker before you trust a block list.
User-agent-only detection
Any script can claim to be GPTBot. OpenAI publishes IP ranges at openai.com/gptbot.json so a visit can be checked; Known Agents documents spoof verification, Vercel separates verified from spoofed bots, and Cloudflare's Free plan identifies crawlers by user agent string with deeper detection on paid plans. Ask what a tool does when the string and the IP disagree.
Bots counted, not named
A single bot-traffic number hides the split that matters: training crawlers shape the next model, answer crawlers arrive right before a citation. If the report cannot show OAI-SearchBot and GPTBot as separate lines, it cannot tell you which half moved.
A window measured in hours
Vercel's Firewall Traffic page defaults to the past hour and stops at 24 hours; longer analysis needs Observability Plus on a paid plan. Crawl patterns take weeks to read. Check the retention on the plan you would buy.
Analytics for one network only
Cloudflare AI Crawl Control needs a Cloudflare zone and Vercel's bot data needs a Vercel deployment, with no log import for other hosts documented by either. If your sites span hosts, you need a log-based tool.
An AI crawler analytics tool shows which AI bots fetch your pages, what they got, what was blocked, and, in the better tools, what happened next. Prefer’s Agent Analytics ranks first on this list because it is the only tool here that documents the last step: each page’s crawls tied to whether the page was cited. The other seven split into three kinds. Cloudflare and Vercel see every request at the edge of their own networks. Known Agents (formerly Dark Visitors) is a dedicated product that works on any host. Otterly, PromptWatch, Qwairy and Profound sell a crawler module as a tier inside an AI-visibility platform.
One disclosure up front: Prefer is our product. We did not open accounts with the other seven tools. Every fact below comes from the vendor’s own documentation or pricing page, read on the date in each entry, and each entry names where the rival beats us.
AI crawlers read the raw HTML your server returns and do not run JavaScript, so Google Analytics and every other tag-based tool never see them. The data has to come from one of two places: your server or CDN logs, which is how Prefer and Known Agents work, or the edge itself, which is how Cloudflare and Vercel work. That is also why two of the three kinds of tool on this page are tied to a host. Cloudflare AI Crawl Control needs a Cloudflare zone and Vercel’s bot data needs a Vercel deployment; neither documents a way to import logs from elsewhere.
Two columns. Your analytics tag sees, runs in a browser: Human visitors, including people referred from an AI answer, Pages after JavaScript has run, Nothing from GPTBot, OAI-SearchBot or PerplexityBot. Logs and the edge see, every http request: Each crawler by user agent, with the IP behind it, The status code the bot got: 200, 403, 404, Which pages were fetched and which never were.
AI crawlers do not run JavaScript, so tag-based analytics miss them entirely. Crawler analytics is a server-log or edge measurement. Conceptual comparison.
The second thing a tool has to get right is the split between crawlers. GPTBot and ClaudeBot collect pages that may train future models; OAI-SearchBot, Claude-SearchBot and PerplexityBot fetch pages so an assistant can answer and cite them live. Google-Extended is a robots.txt token that governs Gemini training and grounding without touching Google Search. The vendors now label this split themselves: Cloudflare’s categories are AI Crawler, AI Search and AI Assistant; Vercel’s directory marks gptbot as ai_crawler and oai-searchbot as ai_assistant, with a note that OAI-SearchBot does not crawl for training; Known Agents separates AI Data Scraper from AI Search Crawler. Prefer reports answer crawlers and training crawlers as two lines, because a spike in the first usually precedes a citation and a spike in the second does not change today’s answers.
A robots.txt allow-list permitting 7 AI crawlers, each with an Allow check: GPTBot (OpenAI, training), OAI-SearchBot (ChatGPT, live answers and citations), ChatGPT-User (ChatGPT, user-initiated fetch), ClaudeBot (Anthropic, training), Claude-SearchBot (Claude, live answers), PerplexityBot (Perplexity, live answers), Google-Extended (Gemini, training and grounding (robots token)). Roles as documented by OpenAI, Vercel's verified bots directory and Cloudflare's bot categories, September 2026. Blocking a training crawler does not block the search crawler from the same company, and the reverse.
A robots.txt allow-list for the crawlers a good analytics tool should name, with the job each one does. Allowing is necessary, not sufficient, for a citation; a blended bot count hides which half moved. Check what your robots.txt allows
This is where blocking goes wrong. Cloudflare’s managed robots.txt disallows eight AI crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider and three more) and leaves the search crawlers alone, which is the sensible default. A hand-written User-agent: * with Disallow: / blocks OAI-SearchBot too, and Cloudflare’s own docs note that robots.txt compliance is voluntary either way. Prefer’s free crawler access checker reads your file and returns an allowed or blocked verdict for 14 named crawlers, and the robots.txt generator writes one that keeps the answer crawlers in. Whether to block at all is its own question, answered at should I block AI crawlers?
Where the crawler modules sit inside visibility platforms#
Four of the eight tools sell crawler analytics as a plan inclusion rather than a product, so the useful comparison is the tier and its price. Otterly is the cheapest published entry at $189 a month, Profound has no published brand price at all. For these four, the pages we read document the tier and, in Profound’s case, three feature names (crawler monitoring, AI traffic attribution, technical optimization), not what the view shows per crawler or per page, so they stay out of the capability table and the scores above.
Tool
Tier with the crawler module
Price
What the vendor documents
Checked
Otterly
Standard (100 prompts)
$189/mo
Agent Analytics, API and MCP access; not on $29 Lite
23 Sep 2026
PromptWatch
Professional (150 prompts)
$245/mo
Agent Analytics; crawler logs metered 200K to 100M rows by tier; not on $95 Essential
23 Sep 2026, rows 22 Aug 2026
Qwairy
Business (up to 800 prompts, 20 workspaces)
$519/mo
Crawler Analytics and Looker Studio; Starter $89 and Growth $229 do not list it
26 Sep 2026
Profound
Enterprise
Custom
Crawler monitoring, AI traffic attribution, technical optimization; integrations from Akamai to WordPress
Counting crawls is the start. The reason to count them is to find the pages AI cannot read and fix them. Prefer’s Agent Analytics follows every page through four stages, published, discovered, answer-crawled and cited, and reports the median days between them, so a page stuck at “discovered” is a page only training bots have seen. Each failure it surfaces (a bot blocked in robots.txt, a 404 served to a crawler while people see the page, an edit the bots have not re-fetched, a new page no bot has visited) routes to Action Center as a fix with a rollback point.
A 4-step path. 1. undefined (Live on your site, in the sitemap, linked internally.); 2. undefined (Any AI bot fetched it. Often a training crawler first.); 3. undefined (OAI-SearchBot, PerplexityBot or Claude-SearchBot read it while writing an answer.); 4. undefined (The page appears as a source in an AI answer. The stage a crawl count alone cannot show.).
The pipeline Prefer's Agent Analytics reports per page. Known Agents and Cloudflare (paid plans) report the visit that follows a citation instead; Vercel reports the crawl only. Conceptual summary.
The other tools stop earlier, honestly and by design. Known Agents and Cloudflare’s paid plans report the human visits that AI answers refer to your site, which is the outcome one step past the citation, and neither documents the citation itself. Vercel reports the crawl and the rule that applied. If you already run answer monitoring, the crawl data belongs next to it; our AI search monitoring tools list covers that half.
Summary graphic of 4 items: 1. Where is the site hosted?: Cloudflare or Vercel: their built-in view is free. Anywhere else, or across hosts: a log-based tool. 2. Count, visit, or citation?: Counts everywhere. Referred visits on Known Agents and Cloudflare paid. Citations per page on Prefer. 3. Already on a platform?: Check your tier: Otterly $189, PromptWatch $245, Qwairy $519, Profound Enterprise. 4. Block or be found?: Edge tools are strongest at blocking. Visibility tools turn a blocked bot into a fix.
The four buying questions for AI crawler analytics tools. Conceptual summary.
Already on Cloudflare: turn on AI Crawl Control today.On Vercel: set the AI Bots ruleset to Log and read the Traffic page.On any other host, or with no answer tracking yet: Known Agents, free.Already paying for Otterly, PromptWatch, Qwairy or Profound: use the module on your tier before buying anything.Want to know whether the crawl became a citation, and have the fix shipped: Prefer, ours, on plans from $24 a month with a 7-day free trial. The honest caveat is that our published plan facts do not state which plan Agent Analytics starts on, and our module does not document AI referral tracking, which two of the free tools do.
For the wider field of tools that measure the answers rather than the crawls, see the best AI visibility tools.
People also ask
The questions buyers ask next.
A tool that reads the requests AI crawlers make to your site, from server logs or at the CDN edge, and reports which bots came (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and the rest), what they fetched, what was blocked or errored, and, in some tools, which AI answers later sent visitors. Prefer's Agent Analytics goes one step further and ties each page's crawls to whether it was cited.
No, and that is why Prefer's Agent Analytics is log file analysis. AI crawlers read the raw HTML your server returns and do not run JavaScript, so the analytics tag never fires for them; Cloudflare and Vercel see the requests at the edge before any tag runs. Google Analytics does see human visitors referred from ChatGPT or Perplexity, which is a different measurement.
At minimum the search crawlers that fetch pages for live answers: OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot and Bingbot. Prefer's free robots.txt generator writes the rules and the crawler access checker tests the file you already have. Whether you allow the training crawlers (GPTBot, ClaudeBot, Google-Extended) is a separate choice.
Prefer's GPTBot guide shows how to check by hand: match the user agent string, which for GPTBot includes a version number and a link to OpenAI's crawler page, and the request IP against the ranges OpenAI publishes at openai.com/gptbot.json. A GPTBot user agent from an IP outside those ranges is a spoof. Known Agents, Vercel and Cloudflare's paid plans do this check for you.
What is the best AI crawler analytics tool in 2026?
Prefer, our product and our pick, if you want the crawl tied to the citation: Agent Analytics reads your server logs, names each AI crawler, separates answer traffic from training traffic, lists the pages AI has never fetched, and follows each page from published to cited, on plans from $24/mo. For a standalone tool, Known Agents (formerly Dark Visitors) has a free plan and 2,000+ named agents. On Cloudflare, AI Crawl Control is free on every plan.
How do I see which AI crawlers visit my site?
Read your server logs or your CDN's bot view, not your analytics tag, because AI crawlers do not run JavaScript. Prefer's Agent Analytics reads the logs for you and lists GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, PerplexityBot, Googlebot and Google-Extended with volume, last-seen time and status codes. On Cloudflare, open AI Crawl Control; on Vercel, set the AI Bots ruleset to Log and open the Firewall Traffic page; on any host, Known Agents accepts logs through a connector or its API.
Should I block AI crawlers?
Usually no. Prefer's answer to this question is that blocking search crawlers such as OAI-SearchBot or PerplexityBot removes you from the AI answers buyers read, and does not erase what a model already learned. Block specific paths you must protect, keep public pages open, and treat training crawlers (GPTBot, ClaudeBot) as a separate choice. The full reasoning is in our answer page, Should I block AI crawlers from my site?
What is the difference between GPTBot and OAI-SearchBot?
Both belong to OpenAI, and they have different jobs. Prefer's Agent Analytics reports the two as separate lines, training and answer. GPTBot collects public pages that may train future models. OAI-SearchBot fetches live pages so ChatGPT can answer and cite them; Vercel's directory notes it does not crawl for training. Blocking one does not block the other, so a robots.txt line aimed at GPTBot leaves your ChatGPT citations intact, and blocking OAI-SearchBot removes them.
Can I see AI crawlers in Google Analytics?
No, and Prefer's Agent Analytics uses log file analysis for exactly that reason. Google Analytics runs as JavaScript in a browser, and AI crawlers read raw HTML without running scripts, so they never register. The edge tools from Cloudflare and Vercel count the requests before any tag runs. What Google Analytics can show is human visitors who arrived from an AI answer, which Known Agents and Cloudflare's paid plans also report.
How much do AI crawler analytics tools cost?
Prefer's plans start at $24/mo with a 7-day free trial, and our published facts do not state which plan Agent Analytics starts on, so ask. Cloudflare AI Crawl Control is included on every Cloudflare plan including Free. Known Agents is free for 100K events a month, then $29 and $299. Vercel's AI Bots ruleset is free; querying bot names needs Observability Plus at $1.20 per 1M events on the $20 Pro plan. Inside visibility platforms the module starts at $189 (Otterly Standard), $245 (PromptWatch Professional) and $519 (Qwairy Business); Profound is custom Enterprise. All checked between 22 August and 26 September 2026.
Is Dark Visitors the same as Known Agents?
Yes, Dark Visitors renamed itself Known Agents, and Prefer lists it under the new name. Its old domain, darkvisitors.com, redirects to knownagents.com, the agent pages carry the suffix formerly Dark Visitors, and the company says existing integrations keep working without changes. It remains the dedicated agent-analytics product on this list, with a free plan and a 2,000+ agent directory.
Answer a few questions and the shortlist writes itself.
01
Where is your site hosted?
On any host, or across several, Prefer reads your server or CDN logs, and Known Agents takes logs through a connector or its API. On Cloudflare, start with AI Crawl Control; it is free on every plan and sees every request. On Vercel, turn the AI Bots ruleset to Log and read the Traffic page, then decide whether Observability Plus is worth it.
02
Do you need to know what happened after the crawl?
If a visit count is enough, Known Agents (free) and Cloudflare (paid plans) report visitors referred from AI chat platforms. If you need to know whether the page was cited, Prefer is the only tool on this list that documents a per-page path from crawled to cited, and Profound lists AI traffic attribution on Enterprise.
03
Do you already pay for an AI-visibility platform?
Then check the tier. Prefer includes Agent Analytics as its crawler module (ask which plan it starts on); Otterly includes Agent Analytics from $189 Standard, PromptWatch from $245 Professional, Qwairy from $519 Business, and Profound on custom Enterprise. If the module is on your tier, use it before buying a second tool; if it is two tiers up, a free edge tool is cheaper.
04
Do you want to block bots or be found by them?
Blocking and firewall control is where Cloudflare and Vercel are strongest. If the goal is citations, the crawler data has to lead to a fix, and Prefer routes each one to Action Center: opening robots.txt to the answer crawlers, fixing a 404 served to a bot, getting a stale page re-crawled. On the other tools, the fix is yours to ship.
Get your free AI visibility report in about 10 minutes.
See how answer engines describe your brand today, and where the openings are to outpace the competition.
Cookie preferences
5 cookies across three groups. Necessary ones stay on. Change the rest any time from the footer.
Strictly necessaryAlways on
Sign-in, security, and remembering this choice. The site does not work without them.
1 cookie
Analytics
Anonymous page and report usage so we can see what to improve. Nothing is tied to you.
3 cookies
Marketing
Lets us credit the partner who sent you, so they are paid for the referral, and measure which campaigns bring people here. One cookie, no ad profiling.