Blocked crawler81
Allow GPTBot and PerplexityBot in robots.txt
12 money-page URLs are invisible to the engines that read them
Product · Measure
See every AI crawler that visits your site.
Agent Analytics reads your server logs and shows which AI bots crawl your pages, which pages they never reach, and how long it takes a new page to go from published to cited.
Answer-time fetches690 ▲ 24%of 1,240 AI crawls this month
Answer crawlers69056%
Training crawlers55044%
The numbers
Four numbers, one question: can AI read you?
Every module below answers part of it: who crawls you, what they get, which pages they miss, and how fast a new page earns a citation.
AI crawls1,240 ▲ 18%
Total AI bot fetches of your site in the last 30 days.
Across 8 crawlersAnswer-time fetches690 ▲ 24%
Fetches by bots that read a page while answering. The signal before a citation.
56% of all AI crawlsPages reached86 ▲ 6%
Pages an AI crawler actually fetched, out of 142 indexed.
56 pages never crawledTime to first crawl3.2d ▼ 0.6d
Median days from publishing a page to the first AI bot fetch.
8 days to first citationLive demo data: Ledgerly, an expense management brand, June 10 to 16, from raw server logs.
Crawler traffic · Module 01
Not all AI bots are worth the same.
Answer crawlers fetch a page while an assistant is writing an answer, so they arrive right before a citation. Training crawlers harvest for the next model. Agent Analytics separates the two, so you know which traffic actually matters today.
Crawls over time
Daily AI bot fetches. Answer crawlers solid, training crawlers muted.
Answer crawlers5 bots69056%
OAI-SearchBot, PerplexityBot, Claude-User, ChatGPT-User, Googlebot. A spike here usually precedes a citation.
Training crawlers3 bots55044%
GPTBot, Google-Extended, ClaudeBot. They shape the next model, not today's answers.
Filter the whole product by crawler type. Most AI analytics tools report one blended bot-traffic number, which hides the only half that moves a citation.
The crawlers · Module 02
Every bot in your logs, and what it feeds.
Eight AI crawlers, each mapped to the assistant it serves, with volume, share, last-seen time and whether your robots.txt lets it in.
Crawlers reaching you
Every AI bot seen in your logs, and the assistant it feeds.
CrawlerFeedsTypeShareFetchesLast seenAccess
New crawlers appear as vendors ship them. When one is blocked, Agent Analytics tells you which pages it could not read.
Citation pipeline · Module 03
Follow a page from published to cited.
Every page moves through four stages: published, discovered by any bot, fetched by an answer crawler, then cited. The funnel shows how many pages make it and exactly where the rest drop out.
The citation pipeline
Where 218 published pages sit today.
−30never crawled
−48no answer crawl
−22not cited
8 daysmedian publish to citation54%of published pages get cited48 pagesthe biggest single leak4 pageslost a citation this month
One page, opened up
Month-end close guide/guides/month-end-close
- PublishedJun 2Live on ledgerly.co
- DiscoveredJun 4GPTBot fetched it first
- Answer-crawledJun 6OAI-SearchBot read it mid-answer
- CitedJun 8ChatGPT now cites it on 4 prompts
Click any stage in the app to filter straight to the pages stuck there.
Page coverage · Module 04
Find the pages AI has never read.
A page AI cannot fetch cannot be cited. Coverage lists every URL with the bots that fetched it, the status code they got, and whether your last edit has been re-crawled yet.
Page coverage
Every URL, the bots that fetched it, and what to do next.
Needs attention6Never crawled2No answer crawl2Crawled, not cited4Regressed1Cited4
PageCrawl stateFetched byCrawls 30dResponseWhy it is here · next step
Security & compliance/securityNever crawled–––Live 3 weeks, never fetched by any AI bot. Invisible to answers today.Submit URL →
Corporate card guide/blog/corporate-card-guideFetch error
3404Returns 404 to GPTBot. AI sees nothing at this URL.Inspect →
QuickBooks sync FAQ/integrations/quickbooks-syncAwaiting re-crawl
22200Edited 2 days ago, last crawled 5 days ago. AI is reading the old copy.Request re-crawl →
Ramp comparison page/compare/ramp-alternativeNo answer crawl12200Only training bots have fetched it, so it cannot be cited yet.Ping crawlers →
Month-end close guide/guides/month-end-closeFresh
64200Crawled daily and cited in ChatGPT. Healthy.View →
Xero integration/integrations/xeroFresh51200Cited in Perplexity since June 14.View →
Silent failures · Module 05
The failures nobody logs.
A blocked bot, a 404 or an unrecrawled edit costs you citations quietly. Nothing breaks, no alert fires, you just stop appearing. Agent Analytics surfaces each one with the fix.
Blocked
GPTBot disallowed on /pricing
12 money-page URLs are blocked in robots.txt. The engines that answer pricing questions cannot read them.
Review the diff →Error
404 served to AI crawlers
One guide returns 404 to bots while rendering fine for people. A redirect fixes it.
Inspect →Stale
Edit never re-crawled
You rewrote the page 3 days ago. The last bot fetch predates it, so AI still quotes the old copy.
Request re-crawl →Unseen
Shipped page never fetched
A new comparison page has been live 2 days with no answer crawler on it. Its citation cannot land yet.
Ping crawlers →Every issue here routes to Action Center as a fix the agent can ship, with a rollback point saved.
From signal to fix
Crawler problems have one-click fixes.
Agent Analytics finds them, Action Center ships them. Technical crawler work is the safest thing to automate, so the agent handles it and saves a rollback point.
Never crawled68
Publish an llms.txt at the site root
Gives answer engines a canonical map of your best pages
Fetch error64
Redirect the 404 served to GPTBot
One guide is unreadable to bots while fine for people
- 01Detect
- 02Fix
- 03Re-crawl
- 04Verify
“We found ClaudeBot had been 403'd on our blog for four months. Nobody knew, nothing alerted, we were just missing from Claude. One rule change and the citations came back in nine days.”
Questions
Asked plainly.
What is Agent Analytics?
Prefer's AI crawler analytics. It reads your server log files and shows which AI bots crawl your site, which pages they miss, what status codes they get, and how long a new page takes to earn a citation.
Which AI crawlers does it track?
GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, PerplexityBot, Googlebot and Google-Extended today. New crawlers are added as vendors ship them.
What is the difference between answer crawlers and training crawlers?
Answer crawlers like OAI-SearchBot and PerplexityBot fetch a page while an assistant writes an answer, so they arrive just before a citation. Training crawlers like GPTBot harvest pages for future models. Only answer traffic moves today's answers.
Should I block AI crawlers in robots.txt?
Not the answer crawlers. Blocking them removes you from AI answers entirely. Training crawlers are a separate choice, and Agent Analytics lets you see the traffic from each before you decide.
How do I get AI to crawl a new page?
Publish it in your sitemap, link to it internally, list it in llms.txt, and make sure robots.txt allows the answer crawlers. Agent Analytics shows the median time to first crawl so you can tell whether a page is late.
Does it need code on my site?
No. Agent Analytics is log file analysis, so it reads server or CDN logs directly. That means it sees bot traffic that never runs JavaScript, which is most AI crawler traffic.
How is this different from Google Search Console?
Search Console reports Googlebot and Google's index. Agent Analytics reports every AI crawler, including GPTBot, PerplexityBot and ClaudeBot, and ties each fetch to whether the page went on to be cited in an AI answer.
Get your free AI visibility report
in about 10 minutes.
See how answer engines describe your brand today, and where the openings are to outpace the competition.