Done for you: our team does the AEO work for you.Learn more
PreferBot

The bot behind Prefer, in plain words.

Prefer reads the pages AI engines cite, so we can show brands what those answers are built on. PreferBot says who it is, follows robots.txt, and never gets around a site that says no.

How to recognise it

Every request PreferBot makes carries this user agent:

PreferBot/1.0 (+https://tryprefer.com/bot)

What PreferBot is

PreferBot is how Prefer reads web pages. Prefer measures which pages AI engines such as ChatGPT, Gemini, Perplexity and Google AI Overviews cite in their answers. When an engine cites a page, PreferBot may read it to learn what the page covers: whether it ranks products or services, who wrote it, and when it was last updated.

What it reads, and what it does not

PreferBot reads public pages that an AI engine cited in an answer, one page at a time, and the robots.txt of each site first. It does not follow links, crawl whole sites, log in, submit forms, run scripts or solve challenges. If a site turns it away, it stops there.

What it keeps

PreferBot keeps the facts it needs about a page, such as its title, its headings, the entries in a ranked list and at most three short quotes, each credited and linked to the page. It does not store the page itself.

A light touch

PreferBot reads only pages that matter to an answer, and waits before it tries a page that did not respond. When a Prefer customer asks us to review their own website, we read it on their behalf, and may use a standard browser identity if their site blocks automated tools.

How to opt out

Add these lines to your site's robots.txt and PreferBot will not read it:

User-agent: PreferBot
Disallow: /

PreferBot checks robots.txt before every page and follows path rules too, so you can close off part of a site instead. Where no group names PreferBot, the rules for all agents (User-agent: *) apply. If your robots.txt cannot be reached, PreferBot does not read the site.

Questions

If PreferBot is causing you a problem, or you have a question about it, email [email protected] and we will look into it.

Get your free AI visibility report
in about 10 minutes.

See how answer engines describe your brand today, and where the openings are to outpace the competition.