Prefer reads the pages AI engines cite, so we can show brands what those answers are built on. PreferBot says who it is, follows robots.txt, and never gets around a site that says no.
Every request PreferBot makes carries this user agent:
PreferBot/1.0 (+https://tryprefer.com/bot) PreferBot is how Prefer reads web pages. Prefer measures which pages AI engines such as ChatGPT, Gemini, Perplexity and Google AI Overviews cite in their answers. When an engine cites a page, PreferBot may read it to learn what the page covers: whether it ranks products or services, who wrote it, and when it was last updated.
PreferBot reads public pages that an AI engine cited in an answer, one page at a time, and the robots.txt of each site first. It does not follow links, crawl whole sites, log in, submit forms, run scripts or solve challenges. If a site turns it away, it stops there.
PreferBot keeps the facts it needs about a page, such as its title, its headings, the entries in a ranked list and at most three short quotes, each credited and linked to the page. It does not store the page itself.
PreferBot reads only pages that matter to an answer, and waits before it tries a page that did not respond. When a Prefer customer asks us to review their own website, we read it on their behalf, and may use a standard browser identity if their site blocks automated tools.
Add these lines to your site's robots.txt and PreferBot will not read it:
User-agent: PreferBot
Disallow: / PreferBot checks robots.txt before every page and follows path rules too, so you can close off part of a site instead. Where no group names PreferBot, the rules for all agents (User-agent: *) apply. If your robots.txt cannot be reached, PreferBot does not read the site.
If PreferBot is causing you a problem, or you have a question about it, email [email protected] and we will look into it.
See how answer engines describe your brand today, and where the openings are to outpace the competition.