For most coaching sites: allow the search and user-request crawlers from every AI company, because they’re how assistants find and quote you, and decide separately about training crawlers. Blocking OAI-SearchBot, Claude-SearchBot or PerplexityBot keeps your pages out of those assistants’ search answers.
Three kinds of AI crawler
AI companies now run separate crawlers for separate jobs, and robots.txt lets you treat each one differently.
| Company | Search (find and cite) | User requests (a page someone asked about) | Training |
|---|---|---|---|
| OpenAI | OAI-SearchBot | ChatGPT-User | GPTBot |
| Anthropic | Claude-SearchBot | Claude-User | ClaudeBot |
| Perplexity | PerplexityBot | Perplexity-User | — |
| Googlebot (Search, including AI Overviews) | — | Google-Extended (a robots.txt token, not a separate crawler) | |
| Apple | Applebot | — | Applebot-Extended (a robots.txt token) |
Our recommendation for coaches
- Allow every search crawler. They decide whether you can appear in answers at all.
- Allow user-request crawlers. They fetch a page when someone shares your link or asks about you.
- Training is your call. Allowing it lets future models learn your name and services. Blocking it keeps your writing out of future training, and doesn’t remove you from search answers.
How to set it in robots.txt
Search and user-request crawlers follow your rules for everyone (User-agent: *), so you only need a group for the crawlers you want to block. To block training only:
# Training crawlers: blocked. Delete this group to allow them.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /Avoid adding groups that only say Allow: / for single crawlers: a crawler with its own group ignores the User-agent: * rules, including the parts you meant for everyone. In WordPress, Obi GEO for WordPress lets you choose which AI crawlers to allow and adds the rules to WordPress’s own robots.txt without replacing it.
Check your firewall too
robots.txt is a request; firewalls enforce. Some security and CDN settings block or challenge AI crawlers whatever robots.txt says, and the crawler sees a challenge page instead of your content. If you use bot protection, look for settings about AI crawlers or verified bots. The free check tests this for you.
Sources
- OpenAI: overview of OpenAI crawlers
- Anthropic: does Anthropic crawl data from the web, and how can site owners block the crawler?
- Perplexity: Perplexity crawlers
- Google: AI features and your website
Related: what is GEO? · how to check whether ChatGPT can read your website