Glossary
GPTBot, OAI-SearchBot, PerplexityBot: which AI crawlers to allow
A retrieval crawler fetches your page in real time to answer a question someone is asking at that moment. A training crawler collects text to train a future model. They are different bots with different user agents, and confusing them is the most expensive robots.txt mistake available.
Which is which?
| User agent | Type | Blocking it costs you |
|---|---|---|
| OAI-SearchBot | Retrieval | Removal from ChatGPT search answers |
| PerplexityBot | Retrieval | Removal from Perplexity answers |
| Claude-SearchBot | Retrieval | Removal from Claude answers |
| Bingbot | Retrieval | Removal from Bing and Copilot answers. ChatGPT search also crawls with its own OAI-SearchBot, so Bing is not its only source |
| GPTBot | Training | Nothing in current visibility |
| ClaudeBot | Training | Nothing in current visibility |
Plenty of sites have blocked all AI user agents on principle, reasoning that they do not want to feed model training. That is a legitimate position — but it also removes them from the answers their buyers read, which is usually not what was intended. The two decisions are separable and should be made separately.
Do they run JavaScript?
No. GPTBot, ClaudeBot and PerplexityBot fetch HTML and parse it. They do not execute JavaScript. A site rendered client-side shows them an empty shell. Googlebot does render JavaScript, so a site can rank perfectly well in classic search and be invisible to the engines that answer questions.
GPTBot vs OAI-SearchBot vs ChatGPT-User
OpenAI runs three agents you will see in your logs, and each robots.txt setting works on its own. GPTBot crawls content that may be used to train OpenAI’s models. OAI-SearchBot crawls so pages can appear in ChatGPT’s search answers; OpenAI says sites that block it are not shown there, apart from plain navigational links. ChatGPT-User is not a crawler at all. It fetches a page when someone in ChatGPT asks for it, and because a person starts that request, OpenAI says robots.txt rules may not apply.
So blocking GPTBot keeps you out of training without touching ChatGPT search, while blocking OAI-SearchBot is the setting that costs visibility. OpenAI also says a robots.txt change takes about 24 hours to reach its search systems. (Source: OpenAI’s crawler documentation.)
What about PerplexityBot, Claude and Google?
Perplexity and Anthropic split their bots the same way. PerplexityBot builds Perplexity’s search index and Perplexity-User fetches pages on request. Anthropic has ClaudeBot for training, Claude-SearchBot for search and Claude-User for requests. Google is the exception: Googlebot does the crawling for Search, AI Overviews and AI Mode, and Google-Extended is only a robots.txt token for Gemini model use. Blocking Google-Extended does not remove you from Search or AI Overviews.
A robots.txt for AI crawlers you can copy
This keeps every retrieval crawler in and opts out of training. Delete the second half if you are happy for your pages to be used in training.
# Let AI search engines fetch and cite your pages
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
# Optional: opt out of model training
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /How do you check which AI crawlers can reach you?
robots.txt is only half of it. A firewall or CDN can still return 403 to a bot that robots.txt allows, and nothing in robots.txt will tell you. Our free AI crawler checker sends a live request as each crawler for any wordpress.org plugin, and the free Softrecs AI Crawler Checker plugin runs the same test from inside your own WordPress admin.
What else breaks retrieval?
Anything that stops the fetch completing, and these do not appear in any content audit. An expired TLS certificate fails the handshake before a byte of HTML is read. A WAF rule blocking unfamiliar user agents does the same at the edge. So does aggressive bot protection at the CDN — and because robots.txt is fetched over the same connection, a permissive robots.txt is no protection against it.
The check takes a second: curl -A "OAI-SearchBot" -I https://yoursite.com/. Anything other than a 200 makes every other optimisation irrelevant.
Related: does llms.txt work?, do AI crawlers run JavaScript? and the free AI visibility checker.
Part of the GEO glossary. The measurements referenced here come from the Plugin Visibility Index, which publishes its raw counts and its limits.