Glossary
What is a retrieval crawler?
A retrieval crawler fetches your page in real time to answer a question someone is asking at that moment. A training crawler collects text to train a future model. They are different bots with different user agents, and confusing them is the most expensive robots.txt mistake available.
Which is which?
| User agent | Type | Blocking it costs you |
|---|---|---|
| OAI-SearchBot | Retrieval | Removal from ChatGPT search answers |
| PerplexityBot | Retrieval | Removal from Perplexity answers |
| Claude-SearchBot | Retrieval | Removal from Claude answers |
| Bingbot | Retrieval (indirect) | ChatGPT has no index of its own and queries Bing |
| GPTBot | Training | Nothing in current visibility |
| ClaudeBot | Training | Nothing in current visibility |
Plenty of sites have blocked all AI user agents on principle, reasoning that they do not want to feed model training. That is a legitimate position — but it also removes them from the answers their buyers read, which is usually not what was intended. The two decisions are separable and should be made separately.
Do they run JavaScript?
No. GPTBot, ClaudeBot and PerplexityBot fetch HTML and parse it. They do not execute JavaScript. A site rendered client-side shows them an empty shell. Googlebot does render JavaScript, so a site can rank perfectly well in classic search and be invisible to the engines that answer questions.
What else breaks retrieval?
Anything that stops the fetch completing, and these do not appear in any content audit. An expired TLS certificate fails the handshake before a byte of HTML is read. A WAF rule blocking unfamiliar user agents does the same at the edge. So does aggressive bot protection at the CDN — and because robots.txt is fetched over the same connection, a permissive robots.txt is no protection against it.
The check takes a second: curl -A "OAI-SearchBot" -I https://yoursite.com/. Anything other than a 200 makes every other optimisation irrelevant.
Part of the GEO glossary. The measurements referenced here come from the Plugin Visibility Index, which publishes its raw counts and its limits.