GPTBot: OpenAI's training crawler explained
GPTBot is OpenAI's model-training crawler, not its search crawler. Learn how to separate training, search visibility, and user-requested access.
GPTBot
GPTBot is OpenAI’s crawler for content that may be used to improve generative AI foundation models. It is not the crawler that determines whether a page can appear in ChatGPT search. OpenAI documents separate controls for GPTBot, OAI-SearchBot, and ChatGPT-User.
That distinction matters. A publisher can decline model-training crawling while still allowing OpenAI’s search crawler to discover public pages.
Which OpenAI crawler does what?
| User-agent | Primary purpose | Practical decision |
|---|---|---|
GPTBot | Collect content that may be used for model improvement | Allow or block according to training and licensing policy |
OAI-SearchBot | Discover and surface public content in ChatGPT search | Allow if ChatGPT search visibility is wanted |
ChatGPT-User | Fetch a page for a user-initiated action | Treat separately; user-requested access may not behave like routine crawling |
Blocking GPTBot alone should not be described as opting out of ChatGPT search. Likewise, allowing GPTBot does not guarantee indexing, a citation, or referral traffic.
robots.txt examples
To block training crawling but retain search discovery:
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
To block both purposes, create a separate group for each token. Do not put Disallow: / under User-agent: * unless the intention is to block all compliant crawlers.
Decision checklist
- Confirm whether the content is public, licensed, paywalled, or contractually restricted.
- Decide separately on model training and AI search visibility.
- Apply the rule to every relevant hostname and subdomain.
- Check CDN, WAF, and bot-management rules; robots.txt permission does not override an edge block.
- Re-test after changes and document the policy owner and review date.
How to verify real GPTBot traffic
A user-agent string can be spoofed. Match log requests against OpenAI’s current published IP range files, then look at response codes, requested paths, and WAF actions. A 200 for /robots.txt does not prove content pages were reachable; sample representative HTML and asset URLs too.
Do not use a ChatGPT prompt as the sole test. An answer can rely on older knowledge, third-party search data, or a previously fetched copy. Server and edge logs provide the stronger access evidence.
Limitations
robots.txt is a request to compliant crawlers, not access control. Use authentication and authorization for private material. To control search indexing, use the appropriate crawler policy plus index controls such as noindex; the crawler must be able to fetch a page to read its meta robots directive.
Primary sources
See also: AI crawlers, robots.txt, and AI referral traffic.