Technical SEO for AI Search · How to
Robots.txt for AI crawlers: allow GPTBot and the search bots
Part of the guide Technical SEO for AI search: the 2026 checklist, in order
To allow GPTBot in robots.txt, add a group with User-agent: GPTBot and Allow: /, then repeat any private Disallow lines inside it, because a bot that matches its own group ignores the * group. Do the same for the search bots that decide whether you appear in AI answers: OAI-SearchBot, Claude-SearchBot, Claude-User and PerplexityBot. Google-Extended controls Gemini training and grounding, not Google Search.
Step 1: decide your policy for each bot family
Split AI bots into two families before you write a line. Retrieval bots (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Googlebot, Bingbot) decide whether your pages show up in answers. Training bots (GPTBot, ClaudeBot, Google-Extended) feed model training, and CCBot builds Common Crawl's open dataset, so it is usually grouped with them. Blocking a retrieval bot removes you from that assistant; blocking a training bot does not, per each vendor's documentation.
Robots.txt is step two of the technical SEO for AI search checklist; step one is confirming the bots get a 200 at all.
| Token | Family | Controls | Source |
|---|---|---|---|
| OAI-SearchBot | Retrieval | Appearing in ChatGPT search answers | OpenAI bots page |
| GPTBot | Training | Use in OpenAI model training | OpenAI bots page |
| Claude-SearchBot | Retrieval | Indexing for Claude's search | Anthropic support article |
| Claude-User | Retrieval | Claude fetching pages for user questions | Anthropic support article |
| ClaudeBot | Training | Use in Anthropic model training | Anthropic support article |
| PerplexityBot | Retrieval | Surfacing and linking in Perplexity results | Perplexity bots page |
| Google-Extended | Training and grounding | Gemini training, and grounding in Gemini Apps and Vertex AI; not Google Search | Google common crawlers page |
| CCBot | Dataset (usually grouped with training) | Crawling for Common Crawl's open web dataset | Common Crawl CCBot page |
| Googlebot | Retrieval | Google Search, including AI Overviews and AI Mode | Google AI features page |
The usual choice for a business that wants customers to find it is to allow everything in the retrieval column. The training column is your call: allowing it costs you nothing in search, and blocking it costs you nothing in search either.
Step 2: write the file
Here is a complete file that allows every retrieval bot and every training bot on public pages while keeping app paths private. Replace the Disallow paths with yours.
User-agent: *
Allow: /
Disallow: /account
Disallow: /api/
Disallow: /checkout
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Google-Extended
User-agent: CCBot
Allow: /
Disallow: /account
Disallow: /api/
Disallow: /checkout
Sitemap: https://www.example.com/sitemap.xml
To allow search but block training, move GPTBot, ClaudeBot, Google-Extended and CCBot into their own group with Disallow: /:
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
Disallow: /
Several User-agent lines stacked above one set of rules is valid and keeps the file short. This is how our own robots.txt on postbing.com is built: one * group, then one group naming twelve AI and search bots with the same private paths repeated.
Step 3: get the matching rules right
Most broken AI bot setups come from three rules of the robots.txt standard (RFC 9309) and Google's implementation of it.
- Named groups replace the
*group. Google picks "the group with the most specific user agent that matches the crawler's user agent. Other groups are ignored." RFC 9309 says the*group applies only "if no matching group exists". So aUser-agent: GPTBotgroup with onlyAllow: /lets GPTBot into/accounteven though*blocks it. Repeat private paths in every named group. - Longest path wins, ties go to Allow. Google uses "the most specific rule based on the length of the rule path" and, on a conflict, "the least restrictive rule".
Allow: /blog/beatsDisallow: /for blog URLs. - Paths are case sensitive and prefix matched. User agent names are case insensitive, but paths are not.
Disallow: /integrationsblocks/integrations,/integrations/and/integrations-guide. Check everyDisallowagainst your public URLs; a short prefix can hide a marketing page you want found.
Google-Extended has one more quirk. Google says it "doesn't have a separate HTTP request user agent string"; Google crawls with its normal user agents and reads the Google-Extended token "in a control capacity". You will never see Google-Extended in your logs, and disallowing it does not stop Googlebot. Google also states that "Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal". To keep content out of AI Overviews and AI Mode you use nosnippet, max-snippet or the new Search generative AI control in Search Console, not Google-Extended.
Step 4: deploy and wait the right amount of time
Robots.txt changes are not instant. Google "generally caches the contents of robots.txt file for up to 24 hours". OpenAI says its systems can take "~24 hours" to adjust after an update. Plan for a day before judging the effect.
Make sure the file itself returns 200. If robots.txt returns a 5xx error, Google stops crawling the whole site for the first 12 hours while it keeps retrying, then uses the last good copy for up to 30 days. RFC 9309 tells all crawlers to "assume complete disallow" when the file is unreachable because of server errors. A 4xx (other than 429) is treated as if no robots.txt exists, which means everything is allowed. A misconfigured CDN that serves a 503 on /robots.txt can silently close the door to every bot at once.
Keep the file under 500 KiB. Google ignores anything after that limit, and RFC 9309 requires crawlers to parse at least that much.
Step 5: test what the bots actually see
Read the live file, not the one in your repo. CDNs and hosting platforms can inject rules or serve a different file:
curl -s https://www.example.com/robots.txt | grep -iE -A3 "gptbot|searchbot|claude|perplexity|extended|ccbot"
Then confirm each bot gets a 200 on a real page, because a firewall can block a bot that robots.txt allows. The curl test for AI crawler access has the full loop with each vendor's published user agent. For per vendor detail, read OAI-SearchBot vs GPTBot and ClaudeBot, Claude-SearchBot and PerplexityBot.
What robots.txt cannot do
Robots.txt is a request, not a lock, and it does not remove pages from indexes. Three limits to keep in mind:
- User initiated fetchers may ignore it. OpenAI says robots.txt "may not apply" to ChatGPT-User, and Perplexity says Perplexity-User "generally ignores robots.txt rules".
- It does not deindex. Google says it "can't index the content of pages which are disallowed for crawling, but it may still index the URL". To keep a page out of Google, allow crawling and use a
noindextag. - It is public. RFC 9309 notes that listing paths "exposes them publicly". Do not list secret URLs; protect them with authentication.
With access sorted, the next steps are rendering and sitemaps, covered in order in our technical SEO for AI search checklist. If you publish with Postbing, the safe by design page explains what we do and do not touch on your site.
FAQ
How do I allow GPTBot in robots.txt?
Add User-agent: GPTBot followed by Allow: /, and copy any private Disallow lines from your * group into it. If GPTBot is not named anywhere and your * group allows it, it is already allowed.
Does blocking GPTBot remove me from ChatGPT?
No. GPTBot is OpenAI's training crawler. ChatGPT search uses OAI-SearchBot, which OpenAI controls separately.
Does Google-Extended block AI Overviews?
No. Google says Google-Extended does not affect inclusion in Google Search. AI Overviews and AI Mode are controlled through Googlebot access, snippet controls and the Search generative AI control in Search Console.
How long until bots see my robots.txt change?
Google generally caches robots.txt for up to 24 hours, and OpenAI says its systems can take about 24 hours to adjust. Allow a day.
What happens if my robots.txt returns a 503?
Google stops crawling the site for the first 12 hours, then uses the last good copy for up to 30 days. Under RFC 9309, other crawlers should assume everything is disallowed while the file is unreachable.
Sources
- Google, robots.txt specification: group selection, rule precedence, case, 24 hour cache, 4xx and 5xx handling, 500 KiB limit, indexing of blocked URLs, accessed 16 November 2026
- RFC 9309, Robots Exclusion Protocol:
*group fallback, unreachable means disallow, public paths, accessed 16 November 2026 - OpenAI, Overview of OpenAI crawlers: OAI-SearchBot, GPTBot, ChatGPT-User, 24 hour delay, accessed 16 November 2026
- Anthropic, crawler support article: ClaudeBot, Claude-SearchBot, Claude-User, accessed 16 November 2026
- Perplexity, Perplexity crawlers: PerplexityBot, Perplexity-User, accessed 16 November 2026
- Google, common crawlers: Google-Extended scope and token behaviour, accessed 16 November 2026
- Google Search Central, AI features and your website: Googlebot as the control for AI features, snippet controls, accessed 16 November 2026
- Common Crawl, CCBot: CCBot token and blocking instruction, accessed 16 November 2026
Related reading
Last updated: November 16, 2026. Spotted an error? Email support@postbing.com and we will fix it. Editorial policy.
Try Postbing free
Postbing reads your site, plans a month of topics from what your customers search for, writes in your voice and publishes to WordPress, Shopify, Webflow or Ghost. Your first article is free.
Try Postbing freeNo card needed. No link exchanges, ever.