Technical SEO for AI Search · How to

Robots.txt for AI crawlers: allow GPTBot and the search bots

Part of the guide Technical SEO for AI search: the 2026 checklist, in order

To allow GPTBot in robots.txt, add a group with User-agent: GPTBot and Allow: /, then repeat any private Disallow lines inside it, because a bot that matches its own group ignores the * group. Do the same for the search bots that decide whether you appear in AI answers: OAI-SearchBot, Claude-SearchBot, Claude-User and PerplexityBot. Google-Extended controls Gemini training and grounding, not Google Search.

Step 1: decide your policy for each bot family

Split AI bots into two families before you write a line. Retrieval bots (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Googlebot, Bingbot) decide whether your pages show up in answers. Training bots (GPTBot, ClaudeBot, Google-Extended) feed model training, and CCBot builds Common Crawl's open dataset, so it is usually grouped with them. Blocking a retrieval bot removes you from that assistant; blocking a training bot does not, per each vendor's documentation.

Robots.txt is step two of the technical SEO for AI search checklist; step one is confirming the bots get a 200 at all.

Token Family Controls Source
OAI-SearchBot Retrieval Appearing in ChatGPT search answers OpenAI bots page
GPTBot Training Use in OpenAI model training OpenAI bots page
Claude-SearchBot Retrieval Indexing for Claude's search Anthropic support article
Claude-User Retrieval Claude fetching pages for user questions Anthropic support article
ClaudeBot Training Use in Anthropic model training Anthropic support article
PerplexityBot Retrieval Surfacing and linking in Perplexity results Perplexity bots page
Google-Extended Training and grounding Gemini training, and grounding in Gemini Apps and Vertex AI; not Google Search Google common crawlers page
CCBot Dataset (usually grouped with training) Crawling for Common Crawl's open web dataset Common Crawl CCBot page
Googlebot Retrieval Google Search, including AI Overviews and AI Mode Google AI features page

The usual choice for a business that wants customers to find it is to allow everything in the retrieval column. The training column is your call: allowing it costs you nothing in search, and blocking it costs you nothing in search either.

Step 2: write the file

Here is a complete file that allows every retrieval bot and every training bot on public pages while keeping app paths private. Replace the Disallow paths with yours.

User-agent: *
Allow: /
Disallow: /account
Disallow: /api/
Disallow: /checkout

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Google-Extended
User-agent: CCBot
Allow: /
Disallow: /account
Disallow: /api/
Disallow: /checkout

Sitemap: https://www.example.com/sitemap.xml

To allow search but block training, move GPTBot, ClaudeBot, Google-Extended and CCBot into their own group with Disallow: /:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
Disallow: /

Several User-agent lines stacked above one set of rules is valid and keeps the file short. This is how our own robots.txt on postbing.com is built: one * group, then one group naming twelve AI and search bots with the same private paths repeated.

Step 3: get the matching rules right

Most broken AI bot setups come from three rules of the robots.txt standard (RFC 9309) and Google's implementation of it.

  1. Named groups replace the * group. Google picks "the group with the most specific user agent that matches the crawler's user agent. Other groups are ignored." RFC 9309 says the * group applies only "if no matching group exists". So a User-agent: GPTBot group with only Allow: / lets GPTBot into /account even though * blocks it. Repeat private paths in every named group.
  2. Longest path wins, ties go to Allow. Google uses "the most specific rule based on the length of the rule path" and, on a conflict, "the least restrictive rule". Allow: /blog/ beats Disallow: / for blog URLs.
  3. Paths are case sensitive and prefix matched. User agent names are case insensitive, but paths are not. Disallow: /integrations blocks /integrations, /integrations/ and /integrations-guide. Check every Disallow against your public URLs; a short prefix can hide a marketing page you want found.

Google-Extended has one more quirk. Google says it "doesn't have a separate HTTP request user agent string"; Google crawls with its normal user agents and reads the Google-Extended token "in a control capacity". You will never see Google-Extended in your logs, and disallowing it does not stop Googlebot. Google also states that "Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal". To keep content out of AI Overviews and AI Mode you use nosnippet, max-snippet or the new Search generative AI control in Search Console, not Google-Extended.

Step 4: deploy and wait the right amount of time

Robots.txt changes are not instant. Google "generally caches the contents of robots.txt file for up to 24 hours". OpenAI says its systems can take "~24 hours" to adjust after an update. Plan for a day before judging the effect.

Make sure the file itself returns 200. If robots.txt returns a 5xx error, Google stops crawling the whole site for the first 12 hours while it keeps retrying, then uses the last good copy for up to 30 days. RFC 9309 tells all crawlers to "assume complete disallow" when the file is unreachable because of server errors. A 4xx (other than 429) is treated as if no robots.txt exists, which means everything is allowed. A misconfigured CDN that serves a 503 on /robots.txt can silently close the door to every bot at once.

Keep the file under 500 KiB. Google ignores anything after that limit, and RFC 9309 requires crawlers to parse at least that much.

Step 5: test what the bots actually see

Read the live file, not the one in your repo. CDNs and hosting platforms can inject rules or serve a different file:

curl -s https://www.example.com/robots.txt | grep -iE -A3 "gptbot|searchbot|claude|perplexity|extended|ccbot"

Then confirm each bot gets a 200 on a real page, because a firewall can block a bot that robots.txt allows. The curl test for AI crawler access has the full loop with each vendor's published user agent. For per vendor detail, read OAI-SearchBot vs GPTBot and ClaudeBot, Claude-SearchBot and PerplexityBot.

What robots.txt cannot do

Robots.txt is a request, not a lock, and it does not remove pages from indexes. Three limits to keep in mind:

  • User initiated fetchers may ignore it. OpenAI says robots.txt "may not apply" to ChatGPT-User, and Perplexity says Perplexity-User "generally ignores robots.txt rules".
  • It does not deindex. Google says it "can't index the content of pages which are disallowed for crawling, but it may still index the URL". To keep a page out of Google, allow crawling and use a noindex tag.
  • It is public. RFC 9309 notes that listing paths "exposes them publicly". Do not list secret URLs; protect them with authentication.

With access sorted, the next steps are rendering and sitemaps, covered in order in our technical SEO for AI search checklist. If you publish with Postbing, the safe by design page explains what we do and do not touch on your site.

FAQ

How do I allow GPTBot in robots.txt?

Add User-agent: GPTBot followed by Allow: /, and copy any private Disallow lines from your * group into it. If GPTBot is not named anywhere and your * group allows it, it is already allowed.

Does blocking GPTBot remove me from ChatGPT?

No. GPTBot is OpenAI's training crawler. ChatGPT search uses OAI-SearchBot, which OpenAI controls separately.

Does Google-Extended block AI Overviews?

No. Google says Google-Extended does not affect inclusion in Google Search. AI Overviews and AI Mode are controlled through Googlebot access, snippet controls and the Search generative AI control in Search Console.

How long until bots see my robots.txt change?

Google generally caches robots.txt for up to 24 hours, and OpenAI says its systems can take about 24 hours to adjust. Allow a day.

What happens if my robots.txt returns a 503?

Google stops crawling the site for the first 12 hours, then uses the last good copy for up to 30 days. Under RFC 9309, other crawlers should assume everything is disallowed while the file is unreachable.

Sources

Last updated: November 16, 2026. Spotted an error? Email support@postbing.com and we will fix it. Editorial policy.

Try Postbing free

Postbing reads your site, plans a month of topics from what your customers search for, writes in your voice and publishes to WordPress, Shopify, Webflow or Ghost. Your first article is free.

Try Postbing free

No card needed. No link exchanges, ever.