---
title: Robots.txt for AI crawlers: allow GPTBot and the search bots
description: A copy ready robots.txt for GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended and CCBot, with the matching rules that make most AI bot setups fail.
canonical: https://www.postbing.com/blog/robots-txt-for-ai-crawlers
published: 2026-11-16
updated: 2026-11-16
author: The Postbing Team
---

# Robots.txt for AI crawlers: allow GPTBot and the search bots

To allow GPTBot in robots.txt, add a group with `User-agent: GPTBot` and `Allow: /`, then repeat any private `Disallow` lines inside it, because a bot that matches its own group ignores the `*` group. Do the same for the search bots that decide whether you appear in AI answers: OAI-SearchBot, Claude-SearchBot, Claude-User and PerplexityBot. Google-Extended controls Gemini training and grounding, not Google Search.

Last updated: November 16, 2026

## Step 1: decide your policy for each bot family

Split AI bots into two families before you write a line. Retrieval bots (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Googlebot, Bingbot) decide whether your pages show up in answers. Training bots (GPTBot, ClaudeBot, Google-Extended) feed model training, and CCBot builds Common Crawl's open dataset, so it is usually grouped with them. Blocking a retrieval bot removes you from that assistant; blocking a training bot does not, per each vendor's documentation.

Robots.txt is step two of the [technical SEO for AI search checklist](/blog/technical-seo-for-ai-search-checklist); step one is confirming the bots get a 200 at all.

| Token | Family | Controls | Source |
|---|---|---|---|
| OAI-SearchBot | Retrieval | Appearing in ChatGPT search answers | OpenAI bots page |
| GPTBot | Training | Use in OpenAI model training | OpenAI bots page |
| Claude-SearchBot | Retrieval | Indexing for Claude's search | Anthropic support article |
| Claude-User | Retrieval | Claude fetching pages for user questions | Anthropic support article |
| ClaudeBot | Training | Use in Anthropic model training | Anthropic support article |
| PerplexityBot | Retrieval | Surfacing and linking in Perplexity results | Perplexity bots page |
| Google-Extended | Training and grounding | Gemini training, and grounding in Gemini Apps and Vertex AI; not Google Search | Google common crawlers page |
| CCBot | Dataset (usually grouped with training) | Crawling for Common Crawl's open web dataset | Common Crawl CCBot page |
| Googlebot | Retrieval | Google Search, including AI Overviews and AI Mode | Google AI features page |

The usual choice for a business that wants customers to find it is to allow everything in the retrieval column. The training column is your call: allowing it costs you nothing in search, and blocking it costs you nothing in search either.

## Step 2: write the file

Here is a complete file that allows every retrieval bot and every training bot on public pages while keeping app paths private. Replace the `Disallow` paths with yours.

```
User-agent: *
Allow: /
Disallow: /account
Disallow: /api/
Disallow: /checkout

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Google-Extended
User-agent: CCBot
Allow: /
Disallow: /account
Disallow: /api/
Disallow: /checkout

Sitemap: https://www.example.com/sitemap.xml
```

To allow search but block training, move GPTBot, ClaudeBot, Google-Extended and CCBot into their own group with `Disallow: /`:

```
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
Disallow: /
```

Several `User-agent` lines stacked above one set of rules is valid and keeps the file short. This is how our own robots.txt on postbing.com is built: one `*` group, then one group naming twelve AI and search bots with the same private paths repeated.

## Step 3: get the matching rules right

Most broken AI bot setups come from three rules of the robots.txt standard (RFC 9309) and Google's implementation of it.

1. **Named groups replace the `*` group.** Google picks "the group with the most specific user agent that matches the crawler's user agent. Other groups are ignored." RFC 9309 says the `*` group applies only "if no matching group exists". So a `User-agent: GPTBot` group with only `Allow: /` lets GPTBot into `/account` even though `*` blocks it. Repeat private paths in every named group.
2. **Longest path wins, ties go to Allow.** Google uses "the most specific rule based on the length of the rule path" and, on a conflict, "the least restrictive rule". `Allow: /blog/` beats `Disallow: /` for blog URLs.
3. **Paths are case sensitive and prefix matched.** User agent names are case insensitive, but paths are not. `Disallow: /integrations` blocks `/integrations`, `/integrations/` and `/integrations-guide`. Check every `Disallow` against your public URLs; a short prefix can hide a marketing page you want found.

Google-Extended has one more quirk. Google says it "doesn't have a separate HTTP request user agent string"; Google crawls with its normal user agents and reads the `Google-Extended` token "in a control capacity". You will never see Google-Extended in your logs, and disallowing it does not stop Googlebot. Google also states that "Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal". To keep content out of AI Overviews and AI Mode you use `nosnippet`, `max-snippet` or the new Search generative AI control in Search Console, not Google-Extended.

## Step 4: deploy and wait the right amount of time

Robots.txt changes are not instant. Google "generally caches the contents of robots.txt file for up to 24 hours". OpenAI says its systems can take "~24 hours" to adjust after an update. Plan for a day before judging the effect.

Make sure the file itself returns 200. If robots.txt returns a 5xx error, Google stops crawling the whole site for the first 12 hours while it keeps retrying, then uses the last good copy for up to 30 days. RFC 9309 tells all crawlers to "assume complete disallow" when the file is unreachable because of server errors. A 4xx (other than 429) is treated as if no robots.txt exists, which means everything is allowed. A misconfigured CDN that serves a 503 on `/robots.txt` can silently close the door to every bot at once.

Keep the file under 500 KiB. Google ignores anything after that limit, and RFC 9309 requires crawlers to parse at least that much.

## Step 5: test what the bots actually see

Read the live file, not the one in your repo. CDNs and hosting platforms can inject rules or serve a different file:

```bash
curl -s https://www.example.com/robots.txt | grep -iE -A3 "gptbot|searchbot|claude|perplexity|extended|ccbot"
```

Then confirm each bot gets a 200 on a real page, because a firewall can block a bot that robots.txt allows. The [curl test for AI crawler access](/blog/test-ai-crawler-access-with-curl) has the full loop with each vendor's published user agent. For per vendor detail, read [OAI-SearchBot vs GPTBot](/blog/oai-searchbot-vs-gptbot) and [ClaudeBot, Claude-SearchBot and PerplexityBot](/blog/claudebot-claude-searchbot-perplexitybot).

## What robots.txt cannot do

Robots.txt is a request, not a lock, and it does not remove pages from indexes. Three limits to keep in mind:

- **User initiated fetchers may ignore it.** OpenAI says robots.txt "may not apply" to ChatGPT-User, and Perplexity says Perplexity-User "generally ignores robots.txt rules".
- **It does not deindex.** Google says it "can't index the content of pages which are disallowed for crawling, but it may still index the URL". To keep a page out of Google, allow crawling and use a `noindex` tag.
- **It is public.** RFC 9309 notes that listing paths "exposes them publicly". Do not list secret URLs; protect them with authentication.

With access sorted, the next steps are rendering and sitemaps, covered in order in our [technical SEO for AI search checklist](/blog/technical-seo-for-ai-search-checklist). If you publish with Postbing, the [safe by design page](/safe) explains what we do and do not touch on your site.

## FAQ

### How do I allow GPTBot in robots.txt?
Add `User-agent: GPTBot` followed by `Allow: /`, and copy any private `Disallow` lines from your `*` group into it. If GPTBot is not named anywhere and your `*` group allows it, it is already allowed.

### Does blocking GPTBot remove me from ChatGPT?
No. GPTBot is OpenAI's training crawler. ChatGPT search uses OAI-SearchBot, which OpenAI controls separately.

### Does Google-Extended block AI Overviews?
No. Google says Google-Extended does not affect inclusion in Google Search. AI Overviews and AI Mode are controlled through Googlebot access, snippet controls and the Search generative AI control in Search Console.

### How long until bots see my robots.txt change?
Google generally caches robots.txt for up to 24 hours, and OpenAI says its systems can take about 24 hours to adjust. Allow a day.

### What happens if my robots.txt returns a 503?
Google stops crawling the site for the first 12 hours, then uses the last good copy for up to 30 days. Under RFC 9309, other crawlers should assume everything is disallowed while the file is unreachable.

## Sources

- [Google, robots.txt specification](https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec): group selection, rule precedence, case, 24 hour cache, 4xx and 5xx handling, 500 KiB limit, indexing of blocked URLs, accessed 16 November 2026
- [RFC 9309, Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.txt): `*` group fallback, unreachable means disallow, public paths, accessed 16 November 2026
- [OpenAI, Overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots): OAI-SearchBot, GPTBot, ChatGPT-User, 24 hour delay, accessed 16 November 2026
- [Anthropic, crawler support article](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler): ClaudeBot, Claude-SearchBot, Claude-User, accessed 16 November 2026
- [Perplexity, Perplexity crawlers](https://docs.perplexity.ai/guides/bots): PerplexityBot, Perplexity-User, accessed 16 November 2026
- [Google, common crawlers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers): Google-Extended scope and token behaviour, accessed 16 November 2026
- [Google Search Central, AI features and your website](https://developers.google.com/search/docs/appearance/ai-features): Googlebot as the control for AI features, snippet controls, accessed 16 November 2026
- [Common Crawl, CCBot](https://commoncrawl.org/ccbot): CCBot token and blocking instruction, accessed 16 November 2026

## Related reading

- [Technical SEO for AI search: the full checklist](/blog/technical-seo-for-ai-search-checklist)
- [OAI-SearchBot vs GPTBot vs ChatGPT-User](/blog/oai-searchbot-vs-gptbot)
- [ClaudeBot, Claude-SearchBot and PerplexityBot explained](/blog/claudebot-claude-searchbot-perplexitybot)
- [How to test AI crawler access with curl](/blog/test-ai-crawler-access-with-curl)
- [How to rank in Google AI Overviews](/blog/how-to-rank-in-google-ai-overviews)
