Technical SEO for AI Search · Explainer

OAI-SearchBot vs GPTBot vs ChatGPT-User: which to allow

Part of the guide Technical SEO for AI search: the 2026 checklist, in order

OAI-SearchBot is the OpenAI crawler that decides whether your pages can appear in ChatGPT search answers. GPTBot is a different crawler that collects content that may be used to train OpenAI's models. ChatGPT-User fetches a page when a person in ChatGPT asks for it. They are controlled separately in robots.txt, so you can allow search and block training.

What is the difference between OAI-SearchBot and GPTBot?

The difference is what happens to your content. OAI-SearchBot feeds ChatGPT's search features, so blocking it removes you from ChatGPT search answers. GPTBot feeds model training, so blocking it says your content should not be used to train future models, and per OpenAI it does not affect search.

Bot access is the first item on our technical SEO for AI search checklist, and OpenAI's bots are the first group to check if ChatGPT traffic matters to you.

OpenAI's bot documentation (developers.openai.com/api/docs/bots, read 16 November 2026) describes them in its own words:

Bot OpenAI's description Obeys robots.txt If you block it IP list
OAI-SearchBot "used to surface websites in search results in ChatGPT's search features" Yes "will not be shown in ChatGPT search answers, though can still appear as navigational links" openai.com/searchbot.json
GPTBot "used to crawl content that may be used in training our generative AI foundation models" Yes Signals "content should not be used in training generative AI foundation models" openai.com/gptbot.json
ChatGPT-User Visits started by users in ChatGPT, Custom GPTs and GPT Actions "robots.txt rules may not apply" Not a search opt out; OpenAI says to use OAI-SearchBot for that openai.com/chatgpt-user.json
OAI-AdsBot Checks landing pages for ChatGPT ads Not stated in our notes Relevant only if you run ChatGPT ads openai.com/adsbot.json

OpenAI states that "each setting is independent of the others". The documented example is exactly the common choice: allow OAI-SearchBot to appear in search while disallowing GPTBot.

Which one should you allow?

If you want to be found in ChatGPT, allow OAI-SearchBot. That is the only part of this page that is not a judgment call: OpenAI recommends "allowing OAI-SearchBot in your site's robots.txt file and allowing requests from our published IP ranges" to appear in ChatGPT search.

GPTBot is a business decision with no documented search effect. Three reasonable positions:

  • Allow both. Most small businesses and SaaS sites want their product described accurately by future models and have nothing to protect behind the text. This is the default on sites we run.
  • Allow OAI-SearchBot, block GPTBot. Publishers who license content, or anyone who does not want to contribute training data, can still be cited in ChatGPT search.
  • Block both. Only if you do not want ChatGPT traffic at all. You still may appear as a plain navigational link, per OpenAI.

What robots.txt rules do you need?

Use one group per bot. Here is the "search yes, training no" setup:

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

And "allow both, keep private paths private":

User-agent: OAI-SearchBot
User-agent: GPTBot
Allow: /
Disallow: /account
Disallow: /api/

Two details trip people up. First, a bot that finds a group naming it follows only that group, not your User-agent: * group, so repeat your private Disallow lines inside the bot's group. Second, OpenAI says it "can take ~24 hours" after a robots.txt change for its systems to adjust, so do not test a fix five minutes after deploying it. The full method for every AI bot, with Google and Anthropic included, is in our robots.txt guide for AI crawlers.

Does ChatGPT-User ignore robots.txt?

It may. OpenAI's page says that because ChatGPT-User actions are started by a person, "robots.txt rules may not apply". It also says ChatGPT-User "is not used for crawling the web in an automatic fashion" and "is not used to determine whether content may appear in Search".

In practice that means two things. You cannot rely on robots.txt to stop a user from asking ChatGPT to read a public page. And you should not block ChatGPT-User at your firewall by accident: when someone pastes your pricing URL into ChatGPT and asks a question, this is the agent that fetches it. If it gets a 403, the answer is built without your page.

How do you check each bot can actually reach you?

Robots.txt is only half the door. A firewall, CDN bot rule or rate limiter can return 403 or a challenge page to these bots even when robots.txt allows them. Send a request with each user agent and look at the status code:

for ua in \
 "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot" \
 "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot" \
 "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot"; do
  curl -s -o /dev/null -w "%{http_code}  ${ua##*compatible; }\n" -A "$ua" https://www.example.com/pricing
done

These are the full user agent strings OpenAI published on 16 November 2026; OpenAI notes the version number may change. Expect 200 for each. A spoofed user agent from your laptop is not the same as a request from OpenAI's IP ranges, so if your WAF verifies bots by IP, also check its logs for real OAI-SearchBot requests. Our curl test guide covers reading the results and the cases a header test cannot catch.

Does allowing OAI-SearchBot get you cited?

No. Allowing it makes you eligible; it does not decide whether ChatGPT picks your page. Selection depends on whether your page answers the rewritten searches ChatGPT runs, whether the text is in the raw HTML, and how your page compares with others. Content work for that is covered in how to get cited by ChatGPT, and the full technical list sits in the technical SEO for AI search checklist.

Also note that OpenAI says opted out sites can "still appear as navigational links". Blocking OAI-SearchBot does not make your brand name disappear from ChatGPT; it stops your pages being used as search answers.

FAQ

Is OAI-SearchBot the same as GPTBot?

No. OAI-SearchBot crawls for ChatGPT search results. GPTBot crawls for content that may be used to train OpenAI's models. OpenAI controls them with separate robots.txt user agent tokens.

No. OpenAI says each setting is independent and gives the example of allowing OAI-SearchBot while disallowing GPTBot. Search visibility depends on OAI-SearchBot.

How long does a robots.txt change take to apply for OpenAI?

OpenAI says it can take about 24 hours from a robots.txt update for its systems to adjust.

Can I block ChatGPT-User with robots.txt?

Not reliably. OpenAI says robots.txt rules may not apply to ChatGPT-User because a person starts each visit. It is not used for automatic crawling or for search eligibility.

Where are OpenAI's crawler IP ranges?

OpenAI publishes JSON files at openai.com/searchbot.json, openai.com/gptbot.json and openai.com/chatgpt-user.json. Use them to allow the bots in a firewall.

Sources

Last updated: November 16, 2026. Spotted an error? Email support@postbing.com and we will fix it. Editorial policy.

Try Postbing free

Postbing reads your site, plans a month of topics from what your customers search for, writes in your voice and publishes to WordPress, Shopify, Webflow or Ghost. Your first article is free.

Try Postbing free

No card needed. No link exchanges, ever.