Technical SEO for AI Search · Explainer

ClaudeBot, Claude-SearchBot and PerplexityBot: robots.txt guide

Part of the guide Technical SEO for AI search: the 2026 checklist, in order

ClaudeBot collects pages that may be used to train Anthropic's models; Claude-SearchBot indexes pages for Claude's search; Claude-User fetches a page when someone asks Claude a question. Perplexity runs PerplexityBot for its search results and Perplexity-User for user requests. To stay visible in both assistants, allow Claude-SearchBot, Claude-User and PerplexityBot in robots.txt. Blocking ClaudeBot only affects training.

What does each Anthropic and Perplexity bot do?

Each vendor splits its bots by job: one for training (Anthropic only), one for search indexing, one for live fetches a user asked for. The table uses the vendors' own wording from their documentation, read on 16 November 2026.

Getting these five right is step one of our technical SEO for AI search checklist: if the search bots cannot fetch a page, nothing else on the list matters for that assistant.

Bot Vendor Job (vendor's words) Robots.txt Effect of blocking
ClaudeBot Anthropic "collecting web content that could potentially contribute to their training" Honored Future content "excluded from our AI model training datasets"
Claude-SearchBot Anthropic "navigates the web to improve search result quality for users" Honored "prevents our system from indexing your content for search optimization"
Claude-User Anthropic Fetches pages "when individuals ask questions to Claude" Honored (Anthropic does not list an exception) "prevents our system from retrieving your content"; may reduce visibility in user directed search
PerplexityBot Perplexity "designed to surface and link websites in search results on Perplexity. It is not used to crawl content for AI foundation models." Perplexity recommends allowing it You stop being surfaced and linked in Perplexity results
Perplexity-User Perplexity "supports user actions within Perplexity"; not used for crawling or training "generally ignores robots.txt rules" Robots.txt is not a reliable block

The notable difference from OpenAI: Anthropic states that its bots honour robots.txt and does not carve out the user initiated agent. OpenAI and Perplexity both say their user initiated fetchers may not follow robots.txt. For OpenAI's three bots, see OAI-SearchBot vs GPTBot.

What is the safest robots.txt for Claude and Perplexity?

If you want to appear in answers but keep content out of training, allow the search and user bots and disallow ClaudeBot:

User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
Allow: /

User-agent: ClaudeBot
Disallow: /

If you are happy to be used for training too (the default on sites we run, because we want products described accurately), a single group works:

User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
Allow: /
Disallow: /account

Remember that a bot which matches a named group ignores your User-agent: * group, so copy any private Disallow lines into the named group. The general rules, with Google-Extended and the others, are in our robots.txt guide for AI crawlers.

Does ClaudeBot support Crawl-delay?

Yes. Anthropic's page says "we support the non-standard Crawl-delay extension to robots.txt" and gives this example:

User-agent: ClaudeBot
Crawl-delay: 1

Use it if ClaudeBot traffic is straining a small server, instead of blocking it. Crawl-delay is not part of the robots.txt standard (RFC 9309) and Google ignores it, so a value set for ClaudeBot does not slow Googlebot.

Should you block Anthropic or Perplexity by IP address?

No, use robots.txt. Anthropic warns that IP blocking "may not work correctly or persistently guarantee an opt-out, as doing so impedes our ability to read your robots.txt file." Its IP list is at claude.com/crawling/bots.json if your firewall needs to allow, not block, these bots.

Perplexity takes the opposite angle for firewalls. It publishes perplexity.ai/perplexitybot.json and perplexity.ai/perplexity-user.json and tells WAF operators to "set up automated processes to periodically fetch and update your WAF rules with the latest IP ranges from these endpoints". If your CDN has a "block AI bots" toggle, check exactly which user agents it blocks before switching it on: if the list includes PerplexityBot or Claude-SearchBot, it removes you from those answers, not only from training.

Does blocking ClaudeBot cover subdomains?

No. Anthropic says you must repeat the block "for every subdomain that you wish to opt out from". Robots.txt is per host: docs.example.com/robots.txt and www.example.com/robots.txt are separate files. The same applies in the other direction: if your blog is on blog.example.com, its own robots.txt must allow Claude-SearchBot and PerplexityBot, whatever the main site says.

What changed in October 2026: the AIPREF training signal

Anthropic now also reads a site wide training preference. Its page says: "As of October 30, 2026, Anthropic has begun recognizing a machine-readable signal that lets website owners express a preference about whether their content may be used to train AI models." This follows the IETF aipref drafts.

Named rules still win. Anthropic says: "If your robots.txt file names ClaudeBot, we follow that rule, even if your general setting says something different." So if you already manage ClaudeBot explicitly, nothing changes. Google has said it plans to support the same shared signal in 2027. Perplexity's bot page, read on 16 November 2026, does not mention it.

How do you confirm the bots can reach you?

Anthropic's support page does not publish exact user agent strings, so you cannot copy a full Claude string from the source. You can still test whether your firewall or CDN reacts to the bot names, because most rules match on the token:

for ua in "ClaudeBot" "Claude-SearchBot" "Claude-User" \
  "Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)" \
  "Mozilla/5.0 (compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)"; do
  curl -s -o /dev/null -w "%{http_code}  $ua\n" -A "$ua" https://www.example.com/
done

The Perplexity strings are copied from Perplexity's docs. The Claude lines send only the token, which is a test of name based rules, not a perfect copy of a real request. Expect 200 for all of them. The curl test guide explains what a 403, 429 or 503 tells you and how to check real bot hits in your logs.

Access is the floor, not the goal. Whether Perplexity cites you after that depends on the page; see how to get cited by Perplexity and the technical SEO for AI search checklist.

FAQ

Does blocking ClaudeBot remove my site from Claude's answers?

Not by itself. ClaudeBot is the training crawler. Claude's search indexing uses Claude-SearchBot and live fetches use Claude-User, which you control with separate robots.txt groups.

Does PerplexityBot collect training data?

Perplexity says PerplexityBot is not used to crawl content for AI foundation models. It surfaces and links websites in Perplexity search results.

Can I block Perplexity-User with robots.txt?

Not reliably. Perplexity says that because a user requested the fetch, Perplexity-User generally ignores robots.txt rules.

What Crawl-delay value does Anthropic support?

Anthropic supports the non-standard Crawl-delay directive and shows Crawl-delay: 1 for ClaudeBot as its example. Google ignores Crawl-delay.

Does Anthropic honour robots.txt for Claude-User?

Anthropic says its bots honour robots.txt directives and lists Claude-User among them, with no exception stated. Blocking Claude-User stops Claude retrieving your pages for user questions.

Sources

Last updated: November 16, 2026. Spotted an error? Email support@postbing.com and we will fix it. Editorial policy.

Try Postbing free

Postbing reads your site, plans a month of topics from what your customers search for, writes in your voice and publishes to WordPress, Shopify, Webflow or Ghost. Your first article is free.

Try Postbing free

No card needed. No link exchanges, ever.