Technical SEO for AI Search · How to
Test AI crawler access with curl: a 10 minute check
Part of the guide Technical SEO for AI search: the 2026 checklist, in order
To test whether AI crawlers can reach your site, request one of your pages with curl using each bot's published user agent, and check two things: the status code is 200, and a sentence from the page body appears in the raw HTML. Anything else (403, 429, 503, a challenge page, an empty JavaScript shell) means that bot cannot use your page in answers.
Why test with curl when robots.txt already allows the bots?
Because robots.txt is only one of three doors. A crawler can be allowed by robots.txt and still be refused by your CDN's bot rules, a WAF, a rate limiter or a geo block, and it can get a 200 that contains no readable text if the page is built in the browser. curl shows you the response a bot gets, with no browser and no JavaScript, which is close to what most AI fetchers see.
This test is the first step of our technical SEO for AI search checklist.
The test takes about ten minutes for three URLs: your home page, your blog index and one article. Run it after any CDN, hosting or security plugin change.
The test script
Save this as bottest.sh, set URL to a real page and PHRASE to a short sentence that appears in that page's body text (not in the menu), then run bash bottest.sh.
URL="https://www.example.com/blog/your-article"
PHRASE="a sentence from the middle of the article"
while IFS='|' read -r name ua; do
body=$(curl -s -L -m 20 -A "$ua" -w '\n%{http_code}' "$URL")
code=${body##*$'\n'}
if printf '%s' "$body" | grep -qiF "$PHRASE"; then text=yes; else text=NO; fi
printf '%-16s %s text:%s\n' "$name" "$code" "$text"
done <<'EOF'
OAI-SearchBot|Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot
ChatGPT-User|Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot
GPTBot|Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot
PerplexityBot|Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)
Perplexity-User|Mozilla/5.0 (compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)
Claude-SearchBot|Claude-SearchBot
Claude-User|Claude-User
ClaudeBot|ClaudeBot
Googlebot|Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
EOF
A healthy result looks like this (our own run against postbing.com/pricing on 16 November 2026, first three bots):
OAI-SearchBot 200 text:yes
PerplexityBot 200 text:yes
Claude-SearchBot 200 text:yes
Where the strings come from: the OpenAI and Perplexity lines are copied from their crawler documentation on 16 November 2026 (OpenAI notes the version number may change). Google publishes the Googlebot smartphone string with Chrome/W.X.Y.Z as a placeholder; any current Chrome version works for this test. Anthropic's crawler page does not publish full user agent strings, so the Claude lines send just the bot name. That still catches firewall rules that match on the name, which is how most of them work.
How to read the results
Each status code points to a different fix. Use this table:
| You see | What it usually means | Where to look |
|---|---|---|
200 text:yes |
The bot can read the page | Nothing to fix for this bot |
200 text:NO |
Page loads but body text is missing from the HTML | JavaScript rendering, or a challenge page served with 200 |
301 or 302 (seen when you drop -L) |
The script follows redirects and prints the final code | Fine, but link to and list the final URL in your sitemap |
403 |
Refused by a bot rule, WAF or security plugin | CDN bot settings, WAF custom rules, plugin "block bad bots" lists |
429 |
Rate limited | Rate limit rules; raise the limit for verified bots |
503 |
Challenge or maintenance page | CDN "under attack" or bot fight modes |
000 |
No response (timeout, DNS, TLS) | Network level blocks, geo blocks, DNS |
A 200 with text:NO is the sneaky one. Open the raw response to see why:
curl -s -A "PerplexityBot" "$URL" | head -c 2000
If you see an empty <div id="root"></div> and script tags, your text is rendered in the browser and most AI fetchers will not see it. That problem and its fixes are in how AI crawlers handle JavaScript. If you see words like "checking your browser" or "verify you are human", your CDN is serving a challenge with a 200 status.
Check the headers too
A page can return 200 and still tell crawlers not to use it. Look for an X-Robots-Tag header and a robots meta tag:
curl -sI -A "OAI-SearchBot" "$URL" | grep -iE "^(HTTP|x-robots-tag|content-type|location)"
curl -s -A "OAI-SearchBot" "$URL" | grep -ioE '<meta[^>]+name="robots"[^>]*>'
curl -s -A "OAI-SearchBot" "$URL" | grep -ioE '<link[^>]+rel="canonical"[^>]*>'
You want no noindex or nosnippet on pages you want quoted, and a canonical that points to the page itself. Google's documentation says nosnippet also stops content being used as a direct input for AI Overviews and AI Mode, so a sitewide nosnippet added for another reason quietly removes you from those features.
What a curl test cannot tell you
A spoofed user agent from your laptop is not a real bot. Some firewalls verify bots by IP and treat a fake OAI-SearchBot from a home connection differently from the real one, in both directions: your test may be blocked while the real bot is allowed, or the reverse. Two follow up checks close the gap.
First, count real bot hits by status code in your access logs (combined log format, status in field 9):
grep -E "OAI-SearchBot|GPTBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot" access.log \
| awk '{print $9}' | sort | uniq -c | sort -rn
Mostly 200 and 304 is healthy. A pile of 403s means real bots are being refused even if your curl test passed.
Second, check that a request claiming to be a bot really came from that vendor. OpenAI, Perplexity, Anthropic and Google publish their IP ranges as JSON in the same prefixes format. This script (tested 16 November 2026) tells you which list an IP belongs to:
import ipaddress, json, sys, urllib.request
LISTS = {
"OAI-SearchBot": "https://openai.com/searchbot.json",
"GPTBot": "https://openai.com/gptbot.json",
"ChatGPT-User": "https://openai.com/chatgpt-user.json",
"PerplexityBot": "https://www.perplexity.ai/perplexitybot.json",
"Perplexity-User": "https://www.perplexity.ai/perplexity-user.json",
"Claude bots": "https://claude.com/crawling/bots.json",
"Googlebot": "https://developers.google.com/static/crawling/ipranges/common-crawlers.json",
}
ip = ipaddress.ip_address(sys.argv[1])
for name, url in LISTS.items():
req = urllib.request.Request(url, headers={"User-Agent": "ip-check/1.0"})
data = json.load(urllib.request.urlopen(req, timeout=20))
nets = [p.get("ipv4Prefix") or p.get("ipv6Prefix") for p in data.get("prefixes", [])]
if any(ip in ipaddress.ip_network(n) for n in nets if n):
print(ip, "is in the published range for", name)
break
else:
print(ip, "is not in any published range: treat the user agent as unverified")
Run python3 whichbot.py 104.210.140.130 with an IP from your logs. If your firewall allows bots by IP, feed it these lists and refresh them on a schedule, as Perplexity recommends for WAFs.
What to do after the test passes
A pass means the door is open; it does not mean you will be cited. Fix order after access: make sure the body text, tables and JSON-LD are in the raw HTML, keep one canonical per page, and keep the sitemap's lastmod dates honest. All of it is in the technical SEO for AI search checklist. If a bot failed because robots.txt itself blocks it, the robots.txt guide for AI crawlers has copy ready rules, and OAI-SearchBot vs GPTBot explains which OpenAI bot matters for search.
Articles published through Postbing to WordPress, Webflow, Ghost or our hosted blog are sent as finished HTML, so the text does not depend on browser JavaScript; your host and CDN still decide who gets a 200. See the integrations page for what each connector writes.
FAQ
What status code should AI crawlers get?
A 200 on every public page you want used in answers. A 301 to the final URL is fine. A 403, 429, 503 or a challenge page means that bot cannot read the page.
Is a curl test with a fake user agent reliable?
It reliably tests rules that match on the user agent name, which most CDN and WAF bot rules do. It cannot test rules that verify bots by IP, so also check real bot hits in your access logs.
Why does Anthropic's test line use only the bot name?
Anthropic's crawler support page does not publish full user agent strings. Sending the name tests the name based rules your firewall applies.
Can a page return 200 but still be invisible to AI crawlers?
Yes. If the text is rendered by JavaScript, the raw HTML has no article text. A noindex or nosnippet directive in a header or meta tag can also keep a page out of answers.
How often should I run this test?
After any change to your CDN, firewall, hosting or security plugins, and once a month otherwise. Bot rules can change when your CDN or plugin vendor updates its defaults, not only when you edit them.
Sources
- OpenAI, Overview of OpenAI crawlers: user agent strings and IP list URLs, accessed 16 November 2026
- Perplexity, Perplexity crawlers: user agent strings, IP lists, WAF refresh advice, accessed 16 November 2026
- Anthropic, crawler support article: bot names, IP list URL, accessed 16 November 2026
- Google, common crawlers: Googlebot smartphone user agent, accessed 16 November 2026
- Google, verifying Googlebot and other crawlers: IP range JSON files, accessed 16 November 2026
- Vercel, The rise of the AI crawler (17 December 2024): major AI crawlers do not render JavaScript, accessed 16 November 2026
- Google, robots meta tag specifications: nosnippet and AI Overviews, accessed 16 November 2026
Related reading
Last updated: November 16, 2026. Spotted an error? Email support@postbing.com and we will fix it. Editorial policy.
Try Postbing free
Postbing reads your site, plans a month of topics from what your customers search for, writes in your voice and publishes to WordPress, Shopify, Webflow or Ghost. Your first article is free.
Try Postbing freeNo card needed. No link exchanges, ever.