---
title: Technical SEO for AI search: the 2026 checklist, in order
description: The checks that decide whether Google, ChatGPT, Claude and Perplexity can read and cite your pages: bots, robots.txt, rendering, canonicals, sitemaps, schema.
canonical: https://www.postbing.com/blog/technical-seo-for-ai-search-checklist
published: 2026-11-16
updated: 2026-11-16
author: The Postbing Team
---

# Technical SEO for AI search: the 2026 checklist, in order

Technical SEO for AI search is the work that lets AI assistants and Google fetch, read and quote your pages. In order: let the retrieval bots in (a 200 for OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot), put the full text in the raw HTML, keep one canonical URL per page, publish an accurate sitemap, then add structured data. llms.txt is optional. None of it guarantees citations.

Last updated: November 16, 2026

## Why order matters

Each step only helps if the one before it passes. Schema on a page that returns 403 to OAI-SearchBot does nothing for ChatGPT; a perfect sitemap does not help if the pages it lists are empty shells without JavaScript. Work top to bottom and stop to fix the first failure.

Google's own position keeps this list short for Google: its AI features page says "There are no additional requirements to appear in AI Overviews or AI Mode, nor other special optimizations necessary", and a page must simply be "indexed and eligible to be shown in Google Search with a snippet". The AI assistants add one thing on top: their own crawlers, with their own robots.txt tokens and firewall behaviour.

## The checklist at a glance

| # | Check | How to test | Pass | Detailed guide |
|---|---|---|---|---|
| 1 | Retrieval bots get a 200 | curl with each bot's user agent | 200 on home, blog, one article | Curl test guide |
| 2 | robots.txt allows the right bots | Read the live file | Retrieval bots allowed on public paths | Robots.txt guide |
| 3 | Text is in the raw HTML | curl and grep a body sentence | Sentence found | JavaScript guide |
| 4 | One canonical URL per page | curl the canonical tag, test host variants | Self canonical, variants 301 | This page |
| 5 | Accurate sitemap | Sitemaps report, lastmod spot check | Success, real dates | Sitemap guide |
| 6 | Indexing status of key pages | Page indexing report | Key pages indexed | Not indexed guides |
| 7 | Structured data in the HTML | curl and grep the JSON-LD types | Article and BreadcrumbList present | Schema guides |
| 8 | Snippet and AI controls set on purpose | Headers, meta robots, Search Console | No accidental nosnippet | This page |
| 9 | Optional extras | IndexNow key, llms.txt | Served with 200 | IndexNow, llms.txt guides |

## 1. Let the retrieval bots in

Two families of bots read your site, and only one decides whether you appear in answers. Retrieval bots (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Googlebot, Bingbot) fetch pages for search and live answers. Training bots (GPTBot, ClaudeBot, Google-Extended) collect content for model training. Block a retrieval bot and you drop out of that assistant; block a training bot and, per each vendor, search is unaffected.

Robots.txt is not the only door. CDN bot rules, WAFs, security plugins and rate limiters can return 403, 429 or a challenge page to bots robots.txt allows. The quick test, for three URLs:

```bash
UA="Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)"
for u in https://www.example.com/ https://www.example.com/blog https://www.example.com/blog/some-article; do
  curl -s -o /dev/null -w "%{http_code}  $u\n" -A "$UA" "$u"
done
```

Repeat with each bot's user agent. Expect 200 everywhere. The [curl test for AI crawler access](/blog/test-ai-crawler-access-with-curl) has a loop for nine bots, a text check, a log query and an IP verification script.

## 2. Get robots.txt right for each bot

Name the bots you care about and repeat your private paths in their group. A bot that matches its own group ignores the `*` group: Google picks "the group with the most specific user agent" and says "Other groups are ignored", and RFC 9309 applies `*` only when "no matching group exists". Changes are not instant: Google caches robots.txt "for up to 24 hours", and OpenAI says its systems need "~24 hours".

Know what each token controls. Google-Extended covers Gemini training and grounding, and Google says it "does not impact a site's inclusion in Google Search". OAI-SearchBot controls ChatGPT search; GPTBot controls training. Full rules and a copy ready file are in the [robots.txt guide for AI crawlers](/blog/robots-txt-for-ai-crawlers), with vendor detail in [OAI-SearchBot vs GPTBot](/blog/oai-searchbot-vs-gptbot) and [ClaudeBot, Claude-SearchBot and PerplexityBot](/blog/claudebot-claude-searchbot-perplexitybot).

Also check the file itself returns 200. A robots.txt that returns 5xx makes Google stop crawling the site for the first 12 hours, and RFC 9309 tells crawlers to "assume complete disallow" while it is unreachable.

## 3. Put the text in the raw HTML

Most AI crawlers read only the HTML your server sends. Vercel's crawler analysis (17 December 2024) found "none of the major AI crawlers currently render JavaScript", naming the OpenAI, Anthropic and Perplexity bots. Googlebot does render, after a queue, and Google still says "not all bots can run JavaScript".

Test it on a real page:

```bash
curl -s https://www.example.com/pricing | grep -c "a sentence from the middle of the page"
```

A 0 means the text is built in the browser. Check prices, plan tables, FAQ answers and JSON-LD especially, since those are what answers quote. Fixes, from static generation to prerendering, are in [how AI crawlers handle JavaScript](/blog/javascript-rendering-and-ai-crawlers).

## 4. One canonical URL per page

Every page should have exactly one URL that all signals point to. Google treats canonical preferences as "a hint, not a rule", and lists the signals with their strength: redirects are "a strong signal", `rel="canonical"` is "a strong signal", and sitemap inclusion is "a weak signal". It adds that "these methods can stack". So use all three together.

The checks:

```bash
curl -s https://www.example.com/blog/post | grep -ioE '<link[^>]+rel="canonical"[^>]*>'
curl -sI http://example.com/blog/post | grep -iE "^(HTTP|location)"
curl -sI https://www.example.com/blog/post/ | grep -iE "^(HTTP|location)"
```

Pass: the canonical points at the page itself; the `http`, non `www` (or `www`) and trailing slash variants all 301 to the one version; internal links and the sitemap use that same version. Mixed hosts and slash variants split signals across URLs that say the same thing.

## 5. Publish an accurate sitemap

A sitemap tells crawlers what exists and what changed. Keep it to final, canonical, 200 URLs, under Google's limits of 50 MB or 50,000 URLs per file, and reference it in robots.txt with a `Sitemap:` line. Submit it once in Search Console.

The detail most sites get wrong is `lastmod`. Google ignores `priority` and `changefreq` and uses `lastmod` only if it is "consistently and verifiably" accurate. Set it to the real last edit, and keep it equal to the visible "Last updated" date and the `dateModified` in your Article schema. Never stamp every URL with the build date. Steps, statuses and a checker script are in [how to submit a sitemap to Google Search Console](/blog/submit-sitemap-to-google-search-console).

## 6. Watch indexing status, not just crawling

Google cites only indexed pages, so check the Page indexing report for your key URLs. Two statuses need different fixes:

| Status | What it means | Usual cause | Guide |
|---|---|---|---|
| Discovered, currently not indexed | Known URL, not yet crawled | Crawling expected to overload the site; too many low value URLs | [Discovered, currently not indexed fix](/blog/discovered-currently-not-indexed-fix) |
| Crawled, currently not indexed | Crawled, not added to the index | Page not distinct or useful enough, duplicates, thin template output | [Crawled, currently not indexed fix](/blog/crawled-currently-not-indexed-fix) |

Google is clear that "Don't expect every URL on your site to be indexed". Make sure the pages that sell or answer buyer questions are in, and accept that tag archives and filters may not be.

For Bing and other participating engines, [IndexNow](/blog/what-is-indexnow) lets you push changed URLs the moment they publish. Google does not take part, and its old sitemap ping endpoint now returns 404.

## 7. Add structured data in the HTML

Structured data labels the page; it is not an AI shortcut. Google says "There's also no special schema.org structured data that you need to add" for AI Overviews and AI Mode, and its AI optimization guide says structured data "isn't required for generative AI search". It is still worth having for accurate titles, dates and breadcrumbs.

What we put on every article:

- **Article or BlogPosting** with headline, image, datePublished, dateModified and an author with a name and url. Google lists no required properties. See the [Article schema guide](/blog/article-schema-json-ld-guide).
- **BreadcrumbList** matching the visible trail. Google shows breadcrumbs in desktop results only. See the [breadcrumb schema guide](/blog/breadcrumb-schema-guide).
- **FAQPage**, only if generated from a visible FAQ. Google stopped showing FAQ rich results on May 7, 2026; see [FAQ schema in 2026](/blog/faq-schema-guide) for why we keep it anyway.

Render all of it on the server. A crawler that does not run JavaScript cannot see JSON-LD injected by a tag manager.

## 8. Set snippet and AI controls on purpose

Some controls quietly remove you from AI answers, so check none is set by accident. On Google, `nosnippet` "will also prevent the content from being used as a direct input for AI Overviews and AI Mode", and `max-snippet` limits how much can be used. Search Console also has a "Search generative AI control" to opt a whole site out of AI Overviews and AI Mode (Google's page on it was last updated 21 October 2026). Google-Extended does not do this; it covers Gemini training and grounding.

Check headers and meta tags on a sample of pages:

```bash
curl -sI https://www.example.com/blog/post | grep -i x-robots-tag
curl -s https://www.example.com/blog/post | grep -ioE '<meta[^>]+name="(robots|googlebot)"[^>]*>'
```

If you see `nosnippet`, `noindex` or a small `max-snippet` on pages you want quoted, find out who set it and why.

## 9. Optional extras: llms.txt and performance

llms.txt is a markdown summary of your site for agents. Google says Google Search ignores it and that it "will neither harm nor help" rankings; OpenAI, Anthropic and Perplexity do not mention it in their crawler docs. Publish one if it takes an hour and you will keep it current. See [what llms.txt is](/blog/what-is-llms-txt) and the [llms.txt example and generator](/blog/llms-txt-example-and-generator).

Performance matters through the crawl budget rather than as an AI signal. Google says that if a site slows down or returns 5xx or 429 responses, "the limit goes down and Google crawls less", and that faster pages let it "read more content from your site". Our own rule of thumb for articles: no render blocking third party scripts, images with width and height set, lazy loading below the fold, and article HTML under about 150 KB.

## How to measure the result

Measure whether you are named and cited, not only rankings. In Google, AI feature traffic is counted "within the 'Web' search type" of the Search Console Performance report, so watch impressions and clicks for the pages you fixed. For assistants, run a fixed set of buyer questions in ChatGPT, Claude, Perplexity and Gemini each week, logged out, and record whether you are mentioned and which URLs are cited. Postbing's AI visibility feature is built to ask your site's buyer questions to those four assistants through their APIs each week and record mentions and citations. The number of questions per site depends on the plan.

## FAQ

### What is technical SEO for AI search?
It is the set of checks that let AI crawlers and Google fetch, read and quote your pages: bot access, robots.txt, server rendered text, canonical URLs, sitemaps and structured data. It is ordinary technical SEO plus the AI vendors' own crawlers.

### Which bots must I allow to appear in AI answers?
The retrieval bots: OAI-SearchBot and ChatGPT-User for ChatGPT, Claude-SearchBot and Claude-User for Claude, PerplexityBot for Perplexity, and Googlebot for AI Overviews and AI Mode. Training bots such as GPTBot and ClaudeBot are a separate choice.

### Do I need special schema for AI Overviews?
No. Google says there is no special schema.org structured data needed to appear in AI Overviews or AI Mode. Article and breadcrumb markup still help Google understand the page.

### Is llms.txt required for AI search?
No. Google says Google Search ignores llms.txt, and the major AI crawler docs do not mention it. It is optional and cheap to publish.

### Does passing this checklist guarantee AI citations?
No. It makes your pages readable and eligible. Whether an assistant cites you depends on how well each page answers the question compared with other pages.

## Sources

- [Google Search Central, AI features and your website](https://developers.google.com/search/docs/appearance/ai-features): no extra requirements, indexing and snippet eligibility, no special schema, Web search type reporting, accessed 16 November 2026
- [Google Search Central, AI optimization guide](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide): llms.txt ignored, structured data not required, accessed 16 November 2026
- [Google Search Central, control what you share](https://developers.google.com/search/docs/crawling-indexing/control-what-you-share): Search generative AI control, accessed 16 November 2026
- [Google Search Central, robots meta tag](https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag): nosnippet and max-snippet and AI features, accessed 16 November 2026
- [Google, robots.txt specification](https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec): group matching, 24 hour cache, 5xx handling, accessed 16 November 2026
- [RFC 9309, Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.txt): `*` fallback, unreachable means disallow, accessed 16 November 2026
- [Google, common crawlers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers): Google-Extended scope, accessed 16 November 2026
- [OpenAI, Overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots): OAI-SearchBot, GPTBot, 24 hour delay, accessed 16 November 2026
- [Vercel, The rise of the AI crawler](https://vercel.com/blog/the-rise-of-the-ai-crawler): AI crawlers do not render JavaScript, published 17 December 2024, accessed 16 November 2026
- [Google Search Central, JavaScript SEO basics](https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics): not all bots run JavaScript, accessed 16 November 2026
- [Google Search Central, canonicalization](https://developers.google.com/search/docs/crawling-indexing/canonicalization) and [consolidate duplicate URLs](https://developers.google.com/search/docs/crawling-indexing/consolidate-duplicate-urls): canonical as a hint, signal strengths, accessed 16 November 2026
- [Google Search Central, build and submit a sitemap](https://developers.google.com/search/docs/crawling-indexing/sitemaps/build-sitemap): limits, lastmod, accessed 16 November 2026
- [Google Search Console Help, Page indexing report](https://support.google.com/webmasters/answer/7440203?hl=en): not every URL indexed, accessed 16 November 2026
- [Google, Optimize your crawl budget](https://developers.google.com/crawling/docs/crawl-budget): server errors lower crawling, faster pages, accessed 16 November 2026
- [Google Search Central, documentation updates](https://developers.google.com/search/updates): FAQ rich result removal, accessed 16 November 2026

## Related reading

- [Generative engine optimization guide](/blog/generative-engine-optimization-guide)
- [AI search optimization checklist for content](/blog/ai-search-optimization-checklist)
- [How Postbing writes and publishes articles for Google and AI answers](/)
