---
title: What is llms.txt? What it does and what it does not do
description: llms.txt is a markdown file that summarises your site for AI agents. Google says Search ignores it. Here is what it is, who reads it and whether to bother.
canonical: https://www.postbing.com/blog/what-is-llms-txt
published: 2026-11-16
updated: 2026-11-16
author: The Postbing Team
---

# What is llms.txt? What it does and what it does not do

llms.txt is a plain markdown file at the root of a website (for example `example.com/llms.txt`) that tells AI agents what the site is and which pages matter, as a short list of links with one line each. It is a proposal by Jeremy Howard, not a web standard, and Google says Google Search does not use it. It costs almost nothing to publish, so many sites add one anyway.

Last updated: November 16, 2026

## What is llms.txt, in one paragraph?

llms.txt is a curated table of contents for machines. Where `robots.txt` says what a crawler may fetch and `sitemap.xml` lists every URL, llms.txt picks the handful of pages an AI agent should read first and describes each in a sentence.

The proposal lives at llmstxt.org. It was first published on September 3, 2024 and revised as "v2" on August 10, 2026. The site describes it as "a proposal to standardise on using an /llms.txt file to provide information to help agents use a website." No standards body (IETF, W3C) has adopted it.

## What goes inside the file?

The format is ordinary markdown with a fixed order. Only the first element is required:

1. An H1 with the site or project name. The spec calls this "the only required section".
2. A blockquote with a short summary of what the site is.
3. Optional paragraphs or lists with detail an agent would need (who it is for, what it does not do).
4. One or more H2 sections, each a list of markdown links with a short note after each link.
5. An optional section literally titled "Optional" for links an agent can skip when its context is short.

Version 2 adds two ideas: publish `.md` copies of your important pages, and point to them from each HTML page with `rel="alternate" type="text/markdown"` or from an HTTP `Link:` header. The file can sit at the site root or at any path, in which case it covers the pages under that path.

You can see a filled in file, line by line, in our [llms.txt example and generator guide](/blog/llms-txt-example-and-generator).

## Does Google use llms.txt?

No. Google's AI optimization guide (published May 15, 2026, with an llms.txt note added June 15, 2026) says Google Search "doesn't use" new machine readable files, AI text files or Markdown copies, and that creating an llms.txt file "will neither harm nor help your site's visibility or rankings in Google Search, as Google Search ignores them."

The same guide's key takeaways list llms.txt among tactics you "can ignore" for Google Search. Google's separate page on AI features says there are no extra requirements to appear in AI Overviews or AI Mode: a page needs to be indexed and eligible to show with a snippet. So for Google, the work that matters is the normal work in our [technical SEO for AI search checklist](/blog/technical-seo-for-ai-search-checklist): crawlable pages, text in the HTML, clean canonicals, an accurate sitemap.

## Who does read llms.txt, then?

Coding assistants and agents that are pointed at documentation are the clearest users. A developer can paste `docs.example.com/llms.txt` into an agent and get a compact map instead of a crawl of hundreds of pages. That is the use case the proposal was written for.

For AI search engines the picture is thin. OpenAI, Anthropic and Perplexity document their crawlers and how they treat `robots.txt`, but none of their bot pages we read on 16 November 2026 mention llms.txt. The llmstxt.org page claims thousands of sites publish one and that Lighthouse checks for it; we could not verify either claim, so treat adoption numbers you see quoted as unconfirmed.

| Reader | Uses llms.txt? | What it uses instead | Status, 16 Nov 2026 |
|---|---|---|---|
| Google Search, AI Overviews, AI Mode | No | Googlebot crawl of HTML, robots.txt, sitemaps | Stated in Google's AI optimization guide |
| ChatGPT search (OAI-SearchBot) | Not documented | robots.txt rules for OAI-SearchBot | OpenAI bot docs do not mention llms.txt |
| Claude (Claude-SearchBot, Claude-User) | Not documented | robots.txt rules per bot | Anthropic crawler page does not mention llms.txt |
| Perplexity (PerplexityBot) | Not documented | robots.txt, published IP ranges | Perplexity bot docs do not mention llms.txt |
| Coding agents given a docs URL | Yes, when a user points them at it | The file itself | The original use case on llmstxt.org |

"Not documented" means the vendor's own page does not say. It does not prove the file is never fetched.

## Should you publish one?

Publish one if it takes you under an hour and you will keep it current, and do not expect it to change rankings or citations. That is the honest position. It is cheap insurance for agent use cases and harmless for Google, which ignores it.

Reasons it is still worth the hour:

- **It forces a useful exercise.** Writing one sentence per key page shows you which pages actually explain your product, your prices and what you do not do. Those are the pages AI answers quote.
- **It helps agent workflows.** If developers, partners or your own support bots read your docs through an agent, a short map saves them tokens.
- **It is low risk.** It is a static text file. It does not replace or override `robots.txt`.

Reasons to skip it or keep it tiny:

- **Stale facts hurt.** A price or plan limit in llms.txt that no longer matches your pricing page is worse than no file. If you list prices, update the file when prices change.
- **It is not access control.** Blocking or allowing AI crawlers is done in `robots.txt`, covered in our [robots.txt for AI crawlers guide](/blog/robots-txt-for-ai-crawlers).
- **It does not fix rendering.** If your pages need JavaScript to show their text, many AI fetchers see an empty shell. llms.txt pointing at those pages does not help. See [how AI crawlers handle JavaScript](/blog/javascript-rendering-and-ai-crawlers).

## llms.txt vs robots.txt vs sitemap.xml

The three files do different jobs and you need the first two far more than the third.

| File | Job | Read by | Format | Required? |
|---|---|---|---|---|
| robots.txt | Says which bots may crawl which paths | Every major search and AI crawler | Plain text directives | Effectively yes |
| sitemap.xml | Lists canonical URLs with last modified dates | Google, Bing and others | XML | Strongly recommended |
| llms.txt | Curated summary and key links for agents | Some agents and tools, not Google Search | Markdown | Optional |

If you have an hour, spend it in this order: confirm the retrieval bots get a 200 (our [curl test guide](/blog/test-ai-crawler-access-with-curl) shows how), fix `robots.txt`, make sure the sitemap has accurate dates, then write llms.txt.

## What we do on postbing.com

We publish `/llms.txt` on www.postbing.com because it costs nothing, and we do not claim it helps rankings. Ours is about 3.5 KB: an H1, a one paragraph summary, what the product does, what it will not do, prices in USD and INR, key pages and a contact line. It is served as `text/plain` with a 200. Blogs hosted on Postbing also get an llms.txt and an llms-full.txt generated for them. If you want the details of what we write and publish, start at the [Postbing home page](/).

## FAQ

### Is llms.txt an official standard?
No. It is a proposal published at llmstxt.org by Jeremy Howard in September 2024 and revised as v2 in August 2026. No standards body has adopted it.

### Does llms.txt help SEO on Google?
No. Google's AI optimization guide says Google Search ignores llms.txt and that publishing one will neither harm nor help visibility or rankings in Google Search.

### Does ChatGPT or Perplexity read llms.txt?
Their official crawler documentation, read on 16 November 2026, does not mention llms.txt. They control access through robots.txt rules for OAI-SearchBot and PerplexityBot.

### Can llms.txt block AI crawlers?
No. llms.txt only describes your site. To allow or block a crawler you need rules in robots.txt for that bot's user agent token.

### Where should the file live?
At the root, for example `example.com/llms.txt`. The v2 proposal also allows a file at a sub path, such as `/docs/llms.txt`, covering only the pages under that path.

## Sources

- [llmstxt.org, The /llms.txt file, v2](https://llmstxt.org/): format, author, publish and v2 dates, proposal status, accessed 16 November 2026
- [Google Search Central, AI optimization guide](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide): Google Search ignores llms.txt, neither helps nor harms, accessed 16 November 2026
- [Google Search Central, AI features and your website](https://developers.google.com/search/docs/appearance/ai-features): no extra requirements for AI Overviews and AI Mode, accessed 16 November 2026
- [Google Search Central, documentation updates](https://developers.google.com/search/updates): 15 June 2026 note on llms.txt, accessed 16 November 2026
- [OpenAI, Overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots): crawler controls via robots.txt, accessed 16 November 2026
- [Anthropic, Does Anthropic crawl data from the web](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler): crawler controls via robots.txt, accessed 16 November 2026
- [Perplexity, Perplexity crawlers](https://docs.perplexity.ai/guides/bots): crawler controls via robots.txt, accessed 16 November 2026

## Related reading

- [Technical SEO for AI search: the full checklist](/blog/technical-seo-for-ai-search-checklist)
- [llms.txt example, template and generator steps](/blog/llms-txt-example-and-generator)
- [How to set robots.txt rules for GPTBot, ClaudeBot and other AI crawlers](/blog/robots-txt-for-ai-crawlers)
- [Generative engine optimization guide](/blog/generative-engine-optimization-guide)
- [How Postbing publishes articles with schema and llms.txt included](/)
