LLMs.txt: The New Robots.txt for AI Search

What llms.txt actually is

llms.txt is a plain-text file, written in Markdown, that lives at the root of your website — yoursite.com/llms.txt — and gives large language models a curated, human-readable map of the pages you most want them to read and cite. It was proposed by Jeremy Howard of Answer.AI in September 2024, and the specification lives at llmstxt.org.

The problem it tries to solve is simple. When an AI assistant retrieves a web page, it doesn’t see the clean article a human reads. It sees raw HTML stuffed with navigation, cookie banners, ad slots, related-post widgets, and JavaScript scaffolding. Context windows are finite. The model has to spend tokens parsing all of that noise before it reaches the sentence that actually answers the user. llms.txt is an attempt to skip the noise: a hand-picked, Markdown-clean index that says “if you only read a few pages on this site, read these, and here’s what each one is about.”

The mental model I use with clients: a sitemap is written for search-engine crawlers and lists every URL; llms.txt is written for language models and lists only the URLs that matter, each with a one-line description of why it matters. One is exhaustive and machine-indexed. The other is curated and machine-read.

llms.txt vs robots.txt: opposite jobs

The “new robots.txt” framing is useful, but only if you understand that the two files do opposite things. They are not competitors and they are not substitutes. They sit at different layers of how machines interact with your site.

• robots.txt — a permission and exclusion file, standardised since 1994 and honoured by virtually every reputable crawler. It tells bots what they may not access (Disallow: /admin/) and which AI crawlers you do or don’t welcome (User-agent: GPTBot). It is about gatekeeping.
• llms.txt — a curation and inclusion file, proposed in 2024 and still gaining adoption. It tells AI models what they should prioritise. It is about wayfinding.

Here is the practical relationship. Your robots.txt decides whether an AI crawler such as GPTBot, OAI-SearchBot, PerplexityBot, or ClaudeBot is allowed in at all. If you have blocked those user-agents in robots.txt — which many sites do without realising the consequence — then your llms.txt is irrelevant, because the model you wrote it for is never coming. So the sequence is: open the gate in robots.txt first, then lay out the welcome map in llms.txt. I cover the crawler-access side in more depth in how to rank in ChatGPT, Perplexity, and Google AI Overviews.

One more distinction that trips people up: robots.txt is honoured as a near-universal convention; llms.txt is not yet. Google’s own crawling-and-indexing documentation describes how robots.txt works in detail because it is load-bearing. There is no equivalent official commitment to llms.txt from Google, OpenAI, or Anthropic. That gap matters, and I’ll be straight about it below.

Does anyone actually read it? The honest evidence

This is the question that separates a useful guide from hype, so I’ll answer it plainly: as of mid-2026, no major AI provider has publicly confirmed that it uses llms.txt as a live retrieval or ranking input. Google has said it does not use llms.txt for Search. OpenAI and Anthropic have made no commitment to honour it. There is no published, controlled study showing that adding an llms.txt file measurably increases citation rate in ChatGPT or Perplexity.

What is true:

• Adoption is real but concentrated. Developer-documentation platforms, AI-native tools, and technical-product companies have adopted llms.txt widely. Tools like Mintlify auto-generate it. The standard has momentum in the segments where AI assistants are most used for research.
• The inference layer is different from the index layer. Even if Google Search ignores llms.txt, an AI agent ingesting your site at inference time — via a coding assistant, a retrieval pipeline, or a developer pasting your /llms.txt URL into a prompt — can and does use it. That use is invisible to your analytics, which is exactly why it’s hard to prove with a clean before/after number.
• The cost is near zero. A good llms.txt takes an hour to write and a minute to deploy. The risk of harm is low (with one exception I’ll cover). So the expected-value math is asymmetric: small, known cost; uncertain but potentially meaningful upside if and when the engines formalise support.

My position, the one I give paying clients: build the file, keep it accurate, and do not expect it to move citation rate on its own. It is a credibility signal and a hedge, not a campaign. The things that actually drive AI citations — entity coherence, schema completeness, answer-first content structure, and third-party brand mentions — are covered in the 7 AI search ranking factors, and that is where the real leverage sits. llms.txt is the cheap finishing touch you add once the foundations are in place.

How to write an llms.txt file (the structure)

The format is deliberately minimal. It is Markdown, parsed loosely, with a small set of conventions from the spec. Here is the anatomy, top to bottom.

• H1 — your brand or project name. Exactly one. This is the entity the file is about.
• Blockquote — a one-line summary. A single > line that says, in plain language, what you do. This is the description a model is most likely to lift verbatim, so write it like the sentence you’d want ChatGPT to say about you.
• Optional context paragraphs. A few sentences of additional framing, with no headings, before the link sections begin.
• H2 sections — grouped links. Each section groups related pages. Inside, a Markdown bulleted list of links in the form [Title](https://absolute-url) — short description.
• An optional ## Optional section. The spec reserves a section literally named “Optional” for links a model can skip if it’s short on context. Put nice-to-have pages here.

A minimal, correct example for a consultant’s site looks like this:

• # Akshay Nigam
• > AI-powered digital growth consultant and AI-visibility architect helping founders get cited by AI search engines.
• ## Core guides
• - [What is GEO?](https://aknigam.com/insights/what-is-geo) — the definitive 2026 guide to generative engine optimization.
• - [AI Search Ranking Factors](https://aknigam.com/insights/ai-search-ranking-factors) — the seven factors that drive AI citations.
• ## Services
• - [AI Visibility](https://aknigam.com/system/visibility) — the consulting programme for getting cited by AI.

Three rules I enforce on every file I write. First, use absolute, canonical URLs — never relative paths and never a URL that 301-redirects, because every redirect hop is a chance for a model to grab the wrong canonical. Second, keep descriptions factual and tight; this is not ad copy, it’s metadata. Third, list only pages you are proud to be cited from. llms.txt is a curation file — if you dump your whole sitemap into it, you’ve missed the point and diluted the signal.

llms.txt vs llms-full.txt

You’ll see a companion file referenced: llms-full.txt. The distinction is worth understanding before you build either.

• llms.txt is the index. Links and descriptions. A map. Small. Start here.
• llms-full.txt is the corpus. The full Markdown text of your priority pages, concatenated into one file, so a model can ingest the actual content in a single fetch without crawling each page individually.

For most sites, llms.txt alone is the right move. Add llms-full.txt only when your priority content is stable and you genuinely want to make direct ingestion frictionless — documentation sites and reference libraries are the natural fit. The trap with llms-full.txt is staleness: a giant text file of your content frozen at one moment in time will drift out of sync with your live pages, and now you’re maintaining two sources of truth. If you can’t commit to keeping it current, don’t ship it.

A real mistake worth learning from

I’ll show my own work here, because the failure is more instructive than the success. When I audited the llms.txt on my own site, I found that the “Services hub” line pointed AI models at a URL that 404’d — it referenced /services, but the real hub had moved to /ai-services. The file had been written once and never re-audited against the live site.

Sit with why that’s bad. The entire reason you publish an llms.txt is to feed AI models accurate canonical URLs so they cite you correctly. A broken link in that file is the worst of both worlds: you’ve gone to the trouble of curating a map, and then the map points off a cliff. A model that follows it either fails to retrieve the page or, worse, falls back to a stale cached version and describes you wrong. A broken llms.txt is worse than no llms.txt, because it converts good intent into a confident bad signal.

The fix was trivial — correct the URL, restructure the file into clear “System” and “Services” sections, confirm the file was saved as UTF-8 so the em-dashes rendered, and purge the server cache so the corrected version was actually served. But the lesson is procedural, not technical: audit your llms.txt links on the same cadence you audit your sitemap. Every URL must resolve, every URL must be canonical, and the file must reflect the site as it is today, not as it was the day you wrote it.

How to deploy and verify it

Deployment is the easy part; verification is where people get lazy. The full loop:

• Upload to root. Place the file so it resolves at https://yoursite.com/llms.txt — site root, not a subfolder. On WordPress you can drop it in public_html directly, since it’s a static file the CMS doesn’t need to render.
• Check it serves as plain text. Open the URL in an incognito window. It should render as raw text, return a 200 status, and use UTF-8 encoding so special characters don’t turn into mojibake.
• Purge your cache. Static .txt files are aggressively edge-cached. If you’re on LiteSpeed, Cloudflare, or any CDN, an edit can sit invisible behind a stale cache for hours. Purge after every change and re-verify in incognito — I learned this the hard way.
• Link it from your <head>. Add a discoverable reference so models and tools can find it without guessing the path.
• Re-audit the links. Click every URL. Confirm each is live and canonical. Do this monthly, or whenever your site structure changes.

One thing not to do: don’t register your llms.txt as a sitemap in robots.txt or Google Search Console. It is not an XML sitemap, and feeding it to GSC as one throws a format error. Keep your real sitemap_index.xml for search engines and let llms.txt be discovered via the head link and the root path. I made and corrected that exact mistake on my own robots.txt — the second Sitemap: line pointing at llms.txt had to go.

Where llms.txt fits in a real AI-visibility programme

I want to be careful not to oversell a one-hour file. In my work, llms.txt is a step, not a strategy. It belongs in the same workstream as your schema graph, your entity coherence, and your crawler-access policy — the plumbing layer of generative engine optimization. It is genuinely useful as a credibility and wayfinding signal, and it costs almost nothing. But if you publish an llms.txt and skip the real work, you’ve polished the doorknob on a house with no foundation.

The honest priority order I’d give any founder: (1) get your crawler access right so AI bots can reach you at all; (2) build a complete, valid schema graph so models understand what your entities are; (3) restructure your priority pages answer-first so they’re lift-able into a synthesized answer; (4) earn third-party brand mentions so models see external corroboration; and then (5) add llms.txt as the curated map on top. If you want to see whether your foundations are in place before you bother with the doorknob, my AI Visibility Audit Worksheet walks you through the self-assessment, and the deeper engineering playbook is in the practical guide to GEO and AEO.

And measure it like everything else. You won’t see llms.txt in your rank tracker, and you mostly won’t see it in GA4. The signal shows up as AI-engine referral sessions and as citation appearances in your manual query-set testing — the kind of attribution discipline I describe in the measurement audit. If you can’t measure citations, you can’t tell whether any GEO tactic, llms.txt included, is working.

The bottom line

llms.txt is real, it’s cheap, and it’s sensible — a curated, machine-readable map that says read these pages first. It is the inclusion-side complement to robots.txt’s exclusion side. It is not a confirmed ranking signal, no major engine has committed to honouring it as one, and it will not, on its own, get you cited by ChatGPT or Perplexity. Build it because it’s good hygiene and an asymmetric hedge, keep every link in it live and canonical, and put the hard hours into the foundations that actually move citation rate. Do that, and llms.txt becomes the tidy finishing layer on a system that’s already working — which is exactly what it should be.