llms.txt: What It Is and Whether It Actually Works

llms.txt is a proposed standard: a markdown file at your site root that gives AI systems a curated index of your most important content. The honest status report is that adoption by site owners has been enthusiastic, while confirmed consumption by the major AI engines remains unproven. It costs little, might help, and should sit far below crawlability and content structure on your priority list.

That is the summary. The details matter, because llms.txt has become a litmus test for whether GEO advice is evidence-based or cargo cult.

What the spec actually says

The llms.txt specification, proposed by Jeremy Howard in late 2024, defines a markdown file at /llms.txt with an H1 title, a short summary, and sections of curated links with one-line descriptions:

# Example Corp

> B2B expense management for mid-market teams.

## Docs

- [Quickstart](https://example.com/docs/quickstart): Set up in ten minutes
- [API reference](https://example.com/docs/api): Full endpoint documentation

## Product

- [Pricing](https://example.com/pricing): Plans and what each includes

The reasoning is sensible: context windows are finite, HTML is noisy, and a curated map could help an AI system find canonical answers fast. A companion convention serves markdown versions of pages at .md URLs.

What the evidence says

Site-owner adoption tells us nothing about engine behavior, so the load-bearing question is whether AI crawlers request the file. On that question the record is thin: no major engine has documented that it consumes llms.txt, and publicly shared log experiments from sites that deployed it report little to no fetch activity from the major AI crawlers. Google's search documentation for AI features points to standard indexing and says nothing about llms.txt, and neither the OpenAI nor Perplexity crawler docs mention the file.

Weigh that against the cost side: the file is cheap to generate, trivial to serve, and harmless under every documented crawler behavior. Some AI-adjacent tools and smaller agents do read it, and nothing prevents major engines from adopting it later.

An honest recommendation

Ship it if it costs you an hour; skip the vendor packages that sell it as a visibility strategy:

• Generate it from your sitemap or docs tree so it stays current without manual upkeep. Stale curation is worse than none.
• Point it at your genuinely canonical pages: docs, pricing, comparison pages, the content you want quoted.
• Do not report it as an optimization. Until engines document consumption, llms.txt is a lottery ticket, and lottery tickets do not belong in strategy decks.

The work that measurably feeds AI answers sits elsewhere: pages the crawlers can fetch and parse, structure that extracts cleanly, and presence on the third-party sources engines cite. That ordering is the whole argument of our GEO audit guide, and the crawler behavior underlying it is documented in the marketer's guide to AI crawlers.

How to test it yourself

The llms.txt debate is settleable per-site with your own data:

1. Deploy the file.
2. Watch for fetches of /llms.txt by AI user agents. Edge-level crawler tracking makes this visible without log spelunking.
3. Watch your mention and citation rates across repeated runs for the weeks before and after, per the sampling method in why AI answers change.
4. Report what you find, either way.

If the bots never request the file, you have your answer for your site, this quarter, and it cost you nothing but the hour.

FAQ

Is llms.txt like robots.txt? Opposite jobs. Robots.txt restricts crawler access and is a documented standard engines parse. llms.txt invites attention to curated content and awaits adoption.

Does Google use llms.txt? Google has not documented any use of it for search or AI surfaces, and its guidance directs sites to standard crawlability and indexing.

Should I generate .md versions of my pages too? If your stack makes it free, it is a reasonable companion experiment. Measure fetches before crediting it with anything.

Measure the experiment instead of debating it

Citlyze shows whether AI crawlers fetch your llms.txt and whether your answer visibility moved, so the debate becomes a readout. Crawler analytics and prompt tracking ship together on every plan, from $29/month at citlyze.com/pricing.