llms.txt: The Proposed AI-Readable Site Summary, and Its Adoption Gap

llms.txt is a convention proposed in 2024 by Jeremy Howard of Answer.AI for a Markdown file served at a site's root path, `/llms.txt`. It offers a curated, machine-readable overview of the site's most important material — key pages, documentation, API references — so that a language model or agent can orient itself without crawling and parsing the entire site. The motivation is genuine. HTML pages carry navigation, styling and advertising that is noise to a model, and context windows are finite. A hand-curated Markdown index is cheaper to consume, and it lets a site owner state what matters instead of leaving that to a crawler's heuristics. Adoption is where the picture diverges sharply from the advice written about it, and this is the part most write-ups omit: - Thousands of sites now serve the file, including a number of prominent technology companies. - **No major AI provider has confirmed that its crawlers consume it normatively.** In July 2025 a Google Search representative stated that Google does not support llms.txt and had no plans to, with a colleague likening it to the long-discredited keywords meta tag. - Measurement supports the scepticism: in one study spanning roughly 500 million AI bot visits over a 90-day window, only a few hundred requests targeted `/llms.txt` at all. The honest characterisation is therefore a **proposed convention with real supply and almost no demonstrated demand** — not a standard, and not something answer visibility currently depends on. Publishing one costs little and may position a site for a future crawling protocol, but treating it as a citation or ranking lever is unsupported by anything public. The shape of this is familiar: a plausible, cheap, easy-to-recommend technical addition accumulates a body of confident advice well ahead of any evidence that it works. Compare Why the Semantic Web Underperformed Expectations, where machine-readable markup similarly outran the consumers meant to use it, and The Vendor Echo Chamber: When Every Search Result Is the Vendor's Own Marketing for how such advice becomes self-reinforcing. For markup that search engines demonstrably do consume, see Rich Result (Search) and JSON-LD.