What llms.txt is and who actually reads it

llms.txt is an optional Markdown file that gives a website’s name, a short summary and links to its most useful pages, so AI agents can find them without crawling the whole site. A bot reads the file only when it requests that address, and in Ahrefs’ server-log study of May 2026, 97% of published llms.txt files received no request at all.

What is llms.txt?

llms.txt is a Markdown index, at /llms.txt or under a path such as /docs/llms.txt, that tells AI agents what a site covers and which pages hold the detail. The llms.txt proposal, published by Jeremy Howard on 3 September 2024 and revised on 10 August 2026, makes one part mandatory: an H1 heading with the name of the site or project.

Everything else is optional: a blockquote summary and sections of links under H2 headings, each link with a line on what the page holds. A section named Optional lists links an agent can skip when it needs a shorter context. The proposal also asks sites to serve a Markdown copy of each page agents may need, at the same address with .md added.

Jeremy Howard, co-founder of Answer.AI and fast.ai, designed the file for agents that need a site’s facts while they help a person. “Our expectation was that llms.txt would mainly be useful for inference rather than training, and that is how it has been used,” he wrote in the proposal, which names software documentation, read by coding agents, as the main use.

Ahrefs measured how far the file has spread: 28% of the 137,210 domains in its May 2026 study published a valid llms.txt file. Ahrefs calls that share an upper bound, since the sites using its analytics tool lean more technical and SEO-focused than the average website.

Who requests llms.txt files

SEO audit tools request llms.txt files more than any other visitor, and the bots that fetch pages for live AI search answers request them least. Ahrefs’ study of 137,210 domains, published by Louise Linehan on 15 June 2026, sorted every llms.txt request of May 2026, about 22,000 in all, by the user agent that sent it:

SEO audit tools sent 21.7% of the requests, other and unidentified bots 14.9%, general web crawlers 13.1% and tech profiling tools 11.6%. AI agents and agentic infrastructure sent 10.5%, GEO/AEO scoring tools 5.8%, AI training crawlers 5.3% and llms.txt discoverability bots 3.6%. Service and social bots, which build link previews, sent 2.9%, research bots 2.7%, AI assistants fetching for one user 2.5% and AI retrieval bots 1.1%, while humans made 4%.

The four AI groups sent 19.5% of requests between them. OpenAI’s training crawler GPTBot made 4.51% on its own, and OAI-SearchBot, the crawler that getting cited in ChatGPT answers depends on, made 0.74%. Ahrefs also found that Slackbot, a chat app’s link-preview bot, requested llms.txt files more often than PerplexityBot did.

How many llms.txt files get read

Only 3% of llms.txt files drew any request in May 2026, in Ahrefs’ study of 137,210 domains. Of the 38,360 domains with a valid file, about 1,100 received every llms.txt request Ahrefs measured, and the other 97% received none at all:

28% of the domains published the file, and 3% of those files drew any visitor.

In Ahrefs’ data, no AI bot requested an llms.txt file that did not exist. Requests for missing files came 98% from humans, and the AI bot share of those requests was zero. Ahrefs concludes that AI tools fetch llms.txt when a link, an index or a user instruction points them to it.

Ahrefs notes that a fetch does not show a bot acted on the file, so every share in its study is a ceiling. A site can test its own case by filtering its server logs for /llms.txt, the same logs that record AI crawler visits and whether ChatGPT cited the site.

What AI companies say about llms.txt

Google Search ignores llms.txt files, and Google’s guide to optimizing for generative AI features, last updated on 10 July 2026, says such a file makes no difference, good or bad, to a site’s visibility or rankings there. The guide adds that site owners need no new AI text files, markup or Markdown to appear in Google Search, its AI features included.

The same guide calls it fine to keep llms.txt for other services, and how Google AI Overviews choose pages comes down to indexing and snippet eligibility. Chrome’s Lighthouse checks for the file in its agentic browsing audits, and the Lighthouse documentation flags a page only when llms.txt returns a server error, marks a missing file as not applicable and calls the file optional for now.

OpenAI ties ChatGPT search to robots.txt. OpenAI’s crawler documentation says sites opted out of OAI-SearchBot do not appear in ChatGPT search answers, and the page points to OpenAI’s own llms.txt only as an index of its developer docs. Peec AI’s review of llms.txt, written by its CPO Malte Landwehr on 30 July 2025, found that no LLM-driven search or answer engine had announced support for the protocol.

What to fix before llms.txt

Access to the site comes first, because a crawler kept out by robots.txt, a CDN or a firewall reads neither the pages nor the llms.txt file. The robots.txt rules and firewall checks for each AI bot are set out in the AI crawlers list.

Readable page text comes second. AI crawlers fetch pages directly and in large numbers, and Vercel’s study of AI crawler traffic, published on 17 December 2024, found that none of the major AI crawlers rendered JavaScript. Vercel counted these fetches across its network in the month before:

Googlebot made 4.5 billion fetches across Gemini and Search, GPTBot 569 million, Claude 370 million, AppleBot 314 million and PerplexityBot 24.4 million. Googlebot and AppleBot rendered JavaScript and the other three did not. Each page’s answer therefore belongs in its HTML, near the top, because Claude search passes about an eighth of a page to its model.

llms.txt comes last. An llms.txt guide on agenzy.lt sets the order: robots.txt, then the CDN and firewall, then schema and answers in page text, then the file itself. That technical work lets AI crawlers in, and more frequent recommendations come from content and from mentions on other sites, the work described in what GEO means for brands.

What it means

The head of marketing should have the web team clear access first and leave llms.txt for last, since the file does not raise AI visibility and a proposal that sells it as the fix sells something false. In order:

1. Check robots.txt, CDN and firewall rules for the AI search crawlers and fetchers, and read the server logs for refused requests.
2. Check that each main page answers its question in HTML text near the top, readable without JavaScript.
3. Publish llms.txt where AI agents or coding tools use the site, such as API documentation, and link to it from those pages.
4. On other sites, accept the file the site builder generates, or leave it out.
5. A month after publishing, filter the server logs for /llms.txt to see who requested it.

FAQ

Does llms.txt replace robots.txt?

llms.txt leaves access to robots.txt, because llms.txt has no power to allow or block a crawler. Robots.txt tells each crawler which paths it may visit, and the shared AI training opt-out that Google and Microsoft plan for 2027 is a robots.txt line too. An llms.txt file offers directions, and each bot decides whether to follow them.

Is an llms.txt file a security risk?

An llms.txt file can be a security risk, because AI agents are designed to take it in and trust it. The largest research crawler requesting llms.txt files in Ahrefs’ May 2026 data called itself prompt-injection-survey/1.0, which Ahrefs reads as someone testing the file as a way to slip instructions to agents. Ahrefs advises keeping llms.txt under version control, limiting who can edit it and filling it with plain links and descriptions.

Do website builders create llms.txt automatically?

Some do. Ahrefs reports that Wix already generates llms.txt files and that Framer and Lovable scan sites for them, and the llms.txt proposal says documentation platforms generate the file automatically. A site on such a platform should read the generated file once, because an old price or a retired page in it misleads every agent that reads it.