llms.txt: What It Is, Who Actually Reads It, and Whether You Need One

What is llms.txt?

llms.txt is a proposal by Jeremy Howard, first published on 3 September 2024 at llmstxt.org. The spec describes it as a way "to provide information to help agents use a website." In practice it is a short markdown index that "offers brief background information, guidance, and links to detailed markdown files."

Three things matter about how the author intended it to work:

• It is read on demand. The spec says the file is used "when an agent needs information about a topic while assisting a user", and that it should "mainly be useful for inference rather than training".
• It is not a robots.txt replacement. In the spec's own words, "robots.txt and llms.txt have different purposes." robots.txt controls access. llms.txt is a reading list.
• Its home turf is documentation. The spec says the files "are used most heavily for software documentation, where coding agents follow them to find API references and tutorials."

That last point explains most of what our logs show.

Is llms.txt actually used? What our server logs show

We pulled seven days of request data for loudface.co from Cloudflare, which sits in front of our site, covering 25 September to 2 October 2026. We looked at five paths: the two llms files, robots.txt, the sitemap and our AI fact sheet page. Then we grouped every request by user agent.

The AI crawlers were active on our site that week. ClaudeBot read robots.txt 338 times and the sitemap 205 times. OAI-SearchBot read robots.txt 215 times. Not one of them asked for llms.txt.

So who did? Of the 56 requests to /llms.txt:

• 25 came from SEO audit and "AI readiness" tools, led by SEOJuice's bot with 21.
• 8 came from tech-profiling crawlers such as BuiltWith and Dataprovider.com.
• 14 came from browsers, generic clients and curl, which is most likely people (or scripts) checking the file by hand.
• 9 came from AI companies: 5 from the agent probe of Andi, an AI search company, 2 from Claude-User and 2 from Bytespider, the crawler run by ByteDance.

Claude-User is Anthropic's user-triggered fetcher, so those 2 requests mean a person's Claude session went to our file, probably because someone asked it to. That is the use the spec describes. It is also 2 requests in a week.

llms-full.txt, the long version of the file, got 9 requests. 2 came from Andi. None came from OpenAI, Anthropic, Perplexity or Google.

The limits of this data. This is one site, one week and five paths. Cloudflare samples its analytics at high volume, and its docs say it "returns an estimate derived from the sampled value", so the 5,000 robots.txt figure is the most likely to be a scaled estimate. Low-volume paths like llms.txt are likely close to exact. Also, 12 of the 56 llms.txt requests were redirects from loudface.co to www.loudface.co, so the number of distinct fetches is lower than 56. If you want to run the same check on your own site, our guide to reading server logs for AI-bot traffic walks through it.

What the bigger studies found

One site proves nothing alone. Three larger studies point the same way.

Ahrefs, 137,210 domains (May 2026). Ahrefs checked the domains using its web analytics and found 28% publish an llms.txt file, and "97% of those files received zero traffic in May 2026." Of the requests that did arrive, 96% came from bots. AI retrieval bots such as OAI-SearchBot and PerplexityBot made only 1.1% of them, and "Slackbot alone fetched llms.txt files more often than PerplexityBot did." GPTBot was the top named AI reader and Claude-Code, Anthropic's coding agent, was second. Ahrefs also found zero AI-bot requests for llms.txt files that do not exist, which means the bots are not probing for it.

EZY Research, 83 sites over 12 weeks (April to July 2026). EZY compared robots.txt fetches with llms.txt fetches from verified crawlers. OpenAI's crawlers: 3,990 against 7. ClaudeBot: 3,120 against 9. PerplexityBot: 775 against 0. Googlebot: 5,125 against 67. The one exception was Meta-ExternalAgent, at 172 robots.txt fetches against 193 llms.txt fetches. EZY sells llms.txt generation, so read its upbeat conclusion with that in mind. We saw zero Meta requests to llms.txt in our week.

SE Ranking, about 300,000 domains (November 2025). SE Ranking tested whether the file predicts AI citations. Only 10.13% of domains had one, and "Both statistical analysis and machine learning showed no effect of LLMs.txt on how often a domain is cited by LLMs," and we have not found a later study that shows the opposite, though this one predates the v2 spec.

Put the four datasets side by side and the pattern is consistent. The crawlers that feed ChatGPT, Claude, Perplexity and Google read robots.txt thousands of times and llms.txt a handful of times, if at all.

What Google says about llms.txt

Google has been unusually direct. Its AI optimization guide, updated in July 2026, says: "You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn't use them." It adds that "It's completely fine if you decide to create and maintain LLMS.txt files (or other similar files) for other services or systems that use these files."

That covers AI Overviews and AI Mode. Google's separate page on AI features says "There are no additional requirements to appear in AI Overviews or AI Mode, nor other special optimizations necessary."

John Mueller put it more bluntly back in April 2025: "To me, it's comparable to the keywords meta tag". He also said "you can tell when you look at your server logs that they don't even check for it." Our logs agree with him.

Then why does Lighthouse check for it? Lighthouse, the page audit tool built into Chrome, added an llms.txt audit under its "agentic browsing" checks, documented in May 2026. Read the details before you panic about a score. Lighthouse only flags a page "if a server error occurs when attempting to retrieve the llms.txt file". If the file is missing, "the audit is marked as Not Applicable (N/A), as providing the file is optional at the moment." Mueller's explanation of the split is that the file is "not done for search". It is for agents and coding tools, not Google Search.

What changed in llms.txt v2 (August 2026)

The spec got its first major revision in August 2026. The changes page gives the reason: "Thousands of sites now publish an llms.txt file, documentation platforms generate one automatically, and coding agents use them reliably." The main changes:

1. Discovery links. A page can now point agents to its covering llms.txt with rel="describedby", and to its markdown version with rel="alternate" and type="text/markdown". You can send these as HTML link elements or as an HTTP Link header.
2. Two naming options for markdown pages. A markdown copy of a page can live at page.html.md or page.md.
3. Files in subfolders. A file covers the pages under its path, and the most specific file wins. A docs site can have one llms.txt per product.
4. Plain consumption rules. Agents read or search the file to find what they need, then follow the links.
5. The "Optional" section lost its special meaning. The context-expansion tooling is gone from the proposal, so an "Optional" heading is now just a heading.

Discovery is the change to watch. Ahrefs found that agents fetch the file when something tells them it exists. v2 gives sites a standard way to do the telling.

What does an llms.txt file look like?

An llms.txt file is plain markdown in a fixed order. Here is the top of our own file at loudface.co/llms.txt, shortened:

# LoudFace

> LoudFace is a B2B SaaS organic growth agency. ...

## About

- [Pricing](https://www.loudface.co/pricing): Retainer plans, ...
- [Methodology](https://www.loudface.co/methodology): The Answer Chain, ...

## Services

- [SEO & AEO](https://www.loudface.co/services/seo-aeo): Search ...

Each part has one job:

The full file has one H1, a blockquote, five H2 sections and 173 links.

What about llms-full.txt? It is a single markdown file with the full text of a site, so you can paste one URL into an AI tool and load everything at once. Mintlify says it developed the format "in collaboration with customer Anthropic". It is a popular convention, but the current v2 spec text does not mention it. Mintlify also claims agents visit llms-full.txt more than llms.txt, without a sample size. Our week does not back that up: 9 requests for llms-full.txt against 56 for llms.txt, and none of the 9 from OpenAI, Anthropic, Perplexity or Google.

How to create an llms.txt file

You probably do not need to write one by hand. The spec lists tools that generate it for you:

• Docs platforms: Mintlify and GitBook generate llms.txt for every site they host.
• WordPress: Yoast SEO generates and maintains the file, and AIOSEO has an llms.txt generator.
• Website builders: Wix generates one for every Wix site.
• Static site and docs frameworks: VitePress, Docusaurus and Drupal have plugins.

If you write it yourself, follow the spec's advice: "Use concise, clear language." Then test it the way the spec suggests, by asking an agent questions about your content and "giving it only your llms.txt as a starting point."

How do you submit it? You do not. There is no submission process at any AI company. Put the file at yourdomain.com/llms.txt, link it from your footer, and add the v2 rel="describedby" link if you want agents to find it from any page.

One warning. Ahrefs notes that "potential bad actors are already probing llms.txt for prompt injection." Agents are built to trust the file, so treat it like code. Keep it in version control and check what your generator writes into it.

llms.txt vs robots.txt vs sitemap.xml

If an AI crawler cannot reach your pages, llms.txt will not fix it. robots.txt decides that. If you want a crawler to find a page, the sitemap and your internal links do that work.

Should your B2B SaaS publish an llms.txt?

It depends on who uses your site.

Developer platforms and API products: yes. If developers point Claude Code or other coding agents at your docs, those agents are the file's real audience. Ahrefs' verdict: "If your customers use coding agents, or if agents act on your site, the file stands a real chance of being read." Ship it with markdown versions of your docs pages.

Marketing sites: optional, and low priority. It takes little time, and your CMS may generate it for you. It will not hurt you. But if your goal is to be named in ChatGPT, Perplexity or AI Overviews answers, Ahrefs calls the file "largely decoration", and every study above backs that up. loudface.co publishes both files, and in the week we measured, the AI crawlers walked right past them.

What to do instead if you want AI citations

The crawlers spend their requests on robots.txt, sitemaps and your actual pages. Put your effort where they look:

1. Check that AI crawlers can reach you. Read your robots.txt and your CDN's bot settings. A blocked GPTBot or OAI-SearchBot costs you more than a missing llms.txt ever will.
2. Put the answer at the top of the page. Engines lift the title and the first block of text far more than anything below it. Our guide to the 60-word block that triggers AI Overviews shows the shape.
3. Publish your key facts on a real, linked page. Our AI fact sheet is a normal HTML page, and OAI-SearchBot fetched it 21 times in the same week it ignored llms.txt. It is also linked from other pages, so we cannot credit llms.txt for those visits.
4. Use schema where it helps extraction. Our breakdown of schema markup for AEO covers the five types worth adding.
5. Measure answers, not files. Track which prompts name you, engine by engine. If you want a baseline, our free AI visibility audit shows where you stand in ChatGPT, Claude, Gemini and Perplexity.

An llms.txt file is a nice-to-have for agents. Getting cited is still about pages worth citing, on a site the crawlers can reach. If you want a team to run that work for you, see our GEO service.