Nobody Is Fetching Your llms.txt (And Google Says It Never Will)

There is a file you have probably been told to add. Maybe you already did. Maybe someone quoted you for it.

It is called llms.txt, it sits at the root of your site, and it is supposed to hand AI crawlers a clean summary of what you do so they cite you in answers.

We have one on this site. We wrote it in about an hour. Then the 2026 server log data started coming out, and it says something uncomfortable: almost nothing is fetching it. Not GPTBot, not ClaudeBot, not PerplexityBot. Meanwhile Google has been on record since 2025 that it will never support the format.

Here is what the logs actually show, and the direct answer on whether to write one anyway.

What is llms.txt supposed to do?

It is a markdown file at your site root that describes your site to a large language model in plain text: what you do, which pages matter, and where the important documentation lives.

The pitch is reasonable on its face. Crawlers waste effort parsing navigation, cookie banners and JavaScript. A curated markdown summary is cheaper to read and harder to misunderstand. Give the model a clean map and it cites you more accurately.

It is modelled on robots.txt and sitemap.xml, two files that genuinely work. That resemblance is most of why people believe in it.

What do the crawler logs actually show?

They show it is being ignored at a rate that is hard to argue with.

Across 19 sets of customer server logs covering February to April 2026, major AI crawlers requested /llms.txt on 3 of 19 sites. Forty-one requests in total, against roughly 1.1 million AI crawler page fetches on those same sites (MaxAEO). That is about 0.004% of crawler activity.

It is not one dataset saying this.

• WISLR logged 12,099 bot requests over 48 days in March 2026 and found zero requests for /llms.txt or /llm.txt (WISLR).
• Evil Martians checked two high-traffic sites via CDN logs in April 2026 and found zero requests to the markdown files from GPTBot, ClaudeBot or PerplexityBot, even when those files were explicitly listed inside llms.txt.
• Per-bot behaviour in one log set: ClaudeBot fetched robots.txt 11 times and sitemap.xml 11 times, and llms.txt zero times. PerplexityBot fetched robots.txt four times and llms.txt zero times. GPTBot fetched sitemap.xml once and llms.txt zero times.

Read that last one again, because it is the finding. The bots are not ignoring your files. They are fetching robots.txt and sitemap.xml constantly and skipping llms.txt specifically.

They are not ignoring your files. They are fetching robots.txt and sitemap.xml on repeat, and skipping llms.txt specifically.

What has Google actually said?

That it does not support the format and is not planning to.

Gary Illyes said so at Search Central Live in July 2025, and Google's own AI features optimization guide tells you to skip it. John Mueller has compared llms.txt to the keywords meta tag, which is the canonical example of a signal so easy to game that no search engine uses it (Search Engine Journal).

There is one wrinkle that gets quoted as a contradiction. Lighthouse ships an experimental Agentic Browsing audit that checks llms.txt handling. That is a different product answering a different question: how a browsing agent navigates your site, not how Search ranks it. It is not Google quietly reversing itself, and anyone citing it as proof that Google secretly uses the file is overreading a checklist item.

So why is everyone still recommending it?

Three reasons, and only one of them is about you.

It is cheap to recommend. An hour of work with no downside is the safest advice anyone can give. Nobody gets blamed for it.

It resembles things that work. robots.txt and sitemap.xml are load bearing, and llms.txt looks like their sibling. The analogy does the persuading before the evidence gets a turn.

It is sellable. "AI search optimisation" is a service line, and a deliverable you can hand over in a day is a good service line. That is not fraud, but it is worth knowing which incentive is behind the advice.

Most guides recommending it were also written before the log data existed. The spec was proposed in 2024, the enthusiasm peaked in 2025, and the server logs only started saying anything definitive in the first half of 2026.

Should you write one anyway?

Yes, once, in an hour, and then never think about it again. Do not pay anyone for it, do not measure anything by it, and do not let it displace work that has evidence behind it.

That is a genuinely small yes. Here is the reasoning.

The cost is an hour. The evidence says the current return is approximately zero. But adoption could change, the file costs nothing to serve, and if a crawler does start reading it you are already there. That is a cheap option on an uncertain future, which is a reasonable thing to buy for an hour.

What it is not is a strategy. If someone is selling you AI search visibility and the deliverable is this file, you are buying an hour of markdown at consultancy rates.

Spend the rest of that day on what the crawlers demonstrably do fetch. The same logs that show llms.txt being skipped show robots.txt and sitemap.xml being pulled over and over. So:

1. Make sure your HTML actually contains your content. If your pages render client-side, a crawler that does not run JavaScript sees an empty shell. This is the single biggest one and it is invisible until you check.
2. Keep sitemap.xml complete and current, including every post. Bots fetch it repeatedly.
3. Name the AI crawlers explicitly in robots.txt. A crawler matching its own named group ignores the wildcard group entirely, so a rule you think applies to everyone may not.
4. Write answers that survive extraction. Direct answer in the first 40 to 60 words under each heading, real numbers with sources, question-shaped headings.
5. Then, if you have an hour spare, write llms.txt. Last on the list, not first.

If someone is selling you AI visibility and the deliverable is llms.txt, you are buying an hour of markdown at consultancy rates.

Common mistakes

Treating llms.txt as an AI visibility strategy. It is a file. The evidence says it is currently unread. A strategy is what you do about the four items above it on that list.

Assuming Google uses it because Lighthouse checks it. Different product, different question. Google Search has said the opposite, repeatedly, on the record.

Listing markdown files inside it and assuming they get fetched. Evil Martians tested exactly this and found zero requests to those files. Linking a resource does not make anything read it.

Writing one and calling AEO done. The things that move AI citations are content quality, extractable structure and being crawlable. Those are hard, which is part of why a one-hour file is such an appealing substitute.

Believing a guide that never mentions log data. Most of the recommendations in circulation predate the evidence. Check the date on anything you are reading about this, including this post.

Key takeaways

• Across roughly 1.1 million AI crawler page fetches on 19 sites, llms.txt was requested 41 times, on 3 of those sites.
• One 48 day log set of 12,099 bot requests recorded zero fetches of the file.
• ClaudeBot, PerplexityBot and GPTBot were all fetching robots.txt or sitemap.xml in the same logs where they never touched llms.txt.
• Google has said since July 2025 that it does not support the format and is not planning to, and has compared it to the keywords meta tag.
• The Lighthouse Agentic Browsing audit is a different product answering a different question, not a reversal.
• Write one anyway if it costs you an hour, because it is a cheap option if adoption changes. Do not pay for it and do not measure anything by it.
• Spend the rest of the day on crawlable HTML, a complete sitemap, explicit robots.txt groups and extractable answers, which is where the evidence actually points.

We keep an llms.txt on this site for exactly the reason above: it cost an hour and it might matter later. What actually changed our numbers was making every route serve real HTML instead of an empty shell, which is the unglamorous item at the top of that list. If you want that kind of work done properly rather than sold as a package, that is what we do.