llms.txt is a plain Markdown file at a site's root that lists the pages most useful to large language models. Adoption is rising, but evidence that AI systems read the file is thin, so treat it as a cheap extra and not a ranking lever. Crawl access, documentation quality and canonical URLs come first. Sona AI Visibility ties AI citations and prompts to pipeline and revenue, and tracks the AI crawler activity that shows whether the file is ever fetched.
What is llms.txt and what does it do?
llms.txt is a plain Markdown file, served from a website's root, that gives large language models a short, curated index of the pages most worth reading, each with a one-line description. It is a reading list written for machines, not a set of rules.
The file's job is selection. A model answering a question about your product cannot read an entire site, and a crawler left to guess can land on a changelog from three years ago. A file at /llms.txt names the quickstart, the API reference, the pricing page and the current policies, so the model starts from pages you chose.
Take a SaaS company with a docs site. Its file lists a short set of pages under headings such as Docs, Pricing and Policies, and each link carries a note on what the page answers. The owner is the documentation lead or the marketer responsible for how the product is described in AI answers.
The boundaries are narrow. The file is a proposal associated with Jeremy Howard of fast.ai, not a ratified standard, and it grants or denies no access. People also type llm.txt, llm txt and llms txt; the proposed filename is llms.txt, with the s. A file named llm.txt follows no proposal.
The cost is an afternoon of editing. The payoff is unproven, so this article separates what the file does from what anyone has measured.
What does a valid llms.txt file look like?
A valid file is Markdown saved as plain text and served from the root path of the domain. It opens with one H1 naming the project, adds a blockquote summary, and then lists links under H2 headings. The H1 is the only required element; the rest is convention.
For a fictional company, Acme Analytics, the file runs in this order:
- H1 title. One line, such as # Acme Analytics.
- Blockquote summary. One or two sentences after a > marker stating what the product does and who it serves.
- Context paragraphs. Optional notes, such as which API version is current.
- H2 sections of links. Each bullet holds a Markdown link, written as [Quickstart guide](page address), and a short note after a colon on what the page answers.
- An Optional section. A final H2 named Optional for secondary pages a model can skip when its context window is short.
Validity is mostly mechanical. The file must return a 200 status, load as text rather than an HTML shell, and avoid redirecting to a login page. Link text should describe the page, and the addresses should be canonical URLs, so a model never lands on a duplicate.
Keep it short. A file that lists every page duplicates sitemap.xml and defeats the point of curation. List the pages that answer the questions buyers ask most.
How is llms.txt different from robots.txt, sitemap.xml and llms-full.txt?
llms.txt curates, robots.txt permits, sitemap.xml enumerates, and llms-full.txt inlines. Each file has one job and one audience, and none of the four substitutes for another. The table compares them.
| File | Primary job | Format | Read by | Controls access? |
|---|---|---|---|---|
| llms.txt | Curated index of the pages worth reading, with short descriptions | Markdown | LLMs and AI agents | No |
| robots.txt | Crawl rules by user agent, using Allow and Disallow directives | Plain-text directives | Crawlers | Yes, for crawlers that comply |
| sitemap.xml | Exhaustive list of indexable URLs with metadata such as last modified date | XML | Search crawlers | No |
| llms-full.txt | Full text of the key content in one file | Markdown | LLMs and AI agents | No |
The most common mistake is treating the index file as an access control. It carries no Disallow line and no user-agent matching, so it can neither block GPTBot nor admit it. Access is set in robots.txt and in server or CDN rules. A page that robots.txt disallows should not appear in the index either, because the two files would contradict each other.
The second confusion is with sitemap.xml, which lists every indexable URL for search crawlers. The index file selects instead of enumerating, and its descriptions give a model context that a bare URL cannot. llms-full.txt goes a step further and carries the text itself. That saves a fetch, but the file grows with the content and costs more to maintain.
What problem is llms.txt meant to solve for AI systems?
The proposal behind the file addresses a retrieval problem. A language model has a context window, the amount of text it can process in one request, and an HTML page wraps its content in navigation, scripts and banners. The file points the model to clean, relevant pages first.
Without a curated index, a system choosing among a site's pages relies on links and the sitemap. A sitemap lists everything and ranks nothing. robots.txt says what may be fetched, not what is worth fetching. Neither tells a model that the version 2 guide supersedes version 1.
Consider a buyer who asks an AI system whether Acme Analytics supports single sign-on. An entry for the security page, described as covering SSO configuration and audit logs, gives a system that reads it a direct route to the answer instead of a search across the docs.
The value is editorial. A person who knows the product writes the shortlist and the descriptions once, and every system that reads the file inherits that judgment. Descriptions carry weight: a link labelled Pricing tells a model less than one noting that the page lists plan tiers and billing terms.
The benefit has a limit. It exists only if a given AI system fetches the file, and the measured evidence on that point is thin.
Do major AI systems actually read llms.txt?
Available request-log measurements suggest llms.txt is rarely requested, and no published evidence gathered here shows that major answer engines consistently use it when generating answers. A crawler request is not proof that the file shaped an answer, and nothing obliges any system to read it. Published request logs show most files drawing no requests at all, and the traffic that arrives comes mostly from non-AI tools. The proposal has no enforcement mechanism, so reading is voluntary.
The table collects third-party measurements. Samples and dates differ, so no single row settles the question.
| Measurement | Figure | Basis | Published |
|---|---|---|---|
| Files with zero requests | 97% | Published files, May 2026 measurement period | Jul 22, 2026 (updated Oct 8) |
| Requests made by bots | 96% | All requests to the files | Jul 22, 2026 (updated Oct 8) |
| Bot traffic from non-AI tools | 77% | Monitors, SEO crawlers and scrapers | Jul 22, 2026 (updated Oct 8) |
| Named AI agents | 1.1% | Share of requests in one dataset | Jul 11, 2025 (updated Oct 7, 2026) |
| GPTBot | 4.51% | Share of measured requests | Jul 2, 2026 (updated Oct 13) |
| ClaudeBot | 0.80% | Share of measured requests | Jul 2, 2026 (updated Oct 13) |
| DeepSeekBot | 0.02% | Share of measured requests | Jul 2, 2026 (updated Oct 13) |
Read the pattern, not any one figure. Most published files went unrequested in the measurement period, and most bot traffic came from monitors, SEO crawlers and scrapers rather than AI systems. Named AI crawlers appear, but a crawler's share of traffic does not show that it consumes the file when composing an answer.
Your own logs are better evidence. Sona AI Visibility connects AI citations and prompts to pipeline and revenue on one account timeline, and its Agent Analytics tracks AI crawler activity from server logs or a lightweight edge worker, with no JavaScript tag, so ad blockers do not hide it. That shows which pages AI bots fetch and which they ignore, the check that decides whether curating an index is worth the effort.
How many websites publish llms.txt, and which kinds of sites use it?
Estimates of adoption range from 8.6% to 51.8% of sites depending on the sample, and the highest rates sit with developer-tool and SaaS companies. The spread comes from who is counted, not from disagreement about the file.
| Sample | Share publishing | Count or base | Published |
|---|---|---|---|
| Top 100,000 domains | 8.6% of all domains; 11.4% of those that responded | 8,598 domains | Sep 13, 2026 (updated Oct 4) |
| Fixed panel of reachable hosts (digitalapplied.com) | 51.8% | 113 of 218 hosts | Aug 7, 2026 (updated Oct 2) |
| Tranco top 1,000 (digitalapplied.com) | 8.7% | 87 of 1,000 domains with llms.txt or llms-full.txt | Aug 7, 2026 (updated Oct 2) |
| Tranco top 1,000, reachable only (digitalapplied.com) | 15.8% | 549 reachable domains | Aug 7, 2026 (updated Oct 2) |
| Developer-tool companies (digitalapplied.com) | 68.9% | Same tracker | Aug 6, 2026 (updated Oct 8) |
| SaaS companies (digitalapplied.com) | 66.7% | Same tracker | Aug 6, 2026 (updated Oct 8) |
| 137,210 domains | 28% | Domains analyzed | Jul 22, 2026 (updated Oct 8) |
| Websites with the file | Count only | 4,088 in June 2025; 36,120 in May 2026, an 8.8-fold increase | Jul 11, 2025 (updated Oct 7, 2026) |
The denominator explains most of the gap. In the top-100,000 sample, 8.6% of all domains publish the file, but 11.4% of domains that responded do. In the Tranco top-1,000 tracker, 8.7% of all domains becomes 15.8% once only the 549 reachable domains count. The 51.8% figure comes from a fixed panel of reachable hosts, not a random draw, so compare rows within a sample rather than across samples.
Sector matters more than sample choice. Per the digitalapplied.com tracker, adoption reached 68.9% among developer-tool companies and 66.7% among SaaS companies. Documentation-heavy sites are the natural fit for a curated index. Publishing is common among them; being read is a separate matter.
Does publishing llms.txt improve AI citations or visibility?
No measurement gathered here shows that publishing llms.txt raises AI citations or visibility. The data counts how many sites publish the file and how often it is requested, not whether citations follow.
Any claim of lift has to hold across engines. Sona's own AI visibility data, September 2026, covers eight answer surfaces: ChatGPT, Google AI Overviews, Perplexity, Google AI Mode, Gemini, Mistral, Qwen and Claude. Those surfaces retrieve differently, so a file read by one crawler cannot be assumed to move all eight.
To test the file on your own site, change one variable:
- Fix a prompt list of buyer questions and run it daily to set a baseline. A team that runs the same list every day sees a pattern a weekly snapshot would have averaged away.
- Publish the file and change nothing else on the site.
- Keep running the list daily. A sustained shift is a trend; a one-day spike is noise.
- Re-audit key pages monthly or quarterly. That runs on a slower clock than the prompt list.
A visibility score tells you the outcome; only a technical audit tells you the cause. If a page is not cited, an index file will not say why. Sona AI Visibility runs a page-level technical audit across six categories: Crawlability, Performance, Security, Content Structure, Content Quality and Accessibility. Crawlability covers robots.txt, llms.txt and the canonical tag. Each page scores out of 100, each finding is ranked critical or medium with the fix attached, and a watch re-runs the audit so the score history shows whether the fix landed.
Does a SaaS company need llms.txt, and what should come first?
No, llms.txt is optional for a SaaS company, not a prerequisite, and crawl access and page quality should come first. The file only points to pages. If a bot is blocked or a page is unreadable, the pointer changes nothing.
Peers already publish. Per the digitalapplied.com tracker, adoption ran 68.9% among developer-tool companies and 66.7% among SaaS companies, so the file is common practice in the category. That makes it a cheap step, not a differentiator.
A sensible order:
- Confirm bots can reach your pages. Check robots.txt and server or CDN rules against the AI crawlers you want to admit, and fix any block that is not intended.
- Fix page-level problems. Run the page-level technical audit in Sona AI Visibility. It shows whether AI engines can crawl and parse the site, not whether you appear in answers.
- Publish the index file. Include entries for docs, pricing, security and the API reference.
- Measure what the pages earn. Sona AI Visibility connects AI citations and prompts to pipeline and revenue on one account timeline, so a mention becomes something you can measure rather than a number to report.
Done in that order, the file is the last small step on a sound base, not a substitute for one.
How do you create, host and maintain an llms.txt file?
Create the file by hand or from a draft, host it at the domain root, and review it whenever documentation changes. Maintenance is the part teams neglect.
- List the pages. Choose those that best answer buyer questions: product overview, quickstart, API reference, pricing, security and policies.
- Draft the file. Write the H1, the blockquote summary, then H2 sections of links with a one-line note each. Edit any generated draft by hand.
- Host it. Save it as UTF-8 plain text and upload it to the domain root so it resolves at /llms.txt. A subdomain needs its own copy.
- Verify it. Confirm the address returns a 200 status, the content type is text, and every link resolves without a redirect chain.
- Add llms-full.txt if useful. That companion holds the key content in one file, and it needs the same upkeep.
Treat the file as part of the release process. A stale index sends any system that reads it to deprecated pages, and an auto-generated file inherits whatever the docs tool exports. Review it monthly or quarterly, and again whenever a page is renamed, removed or superseded. Give it a named owner.
Frequently Asked Questions
Is llm.txt the same as llms.txt?
People type llm.txt, llm txt and llms txt, but the proposed filename has the s after llm and is served from the site root at /llms.txt. A file named llm.txt follows no proposal, so a system looking for the real filename will not find it.
Where does the llms.txt file go?
At the root of the domain, in the same place as robots.txt. Subdomains need their own file if they should be covered. A copy placed in a subfolder is not at the root path, so it sits outside the location the proposal describes.
Does llms.txt block or allow AI crawlers?
Neither. llms.txt is a curated reading list and carries no Allow or Disallow directives, so it cannot admit or turn away a crawler. Whether a compliant crawler may fetch your pages is governed by robots.txt, and server or CDN controls decide whether a request is served at all.
Summarize this article with AI: ChatGPT · Claude · Perplexity · Google AI Mode
Last updated: October 2026
