Skip to content
AI for SMBs

How to create an llms.txt: what it is, who reads it, how to test it

Wouter·29 May 2026·7 min read
TL;DR

llms.txt is a markdown text file at /llms.txt that tells an AI agent in brief what is on your website, with links to the pages that matter. It is a proposal from 2024 that is used mostly for software documentation. Google Search ignores the file, so do not expect a better position from it. You build it with a heading, a summary and a list of links, and you test it with Lighthouse.

You often read that every business needs an llms.txt, because otherwise ChatGPT cannot find you. An earlier version of this article itself said that AI models automatically look for the file. The documentation of Google, OpenAI and Anthropic does not say that. The file exists and it has a clear purpose. What it does for your visibility is a smaller story than many guides promise. Below is what the sources say, how to structure the file and which mistake we found in our own.

What is llms.txt?

llms.txt is a markdown text file that lives at yourdomain.com/llms.txt and tells an AI agent in brief what can be found on your website. Jeremy Howard proposed it in September 2024. The current text of the proposal is the second version, updated in August 2026.

The idea behind it: a web page is built for people, with navigation and JavaScript wrapped around the content. According to the proposal, an agent is best served by concise information in one fixed place. The intention is that it reads the file, searches for what it needs and then follows the links.

The proposal reports that thousands of sites publish such a file and that it is used most heavily for software documentation, where coding agents follow the links to references and tutorials. OpenAI and Anthropic have one for their own developer documentation: developers.openai.com/llms.txt and platform.claude.com/llms.txt. For a business, the proposal names an outline of your structure and policies as a use.

Does an AI search engine read your llms.txt?

Google Search does not use your llms.txt, and the crawler documentation of OpenAI and Anthropic does not say they read it. On 15 June 2026 Google added a clarification to its guide to generative AI in Search: Search does not use such files, and an llms.txt will "neither harm nor help" your visibility or rankings. Maintaining a file for other services that do use it is "completely fine", Google says.

OpenAI and Anthropic describe robots.txt in their crawler documentation as the way to control access. Those pages say nothing about reading an llms.txt on your site (read on 14 November 2026). What is established: OAI-SearchBot finds pages for ChatGPT, and according to OpenAI's documentation sites that opt out of this crawler are not shown in ChatGPT search answers, though according to the same page they can still appear as navigational links.

So none of these sources says an llms.txt gets you into an AI answer. What OpenAI does name as a condition is that its crawler may fetch your pages. That point and the rest of the basics are in our agent-ready website checklist.

Is llms.txt the same as robots.txt?

No, the two files do different jobs: robots.txt controls access and llms.txt gives explanation. The proposal puts it this way: robots.txt lets automated tools know what access to a site is considered acceptable, and the information in llms.txt is used at the moment an agent needs to know something about a topic while assisting a user.

You cannot forbid anything in llms.txt. To block or allow a crawler, you use robots.txt. A sitemap does not replace the file either: a sitemap lists all pages, and llms.txt is a short selection with notes.

What does an llms.txt look like?

An llms.txt starts with a heading that holds the name of your site, and that is the only required part. After it come, in this order:

  • A summary as a blockquote (lines that start with >), with what a reader needs to know to understand the rest.
  • Notes in plain paragraphs or lists, without subheadings.
  • Sections with an H2 heading, each with a list of links. A line in such a list is a markdown link, optionally followed by a colon and a short note.

A section named "Optional" is, by convention, meant for links an agent can skip when it has little room.

# Example Bookkeeping

> Bookkeeping firm in South Holland for freelancers and small companies.
> Bookkeeping, VAT returns and annual accounts.

New clients start with a 30-minute introduction.
We do not do payroll.

## Services
- [Bookkeeping for freelancers](https://www.yourdomain.com/bookkeeping): what is included and what it costs
- [Annual accounts](https://www.yourdomain.com/annual-accounts): process and lead time

## Contact
- [Book an introduction](https://www.yourdomain.com/contact): online calendar and phone number

## Optional
- [About us](https://www.yourdomain.com/about): who works here

The name and the addresses in this example are made up. Our own file is at mindsora.io/llms.txt. It is longer and deviates from this structure: it has subheadings and list items without a link.

The proposal wants links to point to text an agent reads well, such as a markdown version of the page. This example and our own file point to regular web pages.

How do you create an llms.txt in five steps?

You create an llms.txt by writing one text file, placing it in the root of your site and then testing it. The steps:

1. Write the file. Use the structure above. The proposal gives three writing rules: use concise, clear language, add a short description to each link and avoid ambiguous terms and jargon you do not explain. Only put in facts that are also on your website.

2. Write links in full. Every reference is a markdown link: the link text in square brackets and the full address after it in parentheses, as in the example above. A bare path such as /services or a bare address without link text is not a link for the proposal or for Lighthouse. This is the mistake we made ourselves; see below.

3. Place it in the root. Lighthouse looks for the file at yourdomain.com/llms.txt. The proposal also allows a file in a subfolder. On a site that is built from files, like ours, it sits in the public folder. According to the list on llmstxt.org, the WordPress plugins Yoast SEO and AIOSEO create the file for you, and Wix generates one for every Wix site. In that case, check what is in it.

4. Test it. Open the address in your browser: you should see plain text. Then run the Lighthouse audit, for example through PageSpeed Insights. The proposal also advises asking an agent questions about your content while giving it only your llms.txt as a starting point.

5. Keep it up to date. A new service, a changed price or a removed page has to change in the file as well. A file that says something different from your website misleads the agent that reads it.

How AI-ready is your business?

Take the free AI Readiness Quiz: 7 questions, 2 minutes.

Take the quiz →

What does Lighthouse check in your llms.txt?

Lighthouse checks three things in the file: whether it has an H1 heading, whether it has at least one markdown link and whether it is at least 50 characters long. That is in the source code of the audit (read on 14 November 2026).

On a server error (5xx), a 429 or a failed request, Lighthouse fails the audit. On a 404 or another 4xx the audit is not applicable, because according to Chrome's explanation the file is "optional at the moment". Chrome calls the whole agentic browsing category experimental in its scoring notes.

So the audit only says something about the form. It does not measure whether an agent gets anything out of your file.

What went wrong in our own file?

Our own llms.txt failed the Lighthouse audit when we ran it on 13 November 2026. The message was that the file contained no links. That was correct: it had 34 lines with a bare address and 23 blog references as a bare path, and not a single markdown link. A person reads past that. The audit looks for the form [text](address) and did not find it.

We converted the references to markdown links with a full address: 69 links in a file of 318 lines. A day later the audit passed (own measurement through PageSpeed Insights, 14 November 2026). Our test suite now runs a test that repeats the three Lighthouse requirements and also checks that every link points to something that exists in the site.

What we have not measured: whether AI services fetch our file and whether it changes anything about how they write about us.

When is an llms.txt worth it?

An llms.txt is worth it when you have a concrete use for it and are willing to maintain it. That use can be documentation that coding agents rely on, or a fixed overview of your services and policies that an agent can read in one go.

Do not start with it if your pages are not in Google yet or if your main text only appears after the page has loaded. Fix that first.

You can measure it yourself right now with the free Agent-Ready Grader: your website gets a score from 0 to 100 and a list of concrete fixes. That grader is our own instrument. It gives llms.txt 20 of the 100 points. That is our own weighting; Google Search does not use the file.

Frequently asked questions

Is llms.txt an official standard? No. The author calls it a proposal, and Chrome speaks of "an emerging convention".

Does llms.txt affect my position in Google? No. According to Google the file neither helps nor harms your visibility and rankings in Search.

Can I combine it with structured data? Yes. The proposal says the file can reference the structured data on your site. Make sure both say the same as your pages.

How long can the file be? The proposal names no maximum. It does want the file to stay small enough to be read in one go, with the detail behind the links.

Further reading

The Agent-Ready Audit (€697 one-off, excluding VAT) is listed in our services overview. Want to work out how many hours a recurring task costs you? The free AI Readiness Scan does that: 7 questions about your business processes.

Ready to automate?

Book a free consultation and discover what AI automation can do for your business. Or take the 2-minute AI Readiness Quiz first.

Book a call

More articles