What is an llms.txt file?

The format was proposed in September 2024 by Jeremy Howard of Answer.AI. The idea: language models work within limited context, and a typical web page is full of navigation, scripts and layout that get in the way. A single plain Markdown file listing a site's key pages, with one-line descriptions, would let an AI tool find the right content quickly, especially at the moment of answering a question.

It took off fastest with developer documentation. Several documentation platforms and site builders can now generate one automatically, which is why you see it on many software sites.

Robots.txt and sitemaps are part of established crawler infrastructure. Llms.txt is a suggestion. If you want to allow or block AI crawlers, that still happens in robots.txt (for example, rules for GPTBot, OAI-SearchBot or PerplexityBot), not in llms.txt.

The proposed structure, in order:

1. An H1 with the site or company name. This is the only required part.
2. A blockquote with a one or two sentence summary of what the business does.
3. Optional plain paragraphs with any context a reader needs.
4. H2 sections of Markdown links, each with a short note: - [Page title](https://...): what it covers.
5. An optional section titled Optional for lower-priority links that can be skipped.

A simple example for a service business:

# Example Clinic

> Physiotherapy clinic in Dubai offering sports injury rehab, post-surgery recovery and home visits.

## Services
- [Sports injury rehab](https://example.com/sports-injury): assessment, treatment plans, typical session count
- [Home visits](https://example.com/home-visits): areas covered, booking, pricing

## About
- [Our team](https://example.com/team): physiotherapists, qualifications, languages

The proposal also suggests offering clean Markdown versions of key pages, though few business sites do.

Use this decision rule.

• Add it if you run documentation, an API or a product that developers and AI coding tools use. That is where it is most likely to be read.
• Add it if your fundamentals are already in place and it takes under an hour. There is little downside.
• Skip it for now if your pages are not indexed, your business details are inconsistent or AI crawlers are blocked. Those problems matter far more.

Google has said publicly that it does not use llms.txt for Search, and the major AI providers have not documented it as a citation input. Any benefit today comes from AI tools and agents that choose to read it, not from ranking or citation systems.

• Treating it as a fix for invisibility. If ChatGPT does not name you, llms.txt will not change that.
• Pasting marketing copy. Descriptions should say what each page covers, plainly.
• Letting it go stale. A file listing pages that now 404 is worse than none.
• Thinking it restricts access. It does not. Use robots.txt for that.

Here is a hypothetical. Say you run a SaaS company in Berlin selling scheduling software, with public API documentation. Developers increasingly ask AI coding assistants how to integrate with you. An llms.txt listing your quick-start guide, authentication docs, main API reference pages and changelog, plus Markdown versions of those pages, gives those tools a clean path to the right content. That is a sensible half-day of work.

Now say you run a three-branch dental practice. Your Google Business Profiles list two different phone numbers and the services pages are a single bullet list. An llms.txt file would be the least valuable hour you could spend. Fix the listings and the pages first.

We treat llms.txt as a finishing item in a GEO engagement, after crawler access, indexing, entity consistency and page structure are sorted. When it makes sense, we generate it from your sitemap and page metadata, write the descriptions by hand, and set it to rebuild automatically when key pages change, so it does not drift out of date.

When we build or rebuild a site through Web Design, adding a maintained llms.txt is part of the launch checklist. For existing sites, it is part of the AI visibility check: we tell you whether it is worth doing for your business, and what to do first if it is not.

Get my AI visibility check →