llms.txt: what it is and whether your site needs one
llms.txt is a small markdown file at your site root that gives AI systems a curated map of your most important pages. It costs ten minutes and might help. Here is what it actually is, what it definitely is not, and an honest way to decide.
What llms.txt is
Proposed in September 2024 by Jeremy Howard of Answer.AI, llms.txt is a plain markdown file served at yoursite.com/llms.txt. The idea: AI systems have small context windows and your site has navigation, footers and noise, so hand them a clean, prioritized list of what matters - a table of contents written for machines.
A minimal, honest example:
# Seoprog
> Single-page SEO and AI-visibility checker for small sites.
## Main pages
- [Home](https://seoprog.com/): what the tool does, pricing
- [Privacy](https://seoprog.com/privacy): data handling
- [Terms](https://seoprog.com/terms): service terms
## Guides
- [AI visibility](https://seoprog.com/guides/ai-visibility-check): can assistants read your site
What llms.txt is not
- Not a standard. No search engine or major AI vendor has committed to reading it. Support is sparse and mostly unannounced either way.
- Not access control. It cannot block or allow anything - that is robots.txt's job.
- Not an SEO factor. There is no evidence it influences rankings anywhere. Anyone selling llms.txt as a ranking tactic is selling you something.
Should you add one?
Yes, if any of these apply: your product is developer- or AI-adjacent (your audience expects you to have one), your site has real depth worth mapping (docs, guides), or you simply want every low-cost signal in place. Skip it without guilt if your site is three pages of marketing - the file would just restate your navigation.
How to write a good one
- Keep it short. One H1 with your name, one blockquote describing the site in a sentence, then grouped link lists with one-line descriptions.
- Curate, do not dump. This is your judgment about what matters, not a sitemap mirror - the sitemap already exists for completeness.
- Describe honestly. The descriptions are what a machine may repeat about you; write them like answers, not slogans.
- Keep it current. A stale llms.txt pointing at dead pages is worse than none. Update it when your site structure changes.
Serve it as plain text or markdown at the root, exactly like robots.txt. No registration, no ping - systems that want it will look for the path.
Frequently asked questions
Is llms.txt an official standard?
No. It is a community proposal from late 2024. Adoption by major AI crawlers remains limited and unconfirmed - which is exactly why the honest pitch is "cheap courtesy," not "ranking factor."
Will llms.txt improve my rankings?
No evidence says so, for Google or for AI assistants. Treat it as a ten-minute courtesy file with possible upside and no downside.
What is the difference between llms.txt and robots.txt?
robots.txt is enforcement - crawlers respect its allow and disallow rules. llms.txt is a hint - a curated reading list for systems that choose to consult it.
See how your page scores
Run the free check - it reads your page the way Google and AI assistants do and gives you a plain-English fix list. No sign-up for your first check.
Check a page free