llms.txt: Proposal, Support, and a Low-Risk Check

Updated September 20, 2026· 7 min read

llms.txt is a proposed Markdown file that gives agents a short, curated map of a website. Jeremy Howard’s llms.txt proposal (v2 page, published September 3, 2024, updated August 10, 2026) places it at /llms.txt or under a path such as /docs/llms.txt. The required piece is an H1 name. Optional pieces are a blockquote summary, notes, and H2 sections of links, including an Optional section for secondary URLs. Some sites also publish llms-full.txt.

Teams sometimes include a file like this inside AI-powered LLM SEO services. That does not make the file a ranking tactic. For Google Search, including AI Overviews and AI Mode, the current primary statement is that Search does not use llms.txt. Answer Engine Optimization still depends on crawlable pages and useful answers.

What the file is for, and what it is not

The proposal’s job is discovery for agents: a small index an assistant can read, then follow links to Markdown versions of pages (page.md or page.html.md). It is not a compliance standard, not a robots directive, and not a promise of citations.

File	Job	Who documents support	Status
robots.txt	Allow or disallow crawling	Search engines, as a long-running convention	Established crawl control
sitemap.xml	List URLs for discovery	Google, Bing, and others	Established search discovery
llms.txt	Curated Markdown overview for agents	Proposal plus individual publishers and tools	Proposed convention

Google Search. Google’s guide to generative AI features on Search (updated July 10, 2026) says you do not need new machine-readable files, AI text files, markup, or Markdown to appear in Search, as Google Search itself does not use them. Creating llms.txt for other systems “will neither harm nor help” Search visibility or rankings. Google also says there is no special AI markup and no ideal page length. Search Engine Journal’s 2025 write-up of Google comments is older reporting; use the 2026 Search Central guide as the current Search statement.

Chrome Lighthouse. Chrome’s llms.txt audit (updated May 5, 2026) requests /llms.txt. A 404 is Not Applicable because the file is optional. A server error when retrieving it fails the audit. That is an agent-readiness check, not a Search ranking signal. The Agentic Browsing category is documented as experimental.

Other publishers. The proposal notes that OpenAI, Anthropic, and Gemini publish llms.txt on their own developer documentation. Those files retrieve as Markdown indexes (OpenAI API docs llms.txt, Anthropic docs llms.txt, Gemini API docs llms.txt). That establishes publisher adoption. It does not establish whether a specific consumer product checks third-party files, so verify support in that product’s current documentation.

Measure citations with tools such as Rank Prompt instead of assuming the file caused a mention.

Low-risk implementation and verification

1. Confirm robots.txt permits crawling and sitemap.xml lists the canonical, indexable URLs you want discovered.
2. If you publish llms.txt, follow the proposal order: H1, optional summary, then link lists to canonical, public, crawlable URLs (not logins, staging hosts, or redirect chains). Keep the underlying HTML, internal links, and structured data.
3. Keep summaries aligned with claims on the linked page.
4. Fetch /llms.txt yourself. Expect 200 if you publish it, 404 if you do not, and treat 5xx as a server bug.
5. Optional: ask a coding agent to navigate from the file alone. That tests agent discovery, not Google ranking.

A short file can be enough. This sample uses real public canonical URLs because Anderson Collaborative does not publish invented .md variants:

# Anderson Collaborative Knowledge Base
> Practical marketing and advertising reference guides.

## AI search
- [What is llms.txt?](https://www.andersoncollaborative.com/knowledge-base/what-is-llms-txt-ai-crawler-guide/): Proposal, current support, and verification limits.
- [Answer engine optimization](https://www.andersoncollaborative.com/knowledge-base/answer-engine-optimization-aeo-definition-how-it-works/): Answer structure, evidence, and measurement.

A 200 response means the file was served. It is not evidence of ranking, AI Overview inclusion, or a citation.

Frequently Asked Questions

Do I need an llms.txt file for Google Search?

No. Google Search says it does not use llms.txt or similar AI text files for Search, including AI Overviews and AI Mode. Keep robots.txt, sitemaps, and crawlable HTML first.

Does every AI platform ignore llms.txt?

No. The Google Search statement is scoped to Google Search. Chrome Lighthouse can audit the file for agentic browsing. OpenAI, Anthropic, and Gemini publish llms.txt on their own developer docs. That is not the same as a public guarantee that each consumer chatbot uses third-party files.

Is llms.txt the same as robots.txt?

No. robots.txt is a crawl-access convention search engines have used for decades. llms.txt is a later Markdown overview that agents may read on demand. Nothing in the proposal requires a crawler to honor it.

Will adding llms.txt hurt Google SEO?

Google Search says maintaining the file neither helps nor harms Search visibility or rankings. A server error (5xx) when the file is requested can fail a Lighthouse agentic-browsing audit. A 404 is treated as not applicable because the file is optional.

Sources

• llmstxt.org, The /llms.txt file, v2, updated August 10, 2026, accessed September 20, 2026.
• Google Search Central, Optimizing your website for generative AI features on Google Search, updated July 10, 2026, accessed September 20, 2026.
• Chrome for Developers, Lighthouse llms.txt audit, updated May 5, 2026, accessed September 20, 2026.
• OpenAI, API docs llms.txt, accessed September 20, 2026.
• Anthropic, Developer documentation llms.txt, accessed September 20, 2026.
• Google AI for Developers, Gemini API docs llms.txt, accessed September 20, 2026.
• Search Engine Journal, Google says llms.txt is purely speculative for now, 2025 reporting, superseded for Search policy by the July 2026 guide above.
• Rank Prompt, accessed September 20, 2026.