Every SEO newsletter in your inbox right now is telling you the same thing: add an llms.txt file or get left behind by ChatGPT, Perplexity, and Gemini. It’s being sold as the natural next step after Google AI Overview SEO — the new file every AI-ready site supposedly needs. Half of the newsletters pushing it are also selling a generator tool to build one for you. Almost none of them mention that Google has already said, on the record, that it doesn’t use the file at all.
That gap between the hype and the actual evidence is exactly where this post lives.
What llms.txt Actually Is
llms.txt is a plain Markdown file you place at your site’s root — yoursite.com/llms.txt — that gives AI systems a short, curated map of your most important pages, in the same spirit as robots.txt or sitemap.xml. It was proposed in September 2024 by Jeremy Howard of Answer.AI (the team behind fast.ai), specifically to solve a problem AI crawlers actually have: modern websites are full of navigation menus, tracking scripts, and JavaScript that a model has to wade through just to find the actual content, and all of that eats into a limited context window before the model ever gets to the point.
A basic file follows a fixed structure: an H1 with your site or company name, a one-line blockquote summary, then a handful of H2 sections listing your most important pages with a short description of each. No HTML, no clutter — just what matters.
It’s part of a broader pattern worth watching: standards built specifically so AI systems can understand a site without scraping it blind. The Universal Commerce Protocol is doing the same thing for AI shopping agents that llms.txt is doing for AI search and coding tools — structured, machine-readable context in place of raw HTML guesswork.
llms.txt vs. robots.txt: Different Jobs, Same Root Folder
The “new robots.txt” framing is catchy, but it’s a little misleading about what each file is for.
- robots.txt is a permissions file. It tells crawlers where they are and are not allowed to go — block the admin panel, block checkout pages, allow everything else.
- llms.txt is a curation file. It doesn’t restrict anything or grant access to anything an AI couldn’t already reach — it simply points to the pages you’d want an AI system to read first, if it’s going to read anything at all.
That distinction matters because it sets expectations correctly: llms.txt can’t hide anything, and it can’t force a model to prioritize your content. It’s a suggestion, not a directive.
The Part Nobody Wants to Put in the Headline
Here’s the honest state of adoption, because “optimizing for Perplexity & Gemini” only means something if those platforms are actually reading the file.
Google has explicitly declined to support it. In mid-2025, Google’s Gary Illyes confirmed the company doesn’t use llms.txt and has no plans to, and John Mueller compared it directly to the old keywords meta tag — something site owners could write to describe themselves, with no guarantee anyone was checking it against reality. Google’s own generative AI search guidance is blunt on this point: publishers don’t need new machine-readable files, AI text files, or Markdown versions to appear in AI Overviews or AI Mode. That applies to Gemini too, since it draws on the same retrieval and Knowledge Graph infrastructure as Google Search rather than checking for llms.txt at the domain root.
Perplexity is the more genuine case, but still not a confirmed ranking input. Perplexity relies heavily on real-time web retrieval rather than a pre-built index, which is why it’s often described as one of the more “crawl-friendly” AI search engines for publishers — clean, well-structured content generally fares better in its answers. But Perplexity’s own crawler documentation focuses on crawler access and robots.txt controls, not on llms.txt as something it looks for or depends on. Independent traffic studies back this up: large-scale reviews of AI-bot server logs consistently find that crawlers like PerplexityBot and ClaudeBot request llms.txt at negligible rates compared to normal page crawling.
Where it’s genuinely useful today is with AI coding and agent tools, not consumer answer engines. Tools like Cursor, GitHub Copilot, and Claude Code actively fetch /llms.txt when a developer is working against a library or API’s documentation, because it cuts token cost and retrieval noise dramatically compared to parsing full HTML docs. If your business is a SaaS product, API, or developer tool, this is the strongest, best-documented use case for shipping one — and it’s the same underlying shift we cover in Agentic AI for business: software that acts on a site’s behalf instead of a human clicking through it.
So the accurate framing isn’t “add this and rank in Gemini.” It’s: llms.txt is real, low-cost infrastructure for the emerging agentic web, with confirmed value for dev-facing tools and unconfirmed value for consumer AI search — and that’s worth knowing before you spend a week’s sprint on it.
Where This Fits Into a Real GEO Strategy
None of that means skip it. It means place it correctly. llms.txt is a cheap, five-minute addition to a real Generative Engine Optimization stack — not the strategy itself. The things that actually move the needle on AI citations are the same things that move traditional SEO: clear structured data, answer-first content, and genuine authority signals across the web, not a single file at your root.
How to Build One That Isn’t Junk
- Pick 10–30 pages, not your whole sitemap. llms.txt should read like a curated table of contents — your best product pages, docs, and cornerstone blog content — not a dump of every URL you own.
- Write the H1 and one-line summary like you’re briefing a new analyst. If the summary could describe four other companies, rewrite it.
- Organize by H2 section (Product, Resources, Docs) with a one-line description per link, following the format at the original specification, llmstxt.org.
- Test every link. A broken URL in llms.txt gives an AI system incomplete or wrong context about you — worse than not having the file at all.
- Keep it lean. Aim to stay well under 50KB. If you have deep documentation, a separate llms-full.txt can hold the complete version.
- Host it at the true root (yoursite.com/llms.txt), verify it loads as raw Markdown in a browser, and check it sits alongside your robots.txt and sitemap.xml — not buried in a subfolder.
FAQ
Does llms.txt help me rank on Google or appear in AI Overviews?
No. Google has stated directly that it doesn’t use llms.txt as a signal for Search, AI Overviews, or AI Mode, and Gemini relies on the same underlying systems.
Is llms.txt required to be cited by ChatGPT or Perplexity?
No. Neither platform documents llms.txt as a requirement or confirmed citation signal, though Perplexity’s real-time retrieval model makes it more plausible that clean llms.txt content gets picked up incidentally.
What’s the actual difference between llms.txt and robots.txt?
robots.txt controls what crawlers are allowed to access; llms.txt curates what you’d want an AI system to read first. One is permissions, the other is a suggestion — neither can restrict or guarantee anything on its own.
Is llms.txt worth building if I’m not a SaaS or dev tools company?
It’s low-risk either way, but the strongest documented benefit right now is for developer-facing sites and API documentation. For most other businesses, time is better spent on structured data, answer-first content, and authority-building — with llms.txt as a quick add-on, not the main project.
How long does it take to set up an llms.txt file?
For most sites, writing and hosting a basic llms.txt file takes well under an hour once you’ve picked your priority pages — it’s a plain text file, not a development project.
Do I need both robots.txt and llms.txt, or just one?
Both, if you want full control — they do different jobs. robots.txt manages access and blocking; llms.txt curates and describes what’s worth reading. Neither one replaces the other.
What’s the difference between llms.txt and llms-full.txt?
llms.txt is a short index pointing to your most important pages; llms-full.txt is a complete Markdown export of your entire site’s content in one file. Most businesses only need the standard llms.txt — llms-full.txt is mainly used by documentation-heavy sites and dev tools.
Can a bad or spammy llms.txt file hurt my site?
It won’t get you penalized the way keyword-stuffed pages can, since it’s not a confirmed ranking signal in the first place. But a sloppy or inaccurate file can mislead an AI system reading it, or hand competitors a clean map of your content strategy — so treat it with the same care as any public-facing document.
Which companies already have an llms.txt file?
Adoption is still early — independent studies put it at roughly one in ten sites as of 2026 — but it’s most common among developer tools, SaaS platforms, and documentation-heavy sites like Anthropic, Cloudflare, Vercel, and Zapier, where AI coding assistants get the clearest, most measurable benefit.
Does llms.txt cost anything to create or maintain?
No. It’s a free, static text file — no subscription, no plugin required, though free generator tools exist if you’d rather not write it by hand. The only ongoing cost is remembering to update it when your priority pages change.
Where To Actually Spend Your Time
llms.txt is worth an afternoon, not a strategy offsite. Ship it, keep it accurate, and put the rest of your energy into the content and authority signals that every AI platform — confirmed or not — is actually built to reward.
Ready to see what a real GEO strategy looks like for your site, beyond the file everyone’s talking about? Talk to our Experts.
