Skip to content
Leadee
Insight · Technical readiness

llms.txt: What It Does and Does Not Do

llms.txt is a Markdown file at your site root that gives AI agents a short, curated map of your key pages. It does one job: help an agent find your best content quickly when a user asks about your topic. It does not control crawling, does not change Google rankings, and has no proven effect on AI citations. Get the format right, then spend your effort on content.

By the Netbase JSC team · Updated

Key facts

llmstxt.org, 2024 (checked 2026-11-13)
The llms.txt proposal was published by Jeremy Howard on 2024-09-03. It says the file helps "agents use a website", that an H1 with the site name "is the only required section", and that the file is used "on demand, when an agent needs information about a topic while assisting a user". (llmstxt.org, 2024 (checked 2026-11-13))
SE Ranking, 2026 (checked 2026-11-13)
In a study of nearly 300,000 domains published 2026-10-15, only 10.13% had an llms.txt file, and the researchers found no correlation between having one and how often AI engines cited the domain. Removing the llms.txt variable from their model made it more accurate. (SE Ranking, 2026 (checked 2026-11-13))
Google Search Central, 2026 (checked 2026-11-13)
Google says you "don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search", including its AI features, and that an llms.txt file will neither help nor hurt your visibility there because Google Search ignores it. (Google Search Central, 2026 (checked 2026-11-13))
OpenAI, 2026 (checked 2026-11-13)
OpenAI's crawler docs tell site owners to control OAI-SearchBot, GPTBot and ChatGPT-User through robots.txt and published IP lists. They do not say that any of these bots reads a site's llms.txt. (OpenAI, 2026 (checked 2026-11-13))
Leadee test, Nov 2026
In our test of 2026-11-09, 25 of the 52 third-party domains cited most in one tracking round served an llms.txt and 20 did not. The llms.txt on cmsmart.net (CMSmart, part of Netbase JSC) had 48 format problems. (Leadee test, Nov 2026)

Does your llms.txt pass the format check?

The free llms.txt generator drafts a file from your own page titles and descriptions, then checks it with the same validator as Leadee's free scan, line by line. Rule-based, no account. A valid file is a quick fix; it will not make an AI engine cite you.

Leadee is not affiliated with, endorsed by, or sponsored by OpenAI. “ChatGPT”, “GPTBot” and the other names on this page are OpenAI's own, used only to describe what this tool measures.

What does llms.txt do?

llms.txt is a plain text file, written in Markdown (a simple formatting style with # headings and `text links), placed at yoursite.com/llms.txt`. It lists the pages you most want an AI agent to read, with a one-line note on each. An AI agent here means software, such as a chat assistant, that fetches web pages while it answers a user.

The llmstxt.org proposal says the file is read "on demand, when an agent needs information about a topic while assisting a user". So its real jobs are small and practical:

  • It gives an agent a short, curated table of contents, so it can find your key pages without crawling the whole site.
  • It lets you point to clean versions of pages, such as Markdown copies, that are easier for a language model to read than full HTML.
  • It says, in your own words, what your company is and does, in one place an agent can read in a single request.
  • It marks lower-priority links under a ## Optional section, which an agent with little room can skip.

What does llms.txt not do?

Most myths about llms.txt come from treating it like robots.txt or a ranking signal. It is neither. Here is what the file does not do, based on what the proposal, Google and OpenAI publish:

  • It does not control access. robots.txt tells crawlers what they may fetch; llms.txt cannot block or allow any bot. To manage AI crawlers, see AI crawlers and robots.txt.
  • It does not opt you out of AI training. OpenAI documents GPTBot and robots.txt for that, not llms.txt.
  • It does not replace your sitemap. A sitemap lists every page for search engines; llms.txt is a short, hand-picked list. See sitemaps for AI search.
  • It does not affect Google rankings, AI Overviews or AI Mode. Google says Google Search ignores the file.
  • It has no proven effect on AI citations. The largest independent study found no link between having the file and being cited.
  • It does not answer buyer questions for you. AI search cites page passages, not your index file.

What does a valid llms.txt file look like?

The llmstxt.org format has one required element: an H1 heading with your site or company name on the very first line. Around it sits a conventional shape: a one-line blockquote summary right after the H1, optional free-text paragraphs, then one or more ## Section Name headings. Each heading is followed by a bullet list of links in the exact form `- Link title: optional description`.

A section named ## Optional is a convention from the proposal itself. It marks links an AI agent with a short context window (the amount of text a model can hold at once) is free to skip.

  • # Acme Manufacturing: the required H1, your company or site name, nothing else on that line.
  • > Acme makes precision-machined aluminium brackets for the aerospace and automotive industries.: the one-line summary.
  • ## Products followed by `- Aerospace brackets: CNC-machined to AS9100 spec`: a section with one correctly formatted link.
  • ## Optional followed by `- Careers`: secondary links an agent can skip.

What are the most common llms.txt errors?

Leadee's checker parses the file the way an AI agent's parser would. It reports a specific problem string for each thing it finds wrong. These are the exact messages you will see in a scan result, not a paraphrase.

Errors are common even on sites that took the trouble to publish a file. In our test of 2026-11-09, the llms.txt on cmsmart.net (CMSmart, part of Netbase JSC) had 48 problems: lines under "Key facts" and "Contact" were plain bullets, not `- title` links. Of the 25 third-party files in the same test, 10 passed with zero problems.

  • "missing H1 title — the file must start with '# Site Name'": the first non-blank line was not a single # heading; a blank line or a ## section before the H1 triggers this.
  • "missing summary blockquote ('> one-line description') after the H1": no > blockquote line directly follows the title.
  • "no '## Section' with a link list found": the file has an H1 and maybe prose, but no ## heading at all.
  • "malformed link under '<section name>' — expected '- title: description'": a line under a section starts with - but is not a real Markdown link, often a bullet of plain text or a link missing its closing ).
  • "section(s) found but no valid links in any of them": every ## section is empty, or every line under it failed to parse as a link.

What is llms-full.txt, and do I need it?

llms-full.txt is the same idea at a larger scale. Instead of a short index of links, it holds the full text an agent would need, such as summaries, key facts and sometimes whole page bodies, in one file. An agent with a large context window can then read your site's substance without another fetch.

The llmstxt.org proposal does not define llms-full.txt; it is a convention some documentation sites added. Leadee's check gives a small score bump for having both files, because the longer file costs little once the short one exists and some agents prefer one consolidated document.

Does llms.txt affect whether AI engines cite you?

The best independent measurement says no, not in any way anyone has shown. SE Ranking's study of nearly 300,000 domains (published 2026-10-15) found 10.13% adoption and no statistical link between having the file and how often a domain is cited in AI answers. Removing llms.txt from their prediction model made it more accurate, which is the opposite of what a real lever would do.

Our own smaller test points the same way: woocommerce.com was cited 103 times without an llms.txt. Presence is not use, though: none of this shows whether an engine fetched any file. Leadee scores llms.txt as a low-severity check for that reason. Fix it, but fix crawler access, schema markup and clear answers first. For the ChatGPT-only view, read does llms.txt help with ChatGPT?.

Does Google use llms.txt for ranking or AI Overviews?

No, and Google says so directly. Google's AI optimization guide says you "don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search", and that covers its generative AI features too. It adds that it is "completely fine" to keep an llms.txt for other services, but that the file will not help or hurt your Google visibility because Google Search ignores it.

The same guide tells site owners they can ignore "creating unnecessary AI text files (like llms.txt)". Google points instead to the usual fundamentals: crawlable pages, useful content, structured data that matches the page, and a current sitemap.

Where does llms.txt fit in a query fan-out content plan?

Query fan-out is how AI search handles one question: it splits it into several related searches, then writes one answer from what it found. Those searches land on page passages in a search index, not on your llms.txt. So the file cannot answer the sub-questions about price, fit or proof that decide who gets cited. Our service article, query fan-out explained, sets out the content strategy that does.

Use the two together. Run a buyer prompt through the free Query Fan-Out tool to list its likely sub-questions, give each one an owner page, and structure that page with our guide to content structure for passage retrieval. Then list those owner pages in your llms.txt, so an agent that does read the file finds them first.

What have we not verified?

  • Whether any major AI search engine fetches llms.txt when it builds an answer: none of OpenAI, Google, Anthropic or Perplexity has published that it does.
  • Whether fixing a malformed llms.txt changes how often a site is cited: we have not run a before-and-after test.
  • How agents treat llms-full.txt compared with llms.txt: the proposal does not define the longer file.

Check this on your site

A valid llms.txt is a small fix. The bigger lever is answering the sub-questions behind each buyer prompt. The free Query Fan-Out tool lists them, and the service article shows the content strategy.

Tools & technical readiness

llms.txt Generator

Answers: Enter your domain and Leadee drafts an llms.txt in llmstxt.org format from your own titles and descriptions — no AI rewriting. Validate an existing file too.

No sign-upGenerate llms.txt — llms.txt Generator

Questions

Is llms.txt the same thing as robots.txt?

No. robots.txt tells crawlers which paths they may or may not fetch. llms.txt is a curated index of your key pages for AI agents to read. A site needs both for different jobs; see [AI crawlers and robots.txt](/learn/ai-crawlers-robots-txt) for the crawler-access side.

Has any major AI vendor confirmed it reads llms.txt?

Not that we have found documented. Google says Google Search ignores the file. OpenAI documents robots.txt and IP lists as the way to control its crawlers and does not mention llms.txt there. We found no statement from Anthropic or Perplexity either.

Can llms.txt stop AI models from training on my content?

No. llms.txt has no access rules. To opt out of training by OpenAI, disallow GPTBot in robots.txt, which OpenAI documents as the control for training. Other vendors name their own crawlers.

What's the single most common mistake in an llms.txt file?

In the files we have checked, it is link lines that are not real Markdown links, such as plain bullets of facts under a section. A missing or misplaced H1 is another; a blank line or a `##` section before the title triggers the "missing H1 title" error.

Should I generate llms.txt automatically from my sitemap?

You can, but curate it rather than listing every URL. The format's point is a short, prioritised list an agent reads quickly, closer to a table of contents than a full site index.

If llms.txt barely matters, why does Leadee check for it at all?

Because some agents may read it, it costs almost nothing to get right, and a malformed file is a quick, free fix. It just should not be where you spend your first hour of effort.

Do I need to update llms.txt every time I publish a new page?

Only for pages important enough to belong in a short curated index. New blog posts rarely qualify; new product or service pages, and the owner pages from your fan-out map, usually do.

Can GEO guarantee results?

No. AI answer engines change their output frequently, and no vendor — including Leadee — can guarantee a ranking or a mention. Leadee publishes confidence intervals on every metric so you can see how much a result can vary run to run.

How does Leadee measure AI visibility?

Leadee runs your prompt set against each AI engine on a weekly to daily schedule, depending on plan, through official vendor APIs (OpenAI, Gemini, Perplexity, Anthropic, DataForSEO) — never by scraping consumer chat interfaces — and reports mention rate, citation rate, average position, share of voice and sentiment with a 95% confidence interval.

Who do I talk to?

Email support@leadee.ai or use Talk to an expert at https://leadee.ai/contact-sales — the Netbase JSC team behind Leadee replies by email or WhatsApp — or WhatsApp the CEO directly at https://wa.me/84937869689.

See where your own site stands

Run a free scan to check this (and everything else Leadee tracks) on your own site.

Run a free scan