How to Write an llms.txt File: A 6-Step Walkthrough
An llms.txt file is a plain-markdown manifest that sits at the root of your domain and lists the pages of your site you would most like AI assistants to read. Whether a given assistant fetches it is up to that assistant. Unlike a sitemap, it is written for language models rather than search spiders — it uses headings, short link lists and one-line descriptions instead of raw URL dumps. Written correctly, it gives any assistant that does read it a clean map of your best content. Written carelessly, it gets ignored. This walkthrough covers the exact six steps to draft, structure, validate and publish one.
What you'll need before you start
llms.txt has no account tier, no paid plan and no vendor dependency — but it does have prerequisites. Skipping them is the most common reason a file that looks correct on paper still gets dismissed by crawlers.
- Root-level file access. The file must resolve at https://yourdomain.com/llms.txt — not in a subfolder, not behind a CMS page builder. If you use Shopify, Cloudflare Pages, Vercel or a similar host, confirm you can publish a static file at the domain root.
- A shortlist of canonical URLs, not every URL. Ten to forty pages is a healthy target. The point is curation, not completeness — the full sitemap handles the long tail.
- One-line descriptions for each URL. Not marketing copy — a factual sentence explaining what the page contains.
- Your existing robots.txt. You need to read it before you touch it. Half of llms.txt problems are actually robots.txt problems wearing a different name.
- A plain-text editor. Anything that outputs UTF-8 markdown without smart quotes works. Word processors that silently convert straight quotes to curly ones will break the links.
For a broader orientation on how AI-readable files sit next to sitemaps and robots.txt in a modern stack, the SeoAiX learning hub walks through the whole picture.
Step-by-step walkthrough
The six steps below are the same sequence we use when preparing a site for AI crawler readiness. Each step has one output: a single section of the file you will assemble at the end.
Step 1 — Assemble your canonical URL shortlist
Pull URLs from your sitemap, but cut ruthlessly. Keep the pages you would want an AI engine to quote in a sentence: pillar guides, service explainers, product category pages, and the two or three articles that already attract the most citations from other sources. Exclude tag pages, pagination URLs, duplicate locale variants and anything with a query string.
The shortlist is the editorial spine of the file — if it is noisy, the file is noisy.
Step 2 — Write one factual description per URL
Each link gets a short sentence that describes what is on the page, not what it is for. Compare these two:
- Weak: "Our award-winning SEO platform — the best in the industry."
- Strong: "Explains how SeoAiX injects Article, Product and FAQPage schema across Shopify product and blog pages."
The strong version reads like a caption. That is the register AI engines handle best — descriptive, specific, no adjectives that a language model has to discount.
Step 3 — Draft the file in markdown
The llms.txt convention is intentionally minimal. A conforming file looks like this:
# Acme Analytics
> Cloud analytics for B2B SaaS teams. Handles event ingestion,
> funnels and cohort reporting at the warehouse layer.
## Docs
- Getting started: Install the SDK and fire your first event.
- Warehouse sync: Connect Snowflake, BigQuery or Redshift.
- Pricing: Per-event tiers and overage rules.
## Blog
- 2026 benchmark report: Median query latency by plan tier.
Three structural rules matter more than anything else: an H1 with your site or product name, a > blockquote summary of roughly one to three sentences, and then H2 sections of markdown link lists. Each list item is [label](url): description. That is the whole schema.
Pro tip: write the blockquote summary last. It is the single most-quoted line by AI engines, and it is far easier to compress once the list underneath it is final.
Step 4 — Generate a first draft if you would rather not write it cold
Handwriting llms.txt is fine when you have a dozen URLs. Past thirty, the error rate climbs — duplicated titles, trailing slashes, descriptions that drift into marketing copy. A purpose-built tool removes that drift by pulling from your sitemap and enforcing the markdown shape for you.
The SeoAiX llms.txt generator does exactly that: it takes a domain, groups URLs by section, and outputs a conforming file ready to copy. It pairs naturally with the wider AEO and schema features if you are building the file as part of a larger AI-readability pass.
Step 5 — Validate before you publish
Run through this list every time you edit the file:
- Every URL returns HTTP 200 — paste each into a browser and confirm.
- No smart quotes. Apostrophes and quotation marks must be straight ASCII.
- Link text is human-readable, not the raw URL.
- Descriptions are one sentence, factual, and free of superlatives.
- H2 section names are consistent across the file.
- The blockquote summary names the organisation and what it does.
Step 6 — Publish at the domain root and set crawl headers
Upload the file so it resolves at https://yourdomain.com/llms.txt, served with content type text/plain; charset=utf-8. Do not redirect it. Do not password-protect it. If your host allows custom headers, confirm the file returns 200 without a hop.
Confirm your robots.txt permits fetching the same paths. AI crawlers cannot cite what they cannot reach, and a block in robots.txt overrides a friendly llms.txt every time.
How to verify it worked
Verification is mostly structural — you are checking that the file is reachable, well-formed and consistent with the crawl permissions around it.
- Fetch it directly. Open yourdomain.com/llms.txt in a browser and confirm it renders as plain text with no HTML wrapper.
- Check the status code. A 200 is required. A 301 to a different path, a 403 from a WAF rule, or a 404 from a case-sensitive server all mean the file will be skipped.
- Cross-check robots.txt. Run /robots.txt and confirm that every path you listed is not disallowed for user-agents like GPTBot or ClaudeBot.
- Watch your server logs. Within a few weeks of a correct publish you should see requests from AI user-agents hitting the file path. If no fetch hits appear at all after four to six weeks, the most probable cause is a redirect or a content-type mismatch.
Failure mode to watch for: the file renders correctly in the browser but arrives with a text/html content type because your CDN rewrites unknown extensions. The bytes may be fine and the file will still be ignored. Inspect the response headers, not just what is on screen.
Troubleshooting common issues
The file is live but AI crawlers never request it
Often this is a crawl-permission problem, not an llms.txt problem — and some assistants simply do not request the file. Check your robots.txt for Disallow: / on AI user-agents, and confirm no WAF rule is silently serving a challenge page to bot traffic. Verify with a fetch from a clean network before assuming the file itself is the issue.
Links resolve for you in a browser but not for a fetch tool
Browser autocomplete will happily add the missing www. or upgrade to HTTPS. A crawler will not. Pin every URL in the file to the exact canonical host and scheme — mismatches here are the second most common cause of dropped citations.
The file was copy-pasted from a document and broke
Markdown links need straight parentheses and ASCII brackets. Rich-text editors quietly substitute curly characters. Open the file in a code editor and search for the characters [ and ] — if you cannot find them as plain ASCII, retype the affected lines.
You have llms.txt and llms-full.txt and neither behaves
The two files have different jobs. llms.txt is the curated index. llms-full.txt is the full-corpus variant some crawlers support, with entire page contents inlined. If you are unsure which to serve first, ship llms.txt alone — it is the only one of the pair with broad tooling support, and it will not confuse a crawler that does not understand the full variant.
Your descriptions keep getting quoted out of context
If an engine lifts a fragment from your llms.txt and it reads badly as a standalone sentence, the fragment was too promotional. Rewrite descriptions as factual captions — subject, verb, object. The file is not a sales channel, and it is not indexed the way a landing page is.
Common questions
Is llms.txt officially supported by Google or OpenAI?
No. There is no standards body backing llms.txt, and as of early 2026 the major model providers have not published a formal commitment to consume it. It is a community convention with growing adoption, not a ranking signal. Treat it as a low-cost, low-risk signal that may help AI crawlers parse your site more cleanly — not as a guaranteed lever.
What should an llms.txt file actually contain?
Three things: an H1 with your site or product name, a one-to-three sentence blockquote summary of what the site is, and H2 sections of markdown link lists. Each link is [label](url): one factual sentence. Anything beyond that — extra metadata, JSON blocks, nested lists — is optional and inconsistently supported.
Does adding llms.txt improve my rankings on Google?
No. Google Search does not consume llms.txt, so it has no bearing on standard organic ranking. Its purpose is discoverability inside AI answer engines, which cite sources in a different way. If your goal is traditional SERP performance, the priorities remain content quality, page experience and conventional structured data.
How many URLs should llms.txt list?
Curate tightly. Ten to forty of your strongest, most citable pages is a workable range. Listing every URL from your sitemap defeats the purpose — the file is a signal of editorial priority, not a mirrored index.
How often should I update the file?
Whenever the underlying content changes: a pillar page is rewritten, a product category is retired, a new flagship guide ships. A quarterly audit is a sensible floor. Stale links in the file are worse than missing links because they waste crawl attention.
Going further: related workflows
llms.txt rarely lives alone. Two adjacent files do most of the heavy lifting alongside it:
- AGENTS.md — a machine-readable description of how automated agents should interact with your storefront, including contact endpoints and purchase constraints. Sites running AI-assisted commerce need one, and the AGENTS.md walkthrough covers the Shopify-specific format.
- Schema markup — JSON-LD structured data gives AI engines explicit facts about products, articles and FAQs. It is separate from llms.txt and complementary to it. The schema injection features page explains what gets deployed on product, collection and blog pages.
If you want a fixed starting point rather than a blank file, the free llms.txt generator produces a conforming draft from any domain in under a minute.
When this approach is the wrong fit
llms.txt is a poor use of time in three situations. If your site has fewer than ten substantial pages, a well-written sitemap and clean on-page structure matter more. If you are under active penalty or the content itself is thin, the file will not compensate. And if your audience searches exclusively through Google and never via an AI assistant, the marginal value is close to zero — the file exists for engines that read markdown, not for the classic index.
Treat it as one well-shaped signal in a broader AI-readability stack, not as a standalone fix.