llms.txt Generator

llms.txt is a proposed standard for a small markdown file you host at the root of your domain (next to robots.txt) that describes your site for AI crawlers. It was introduced by Jeremy Howard in 2024 and has been picked up by a growing list of search agents that prefer structured guidance over blind crawling.

The structure is deliberately simple. A site name and tagline at the top, a short intro paragraph, then one or more markdown sections, each containing a bulleted list of links with one-line descriptions. That’s the whole spec. A well-written llms.txt file fits on one screen and gives an AI crawler a curated shortlist of your best content along with context about what each page covers.

The payoff is citation quality. When an AI agent needs to answer a question about your domain, it has two options: crawl every page and guess at relevance (expensive, slow, and error-prone), or read your llms.txt and follow the curated trail you laid out. The second path produces better citations because you—the author who actually understands the content—get to decide which pages best represent your site for which topics.

This is the same idea robots.txt and XML sitemaps use for search engines, except aimed at a different audience with different needs. Search engine crawlers want every indexable page; AI agents want a curated set of representative pages that cover your site’s main topics well.

The interface is a familiar two-pane form. Type into any field on the left, the markdown preview on the right updates immediately. When the output looks right, click the download button in the preview header to save the file as llms.txt, then upload it to your server’s document root.

1. 1

   Start from the seed example (optional)

   Above the form there’s a Load XooCode example button. Click it to see a fully-populated llms.txt built from XooCode’s own site structure. Useful as a shape reference even if you clear it and start over for your own site.
2. 2

   Fill in site identity

   The top block asks for your site name (required), an optional tagline, and an intro blockquote. These become the # SiteName, italicized tagline, and > Intro paragraph at the top of the file. Keep the intro to two or three sentences maximum — this is the first thing an AI agent reads and it sets expectations for everything below.
3. 3

   Add link sections

   Click Add section to create a new heading with its own list of links. Typical sections are “Docs”, “Examples”, “Tools”, and “Optional” — the last of which is reserved by the spec for links the crawler can safely skip when token budget is tight.
4. 4

   Add links to each section

   Every link has a title, an absolute URL, and an optional one-line description. The description is what the crawler actually uses to decide whether to follow the link, so write it like a search snippet: concrete, specific, no fluff. “A Recipe markup example covering HowToStep composition and ISO 8601 durations” beats “Recipe example” by a mile.
5. 5

   Download and deploy

   Click the download button to save the file as llms.txt, then upload it to your server’s web root so it’s accessible at https://your-site.com/llms.txt. No special headers or config are required — serve it as text/plain; charset=utf-8 (most servers do this by default for .txt files) and you’re done.

The spec is short enough to fit inline. Here is the full structure the generator emits, with the required and optional blocks called out:

• # Site Name — an H1 heading. Required.
• > One-line tagline — a blockquote directly below the H1. Optional.
• An intro paragraph or two describing what the site is. Optional.
• ## Section Heading — an H2 marking a named group of links. Any number of sections.
• - [Title](url): description — bulleted markdown links under each H2 section. Descriptions are optional but strongly recommended.
• ## Optional — a reserved section name. Crawlers are free to skip the links under this heading when token budget is tight.

The Optional section name is reserved by the spec. When an AI crawler sees it, the agent is free to skip its contents when it needs to trim input tokens. Put nice-to-have references there — deep archives, tangential reading, older posts — so the crawler can read the rest of your file even on a tight budget.

Every other section name is author-defined. Conventional sections are Docs, Examples, Tools, and Guides, but the spec doesn’t require or forbid any specific names. Pick the names that make your site legible.

llms.txt lives at the document root of your site, the same place robots.txt and sitemap.xml live. For a site at example.com, the file must be reachable at exactly https://example.com/llms.txt — no subdirectories, no query strings, no redirects.

How you get it there depends on your stack. For a static site, drop the file in the public/ or static/ directory of your build and it’ll ship on your next deploy. For a framework like Next.js, put it in app/llms.txt/route.ts or public/llms.txt depending on whether you want it dynamic or static. For a traditional server, upload to the web root via FTP or SSH.

Serve the file as text/plain; charset=utf-8. Most web servers set this by default for .txt files, but it’s worth confirming with curl -I https://your-site.com/llms.txt after deploying — a text/html content-type will still work but signals less confidence that you know what you’re serving.

The llms.txt proposal is new (2024) and adoption is still in progress. As of this writing, the agents and tools that explicitly support or recommend llms.txt include:

• Anthropic’s Claude — Anthropic has published their own llms.txt file at docs.claude.com, signalling first-class support.
• Perplexity — Perplexity’s search agents read llms.txt when crawling sites for answer synthesis.
• Documentation tools — Tools like Mintlify, Fern, and Context7 read llms.txt files to build documentation indexes for AI assistants.
• Cursor, Windsurf, and other AI IDEs — These tools consume llms.txt when the developer adds a library as context, so their code suggestions can be grounded in the library’s own curated links.

ChatGPT, Gemini, and Bing don’t officially read llms.txt yet, but the file is cheap enough to ship that waiting for universal support is a false economy. The incremental cost of adding one is a few minutes with a generator; the value curve ramps as each agent adds support without requiring any change on your end.

There are three root-level files that AI tools sometimes read, and the differences between them are worth understanding.

If you have bandwidth for only one of the three, start with llms.txt. Add llms-full.txt later if you have an audience of agents that can’t fetch URLs. Ship robots.txt separately because it does a different job entirely.

The llms.txt Generator is tightly scoped to building one kind of file. Everything it doesn’t do is deliberate.

Three XooCode tools work together for root-level site metadata. Use all three for a site that communicates well with both traditional search engines and AI agents.

The generator follows the spec as published. Consult these sources when you need details the tool doesn’t cover.

• llmstxt.org — the proposal site. The spec lives here and is the definitive source for format rules.
• AnswerDotAI/llms-txt — the reference repository. Example files from adopters, discussion threads, and the open proposal for changes to the spec.
• Anthropic’s llms.txt — a real-world example from Anthropic’s documentation site. Good shape reference for what a well-curated llms.txt looks like on a large docs site.
• robotstxt.org — the robots.txt spec for comparison. Different file, different purpose, same root-level deployment pattern.