I Probed a Million Sites for llms.txt. Half the 200s Were Junk, and My Accept Header Invented More.

llms.txt is a proposed file, like robots.txt, that a site can publish to tell language models what's on it. I wanted to know how many sites actually have one, so I fetched it from the Tranco top 1,000,000 domains.

The adoption number is not the interesting part of this post. The interesting part is that the run passed every check I had and was still wrong. It agreed with its own smoke run to about 99%. Then I re-fetched a sample with a different client from a different network, read the stored bodies, and found three systematic errors. One came from an Accept header I'd added on purpose.

If you run crawls that produce a dataset someone else will quote, this is about how to find out you're wrong before they do. The adoption findings themselves live on the Crawlora dataset page; I'm not repeating them here.

TL;DR

• A 200 at /llms.txt is not a hit. In my calibration sample, 47.0% of the 200s were HTML soft-404 pages, and 47.1% were real markdown files. Only 2 of 448 valid files had an HTML Content-Type, so you have to validate from the body.
• Two runs of the same worker agreeing proves repeatability, not correctness. My smoke run shared the bug with the full run, so it agreed on the false positives.
• Naming text/markdown in Accept made some servers convert their homepage or a soft-404 into well-formed markdown, and my worker stored it as a valid file. A different client found it.
• A redirect that fails on a DNS lookup looks like a clean 301 unless you record the error next to the status. I hit that once on my laptop and once in production.
• "Adoption" depends on the denominator. I got 11.6% and 13.3% from the same data, and both are defensible.

Calibrate small first

Before the full run I probed 4,918 domains: 3,700 random reachable ones in five rank bands, plus control groups (sites Common Crawl had found a file on, defended sites, and unreachable ones). It ran from a pod in a datacenter, using curl-impersonate, 32 domains in parallel, in 1,665 seconds. The pod was CPU-throttled at 0.5 cores and managed about 3 domains per second. That sample is where the first two lessons came from.

Status codes lie. Of 3,557 live random domains, /llms.txt returned 404 for 60.6%, 200 for 26.7% and 403 for 8.5%. Of the 951 200s, 47.1% were valid markdown and 47.0% were HTML soft-404s. The rest were empty files, binaries, a robots.txt at the wrong path, and a little JSON. So I validate every file from its body.

A headline rate can be one vendor. The first estimate of adoption was 11.52% of reachable domains. Of that, 3.64 points were a single Shopify-pushed template titled Agent Instructions - <store>, and 81 of the 87 sampled Shopify sites published it. Without that template the rate was about 7.9%. A number that big needs a "who wrote these files" split next to it, or it just measures one company's rollout.

The calibration also taught me not to probe from my laptop. My Mac's resolver hands out fake 198.18.x.x addresses, which my crawler's SSRF guard correctly refuses to follow. The statuses came back as bare 301s with the error swallowed. A redirect blocked by the guard looks exactly like a clean redirect. Hold that thought.

The full run

The full run covered 1,000,000 domains on a pinned Tranco list (the same list id my other censuses use, so ranks join across datasets). I split it into 200 shards of 5,000 domains and ran four pods at concurrency 150 with no proxy. Final result: 798,540 reachable domains. For /llms.txt: 92,631 present, 603,639 absent, 77,579 blocked and 24,691 errors.

It didn't run straight through. After about 140,000 domains the pods' /metrics scrapes started timing out, because I had a metric labelled by the upstream host and a census visits a million hosts. I scaled to zero, bounded the label, and resumed under the same run id. The Redis cursors continued and the stored documents were kept. After the fix a pod exposed about 715 samples and scraped in 10 to 150 ms. The first ~140k domains were probed by the unfixed image, but the probe and analyzer code was identical in both.

I estimated about seven hours at roughly 39 docs/s, and I did not record the actual total wall-clock time, so I won't give one.

99% agreement, and wrong anyway

Before the full run, I ran the worker over the same 4,918 domains as a smoke test. It matched the calibration closely: valid /llms.txt was 11.52% in both, and per-domain agreement was 97.4%. Run twice, an hour apart, the worker agreed with itself at 99.0%, so the noise floor is about 1%.

That smoke validation did find real defects. My binary sniff read the first 4,096 bytes and cut mid-character, so any UTF-8 file over 4 KB with a multi-byte character at the cut read as binary (25 of about 1,000 present files). The calibration had the same flaw, which is part of why the two agreed so well. An HTML page that opened with a comment slipped past the HTML check, and [x](y) inside its JavaScript satisfied the markdown check, so a parked page counted as a valid llms.txt and as 99 "present" robots.txt files. And "robots.txt present" read 72.4% against 68.1% in the calibration, because comment-only files and plain text pages counted. All fixed.

After that I considered the pipeline validated. It wasn't.

After the full run, I took 1,341 domains stratified by state and re-fetched them with curl-impersonate from a different box on a different network. Then I read the stored bodies. Three more problems appeared.

Three errors the cross-check found

1. Redirects dropped under DNS load. In the first half of the run, 2.4% of probed files were stored as error with a final status of 301 or 302, against 0.6% later. The cause was my SSRF guard resolving each redirect target through a cluster DNS path that was saturated at that point. The lookup failed, the redirect was dropped, and the file was recorded as an error. It was the laptop bug again, at scale. I only saw it because I broke the quality metrics down by probed_at hour; the run-wide average hid it. I re-probed 38,302 domains with node-local DNS. 10,133 former errors became absent and 1,199 became present.

2. My Accept header asked for markdown. I'd sent Accept: text/plain, text/markdown, */*, which seemed polite for a markdown file. But a growing number of servers, Cloudflare's "Markdown for Agents" among them, convert any HTML page to markdown for a client that asks. So a homepage, or a "page not found" page, arrived as well-formed markdown and I stored it as a valid file. This affected about 1% of valid files. 625 redirected to another page with front matter, and 72 were titled "page not found". The reverse also happened: a real file served as text/html came back re-escaped and read as absent.

The probe now sends text/plain, */*;q=0.8, and a comment in the code explains why text/markdown must not appear. I re-probed 5,270 candidates. 983 valid files became absent and 51 absent became valid. 969 of the Shopify stores in that batch answered with a 429 to the burst of re-probes, homepage included, so I discarded those results and kept the originals. A re-probe that comes back newly blocked must never overwrite an established state.

3. Conversion regardless of the header. About 0.1% of pages were still converted even without asking. The analyzer now rejects a leading front-matter block with no heading, an H1 reading "not found" or "404", and a redirect to a URL with no "llm" in it and no .txt or .md extension.

Across the whole cross-check, present files agreed 92 to 97% with the other client, absent 94 to 95%, and blocked 91%, and nearly every disagreement was the other client being refused. I also compared my per-crawler robots.txt stances with what Python's urllib.robotparser says about the independently fetched files: 99.7% agreement over 651 sites and 7 crawlers. And the file "kinds" lined up with a separate tech-stack census: the Shopify template appeared on sites that census marks Shopify 90% of the time, Yoast files on WordPress 92%, Wix files on Wix 97%.

One finding I can't resolve: of domains I stored as unreachable, 5.5% answered the other client. So reachable = false means "unknown", not "dead", and about 1% of all domains in the run are reachable sites I couldn't reach.

The denominator

The public page says 13.3%: 92,631 files out of 696,270 domains where I got a definite present or absent answer. My earlier internal note said 11.6%, which divides by all 798,540 reachable domains. That second figure counts the blocked and error domains as if they had no file, and they didn't. They're unknown. Neither number is wrong, but they answer different questions, and quoting one without naming its denominator is how a 1.7-point gap becomes an argument.

I also can't show you a before-and-after on the headline from the pre-fix run, because the recheck indices and the smoke index were deleted once the merged data was validated. The numbers above for each defect come from the notes I wrote at the time.

What I'd do again

• Cross-validate with a different client from a different network, and read the stored bodies. Two runs of your own worker share your bugs. The Accept header was invisible to every check that used it.
• Break every quality metric down by time of probe. The DNS artifact was 2.4% in one half of the run and 0.6% in the other. The overall average looked fine.
• Record the error even when you also have a status. A blocked redirect and a clean redirect should never look the same in your data.
• Don't let a re-probe overwrite an established state with "blocked". A rate limit says something about your recheck, not about the site.
• Never probe from your laptop. And if you must move results off a pod when kubectl crawls at about 5 KB/s, write them as base64 chunk documents into a scratch index, read them back, and verify the sha256.

What this still doesn't tell you

All of this is one vantage point: datacenter IPs, no proxy, one date. 77,579 domains answered "blocked", and some of those would probably serve a file to a residential connection. I didn't test that. A hand-written file with front matter and a converted page that starts with a heading can't be told apart by content, so a few dozen of each may be misfiled. And roughly 145,000 unreachable candidates were never re-checked from the other vantage point, where I'd expect to gain about 11,000 domains.

The data is browsable at crawlora.net/datasets/llms, including the file kinds and the robots stances. If you re-run any of this and get a different number, I'd like to know which trap you found.