The web wrote 605,000 robots files. It said far fewer things.

01 — The corpus

What 605,000 files add up to

Every robots.txt served by the Tranco top million, parsed line by line: 31,819,812 lines, 13,546,143 disallow rules, and 0.85 GB of plain text written almost entirely for machines to read.

The average file runs 52.6 lines. Of the disallow rules, 28.9% contain a wildcard and 8.0% end in a file extension — the web blocking individual files rather than directories.

02 — Boilerplate

The web is a photocopy

Strip the comments, lowercase the directives, sort them, and hash the result. Two files with the same hash express exactly the same policy. What comes back is the single most striking fact in this corpus.

16.4% of every robots.txt on the web — 99,015 files — reduces to just 40 distinct signatures. The most common one alone accounts for 30,013 sites. And the signatures at the top are not elaborate: most are two lines long.

Four of the ten most common robots.txt files on the internet say only “everyone, everything” or “everyone, nothing.” They are the two shortest possible opinions, and together they cover tens of thousands of sites.

Note what this does to the coverage report next door. When 16.4% of files are identical to some other file, most robots.txt “decisions” are not decisions at all. They are defaults, inherited from a CMS, a host or a CDN — which is exactly what the rank-independence of AI blocking implied from the other direction.

03 — A new directive

Something appeared that the standard has never heard of

The most common signature in the corpus is not a robots.txt in the traditional sense at all. It carries a directive that does not exist in RFC 9309.

allow: /
content-signal: search=yes,ai-train=no,use=reference
disallow: /
disallow: /
disallow: /
disallow: /

Content-Signal appears in 58,034 files. Rather than blocking a crawler by name, it declares what the content may be used for — search=yes, ai-train=no — and leaves the crawling itself permitted. It is a licence expressed in a file that has no licensing mechanism, aimed at companies under no obligation to read it. A second contender, llm-policy, appears in 2,314 more.

04 — The comments

190 MB of prose nobody reads

46.4% of files carry at least one comment — 3,363,309 lines of natural language, addressed to an audience of parsers that discard them.

The largest category is the least human. Over half of all commented files mention a plugin, a generator, or an instruction not to edit — the comments are themselves machine output. Legal and rights language comes second, AI and training third.

Ranking the vocabulary makes the same point. These are the most frequent words across every comment in the corpus:

Not one of them is addressed to a person. content, signals, corresponding, operator and rights are the vocabulary of the Content Signals boilerplate; cloudflare, managed and shopify name the machines that wrote the file. Only 396 files in the entire corpus — 0.1% of commented ones — contain a recruiting message, the classic robots.txt easter egg. It is effectively extinct.

05 — What relates to what

Crawlers travel in constellations

Site owners do not evaluate crawlers one at a time. Grouping user-agents by how often they are named together — connected components over a Jaccard overlap graph — recovers distinct blocks that get chosen as units.

None of these groups was defined in advance; they fell out of the co-occurrence structure. And they map cleanly onto real artefacts. The largest is the modern AI blocklist shipped by CDNs and CMS plugins. Another is the SEO-tool tier — AhrefsBot, SemrushBot, MJ12bot, DotBot — a decade-old copy-paste block that predates the AI ones entirely. A third is just Googlebot and Bingbot, the pair everyone writes rules for on purpose.

You can date a robots.txt by which constellation it contains. The SEO block is the sediment of one era; the AI block is the sediment of the next.

Lift above 1 means the pair appears together more often than independence would predict. A→B is the share of sites naming A that also name B.

06 — Fingerprints

Paths that always arrive together

Run the same clustering over disallowed paths and something different emerges: not policy, but platform. Each cluster is a piece of software's default output.

These are identifiable on sight. Shopify ships /checkout, /cart/, /orders and the /collections/*+* filter patterns. Drupal ships /node/add/, /comment/reply/, /filter/tips and /core/, each duplicated behind /index.php. WooCommerce ships the /wp-content/uploads/wc-logs/ trio. The cohesion figures — most above 0.95 — mean the paths are almost never seen apart.

A robots.txt is therefore a reliable fingerprint of the stack underneath it, whether or not the site meant to disclose that.

07 — Meaning

What the rules are actually about

Classifying all 13,546,143 disallow rules by what they refer to gives the web's blocking priorities in aggregate.

The shape agrees with the coverage report's finding that the web blocks infinite spaces and private surfaces. Search and filtering, accounts, and admin interfaces dominate. Blanket whole-site blocks are the second largest single category, driven by the parked and unlaunched tail.

Which paths each crawler is steered away from

Crossing agents against paths shows what sites are protecting when they name a specific crawler. Lift compares against that path's base rate across the whole corpus.

The AI crawlers cluster on the same three targets: the site root, account areas and admin paths. When a site singles out an AI crawler, it is usually not curating which sections may be read — it is closing the whole door, or the doors that were already shut to everyone.

08 — Language

An overwhelmingly English, overwhelmingly ASCII file

Despite covering a million domains across every country, the text is near-uniform.

Non-Latin scripts appear in a fraction of a percent of files. robots.txt is a technical artefact written in the lingua franca of technical artefacts — and increasingly it is not written by hand at all, which makes the language question mostly moot.