Archive
The whole corpus, in a search box
Six gigabytes of robots.txt, reduced to the only part that repeats — its vocabulary — and small enough to hand to a browser. Everything below runs on your machine. There is no server to ask.
What you are searching
A corpus that is mostly repetition
The archive fits in a page because robots.txt is written in a small language. 605,046 files reduce to 437,571 distinct bodies, and the 800 most-duplicated of those are served by 23.2% of the web.
Crawler names compress hardest of all: the entire web addresses just
23,808 distinct user-agent tokens, and all
of them are in this page. Paths are the opposite — every site has its own /wp-content/uploads/wpo
— so the 60,000 shipped here are the ones that recur, covering
67% of the disallow rules written on the web but nothing like
67% of the distinct strings.
Each row carries a ten-cell strip: where in the popularity ranking that term is used,
from rank 1 on the left to rank 1,000,000 on the right. Each strip is scaled against its
own busiest band, so it shows shape rather than volume — the count beside it is the volume.
A term used evenly across the web looks flat. /wp-admin/ leans right, into the long tail where WordPress lives.
content-signal leans hard right too, for a different reason — it arrived
through a CDN default rather than through anyone's editorial policy.
Clicking any term searches for it, which is the quickest way from a name to the actual files it appears in.
One category is missing from the crawler list on purpose.
23,443 files — 3.9% of the
corpus — arrive with no line breaks a parser can find, so the whole file collapses into a
single directive and the rest of it becomes the user-agent's name. They produce
4,142 entries that look like crawlers and are actually entire
robots.txt files, of which the most common is
lwp::simple.
A crawler reading one of these obeys nothing in it. They are counted, but they are not
crawler names, so they are not in the list.
Method & caveats
Built from the same crawl as the robots.txt report and
the pixel map: one GET to /robots.txt
per domain across the Tranco top-1M, 605,046 of which returned a parseable file.
Everything is counted once per site — a file repeating Disallow: / across forty
user-agent groups is one site's opinion, not forty.
Paths are stored exactly as written, because the variants are the point: the archive
distinguishes /admin from /admin/ and
/collections/*+* from /collections/*%2B*. Strings longer than 80
characters are truncated. Canonical files are grouped by exact byte-identity of the body and
shown truncated at 1,400 characters, with the full text one click away.
Two honest limits. The path vocabularies are the most common 60,000 and 15,000 entries, not all of them — a long tail of single-site paths is not carried, so a search that finds nothing means "not common", not "not present". And the counting pass trims its working set as it goes, which cannot lose anything frequent but means the number of distinct paths in the corpus is not something this page can state. Crawler names, directives and file counts are exact.
The payload is 0.73 MB of deflated JSON inlined as
base64 and inflated in the browser with DecompressionStream. That makes this
page about 1.0 MB to download, which is the bargain being
struck: once it has loaded, it is an archive you can save, carry offline,
and search with the network switched off. Generated
2026-08-12 by src/archive-extract.js and
src/report-archive.js.