A million robots.txt files, one pixel each

01 — The whole corpus

Every domain, in rank order

Rank 1 is the top-left pixel. Rank 1,000,000 is the bottom-right. Each row is 1,000 consecutive places in the ranking, so vertical position is popularity — and any horizontal banding is a change in behaviour as the list gets deeper.

Hovering reads a single domain's record back out. Clicking a key fades that class into the paper, which is the fastest way to see where one behaviour lives: fade everything but blanket Disallow: / and the bottom third of the picture is visibly denser than the top.

Two things only show up at this resolution. The share of domains that answer at all barely moves across a thousandfold range of popularity — the web's least famous million behaves almost exactly like its most famous thousand. And blanket blocking climbs steadily into the tail, where the corpus fills with parked domains and unlaunched projects that block everything as a side effect of not existing yet.

The third thing is the ochre streak marked at rank 745,000, and it is the reason to draw the picture at all: it is not visible in any aggregate. Across those five thousand consecutive places, 21% to 37% of domains have no rule for * at all, against a corpus average of 3.9%. They are machine-generated domains sitting at adjacent ranks — 003bingo.com, 006bingo.com, 01d98y.com — and they all serve the same file: 1,248 bytes in which every single line is a comment.

It is the Cloudflare content-signals preamble — the paragraphs of prose declaring that ai-train=no is a reservation of rights under Article 4 of the EU copyright directive — with the rule block that would have enforced any of it absent. There are 21,359 of these files in the corpus, 1,248 bytes each, 75.8% of every file that parses to zero rules. A legal notice, addressed to a machine, in a format that has no way to read it.

02 — Who gets blocked with whom

The blocklists travel in packs

Take the 103,045 sites that block at least one AI crawler. If a site blocks the crawler on the left, how often does it also block the one along the top?

An independent decision about each crawler would produce a pale, patchy grid. What is actually there is a solid block in the top-left corner: eight crawlers blocked together between 79% and 99% of the time.

Everything below that block is dark and everything to the right of it is pale, and the asymmetry is the finding. The eight are a floor almost every blocklist starts from; the rest are names a minority adds on top. Sites that block anthropic-ai also block GPTBot 93% of the time — sites that block GPTBot return the favour 19% of the time. The one crawler that sits outside the pattern entirely is PetalBot, whose blockers overlap the eight only 31–55% of the time: Huawei's crawler is refused by a largely different population.

03 — Where the lists come from

Half the web's AI policy is one file

Collapse each site's blocklist to the exact set of crawlers it refuses, and count how many sites share each set. The distribution is not a long tail — it is one spike.

The top row is 51,936 sites — 50.4% of every AI-blocking site in the top million — carrying byte-for-byte the same list of eight names. It is not a coincidence and it is not a consensus. Of those files, 99.6% contain the marker # BEGIN Cloudflare Managed content, and 54,542 files in the corpus carry that block in total. It is a switch in a dashboard.

The single most consequential author of robots.txt on the web today is not a site owner. It is a default.

The same file explains the matrix above: the eight-crawler clique is Cloudflare's list, and the tight second cluster is what people write when they write it themselves. It also explains an oddity in the report — CloudflareBrowserRenderingCrawler appears as one of the most-addressed user-agents on the web despite almost nobody having heard of it, because the managed block names it alongside the eight.

Read the other rows and the shape of hand-authored policy appears in the negative: single-name lists (4,904 sites block only PetalBot), then a scatter of long, idiosyncratic lists — the 4 rows here naming sixteen crawlers or more are sites maintaining a blocklist by hand, and there are 30,051 sites outside these 14 rows spread across thousands of distinct sets.