Reports
4 published · newest first
A note works something out in an afternoon. A report goes and measures the thing first — a corpus, a crawl, a pile of files — and then works it out. Each one is a single page: the numbers, the charts, and a method note saying exactly what was counted and what the count is worth.
Ref
Report
Subject
Published
R·04
The whole corpus, in a search box
Six gigabytes of robots.txt compressed to its vocabulary and
made searchable in the browser — every crawler name the web knows, and the files
they live in. No server to ask.
crawling, archives, the web
11 Aug 2026
R·03
What robots.txt actually says
The same corpus read as text rather than policy: 16% of the web
turns out to share forty files, the comments are mostly written by plugins, and
crawlers are named in constellations nobody chose one at a time.
crawling, language, the web
10 Aug 2026
R·02
A million robots.txt files, one pixel each
The same crawl, drawn instead of counted: every domain as a single
pixel, and the handful of blocklists the whole web turns out to be copying.
crawling, visualisation, the web
10 Aug 2026
R·01
The state of robots.txt
One
GET to a million domains. What the busiest sites
on the web allow, what they block, and — increasingly — who they refuse by name.
crawling, machine reading, the web
7 Aug 2026