What the web's busiest sites refuse to let you crawl

01 — Coverage

Asking 1,000,000 sites a simple question

A request for /robots.txt is the most basic thing a crawler does. Only 60.5% of the top 1,000,000 domains answered it with an actual robots file.

The rest split into instructive failure modes. Some domains in a popularity ranking aren't websites at all — CDN edges, nameserver infrastructure, and tracking hosts inflate the DNS failures. Others actively refuse: 74,194 returned an HTTP error, most of them 403 from bot-protection layers that block the very file designed to tell bots what to do. And 47,958 served an HTML page with a 200 status — a soft 404 that a naive parser would happily read as a robots file with zero rules.

02 — Default posture

Most of the web is open, conditionally

For the generic User-agent: * group — the rules that apply to a crawler nobody has heard of — the dominant answer is yes, but not there.

Only 48,681 sites (8.0%) slam the door on unknown crawlers entirely. The typical file is a scalpel, not a wall: a median of 8 rules across 1 user-agent group, carving out the parts of a site that are expensive to serve or worthless to index.

03 — The AI question

The new names on the blocklist

The most significant change to robots.txt in twenty years is not a new directive. It's a new class of user-agent — and site owners are naming them explicitly.

154,160 sites (25.5% of those serving a robots file) name at least one AI or LLM-training crawler by name. Of those, the overwhelming majority are not writing rules to welcome them: 103,045 sites block at least one outright.

When a site bothers to mention an AI crawler at all, the mention is usually a refusal — a block rate far above anything the traditional search crawlers see.

04 — For contrast

Nobody blocks Googlebot

Run the same measurement against the crawlers that have been around for decades and the picture inverts.

Search crawlers get named often and blocked rarely — being in the index is the point. The interesting middle ground is the SEO-tooling crawlers: AhrefsBot, SemrushBot, MJ12bot and friends provide no traffic in return for the bandwidth they consume, and their block rates look much closer to the AI crawlers' than to Google's.

The archival crawlers sit at the same end. ia_archiver is blocked by 76.1% of the sites that name it — though that number should be read with care: ia_archiver was Alexa's crawler, retired years ago, so much of what we're seeing is fossilised rules nobody has revisited. robots.txt accumulates; it rarely gets pruned.

05 — Rank effects

How far down does the policy go?

Split the corpus by popularity rank. The obvious hypothesis — that elaborate robots.txt policy is a luxury of large sites — is only half right.

File complexity behaves exactly as expected. The top band averages 4,232 bytes against 1,318 at the bottom, a 3.2× spread, and the median file in the tail carries just 8 rules across 1 user-agent group. Sophistication tracks investment, and investment tracks traffic.

AI blocking does not follow that pattern. It runs from 23.7% in the top band to 16.0% at the bottom — a decline of only 1.5× across a thousandfold range of popularity. A site nobody has heard of is nearly as likely to name and refuse an AI crawler as one in the top thousand.

Blocking AI crawlers is the least elitist policy on this page. It reaches essentially the whole web, not just the part of it with a traffic team.

That reach is the finding. Almost every other robots.txt behaviour is a proxy for engineering effort. AI blocking barely is, which suggests it mostly isn't hand-written — it arrives pre-installed, in CMS defaults, WordPress and Shopify plugins, and one-click toggles from CDN providers. A small site inherits the same blocklist a large one deliberates over. The policy diffused across the web faster than the expertise to write it.

Blanket Disallow: / traces a different curve entirely — a U. It is 7.2% at the top, bottoms out at 3.0% around 1k–10k, then climbs back to 10.2% in the deepest band. The two ends are doing opposite things for opposite reasons. Large sites wall off unknown crawlers deliberately, because they have infrastructure to protect. The deep tail is thick with parked domains, staging servers, expired projects and "coming soon" pages — sites that block everything not as policy but as a side effect of never having been launched. The healthy, ordinary web sits in the trough between them.

06 — What gets blocked

The anatomy of a disallow

Strip out the blanket Disallow: / and a remarkably consistent vocabulary emerges. The web blocks the same handful of things everywhere.

Three motives cover nearly all of it. Infinite spaces — search results, filters, calendars, sort parameters — where a crawler can generate URLs forever and index nothing of value. Private surfaces — admin panels, carts, checkouts, account pages — that leak nothing useful and sometimes leak something sensitive. And machinery — print views, build assets, tracking redirects — that was never meant to be a page.

One entry gives the game away. /wp-admin/ appears in 76,453 files, 12.6% of the whole corpus and the single most disallowed path on the web. That is not a decision anyone made; it is WordPress shipping a default. A large share of the web's robots.txt rules were written once, by a CMS vendor, and inherited wholesale — the same mechanism that put AI blocklists on small sites in the section above.

07 — Who gets addressed

CCBot is the most-addressed crawler on the web

Rank every user-agent token by how many sites name it, set aside the universal *, and the result is the clearest single statement this dataset makes.

The 8 most-named crawlers on the web are, without exception, AI crawlers. CCBot leads at 119,287 sites — 1.7× as many as name Googlebot, which sits at 10th. In under three years, AI crawlers have accumulated more site-specific robots.txt rules than search engines did in twenty-five.

The web has spent a quarter century writing rules for crawlers that send traffic. It is now writing far more rules, far faster, for crawlers that don't.

A caveat on reading this chart: naming a crawler is not the same as blocking it, and the tokens here are counted exactly as written. A site that writes User-agent: Applebot has addressed Apple's search crawler, not Applebot-Extended, its separate AI-training opt-out — they are different products and this analysis keeps them apart. Sites frequently conflate the two, which is its own kind of finding.

08 — Hygiene

Structure, size, and sloppiness

robots.txt has no validator, no build step, and no error message. It shows.

Path syntax tells a similar story. Wildcards (*) and end-anchors ($) were vendor extensions for two decades and only became standard in RFC 9309 in 2022 — yet 34.9% of files use a wildcard and 8.5% use an anchor. A crawler implementing the literal 1994 draft would read those patterns as literal characters and match nothing, quietly ignoring rules the site owner believes are in force. Allow, likewise absent from the original draft, now appears in 57.8% of files.

Non-standard directives are common enough to be worth naming. Crawl-delay is understood by Bing and Yandex but ignored by Google; Host and Clean-param are Yandex extensions; Noindex in robots.txt has never worked and was formally killed by Google in 2019 — yet it persists.

Directives no major crawler implements, seen in the wild: content-signal llm-policy llms display width background height padding margin https.

Crawl-delay values cluster on round numbers: 10s (37,167), 1s (16,104), 5s (11,519), 30s (6,436), 2s (5,503), 20s (4,150).

09 — Extremes

The outliers

A few sites treat robots.txt as a configuration file of record.

And the sites naming the most AI crawlers individually — each of these maintains an explicit, hand-curated blocklist rather than a blanket rule: