Support me on Patreon to write more tutorials like this!
Most "I'll just scrape it" projects don't die on parsing HTML. They die when the page turns out to sit behind Cloudflare and needs an entirely different approach than the one you built. So I wanted a real number: how much of the web actually runs managed bot defense?
I scanned the Tranco top 1,000,000 domains to find out. The answer I published is 53.5% of reachable sites, from 998,497 domains probed, 818,614 of them reachable.
But that number is the fifth answer, not the first. The first four were wrong — not by rounding, but by enough to change the conclusion. Each was wrong for a different reason, and each reason is a trap that applies to anyone measuring anything about live websites at scale.
This post is about the four corrections. If you only take one thing from it, make it the last section: a headless browser is almost certainly not the tool you need.
TL;DR
- I read only the first 240 bytes of each response body, so CAPTCHA detection was effectively dead.
g-recaptchasits at byte 7,167 of Fiverr's 403 page. - I counted any Akamai CDN site as bot-managed, because I matched on edge headers that ship on every Akamai Ion property.
- I counted a 201KB HTTP 200 as a successful fetch. It was a block page.
- I scored sites that silently fingerprint you while serving real content as completely unprotected. TikTok was one of them.
- After all four fixes: 53.5% of reachable sites carry a managed wall, but only 9.8% actually challenged my request. Most walls are asleep.
The setup
The scanner is a Go worker. One homepage GET per domain against https://<domain>/, with a www. fallback when the apex returns nothing at all. It walks a cheapest-first ladder and stops at the first success:
- A plain HTTP client
- The same request with a Chrome-impersonated TLS fingerprint (via
imroc/req/v3) - A proxied retry through a rotating egress pool
There is no clever escalation heuristic. It runs the cheap thing, and if that passes, it never runs the expensive thing. That single decision is why the whole census fits on a handful of preemptible nodes.
Everything about it is built to be killed mid-run: batches of 1,000, a cursor in Redis, and an Elasticsearch _id of domain|run_id so re-doing an in-flight batch is idempotent. It ran at concurrency 150 per pod, nine pods, on a five-node cluster of 3.5 vCPU boxes. The pods reserve 250m CPU and 512Mi of memory, and actually use about 20m and 140Mi — it's I/O-bound, not CPU-bound, which is the whole reason this is cheap.
Detection is signature matching, not machine learning: 26 vendor signatures over response headers, Set-Cookie names (never values), Server strings, and body markers.
That's the easy part. The hard part is that all four of my wrong answers came from the classifier, not the crawler. The infrastructure worked. My measurement was broken.
Correction 1: I was reading 240 bytes of each page
The probe layer stored a BodyPreview for marker matching. It was truncated to 240 characters, and that preview was the only body the classifier ever saw.
Two hundred forty characters is roughly the opening <head> tag. Every body-based signature — every CAPTCHA marker, every embedded SDK reference — was being matched against almost nothing. The scan wasn't finding few CAPTCHAs. It was structurally incapable of finding any.
The proof is easy to state:
g-recaptchaappears at byte 7,167 of Fiverr's 403 page.- DataDome's
captcha-deliverymarker appears around byte 506 on idealista.
Both are past a 240-byte cutoff. Every CAPTCHA number I produced before this was closer to noise than to a measurement.
Raising the preview to 64KB fixed it. The lesson is dull but expensive: if a signal lives in the body, verify your pipeline actually retains enough body to contain it. I had unit tests for the matcher. They passed, because they fed it whole documents. Nothing tested the thing that fed the matcher in production.
Correction 2: A CDN is not a bot wall
Akamai's edge products set headers like x-akamai-transformed, akamai-grn, and x-akamai-request-id. I matched on those and called the result Akamai Bot Manager.
Those headers ship on any site behind Akamai Ion. Bot Manager is a separate product. So the scan was reporting a bot-management deployment every time it saw a CDN.
I found this from the million-site data itself, which is the part I'd defend. When I sampled what the scan had labeled Bot Manager, the list included meraki.com and iowa.gov — sites that plainly aren't running aggressive bot defense on their homepage. The false positives were obvious enough at scale to notice, and invisible in any single spot-check.
Akamai Bot Manager actually announces itself with _abck or bm_* cookies, or bmak. in the body. Splitting "Akamai (edge)" from "Akamai Bot Manager" moved thousands of sites between buckets. In the final data they sit at 5,185 and 4,810 sites respectively — close in size, entirely different in meaning.
The same discipline cuts the other way. F5 BIG-IP sets bigipserver<pool> and f5_ cookies, and it would be easy to count those too. I deliberately don't: the value encodes a backend pool. That's load-balancer session persistence, not bot defense. Counting infrastructure as defense is how you get a headline number that's 10 points too high.
Correction 3: A 201KB HTTP 200 that was actually a block
This one is my favourite, because everything about it looks like success.
Request microsoft.com from a datacenter IP with curl and you get HTTP 200, about 201KB of HTML, complete with Microsoft's real site chrome. Every signal I was checking said: fetched fine, no wall, move on.
Read the body and it says:
your current User-Agent string appears to be from an automated process
It's a soft block — internally PAGENAME=smarterror.aspx, titled "Your request has been blocked," served as static content from Akamai NetStorage. A real browser from a residential IP gets the actual homepage. curl gets the error page regardless of what User-Agent string you set, because the tell is the TLS fingerprint and the datacenter IP, not the UA header.
The bug this exposed was worse than the one site. The worker wasn't passing block markers into the probe layer at all, so every content-based deny page that returned a 2xx counted as reachable and unprotected. Sites that were politely refusing me in plain English were recorded as open.
The fix: pass the challenge markers through, add "your request has been blocked" to them, and treat a non-contentful hard-deny body as forbidden even on a 200.
The general rule I'd give anyone building this: a status code is not an outcome. If you're measuring reachability, you have to read what came back.
Correction 4: Silent fingerprinting scored as "open"
The classifier only mined body signals from responses that looked like challenges — non-contentful pages, interstitials, error bodies. That seemed reasonable. It quietly encoded an assumption that turned out to be false: that a protected site tells you it's protecting itself.
Plenty don't. They serve you the real page, in full, while an embedded SDK fingerprints the session for later. Under the old logic those sites landed in the cleanest bucket available — reachable, no vendor, no protection.
TikTok made this obvious. It served a 388KB fully-rendered page containing the webmssdk reference eleven times. Its own proprietary detection was all over the response, and my scan classified it as having no protection at all.
The fix was a separate class of signature for always-on SDK references that are safe to match against a fully rendered page, rather than only against challenge bodies. It's a small change in the matcher and a meaningful change in what the census means: "no protection detected" now means something closer to what a reader assumes it means.
The ceiling I can't fix, and why the number is a floor
After four corrections I'd like to tell you the number is correct. It isn't — it's a bound, and being honest about the direction matters more than the decimal.
A passive homepage GET can only see vendors that announce themselves unprompted. DataDome, PerimeterX, Kasada, AWS WAF, and Shape largely don't. They emit their cookie or header under challenge — meaning a scan that never triggers a challenge never sees them. Validating against known-protected sites put my recall at roughly 55%, with essentially no false positives.
So read the headline this way:
- 53.5% protected is a conservative floor. The true figure is higher, and it's higher by an amount I can't measure with this method.
- The difficulty grades are a conservative ceiling. Cross-checking against a real browser engine showed the passive heuristic over-estimates how hard sites are: leboncoin and wayfair grade as very hard passively but are reached by a matched Chrome TLS fingerprint; Kohl's needs an anti-detect browser; only genuine behavioral/VM defenses like Kasada on realtor.com and PerimeterX on therealreal.com actually defeat a production engine.
One more caveat worth stating out loud, because it undercuts my own homepage-only method: probing deeper pages on 50,000 domains showed 20.0% have at least one page strictly harder to reach than their homepage, and 6.0% have a scrapeable homepage but a blocked interior page. Homepage-only measurement systematically flatters the web.
What the corrected data says
From the completed run over 998,497 domains, 818,614 reachable:
| Finding | Sites | Share of reachable |
|---|---|---|
| Carries a managed wall | 437,857 | 53.5% |
| Cloudflare | 368,609 | 45.0% |
| Google reCAPTCHA | 61,238 | 7.5% |
| hCaptcha | 28,249 | 3.5% |
| Unidentified WAF | 17,383 | 2.1% |
| Cloudflare Turnstile | 10,910 | 1.3% |
| Imperva (Incapsula) | 5,249 | 0.6% |
| Akamai (edge only) | 5,185 | 0.6% |
| Akamai Bot Manager | 4,810 | 0.6% |
Cloudflare is 84% of every protected site in the sample. Whatever else this data says, it mostly says "Cloudflare."
But the number I'd actually put on a slide is this one: of the 818,614 reachable sites, only 79,835 — 9.8% — actively challenged my request. The other 358,022 walled sites ran their vendor passively and handed a matched request a clean 200.
Having a wall and using a wall are different things, and the gap between them is enormous. Roughly four out of five sites that look protected simply let a well-formed request through.
The finding that changed how I build crawlers
Everything above is measurement hygiene. This last part is the practical payoff, and it comes from a companion run over about a thousand hand-picked sites — the kind people actually scrape — where I could afford to test each one against a real browser.
Two results, both against my expectations:
67% needed no browser at all. A plain HTTP client or a matched TLS fingerprint reached two-thirds of them.
For the sites that did block a passive request, a real headless browser recovered only 22%. The other 78% blocked the browser too.
That second number is the one worth internalizing. The reflex when you hit a block is to reach for Puppeteer or Playwright, on the theory that the site wants JavaScript executed. But if a browser fails on 78% of hard sites, then whatever is stopping you mostly isn't JavaScript rendering — it's IP reputation, TLS fingerprint, and behavior. Launching a browser adds enormous cost per request and, for four out of five hard sites, changes nothing.
The measured shape of the ladder says the same thing from the other direction. Across the census, direct HTTP handled about 52.9% of probes and Chrome-impersonated TLS another 13.6%; only around 12.7% ever needed a proxied attempt at all.
So the ordering I'd defend for any crawler at scale:
- Plain HTTP, always first.
- A matched TLS fingerprint. This is the single highest-leverage upgrade available and it costs almost nothing.
- Rotating egress, only for what's still failing.
- A browser, last, and only after you've confirmed the block is actually JS-shaped.
Most crawler architectures I see invert this — a browser pool at the center, because the browser is the thing that always works. It doesn't always work. It works 22% of the time on the cases you'd reach for it, and it costs you on every single request that would have succeeded without it.
Data
The full per-domain dataset is published under CC BY 4.0: anti-bot-adoption-index-data. The productized version of this study, with the browsable index and per-vendor breakdowns, lives at Crawlora — same underlying scan, different question.
If you re-run any of this and get a different number, I'd genuinely like to know which of the four traps I'm still in.