Your Crawler Won't Crash. It'll Wedge.

A crawler that crashes is a good day. You get a stack trace, a restart, an alert, a line in the log that says what happened.

The bad days are the ones where nothing crashes. Pods stay Running. CPU looks normal. Liveness probes pass. Throughput drops to something just plausible enough that nobody notices for a week. The system has stopped doing its job and it is telling you, in every signal you have, that it is fine.

Here are five of those from the last couple of months on a distributed crawler fleet — three regions, a shared browser pool, an adaptive rate controller. They have almost nothing in common at the code level. One is a channel deadlock, one is a circuit breaker, one is leaked browser tabs. What they share is the shape: the failure was invisible, and the instrument that would have shown it was available the whole time.

TL;DR

• A rate controller wedged seven times in eight days. It was behaving exactly as designed on every one of them. The bug was that the correct branch had no log line, so "working" and "hung" were indistinguishable.
• Every restart "fix" for that was a placebo. The code's own comment said so, a month before anyone read it.
• A liveness probe that could not fail by construction — the heartbeat fired unconditionally on one path and was never called at all on the other.
• A 30-minute hang blamed on proxies and transports through three failed fixes. It was a plain Go channel deadlock. One goroutine dump found it in seconds.
• Around 30 leaked browser tabs silently take a rendering backend out of service. Closing the last tab bricks it unrecoverably.
• The tell that ties four of them together: dead silence is a signal. A system in trouble usually still makes noise.

1. The controller that was working perfectly

The adaptive rate controller decides how fast to crawl. Every 60 seconds it reads the last five minutes of outcome buckets from Redis and decides whether to step the interval up, down, or hold.

Starting on 08-01 it began wedging: interval pinned at its 120-second ceiling, never coming back down, throughput stuck at the floor. It happened seven times in eight days. Each time somebody restarted the controller pod and each time it eventually recovered, which everyone read as "the restart fixed it."

On the eighth occurrence I took a live goroutine dump off the wedged pod before touching anything — :9100/debug/pprof/goroutine?debug=2 through a kubectl port-forward. No code change was needed to get it. It had simply never been tried in any of the seven prior occurrences.

The dump showed goroutine 166 sitting in a completely ordinary select on the ticker channel. Not blocked in Redis. Not deadlocked. Ticking every 60 seconds exactly as designed, the entire time.

The real explanation was in decideInterval(), which has two paths:

• The normal ratio-based path, taken when total >= minSamples, which always logs its decision.
• A low-sample recovery ratchet for total < minSamples, which steps the interval down only if blocked <= recoveryMaxBlocked, and otherwise returns the current value unchanged.

controlOnce() only logged the first branch. The second one returned silently.

Now count the samples. A region pinned at the 120-second ceiling with two replicas produces roughly five samples per five-minute window — against a minSamples of ten. It is structurally impossible for that configuration to collect enough samples to reach the logging path. And when a couple of those few samples come back blocked, the ratchet can't fire either. Both gates fail on every single tick, forever. It is not a race or a fluke; it is a deterministic, self-reinforcing steady state.

So the process went dead silent while doing precisely what the code told it to do.

The part that still stings: the trap was already documented in the code's own comment above the function, added by an earlier session — "restarting the controller doesn't help, it just re-reads the same frozen value and falls into the same trap." Seven rounds of restarts, and the answer was sitting three lines above the bug.

The fix was a logging statement. decideInterval's return value and all its control flow are untouched. The branch now emits low-sample hold (X/Y blocked, need N samples, recoveryMaxBlocked=R) on every tick, so the state is visible instead of silent.

I want to be clear about what I did not do, because it was tempting. The obvious "fix" is to loosen the ratchet gate so it can drop the interval more easily. But blocked was running at nearly 100% of total in every observed window — that pressure was real. Loosening the gate would have meant ratcheting down regardless of genuine backend pressure, making the underlying contention worse. The system's behaviour was right. Only its silence was wrong. Fixing the observability and leaving the policy alone was the whole change.

2. The leak underneath it, and the mitigation that never shipped

The pressure that kept the controller pinned was itself a bug, one layer down.

The browser-automation engine leaks pool slots. When a browser crashes mid-solve — newPage: Target page, context or browser has been closed — its slot doesn't reliably return to the available set. Usable capacity erodes toward zero over a pod's lifetime, and requests start failing with Browser pool saturated.

Those failures were classified as a capacity outcome, which the rate controller counts identically to a real Cloudflare block. Defensible in principle: a degraded pool genuinely is degraded. Misleading in practice: the controller throttled itself hard against what it read as aggressive bot defense, when the actual problem was that we had leaked our own capacity away.

That is how you get a crawler that slows itself to a crawl in response to a bug in itself.

There was an earlier round of this. On 07-30 the same ratchet required blocked == 0 exactly before it would step down, so a single capacity hit in a window froze the interval permanently. That got fixed with a tunable. On 07-31 the leak itself was diagnosed and the mitigation written up: a recycler CronJob doing a rolling restart every six hours, documented in the design doc as added.

When I checked the live fleet, the CronJob did not exist in any region. Either it was never applied or it was applied and reverted. Meanwhile the region it was supposed to protect had been sitting at a 100% blocked ratio for three and a half hours straight.

The code fix was live. The operational mitigation existed only in a document. That gap is worth more attention than either bug — a doc that says "added" is not a deployment, and nothing in the system was checking.

The diagnostic that separates the two cases, for next time: don't trust the blocked ratio to mean real pressure. Hit each engine pod's own /stats endpoint and look at {browsers, available, busy, restarts}. Zero solver activity plus most pods reporting available: 0-1 out of 2 is the leak, not demand.

3. The health check that could not fail

Same controller, different way to die.

When the Redis pod restarts it comes back with a new IP. The controller kept dialing the old one, got connection refused, and its circuit breaker latched open — permanently, because it never re-resolved DNS. Every tick after that logged redis circuit breaker open. It could no longer read the block buckets or write the interval keys.

Here is why this survived so long. The crawlers reconnect to the new Redis IP without trouble and keep running. With the interval keys missing they fall back to their configured 30-second floor. So throughput looks entirely normal from the outside. The only casualty is the adaptive controller — the component whose entire job is to not be running at a fixed rate.

The obvious response is a liveness probe. There was one. It pointed at /metrics, which answers whether the HTTP server is up, not whether the worker is doing anything.

And when I looked at the heartbeat that /livez was supposed to consult, it turned out heartbeat.Beat() was called unconditionally, regardless of Redis errors in one worker, and never called at all in the other. So no liveness probe could have caught this, or any other cause of a stuck breaker. The health check reported healthy by construction. It was decoration.

Both were fixed and the probe repointed at /livez fleet-wide.

4. Three wrong fixes and one goroutine dump

A market-enumeration job hung for exactly 30 minutes, every run, hard-killed at the Kubernetes activeDeadlineSeconds.

Three fixes were attempted, all aimed at the network:

1. A soft deadline with context.WithTimeout.
2. Context-aware semaphore acquisition.
3. Racing every search call against its own time.After(60s).

All three failed identically. The process stayed completely silent — no logs at all — for the full 30 minutes each time.

In hindsight that silence was the entire diagnosis. A real network hang still makes noise: retries, timeouts firing, errors. Ten to sixteen concurrent goroutines racing 60-second timers should have produced something within a minute. Dead silence past that point meant execution wasn't reaching the timer logic at all.

A live pprof goroutine dump against a hung pod found it immediately, and it had nothing to do with networking:

foundCh was a buffered channel sized len(boxes) * 20 — 320 slots, from an assumption of roughly 20 results per box. It was drained by a for f := range foundCh loop placed after wg.Wait() returned. The dump showed eight goroutines blocked for six-plus minutes on a plain foundCh <- ... send. Their searches had already succeeded; they were holding real data and waiting to hand it off to a channel nobody was draining yet. And wg.Wait() could never return, because the producers it was waiting on were exactly the ones blocked on the channel.

One market — Austin — returned 640 results against a buffer sized for 320. The capacity assumption was simply wrong for real traffic, and being wrong turned a working pipeline into a deadlock.

A sibling function never had this bug, for one structural reason: its collector drains concurrently with dispatch rather than after it.

The fix removed wg.Wait() from the search path entirely. Each market gets one hard timer, and a collector drains an outcomes channel sized exactly one slot per box — not per listing — so it structurally cannot overflow, then moves on regardless of stragglers. All ten markets now finish in two minutes total, tracking 5,707 listings with zero errors, against hard-failing at 30 minutes every time before.

The durable change wasn't the fix. It was wiring net/http/pprof into the shared worker metrics server, so this diagnostic is available on any worker without a code change or a redeploy.

5. Thirty tabs

The rendering backends rot. Not crash — rot.

Roughly thirty leaked, unclosed CDP tabs accumulate and the endpoint starts refusing connections. The pod is Running, the process is alive, and it will not serve you. Separately, several backends were live-pinned reporting 0/0 capacity, which meant they looked idle rather than broken.

The mitigation is a tab reaper on a three-minute cycle, closing only tabs that have been stale for at least one interval. With one hard-won guard: it keeps a floor of two tabs, because testing found that closing a backend's last tab bricks it unrecoverably. The obvious implementation — reap everything stale — is worse than the disease.

What I actually changed

Not much code, in the end. Mostly how I approach the first ten minutes.

Take the goroutine dump first. It is the cheapest, highest-information action available and I keep not reaching for it. It would have settled incidents 1 and 4 in seconds. In incident 1 it had been available through a port-forward for all seven prior occurrences and nobody tried it. Wire pprof into every worker now, before you need it, so the answer is a port-forward away rather than a deploy away.

Treat dead silence as a signal, not an absence of one. A system in trouble usually still makes noise. When output stops entirely, past the point where timers should have fired, suspect a synchronization primitive — a channel, a mutex, a waitgroup — before you suspect the network. Three plausible network-layer theories cost more than one dump.

Log the boring branch. Incident 1 existed entirely because the correct path was silent. If a decision function has a branch that returns "no change," that branch needs a line saying so and why. Silence is indistinguishable from a hang, and you will guess wrong for a week.

Make health checks capable of failing. A probe that hits /metrics, or a heartbeat that fires unconditionally, is worse than no probe — it produces confident green signals about a component that has stopped working. Ask what specific failure your probe would catch. If you can't name one, it isn't a health check.

A restart that "works" is a hypothesis, not a fix. Seven of them in a row should have been the alarm. If you can't say what the restart changed, you haven't found the bug — you've found a state that clears on its own.

"Documented as added" is not deployed. The recycler CronJob existed in a design doc and in nobody's cluster. Check the fleet, not the file.