My Canaries Kept Passing and My Rollouts Kept Failing

I run a worker that refreshes per-creator post statistics for a dataset of social-media creators. It fetches each creator's recent posts, computes aggregates, and writes them to Elasticsearch. A circuit breaker watches the success rate over the last 200 fetch attempts and stops the worker when it falls too low.

Over six days I tried three changes to that worker's reliability or routing. Each passed its first test, and each then failed in production. I can prove the cause of the first failure. The other two I can only describe.

Most of this post is about the first one, because it ended with a test I'd write first next time.

TL;DR

• A canary measures the targets it happens to be handed. If the queue has a stretch of targets that can't succeed, a canary that never meets that stretch says nothing about it.
• A circuit breaker that watches yield can't tell "the source is down" from "the next 200 targets have nothing to give". Mine opened on the second.
• The fix was tested negative-first: feed the worker a long run of known-unavailable targets before any good ones and check the breaker stays closed, then break the network and check it still opens.
• Twice more, a probe on known-good targets looked clearly better and then didn't carry over to sustained work. I don't know why.

The canary that passed

On 2026-10-15 I rolled out a change that skips creators who are provably unavailable (private accounts, accounts with no posts) instead of burning fetch attempts on them. I ran it first as two paired canary pods on the real queue, starting at 02:17:58 UTC.

At the one-hour reading they had 945 persisted writes, 35% and 44.5% rolling yield, closed breakers and no restarts. At about 80 minutes: 1,149 writes, 75.5% and 72.5% yield, still no breaker opening and no restarts.

That is a good canary. I promoted it.

The rollout that failed

Both standing pods opened their breakers at 9.5% yield. By 03:57 UTC they were open, and at 06:03 they were still open with 543 and 539 persisted writes in their whole lifetimes.

The pods were up. When I checked the failed creators independently, they really were unavailable, while creators the worker had already written successfully still returned valid posts. The source was fine. One failed profile matched a private account. The others returned errors with no usable identity, so I couldn't call them permanently gone.

The queue isn't uniform. My reading is that the canary pods drew a stretch of creators that mostly worked and the standing pods drew a stretch that mostly didn't. (The note I wrote afterwards calls the canary a "good-cohort canary", but it doesn't record the cohorts themselves.) Yield depends on two things: whether the source is healthy, and what the queue is feeding it. My breaker measured only the product. When the queue side collapsed, the breaker read it as a broken source and opened.

The preflight I'd just shipped wasn't enough. It skipped provably-unavailable creators, but it can't safely classify errors that come back with no identity or only a bare status. A status-only "not found" page proves nothing: one of my own known-good controls returned exactly that. So the breaker still saw runs of failures it couldn't explain.

An earlier rollout of the same worker had already degraded after its own short canary. The note doesn't record that one's numbers, so I'm treating it as context, not evidence.

The test I should have written first

The fix lets the breaker ask a second question before it opens. When yield falls, the worker fetches up to three creators it wrote successfully in the last 24 hours, through the same source. If those still work, the source is fine and the low yield is the queue's problem, so the breaker stays closed. The check is capped at eight seconds, its proof expires after a minute, and it never touches the real accounting: control results don't raise raw yield, write aggregates, bump the persisted-write counters, or spend a creator's attempts.

The part worth writing down is how I tested it, since my first canary design was what had failed.

Negative first. I built a 200-creator run that put 190 independently confirmed unavailable creators ahead of ten proven-positive ones. Filling the 200-attempt window with nothing but failures is exactly the condition that opened the breaker in production. The run scanned 200, aggregated ten, missed 190 and tombstoned none. Raw yield sat at zero, the breaker stayed closed, and three separate control checks passed. Reading Elasticsearch back, all 190 negatives were unchanged and all ten positives had advanced.

Then the inverse, because a guard that never lets the breaker open is also a bug. I ran a one-shot pod with only its own proxy hosts overridden to an unreachable address. The control check failed, the original breaker opened, and the run finished with 200 scanned and zero aggregates. The guard holds the breaker shut when the queue is bad and lets it open when the network is.

(My first attempt at this test used 220 targets, and the worker's 200-ID validation rejected it before it fetched anything. Annoying, and a small proof that the validation is real.)

The standing gate then ran for a full hour: 57 observations from 06:32:59 to 07:33:45 UTC, 866 additional writes across the two pods, no restarts and no breaker openings. Raw yield dipped to 11.5% and 13% in an unavailable stretch of the queue and recovered without anything pausing. Final yields were 85.5% and 87.5%.

That's one hour. It's a better test than the 80-minute canary, but I'd call it "passed its gate", not "stable".

The same shape, twice more

Four days later I compared egress routes for the same worker, and the same thing happened on a different axis.

I probed three routes against known-good creators: two creators' recent-post windows under six different session contexts, 12 windows per route.

The country-pinned route looked like a clear win. (It also had 13 alive nodes at the time against 718-812 in each default-pool instance, which I'm noting and not interpreting.) So I ran a same-load production A/B: one pod on the default route, one on the new one, same image, same queue, same guards.

The two breakers opened 33 seconds apart. The default pod's opened at 21:23:51 UTC after 6 writes and 109 failed fetches. The new-route pod's opened at 21:24:24 after 7 writes and 139 failed fetches. I rejected the new route early.

The 33-second gap points to a shared cause rather than a route effect, but that's my inference. The note I wrote at the time says only that both source windows stayed unstable, and it doesn't establish a root cause.

The next day I did it again with a sticky-IP route. Source-only probes validated 8 of 10 windows on the sticky route against 1 of 10 on the default. The live canary wrote 23 records, then sat breaker-open at zero rolling yield for over 18 minutes, with eight status-zero fetch failures in its logs. Pods on the default route in the same period recovered intermittently and kept writing. I rejected it and removed it.

I don't know why these probes won and the pods lost. Three explanations fit and I can't separate them: the probes ran on known-good targets and production didn't; short windows are noisy; or the source itself was unstable that day. What I can say is that a probe on friendly targets answered "can this path ever work?", and I read it as "will this path sustain the queue?"

Keep the failed control

In one paired comparison, the old worker's breaker was already open at the first measurement frame, which made that window a failure. The tempting move is to restart the observer until the window looks clean. I kept the original journal and its failed verdict instead. The repair replaced that pod, so the next pair is a new scope, recorded as one, not a reset of the old.

If you restart the observer to erase an open breaker or an inconvenient window, you get the answer you wanted and lose the one the system gave you.

What this did not fix

The guard stopped the false breaker-openings. It didn't make the worker fast.

On 2026-10-19, an accepted 30-minute window of the c4 configuration persisted 719 and 630 records across two pods, a gross rate of 2,717.9 writes per hour. At the same time the index held 11,397,707 candidates, 9,249,415 of them remaining. The six-hour planning window showed net candidate growth of 10,445 per hour, so draining the backlog in 12 days needs about 42,561 writes per hour (that's the planning note's arithmetic, against a 12-day goal). The configured start-rate ceiling for that setup was 7,200 per hour, so even perfect yield couldn't get there.

That's a separate problem, and it's still open.

What I'd do differently

• Don't draw canary traffic only from what's easy. Oversample the failures on purpose, or run a negative-first cohort alongside the real one.
• Test the safety mechanism in both directions: it should stay closed when the queue is bad and open when the network is.
• Decide the gate before the run. A canary that is "stable" after 80 minutes is a result about those 80 minutes.
• Treat a probe's win as a hypothesis about the path, not a forecast for the queue.
• Keep the failed control.