Support me on Patreon to write more tutorials like this!

I have a worker that fills in post statistics for TikTok creator profiles. At the start of this story its drainable backlog was about 9.2 million profiles, it ran as two pods with 8 fetchers each, and the platform it reads from had been blocking me for days. I had fixed everything I could think of on my side: the signer, the way an empty account is classified, the circuit breaker, the health probe. Concurrency was the only lever left.
So I wrote a runbook with rules for turning it up and for turning it back down, wrote the rules down before touching anything, and then followed them. The first raise passed every gate condition. By the next morning it had made things worse than the baseline it started from, and it had grown the backlog.
I'm writing it up because the failure wasn't a rule I broke. It was something the rules couldn't see.
TL;DR
- I raised concurrency from 8 to 12 on 2026-11-21 at 22:55 after all five RAISE conditions measured green on a fresh read.
- Fifteen minutes later attempts were up 21.8% and written records up 53%. At 30 minutes the gain had shrunk to +30%.
- By the next morning, over a 145-minute read, four of the six LOWER conditions were tripped: 17.4% of work shed, 11 sustained errors, answered rate 0.080 to 0.095 per pod, the breaker cycling open about every 20 minutes. Attempts per minute had fallen from 213 to 30.4.
- While
-c 12ran, the drainable set went from 9,245,305 to 9,360,616, a regression of 115,311 (derived), because intake kept running while the breaker shed. - Part of the "−30% written" headline is the queue getting harder, not the raise. The rollback is justified by the shed rate, the breaker and the attempts collapse, which the queue's makeup can't explain.
The runbook
The worker runs as 2 pods with 8 fetchers each and a minimum interval of 1,500 ms per fetcher. At the baseline I captured on the afternoon of 2026-11-21 it made 209.3 upstream attempts a minute across the fleet, against a pacing ceiling of 320 a minute per pod. So the pods were running at about a third of what their own pacing allowed, and the average request took 1.336 seconds, which would allow roughly 359 a minute per pod. The fetchers weren't saturated. The open question was whether more of them would buy net drain or just more soft blocks.
The decision rules are in the runbook and I applied them literally:
- RAISE one step only if all of these hold: the answered rate (the breaker's own gauge) at least 0.17, shed share under 3%, no breaker open in the last hour, average latency at most 1.50 s, net drain negative or flat, and the previous raise had been observed for a full day.
- LOWER one step if any of these: answered rate under 0.12 for 30 minutes, soft-block share over 45% on a 1-hour window, latency over 1.80 s, any sustained errors, a breaker open for more than 30 minutes, or shed share over 10%.
- One step per day, never concurrency and the image in the same run.
The first step was 8 to 12 fetchers.
The raise
The gate, re-read immediately before the change, was green: answered rate 0.217, shed 0.0%, no breaker open in the last hour, latency 1.363 s, and no earlier raise still unobserved.
Fifteen minutes after the roll, with the live args confirmed on both pods:
-c 8 (1 h) | -c 12 (15 min) | change | |
|---|---|---|---|
| attempts per minute | 225.6 | 274.7 | +21.8% |
| written per minute | 1.37 | 2.10 | +53% |
| soft-block share | 29.9% | 26.7% | down |
| average latency | 1.363 s | 1.433 s | +5% |
I recorded the result in the runbook. The useful observation in that entry is that more fetchers bought less than proportional work: fetchers rose 50% and attempts only 21.8%, about 44% of the linear expectation. I recorded that the source was already rate-sensitive somewhere above 275 attempts a minute for the fleet, and I recorded that -c 12 was "clearly a win and is being kept".
Thirty minutes in, the gain had already settled to about +30%. I corrected the entry. It was still a gain.
The next morning
At 10:00 the next day I ran the daily check. Four of the six LOWER conditions were tripped, and two were not:
| LOWER condition | threshold | measured | |
|---|---|---|---|
| answered rate, 30 min | under 0.12 | 0.080 and 0.095 per pod | tripped |
| soft-block share, 1 h | over 45% | 28.5% (6 h) | ok |
| average latency, 1 h | over 1.80 s | about 1.66 to 1.77 s | ok |
| sustained errors | over 0 | 11 events | tripped |
| breaker open | over 30 min | cycling open every ~20 min | tripped |
| shed share | over 10% | 17.4% (927 of 5,328) | tripped |
And against the -c 8 numbers it had come from:
-c 8 baseline | -c 12 overnight | |
|---|---|---|
| attempts per minute | 213 | 30.4 |
| gross settles per minute | 32 to 35 | 10.9 |
| written per minute | 1.37 | 0.96 |
| shed share | 0% | 17.4% |
| breaker | closed | cycling |
The raise had ended up worse than the baseline, not just neutral. My reading, which I did not test further, is that the extra four fetchers push the fleet past what the source tolerates at steady state. The answered rate falls under the breaker's floor, the breaker sheds work, and throughput collapses.
I rolled back to 8. Shed went to zero immediately, and by the time I wrote the entry one pod's breaker had already closed.
What a gate can't see
The gate measured the system as it was before the change. A raise changes the system, and the part of it that matters here, a feedback loop between the source's tolerance, my breaker and my shed rate, takes hours to show up. The 15-minute read and the 30-minute read were both taken before it did. The first was a wave and so was the second.
The runbook already said not to raise to 16 on 15-minute numbers. What I hadn't written was the stronger rule: do not judge a raise at all inside one hour, and never quote a sub-hour gain as a level. The +53% went into two commits and a rollback before any long window existed.
The experiment had a cost
While -c 12 ran, the drainable set went from 9,245,305 to 9,360,616. That is a regression of 115,311 profiles (derived from the two figures). Net drain had been negative before the raise, between −25 and −28 per minute, and it was positive for this window, because discovery kept adding profiles while the breaker shed. I'd been measuring the gain in throughput during the experiment. The number that matters is the backlog before and after, and that was larger than any gain the raise produced.
The comparison is partly confounded
I want to be careful with the "−30% written" headline, because the queue is not a constant. In the 2026-11-21 baseline table, written was 2.37 a minute. By the 1-hour window right before the raise, at the same concurrency, it was 1.37. That is a 42% fall (derived) with no change to the worker. The explanation recorded the same evening is that the drain was walking a follower-sorted population into its postless tail: written decelerated from 1.99 a minute over 6 hours to 1.51 over the last hour while the drain rate held.
So some of the gap between 1.37 and 0.96 is the queue getting harder, and I can't separate that from the raise. The rollback doesn't rest on it. It rests on what composition can't explain: shed share going from 0% to 17.4%, the breaker cycling, the answered rate more than halving, and attempts per minute falling from 213 to 30.4.
A measurement trap in the same run
For the first 40 minutes of the check, every Prometheus one-hour figure for this job read zero. That looks identical to a dead worker. Prometheus had been scraping stale pod IPs from an earlier rollout, one pair of addresses where the live pods had a different pair. The up{job=...} series is what exposed it.
The runbook now requires up == 1 before any Prometheus number is trusted, and falls back to reading the pod's own metrics endpoint and taking a delta over 60 to 120 seconds when it isn't. The counters in my rollback record came from there.
What I changed
- The runbook marks rung 1 (
-c 8to-c 12) deprecated, and the next rung (a shorter minimum interval) deprioritized, because it raises a ceiling this evidence says isn't mine to raise. - A raise isn't a raise until 30-minute, 3-hour and 24-hour windows all agree.
- The experiment's cost is measured as backlog before versus after, not as throughput during.
- The manifest carries the rollback record inline, so nobody restores
-c 12without a full-day measurement.
What I don't know
I have one experiment and one night. I don't know where in the 145-minute window the collapse began, or what would have happened at 10 fetchers. The cause is a hypothesis about the source's tolerance, not a tested mechanism. I also don't have a cost figure for any of it. And the 15-minute and 30-minute figures exist only as the numbers I wrote down at the time, not as a time series, so I can't plot the turn.