The Retry I Added Converted 6 of 2,840 Empty Pages

I run a worker that refreshes post statistics for a large set of creators on a social platform. For each creator it fetches a page of recent posts, computes aggregates and writes them to Elasticsearch. An empty page is a common answer, and it is ambiguous: the creator may have nothing to show, or the platform may be throttling the identity I fetched with.

My policy for an empty page was to rotate the client identity and ask again straight away. It looked like the most defensible line in the worker. A retry costs one request and might rescue a creator. This post is about the day I put a counter on it.

TL;DR

• In a one-hour window the immediate retry turned 6 empty pages into non-empty ones out of about 2,840. That is 0.2%. It used about 27% of the page-level requests.
• I kept the identity rotation and dropped the immediate re-ask. Page calls fell about 15%, and the write rate rose somewhere between 16% and 24% depending on the window. My windows were sequential, so don't read that as a clean causal number.
• The same change made a failure ratio look worse: the soft-blocked share of raw attempts went from roughly 58 to 62% up to about 69%, while output went up.
• The dead-letter pool kept growing at the same pace afterwards, about 10 thousand creators a day.

What the retry was supposed to do

An empty answer from the platform often seems to mean one identity has been throttled, and rotating to a fresh identity usually restores it. My earlier notes record that turning rotation off made the soft-block share worse, so rotation earns its place and I kept it.

The retry is a separate decision. It says: after rotating, ask for the same page again immediately, instead of leaving that creator for the queue's normal backoff. I don't have a date for when I added it, and I have no record of anyone measuring it. I could first see its hit rate on 2026-11-03, when a counter that splits page outcomes into first attempts and retries was live.

The hour I looked

Here is one hour of that counter, in pages:

Every retry follows an empty first attempt, and the retry row adds up to 2,836 outcomes. My notes round that to about 2,840. Six of them came back with data, which is 0.2%. The other 2,804 were empty again.

The cost side comes from a different counter. In the previous hour the worker made 10,475 page-level calls, of which 7,630 were first attempts and 2,845 were retries. So 27% of the page requests went to an operation that worked about twice in a thousand, and each of those retries also meant a second set of transport attempts.

My notes read the result carefully. The useful part of the rotation is probably the fresh identity for later calls, not the immediate re-ask.

The experiment

I added a flag to skip the retry while still rotating, so the creator goes back to the queue and the worker's own backoff retries it later. It went live on 2026-11-03 at about 15:20 UTC. Here is what I measured, against the hour before:

"Written" means aggregate documents written to Elasticsearch. The same upstream budget produced more of them with fewer requests.

I took a second look at 48 hours. Output from the app-endpoint source was 57.1 written a minute, against 49.0 over the 72 hours before that and 48.5 in the stretch right before the change. The 48-hour window is a blend, because 4,167 retry calls still happened before the flag took effect. Of those, 15 returned data, which is 0.36%, and no retry has run since.

Why I don't trust the exact percentages

My windows are not concurrent. I compared an hour before with 14 hours and 48 hours after, on pods whose throttle drifts. In an earlier test the same configuration wrote 46.6 a minute in one 10-minute window and 36.0 in the next, so I treat anything under about 25% from a single short window as unresolved. The first post-change reading, 62.8 against 50.7, is 24% and sits right at that line. The longer windows, 59.4 and 57.1, come out about 17% above their baselines, and they point the same way. I trust the direction. I don't trust the exact figure.

I can't run it as a concurrent A/B either, because my pods share one app-level budget. When I added a third pod, two of the three collapsed together. A treatment pod next to a control pod would contaminate each other.

The dashboards disagreed

The change was a good one, and three of my own instruments said otherwise for a day.

First, a ratio moved the wrong way. With retries gone, the soft-blocked share of raw attempts rose from roughly 58 to 62% to about 69%. That wasn't a regression. The retries had been adding failed attempts to the denominator, and with them removed only first attempts were counted. I wrote the new expectation into the docs so nobody would mistake the higher number for damage.

Second, two counters I added for this experiment, retries skipped and attempts by phase, were incremented in code but never registered with the metrics registry. They never appeared in a scrape. I added them to the registration list with a regression test that gathers the registry. After the next deploy the skipped counter was visible, at 287.

Third, my yield alert. Removing retries moved the baseline of the app-endpoint source from about 0.38 to 0.32, and the warning at 0.30 sat pending for part of a day while throughput was up. I lowered it to 0.25. My first attempt to ship that inside an image-pin commit lost the file change, so the manifest changed and the live alert rule didn't. This time I checked the worktree, the commit, the live mounted ConfigMap and the rule's query.

An alert from the same week had the same kind of flaw. I had written one that compared pod age against a metric this cluster doesn't expose, so its expression was empty and could never fire. The commit that fixed it says exactly that. In each of these cases an instrument was telling me something other than what I'd assumed.

What it didn't fix

The dead-letter pool, creators that exhausted their attempt budget, did not shrink. It was growing at about 10.5 thousand a day, then 9.7 thousand, and about 10.8 thousand after the change. Once I added a counter by class, the whole pool turned out to be one kind: creators whose pages keep resolving to the ambiguous empty answer, not transport errors.

That tells me removing the immediate retry didn't make the dead pool worse. It doesn't tell me whether any individual creator who would have been rescued by a retry was recovered later by the queue's backoff. I don't have that number.

I also don't know which half of rotate-then-return does the work. I kept rotation because turning it off hurt. I removed the retry because it paid back about two pages in a thousand. Whether a longer delay before the next attempt would help is a test I haven't run.

The rule I took from it

Every retry should have a hit-rate counter before it ships: how many first attempts failed, how many retries succeeded, how many requests the retries cost. The counter is a few lines. Mine came after the retry, and I can't say by how long. In the hour I measured, about a quarter of my page requests went to asking again for something that mostly wasn't coming.