Support me on Patreon to write more tutorials like this!
I keep several crawler fleets writing into one Elasticsearch node. On 2026-10-26 I reviewed it and the numbers were bad: 96.5% mean host CPU over 24 hours, average write and search queues of 474 and 53, and peaks of 2,667 and 154.
My instinct was to look at the collectors, since they're the thing I'm always tuning. The collectors weren't the biggest load. This post is about the three things that were, how I cut each one, and the honest ending, which is that the write queue was still deep afterwards.
TL;DR
- Elasticsearch counts work, not intent. A queue of 1,400 pending writes with zero rejections is latency, not an alarm, and it tells you nothing about who's causing it.
- Three loads I wasn't watching: detached background rewrites, a maintenance pass, and idle workers polling the node for work that wasn't there.
- Two of the fixes cut their own counters by 95% or more (98.6% and 95.2%). The third roughly halved refill cycles. None of them cleared the write queue.
- "Requested", "offered" and "completed" are different numbers, and only one of them is work.
The baseline
The review's other numbers narrowed it down. In a 99-second sample, my Google Business scan tasks consumed 85.9% of shard query time. Those are the task documents the place collectors pull work from. The queue lives in Elasticsearch itself, with a Redis cache in front, and refilling that cache is a search. Separately, the job-postings geo enrichment had 58 detached update-by-query tasks running.
So there were at least two suspects before I touched anything: the collectors' own queue refills, and the pile of detached rewrites.
Load 1: a rewrite nobody was watching
The geo enrichment was launching update-by-query tasks detached, fire and forget. Fifty-eight of them were alive at once.
I replaced them with bounded batches, and let the old tasks drain before the new bounded writes began. Then I measured. A 121-second sample before the new image measured 963.8 job-index actions a second. A 275.6-second sample after the old tasks had drained measured 13.49 a second, with the posting document count unchanged. That's a 98.6% drop (derived from the two figures).
That is an observation, not a benchmark; it's two windows on a cluster with other traffic. The posting document count stayed the same, so these were updates to existing postings, not new ones. They are counts of processed postings, not new jobs or new coordinates.
Load 2: a maintenance pass with no cap
An archive maintenance pass rewrites a large batch of documents, and it competes with everything else for the write pool. I capped it at 50 document updates a second. At the first check it had confirmed 33,668 child updates, and its three slices each ran at 16.666666 updates a second, for a total of 50. Fifty thousand updates at 50 a second needs 1,000 seconds, so I gave the supervising job 1,800 seconds.
The detail I like: the pass has a persisted Elasticsearch task ID that survives a Pod deadline. Capping the rate didn't mean redoing the work.
Load 3: workers asking whether there was work
Two repair consumers had nothing to complete. Their loop was to ask Elasticsearch for more IDs, get an empty answer, and ask again. In a 02:15 to 02:22 UTC sample they made about 3.95 empty refill cycles a second between them and completed no actions.
The fix wasn't to stop them asking. It was a shared lease: whichever worker refills holds a queue-scoped lease, and the others skip. I extended it from 3 seconds to 10 on the farseed and main hydration lanes first, then on six more, and left batch sizes, images and worker counts alone.
A third consumer, rated-zero, started at about two empty refill cycles a second and zero successful actions. After I turned the shared lease on for it, the settled numbers were: successful empty refills went from 1.99995 a second to 0.09630, a 95.2% reduction. Lease acquisitions were 0.09630 a second and held-lease skips were 1.90370 a second. Add those two and you get 2.0, the same as before. The workers still tried twice a second. They just stopped bothering Elasticsearch about it.
On the farseed lane, refill cycles went from 0.322 a second to 0.153 and requested IDs from 165.5 a second to 95.1. Completions rose in that window (12.5 to 35.1 a second), but the notes say other traffic changed too, so I won't credit the lease.
Three numbers that look alike
The scan-task lanes taught me to be careful with these:
Requested IDs are what workers asked the queue for: at baseline the farseed lane requested 248.6 a second. Offered IDs are what the queue handed back. Completed actions are what the workers finished, and in the first baseline farseed completed about 16 a second.
Requested is about 15 times completed there (derived). Cutting requests and watching completions hold is a good sign. Cutting requests and calling it a throughput win would be a mistake, and the notes I wrote at the time say not to equate processing actions with distinct businesses either.
The honest ending
After the lease change was on all eight collection lanes, the 03:00 UTC five-minute sample showed 83.8% CPU, a search queue of 0 and a write queue of 1,571.78. Before the canary it had been 95.1%, 12.1 and 1,421.4.
CPU and the search queue improved. The write queue is higher. The note I wrote that day says it plainly: the write backlog has not cleared.
A later canary for the rated-zero consumer was followed by 85.0% CPU and a write queue of 1,377.9, against 74.7% and 1,732.3 immediately before. Those are fluctuating cluster windows with other workloads running. I can't tell you how much of that movement came from the canary. I won't try.
The same lesson from another job
The same week, a different job showed me what a deep write queue does. A place census ran at 0.2 to 0.3 tiles a second against a 1 per second cap, even though the upstream answered in a 2.3-second median. The time went into five small Elasticsearch calls per tile, queued behind everyone else's writes. The shared write pool showed 16 active threads and between 300 and 2,100 queued, with zero rejections. My own share was about 12 of 700 to 900 documents a second.
Raising concurrency from 2 to 8 made it worse. In 2.5 minutes, 17 tile-state writes failed against 3 done, because they hit their 10-second deadline and a failed completion redid the upstream search. The fix was fewer round trips per tile, a longer deadline on idempotent state writes, and treating a deep queue with zero rejections as a latency problem, not an overload signal.
What I'd say about the node now
I have no controlled before-and-after. Every comparison above is an observation window on a cluster with other traffic, and my own notes call them that. What I have is a list of three loads I'd never had on a dashboard, two counters that dropped by 95% or more and one that roughly halved, and a write queue that sat between about 1,400 and 1,700 deep in every window I recorded on 2026-10-27.
That is where my numbers stop. I wrote this on 2026-10-30 without re-reading the node since 2026-10-27, so I can't say what the queue looks like today. A queue that deep with zero rejections is a node that is slow, not a node that is failing, and I don't yet know which way it went.
If I were doing this on any shared datastore again, I'd ask who else is on the node and what each of them does per second before tuning the workload I can see. All three loads here were mine. I'd just never counted them.