I Blamed Three Things for My Crawler's Refill Rate. The Biggest One Was a Worker I Had Shipped Four Days Earlier

I keep an index of TikTok creator profiles. In early November it held about 14.3 million candidate profiles. 4.4 million had been analysed and 9.9 million were waiting. The analysis side was doing roughly 229 thousand profiles a day, so on paper the backlog was about six weeks of work. In practice it was barely shrinking, because discovery kept adding candidates almost as fast as analysis removed them.

It took me four days to find out why. In that time I gave three explanations and acted on two of them. None came from counting what was actually being written to the index. The biggest cause was on none of my lists, and a query I could have run on the first day would have shown it.

TL;DR

• The refill rate went from about 59 thousand new candidates a day to about 199 thousand. Two things changed that day: a cluster fix that restarted my cron jobs, and a new discovery worker I deployed. I wrote down the first one.
• Counting new profiles by their source field, the worker supplied 76 to 80% of all new profiles every day from 11-08, while everything else was falling, not rising.
• The worker ships its whole day's output in one hour after UTC midnight. A 24-minute check I made at 14:00 UTC could not have seen it. It saw 666 new profiles, none from the worker.
• The drop from 199 to 136 thousand a day that I credited to the feeders is mostly calendar: one review window held three of the worker's bursts, the next held two.
• Scaling the worker to zero cut new profiles from about 130 thousand a day to 27 thousand and made the drain about 2.5 times faster.

Where the numbers stood

Between my reviews of 2026-11-05 and 2026-11-07 the candidate pool grew by 119,909, about 59 thousand a day. Between 11-07 and 11-09 it grew by 397,858, about 199 thousand a day, 3.4 times as fast. Analysis ran at about 229 thousand a day across that second window. The net was that the backlog shrank by only about 30 thousand a day, a drain of roughly 324 days. With no new discovery it would have been about 43.

Two things had changed on 11-07. The cluster's Job controller, which had stopped creating pods for my cron jobs in late October, started working again, so the discovery cron jobs ran again. And that same day I deployed a new discovery worker. My review note for the 11-09 window explained the jump as the discovery pipelines "running at full rate again now that P0 recovered." It does not mention the new worker.

The worker walks my index of resolved videos, fetches a page or two of comments for each, and adds any commenter that isn't already in the index. Before I built it, a canary over 120 videos found 3,774 distinct commenters, 94.6% of them absent from the index. Its first production run took 200 videos and produced 3,496 new profiles in about four minutes. It had a cap of 2,000 videos per UTC day, which it hit in about 25 minutes, so within ten minutes of deploying I raised the cap to 5,000.

What the index said about the jump

I grouped new profiles by day and by source. The "everything else" column is the total minus the worker.

If restarted cron jobs had driven the jump, I would expect the other sources to rise on 11-07. They fell, from about 64 thousand on 11-06 to 38 thousand on 11-08. Part of that fall was a separate source that had been feeding 26 to 41 thousand a day until 11-06 and stopped within a day. The jump was the new worker. I never grouped by source while it was happening, so I explained it with the event I knew about.

Explanation one: the feeders

On 11-09 I scaled the two continuous feeder deployments to zero. My commit message said they "are the source of new creators." Twenty-four minutes later I measured the candidate count: up by 658, about 40 thousand a day. I projected that the backlog would now shrink by 170 to 180 thousand a day and finish in about 54 days.

Two days later the 48-hour figure was different. Candidates had grown by 267,329, about 136 thousand a day, and the backlog was shrinking by 62 thousand a day. I credited the feeders with the fall from 199 to 136 thousand a day.

The index says that was mostly the calendar. Counting new profiles in windows that approximate each review, the first window (about 11-07 12:00 to 11-09 12:00 UTC) holds 388,219, of which 317,675 are the worker's. That is three of its bursts: its first afternoon, then the 11-08 and 11-09 midnights. The second window (11-09 14:00 to 11-11 13:00 UTC) holds 257,322, of which 206,730 are the worker's, two bursts. Both totals are within 4% of the candidate figures I was using. The rest, everything that wasn't the worker, fell from about 35 thousand a day in the first window to about 26 thousand in the second. That is the most the feeders could have been worth, and the end of the separate source above is in there too.

Explanations two and three

My next two explanations came from the same days' notes. The first was that the cron jobs which call the same enqueue functions had resumed when the Job controller recovered. I suspended three of them. The table above says they were not what changed on 11-07, and I never isolated their effect, because I made the last change fourteen minutes later.

The second came from my job-kind breakdown. In the last two hours of one window, the only substantive job type executing was one that resolves the handles mentioned in analysed posts, about 10.7 thousand an hour. I wrote that those jobs "produce ~140 k new candidates/day," 74% of the pre-pause inflow. That took the whole measured inflow and assigned it to the one job type I could see. The note I wrote the next day says the figure "was overestimated."

The breakdown could not have shown me the real source. The comment worker doesn't go through the job queue. It walks the video index with its own cursor, so a table of job kinds never contained it.

The query I should have run first

Every document in the index carries a source field, written by the code path that created it. I grouped the new profiles of the 48 hours to 11-11 by it:

The field had been there all along. I had been reasoning about which workloads I remembered running, and the index could have told me which ones were writing.

Why the 24-minute number lied

The worker's day counter is keyed by UTC date and its cap is on videos per day. At the pace I measured, the cap is reached in about an hour. After that it does nothing until the next UTC midnight. Before 11-11 it didn't even sleep. It re-entered every 60 seconds and logged the same line for the rest of the day.

I counted the worker's new profiles per hour for 11-08 through 11-11. Between 96 and 99% of each day's output landed in the 00:00 UTC hour: 117,887 of 121,031, then 110,899 of 112,166, then 99,850 of 104,009, then 101,364 of 102,721. The rest trickled into the following hour.

My 24-minute window ran from 13:51 to 14:15 UTC on 11-09, about thirteen hours after that day's burst. The index holds 666 new profiles for those 24 minutes: 476 from search, 189 from discover, one from an outside feed, none from the worker. I measured the idle half of the duty cycle and read it as a rate.

The same shape explains the 48-hour attribution. The worker's output for 11-10 and 11-11 adds to 206,730 (104,009 plus 102,721). That is within 13 profiles of the 206,743 in my 48-hour source table. Those two days include 1,357 profiles that landed in the 03:00 UTC hour of 11-11, which I assume came from a short experiment on its scan order before the cut.

What happened when I turned it off

On 11-11 at about 13:55 UTC I scaled the comment worker to zero.

• New profiles went from 130,476 on 11-10 to 27,103 on 11-12, down 79%. All of the 27,103 came from search and discover.
• By my candidate gauge, analysis held: about 199 thousand a day before, 180,343 in the 24 hours after.
• The backlog's rate of shrinking went from 59 thousand a day to 149,028, about 2.5 times faster.

At the new rate my note projected about 65 days to finish the 9.73 million backlog, against about 166 before. Another note, using a different window, said 54. That is one day of data.

I stopped the best discovery source I have on purpose. It had added 524,424 profiles by then, and it's the thing I would turn back on first. I wrote down the revert and a warning that its cursor needs copying before it resumes.

What I don't know

My index counts and my candidate gauge don't match exactly. They differ by 2 to 4% in the two review windows, so I use the index for who produced a profile and the gauge for how fast the backlog drains.

I can't split the roughly 9 thousand a day by which "everything else" fell between the two windows. The feeders, the cron jobs and a separate source ending all sit in it.

I didn't read the worker's pod logs. The index told the same story hour by hour.

While applying the cut I found a separate fault. An earlier patch had added a second suspend key to one manifest, so every kustomize build of the live overlay was failing on a duplicate key. The apply before mine would have failed silently.

The rule I took from it

When a rate surprises you, count the new records by the field that says what created them before you touch anything. A breakdown of which jobs are running only describes the workloads you already know about, and it will happily assign the whole effect to the one you can see.

A second rule is about time. A worker with a daily budget has a duty cycle, and a measurement shorter than a day measures wherever in the cycle you happened to land. The same goes for comparing two windows: count how many cycles each one contains before reading a difference as an effect. I should have asked for the worker's reset time before I believed 24 minutes.

I also credited the feeders with a drop that was mostly a calendar effect. A partial answer made it easy to stop looking.

Update 2026-11-13: this post originally said the duty-cycle explanation was a reconstruction I had not checked, and that the feeders were worth about 63 thousand a day. I then queried the index per hour and per day by source. The text above is revised to the measured numbers. The feeder figure and the "each explanation was partly true" framing did not survive.