What Parallel Coding Agents Did to Go's Build Cache

Last week my laptop got down to 357 MiB of free disk. Go's build cache, ~/Library/Caches/go-build, was 202 GB. Running du -sh on it took more than two minutes.

Nothing had gone wrong in the usual sense. No build was broken and no test was failing. I'd just spent a few weeks running several coding agents at once against the same Go repository, each in its own git worktree, and it turned out Go's caches assume something that setup breaks: that there's one checkout, at one path, being built over and over.

This post is about what that assumption costs, how I found it, and the small changes that fixed most of it. None of it is specific to AI agents. Anything that builds one Go module from many directories at once, like CI runners with random workspace paths, will hit the same problems.

TL;DR

• Go's build cache includes each package's source directory in its key unless you pass -trimpath. A fresh worktree recompiles the whole in-module dependency closure cold, and leaves behind cache entries nothing will ever read again.
• Go's test cache records every file a test opens by absolute path, fingerprinted by size and mtime, not by content. No flag changes that. A test that reads testdata/ from disk can't reuse a result from another worktree, and regenerating a file with identical bytes still invalidates it.
• On a fresh worktree whose commit another worktree had already built, the release gate went from 60.7 minutes and 1,121 re-run test packages to 8.2 minutes and 196 with -trimpath, then to 4.2 minutes and 25 after embedding test fixtures.
• The cleanup job that should have kept the cache in check never ran, because its "is a build running?" guard was true essentially all the time. The fix was an age-based trim that's safe to run during builds.
• The rule I ended up with: a Go package's test results only cache across worktrees if its tests open no files at all.

The setup

The repository is the API monorepo behind my scraping product. It's one Go module with roughly 780 packages, and a lot of the work on it is adding new scraper families, which are self-contained enough to hand out in batches. One batch last week was 23 news-site families. Each agent gets its own git worktree, builds and tests its slice, pushes, and a coordinator merges the batch and runs the full release gate (make verify) once at the end.

All of it runs on one Mac with 8 cores, alongside whatever other sessions I have open.

The first problem was the obvious one, CPU. Early batches ran about eight agents at once. At eight, each compiling the router and MCP server packages, load reached 74 alongside other sessions' race and coverage runs, and I lowered the cap to three concurrent build agents. Each brief now also says to run Go with -p=2, one Go command at a time, against the agent's own packages only.

The second problem was that each agent, sensibly, wanted to run the full gate before calling its work done. The build lock (a mkdir-based semaphore with two slots) bounds how many full gates run at once, but it can't stop seven agents from all wanting one. In a seven-agent batch, six of the seven were waiting on the lock at any given moment, making no progress. The fix was a rule, not code: contributors run scoped checks against their own packages, the pre-push hook runs a diff-scoped check automatically, and the full gate runs once, by the coordinator, after everything has landed.

Even with that rule, an eight-agent batch the next day pushed load to 260–290. Creating the eight worktrees took more than ten minutes on its own, and two agents sat for over 600 seconds with no progress. That one wasn't a bug. It was an 8-core machine with far more work queued than it had cores. I waited it out and resumed them.

The fix that made it worse

When the disk fills during a build, Go reports no space left on device, and the tempting response under fan-out is to give every agent its own GOCACHE and GOMODCACHE "to be safe." Then no agent can corrupt another's cache, and one agent's eviction can't break another's link step.

That works against you. N isolated caches means N agents each doing a full cold download and build instead of sharing one warm cache. During one batch the shared cache reached 119 GB, plus however many isolated copies agents had created. My notes say that was very likely what filled the disk that day, and the knock-on effect was worktrees being force-deleted under agents that were still using them. Isolation is now reactive only: do it after an actual eviction error, and delete the isolated cache yourself when the task is done.

The cleanup job that never ran

I already had a cleanup job. It runs every two hours and has a step that clears Go's build cache. That step had a guard: skip it if a Go compile is active, so you don't delete an entry a running build is about to read.

The guard was correct. On this machine it was also true almost all the time. During one pass I counted eight or more live compile and go test processes across different sessions. So the job kept running, reported success, and never touched the cache. The same "correct but always active" guard turned out to be skipping the per-task scratch directories too, for the same reason.

Two data points show how fast the cache grows without it. On one evening it was back to 89.9 GiB about eight hours after I'd last cleared it. The next time I looked, it was 202 GB.

The emergency fix is to mv the cache directory aside and delete the old one in the background. Moving is instant, so running builds see an empty cache instead of half-deleted entries. There's one catch: Go creates the 00 to ff subdirectories once, when a process starts. A build that's already running when you move the directory will try to write into subdirectories that no longer exist. So recreate them right after the move:

mv ~/Library/Caches/go-build ~/Library/Caches/go-build.old
mkdir ~/Library/Caches/go-build
for i in $(seq 0 255); do mkdir -p ~/Library/Caches/go-build/$(printf %02x $i); done
nohup rm -rf ~/Library/Caches/go-build.old >/dev/null 2>&1 &

On one of these passes, eight Go processes were running when I moved the directory. Within seconds they had already written new files into the fresh directory.

The permanent fix came from reading how Go itself trims the cache. When a build uses a cache entry, Go refreshes the entry's mtime, but at most once an hour (mtimeInterval in cmd/go/internal/cache/cache.go). Go's own trim deletes entries by mtime age. So an entry that hasn't been touched in two hours isn't being used by any running build, and deleting it costs at worst a cache miss. The job now trims entries older than six hours on every run, outside the guard, and drops to two hours if the cache is still over 20 GB. It has a hard two-hour floor.

It has a limit, and I wrote it down: if agents write more than the free disk space within two hours, it can't keep up, and the mv is still the answer.

That covered the disk. It didn't explain why the cache was growing by tens of gigabytes a day in the first place.

Why every worktree rebuilt everything

The clue was in the pre-push hook. When you push a commit that isn't your current checkout, the hook checks it out into a brand-new temporary worktree and runs the checks there. Those checks were slow in a way that didn't make sense for a small change.

I timed one package, go test -vet=all -run '^$' with no tests actually run, just compile and vet:

Same code, same commit, same cache. The only difference was the directory. Without -trimpath, Go folds each package's source directory into its build-cache key, so a new path is a new key. A fresh worktree recompiles the whole in-module dependency closure from scratch, and a change to the router package pulls in most of the module. Every one of those compiles writes cache entries keyed to a temporary path that will never be built again. That's where the tens of gigabytes a day were going.

The pre-push fix was to stop using fresh paths. The hook now keeps four fixed checkout slots per clone and switches them with checkout --force --detach plus clean -ffd, so only changed files get rewritten. An end-to-end hook run for a small commit took 55 seconds with a new slot and 14 seconds with a reused one.

The release gate couldn't use fixed slots, because it runs from a fresh worktree by design. So the gate now runs Go with -trimpath, which drops the source directory from the build-cache key. A new worktree can then reuse whatever any other worktree already compiled.

That fixed compilation. Tests were a separate problem.

The test cache is keyed on paths too, and on mtimes

With -trimpath in place, a fresh worktree on an already-built commit still re-ran 196 test packages, 1,057 test-seconds of work that had already been done somewhere else.

Go's test cache works differently from the build cache. When a test runs, Go records every file it opens and every environment variable it reads, and the cached result is only reused if those are unchanged. Two details of how it records files matter here:

1. Files are recorded by absolute path. A test that reads testdata/fixture.json records /path/to/worktree-a/.../testdata/fixture.json. In worktree B that's a different file, so there's no hit. -trimpath doesn't affect this, and no flag does.
2. Files are fingerprinted by size and mtime, not content.

The second one explained another thing I'd seen. The gate regenerated the API docs (docs/swagger.json) in place on every run to check they were up to date. The output was byte-identical, but every write gave the file a new mtime, and that invalidated the cached result of every package whose tests read it. The docs step now generates into a temp directory and only replaces files whose content changed.

For the fixture reads, I looked at the 196 packages and 174 of them had one thing in common: os.ReadFile("testdata/..."). The fix was to embed the fixtures:

// testdata_embed_test.go
package foo

import "embed"

//go:embed all:testdata
var testdataFS embed.FS

Embedded files are compile inputs, keyed by content. They share across worktrees like any other build output, and the test no longer opens anything on disk. 215 packages got this change. An architecture test at the repo root now fails on any direct os.ReadFile or os.Open of testdata in a test, so it doesn't creep back. (Use path, not filepath, with an embed FS. Embed paths are always slash-separated.)

Here's the whole sequence, on a fresh worktree whose commit another worktree had already built. "Net" is wall time minus time spent queued for the build lock:

This is the best case, and I want to be clear about that. Another worktree had built the same commit first. That's the normal release flow here: verify in one worktree, then build from a clean one, and the coordinator re-verifies. A commit nobody has built yet still compiles its changed packages and their dependents. What it gains is a single test pass, a shared cache for the standard library and dependencies, and one lock acquisition instead of six.

The remaining 25 packages mostly can't be fixed by embedding. Some tests scan router source files, some read config, some load production data files. go:embed can't reach a parent directory, so an embedded copy has to be exported by the package that owns the file. I did that for the API spec (docs.Spec) and it saved about 35 test-seconds. The linker drops an embed that nothing reachable references, so the API binary is 243,354,514 bytes with or without it. That only holds while production code doesn't use it, which another test enforces.

Two notes if you want to check this in your own repo:

• A package only caches across worktrees if its tests open no files at all. That includes the os.Stat("go.mod") walk that repoRoot() helpers do to find the module root.
• Use GODEBUG=gocachetest=1 to see why a test result was or wasn't reused. Don't use -count=1 to test anything about caching. It doesn't just skip reading cached results, it also doesn't store them, so it can't show you a hit.

The lock that lost its place

One more line in that first table deserves its own section. Of the 42.8 minutes the old gate spent waiting for the lock, 39 were in a single wait between two stages.

The gate used to take the lock separately for each stage: vet, race, cover, doc check, and so on. The lock has no queue order. When one stage finished and released its slot, a waiting agent could grab it before the next stage re-acquired, and a half-finished gate could sit behind the entire backlog. Now the gate holds one slot for all its stages. The holder also touches its slot every 60 seconds. Previously a slot was reclaimed purely by age, and on one occasion a live, slow race build had its slot taken because it looked stale.

Two smaller traps that had the same shape

A healthy job that always reported failure. The cleanup job's last exit code was 1, and it had been 1 for every run in the current log. The job was working. It pruned each registered Docker buildx builder, and one of them pointed at a container that no longer existed. That error, 89 times in the log, set the exit status on every run. So the one signal I'd look at to see if cleanup was broken said "broken" permanently, which made it useless for noticing a real failure. The fix was removing the dead builder.

An empty answer from a broken check. The cleanup runbook uses find -newermt '-60 minutes' to check whether anything wrote to a directory recently before deleting it. In the coding agent's shell, find is wrapped to run bfs, which only accepts absolute timestamps and exits with an error on '-60 minutes'. Every probe had 2>/dev/null on it. So every probe printed nothing, and the scripts read nothing as "no recent writes." I caught it because one probe ran without the redirect. The deletions that day also rested on other checks that did work (no open file handles, no process referencing the path, old mtimes from stat), so nothing live was hit. The lesson is broader than find: never discard stderr on a check whose empty output means "safe to delete."

Both are the same failure as the cleanup guard: a check that returns the same answer whether or not anything is wrong.

What I'd tell someone setting this up

Build caches assume one checkout, and so do test caches. If you build one Go module from many directories, whether that's agent worktrees, CI runners with random workspace paths, or throwaway checkouts in a hook, use -trimpath for anything whose output you want to share, reuse fixed paths where you can, and embed test fixtures instead of reading them from disk.

Content-keyed caches beat path-keyed ones, and mtime is not content. Anything that rewrites identical files, like code generators or formatters, invalidates every cache keyed on mtime downstream of it. Generate into a temp directory and replace only what changed.

Adding agents adds contention before it adds throughput. Past three concurrent build agents on eight cores, I got load averages in the hundreds and agents waiting on each other. Letting every contributor run the full gate turned the lock into a queue. One consolidated gate at the end was faster than N parallel ones.

Check your guards. A cleanup guard that's always active, a cron job that always exits 1, and a probe that always prints nothing all look fine in isolation. Each one hid a real problem. For any check you rely on, confirm it can actually produce the other answer.