Support me on Patreon to write more tutorials like this!

One of the endpoints in my crawler returned 429 Too Many Requests to every request from the day its monitors went red until I corrected it five days later. I gave two explanations in that time. The first was a quota on the app's embedded API key. The second was that the app had started attaching Google Play Integrity attestation, which a server can't produce, so I wrote the endpoint off as lost.
Both explanations were built from facts I had actually observed. Both were wrong. The cause was the User-Agent header, which my HTTP client had never set. The library filled it in with its own name, and the server didn't like that.
Nobody had removed one candidate requirement from a request and watched the answer change. That is the part I'd like to get across, because I've since found the same exposure in 318 other places in the same codebase.
TL;DR
- Etsy's mobile API (the one the Android app talks to) returned the same generic 429 body to every correctly signed request from two networks, with no
Retry-Afterheader. I read it as a per-key throttle on 2026-11-15 and as Play Integrity attestation on 2026-11-16. - Sending the request again with an app-shaped
User-Agentreturned real data: 3 of 3 in an interleaved A/B, against 0 of 3 with the library's default. Eight more requests, 5 seconds apart, all returned 200, so there was no quota to speak of. - The attestation story was wrong in a way I could have seen on day one. The app's integrity code only runs after a
401that carries aWWW-Authenticatechallenge, and the 401 this API returns has none. - My client library sends
req/v3 (https://github.com/imroc/req)whenever the caller sets no User-Agent. 318 files in the repo build a client that way. The five upstreams I sampled all answered 200 to both that string and a Chrome one, which is why it never showed. - I added a test that fails on any new file that does this, as a shrink-only list rather than a blanket exemption.
The outage
All seven of the monitors for the endpoint went red on 2026-11-15. Replaying the requests from the cluster showed openapi.etsy.com answering 429 to every attempt, with a body that read Send fewer requests per minute and/or limit requests per user that you send at the same time., and no Retry-After or rate-limit headers.
The obvious suspect was the device id. In August a missing x-etsy-device header had put requests into a tightly throttled bucket. So I sent the same signed request twice, once with our shared device id and once with a fresh install-shaped one. From inside the cluster and from my laptop, both returned 429. The limit didn't depend on the egress IP or the device id, so I concluded it applied to the app key itself, probably because the key came from an old build of the app (7.88.0).
That was a reasonable inference from what I had. Real usage of the endpoint was zero requests in 24 hours, so it was a monitor-only outage, and I left the routes failing closed and wrote down the next step: re-extract the key from a current build.
The second explanation
The next day I did that. The key from the current build (8.3.0) was byte-identical to the old one, so it wasn't stale. A request the server rejects with a valid key and a valid signature has to be missing something, so I traced the new app's request path to find out what.
I found something real. The app has an integrity interceptor that gets a challenge from Etsy, asks Google Play Integrity for a token bound to it, and the signing code appends an assertion to the signature header when that token is present. Our transport never sent one. A generic "too many requests" with no reset time, on an otherwise valid signed request, looked like unattested traffic being soft-throttled.
I wrote it up as a hard wall: a Play Integrity token is minted on a certified device and can't be forged or replayed from my egress. The disposition included this sentence: "do not spend probe budget on the throttled key." In other words, I wrote down an instruction not to run the experiment that would have settled it.
The control I hadn't run
Four days later I sent the same signed request twice and changed one header. The results, against the listing-reviews route, from my laptop and from a server on a different network, paced 8 to 10 seconds apart:
| What I changed | Result |
|---|---|
| A User-Agent shaped like the app's own | 200, real reviews |
The same request with Go-http-client/2.0 | 429 |
The same request with req/v3 (https://github.com/imroc/req) | 429 |
| Interleaved A/B, three rounds, nothing else different | app UA 3 of 3 returned 200; Go UA 0 of 3 |
| Eight consecutive requests with the app UA, 5 seconds apart | 8 of 8 returned 200 |
| Either of the app's two UA tokens alone | 429 each |
| Both tokens present, with different version and API-level values, lowercased, or embedded in a longer browser-style UA | 200 each |
x-etsy-device removed, app UA kept | 429 |
| A made-up Play Integrity assertion appended to the signature | 403 Invalid request signature |
So the server wants a User-Agent that contains two tokens the app adds to the platform UA. It does not check their values. It also still wants the device header from August. A 429 for a correctly signed request is this API's answer for "this request is not shaped like the app". It uses the same body for every such refusal, and removing the device header and removing the UA are indistinguishable from the response alone.
Seven routes returned real bodies under the app UA and returned 429 without it: search, listing, shop quickview, shop reviews, shop listings, public shop search and listing reviews.
The 8-of-8 run is what kills the quota theory. A rate limit doesn't let you through eight times in 40 seconds and then block the library's own name. The last row is what kills the attestation theory: with the right UA, no assertion is needed, and bolting one on breaks the signature.
What I should have seen on the first day
The integrity interceptor is reactive. It calls chain.proceed(request) first, and only does anything if the response is a 401 carrying a WWW-Authenticate header with a challenge. Then it fetches a token and retries once. So attestation is a recovery path for a challenge, not a requirement on every request.
The 401 this API returns for a missing key has no WWW-Authenticate header at all. So the interceptor would never have fired against my traffic. I had traced a code path my requests never take and attributed the 429 to the one difference I could name.
A trace that explains a refusal isn't a test that the thing it traced is required. The test is to take that thing away from a request that works, or add it to one that doesn't, and watch what changes.
It was my library
Why was the UA the library's own name? My HTTP package has two families of client profile. The browser-impersonating profiles bring their own User-Agent along with the TLS fingerprint. The plain ones use stock Go TLS and send whatever the caller set, which is nothing if the caller sets nothing. When the caller sets nothing, the vendored HTTP client falls back to req/v3 (https://github.com/imroc/req). That string is written by the library when a request has no User-Agent header (third_party/req/transport.go in my repo), and I deliberately don't force one in the shared code, because a forced UA breaks every non-Chrome profile.
The Etsy transport used one of the plain profiles and never set a UA, so it had been announcing itself as a Go library the whole time. I don't know when the server started to care.
318 files
That made me count. On 2026-11-20 I found 318 non-test files in the repo that build a client of the kind that leaves the UA to the caller and never set one. Those files send a library signature that no browser or app has ever sent.
I sampled five of them live: DuckDuckGo, Google suggest, OpenLigaDB, Kraken and the BBC feeds. Every one returned 200 for both the library default and a Chrome UA. That is exactly why the problem is invisible. A missing header isn't an error at any layer. The request succeeds, the body parses, the endpoint looks healthy, and the failure only appears when one upstream adds bot management and the same call starts returning 403 or 429 with nothing pointing at the header.
I did not want to fix 318 files at once on the strength of one incident, and I did not want to ignore them either. So I wrote a test, with three decisions in it:
- A shrink-only baseline. The 318 existing files are listed in
testdata/ua_required_baseline.txt. A new offender fails the test, and the message says never to add to the list. Fixing one makes the test report the entry as stale and ask you to delete it. The debt has a dated, recorded list instead of an exemption nobody revisits. - Parse the code, don't regex it. My first version stripped comments with a regular expression and silently passed files whose HTML
acceptheader contained*/*, because it read that as the end of a block comment and ate the rest of the file, including the call I was looking for. The test now uses Go's parser and AST. - Exemptions have to say what was measured. A file can opt out with a
user-agent-exemptcomment, and the comment's reason must name the measurement that justified it.
The rule I took from building it: prove each branch (an offender, an exempt file, a covered file) on tiny synthetic files before trusting a pass over four thousand real ones.
A second bug the 429 had hidden
The 429 was also standing in front of a bug nobody could see. Once requests returned 200 for the first time, search came back with the right count and an empty results array. The response is a tree of components, not the flat list my parser expected, and no earlier session had ever captured a 200 body for it. The parser decoded the count correctly and the results as always empty. Paging was also not offset-driven: offset and page alone re-served the same first 36 cards. The response's Link: rel="next" header carries the full parameter set for the next window, so I walk that chain.
What I'd do differently
- Run the subtraction before writing the explanation. Change one header against an otherwise identical request. The probe needed a laptop and a signed request I already had. I wrote up static analysis instead.
- Treat a generic 429 as a shape verdict until it proves otherwise. Quotas and attestation both leave distinguishing signs, such as a
Retry-Afteror a challenge header. A generic body has none. - Ask for a positive and a negative control. "The gate is X" needs X present giving success and X absent giving the same refusal. I had flipped neither.
- Interleave the conditions. This API re-randomizes ad ordering per request, and batched probes let a changing baseline look like an effect. A/B/B/A doesn't.
- Be suspicious of a note that tells you not to test. Mine said not to spend probe budget. A wall you can't probe is a wall you haven't confirmed.
- Treat a library default as a failure mode. The "safe fallback" is the thing the upstream will eventually refuse.
What I don't know
The evidence is small, and it dates from 2026-11-20. Three interleaved rounds and eight sequential requests, from two networks, on one route in the A/B itself, with the other six routes checked once each. It is enough to show that the UA matters. It isn't enough to characterise the rule, and the server's check might change.
I don't have a figure for how long this endpoint was wrongly written off before the correction, beyond the five days between the monitors going red on 2026-11-15 and the correction on 2026-11-20, and since real use was zero requests in the 24 hours before the first outage note, it cost me monitors, not customers. I also haven't measured how many of the other 318 will ever matter. The five I sampled didn't.