My \"HTTP/1.1\" Go Client Advertised h2, and the Fix-up for It Silently Stopped Working

November 22, 2026 (Today)

Support me on Patreon to write more tutorials like this!

profile picture
Tony Wang

I have an HTTP client in my crawler called StandardProxiedHTTP1. Its doc comment says it is a fresh standard-library HTTP/1.1 client. It advertised HTTP/2.

That alone is a small Go trap. What turned it into an outage was a second piece of code I had written about three weeks earlier to patch exactly that trap. It checked whether the client's transport was a *http.Transport and, if not, returned the client unchanged without a word. On 2026-11-15 I added a wrapper around the transport. The check started failing, the patch stopped doing anything, and a registration call that the service depends on failed on every attempt I measured until I found it on 2026-11-19.

TL;DR

  • ForceAttemptHTTP2 = false on a cloned http.DefaultTransport does not stop the TLS handshake from offering h2. The ALPN list is already filled in by then. The server picks h2, my HTTP/1.1 reader gets binary frames and reports malformed HTTP response.
  • My earlier fix-up did client.Transport.(*http.Transport) and returned the client on failure. A later wrapper type made the assertion fail, so the fix-up quietly did nothing.
  • Registration counters read 20 failures and 0 successes, with 60 transport retries, and no pre-minted standby identity on any of the five replicas.
  • No alert fired, because the service kept answering with the identity it already had, and the one rule that watched for a missing standby had never been able to evaluate.
  • I found it by repairing that dead rule. Once it could fire, it did.
  • The fix pins both http.Protocols and ALPN inside the accessor, lets callers walk to the real transport through wrappers, and makes the old patch complain loudly when it can't reach it.

How I found it

I wasn't looking for this. On 2026-11-19 I was tidying the alerting for another worker, where five rules fired for one condition, and while validating the rule set I noticed several rules with health = "err". Prometheus was skipping them on every evaluation.

One of them was TiktokMobileIdentityStandbyMissing. It errored with vector contains metrics with the same labelset after applying alert labels. The alert overrode the real per-replica instance label with a constant, which collapsed five series onto one labelset. I aggregated it to a single fleet-level series. It went to health = ok, and within one evaluation window it went to firing.

Firing meant standby_present was 0 on all five replicas. That could have meant slow provisioning. So I looked at counters the alert doesn't mention:

registrations_total{outcome="failure"}                 20
registrations_total{outcome="success"}                  0
registration_retries_total{outcome="transport"}        60
standby_replenish_failures_total                       7 per pod

Zero successes, cumulatively, is a different statement from a high failure rate. It turned "provisioning is slow" into "this has never worked on these pods". The log line on every attempt said why:

net/http: HTTP/1.x transport connection broken:
  malformed HTTP response "\x00\x00\b\a\x00\x00..."

That string is not garbage. The first three bytes of an HTTP/2 frame are its length, and the fourth is its type. Here the length is 8 and the type is 7, which is GOAWAY. An HTTP/1.1-only reader was being handed an HTTP/2 frame by a server that believed it was speaking HTTP/2.

Bug one: the flag does not mean what it says

The accessor was:

transport := http.DefaultTransport.(*http.Transport).Clone()
transport.ForceAttemptHTTP2 = false
transport.DisableKeepAlives = true

I printed the transport's state on the build host (Go 1.26.6) before and after:

DefaultTransport.ForceAttemptHTTP2   = true
DefaultTransport.TLSClientConfig     = <nil>

--- after clone + ForceAttemptHTTP2=false ---
ForceAttemptHTTP2 = false
TLSClientConfig   = &{... [h2 http/1.1] ...}
NextProtos        = [h2 http/1.1]

Two behaviours combine. Clone() runs the lazy HTTP/2 setup on the shared source transport as a side effect, and that setup writes h2 into the TLSClientConfig.NextProtos list. The clone then inherits a config that already offers h2. ForceAttemptHTTP2 only decides whether the transport installs an h2 round tripper, not what the handshake offers.

So the client offered h2 first, the edge accepted, and then my client sent an HTTP/1.1 request line into an h2 connection. The edge answered, correctly, with a GOAWAY. The server did what it should. The only party that lied was my client, in ALPN.

The correct pin needs both halves:

protocols := new(http.Protocols)
protocols.SetHTTP1(true)
transport.Protocols = protocols
if transport.TLSClientConfig != nil {
    transport.TLSClientConfig = transport.TLSClientConfig.Clone()
} else {
    transport.TLSClientConfig = &tls.Config{}
}
transport.TLSClientConfig.NextProtos = []string{"http/1.1"}

Protocols stops the h2 round tripper being installed. NextProtos stops the server ever choosing h2. Setting only the first is exactly the bug.

Bug two: a guard that could not fail loudly

I had met this symptom before. On 2026-10-24 I added a patch at the one call site that saw it, in the registration code:

transport, ok := client.Transport.(*http.Transport)
if !ok || transport == nil {
    return client        // silently does nothing
}
// ... pin Protocols and NextProtos ...

It worked when I wrote it. On 2026-11-15 I merged a wrapper that gates each physical attempt, and when a proxy is configured, which it always is in production, the accessor began returning a client whose transport is a wrapper type. I checked the two shapes directly:

ungated transport type: *http.Transport
gated   transport type: request.contextPhysicalAttemptUnsupportedRoundTripper

The assertion failed, return client ran, and the h2-capable client shipped. There was no log line, metric or error. The function did exactly what its if said.

Anyone who adds an instrumented round tripper, a retry shim or a gate in front of a transport turns this pattern into a no-op. The mistake was making the failure branch quiet.

Reproducing it without the platform

The mechanism needs no network. Against a local httptest server with EnableHTTP2 = true, using the pre-fix shape:

pre-fix  ALPN=[h2 http/1.1]
server:  http2: server: error reading preface from client ...: bogus greeting "GET / HTTP/1.1..."
pre-fix  request error: net/http: HTTP/1.x transport connection broken:
         malformed HTTP response "\x00\x00\x1e\x04\x00\x00..."
fixed    ALPN=[http/1.1]
fixed    proto=HTTP/1.1

The server-side line is the same event as the production GOAWAY, seen from the other end. Two entries in a list decide everything.

Why nothing alerted

Two blind spots had to line up.

The first is that the failure only degraded one path. When rotation fails, the service keeps answering with the identity it already has, and logs one warning per process. Every route kept returning 200, so no error-rate, latency or yield alert moved. Stale but functional is the hardest kind of degradation to notice without a counter that targets it.

The second is that the counter that would have shown it was on a dead rule. standby_present = 0 is exactly the signal for "no pre-minted identity". The rule on it could never evaluate, so Prometheus skipped it, and a rule that cannot evaluate looks identical to a rule with nothing to report.

The fix and how I checked it

  • The accessor pins http.Protocols and NextProtos itself, so its name matches its behaviour for every caller. Another caller of the same accessor got the fix for free.
  • The gate's wrapper type gained an UnwrapTransport() method, and the old patch now walks the wrapper chain.
  • The old patch logs http1_unwrap_failed instead of returning silently. I kept it as a second pin, but it is now loudly redundant instead of quietly load-bearing.
  • Two regression tests assert the transport is HTTP/1.1-only with ALPN exactly [http/1.1], both directly and through the gate.

An in-cluster probe against the real endpoint showed the fixed client getting HTTP/1.1 with a 200, and a plain Go client as a control getting HTTP/2.0 with a 200. That control matters: h2 was never forbidden. The bug was the mismatch between what the client promised and what its reader could parse.

The probe only explains why. The acceptance test is the product, so I read the counters after the rollout:

measurebeforeafter
registrations, failure200
registrations, success01
standby_present0 on all 5 replicas1 on all 5 replicas
standby replenish failures7 per pod0 on all 5 pods
TiktokMobileIdentityStandbyMissingfiringresolved

What I'd do differently

  • Print the ALPN list, don't read the flag. The flag name is unambiguous and the behaviour isn't.
  • Don't write a guard that falls back silently. If a fix depends on reaching a concrete type through a chain, walk the chain, and fail loudly when it ends somewhere unreachable.
  • Give every degraded-fallback behaviour its own counter and its own rule, and check that the rule can evaluate. Looking at rule health in the /api/v1/rules payload on a schedule would have shown me the dead rule long before this incident.
  • Treat "never worked" as structural. A success counter that has never moved is a different finding from a high failure rate.

What I don't know

The "four days" is inferred, not measured. The wrapper landed on 2026-11-15 and the fix shipped on 2026-11-19, and the counters I read show zero successes, but those counters are per pod lifetime and I did not reconstruct every deploy in between. Some of the figures in my own notes mix lifetime counts with 24-hour ones, so I've kept them separate here. I also haven't checked how many other places in my codebase set ForceAttemptHTTP2 = false and call it HTTP/1.1.