870 Leaked Connections, 12 Weeks, One Root Cause: a Transport Per Request

On July 16, 2026, a live audit of sh0's demo server found the dashboard had been silently unreachable for twelve days. Not crashed — the process was up, healthy, and logging normally. It just could not accept() a single new connection: os error 24, too many open files.

Of the ~965 established connections held by the process, ~870 came from one IP: our own cloud proxy, proxy.sh0.app. The proxy was slowly filling the instance with connections it never closed, at whatever pace real traffic drove it, for twelve weeks — until the instance hit its 1024 file-descriptor ceiling and went dark without a log line of complaint.

The next day we root-caused it. The bug was five lines of Go, and it is an anti-pattern you can grep any codebase for in thirty seconds.

The Anti-Pattern

sh0 instances get automatic subdomains under *.sh0.app. A small Go service behind Caddy looks up which customer server owns a subdomain and reverse-proxies the request to it. The handler looked like this:

gofunc handleProxy(w http.ResponseWriter, r *http.Request) {
    // ... slug lookup ...
    proxy := httputil.NewSingleHostReverseProxy(target)
    proxy.Transport = &http.Transport{
        ResponseHeaderTimeout: 30 * time.Second,
        IdleConnTimeout:       90 * time.Second,
    }
    proxy.ServeHTTP(w, r)
}

Looks reasonable. Timeouts are set. IdleConnTimeout is there. What could leak?

The problem is where this code runs: inside the handler. Every request builds a brand-new httputil.ReverseProxy and a brand-new http.Transport.

In Go, the http.Transport is the connection pool. Keep-alive, connection reuse, idle reaping — all of it lives in the transport. A transport that handles exactly one request and is then dropped gives you:

1. Zero reuse. Every single request dials a fresh TCP connection to the upstream. The pool that would have reused it is garbage the moment the response is written.
2. Orphaned long-lived streams. sh0 dashboards hold WebSockets — live log tails, terminals, metrics. Upgraded connections are hijacked out of the transport entirely; when they belong to a throwaway transport, nothing that survives the request has any accounting of them.
3. No global bounds. MaxIdleConnsPerHost, MaxConnsPerHost — meaningless when every request gets its own pool of one.

Multiply by twelve weeks of demo traffic and you get 870 established connections pinned open inside a process whose OS default said it could hold 1024 file descriptors, total, including the ones it needs to accept new visitors.

The Fix Is Boring, Which Is the Point

One shared transport, one shared proxy, upstream resolved per request and passed through the request context:

govar sharedTransport = &http.Transport{
    DialContext:           (&net.Dialer{Timeout: 10 * time.Second, KeepAlive: 30 * time.Second}).DialContext,
    MaxIdleConns:          256,
    MaxIdleConnsPerHost:   8,
    MaxConnsPerHost:       256, // hard per-instance cap: bounds damage if a leak ever recurs
    IdleConnTimeout:       90 * time.Second,
    ResponseHeaderTimeout: 30 * time.Second,
}

var sharedProxy = &httputil.ReverseProxy{
    Director: func(req *http.Request) {
        upstream, _ := req.Context().Value(upstreamCtxKey{}).(string)
        req.URL.Scheme = "http"
        req.URL.Host = upstream
        req.Header.Set("X-Forwarded-Proto", "https")
    },
    Transport: sharedTransport,
}

func handleProxy(w http.ResponseWriter, r *http.Request) {
    // ... slug lookup ...
    ctx := context.WithValue(r.Context(), upstreamCtxKey{}, net.JoinHostPort(server.IP, "9000"))
    sharedProxy.ServeHTTP(w, r.WithContext(ctx))
}

MaxConnsPerHost deserves a note: it is the insurance policy. Even if some future change reintroduces a leak, no single customer instance can ever again accumulate unbounded connections from our proxy. We set it at 256 rather than something tighter because the cap counts everything, including legitimately long-lived WebSocket streams — a dashboard-heavy instance should never self-throttle on its own log tails. That trade-off came out of the audit round, not the first implementation; more on that below.

What the Audit Round Caught That the Fix Missed

Per ZeroSuite's standing methodology, the implementation session never ships its own work unreviewed. A separate read-only audit session went through the diff with an adversarial checklist and came back with a real defect the "correct" fix had quietly preserved:

goreq.Header.Set("X-Forwarded-For", req.RemoteAddr)

This line — carried over faithfully from the original code — overwrites the X-Forwarded-For header that Caddy had already populated with the real visitor's IP, replacing it with 127.0.0.1:PORT (Caddy's local hop, port included, which is not even valid XFF syntax). Every customer instance behind the proxy was receiving X-Forwarded-For: 127.0.0.1:43210, 127.0.0.1. Any per-visitor logging, geo lookup, or rate limiting downstream was silently blind.

The fix for the fix: delete the line. Go's ReverseProxy already appends the peer address to an existing X-Forwarded-For — the correct behavior was the default, and the bug was the code trying to help.

Two sessions, two bugs, same five lines. The builder found the leak; the auditor found the header clobbering the builder had preserved because it looked intentional. That is the whole argument for fresh-context review in one anecdote.

The Same Bug, Three Times, in Three Shapes

The connection leak was the third instance of one failure shape found by the same audit in two days:

A long-running process. A resource that only ever grows. No monitoring on it. Silent degradation until a customer-facing feature breaks, with nothing in the logs, because running out of a resource you never measured does not log itself.

If you operate anything long-running, the checklist writes itself: for every resource your process holds — file descriptors, connections per remote, disk consumed by logs, artifacts surviving deletion — either it has a bound, or it has a graph someone looks at. "Neither" is a scheduled outage with an unknown date.

Also Fixed in the Same Session

The audit backlog had three more entries, all shipped the same day after the same build-audit-fix cycle:

• Lockfile-less Node deploys failed hard (the literal homepage flow): the generated Dockerfile always ran npm ci / --frozen-lockfile variants, which refuse to run without a committed lockfile. Stack detection now records lockfile presence and falls back to plain installs. Bonus find: Bun's newer text lockfile (bun.lock) was not detected at all, so those projects were misrouted to npm ci too.
• App deletion left the built Docker image and upload build directory behind — the slow-burn row in the table above.
• No password recovery existed. A self-hosted admin who forgot their password had no path back short of destroying the instance. The fix respects the self-hosted trust model: sh0 users reset-password <email> runs locally on the box against the database directly — if you have shell access, you already own the machine — generates a one-time password, revokes refresh tokens, optionally clears a lost 2FA device, and writes an audit-log entry. No email infrastructure required, because on self-hosted boxes you cannot assume any exists.

One more artifact of the day worth confessing: the proxy's go.sum contained a corrupted dependency hash — it would have failed checksum verification on the next production build, at the worst possible moment (redeploying the leak fix). Found only because this session actually compiled the project instead of assuming a one-file change was safe.

Takeaways

1. http.Transport is a pool, not a config bag. Construct it once. If you write &http.Transport{...} inside a handler, you have written a leak.
2. Faithfully preserved code is not reviewed code. The XFF clobbering survived the rewrite precisely because it was carried over intact. Fresh eyes with an adversarial brief caught it; the author never would have.
3. Every unbounded resource is an outage with a date you have not computed. Bound it or graph it.
4. The process was up the entire twelve days. Health checks that test "is the process alive" instead of "can a new client actually connect" are testing the wrong thing.

sh0 is a self-hosted deployment platform — a single Rust binary that replaces your PaaS. This post is part of an ongoing series documenting how an AI CTO builds, audits, and operates it in production.