The Autoscaler That Compared a Space to a T: Two High Bugs, One Day, Before and After

Two High bugs sat at the top of sh0's live-audit registry on the morning of 2026-09-10. By the evening both were closed, and closed the only way this project accepts: with a dated observation on a real machine, before and after, same applications, same probe. This is what it took.

Bug one: an autoscaler that never decided

sh0's autoscaler wakes every 30 seconds, reads the last two minutes of CPU and memory samples for each application with a scaling policy, and scales up or down. On the demo box it had been running for weeks. It had never scaled anything.

The metrics table stores recorded_at through a column default:

sqlrecorded_at TEXT DEFAULT (datetime('now'))   -- 2026-09-10 00:43:04

The autoscaler built its window bound in Rust:

rustlet since = (Utc::now() - Duration::seconds(120)).to_rfc3339();  // 2026-09-10T00:41:28+00:00

Then the query said recorded_at >= ?. The column is TEXT, so SQLite compares strings. At index 10 the stored rows carry a space (0x20) and the bound carries a T (0x54). Every row of the day sorts below the bound. Zero rows, always, on every application. should_scale_up and should_scale_down were both permanently false.

The fun part: the field right next to it, last_scale_at, was RFC3339 on both sides and worked perfectly. That is how the cooldown had been proven live the day before while the thing it was cooling down had never fired once.

The fix is deliberately not a migration. Every installed box writes the column through the SQL default, so the bound moves to the stored shape instead, through one construction point:

rustpub fn bound_seconds_ago(secs: i64) -> String {
    (Utc::now() - Duration::seconds(secs)).format("%Y-%m-%d %H:%M:%S").to_string()
}

The unit test inserts a row through the default and reads it back with that bound. It also pins the defect: a bound made of the same digits with a T in place of the space must find nothing. The first draft of that assertion used now - 120s and would have flaked for two minutes after every UTC midnight, because the date part changes first. The adversarial audit caught it.

Bug two: fifty seconds of 502 on every redeploy

The second bug had already been "fixed" once. The public *.sh0.app path goes through a small reverse proxy inside sh0 that caches domain -> ip:port for 60 seconds. A redeploy replaces the container, so the IP changes, so the cache must be purged. The September 6 fix added a purge. The September 9 replay measured the exact same 50-second outage.

The diagnosis from that replay was right and it was about order, not keys. The purge ran inside route_container, which runs before the old container is stopped and before finalize_app writes the new container_id to the database. Any request landing in between re-resolved through the database, found the old container, and re-cached its doomed upstream with a fresh 60-second TTL. The purge could only work if nobody visited the app during the deploy, which is the one case where the bug had no victim.

Reading the code added a detail the fix of the 6th had missed: the git path, the one every git push deploy takes, never called route_container at all. It routed inline. It had never purged anything.

The fix moves the database write before the purge and both before the stop, on all five deploy paths, and a source-order test reads pipeline.rs and refuses any path that inverts it. The proxy gets a defense for the replacements the pipeline never sees: a cached upstream that refuses the connection is evicted and re-resolved once; an upstream that did not answer is never cached; a timeout is never retried, because a POST that timed out may have been received.

The proof, and the app that refused to reproduce the bug

The rule here is that "fixed in code" is not a state. A fiche closes on an observation with the command and its output, or it stays open.

The morning went to baselines on the box under v1.7.1. For the proxy bug the first attempt used traefik/whoami: thirty probes, thirty 200, no bug. Same binary the replay had used the day before. The difference was the container: whoami handles SIGTERM instantly, so the window between purge and death lasted milliseconds and a probe every three seconds never landed in it. The Go app from the replay ignored SIGTERM and docker stop waited its full thirty seconds. The bug needs a container that dies slowly. With the exact repository from the 9th, the baseline gave 000, 000, 502, 502, 502 from +12s to +55s. A Python http.server running as PID 1 gave two 000. Those are the two apps the afterwards was measured on.

For the autoscaler: three nginx replicas at rest, a policy of 1 to 3, and a watch every 30 seconds. Under v1.7.1: 36 samples inside the stored window, three replicas for four and a half minutes, zero decisions.

Then the release candidate. The verification agent ran the full sweep (975 tests), the tag went up, GitHub took two hours to get to the build, and the pinned install landed at 09:06:56 UTC.

09:07:01  INFO  Autoscaling down  current_replicas=3 target_replicas=2
09:08:31  INFO  Autoscaling down  current_replicas=2 target_replicas=1

First tick after the restart. The proxy probes: thirty 200 on the git path, thirty 200 on the Dockerfile path, and for the defense, docker stop gave one 404 in five seconds, docker start gave 200 in three milliseconds with no wait for the TTL.

What the campaign found that nobody was looking for

The counter-proof for the autoscaler was supposed to be scale-up under load. Forty parallel curl loops for 150 seconds pushed 106 MB out of the container. The CPU metric stayed at 0.0 across all 22 samples. Scale-down works because zero is always below the threshold. Scale-up on CPU can never fire. That is a new Medium fiche, with the file to look at and the observation that will close it, and it does not reopen the one just proven: the proof owed was the descent to min_replicas, and that happened.

The mechanics that made one day enough

• The decision session in the morning replayed its own prompt against the code before believing
• it and found six drifts, one of which was that the prompt classified a toolchain-blocking bug as
• acceptable debt.
• The adversarial audit ran before the commit, in a read-only agent, and returned five fixes. Four
• were applied before the compiler ever ran.
• The verification ran in a background agent whose output never entered the main context.
• The version was bumped before the RC tag, because the previous RC had reported the old version
• string on the box and nobody could tell what was running.
• The box was returned to zero applications, and the baseline logs are in the repository next to
• the fiches.

The registry moved from 22 open to 21. One High remains, blocked on a hardware question that belongs to the CEO. The launch clock still has not started.