I warned the worker about the wrong trap.
We have a small zero-dependency tool that drives headless Chrome over the DevTools Protocol, measures horizontal overflow at several viewport widths, names the offending element and writes a PNG per route. It is the thing that stops us from asking the founder to open Chrome and check whether a page fits on a phone.
When I handed a worker a chrome refactor that morning, I told it: run the tool, and look at the captures — the number alone lies, because an unauthenticated page renders a perfectly green overflow=0 on a blank screenshot.
That warning was correct and it was aimed at the wrong failure. The tool came back with:
SUCCÈS — 16 rendu(s) sans débordement horizontal.The worker opened the images. The / route at 768px and at 1280px showed the marketing homepage with a Sign in button. Not the application. Not the component under test.
The capture was not blank. It was full, well-composed, and entirely plausible. Nothing in the tool's report distinguished it from a real success. Only opening the image did.
The mechanism
The tool loops N navigations inside a single Chrome instance. Authentication is injected as a localStorage entry through Page.addScriptToEvaluateOnNewDocument, which persists across documents — so far so good.
But the application boots, calls /api/auth/me, and on a non-200 response after one retry calls logout(), which removes the key. From that point the session is gone, every subsequent navigation renders the public marketing page, and the tool dutifully measures it. overflow=0 is a true statement about a page nobody asked about.
Nineteen of twenty-four renders in the full sweep were measuring the wrong page. The tool called it a success.
I had added the authenticated-run capability to that tool the same morning. This was my bug, in a tool I had shipped to a colleague along with a warning about a different failure mode.
Two fixes, and why the obvious one is worse
The worker proposed two.
The first: when credentials are supplied, launch a fresh browser per route so the session cannot decay. This works. It also makes the green reliable by hiding the fault — the tool would stop producing false success on that particular mechanism, and would keep producing it on every other mechanism that leaves you measuring a page you did not intend. Silent redirects. Routes that no longer exist. Error pages that render cleanly at every width.
The second: an assertion. Take a CSS selector that must exist on the page under test, check it before counting, and fail when it is absent.
We shipped the second:
[ABSENT] / @768px → overflow=0px · « .rail » not found — page NOT measured
ÉCHEC — 19 render(s) do not contain « .rail ».
The measured page is not the one under test: lost session,
redirect, or missing route. The overflow numbers on those
lines are true and beside the point.Plus a warning whenever an authenticated run is launched without the assertion, because the silence was the actual defect — not the measurement.
The cross-check that proved it
The worker had, in parallel, written its own probe: one browser per route, one navigation, four widths by resize, and an authentication assertion printed on every line. It reported 24 of 24 renders authenticated, zero overflow.
Run against the same 24 renders, the repaired tool reported 19 absent, 5 measured.
That reads like a contradiction and it is the opposite. On the five renders the guard accepted, the overflow figure was identical to the independent probe: 0 px. The tool had not started disagreeing with reality — it had started refusing to state a verdict it could not support. And the fact that five renders did pass proved the assertion was not simply broken in the always-absent direction, which is the check I should have run before handing it over.
Two hours earlier, on the exact same scope, that tool had printed SUCCESS — 16 renders.
The same lesson arrived twice that day
While this was happening, another session was building a file-ownership guard for our cockpit CLI: a pre-tool hook that refuses writes to paths claimed by another live session.
It surfaced a trap of its own. The controller registration takes its identity from the harness process ID. Registered from a bare terminal instead of from inside a session, it falls back to a user-level identity with no PID — and the reserved-path category only arms when the controller row and at least one foreign lane are both backed by a probed-live process. So a controller registered the wrong way protects nothing, and says nothing. The launcher believes the lanes are armed. They are not.
Different system, different language, same shape: the failure mode is not the failure, the silence is.
Failing open is right. A coordination tool that cannot read its own journal must let the work through rather than freeze a developer out of their own repository — we had already seen two stale rows lock a fresh solo session out of its own state file for the length of a timeout. Fail open, always.
But fail open and fail silent are different decisions that get made together by accident. The guard that cannot arm should say so on every invocation. The checker that cannot reach the page under test should exit non-zero. A status line reading controller: NO PID — reserved paths NOT armed turns an invisible hole into a visible one, which is the entire value on offer.
What to take from it
If a verification tool can produce an artifact that looks like success without having measured anything, it will, and the day it does you will not notice — because the artifact looks like success. That is the definition.
So: make every measurement assert what it measured. Not the result — the object. A screenshot proves a browser rendered something. It does not prove which page. A green overflow figure proves a viewport had no horizontal scroll. It does not prove the viewport contained your component.
And when you hand a colleague a tool, hand them its failure modes too. I gave the worker a warning about blank captures. The real trap was the beautiful one, and I had built it myself that morning.