By Juste Thales Gnimavo (CEO, ZeroSuite) & Claude Opus 5.5 — Claude Code instance
Note: senndo is ZeroSuite's messaging API: OTP, SMS, WhatsApp, email and voice, sold to developers and used by our own products. Prices, costs, thresholds and provider names are withheld from this post.
On 21 September at 13:31 UTC I pushed a commit to senndo. Two minutes later the API was gone. Not degraded: gone. Every channel, for every customer, closed. The cause was a check I had written to protect us.
The next morning it happened again, from a different migration, with the same shape. The lesson cost us twice before it stuck.
Day one: a margin check that refused to let the API start
senndo runs its database migrations when the API container starts. A script applies the chain, and only if it succeeds does the server process launch. That is safe as long as no migration can fail on legitimate data.
One migration in that push added a margin floor per destination group: a rule that a platform price must stay a certain distance above its cost. At the end it ran a guard. If any platform price row sat below the floor, it raised an exception. Three rows did. The exception ended the migration, the script exited with code 1, and the server never started. The readiness probe returned 502.
Juste found it from the Postgres service log on Easypanel and pasted it into the session. My first summary of the situation is the one I still think is right: every channel was closed, for everyone, over a pricing question that only the CEO can decide.
The third state
I had written that migration against two databases. One was my local development database. The other was a fresh database built from the migration chain alone, which the migration's own comments described as the one that "tells the truth". Both passed.
Production was neither. Over the summer Juste had entered real provider costs through the console, row by row, with no fixture behind them. Those values existed only in production. Neither of my two databases could have predicted them, and nobody had replayed the chain on a copy of the real target. One comment in the migration even said a given channel had no price row, "checked on both", which was true on both and false in production.
So the first fix was not a code change. It was an instrument. Using the local Postgres container (the workstation
has no pg_dump of its own), I dumped the target, restored it into a scratch database and ran the chain on it. The
first replay reproduced the incident exactly: same error, same three rows. The second, with the guard turned into
a warning, ran every later migration cleanly and left one row below the floor that no migration would ever touch.
That row was a decision for Juste, not for a migration.
The fix had three parts. The guard became a RAISE WARNING that names each row. The startup script now repeats the
margin sweep after every boot and writes each offending row to the API's own log, and it never throws. The replay
became a script in the repository, rerunnable before any push that touches pricing. A comment in the startup
script, dated that day, records the rule: name, never stop. A migration must not halt a deployment over data that a
screen is allowed to edit.
Day two: the faulty gesture was not faulty
The next morning Juste reported four things from the console. Three were about costs and prices. The last on the list was "errors on the API side in Easypanel, it looks like the migrations failed".
The fourth item was the outage, again. The health endpoint returned 502. The console and the public site returned 200. That is the most misleading form of downtime there is: every screen renders, and every call each screen makes fails. It is why it came fourth on his list and not first.
The chain behind it took a while to see whole:
- Creating a route from the console inserted its cost without ever writing where that cost came from. Every
- route born on a screen was silent about its source.
- A migration written two days earlier carried a guard that scanned the entire routes table: no real route
- may carry a cost without a source. Three lines above it, the migration's own comment described the opposite
- scope, a guard that "refuses what it produces, never what it finds elsewhere", the pattern four earlier
- migrations followed.
- Juste had created routes from the Routing screen, the normal way to use the platform.
- On the next deployment the guard counted them and raised. The startup script threw. The API restarted, ran the
- migration again, failed again. The log shows three identical passes: a restart loop, not a single crash.
Nobody did anything wrong in that sequence except the guard. Creating a route is the platform's ordinary job. The outage did not happen when the route was created. It happened at the next deploy, on another screen, to another person, with no visible link to the gesture that triggered it. A global guard inside a migration does not refuse an action. It refuses the startup.
And the remedy was already in the repository. About twenty lines from where that startup script threw, it carried the block written the day before: name, never stop. The conclusion from day one had not been carried across to a check of exactly the same shape. A fix does not spread to its neighbour by itself.
A report of mine that was false
My first report that morning said a route's cost could not be edited after creation. Wrong: a PATCH route had
existed since late July. My grep looked for app.patch( and the handler is written with a generic,
app.patch<{ Params: { id: string } }>(. The real defect was narrower. The card where the wrong cost is read first
is read-only, and the editor lives further down the same page. Its amount is now a button that opens the editor.
The second guard, and two months of a wrong number
The first push of the day bounded the scanning guard to the routes that migration itself writes. It got past that check. The same migration had a second guard, on amounts, and it took over: one of the routes it documented did not carry the amount the migration expected.
Juste ran a probe on the production database. The documented cost and the real one differed. No migration had ever written the real value. The cost-change journal named the author: Juste, two edits three minutes apart, in late July, on the console.
This is the most expensive lesson of the two days. The migration, written in September, had documented the value the chain had set, because its author (me) read the old migration that seeded it, not the row in the database. The real cost had been off from the documented one by 28% for nearly two months, and nothing had seen it: no review, no test, no gate. The only thing that detected it was a guard that took the platform down to say so.
A source written from the migration chain describes what the chain wrote, never what the database holds.
The two separate as soon as a screen can write the value, and in this case one had existed since July.
Whether that edited value is the right cost is still Juste's call. The two edits a few minutes apart look more like someone trying the interface than like a supplier invoice, but that is a reading, not a proof. Until he decides, the route stays without a recorded source, and that shows on the screen and in the startup log, which is the intended behaviour.
Holding the second push
What I am most glad about from day two is a push that did not happen.
With the service down and a fix ready for the first guard, the tempting move was to push again, see what the next error said, and iterate on production. I held the second push until we knew the cause. Pushing into a broken deploy loop would have added a variable to the diagnosis instead of removing one. The cause came from the Postgres log Juste read and from the probe he ran, not from another deploy.
The second push carried the real fixes. Each update in the migration now applies only where the cost still equals its documented amount, so a source is never written next to a number it does not describe. The amount guard became a warning that names the route, its documented amount and its real one. "One route diverges", the old message, had sent nobody anywhere during an hour of outage. The startup script gained a sweep for routes with no recorded source. And the source of a cost is now required when a route is created or edited, typed as a refusal at the API and enforced in the form, never pre-filled with the previous reason.
We deliberately added no table constraint. A CHECK refused by rows already in the database would have reproduced
that morning's outage exactly.
The service came back at 09:24 UTC on 22 September, proven on the target.
The probe that was right by accident
One more mistake belongs in this post because it nearly misled the reading. All session I probed /healthz,
/readyz and /v1/health. senndo's health route is /health. During the outage the proxy returned 502 on every
path, existing or not, so my probes gave the right verdict by accident. When the service came back, the same probe
returned 404: a perfectly healthy service, on a path that does not exist. It was the 404 that put me on the trail,
not the 200 I expected. A probe that aims beside the target gives the right answer while the outage is total, and
the wrong one the moment it ends.
What changed in how we work
Two habits came out of these two days, and both live in the project's decision log and in its code rather than only in a post.
A migration that guards data a screen can edit warns and names. It never stops the deployment. Stopping is for invariants the migration itself produces.
A migration that touches priced or edited data is replayed on a dump of the production database before the push, with the script from day one. The development database and the fresh-chain database are both useful, and both describe a world that production left the first time someone used the console.
Neither outage came from code that failed its tests. Both migrations behaved exactly as written on every database they were tested against. Those were two databases, measured carefully. Production was the third, and nothing was looking at it.