A production hotfix drill in nine minutes of machine time

On 2026-07-17 the Software Factory ran its first production-down drill and closed it in about 9 minutes (measured) of machine time. Drill is the operative word: the incident was seeded on true-crime-ranker's local production simulation, nothing was pushed, and no user ever saw it. The seeded bug was the kind that does not page anyone. A half-renamed sort key made every ranking sort on a constant 0.0, no exception was thrown, and 1 test failed (measured).

Before

There is no honest baseline number here yet. The comparison is a human noticing a silent bad deploy, reproducing it, tracing it to the half-rename, writing a regression test, and shipping the patch. Nobody timed that here and no ledger records it, so the before row reads about 2 hours (my estimate) until Noah puts his own number on it. Inventing a tidy "three hours" would make every other number in this post worth less.

What we did

factory-hotfix (sonnet) ran first, in DIAGNOSE mode. It reproduced the failure with captured output, localized it to the exact commit and lines, checked the frontend contract before choosing a revert over finishing the rename, and wrote the risk, the rollback, and a named debt ticket for the missing contract test. It edited no product code and hard stopped at the human approval gate. DIAGNOSE took 125 seconds (measured).

The approval was mine, relayed and recorded in the run ledger with an approval ref. Then factory-hotfix switched to RACE mode and three racers implemented the approved approach in parallel, one git worktree each, with ruff and pytest running as gates inside every worktree. Race plus adjudication took 423 seconds (measured).

factory-gatekeeper (opus) closed it out. Racer claims carry no weight at that node, so it re-ran the evidence itself and proved the winning regression test fails on the pre-fix code and passes on the fix. Verdict GO, 0 blockers, 2 documented non-blockers with real approval refs (measured). The winner merged to the local prod branch and the simulation came back at 9/9 tests green (measured).

The factory is built with Claude, Claude Code, MCPs, Skills, and Agents. This lane used a Claude Code Workflow tool script, the two agents above, git worktrees, the ruff and pytest gates on the true-crime-ranker tenant, and model tiering by node: haiku running gates, sonnet racing, opus gating.

After

Detection to validated fix was 548 seconds of machine time, 125 s to diagnose plus 423 s to race and adjudicate, about 9 minutes (measured). The human approval gate sits between those two numbers and is not counted in them. The drill also generated its own follow-up: the debt ticket the diagnosis named, the missing sort-key contract test, shipped four days later as true-crime-ranker PR #4, 2 files, +65/-3, 13/13 tests, merged 2026-07-21 (measured).

What broke

All three racers returned the identical 2-line fix (measured). The diversity landed in the tests, not the code, so the race bought certainty rather than speed. One racer would have produced the same patch.

True first-to-green was never implemented. The race used a barrier and then picked the winner by finish order plus authoritative gates, so "first green wins" is still a design note, not running code.

The M2 race script never entered the pushed repo, and the run ledger is gitignored and machine-local, so the 548 seconds cannot be re-derived from a ledger on another machine. Everything above traces to the drill results file, which is why that file is named in the receipt.

Still manual, by design: the approval gate between diagnosis and race is a human and stays that way. So is the merge to real production, because the factory never merges or pushes. Filing the intake, a GitHub issue with a factory:hotfix label, is manual, and so is the cron wiring for unattended runs.

What you can copy

• ›Split the incident lane into two nodes: a diagnosis agent that is forbidden from touching product code, and a build step that only starts after a human approves the approach. The hard stop is what makes the speed safe to keep.
• ›Make the gate re-run the evidence instead of reading the builder's summary. Proving the regression test fails before the fix and passes after it is the only reason the 9 minutes means anything.
• ›Skip parallel racers until you have evidence they diverge. Three identical patches is a bill, not a result. Spend that budget on the gate.

A receipt, not a case study: NSA Designs publishes the gap in the same post as the number. If you want this lane on your own stack, the Readiness Sprint is where it starts.