AI coding agents are good at producing motion. They inspect a repository, edit files, run a test, and return a confident summary. The hard problem begins when that activity becomes a claim: this change is delivered.
“Done” sounds harmless in a chat window. In a software team, it carries hidden commitments. The request was understood. Blocking decisions were answered. The change was isolated. The test failed for the intended reason before the fix, passed after it, and still caught the bug when the fix was removed. Review covered the whole change, not only the latest diff. Checks that could not run were named rather than blurred into success.
More agents do not make those commitments true. They make it possible to produce unsupported completion claims faster.
A factory needs a delivery contract
A useful software factory can isolate work, build changes in parallel, collect before-and-after evidence, and send the result through review. Those capabilities improve throughput and make agent work easier to inspect.
They still leave one question unanswered: what must be true before a run may call itself delivered?
Factory treats that as an executable contract rather than a sentence in a prompt. A run carries testable acceptance criteria, open decisions, proof receipts, review outcomes, and a requested terminal state. The gate reads that record and can refuse the terminal claim.
The failure that became the first gate
The failure shape behind Factory was ordinary enough to be dangerous. A run reached delivered while five blocking questions remained unanswered. Two acceptance criteria were marked partly met, a phrase that sounded encouraging but had no enforceable meaning.
The prose rules already said not to do this. The run did it anyway.
Factory now accepts only three acceptance-criterion results: met, unrunnable, or failed. An unanswered blocking question prevents delivery. An unrunnable check is not silently promoted to success; it requires a named gap and explicit acceptance before the run can use delivered-with-gaps.
[false-delivery] exit=1
GATE: BLOCKED
5 blocking questions unanswered
2 acceptance criteria invalid“Blocked” is a product feature
A reliable agent needs more than success and failure. Factory uses four explicit terminal states:
delivered: every acceptance criterion was proven and the terminal gate passed.delivered-with-gaps: named checks could not run and the user explicitly accepted those gaps.blocked: a missing fact, unanswered decision, or exhausted bounded loop prevents correct delivery.intentionally-unchanged: diagnosis showed that changing code was not the right treatment.
These states prevent a common collapse of meaning. “I changed something,” “I could not verify something,” and “nothing should change” are different outcomes. Treating all three as done makes the human reconstruct the truth from a long transcript.
The gate must be allowed to fail too
An executable gate is not automatically trustworthy. It is software, and software can encode the wrong rule.
While preparing Factory’s public proof, the gate exposed its own defect. It validated the terminal requested on the command line but did not compare that request with the terminal already recorded in the run state. A caller could ask it to validate delivered against a run recorded as blocked. If the other checks were clean, the old gate returned PASS.
The terminal-state mismatch case study preserves the baseline output, the regression, and the fixed refusal. That bug is part of the product story, not an embarrassment edited out of it. A quality system earns trust by making its claims falsifiable—including claims about itself.
Proof should survive the demo
A polished video can make almost any agent workflow look reliable. The more useful standard is whether another engineer can inspect the record after the presentation ends.
- The acceptance matrix maps each requirement to evidence, a named gap, or failure.
- Receipts preserve the command, result, and commit outside compact carried state.
- The test must fail before the fix, pass after it, and fail again when the fix is removed.
- Milestone review and whole-change review cover different failure scopes.
- The terminal gate emits the exact reasons a completion claim is accepted or refused.
The unfair advantage is honest refusal
Model capability will keep improving. Agent orchestration will become easier to copy. Neither removes the need to decide what counts as delivered.
The durable advantage is a workflow built from real failure modes and converted into executable constraints: unanswered decisions, tests without teeth, self-approved permission fallbacks, unbounded retries, oversized state, and locally clean milestones that break a branch-wide invariant.
Factory is not another conveyor belt for producing code. It is the quality system beneath the conveyor belt. Its job is to make delivery claims inspectable and to refuse the ones the evidence does not support.
Your coding agent does not need permission to sound confident. It needs the right to say exactly why the work is not delivered.