Failures That Look Like Success: Confabulation, Exit Codes, and Engineered Refusal

Early in the rebuild I have been describing in this series, the synthesis pipeline that replaced raw document piles with small governed knowledge bases for major US retailers, we asked a vision model to transcribe an image-heavy deck in one pass. The output came back clean. It read the way a good transcription reads: confident bullet points, specific metrics, a structure that matched the deck’s rhythm. It was also partly invented. The model had fabricated plausible bullets, made up metrics, and replaced the real content of three slides with a generic category list. Nothing in the output marked where transcription stopped and invention began. It read smoothly, and the smoothness is what made it dangerous.

We caught it because we went looking. The counterpart test converted the same deck page by page at high resolution, under a prompt that required verbatim transcription and forbade inference, and it reproduced all three slides exactly, down to a typo preserved from the source. The comparison settled two things at once. It established what the deck actually said, and it established that the first output’s fluency had carried no information about its fidelity. The design consequence was structural: per-page conversion became a fixed requirement of the pipeline, for the unit-scope reasons the second piece in this series covers. What stayed with me was the shape of the failure more than the fix. Had we not held the output against ground truth, the invented metrics would have flowed into synthesis, into the knowledge base, and eventually into a production answer with a source list attached, carrying the full authority of a system built to be traceable.

This closing piece of the series collects what the build taught us about failure, because the hardest lessons shared a property worth naming: nothing crashed. Conventional software trains an instinct that failure announces itself, in an exception, a stack trace, a red build, a nonzero exit. Verification built on that instinct watches for visible breakage. In this build, the failure modes that mattered produced smooth prose, clean exit codes, and confident answers, and a verification posture aimed at breakage would have missed every one of them. The conclusion we kept arriving at, in different parts of the system, is that a trustworthy agent system has to be engineered on the assumption that its most convincing signals are the ones most in need of verification.

Exit codes, in both directions

The clearest demonstration came from ordinary tooling rather than from a model. An upstream seed parser was written to match a format that its own companion tool never writes. It matched zero lines, returned an empty result, and exited 0. Undetected, that empty result would have led the pipeline to re-identify every topic as new and orphan the live articles in production. The build’s records call the failure silent by construction, and the phrase is precise: every component behaved as written, the composition was wrong, and no signal in the chain was capable of noticing. The converter proved the same point from the other side. It writes failed-page markers into its own output and still exits 0, and it can fail report generation entirely, leaving an exit 0 with no report at all. An exit code asserts that a process ended. Everything beyond that is inference, and in this build the inference failed in both directions.

The driver therefore branches on the machine-readable run report, never on the exit code. The report records what actually happened inside the run, and it is the artifact the next stage’s decisions are based on. But the reports earned their own scrutiny. After live runs, the auto-generated reports needed hand-written corrections; one counted 417 errors that turned out to be logging artifacts of an append-only design, with zero real failures behind them. The two fixes point the same way. A success signal is only as good as the machinery that produces it, and reading a signal correctly requires knowing how it is made: an exit 0 can hide real failure, and an error count can manufacture failure that never happened. The operating habit the build settled into is layered. Trust reports over exit codes, and evidence over reports.

Refusal at the edge of knowledge

The same posture had to be engineered into the product, because the end-user version of the confabulated slide is a fluent answer to a question the knowledge base cannot support. The production agent is read-only. It runs with web search off, it answers only from its synthesized knowledge base, and when the information is absent it states that it does not have it rather than speculating. None of that behavior is assumed from model quality. The refusal is configuration and prompt contract, and it is observed as live behavior rather than taken on faith, for the same reason the driver reads run reports: a behavior the system depends on needs evidence, and a behavior that only exists as an expectation will eventually be missing exactly when it matters. The pipeline applies the same standard to the models it employs internally. When a model hard-refuses a conversion task, the refusal is recorded as a first-class failure class and never ships silently. Declining to answer, and keeping a record of the decline, is what honesty looks like at the edge of a system’s knowledge.

The fair objections deserve direct answers. The first is that three incidents from one build are anecdotes rather than data, and that is true; I present them as my own operating experience, not as industry statistics. What generalizes is the class they belong to. Any system that produces fluent output can fail fluently, and the incidents you catch are the only evidence you will ever have about the ones you would not have caught. The second is cost, and the machinery is real: per-page conversion pays more per document, report verification is engineering, and the corrections were written by a person. Against that stands the cost of the alternative, which is failure that is invisible by design, discovered downstream in a production answer or an orphaned knowledge base, where no re-run repairs the damage cheaply. The third is that a refusing agent frustrates users who expect an answer. Occasionally it does. But a wrong answer delivered with a confident source list does deeper damage than a declined one, because it spends the credibility that the whole knowledge-engineering investment exists to build. The fourth is that newer models confabulate less, which is also true and changes nothing about the design. A system that assumes model quality has no way of noticing when the assumption fails, and the verification that looks redundant around a good model is the only instrument that can tell you the model is good.

For leaders operating agent systems, the lessons convert into questions that require no code review. Ask what evidence stands behind the system’s success signals: when a pipeline reports a clean run, ask what produced the report and what it would have looked like if the run had failed silently. Ask what the system does at the edge of its knowledge, and whether anyone has observed it decline rather than guess. And ask where the verification effort is pointed, because the instinct to double-check the ragged output and wave through the polished one is exactly backwards in these systems. Fluency and fidelity vary independently, and the most convincing output deserves the most scrutiny precisely because it is the one nobody else will challenge.

This series began with a knowledge base that could not be trusted because it had been accumulated rather than engineered, and it ends here deliberately. The provenance ledger, the recorded conflict rules, the governed write path, and the engineered refusal are one posture applied at different points in the system: none of it takes the system’s own output on faith. The build’s most useful discovery was the standard all of those mechanisms serve. A system that can surface its own failures is trustworthy. A system that has merely not failed visibly is untested, and the difference between those two conditions is everything this series has described.