Trust as Machinery: Provenance, Conflict Resolution, and Citations That Survive to the End User

Every agent deployment eventually produces the same meeting. An answer the agent gave, a number, a policy, a claim about how something works, disagrees with what someone in the room believes, and the conversation turns from what the agent said to why it said it. That question is reconstructive: which sources produced this claim, what happened to them on the way to the answer, and what decided that this version was the one worth stating. In the deployments I have seen, the honest response is that nobody can say. The model read whatever retrieval surfaced and produced a sentence, and the path from source to statement dissolved the moment the answer was rendered. In the rebuild I have been describing in this series, a synthesis pipeline that distills raw document piles into small governed knowledge bases for major US retailers, we treated that question as a design requirement rather than a support escalation. This piece is about the machinery that answers it.

The instinct most teams follow is to treat provenance as a prompting problem. The system prompt gets a bullet: always cite your sources. In a demo the instruction appears to work, because the model produces something citation-shaped often enough to pass review. The build’s own records state the difficulty plainly. The question on the table was whether the model writing each knowledge article should maintain its own sources section, and the recorded answer is that an instruction to a model is a bullet in a prompt that nothing enforces. The model may comply, will usually comply, and occasionally will not, and the system contains no mechanism that notices the difference. Retrieval systems that display their retrieved passages carry a quieter version of the same gap. The citation points at what was available to the model rather than at what the model actually used, and the model remains free to blend, drop, or embellish between the retrieved text and the rendered answer. If trust in the system has to survive an audit, a legal question, or an executive who wants to know where a number came from, traceability cannot live in a request. Whatever must be provable has to be carried by machinery.

In this build, the machinery is a per-brand lineage ledger, and it is the spine of the pipeline rather than a reporting layer added afterward. The ledger is a content-addressed build graph. Every artifact at every stage, the raw source file, the converted document, the identified topic, the synthesized article, is a node, and every node carries a content hash of what it is, an inputs hash of what it was made from, and derivation edges pointing back at the nodes it came from. Provenance in this design is a property of the data structure. The work cannot be performed without leaving the record, because the record is how the pipeline knows what work exists.

The decision to key identity on content hashes rather than timestamps sounds like an implementation detail until you consider how routinely timestamps lie. The recorded rationale in the build is blunt: a bulk zip extract destroys the modification times of every file it touches without corrupting change detection for a system that only trusts the bytes. The same discipline points inward at the pipeline itself. Each node’s recipe pins the model that ran, the version of the prompt it ran with, and a content hash of the tool tree, so a locally patched script cannot quietly produce artifacts attributed to the code in the repository. When someone asks what produced a given article, the ledger’s answer is exact: these sources, through these stages, by this code, under this prompt, with this model.

The graph earns its keep when the corpus changes, which is the normal condition of an enterprise corpus. A changed source re-derives only its affected descendants, so the routine update touches a fraction of the pipeline’s work. A removed source tombstones its dependents all the way down to the published article. Incremental rebuild and honest retraction fall out of the same structure, and the retraction half matters more than it first appears. When a source turns out to be wrong, withdrawn, or superseded, the question of which published knowledge depended on it has a computed answer rather than an investigated one.

When sources disagree

Enterprise corpora disagree with themselves. The first piece in this series described the symptom at answer time: multiple documents stating different numbers for the same metric, with the agent surfacing whichever fragment retrieval happened to match. Synthesis moves that reconciliation to build time, which raises the question of how the reconciliation decides. The answer in this build is deliberately unexciting: recorded preference rules rather than free model judgment. A date found in the document body beats a date inferred from the filename, which beats no date at all, and between dated sources the more recent one wins. When two conflicting sources carry the same effective signal, the pipeline stops pretending to know. The conflict is marked unresolved, and both claims with both sources are recorded in a dedicated conflicts section of the article, visible to anyone who reads it. The recorded purpose of the mechanism is to replace a situation nobody can defend, where the model saw conflicting documents and chose one for reasons no one can reconstruct, with a decision that can be audited: this claim won because of this rule, or no claim won and here is what we know.

The objection I take most seriously is that fixed rules look rigid next to a capable model weighing sources in context. A model can weigh considerations the rules do not encode: which document reads as more authoritative, which phrasing suggests a draft rather than a final. The difficulty is that this weighing happens again on every run, invisibly, leaves no record, and can change its conclusions with the next model version. A recorded rule can be wrong, but it is wrong in public, the same way every time, and correctable in one place. The unresolved category, meanwhile, does the work people hope judgment would do. Where the rules cannot decide, the system shows both sides rather than manufacturing confidence. A knowledge base earns trust by showing its reasoning, and it can only show reasoning it actually has.

The same principle carries citations through to the person reading the answer. Source lists in the published articles are never maintained by the model that writes the prose. They are rebuilt deterministically from the ledger’s derivation edges at ingest time, immediately before an article moves into the shared enterprise knowledge store. The effect shows up at the last mile, which is where provenance usually dies: every answer the production agent returns closes with a source-document list naming the knowledge articles that contributed and the underlying source files behind them. The reader does not have to trust the pipeline. They can see the documents standing behind the claim, and they can go look.

One boundary has to be stated plainly, because overstating traceability is its own failure. Attribution in this system is at article level. The ledger knows which sources produced an article; it has no per-claim citation edge, so it cannot prove which individual sentence came from which individual document. The build records that limit explicitly rather than implying finer granularity, and I would hold any system to the same standard. Some audit regimes will want per-claim attribution, and for them article-level lineage is the foundation to build on rather than the finished answer. What a trust layer must never do is imply precision it does not have. A provenance system that overstates its own coverage has reproduced, inside the machinery meant to establish trust, exactly the confident-but-unverifiable behavior it was built to eliminate.

What to ask of a system you are asked to trust

The question I now ask of any agent proposal is what carries its traceability. If the answer is a prompt, the system has trust by request, and the request will be honored until the one occasion when it matters that it was not. The follow-up questions are concrete and require no code review. Is provenance recorded as data while the work happens, or reconstructed as prose after the fact? When sources conflict, can anyone state the rule that decided the winner, and where do the undecided cases go? Are the citations shown to the end user built from that record, or written by the same model whose output they are supposed to verify? Is the granularity of the attribution stated honestly? The answers separate systems that can survive a challenge from systems that can only survive a demo.

The infrastructure objection, that a content-addressed ledger is heavy machinery for a content pipeline, understates how much of the machinery is shared. The graph that carries provenance is the same graph that makes rebuilds incremental and retraction computable. The pipeline needed it to operate, and trust rides on structure the operation already required. I would defend the machinery even if it were pure overhead, because the alternative is an agent whose answers cannot be defended in the meeting where it counts.

There is one more layer to the chain: the knowledge this pipeline produces still has to be written into a shared production store, and the discipline governing those writes is the subject of the next piece. The durable consequence of this layer is a change in what an answer is. From an accumulated pile, an answer is a plausible synthesis whose origins dissolved as it was rendered. From this machinery, an answer arrives with its lineage attached: sources named, conflicts resolved by rules or shown openly, every step from document to claim reconstructable. That is what trust means in an agent system, and none of it can be requested from a model. It has to be built.