Synthesis over Storage: Why We Rebuilt an Agent's Knowledge Instead of Updating Its Documents

The request that started this work was routine. A set of commerce agents had been built the year before for major US retailers, and the capability lead asked us to update them. The agents’ knowledge bases had been assembled the way most agent knowledge bases are assembled today: raw source documents, decks, PDFs, and transcripts, loaded directly into each agent’s private store with no processing in between. Updating them should have meant swapping stale documents for current ones. We rebuilt the knowledge instead, and the reasoning behind that decision is worth walking through, because the instinct we declined to follow is the default instinct almost everywhere right now.

The default model for making an agent knowledgeable is access. Connect the document store, load the knowledge base, add retrieval, and let coverage do the rest. It feels safe because nothing is lost: every document the organization has is available to the agent, so any answer it might need is in there somewhere. The problem is that an agent is only as good as the knowledge engineering behind it, and access is not knowledge engineering. When we looked honestly at how the accumulated version behaved, it was failing on cost, accuracy, and speed at the same time, and all three failures traced to the same structural cause.

The cost failure is the easiest to see. AI systems are billed by the volume of text they process, so when an agent reasons over raw documents, every answer pays to move overlapping material that was never written to be read by a machine. The accuracy failure is quieter. Our corpus contained multiple decks stating different numbers for the same metric, and the agent surfaced whichever fragment retrieval happened to match, producing inconsistent answers with no clean way to trace a claim back to its source. The speed failure follows from the first two: bigger prompts and noisier retrieval meant slower responses and more tool round trips. Underneath all three is a hard constraint. Raw documents are far larger than any model’s usable context window, so long raw passages crowded out the room the agent needed to actually reason.

The structural cause is a coupling. In the accumulation model, cost and quality scale with document bloat rather than with the value of the answer. Add a hundred more decks and every query gets more expensive and every answer gets noisier, whether or not those decks improve a single response. That coupling does not improve with growth; growth is what makes it worse. Any approach that leaves the coupling in place only defers the failure.

What we built instead

We treated the update as an architecture problem. In place of a document refresh, we built a synthesis pipeline: a repeatable process that distills the raw corpus into deduplicated, conflict-aware knowledge articles and moves the results into a shared enterprise knowledge store where other applications can use them too. For the first brand, the raw corpus was roughly 580 megabytes across 120 files. The pipeline reduced it to 41 knowledge articles totaling about half a megabyte, a measured reduction of a little over a thousand to one. The synthesized knowledge base is under one tenth of one percent of the raw volume, and it answers better than the pile it replaced, because it was engineered to answer rather than stored to be searched.

The reduction is a consequence, and the mechanism matters more than the ratio. Synthesis moves the reconciliation work from answer time to build time. In the accumulated model, the agent confronts duplicates and contradictions on every question, invisibly, and resolves them for reasons no one can reconstruct. In the synthesized model that work is done once, under rules, with a record. Overlapping sources are merged. Where two sources conflict on the same fact, deterministic preference rules decide which one wins, and when the rules cannot decide, the conflict is recorded openly with both claims and both sources rather than silently resolved. Every article remains traceable to the source documents it came from, and the traceability is carried by deterministic machinery rather than by an instruction to a model, because an instruction is a request and machinery is a guarantee. That lineage survives all the way to the end user: every answer the production agent gives ends with the list of source documents behind it. When the agent lacks the information, it says so rather than inventing an answer.

Getting there takes more machinery than it sounds like it should. The corpus is far larger than any context window, so the pipeline runs as one orchestrating process that never reads a document itself. It delegates the reading and judgment work to batches of subordinate agents and keeps everything with a single correct answer, the hashing, the change detection, the lineage, in deterministic code. Because the destination is shared production with no undo, writes are governed as carefully as answers: every write is permanent, each one is verified before and after it lands, and a human authorizes the steps that cannot be walked back. Each of those layers has enough depth to deserve its own treatment, and I plan to give them that separately. The point of the map is the shape: raw source material goes in one end, and governed, deduplicated, traceable knowledge comes out the other.

The honest objections deserve a direct response. The first is effort: a synthesis pipeline is real engineering, and connecting a document store is an afternoon. But the comparison is between a one-time build cost and a coupling that taxes every query for the life of the system. Because the pipeline tracks what changed, a changed source re-derives only what depends on it, so the build cost is paid in full once and incrementally after that. The second objection is that synthesis could lose detail or introduce its own errors. It can, which is why the process carries recorded conflict rules, content-hashed lineage, and human gates ahead of the steps that cannot be undone, rather than a bare instruction to a model to “be careful”. The version of the objection I do not accept is that model progress will make the problem irrelevant. Larger context windows make it cheaper to do the wrong thing at greater volume. They do not decouple cost from bloat, and they do not give an accumulated pile a lineage it never had.

What this changes for leaders

The operating implication is a change in what you budget for and what you measure. Treat an agent’s knowledge base as engineered infrastructure, with a build pipeline, provenance, and write discipline, rather than as a drive the agent is pointed at. When someone proposes an agent over enterprise content, ask what the knowledge engineering behind it actually is. If the answer is a connector and an index, expect the cost, accuracy, and speed problems to arrive together as usage grows, because they share a cause. And judge the system by the cost of a trustworthy answer rather than by how much content the agent can reach. Reach measures input. The value is in the output, and in whether anyone can trace how the output was made.

What stays with me from this build is how ordinary the original request was. Update the agents. Most organizations deploying agents over their own knowledge will hear that same request within a year or two, and the ones that treat it as a content refresh will keep paying the coupling, with cost and error scaling alongside a pile that only grows. The durable version of an agent program is the one where knowledge is made rather than accumulated: synthesized under rules, traceable by construction, and written to production under governance. The first rebuilt agent now answers from a knowledge base roughly a thousandth the size of the corpus behind it, and it is better for it. That is what knowledge engineering buys, and it is the part of the agent stack that deserves the investment.