The Orchestrator That Never Reads a Document: Running Agent Knowledge Work Beyond the Context Window

In the first piece in this series I described rebuilding a set of commerce agents’ knowledge for major US retailers: a synthesis pipeline that distilled hundreds of megabytes of raw source documents into a small, governed, traceable knowledge base. This piece is about the engineering problem underneath that description. The corpus was far larger than any model’s context window, and no amount of clever prompting changes that math. The design finding we recorded partway through the build has stayed with me since: the binding constraint of the system was context rather than compute. We were never short of model capacity. The constraint we kept hitting was room for a model to reason well, and the whole architecture followed from taking that constraint seriously.

The common response to that constraint is to wait it out. Context windows have grown every year, so teams treat the limit as a temporary model problem: buy time with retrieval, and the next generation will make the problem irrelevant. The math does not support the patience: a window ten times larger still does not hold this corpus, and a corpus is not a fixed quantity; new decks, transcripts, and spreadsheets arrive throughout the year, and the roster here runs to nineteen brands processed one at a time. More important, fitting everything into one context was never the actual goal. Even if the documents fit, a model reasoning over hundreds of megabytes of overlapping raw material would still pay the cost, accuracy, and speed penalties that motivated the rebuild in the first place. The architectural question is how to coordinate knowledge work that will never sit in one place at one time, and that question does not get easier by waiting.

The answer we landed on is an economy of context, designed as deliberately as any budget. The pipeline runs as a single deterministic orchestrating process, and that orchestrator never reads a document. It never reads a source file, a converted page, a topic, or a sub-chapter body. The content lives on disk, and disk is the coordination layer. When a piece of judgment work needs doing, the orchestrator writes a small dispatch file naming one unit of work, hands it to a model sub-agent, and moves on. The sub-agent reads what it needs from disk, writes its result back to disk, and reports completion as a one-line status message in a fixed grammar. The orchestrator reads only the status lines. Batches are capped so that no more than one batch of sub-agents is live at a time, which keeps failures containable and the run legible. A single brand’s run dispatched roughly 244 sub-agents this way. The coordinating context stays small for the entire run because it carries coordination and never content, and that is what lets one process drive a corpus of arbitrary size.

Underneath the orchestration is a split we enforced across all eleven stages of the pipeline, from preflight and raw-source scanning through conversion, topic synthesis, tagging, and governed ingest. Deterministic code does everything with a single correct answer: hashing, diffing, ledger reads and writes, computing the work plan, generating the dispatch files, running the serial ingest loop. Model sub-agents do only the work that requires judgment, and each one does exactly one unit of it. The split is a case-by-case engineering decision rather than a slogan, and the sharpest example in the build is spreadsheet conversion, which is deterministic code with no model call at all. The recorded rationale was that there is no reason to involve a model that could invent numbers. In a job that consists of copying numbers, that is the entire argument.

Paying for correctness

Where models are involved, the routing to them is itself deterministic, and it is biased in a direction that looks wasteful until you consider the failure modes. Conversion routes are chosen by fixed classification thresholds, characters per page and raster coverage, and the bias runs toward the slower, more expensive vision route. The reasoning is an asymmetry: a wrong fast-lane decision silently loses content, while a wrong vision decision merely costs a bit more. We chose to overpay a little, systematically, to avoid losing content invisibly. Oversized documents are decomposed page by page, so a PDF of more than 300 pages becomes 300 small conversions stitched back together in order. A standalone conversion run processed 199 files in roughly three and a half hours with a median around 29 seconds per file, which is the throughput you get when the units are small and nothing in the loop waits on a person.

The small units are a correctness decision as much as a context decision. Early in the build we asked a vision model to transcribe an image-heavy deck in one pass, and it confabulated: plausible bullet points, fabricated metrics, and a generic category list standing in for the real content of three slides, all delivered in prose smooth enough to pass a casual review. The same slides converted per page, under a transcribe-verbatim prompt that forbade inference, came back exact, down to a typo preserved from the source. Both runs used the same class of model. What differed was the size of the unit and the room the model had to drift. A model given one page and a narrow instruction has nowhere to hide an invention. A model given forty slides and a summary-shaped task will produce something summary-shaped whether or not the content supports it. Per-page conversion became a fixed requirement of the design after that, and the general lesson holds anywhere models do judgment work: unit scope is a quality control, and it is one you set in architecture rather than in a prompt.

The honest objection is that this is a lot of machinery. A long-context model with retrieval over a document store can be stood up in an afternoon, and what I have described is an orchestrator, a dispatch protocol, a status grammar, and a few hundred sub-agent invocations per brand. Two things make the machinery worth it. The first is that at this size the simple alternative does not exist; there is no window to fall back to, so the real choice is between designed coordination and improvised coordination, and improvised coordination across hundreds of units is exactly where content gets lost without anyone noticing. The second is that the machinery is what makes the run repeatable and incremental. Because the orchestrator is deterministic and every unit of work is small and recorded, a failed unit can be retried without restarting the run; how the pipeline handles a changed source belongs to the provenance machinery the next piece covers. The fragility people expect from hundreds of dispatched agents comes from designs where agents converse with each other and accumulate shared state. These do neither. Each one reads from disk, performs one unit of judgment, writes to disk, and ends.

What to ask of an agent architecture

The operating implication for leaders reviewing an agent proposal is a design question that rarely gets asked: where is the context economy? If the answer is that the coordinating process holds the content it coordinates, or that every agent reads everything relevant, the system has a scaling wall built into it, and the wall arrives as the corpus grows rather than announcing itself in a demo. The demo corpus always fits, and the production corpus stops fitting quickly. The durable pattern is the one this build forced on us: content on disk rather than in the coordinating context, judgment done in small bounded units under narrow instructions, and everything with a single correct answer in deterministic code, where it can be tested, rerun, and trusted. The machinery that carries provenance through this pipeline, and the discipline that governs writing its output into a shared production store, each deserve their own treatment, and they are the subjects of the next two pieces.

What stays with me from this part of the build is how much the constraint clarified the design. Treating context as a budget forced every decision the architecture got right: the orchestrator that never reads a document, the one-unit sub-agents, the deterministic spine underneath the judgment work. Larger context windows will keep arriving, and they will make individual units of work bigger and cheaper. They will not remove the need for the economy, because the corpus grows too, and because the room a model needs to reason well is worth protecting at any window size. Context is the resource an agent system actually runs on. The systems that treat it that way, and design its economy on purpose, are the ones that will still be running when the corpus has doubled twice.