Captioning a Product Catalog with a Local Vision Model
Running an open vision model over 23,000 product images on local hardware. The model was the easy part; the deterministic envelope around it did the real work.
A large company’s product catalog arrived as a spreadsheet, about nineteen hundred products across the rows, and most of those products carried a set of images: a lead shot, packaging, size comparisons, nutrition and claims art, marketing renders. The images were referenced only by opaque content-delivery URLs, the kind that end in a random string and tell you nothing about what they point to. There were 23,434 of them. Nobody on the team could say what any particular image actually showed without opening it in a browser, and no one was going to open twenty-three thousand images. What the team needed was ordinary: a readable description of every image, keyed back to the product it belonged to, so the catalog could be searched and audited by content rather than by filename.
I decided to produce it by running an open vision model over the entire catalog on my own hardware. The build that resulted is a small, unglamorous piece of software, and it turned into a clean example of something I usually argue about in the abstract, which is how a deterministic system and a semantic one are supposed to fit together and how much of the real work lives at the seam between them. The model was the easy part. Making its output dependable enough to keep was most of the job.
The model was llama3.2-vision, an open model, run through Ollama on local hardware. Running it locally rather than calling a hosted vision API mattered for two reasons, and both were practical rather than ideological. The first was control over the data. The catalog was a real company’s proprietary product information, and running the model on my own machine meant no image and no row was ever sent to a third party. There was no external service in the loop to trust or to indemnify. The second was cost. Twenty-three thousand images through a hosted vision API, at whatever the per-image price happens to be, is a real bill and a rate limit to work around. On local hardware the marginal cost of describing one more image was electricity and time. What I paid instead was wall-clock time, since the work ran on a single worker at about eight and a half seconds per image, which is why the full run took days rather than hours. For a one-time internal inventory that was an easy trade, and the part worth noticing is that an open model on one machine was good enough to do the actual work, which was not obviously true a year or two ago.
Where the deterministic work ends and the semantic work begins
The task divides cleanly, and seeing the division is half of getting the design right. There is exactly one piece of work here that requires judgment over input no set of rules could enumerate: looking at an arbitrary product image and saying what is in it. That is what a vision model is for. Everything else in the pipeline is deterministic work with a single correct answer. Reading the spreadsheet and finding the image URLs in its cells, downloading each image, conditioning it into a form the model will accept, handing it over, taking the result back, checking it, caching it, and writing the descriptions out as structured files: none of that involves judgment, and all of it needs to happen the same way every time.
Neither side could do the other’s job. A rule-based system cannot describe a photograph it has never seen, which is why this problem waited for capable vision models to exist. And a model cannot be trusted to reliably walk 1,911 rows, resolve every URL, and write an identical file structure on every run, because doing the same mechanical thing correctly twenty-three thousand times in a row is precisely what deterministic code is good at and what a probabilistic system is not. The interesting decisions in the project were almost all about the deterministic side and the boundary it shares with the model.
Making an unreliable model dependable
The prompt was a single line: output a caption fragment of eight to twenty words describing what is visible, with no preface. The instruction to skip the preface was ignored constantly. The model liked to begin its answer with “The image shows.” When I measured this on a fixed set of 35 test images, the prompt I ended up shipping still produced that lead-in on 57% of them, twenty out of thirty-five. I could have kept editing the wording. Prompt-tuning against a probabilistic system is a game you do not finish, because the next batch of inputs finds a new way to misbehave and you are back to editing.
The durable fix was to stop trying to make the model comply and to correct its output deterministically instead. A small normalization step strips one leading boilerplate phrase, “this image shows,” “the image shows,” “there is,” “there are,” and a handful of relatives, whenever one is present. It runs on every caption, it always does the same thing, and it does not depend on whether the model followed instructions that run. This was the first lesson the build taught in concrete form: reliability came from constraining the model and correcting its output afterward, and wording the instruction more carefully did not produce it. An open model on local hardware is less obedient than a large hosted one, so the correction layer was a requirement rather than a refinement.
The harder failure mode was quieter. The model almost always returned something, and a clean return was not the same as a usable caption. Two problems mattered. The model would sometimes fall into a loop and emit a caption like “bottle bottle bottle bottle,” which is fluent, well-formed, and worthless. And it would sometimes describe the wrong thing with complete confidence, which is the failure no cheap check can catch. For the loops I added a deterministic guard: if a caption repeats a word pair three or more times, or if fewer than 45% of its words are unique, it is treated as degenerate and collapsed or discarded. That guard fired 144 times over the full run. Captions that came back empty, or began with “unclear,” or ran shorter than four words were treated as non-results and retried rather than saved. None of this is sophisticated. It is the deterministic side of the boundary refusing to accept input it cannot trust, which is the job that side exists to do.
The same principle chose the model and its settings in the first place. Rather than judge a configuration by whether it produced output, I ran a small harness that put a dozen configurations against the same images and scored each one on whether the output was actually usable: no repetition loops, no dumps of an entire nutrition label, no bulleted lists, and a caption of roughly six to twenty-five words. The configuration I shipped scored fully usable on that test, thirty-five out of thirty-five. Two other prompt styles that read perfectly reasonable on paper scored 57% and 43%, the second mostly by producing captions too short to carry any meaning. Judged by whether the model responded, all three would have looked fine. The distance between responding and answering is the entire reason the quality gate exists, and it is the measurement most worth building before you trust a semantic component at volume.
What made a run of days survivable
By the time the model call worked, it was a small part of the system. The catalog scan alone, reading the spreadsheet and pulling every image URL out of its cells, took about four hours, and the full run took roughly sixty-two, about two and a half days on one machine. A run that long will be interrupted, so the pipeline was built to resume. It keeps two caches: one for the expensive spreadsheet scan, fingerprinted so it is reused only when the spreadsheet has not changed, and one keyed by image URL that stores every caption and every error already produced. Restarting picks up where it left off and re-describes nothing. Ollama also rejects any image over eight megabytes, so before sending, the pipeline shrinks oversized images in steps, reducing dimensions and JPEG quality across up to six attempts until the image fits or is given up. Downloads retry with backoff. Every stage emits a structured event so the run can be watched while it happens and counted afterward. This is the part of an AI system that no demonstration shows and that determines whether the thing actually finishes.
The failures were revealing in a way I had not planned for. Of the 23,434 images, the first pass described 23,394 and failed on 40, a rate of about 0.17%, roughly two in a thousand. When I looked at those 40, none of them were the model’s fault. Twenty-six were images so large they exceeded the eight-megabyte limit even after being shrunk. Twelve were dead links that returned a 404. One was a server error, and one was a corrupt file. Every one of those was a problem with the catalog rather than with the captioning: broken image references and oversized assets sitting in a production product catalog that nobody had audited, surfaced as a side effect of trying to look at every image once. The captioning job doubled as an integrity check on the data it was captioning. That dividend is common when you push real data through a careful pipeline, and it appears only when the deterministic side records its failures instead of swallowing them.
What it means to put a model to work
Two things from this project generalize to almost any similar decision. The first is that an open model running on your own hardware is now a real option for bulk internal work, and for proprietary data the case is strong, because nothing leaves the building and the marginal cost of another item is close to nothing. What you pay is wall-clock time, which for internal work that is not urgent is usually affordable. Teams that default to a hosted frontier model for everything should at least price the local option, especially for high-volume, low-stakes work over data they would rather not send out.
The second is more important. The model was the smallest decision in the project. The decisions that actually determined whether the output was worth keeping all sat on the deterministic side of the boundary: what to reject, how to normalize, when to retry, how to resume a run measured in days, and how to tell a usable answer from a well-formed useless one. A better model would have written better captions. It would not have made any of those decisions for me, and it would not have removed the need to make them. When people ask where to put a model in a workflow, placing it is the easy half of the answer. The work is building the deterministic envelope around it that turns an unreliable component into something the rest of the system can depend on, and that envelope, not the model, is what you own.