A staged multi-model pipeline: 96% accuracy, 4x faster, new models swapped in an afternoon

The brief: A contract-intelligence SaaS — the kind that reads commercial agreements and returns structured data their customers’ legal and procurement teams act on — had hit a ceiling. Their whole product ran on a single large frontier model behind one prompt. It was accurate enough to sell and too slow and too expensive to scale. Field-extraction accuracy plateaued at 82%, the p95 response time on a long contract was nine seconds, and the model bill was their single largest cost line. Worse, every time a better model shipped, swapping it in meant a fortnight of re-prompting and the quiet terror that something would regress in production with no way to know until a customer complained. They asked us to make the pipeline faster, more accurate, and — their words — “able to adopt the next model without holding our breath.”

What the readiness audit found

Two weeks across their ML, product, and customer-success teams. The core finding was structural: they were using one expensive, slow, general-purpose model to do four jobs that wanted four different tools. Roughly 70% of the work was easy classification and extraction that a small fast model could do at a fraction of the cost and latency. About 20% was genuine ambiguity — unusual clauses, conflicting terms — that genuinely needed a frontier model. And nothing in the pipeline checked its own work, so the errors that did happen reached the customer unflagged. They also had a gold-dust asset they were not using: tens of thousands of human-reviewed, corrected contracts sitting in their database.

What we built: one pipeline, four model roles

We rebuilt the single-model pipeline as four stages, each served by the model that actually fits the job, behind a single routing layer:

• Stage 1 — triage and classification (small fast model). Every incoming document is classified by type, jurisdiction, and complexity in well under a second, and routed accordingly. A standard NDA never touches the expensive model; a 90-page bespoke master agreement gets flagged for the heavy path. This one decision removed most of the cost and most of the latency.
• Stage 2 — structured extraction (a fine-tuned small model). We used their corpus of human-corrected contracts to fine-tune a small model for the high-volume common case — the parties, dates, values, renewal and termination terms that make up the bulk of every extraction. On their own data it matched the frontier model’s accuracy at a tenth of the latency and cost.
• Stage 3 — ambiguity reasoning (frontier model). The genuinely hard 20% — unusual clauses, internal contradictions, anything the extractor flagged as low-confidence — is escalated to a frontier model with the full context assembled. It only runs when it is actually needed, so its cost is bounded and its quality is reserved for the cases that earn it.
• Stage 4 — validation (a separate model as critic). A different model, on a different prompt, checks the assembled output against the source document and flags disagreements for human review rather than shipping them silently. Using a distinct model as the critic catches a class of error a model never finds when grading its own work.

A dedicated embedding model underneath the whole pipeline handles retrieval — pulling the relevant prior clauses and the customer’s own playbook into context at each stage. Five model roles in total, each chosen for its stage, none of them load-bearing for the others.

The part they actually asked for: plug-and-play models

None of the four stages calls a model directly. Each calls an internal routing layer with a provider-agnostic tool definition, so the model behind any stage is a configuration value, not a code dependency. Swapping the frontier model in Stage 3, or trying a new small model in Stage 2, is a one-line change.

What makes that safe rather than reckless is the eval gate. From their corpus of human-corrected contracts we built a golden eval set — thousands of documents with known-correct answers — and a harness that scores any candidate model on accuracy, latency, and cost per stage before it is allowed near production traffic. A new model is not “adopted” because it tops a public leaderboard; it is adopted because it beat the incumbent on their eval set, at their stage, with no regression on the hard cases. The breath-holding is gone because the eval gate, not a customer, is now the thing that finds a regression.

How we worked with their team

A three-person Cravings pod — an applied-AI engineer, an ML engineer, and a senior software engineer — embedded with their ML team for fourteen weeks. We brought the staged-routing pattern, the eval harness, and the fine-tuning pipeline. They brought what only they had: which clauses actually matter to their customers, which extraction errors are merely annoying and which are contractually dangerous, and the labelled history that made the fine-tune and the eval set possible.

By week ten their engineers owned the eval harness and had run the first model swap themselves. The pod rolled off at week fourteen onto a light retainer for the quarterly evaluation review. The plug-and-play promise has since been tested twice in the wild: two new models released after we left were evaluated and adopted by their own team in an afternoon each, with zero pipeline changes and zero customer-visible breakage.

What changed

• Field-extraction accuracy: 82% → 96% on their gold standard, driven mostly by the fine-tuned extractor and the independent validation stage.
• p95 response time: 9.0s → 2.1s, because most documents never reach the frontier model.
• Model spend per processed contract: down 54%, from routing the easy majority to a cheap small model and reserving the expensive one for the cases that need it.
• Silent errors reaching customers: down an estimated 70%, as the Stage 4 critic now flags disagreements for human review instead of shipping them.
• Time to adopt a new model: two weeks of re-prompting and held breath → an afternoon behind the eval gate, demonstrated twice post-engagement.
• Engagement cost including Cravings fees, fine-tuning compute, and tooling: well inside one quarter of the model-spend reduction’s annual run-rate — the architecture paid for itself on the cost line alone, before counting the accuracy and speed gains.

Why this kind of work is hard for most teams

The instinct, when a single model is not good enough, is to wait for a better single model. It almost never solves the real problem, because the real problem is that one model is being asked to be cheap, fast, accurate, and self-checking all at once, and no model is all four. The leverage is in the architecture: the right model at the right stage, a fine-tune where your own data beats the general case, an independent critic, and a routing layer that turns “switch models” from a project into a config change gated by an eval set you trust.

This is where Cravings tends to be useful. The hard part of this build was not picking models — it was the unglamorous spine underneath: the eval harness built from the client’s own labelled history, the fine-tuning pipeline, the provider-agnostic routing layer, and the discipline to make every model a swappable part rather than a dependency. That spine is what lets a team stop holding its breath every time the frontier moves — and it is what AI enablement, done properly, actually leaves behind.