Accuracy vs. Latency vs. Cost Tradeoffs in Document Extraction APIs

Accuracy, latency, and cost in document extraction work against each other by design, not because vendors priced things that way, an architectural fact, not a negotiating position. That's an architectural fact, not a negotiating position.

The reason traces back to how modern document AI actually works. Current systems stack OCR, layout understanding, named entity recognition, and large language models on top of each other so the pipeline can read a document the way a person would, holistically, rather than as a grid of characters. Every layer added to that stack buys some accuracy on documents with awkward layouts, and every layer also adds inference time and compute cost. There's no way to bolt on the comprehension without also bolting on the bill.

That tradeoff isn't uniform across a workload, either. Documents that all look the same, standardized forms, templated invoices, repeat filings, don't need the heavy stack; a cheap deterministic path handles them fine. Documents that vary wildly in layout and content are where the expensive path earns its premium. Almost no real production queue is made of one or the other exclusively. It's a mix, and that mix is precisely what gets lost when a team picks a single extraction configuration and applies it to everything that comes through the door. The result is a pipeline that overpays on the easy half and under-extracts on the hard half, and both failures carry real downstream cost. What follows is a way to reason about why each piece of that tradeoff behaves the way it does, so the configuration decision stops being a guess.

Input representation choices and the accuracy-latency baseline

Before any model even runs, the choice of what to feed it, a page image, extracted text, or both, already sets the ceiling on accuracy and the floor on latency. This is the earliest lever in the pipeline, and it's the one teams are most likely to treat as fixed rather than tunable.

A ServiceNow study (arXiv:2609.22628) tested this directly, holding the prompt, the judge, and the scoring pipeline constant across four commercial endpoints and two document corpora. Under oracle conditions, image input beat text input on accuracy at every document length, on both corpora. The reason is straightforward: images retain visual evidence, spatial relationships, layout cues, handwriting shape, that a text-only pipeline simply discards during extraction. A companion 2026 study, Text, Pixels, or Both?, found the gap could run as high as 14.8 percentage points in favor of images over text-only input.

That accuracy doesn't come free. Text-based latency stays roughly flat as documents get longer, while image-based latency climbs steadily with page count.

Image and text don't fail on the same questions. On a meaningful share of items, exactly one representation gets the answer right and the other doesn't. Neither approach subsumes the other; they're complementary failure modes, not a strict hierarchy. A lightweight TF-IDF question-text router that selects representation per question gains accuracy over always-text while cutting median latency relative to always-vision. Routing, not a fixed global choice, turned out to be the efficient frontier.

Most teams never get this far. Representation gets decided once, during initial pipeline setup, and then treated as infrastructure rather than as a parameter that should respond to the document in front of it. A pipeline locked to text-only is quietly leaving accuracy on the table every time a layout-heavy document comes through. A pipeline locked to images is paying a latency tax on the text-heavy documents that never needed it. This is the first point in the stack where the tradeoff can actually be managed instead of just absorbed, and it's invisible to anyone who never looks at representation as something to tune.

Accuracy gaps between APIs on the documents that matter most in production

Vendor accuracy numbers tend to come from clean benchmarks, and clean benchmarks are exactly the documents that don't reveal anything useful. Dense tables, long repeating arrays, handwritten fields: these are where extraction systems actually break, and they're also where a wrong answer costs the most downstream. An aggregate accuracy figure that averages in a pile of easy documents will always understate the real gap between two APIs, because it dilutes the failure with success stories that were never in question.

Published benchmark work on long, repeating array extraction bears this out starkly. One documented result put a commercial extraction product at 68.8% on a long-array benchmark, a number that would look entirely respectable on a clean-document test but signals a real production risk once tables run into the hundreds of rows. The gap between how a system performs on marketing-grade sample documents and how it performs on a client's actual thousand-row invoice schedule can be enormous, and that gap is systematically hidden by the way most benchmarks get built.

There's a second, quieter metric that matters just as much: whether the system finishes the job. A system that times out on the longest documents in a batch and then reports accuracy only on the documents it completed is grading itself on a curve it built. The honest way to account for that is to score a failed run as zero, not to exclude it from the denominator. Run completion and field accuracy are two different numbers, and a vendor that only publishes one of them is telling half the story.

None of this means clean-document accuracy is meaningless, only that it's not predictive. Accuracy on a tidy test set says nothing reliable about accuracy on long arrays, dense forms, or handwriting, because those failure modes live in a different part of the model's behavior. The only test that means anything is evaluation against documents that actually resemble the production workload, not the workload a vendor chose to showcase.

Model choice, pipeline architecture, and volume in determining per-document cost

Cost per document isn't a single number attached to an API. It moves by roughly an order of magnitude depending on which model does the work, how the pipeline is built, and how much volume runs through it, and the cheapest setup for one workload can be the most expensive for another.

A peer-reviewed John Hancock study (arXiv:2609.15706), focused on templated business document extraction, laid out the relative cost ordering across approaches. A non-LLM pipeline that pairs OCR with regex matching is at the bottom of the cost curve, the cheapest option tested. Add a large language model into a hybrid with OCR and the cost roughly doubles. Running a vision-capable model like GPT-4.1 directly on document images costs a similar range to that hybrid, with Claude Sonnet 4.5 moderately more expensive than GPT-4.1. Claude Sonnet 4.5 comes in moderately above GPT-4.1. None of that ordering is exotic; it tracks reasonably well with what each model costs to run per token or per call.

Managed services complicate the picture further because they price by the page rather than by model call. The study cites Azure Content Understanding running near a cent per page, and that structure scales with page count regardless of how complex any individual page actually is.

Self-hosting breaks this logic open. Run a dedicated GPU instance continuously and the marginal cost of each additional document trends toward zero, right up until the instance saturates. Below that ceiling, extra documents are nearly free; cross it, and a second instance has to come online. Where that crossover sits depends entirely on volume and how dense the documents are, which means self-hosting only makes economic sense once volume clears a threshold that's specific to the workload, not a fixed number.

Architecture matters here as much as model choice does. An OCR+LLM hybrid that routes the predictable majority of fields through cheap deterministic OCR, and reserves the LLM call for the fields that are genuinely ambiguous, can match or beat a pure-LLM approach on structured forms while costing meaningfully less. That's not a compromise; it's a design decision that recognizes most fields on most forms don't need a language model's judgment.

Fine-tuning shifts the frontier again. The study found that fine-tuning open-source vision-language models on a moderate labeled sample lifted the best of them above an F1 of 0.98 on structured check extraction, ahead of every zero-shot commercial system tested on that task. Once the upfront training cost is amortized across enough volume, the per-document inference cost comes in lower than the commercial alternatives too. A fine-tuned model is narrow by construction: it's excellent at the document type it was trained on and needs retraining the moment layouts shift meaningfully, and that retraining is a maintenance cost that doesn't appear in the per-document number.

The routing architecture that resolves the tradeoff for most production workloads

The pattern that emerges from all of this points toward one architectural answer: don't pick a single extraction configuration, build a routing layer that sends each document down the path calibrated to its own complexity. That preserves accuracy where it's needed and controls cost and latency everywhere else, instead of forcing one setting to do both jobs badly.

The routing logic follows naturally from what's already been established about document behavior. High-volume, structurally uniform documents, standard forms, templated invoices, get routed to the cheap, fast, deterministic path: OCR plus regex, or a lightweight model at most. Documents that vary in structure, run long, contain handwriting, or carry compliance weight get routed to the higher-accuracy, higher-cost path. And outputs that come back with low confidence on either path get routed to a human, rather than passed through silently. That third branch is what makes the whole system trustworthy: confidence scoring is the mechanism that makes routing decisions defensible, and it is not a nice add-on here.

Some documents need more than one pass. A multi-pass, "agentic" approach, where a vision model reviews and corrects the output of a first extraction pass, costs more time and more compute per document, but it recovers accuracy specifically on the documents where a single pass fails. Extra latency and extra cost are bought back in the form of accuracy on the hardest tail of the workload.

That premium pays off in specific, identifiable situations, not as a default. High document variety across many different senders is one. Validation-heavy workflows, accounts payable, claims processing, compliance review, are another. So is any workload where the manual review rate has climbed past the point where paying for the agentic path is cheaper than paying people to check everything by hand. And so is any workflow where an extracted value triggers a downstream action automatically, because a wrong number there has a real business consequence, not just a cosmetic one.

The representation-routing insight from the ServiceNow study belongs at this same layer, not just at the input stage. Routing image versus text per document, or even per question, rather than committing to one globally, recovers most of the accuracy benefit of always using images, while cutting median latency by roughly 30% relative to an always-vision approach. Routing, in other words, is a decision made repeatedly throughout the pipeline rather than a single decision made once at the top. It's a pattern that repeats at every layer where a cheap path and an expensive path both exist, and it earns its keep at each one.

Where confidence scoring fits in the pipeline

Most extraction systems deployed today return a single value with no indication of how sure the model was when it produced that value. A low-confidence guess and a well-grounded fact come back looking identical, and that's a structural gap, not a minor inconvenience, because routing and human review both depend on exactly the signal that's missing.

It routes uncertain extractions to a human before they ever reach a downstream system, catching the error at the cheapest point to fix it. It separates field-level uncertainty from document-level uncertainty, since a document with one shaky field is an entirely different routing problem than one where the model is unsure about everything on the page. And it feeds corrected values from human review back into the system's ongoing improvement, without requiring a full retrain on customer data every time.

Regulation has made this less of an internal engineering preference and more of a legal obligation. The EU AI Act's transparency provisions took full effect on August 2, 2026, and while the Act's high-risk provisions, covering AI used in hiring, credit scoring, and comparable high-stakes decisions, were pushed to December 2, 2027 under the EU AI Omnibus, the penalties attached to non-compliance run as high as €15 million. Under the Act's explainability requirements, returning a value isn't enough; the system has to be able to show where that value came from. Per-field citations back to the source document are the practical way that requirement gets satisfied. The extended timeline is preparation time, not a reason to wait until the deadline is closer.

There's a structural-reliability piece that sits alongside all of this, too: running LLM output through a JSON Schema validator, or an equivalent typed model, and building in a self-correction loop so the model retries until the output actually validates. That catches malformed output before it ever reaches a downstream consumer, which is a cheaper place to catch it than after it's already broken something else.

Evaluating an extraction API against your actual workload

Everything above points to one conclusion: the only evaluation that means anything runs on documents that look like the actual production workload, not a vendor's benchmark set. Published benchmarks, even open ones a vendor releases voluntarily, are a starting point for narrowing the field. They are not a basis for a purchasing decision on their own.

A few things are worth measuring, in roughly this order of priority:

Run completion rate. Does every document in a representative batch actually finish processing? A system that times out on the longest documents and reports accuracy only on the ones it completed isn't a viable production option, regardless of what its headline number says. Array recall and duplicate handling. For any workload with repeating records, tables, line items, schedules, check whether every expected record shows up in the output. A plausible-looking partial response is more dangerous than an obvious failure, because it passes downstream validation without anyone noticing rows went missing. Field-level accuracy on the field types that actually matter. Aggregate accuracy hides field-specific weak spots. Complex layouts, handwritten entries, and nested tables each stress a different part of the extraction stack, and a system can be strong on one while quietly weak on another. Latency under real concurrency and real document length. Pipelines that process documents as images scale latency with page count, so testing against a median document tells very little. Cost at actual volume. Model the pricing structure, per page, per token, per provisioned instance, against real monthly volume and document density. The cheapest option at low volume is frequently not the cheapest option once volume climbs.

Beyond those five, two checks are easy to skip and shouldn't be. Run the same document set through two versions of a schema and confirm that changing one field doesn't silently shift extraction on an adjacent one; schema stability under iteration is a production requirement, not something to worry about only at demo time. And test the review workflow end to end, not just the confidence score in isolation, to confirm the threshold configured actually catches the errors that show up in real documents rather than flagging borderline-but-correct values instead. For anything in healthcare, financial services, or legal work, verify that field citations point to the right location in the source document and that the audit trail can reconstruct how any given result was produced. Readable output isn't the same thing as an auditable one, and the difference tends to become visible at exactly the moment it's most expensive to discover.