Why Vision Language Models Are Not Enough for Production Document Extraction

Why do high OCR accuracy benchmarks like 99% character recognition not translate to reliable invoice or structured document field extraction in production?

The gap opens because character-level accuracy and field extraction are measuring different things. A 99.2% character recognition score describes how well a system reads raw text on clean, predictable layouts; it says nothing about whether the system correctly placed the invoice number into the invoice_number field on a document nobody designed for a machine to read. Table extraction scores on RD-TableBench, 1,000 complex tables, range from 90.2% down to 64.6% across major commercial systems, and that gap appears before any consideration of multi-page spans, rotated scans, or stamps layered over typed text. Compounding the problem, when a scan is genuinely unreadable a model does not return null: it generates something plausible instead, a dollar figure or ID number that is structurally correct, passes every downstream validation check, and is simply wrong, moving through a billing or database system until a customer finds it.

The headline numbers look reassuring, and that's exactly the problem with them. On purely typed text, leading cloud OCR engines clear 99.2% accuracy. On mixed printed pages, Vision OCR achieves 98.0%. Read those figures again: they describe raw character recognition on clean, predictable layouts. They say nothing about whether a system correctly pulled the invoice number into the invoice_number field, on a document nobody designed for a machine to read.

Table extraction is where the gap between "reads text" and "extracts structure" turns into a canyon. On RD-TableBench, a set of 1,000 complex tables, scores across major commercial systems range from 90.2% down to 64.6%, the gap between a system a business can rely on and one that needs a human checking its work on every single table. On RD-TableBench, a set of 1,000 complex tables, scores across major commercial systems range from 90.2% down to 64.6%, a gap far larger than a rounding difference. That's the gap between a system a business can rely on and one that needs a human checking its work on every single table. RD-TableBench is run by a vendor, not an independent lab, so those numbers describe how the tool performs on that vendor's chosen tables, not necessarily the tables sitting in any given company's document backlog. Anyone weighing an architecture decision on a benchmark like this should run their own documents through it before signing anything, full stop.

Open-source VLM OCR benchmarks, pulled from the consolidated document parsing survey on arXiv (2410.21169), show narrow but real differences at the top of the field: GLM-OCR scores 94.62 overall, PaddleOCR-VL-1.5 is 94.50, FireRed-OCR at 92.94, DeepSeek-OCR 2 at 91.09, and MinerU2.5 at 90.67. These are strong numbers, and they're also aggregate scores across mixed document types. A model can post a 94 overall while still failing badly on the one document category a given business actually processes all day, every day.

VLM failures on real documents: layout complexity, token cost, and format fidelity

Layout variability causes nearly every VLM failure on enterprise paperwork. Visually rich documents, the kind full of tables, stamps, signatures, and inconsistent column widths, stay hard for these models precisely because the layout itself carries meaning that shifts from one document to the next.

Multi-column insurance forms break reading order whenever the columns aren't equal width or get interrupted by an image or a logo dropped mid-page. Handwritten annotations layered over typed text get merged by the model, or the line between the two gets lost entirely, so a margin note ends up read as part of the printed field. Dense financial tables are worse: single-pass VLM OCR on these can hallucinate or drop content, a risk that compounds with layout complexity. Tables that span multiple pages compound the problem, since keeping structure intact across a page break adds another layer of difficulty for current systems. Rotated pages, low-resolution scans, and fax-quality documents add a further wrinkle: combinations of perspective distortion, rotation, and noise present additional challenges for modern VLMs doing structured extraction, and their compounding effects are not well characterized.

Format fidelity is a separate problem, distinct from whether the content itself is correct. TEXOCR-Bench, a set of 2,135 expert-annotated examples, found that even the best-performing model only reaches 78.5% accuracy on producing LaTeX that actually compiles. Models that look excellent at PDF-to-Markdown conversion degrade sharply the moment the task demands global structural constraints: consistent structure across pages, correct float placement, reference links that still point where they should. Markdown forgives a lot. Compilable LaTeX forgives nothing, and that gap says a lot about how much slack exists in most of the formats teams currently benchmark against.

Then there's the cost side, and this is where the throughput pitch tends to fall apart. DeepSeek-OCR's "contextual optical compression" cuts the number of vision tokens needed by a factor of 7 to 20, which is how it processes runs of 200,000-plus pages without the token bill spiraling. Push compression to the top of that range, though, and accuracy falls to around 60%. Cost optimization has a floor, and that floor sits lower than most teams assume when they're only looking at the throughput number.

Hallucination in document extraction is categorically different from hallucination in chatbots

A chatbot that hallucinates gives a wrong answer that a person reads and, often, catches on the spot. A document extraction system that hallucinates does something worse: when a scan is unreadable, the model doesn't return null. It generates something plausible instead, an ID number the right length with a valid-looking checksum, a name slotted into the wrong field, a dollar figure that's structurally correct but simply invented.

A format-valid hallucination should worry engineering teams more than any benchmark score does. It passes every downstream validation check built to catch malformed output, because it isn't malformed. It looks exactly like a real value. It moves through the pipeline, lands in a database or a billing system, and gets caught, if it gets caught at all, by a customer months later rather than by an engineer during review.

Document extraction feels this more sharply than almost any other corner of applied AI, because so many of the fields involved demand exact match, not approximate correctness. A policy number off by one character is wrong, full stop, the same as if the field were left empty. And the math compounds fast across any multi-step agentic pipeline: an agent running at 95% reliability sounds fine in isolation, but a hallucination rate of even a tenth of a percent, multiplied across the volume a real enterprise pipeline processes, adds up to thousands of confidently wrong answers moving through the system with nothing flagging a single one.

How benchmark design shapes which production failures get caught

Benchmarks aren't neutral rulers. What they measure, and how they measure it, decides which failures surface before deployment and which ones slip through to a customer's desk instead.

OmniDocBench started with 981 samples and has expanded across releases, spanning nine document source types from academic papers to handwritten notes. It scores text with Normalized Edit Distance, tables with TEDS, and formulas with CDM. Version 1.6 introduced Multi-Granularity Adaptive Matching, built specifically to strip matching bias out of the scoring process, which says something on its own: even a well-designed benchmark needs revision once teams find the seams in it. OmniDocBench is also starting to saturate. As newer models close in on ceiling performance, the benchmark stops telling "very good" apart from "excellent," which limits how useful it stays for choosing between top-tier systems going forward.

ExtractBench closes two gaps that no earlier benchmark addressed. No end-to-end benchmark had evaluated PDF-to-JSON extraction against the kind of schema breadth an actual enterprise runs, and no benchmark methodology captured the layered nature of correctness in nested extraction, where an identifier needs an exact match, a quantity can tolerate a margin of error, and a name only needs semantic equivalence. It also draws distinctions that matter in practice: different field types carry different correctness requirements, and a single blended error rate can obscure which specific failure modes a system actually has.

READoc is a newer, unified benchmark aimed at realistic document structured extraction. Recent papers cite it as a real-world-oriented complement to OmniDocBench.

The RD-TableBench issue raised earlier isn't a one-off. Vendor-run benchmarks are the norm in this space, not the exception. A benchmark built by the company selling the tool isn't disqualifying on its own, but it is a reason to treat the published number as a starting point for evaluation rather than a conclusion. Any team choosing an architecture should run its own document distribution through the candidate systems before trusting a leaderboard number someone else generated.

The system architecture components a VLM alone cannot provide

The mature architecture taking shape across the field in 2026 is hybrid routing, not universal agentic processing thrown at every incoming file, and teams that skip this step are the ones whose pipelines fall over first. Predictable, high-volume document types go down a cheap, deterministic path. Documents with genuine variance, the ones that would break a fixed template, get routed to the slower, more expensive agentic path that can plan and check its own work. That routing decision, more than any single model choice, usually decides whether a document AI program survives its first budget review.

Classification has to happen before extraction starts, not after. A system that runs one prompt against every incoming document, regardless of layout or type, applies the same tool to fundamentally different problems and calls the resulting mess a pipeline. Splitting documents into known types first is what lets a team route the bulk of the boring cases cheaply and save the expensive reasoning for the small remainder that's genuinely hard.

Combined pipelines that pair text extraction with a vision-language model earn their keep here too. Running OCR first gives a system cost control, an audit trail, and page-level intermediate output compatible with text-only parsing models downstream, none of which a pure VLM approach hands over for free. As models improve, the tradeoff between a pure VLM approach and an OCR-enhanced pipeline may shift, so it should not be treated as a fixed rule to build an architecture around permanently.

A study out of a collaboration involving John Hancock (arXiv:2609.15706) shows the training harness around a model affects extraction reliability more than raw model horsepower does. The study ran eleven systems, spanning commercial models, reasoning models, open-source VLMs, and a baseline approach pairing text extraction with pattern-matching rules, against 750 held-out synthetic checks. Fine-tuning on a moderately sized labeled dataset pushed the best open-source VLMs above an F1 score of 0.98, ahead of every zero-shot commercial system tested on that same task. The model wasn't what made the difference here. The harness around it was.

Evaluating and selecting the right approach for a given extraction task

Start with the document, not the model. A single-column invoice with a fixed vendor list is a different problem than a multi-page loan application scrawled with handwritten margin notes, and no amount of model capability erases that difference. Only good routing does.

Test on the actual document distribution a business will run in production, not on a vendor's benchmark set. RD-TableBench, OmniDocBench, and ExtractBench help narrow a shortlist of candidate approaches, but they don't substitute for feeding a system the messy, specific scans that will actually hit it once deployed.

Separate correctness criteria by field type before scoring anything. An identifier needs exact match. A quantity can tolerate a defined margin. A name might only need semantic equivalence. Treating all three the same way, with one blended accuracy number, hides exactly which fields a given system is unreliable on, and that's the information a team needs before shipping, not after.

Decide, deliberately, where the deterministic path ends and the agentic path begins. Not every document needs a system that plans, checks its own work, and retries. Most don't. The ones that do are usually a small, identifiable slice of total volume, and finding that slice, rather than running everything through the most expensive path by default, is what keeps a document AI system affordable at scale.

Build in a way to catch a confident, well-formatted wrong answer before it reaches a customer. Confidence scores straight from a model, uncalibrated, won't do this reliably. Neither will a validation pass that only checks for malformed JSON. Assume, from the start, that a format-valid hallucination will eventually show up, and design around that assumption with a specific mechanism: targeted spot-checks, exact-match rules on the fields that demand them, or human review triggered by defined risk signals. Catch it there, before a downstream system does it for you.