Benchmarking Methodology Standards for Document AI Accuracy Claims

The Document AI market has a math problem before it even has an accuracy problem. Gartner's inaugural Magic Quadrant for Intelligent Document Processing Solutions, published September 3, 2025, counts more than 100 vendors, all publishing accuracy numbers that engineers have no easy way to check against each other. Meanwhile, market-sizing analysts can't even agree on how big this space is: Grand View Research and Fortune Business Insights put out estimates for the 2025 IDP market that differ by several times over. That's a three-fold gap on the exact same question, produced by firms whose entire job is measurement.

If two research shops measuring an entire market can land three times apart depending on what they count and how, then a vendor measuring its own product on a dataset it built, scored against a metric it picked, is doing something structurally similar, just with more skin in the game. Roughly 80% of enterprise data is in unstructured, document-bound form, according to widely cited industry estimates, so the accuracy of extraction systems isn't a side issue. It's the whole ballgame for a huge share of enterprise automation. The trouble isn't that vendors lie about their numbers. It's that the three quiet decisions behind any benchmark, which documents to test, which metric to score, and which failure modes to leave out, can turn a mediocre system into a headline figure that looks production-ready. This piece lays out what a rigorous benchmark actually requires, and gives engineers and technical buyers a standard to hold vendors to instead of a leaderboard to take on faith.

How benchmark design choices quietly determine the outcome before a model is tested

Three choices, made long before a model sees a test set, decide most of the outcome. First, dataset selection: clean, modern, institutional documents score differently than the water-damaged invoice or the nineteenth scan-generation contract that appears in a real accounts-payable inbox. Second, metric selection: text similarity is not the same question as structural fidelity or downstream usability, and most benchmarks quietly answer the easier one. Third, exclusion: failure modes that a curated test set rarely includes don't get penalized, because they're never tested.

The gap between text similarity and usability is where a lot of the deception, unintentional or not, lives. A parser can report high character-level accuracy while producing output that's structurally useless to whatever system downstream is supposed to consume it. Tensorlake's benchmark analysis makes this point directly: traditional parsing benchmarks measure text similarity while ignoring structural preservation. A table can have every character correct and still be functionally broken if the rows got scrambled.

Documentation compounds the problem rather than fixing it. IBM Research's Auto-BenchmarkCard paper (arXiv:2512.09577, December 2025) found that benchmark documentation is often incomplete or inconsistent, which makes it hard to compare benchmarks across tasks or domains at all. Fields that should be filled in, like evaluation methodology, limitations, and known risks, are frequently left blank. The same paper's conclusion is blunt: an unsuitable benchmark risks leaving failure modes undetected, and that leads to systems being deployed with behavior nobody actually verified.

Most published numbers also carry an assumption that running a prompt once tells you anything stable. A University of Pennsylvania and Stony Brook study (arXiv:2608.06202), covering 401 prompts and 4,812 collected responses, found that repeating the same prompt produced inconsistent responses in up to 21% of cases. A system tested once per prompt can look perfectly stable and still be nothing of the kind. The same study found that turning on web search dropped accuracy by up to 8 percentage points, and that differences of just 2 to 3 percentage points between modalities were statistically significant, sometimes reversing direction depending on conditions. Small gaps in these numbers carry more signal than the big, round headline figures usually do. A benchmark reporting one number, on one document type, at one point in time, is not measuring a production system. It's measuring a moment.

What the performance cliff in OCR models reveals about headline accuracy figures

olmOCR 2 scores 82.3% accuracy on old math scans, and then, on general historical scans within the same benchmark, drops to 47.7%. That's a nearly 35-point collapse for the identical model, on a different slice of the same test suite. GOT-OCR 2.0 manages only 22.1% on olmOCR-Bench overall. TrOCR reports a 5.7% character error rate even after fine-tuning. None of these are fringe tools; they're models that look entirely viable until you check the specific subset where they fail badly.

The pattern behind these numbers is not random. OCR and document-understanding evaluation still centers on modern, Western, institutional paperwork, and that focus hides what happens to system behavior on historical material, degraded scans, and layouts that don't follow a standard template, which is exactly where a lot of production documents actually live. A vendor claiming something like "96.5% average OCR accuracy across diverse document types" in a 2025 benchmark study might be averaging across a distribution engineered to exclude the document classes where the system actually breaks.

The accepted benchmark for excellent printed-text OCR is 98 to 99%. A headline average creeping close to that range can coexist, quietly, with catastrophic failure on a specific document subclass that never gets its own line item in the report. Aggregate accuracy functions as a ceiling on what a system can do, not a floor guaranteeing what it will do. Document-type-specific breakdowns aren't a nice-to-have for a serious evaluation. They're the only way to know what the system is actually doing when it meets a document it wasn't tuned for.

The structural failures that most benchmarks still do not penalize

Reading order is a good place to start, because it's invisible to character-level scoring and decisive for everything downstream. A multi-column layout that gets flattened in the wrong sequence can produce text that's individually correct and collectively nonsense. Tensorlake's benchmark methodology treats reading order as its own measured dimension precisely because character accuracy won't catch it.

Tables tell a similar story with higher stakes. A table with textually correct cells that have been structurally rearranged will feed wrong numbers into whatever calculation happens next, and simple text similarity has no way to notice. Tree Edit Distance Similarity (TEDS) does notice, which is why it matters as a distinct metric. On OmniDocBench's harder table subcategories, some open-source models score below 70% TEDS, while specialized table-focused tools can score substantially higher, a gap that a character-level metric would never make visible.

Silent row dropping is arguably the most dangerous of these failures because it's the quietest. A system that extracts 18 line items out of 20 and drops the other two without any signal will still post a high per-field accuracy score, and it'll also produce a wrong total on every invoice it touches. Benchmarks that don't track recall at the row level simply never catch this.

Field-relationship accuracy is its own separate question: is the vendor's name correctly paired with the vendor's address, and not the customer's? Text similarity has no opinion on that; it's a structural question, not a character-matching one. And confidence scoring, or the absence of it, deserves its own scrutiny. Production systems need to know when a model is guessing. A benchmark that reports only an aggregate accuracy number says nothing about whether the system can flag its own uncertainty, which quietly lets vendors skip building that capability at all. Schema variability adds one more wrinkle: real invoices arrive in hundreds of templates, and a benchmark like Businessware Technologies' March 2025 study, which spans 20 invoices from 2006 through 2020, at least starts to surface template drift. Most academic benchmarks never go there.

What existing benchmarks cover and where each stops short

OmniDocBench, presented at CVPR 2025, is a serious academic baseline: 1,403 PDF files annotated for layout, text, formulas, and tables, scored with TEDS for table evaluation and covering OCR, layout detection, formula recognition, and reading order together. What it doesn't do is tell you anything about production integration, confidence scoring, schema versioning, or whether a system improves over time. It's a snapshot, and a good one, but still a snapshot.

A specialized OCR benchmark goes further into edge cases: 7,010 unit tests spread across multiple category splits covering a range of document types and difficulty levels. Among the major benchmarks, it comes closest to covering historical and irregular material. It still isn't representative of the sheer variety of documents an enterprise sees at scale.

The Omni AI OCR Benchmark, an open-source project from February 2025, evaluates JSON extraction through a document-to-OCR-to-extraction pipeline, scoring accuracy as one minus the ratio of mismatched fields to total fields, plus Levenshtein distance. To its credit, the project's own documentation states that this scoring heavily penalizes correct text that doesn't match the exact layout of the ground truth. That's a self-identified limitation, and it's worth taking at face value rather than treating the resulting scores as clean.

Doc-PP, the Document Policy Preservation Benchmark, uses an LLM as a judge and reports a 93% agreement rate with human annotators across a 100-instance validation study. That's a meaningful confirmation that automated evaluation can substitute for human review, but only once the framework has actually been validated against people, not simply assumed to work.

Tensorlake's enterprise document benchmark covers 100 pages spanning banking, retail, and insurance, scored with TEDS and JSON F1 together. The two-stage approach, structural fidelity first, downstream usability second, is close to a template for what production-minded benchmarking should look like. A hundred pages, though, is a thin sample to hang a general claim about enterprise coverage on.

None of these benchmarks closes the full loop. None of them offers ongoing evaluation against a customer's actual, evolving documents, failure attribution down to the individual field, or a correction mechanism that feeds errors back into the model or the prompt. They're all snapshots of a moment, some more honest about their limits than others.

The metrics that predict whether a Document AI system will hold up in production

Structural fidelity has to come before text accuracy, not alongside it. TEDS on a table answers "is this still a table" before anyone asks whether the text inside it is correct, and that ordering matters because structural errors compound as they move downstream, while character-level errors frequently don't.

Field-level precision and recall, reported as a JSON F1 score, measures two different things: whether the fields extracted are correct (precision) and whether all the fields that should have been extracted actually were (recall). Tensorlake's benchmark found a gap between 91.7% and 68.9% F1 on the same set of enterprise documents, a difference that works out to roughly five extra correctly extracted fields per twenty. Multiply that across millions of documents running through production, and the gap turns into an operating cost rather than a rounding error.

Row-level recall, calculated as total fields extracted over total fields present at the row level rather than the document level, is the only metric that reliably catches silent row dropping. Multi-run consistency affects trust in a benchmark: given that the University of Pennsylvania study found inconsistency in up to 21% of repeated prompts, any benchmark worth trusting has to report variance across runs, not a single pass dressed up as an average.

Document-class disaggregation is non-negotiable. Accuracy has to be reported separately for each document type in the test set, because an aggregate number that blends clean documents with degraded ones, or simple layouts with complex ones, tells an engineer nothing actionable. Confidence calibration deserves the same scrutiny: does a system's stated confidence actually track its real accuracy? A model that claims 95% confidence on fields it gets wrong one time in five is worse than one that honestly flags its own uncertainty, and this is a measurable property, not a subjective one. Industry-specific customization can produce meaningful accuracy improvements over generic systems, a gain that a benchmark testing only generic documents will never reveal, favorably or otherwise.

What rigorous benchmark methodology requires in practice

On the dataset side, the documents need to be real customer material, not a curated academic corpus: degraded scans, skewed text, water-damaged invoices, the full mess of template variance that occurs in an actual inbox. Results need to be reported per document-type stratum, never blended into one aggregate. A 2025 benchmark guideline sets 300 DPI as a reasonable floor for scan resolution in test inputs. And the dataset needs to include the edge cases a system will actually meet in the field: handwriting, multi-column layouts, merged table cells, footnotes, checkboxes, redlined text.

On the evaluation side, multiple runs per document or prompt should be standard, with consistency reported as its own metric next to accuracy, not buried. Evaluation should run in two stages, structural fidelity first (TEDS or an equivalent), downstream usability second (field-level F1), because these stages catch different failure modes and collapsing them into one score erases the distinction. Ground truth has to be built and checked by human reviewers rather than generated by a model and accepted on faith. And the scoring methodology needs to be documented well enough that an outside team could reproduce it: IBM Research's Auto-BenchmarkCard paper notes that documentation today is often so thin that developers end up reading source code just to figure out what was actually measured.

Transparency is its own requirement, separate from methodology. Publish the dataset, or at least a representative sample of it. Name the document types, where they came from, and any preprocessing that happened before scoring. Disclose known scoring limits, the way the Omni AI benchmark does with its exact-layout penalty, instead of leaving them out of the write-up. Report accuracy broken down by document class, by field type, and by confidence band, not folded into a single headline number.

None of this matters if it's a one-time exercise. Production integration means continuous benchmarking against documents that keep evolving, schema versioning so that accuracy gets tracked as extraction schemas change over time, and failure attribution at the field level so engineers know exactly which fields are breaking and why. Human review loops, where corrections feed back into the evaluation set, are what separate a system that improves from one that's frozen at whatever number got published once.

How to evaluate a vendor's benchmark claims using this standard

The first question to a vendor is never "what's your accuracy." It's: on what documents, measured how, against what ground truth, across how many runs?

A handful of signals reliably indicate a benchmark built to flatter rather than inform. A single aggregate accuracy figure with no breakdown by document type obscures exactly the performance cliffs that matter most in production. A benchmark dataset whose methodology is incomplete or inconsistently described, as IBM Research's Auto-BenchmarkCard paper found is common, makes cross-benchmark comparison unreliable. So is a result built on a single run with no consistency metric attached. Table accuracy reported without a structural metric like TEDS leaves row-level and cell-level failures undetected. So is precision reported with no recall alongside it, since a system that extracts fewer fields more carefully can look better on paper than one that actually captures everything present. And confidence scores are only meaningful if there is evidence that they track real accuracy, not simply that the system outputs a number alongside its predictions, not evidence.

What to actually demand is straightforward. Run the benchmark against a sample of the buyer's own documents, not the vendor's curated test set. Ask for field-level failure reports instead of document-level averages. Row-level recall is the metric that catches silent row dropping, and it is worth confirming whether any vendor benchmark tracks it. And require that confidence scores be validated against real error rates on the specific document types in question, not on whatever set the vendor happened to test.

Platforms that treat evaluation as a working part of the product, rather than a one-time marketing artifact, make all of this easier to check. Built-in evaluation sets, accuracy reports broken down by field and by document, schema coverage that accounts for template variation over time, and human-in-the-loop correction that actually feeds back into the system: these aren't extra features bolted onto a good Document AI product. They're what separates a benchmark from a system anyone can trust with real documents.