Extraction Accuracy Benchmarks Dissected by Document Type and Noise Level

Extraction accuracy benchmarks can show how reliable a system will be in production only when the results are split by document type, noise level, and schema complexity. A high aggregate score cannot establish whether a system will extract the right dosage from a clinical record or preserve every quantity in a dense table. The useful evaluation unit is the field under realistic document conditions.

Why aggregate accuracy scores are the wrong unit of measurement for production document pipelines

Buyers often encounter a headline accuracy number before they see the documents or scoring rules behind it. That number appears to offer a direct comparison between systems. It does not establish production fitness. Enterprise buyers report a 15–25 percentage point drop between vendor benchmark performance and results on their own document mix, where dozens of formats, scanner artifacts, handwriting, and dense tables replace clean academic layouts.

The gap is structural rather than incidental because an aggregate combines tasks with different failure conditions. Correctly reading printed prose does not demonstrate that a parser can bind a value to the right table column. Nor does text similarity show whether a blank field was left empty or filled with an invented value. These are different tests of extraction quality. Their relative frequency in a benchmark determines how much each contributes to the final score.

RealDocBench identifies unrealistic documents, the wrong metric, the absence of a paired layout track, and missing cost and latency reporting as gaps in existing evaluations. Its argument reaches beyond document selection to the meaning of success itself. A parser can reproduce most of a page while misreading the cash-to-close figure on a Closing Disclosure. The downstream workflow needs that specific value, so a strong page-level score can coexist with an operational failure.

How document type determines which failure modes are even possible

With clean printed text, the main task is recognition. With standard fonts and enough resolution, recognition is all but solved, so the errors left on clean material say little about the hardest production cases. Structured forms introduce a different task. The system must identify which value belongs to which key, distinguish checkbox states, and keep missing values distinct from explicitly empty fields.

A value is useful only when it remains bound to the correct row and column. Reading every character correctly is therefore insufficient. Dense layouts can fail through structural association rather than recognition, while captions and figures require the parser to distinguish kinds of content rather than treat the page as an undifferentiated text stream.

Handwriting creates a distinct recognition problem, with legible writing and messy or cursive script presenting different levels of ambiguity. With scanning, resolution can make the task harder, while phone capture adds glare and perspective distortion. These categories can overlap on the same page. A scanned form with handwritten entries combines capture quality, character recognition, and field association rather than occupying one position on a simple accuracy scale.

RealDocBench makes these distinctions explicit in its design. Its QA track tests row-and-column-bound values, checkbox states, figure captions, and blank fields that must be returned as empty. Its layout track annotates nine structural classes, including tables, key-value regions, and figures. That separation supplies a vocabulary for diagnosing failure before any accuracy number is considered.

How noise degrades accuracy differently across each document category

Physical noise makes each category harder in its own way because the tasks differ. With enough resolution, printed text starts close to the best recognition accuracy possible in practice. That advantage weakens as image quality falls. Scans below 300 DPI cause approximately a 20% accuracy drop compared with higher-resolution scans, a loss that better capture can prevent but even a high-end model cannot make up for.

Handwriting is less reliable to start with, so compression artifacts, fax rendering, and low contrast make recognition harder still. Messy or cursive script is substantially less dependable. Benchmarks from 2026 place its accuracy at 60–75%, a range where unattended automation is not viable and human review is effectively required.

For forms and tables, degradation also depends on what the field asks the system to resolve. A binary field and a complex categorical field are not interchangeable tests, even when both appear on a clinical document. Success on yes/no fields does not validate open-ended categorical or numeric extraction. When these field types are pooled, their different difficulty levels disappear into the mean, obscuring the relationship between document quality and the required interpretation.

The final effect compounds across the output. A single misread digit can invalidate a financial total despite high character accuracy across the page. As the number of fields grows, there are more chances for an error in a document, so document-level zero-error rates fall even when per-field accuracy stays high. This is the rate that governs touchless processing. Noise must therefore be evaluated not only by how much text remains readable, but also by how often every required field survives extraction correctly.

Why schema breadth and output volume are underappreciated noise sources in extraction benchmarks

The same compounding effect extends beyond physical image quality to the schema imposed on a document. A clean page does not guarantee a reliable structured response when extraction requires many fields and a long output. Schema breadth is an independent failure driver. Benchmarks limited to short answers or narrow schemas can leave this production risk largely untested.

Contextual AI’s ExtractBench evaluates schemas ranging from tens to hundreds of fields and finds sharp degradation as schema breadth grows. On a large financial reporting schema, every frontier model tested failed to produce any valid output. Long documents and broad schemas require long, precisely structured responses that are vulnerable to truncation and formatting errors.

ExtractBench also reports low valid JSON rates across six tested models, separating output validity from field correctness. Even when a response passes the JSON validity test, it can have wrong values on complex schemas. Syntax is only the first gate. Treating a parseable response as successful extraction would overlook the very field errors that the structured output is meant to prevent.

Nested arrays are harder to evaluate because scoring must take care with the order and with matching each item. An omitted item must be distinguished from a hallucinated one, while a minor formatting difference in a name is not equivalent to a wrong identifier. Uniform recursive comparison obscures those distinctions. In this broader sense, production noise includes artifacts introduced by a scanner and the interaction of schema complexity, document length, and layout.

What the benchmark methodology itself gets wrong, and what better designs reveal

Metric choice determines which failures an evaluation can see. Character accuracy measures how well text is recognized, field accuracy measures whether the values asked for are right, and document accuracy measures whether all the required fields are free of errors. These measures answer different operational questions. A field with a single character error may register as 99% correct at the character level while failing the extraction requirement entirely.

Confidence reporting introduces a separate methodological test. A confidently wrong answer can get through review routing and into downstream systems as correct data, but most vendor accuracy reports do not give an explicit count of silent failures. Calibration is therefore part of production reliability. Per-field confidence tied to observed accuracy allows a tax total to be routed for review while the vendor name is accepted, a distinction document-level confidence cannot support.

ExtractBench treats the schema as an executable specification for scoring. Identifiers use exact match, quantities use numeric tolerance, and free text uses semantic equivalence, while omission remains distinct from hallucination. The field’s role determines the appropriate test. This approach avoids treating every difference as equally serious and makes the evaluation sensitive to the requirements expressed in the schema.

RealDocBench addresses another source of distortion by using a uniform extraction-and-scoring protocol and reporting per-page cost and cache-busted latency with accuracy. Its paired QA and layout tracks also distinguish field retrieval from structural understanding. Those measurements expose trade-offs that a single ranking suppresses. An evaluation with appropriate field metrics, realistic documents, and per-domain reporting is consequently a different instrument from a leaderboard score, because it can identify both the location of failure and the conditions under which it occurs.

How the document-type-and-noise framework applies to regulated industry workflows

Regulated workflows raise the stakes because extracted fields can feed autonomous agents that act on them. A wrong cash-to-close figure, dosage, or line-item quantity can therefore turn a transcription error into a financial or legal problem. Document mix sets the dominant risk. RealDocBench draws from mortgage, finance, supply chain, and medical/healthcare documents containing dense forms, checkbox grids, multi-column tables, handwriting, stamps, redactions, and scanner artifacts.

Mortgage extraction shows why field association needs to be measured on its own. In dense fillable forms and checkbox grids, labels, states, and values must be linked correctly, and the cash-to-close figure must be right in its own field. Reading the surrounding page is not enough. A benchmark made up mostly of clean prose says little about those structural requirements, even if its recognition score is high.

Medical extraction combines complex categorical fields, handwritten annotations, and dense tables within the same domain. RealDocBench finds medical/healthcare persistently hardest across eighteen evaluated systems, showing that broad performance does not eliminate domain-specific difficulty. That finding limits what general scores can establish. In particular, strong performance on simple fields cannot stand in for validation of dosage values or more complex clinical categories.

Financial reporting exposes schema breadth more directly. ExtractBench’s 369-field financial reporting schema produced zero valid output across all tested frontier models, demonstrating a failure condition relevant to the schemas used in those workflows. In supply chain extraction, bills of lading, delivery notes, and warehouse receipts come in as phone photographs and faxed forms, so resolution and capture method are central to evaluation, with image quality the main risk.

These differences prevent one industry’s results from serving as a reliable proxy for another’s. A benchmark can include regulated documents yet still weight the wrong structures, fields, or capture conditions for a particular deployment. Domain labels alone are too broad. Predictive evaluation depends on the actual mix within the industry, including which documents contain the hardest fields and how those documents arrive.

What a benchmark result needs to include before it can predict production accuracy

A credible result discloses the document types in its corpus and their proportions, the noise present, the schema complexity, and the mix of field types. It also defines the metric and states whether silent high-confidence failures count as errors. The aggregate must be shown alongside breakdowns by domain and field. Without these details, a buyer cannot tell whether the system did well on tasks like those in the planned workload.

The same scrutiny applies to who ran the evaluation and whether it can be reproduced. Vendor-run and independently audited benchmarks are not equivalent evidence, while a reproducible evaluation harness provides a stronger basis for assessing a claim. Independence does not erase differences in test design. Each accuracy figure remains conditional on its metric, corpus, and evaluation status, so comparisons require those conditions to be aligned or their differences made explicit.

For a team evaluating a candidate system, the practical questions follow directly from those disclosures. Does the corpus include the same difficult forms, handwriting, tables, and capture defects as the production workload? Does the schema reach the required breadth and output volume? Does scoring tell an omitted value apart from an invented one, and does per-field confidence match observed accuracy well enough to guide review routing? These questions establish what the score actually licenses the buyer to conclude.

The practical test is to run the candidate on a sample of the team’s own document mix, stratified by type and noise level, using the intended schema. Scoring should operate at the field level and count silent failures, with document-level zero-error rates exposing the limits of touchless processing. A public benchmark is a starting point. The production-relevant result is performance on the fields, structures, and noise conditions the pipeline will actually face.