How 2024 Benchmark Leaders in Document Extraction API Accuracy Were Separated by Methodology Not Just Score

Document extraction API rankings in 2024 were separated by methodology as well as score. Parsing quality, structured-field correctness, and completeness measured different capabilities, so a leaderboard could not substitute for a production evaluation. For healthcare teams seeking a HIPAA-compliant API with self-hosted deployment, accuracy also left a separate question unanswered, namely whether the deployment met requirements for handling protected health information.

Why leaderboard rankings failed to predict production performance in 2024

A vendor’s benchmark score looked strong, but its output failed when connected to a production pipeline. That was the frustration behind document extraction buying decisions in 2024. Engineering teams committed time on the strength of rankings, then found that the measured capability was not the one their workflow required. The score had answered a narrower question.

Benchmark design drove the gap alongside model quality. What a score meant was set before processing began by the test documents, the rules for what counted as correct, and how incomplete results were scored. Tests chosen by vendors were even harder to compare because they used different document sets and rules for accuracy. Three methodological faults explain the gap most directly, including the conflation of parsing with extraction, metrics insensitive to structure, and scoring that overlooked silent omissions.

Parsing accuracy and extraction accuracy measure different things

Parsing turns a PDF into text or Markdown and, when it works as intended, keeps the reading order and table structure. Extraction returns structured values against a schema. These are distinct operations, even when an API performs both in one workflow. A parser can reproduce every word in a table cell yet lose the cell’s relationship to its row or column. An extraction system using that representation can therefore return the right text as the wrong field.

The distinction changes what an evaluation must measure. TEDS compares table structure, while JSON F1 measures the correctness and completeness of structured output. In operational terms, preserving a table is not the same as producing data that downstream automation can use.

ExtractBench, by Ferguson et al., identifies the broader evaluation gap formally. Its paper states that no existing benchmark jointly covers extraction from PDFs, enterprise-scale JSON schemas, and fine-grained key-level evaluation. A strong parsing result leaves schema-bound behavior untested, including whether values were assigned to the correct keys. If a vendor’s benchmark tested only parsing, its extraction accuracy was never measured.

Why string-similarity metrics reward the wrong behavior

Even within parsing, character fidelity is an incomplete definition of correctness. String-similarity and edit-distance metrics reward output that looks like the reference text. They can therefore favor a result whose words are right but whose relationships are wrong. Structure determines how those words are interpreted downstream. TEDS addresses part of this gap by comparing predicted and reference table trees, capturing whether logical relationships survived.

Horn and Keuper’s formula-parsing benchmark found that character-level matching, or CDM, correlated poorly with human judgments of mathematical equivalence, with r≈0.34 compared with r=0.78 for LLM-as-a-judge. Notation with the same meaning could get a score near zero. Conversely, a syntactically similar formula could score well despite expressing the wrong mathematics.

The same holds for names, quantities, and dates that a system extracts. An honorific, a different unit, or a different date format may keep the same meaning while adding to the edit distance. OmniDocBench, presented at CVPR 2025, illustrates another limitation through its composite of text edit distance, table TEDS, and formula CDM. When scores for different tasks are averaged together, good results on plain text can hide poor results on tables. Such a composite does not reveal whether the component a production workflow depends on is reliable.

Schema-driven evaluation exposes failures that composite scores hide

Schema-driven evaluation uses clear rules for the values a system must return in place of a score for how alike the outputs look. Identifiers require exact matches, quantities allow numeric tolerance, names require semantic equivalence, and arrays require alignment. The schema becomes an executable specification. This approach also separates omission from hallucination rather than treating every incorrect result as interchangeable. Each distinction corresponds to a different kind of downstream failure.

ExtractBench pairs PDFs with JSON Schemas and human-annotated gold labels across economically valuable domains. Its thousands of evaluatable fields span schemas containing tens to hundreds of fields. In baseline evaluations, frontier models produced 0% valid output on a 369-field financial reporting schema.

A parsing leaderboard could not expose that behavior because it did not require the model to satisfy the same structured-output contract. Generic recursive comparisons also miss important distinctions when they apply the same rule to every field. A formatting difference in a name should not be conflated with a wrong identifier, nor should an otherwise correct array fail solely because its ordering differs where ordering is irrelevant.

The counter-intuitive finding was degradation with schema breadth. A document’s page count was therefore a poor proxy for the extraction burden that enterprises needed to test. Marketing centered on long-document handling did not answer whether a system could populate a broad schema reliably. This shifted the meaningful stress test from how much text a model could read to how much structured information it could return correctly.

Silent row drops are a production failure mode that accuracy scores routinely ignored

An invoice pipeline can receive a valid-looking array of line items and still fail reconciliation because some rows are missing. Nothing about the returned rows alone proves completeness. The same failure applies to transactions in a statement or medications in a clinical note. If a benchmark scores only the rows returned, those omissions never enter the calculation. Excluding incomplete runs creates a related distortion by removing evidence of unreliability before calculating average accuracy.

Precision and recall separate these two concerns. Precision measures the correctness of extracted fields, while recall measures completeness against the expected result. JSON F1 accounts for both. A system that extracts five fields perfectly but omits three required fields should not receive the same treatment as a complete result merely because its returned values are correct.

ExtractBench distinguishes omissions from hallucinations at the field level and treats a missing field differently from an explicit null. A skipped field requires a different response from an invented value, while a null must be interpreted against the schema’s expectations. In arrays, alignment and expected-row accounting are therefore essential to evaluating completeness. Without them, structurally valid output can conceal systematically missing data.

How benchmark version changes and dataset contamination further distorted rankings

Metric choice is not the only source of misleading comparisons. A benchmark’s dataset and matching procedure also define its score. When either changes, the result is no longer directly comparable with earlier measurements. OmniDocBench provides a later concrete example of this structural problem. Its major versions changed both document pages and prediction-matching logic, so v1.5 and v1.6 scores are not directly comparable.

The chronology is important to the argument. OmniDocBench’s CVPR 2025 benchmark provides evidence from after 2024 of how changes to benchmark versions make scores difficult to compare. When vendors leave out the version, score differences may come from different pages or matching rules, even if extraction is no better. Cherry-picking a favorable version compounds that ambiguity.

Dataset contamination poses another risk when documents used in a test are also in the training data. High performance can then reflect familiarity with the test documents. Benchmark saturation creates a related interpretive problem. A model scoring 96% on a well-known, saturated benchmark may have learned that benchmark’s particular demands while remaining unprepared for the variation found in production. Such a score is insufficient evidence of reliability on a new workload.

What production pipelines require that no single benchmark captured in 2024

Document workflows span parsing with layout preservation, extraction of schema-bound values, splitting compound files, and classification of document types. Each operation has different failure modes. A field-level extraction score cannot establish whether a compound file was split correctly, just as a parsing score cannot establish schema compliance. End-to-end evaluation also needs completion-rate tracking and support for schema versioning.

Confidence scoring and grounding address the failures that remain after extraction. Source-span traceability connects a returned value to the document evidence behind it, while calibrated confidence signals support review and correction. Without them, a benchmark score says little about whether errors can be audited at volume. Whether a system can work where it is needed also depends on scan-defect robustness, latency budgets, processing volume, and deployment governance.

Healthcare makes the deployment gap especially clear. A workflow that handles protected health information must meet HIPAA requirements for encryption, access control, audit logging, and transmission security. Evaluating those controls, which self-hosted API meets those needs, and whether medical documents can be processed without sending protected health information to third-party servers requires more than an accuracy leaderboard.

Logistics exposes a different gap between average accuracy and operational reliability. Shipment details extracted from bills of lading feed real-time tracking, so errors can compound across downstream systems. Not all fields have equal consequences. Reliability on the fields that trigger downstream actions can be more important than the average across the document. No single 2024 benchmark captured these workflow, reliability, and governance requirements together.

How to evaluate a document extraction API against your actual workload

The lesson for teams selecting an API is to build an evaluation around the operation they intend to deploy. Evaluate parsing and extraction separately, using TEDS for table structural preservation and JSON F1, or a schema-driven equivalent, for extracted-field correctness and completeness. Keep component results visible. This makes it possible to distinguish a reading failure from a structured-output failure.

Use a schema that reflects production breadth. A ten-field test will not expose all the failures that appear with fifty-field or hundred-field schemas. Penalize omissions explicitly by counting missing required fields and dropped rows against the expected result. Incomplete runs must remain visible in the evaluation. Otherwise, the test recreates the same selection bias that made apparently strong accuracy scores poor predictors of production behavior.

Include the actual document mix, especially handwritten annotations on printed forms, non-standard layouts, damaged or low-resolution scans, and multi-language documents. These cases test the variation a vendor’s document set may omit. Test confidence scoring and source-span grounding as first-class criteria as well. An API that returns traceable values with calibrated confidence supports auditing and correction at volume. Without those capabilities, the system has to rely much more on getting every extraction right.

Verify compliance controls independently of accuracy. SOC 2, HIPAA, and zero data retention are distinct deployment criteria, and the applicable requirements should be treated as hard gates for regulated workloads. Purpose-built infrastructure for parsing, extraction, splitting, and classification belongs in the comparison only on the evidence that it meets these operational needs, including schema-driven evaluation, confidence scoring, and compliance controls. A platform label is not that evidence.