Silent Row Dropping in Large-Array Document Extraction
Missing table rows can pass validation while corrupting downstream analytics.
Silent row dropping in large-array document extraction produces output that is structurally valid but silently incomplete. It is a distinct failure mode. Production-grade pipelines must check that every expected row made it through extraction, because correct values, valid JSON, and high confidence cannot prove that a table is complete.
Why large-array extraction is structurally different from ordinary field extraction
Field extraction asks whether a value is correct, while large-array extraction also asks whether all the records that should be there are there. A wrong invoice total, a misread date, or a blank field gives validation something to inspect. A missing array row offers no equivalent object to check. The pipeline can transcribe every returned row perfectly and still deliver a materially wrong result.
For example, a table containing a few dozen rows can have exactly the same JSON structure as the complete table of 78 rows. Both can pass schema validation because the schema describes what a row contains without establishing how many rows the source contains. Correctness therefore depends on count as well as content. This distinction makes the array, rather than the individual field, the essential unit of completeness testing.
The three mechanisms that cause rows to disappear without a trace
The first mechanism is output-token truncation. A completion budget suited to prose can run out before a long array is finished, leaving a shorter table without a clear failure reaching the caller. The number of surviving rows then depends on per-row verbosity. RAGStack’s wilke/ragstack PR #553, September 2026, documents a case in which raising the source count from 10 to 25 shrank the output table to two rows. Richer context consumed more tokens per row, so additional source material produced less tabular output.
The second mechanism is a retry loop that replays the same budget constraint. Detecting a stop_reason of max_tokens does not resolve truncation when the next attempt retains the same insufficient cap. If the response ends mid-array, JSON parsing can return nothing and extraction can fall back to heuristics for that batch. The caller then receives an incomplete result, with only a warning in a log indicating what happened. Tokenmizer GitHub issue #89, October 2026, documents how reasoning-token costs further reduce the budget available for the JSON that follows.
The third mechanism is structural confusion during parsing and layout analysis. Line-based detection can miss tables without borders, coordinate clustering can fail when columns are uneven, and a mix of prose and tables can throw off layout detection. Page-by-page parsing adds another problem by losing column alignment, treating repeated headers as data, or appending sub-tables as flat rows. Traditional OCR captures text without necessarily preserving the relationship between a parent cell and its child table, so flattening can discard overflow rows. These mechanisms operate independently. An extraction can lose rows even when another layer behaves correctly.
How input-side chunking loses rows before the model ever sees them
Input-side chunking is a fourth way to lose rows, separate from both layout recognition and output truncation. A parser may capture a table, yet fixed-size chunking can separate its section headers from the data beneath them. Retrieval then selects fragments of the logical table. The model cannot extract rows that never enter its context, even if it has plenty of output capacity left.
FinMMEval 2026 shows this failure in a question about Johnson & Johnson cash flow. Inspection of the top-five retrieved chunks found that all five were pipe-delimited table header rows from the SEC filing. The model received no dollar figures because similarity-based retrieval selected only the section headers. Better embeddings or re-ranking do not repair the mismatch between chunk format and query intent. Raising max_tokens or checking finish_reason addresses output loss, but leaves this upstream failure untouched.
Why the consequences scale faster than teams expect and hit hardest in regulated industries
In downstream validation, a wrong field can fail a rule and go to human review, while a missing row skips those checks entirely. Every system or agent consuming the incomplete table can therefore inherit the omission without receiving a failure signal. The cost is no longer confined to repeating extraction, because the missing data has already shaped downstream reports and analyses.
In finance, that propagation creates concrete regulatory exposure. VAT returns and SAF-T filings can face penalties or rejected submissions from a single missing field. A missing transaction row also corrupts reconciliation reports and creates audit liability that downstream corrections cannot retrospectively erase. The table may look clean at each handoff while remaining incomplete relative to the source. As the number of documents grows, the same failure can go unmeasured and affect more records while each output looks just as credible.
Why standard accuracy benchmarks cannot detect silent row dropping
An evaluation that measures only returned fields cannot establish whether all source rows survived. Its denominator contains only the rows that appeared. Perfect transcription of that subset can therefore yield an excellent score despite substantial omission. High per-row confidence has the same limitation because it describes only the returned content.
OmniDocBench uses 1,403 PDF files with detailed annotations and evaluates table extraction with TEDS, or Tree Edit Distance Similarity. TEDS compares predicted and ground-truth table trees, penalizing missing or extra rows as well as transcription errors. Its structural score can therefore detect row loss, but it is not a dedicated verification of complete array length.
DTBench, arXiv:2602.13812, February 2026, tests multiple Doc2Table capabilities, including conflict resolution when document segments provide conflicting values for an attribute. That is a meaningful test, but it does not measure completeness. The 2026 systematic literature review case, arXiv:2607.21920, shows what can be lost when completeness goes unmeasured. One-shot single-LLM extraction got only roughly half of OS-related cells correct and both hallucinated rows and left rows out. Of a dozen reportable subgroup rows, several contained errors and two were omitted outright, losses that scoring only the present cells would not adequately expose.
What a completeness-aware benchmark actually has to measure
A completeness-aware benchmark must treat array length as an explicit correctness dimension. At minimum, it compares the extracted count with a ground-truth row count and penalizes omissions directly. A partial extraction from a 78-row source must not pass as successful merely because its remaining values are accurate. Under this evaluation lens, dropped rows should weigh more heavily than minor transcription errors in an otherwise complete table, because omissions can bypass downstream validation.
Count alone, however, is not the entire specification. An array must also preserve the source records and conform to the required schema, rather than replace missing rows with hallucinated ones. LongArray-Extract addresses this broader requirement through its primary metric of per-document extraction accuracy across complete, schema-faithful arrays. Its primary metric is not row completeness specifically, but completeness is part of what makes the document extraction correct. Structured evaluations also need to track accuracy as row count grows, since short tables do not expose the budget and chunking failures that emerge with larger arrays.
How to detect silent row dropping in a pipeline that is already running
Operational detection begins at the completion boundary. Every call should be inspected for a stop_reason of max_tokens, and a response that hits the ceiling should be treated as potentially truncated. This check catches a budget signal that an otherwise usable result can conceal. It cannot prove that a response that finishes cleanly is complete, because rows can be lost during layout and chunking before generation begins.
Expected row counts provide a separate anchor. Where document metadata or a prior parse establishes the source count, post-extraction validation should compare that count with the returned array length and flag discrepancies. Relational checks add coverage by testing whether invoice line-item sums match the stated total, transaction counts match batch headers, and date ranges are contiguous. These checks depend on relationships across the array rather than the apparent quality of isolated cells. They can reveal omissions that confidence scores miss, although they are not a substitute for an expected row count.
The validation layer should run checks at the post-parse, post-extract, and pre-handoff stages. Those boundaries place checks around the transformations through which incomplete data could otherwise pass unchecked. Confidence thresholds can route low-scoring fields to human review, but they address errors in fields that exist. Together, finish-reason inspection, count anchoring, and relational validation expose different failure modes without treating any one check as proof of success.
Pipeline architecture choices that prevent row dropping rather than just detecting it
Semantic chunking that retains a table’s rows in a single context window removes the header-versus-data split behind the FinMMEval retrieval failure. That design’s protection is limited to chunking loss. Multi-page tables must likewise be kept together as logical units, with the links between pages intact. Preserving those relationships avoids asking a later extraction stage to reconstruct context that an earlier stage removed.
Budgets should replace global defaults with configurable per-template settings based on expected maximum row count and per-row verbosity. RAGStack’s fix used configurable output limits, a truncated flag on templated /v1/query responses, and regression tests pinning every shipped table template to a real row-count ceiling. This approach tests how much output capacity the workload needs instead of assuming that a response budget sized for prose will be enough for an array. A truncation flag should also be a first-class API contract so callers can distinguish completion from budget exhaustion.
Parallel extraction with consensus voting adds another check at the orchestration layer. Multiple LLM backends can process identical documents concurrently, with agreed values retained and discrepancies sent for human review. A truncated table will disagree with an extractor that returns more of the source rows, making a previously silent omission visible before acceptance. Consensus does not remove the need for sound chunking or sufficient budgets, because agreement alone does not establish a source-grounded count. Its role is to expose divergent extraction results within an architecture that also controls the causes of loss.
What silent row dropping means for teams evaluating or replacing their extraction pipeline
A pipeline that has never been tested for row completeness has an unknown failure rate on that dimension. Evaluation of the current system should establish whether it returns a row count, distinguishes max_tokens from clean completion, and validates array length against a source document count. A returned count makes output size visible, but only a source comparison establishes whether that size is correct. Without these distinctions, accuracy claims may describe only the rows the pipeline returned.
Replacement evaluation should examine whether omissions receive explicit penalties or disappear behind cell-level scores. It should also establish whether multi-page tables are handled as logical units and whether truncation is exposed as a first-class signal. In financial services, healthcare, logistics, and real estate, these are operational and compliance requirements because tables carry transactional, clinical, and regulatory data. The practical next step is to test the pipeline against source-grounded row counts and inspect its failure signals before relying on its accuracy claims.
