Reading Order Reconstruction in Complex Document Layouts

Reading order reconstruction is what makes a document parsing pipeline produce usable output rather than garbage dressed up as usable output, and the industry still treats it as an afterthought bolted onto OCR. The common assumption among engineers building these systems runs something like this: parsing is OCR plus text extraction, and reading order is just a sort step you apply once the real work is done. That assumption is wrong, and it costs teams real production incidents once complex layouts show up in the input stream.

Document parsing pipelines handle complex multi-column layouts by applying reading order reconstruction on top of, or integrated with, their detection stages, and the approach taken determines whether downstream AI consumers receive coherent text or silently scrambled output. A naive top-to-bottom sort interleaves independent column streams mid-sentence, misassigns figure captions, and conflates marginal annotations with body text, without producing any error that an extraction model, retrieval system, or agent workflow can detect. Purpose-built graph-based methods that score candidate transitions using language model signals recover up to 88% macro edge accuracy on multi-column English layouts versus 75% for geometric sorting, and end-to-end architectures like RT-DocLayout address the handoff between detector and serializer by training layout detection and reading order jointly in a single model, eliminating the stage boundary where errors propagate forward. Difficulty-calibrated training objectives like FocalOrder further close the gap on the hardest transitions, the column boundaries, figure captions, and graphical regions where standard training regimes collapse because easy single-column patterns dominate the supervision signal.

Extraction pipelines read that stream and map spans to fields. Retrieval systems chunk that stream and embed the chunks. Agent workflows reason over that stream as if it were a coherent narrative. If the order is wrong, none of those consumers know it, because nothing about a wrongly-ordered token stream looks structurally different from a correctly-ordered one.

Consider a financial report laid out in two columns, where a paragraph from column two gets spliced between two sentences of column one. An extraction model does not throw an exception when this happens. It calmly pulls the wrong number into the wrong field, and the output looks plausible enough that a reviewer skimming a dashboard might never catch it. Or take a retrieval-augmented generation pipeline that pulls a figure caption into context before the data the figure actually describes. The language model reasons over a context window that reads coherently sentence-to-sentence but describes nothing real, because the caption and its referent got separated by a layout the parser couldn't disentangle. An agent filling out a form field can just as easily grab text from a marginal note that happened to sit close, geometrically, to the field it's populating.

That is the core danger: wrong reading order rarely announces itself. It does not crash a pipeline or return an error code. It produces output that looks fine until someone checks the underlying numbers, which makes it one of the hardest classes of bugs to catch in production without deliberate evaluation built around exactly this failure mode. Recent work on parser-oriented refinement makes the point directly: even when the overwhelming majority of the text content on a page gets recognized correctly, errors in layout retention or ordering can still corrupt the final Markdown or JSON that the pipeline emits. Correct text recognition is necessary but nowhere near sufficient.

The structural cases that defeat simple sorting

The simplest reading order method available is a top-to-bottom, left-to-right scan, often implemented as an XY-cut. On a single-column page, it works flawlessly: no model, no training data, no inference cost, just a deterministic sort that runs in milliseconds. It's easy to see why so many production systems still lean on it as a default.

The trouble starts the moment a page stops being a single column. Multi-column layouts put two or more independent narrative streams in the same vertical band on the page, and a naive top-to-bottom sort interleaves them, stitching half a sentence from column one to half a sentence from column two. Marginalia sits geometrically close to body text while belonging to an entirely separate commentary stream. Interlinear glosses occupy physical space between lines of a different logical layer. Figure captions drift above, below, or beside their figures depending on the page designer's whim, and any heuristic built on proximity alone will misassign them constantly. Complex nested tables, where cells span rows and columns in shapes that refuse to form a rectangle, defeat bounding-box ordering because there is no well-defined box order to recover.

The extreme case, and a genuinely useful one for understanding why geometry alone cannot carry this task, is the Glossa Ordinaria manuscript layout: a central text block surrounded by commentaries that wrap around it in regions that are neither rectangular nor convex. There is no sort direction that recovers the intended reading path through a page like that, because the page was never designed with a single sort direction in mind.

The numbers bear this out starkly. On wrap-around Glossa layouts, a graph-based method developed at Tel Aviv University recovers 95% of ground-truth successor edges on average, against roughly 50% for XY-cut, and on a 140-page multi-column English subset of OmniDocBench, the same method reaches 88% macro edge accuracy against 75% for XY-cut Reading Order Inference for Complex Document Layouts. That gap, nearly halved error on the hardest layouts and a meaningful double-digit gap even on conventional multi-column English text, is exactly where production systems quietly bleed accuracy Reading Order Inference for Complex Document Layouts. Layered on top of the structural ambiguity is the physical reality of scanned and photographed documents: paper warping, perspective distortion, and other geometric artifacts compound the difficulty of determining correct reading order well before any ordering logic even runs, because the layout detector itself has to fight noisy geometry first. Geometry alone is underdetermined in precisely the cases that matter most in production: financial filings, scanned legal contracts, historical archives, anything that isn't a clean single-column PDF OmniDocBench (CVPR 2025). That underdetermination is what specialized models exist to resolve.

The two structural failure modes that standard training regimes do not fix

The first is what researchers call Positional Disparity. Models trained on standard document corpora become excellent at recognizing the deterministic start and end regions of a page, the title, the header, the final paragraph, but their performance collapses in the messy intermediate sections where columns merge, tables interrupt text, and captions compete with body copy for sequence position. Easy layout patterns, single-column paragraphs and simple headers, vastly outnumber difficult transitions in most training sets, so uniform supervision drowns the learning signal from the hard cases in the sheer volume of the easy ones, and this structural imbalance is the reason accuracy collapses on hard transitions. Standard training implicitly assumes a constant difficulty distribution across a page. That assumption is false, letting a model post a strong aggregate benchmark score while being reliably wrong on exactly the transitions that decide whether a production extraction is usable.

The second failure mode lives in the seam between detection and parsing, in the pipeline architectures that dominate the space. A layout detector produces a pool of candidate regions, often redundant, often overlapping, and a heuristic non-maximum suppression step combined with rule-based serialization converts that pool into the sequence the parser actually consumes. Duplicated candidates that survive into the parser input create content redundancy and a garbled reading sequence. Heuristic NMS, meanwhile, sometimes keeps an incomplete detection and discards a better-localized alternative. Content gets lost outright rather than merely reordered. A parser-oriented refinement approach published in April 2026 addresses this handoff directly, inserting a lightweight refinement stage between detector and parser and achieving a Reading Order Edit score of 0.024 on OmniDocBench, a meaningfully tighter number than what the raw detector-to-parser handoff produces on its own Parser-Oriented Structural Refinement for a Stable Layout Interface.

What both failure modes share is that they are architecture and training problems, not annotation problems: more annotated pages alone will not fix Positional Disparity or handoff instability. This sets up the next section: the architectural responses to these two failure modes.

RT-DocLayout as a case study in unified end-to-end architectures

The most direct response to the detector-to-parser handoff is to eliminate the handoff. Pipeline architectures propagate detection errors forward through every subsequent stage, and the fragmentation across multiple stages caps both efficiency and accuracy no matter how well each individual stage is tuned. It has since shipped in PaddleOCR under the name PP-DocLayoutV3, with weights available publicly on HuggingFace at PaddlePaddle/PP-DocLayoutV3.

The architecture is a 33-million-parameter model built on RT-DETR, using a single query-based decoder that produces four outputs simultaneously rather than passing intermediate results between separate models RT-DocLayout (ECCV 2026) RT-DocLayout: Real-Time End-to-End Document Layout Analysis with Reading Order in the Wild. A classification head, implemented as a per-query MLP with focal loss to handle the class imbalance that document layouts naturally produce, identifies what each region is. A bounding box regression head locates it. A mask head, inspired by a segmentation-focused detection model, produces binary segmentation masks rather than coarse boxes, which matters specifically for warped, curved, or tilted document images where a rectangular box either includes background noise or clips part of the actual content.

Robustness was tested across six dimensions spanning OmniDocBench v1.5 (Raw) and Real5-OmniDocBench's five challenging scenarios: Scanning, Warping, Screen-Photography, Illumination, and Skew. Joint learning is the mechanism that makes this work. When geometry and reading order train together in the same model, layout structure informs how regions get ordered, and ordering constraints feed back into which layout hypothesis the model should prefer in the first place, a feedback loop a two-stage pipeline structurally cannot form because its stages never talk back to each other. For any team building a production front end, the frames-per-second figure matters as much as the accuracy figure: a model fast enough to run inline, rather than as an offline batch job, makes a usable architecture rather than a research artifact. RT-DocLayout (arXiv 2606.23344, June 2026, PaddlePaddle Team / Baidu Inc.). A compact 33M-parameter model achieves 92.46% overall accuracy at 132.1 FPS on an NVIDIA A100 GPU, outperforming substantially larger models in both accuracy and speed, as sourced from the RT-DocLayout paper (RT-DocLayout (ECCV 2026), RT-DocLayout, OmniDocBench (CVPR 2025)).

FocalOrder's difficulty-calibrated training attack on the Positional Disparity problem

If RT-DocLayout answers the architecture challenge, FocalOrder answers the training challenge.

An adaptive difficulty discovery process uses an exponential moving average to track, during training itself, which transitions the model is struggling with, rather than relying on a fixed, pre-labeled notion of difficulty. Paired with that is a difficulty-calibrated ranking objective that deliberately upweights the training signal coming from those hard transitions, so the volume of easy single-column patterns stops drowning out the harder cases the way it does under uniform supervision. In practice, the hard cases look like graphical regions, complex tables, and the boundary between a figure caption and the body text that follows it, precisely the transitions where Positional Disparity causes standard models to collapse.

On the Comp-HRDoc benchmark, FocalOrder reaches 97.1% REDS on text regions and 91.1% on graphical regions, with a 0.5 percentage point improvement over the previous best method specifically on graphical region serialization FocalOrder (ACL 2026). Half a percentage point sounds unremarkable until you remember which regions it applies to: graphical regions are the hard case in this benchmark by design, so a gain there carries more weight than the same gain would on an easy metric already near ceiling. FocalOrder also outperforms PaddleOCR-VL's pointer-network approach and matches MinerU 2.5, a considerably heavier multi-stage vision-language model pipeline, across both OmniDocBench v1.0 and Comp-HRDoc. A compact, purpose-built model matches or beats general-purpose VLM pipelines several times its size. It suggests that aligning a training objective with the actual structural ambiguity of documents does more work than simply scaling up a general model and hoping the ambiguity gets absorbed somewhere in its parameters. Scale is not the lever here. Calibration is.

Semantic signals and hybrid approaches for low-resource layouts

Neither joint architectures nor difficulty-calibrated objectives help much when there's no large annotated corpus to train on in the first place, which is the reality for historical documents, non-Latin scripts, and specialized domain layouts that never made it into a benchmark like OmniDocBench Reading Order Inference for Complex Document Layouts.

The Tel Aviv University approach is training-free Reading Order Inference for Complex Document Layouts. Each OCR text line becomes a node in a directed candidate-transition graph, and edges between nodes get scored by a weighted ensemble combining a causal language model's conditional likelihood with BERT's next-sentence prediction signal. Global order gets recovered as a degree-constrained directed path cover over that graph, with a max-regret inference rule specifically built to avoid the greedy "edge-theft" that simpler graph traversal methods fall into. LayoutReader, used as a baseline, transferred poorly onto this task because it was built around word-level granularity while the target documents demanded line-level reasoning, a reminder that a model's transfer assumptions have to match the granularity of the thing it's being pointed at. And the graph-based method was separately verified to be mirror-invariant under both horizontal and vertical page reflections, a robustness property that no geometric heuristic can promise by construction.

The second approach, applied to historical Armenian newspapers, took a hybrid path: semantic zone detection paired with a generative LLM handling the actual ordering decision, trained and evaluated on 66 annotated pages Reading order reconstruction in historical Armenian newspapers. That's a genuinely small dataset by modern standards, and yet the hybrid method reduced ordering errors by up to 76% over the strongest geometric baseline tested, and stayed robust across multi-page documents and under the noisy OCR output that historical newsprint reliably produces Reading order reconstruction in historical Armenian newspapers. Neither approach requires the scale of annotation that a from-scratch deep model would need, and that's the real lesson for teams facing a specialized or historical layout: a well-designed inference procedure built around existing language signals can match or beat a purpose-trained model, without the annotation budget a purpose-trained model demands. There are two contrasting approaches from the research.

Reading order accuracy measurement and the limits of current evaluation

OmniDocBench, introduced at CVPR 2025 and maintained by OpenDataLab, is the closest thing the field has to a shared yardstick. It spans 1,651 PDF pages across ten document types, five layout types, and five languages, annotated across 28 block-level categories and 4 span-level categories, and it includes reading order annotations alongside its OCR, table, and layout evaluation tracks OmniDocBench (CVPR 2025).

Comp-HRDoc, the benchmark FocalOrder reports against, takes a narrower but arguably sharper approach: it evaluates REDS, a sequence-level edit distance metric, separately on Text Regions and Graphical Regions, precisely because the split matters given graphical regions are the hard case. ReadingBank, from EMNLP 2021, remains foundational as the first large-scale dataset built for this task, roughly half a million document images with reading order, text, and layout derived straight from WORD document XML metadata. DocLayNet, from 2022 and updated to v1.2 in 2026, is still the largest layout segmentation benchmark by page count, with 80,863 manually annotated pages across 11 layout categories.

None of that changes the core critique. Aggregate benchmark scores can and do mask the exact systematic collapse in complex intermediate layout sections that Positional Disparity describes, because those sections are underrepresented relative to easy single-column pages in most test sets, the same imbalance that distorts training. TEDS for table extraction and Reading Order Edit Distance are the field's primary metrics, and neither one penalizes content that gets silently dropped rather than merely misordered, which is a critical production failure mode. A vendor quoting only an aggregate OmniDocBench number is not lying, but it is presenting a partial picture: on wrap-around Glossa layouts, the graph-based method from Tel Aviv University recovers 95% of ground-truth successor edges on average versus XY-cut's 50%, and on a 140-page multi-column English subset of OmniDocBench, it achieves 88% macro edge accuracy versus XY-cut's 75% Reading Order Inference for Complex Document Layouts. Active maintenance saw v1.6 add 296 new pages covering complex nested tables, dense mathematical formula layouts, and unconventional layout structures, while the April 2026 update to v1.7 added the Qianfan-OCR leaderboard and skills-based evaluation (dots.ocr, OmniDocBench (CVPR 2025)). The v1.5-to-v1.6 update introduced Multi-Granularity Adaptive Matching (MGAM) to eliminate matching bias.

How does reading order accuracy affect RAG pipelines, extraction, and agent workflows?

Everything upstream in this piece converges on one operational point: a document parsing system is only as trustworthy as its reading order, and that trust has to be measured directly rather than assumed from an aggregate score. A retrieval pipeline built on top of a parser with weak reading order will chunk incoherent text, embed it, and retrieve it with full confidence, because nothing in the embedding step knows the source text was scrambled. An agent filling structured fields from a parsed document inherits whatever ordering mistakes the parser made, with no mechanism downstream to catch a caption that landed next to the wrong figure or a column that got spliced mid-sentence.

The architectures and training methods surveyed here, joint detection-and-ordering models like RT-DocLayout, difficulty-calibrated objectives like FocalOrder, and semantic graph or hybrid LLM methods for low-resource layouts, all point toward the same conclusion from different angles. Geometry gets a document parser most of the way there on the easy majority of pages, and it fails in specific, identifiable, and now well-studied ways everywhere else.