LayoutReader: Pre-training of Text and Layout for Reading Order Detection

1 Introduction

• Existing OCR ordering heuristics can fail on multi-column templates, forms, and invoices, where incorrect sequences can harm document understanding.The introduction specifically notes unacceptable results for receipt and invoice information extraction.Source: §1 · p. 1
• Earlier methods relied on small, domain-restricted datasets or empirical rules, while producing in-house data for deep learning was labor-intensive.These constraints made the approaches difficult to adapt to real-world reading-order problems.Source: §1 · p. 1
• ReadingBank contains 500,000 real-world document images, with reading-order labels extracted from WORD XML and word boxes aligned from PDFs.A coloring scheme aligns XML text with PDF bounding boxes.Source: §1 · pp. 1–2
• LayoutReader performs almost perfectly in reading-order detection and improves line ordering for open-source and commercial OCR engines.The model uses both text and layout information in a seq2seq approach.Source: §1 · p. 2
• The authors present ReadingBank as a large-scale benchmark and LayoutReader as a reading-order model, with dataset and model release planned to support further research.The introduction describes ReadingBank as the first large-scale benchmark for this task.Source: §1 · p. 2

2 Problem Formulation

• Reading order is a well-organized, readable word sequence, which can be difficult to obtain from images containing tables or multiple columns.Most OCR engines do not provide the proper order in these cases.Source: §2 · p. 2
• The task represents each document token as a word with bounding-box coordinates, then sorts the tokens into natural reading order.The coordinates identify each box’s left-top and right-bottom corners.Source: §2 · p. 2
• Figure 2 labels (w, i) as a word and its appearance index, and lists word, color, bounding-box, height, and width fields in the representation.Source: Figure 2 · p. 3

3.1 Document Collection

• The dataset targets English reading-order detection: it keeps pages with more than 50 words from 210,000 English Word documents, then randomly selects 500,000 pages.Collection considered the robots exclusion standard and public-domain licensing.Source: §3.1 · p. 3

3.2 Reading Sequence Extraction

• ReadingBank’s Reading Sequence is the word order extracted from DocX XML by traversing paragraphs line by line and table cells cell by cell.The sequence is extracted before layout information is aligned to each word.Source: §3.2 · p. 3

3.3 Layout Alignment with Coloring Scheme

• To align repeated words with PDF coordinates, the pipeline labels each occurrence, maps its appearance index to an RGB color, and matches the colored PDF word to its DocX counterpart.The appearance index makes each word-and-index pair unique; the mapping function is one-to-one.Source: §3.3 · pp. 3–4

3 ReadingBank

• ReadingBank contains 500,000 document pages with images and word sequences paired with coordinates in reading order.The training, validation, and test sets contain 400,000, 50,000, and 50,000 pages, respectively.Source: §3 · p. 3, §3.4 · p. 4

3.4 Dataset Statistics

• The dataset is split 8:1:1, and the word-count and BLEU-score distributions are consistent across subsets to support data balance.BLEU scores measure how difficult samples are relative to left-to-right and top-to-bottom ordering, using ground-truth reading order as reference.Source: §3.4 · p. 4, Table 1 · p. 4

• LayoutReader uses LayoutLM to encode text and layout information, then modifies generation to produce the reading-order sequence.Source: §4 · p. 4, Figure 3 · p. 5
• The self-attention mask lets source tokens attend to one another while preventing target tokens from attending to rightward context.Source: §4 · p. 4, Figure 3 · p. 5
• The decoder restricts prediction candidates to source-segment indices, with probabilities calculated from source embeddings, hidden states, and a time-step bias.Source: §4 · p. 5

5 Experiments

• The experiments cover ReadingBank reading-order detection, input-order variation, OCR-engine adaptation, and a real-world case study.Implementation uses base pre-trained models, four Tesla V100 GPUs, batch size 4 per GPU, and three training epochs.Source: §5 · p. 5, §5.2 · pp. 5–6

5.1 Comparative Methods

• The comparisons include a heuristic that sorts words left-to-right and top-to-bottom, plus text-only and layout-only LayoutReader variants.Text-only substitutes BERT or UniLM for LayoutLM; layout-only removes token embeddings and uses positional layout information.Source: §5.1 · p. 5

5.3 Evaluation Metrics

• The experiments report average page-level BLEU, measuring within-page n-gram overlap, and ARD, which allows omissions but penalizes them.ARD measures relative distance between common elements in the reference and generated sequences.Source: §5.3 · p. 6

5.4 Reading Order Detection

• 0.2847 higher average page-level BLEU and 6.71 lower ARD: LayoutReader surpasses the comparative methods on ReadingBank’s reading-order test set.Layout-only exceeds text-only by about 0.1 BLEU and about 9.0 ARD; text-only’s ARD drops because omissions are penalized.Source: §5.4 · p. 6, Table 2 · p. 6

5.5 Input Order for Training and Testing

• LayoutReader and layout-only are more robust to shuffled training tokens; all three comparative models perform well with normally ordered evaluation inputs.The study varies the proportion of shuffled training samples; layout information remains consistent under shuffling.Source: §5.5 · pp. 6–7, Table 3 · p. 7
• With r = 0% shuffled training inputs, LayoutReader drops on token-shuffled evaluation inputs, which the authors attribute to overfitting the familiar order.The evaluation order is entirely unseen to models trained without shuffled inputs.Source: §5.5 · p. 7, Table 4 · p. 7

5.6 Adaption to OCR Engines

• LayoutReader adaptation greatly improves text-line ordering for both Tesseract and a commercial OCR engine.The experiments adapt token-level reading order to lines and compare with OCR-engine outputs.Source: §5.6 · p. 7, Table 5 · p. 7, Table 6 · p. 7

5.7 Case Study

• Figure 4 compares commercial OCR and adapted LayoutReader line orders against ReadingBank-adapted ground truth, marking correct and incorrect indices in green and red.The authors report that LayoutReader adaptation improves the OCR engine’s text-line ordering in this example.Source: §5.7 · p. 8, Figure 4 · p. 8

6 Related Work

• Reading-order systems used qualitative rectangle relations, Bayesian classifiers, and ILP ordering predicates, followed by domain-specific knowledge and unsupervised abstract argumentation.Source: §6 · p. 8
• Reading-order detection progressed from qualitative-rectangle rules to Bayesian chain reconstruction, ILP ordering predicates, and unsupervised argumentation.Source: §6 · p. 8
• Li et al. (2020a) used a Graph Neural Network with visual-layout attention and an attention-based sequence decoder to reorder OCR text.Source: §6 · p. 8

7 Conclusion

• ReadingBank contains 500,000 document images for reading-order detection.Source: §7 · p. 8
• LayoutReader, built on pre-trained LayoutLM, significantly outperforms left-to-right and top-to-bottom heuristics and several strong baselines.Source: §7 · p. 8
• LayoutReader can be adapted to OCR engines to improve reading order, and the authors plan to release the dataset and model publicly.Source: §7 · p. 8
• Future work will investigate larger synthesized data, reading-order labels for scanned documents, and whether a few human-labeled samples suffice for domain-specific detection.Source: §7 · p. 8

A Ethical Consideration

• The authors say they crawl publicly available data under website robots exclusions and retain pages they have permission to edit and redistribute.Source: A Ethical Consideration · p. 9
• About 100 pages will be public after manual checking and redaction; access to the full dataset requires further permission.The authors state that dataset data will be protected by an Apache 2.0 license.Source: A Ethical Consideration · p. 9
• The authors expect broad crawl coverage without area preferences or limitations to support generalization to other visually-rich document images.Source: A Ethical Consideration · p. 9