PaperlayerView exact revision on arXiv

Source-linked AI summary

DocLayout-YOLO: Enhancing Document Layout Analysis through Diverse Synthetic Data and Global-to-Local Adaptive Perception

Zhiyuan Zhao, Hengrui Kang, Bin Wang, Conghui He

arXiv:2410.12628v1cs.CV

TL;DR

Document layout analysis must balance accuracy with speed, and existing methods still struggle across diverse document formats. DocLayout-YOLO combines diverse synthetic pretraining data with a global-to-local module, and experiments report improvements in speed and accuracy over existing methods.

  • Problem

    Existing document layout methods struggle to balance speed and accuracy across diverse document formats.

    Source: §1 · p. 1

  • Method

    DocLayout-YOLO combines Mesh-candidate BestFit-generated DocSynth-300K pretraining data with a Global-to-Local Controllable Receptive Module for document layout analysis.

    Source: §1 · p. 2, §3 · pp. 3–4

  • Results

    Across downstream datasets, DocLayout-YOLO significantly outperforms existing methods in speed and accuracy and achieves superior performance on three of four DocStructBench subsets.

    Source: §5.4.2 · p. 7, §6 · p. 9

  • Takeaways & Limitations

    The authors plan to open-source DocSynth-300K to support further research in document layout analysis.

    Source: §1 · p. 2

Abstract

from arXiv · show

Document Layout Analysis is crucial for real-world document understanding systems, but it encounters a challenging trade-off between speed and accuracy: multimodal methods leveraging both text and visual features achieve higher accuracy but suffer from significant latency, whereas unimodal methods relying solely on visual features offer faster processing speeds at the expense of accuracy. To address this dilemma, we introduce DocLayout-YOLO, a novel approach that enhances accuracy while maintaining speed advantages through document-specific optimizations in both pre-training and model design. For robust document pre-training, we introduce the Mesh-candidate BestFit algorithm, which frames document synthesis as a two-dimensional bin packing problem, generating the large-scale, diverse DocSynth-300K dataset. Pre-training on the resulting DocSynth-300K dataset significantly improves fine-tuning performance across various document types. In terms of model optimization, we propose a Global-to-Local Controllable Receptive Module that is capable of better handling multi-scale variations of document elements. Furthermore, to validate performance across different document types, we introduce a complex and challenging benchmark named DocStructBench. Extensive experiments on downstream datasets demonstrate that DocLayout-YOLO excels in both speed and accuracy. Code, data, and models are available at https://github.com/opendatalab/DocLayout-YOLO.

1 INTRODUCTION

DocLayout-YOLO combines diverse synthetic document pretraining with document-specific network design to improve layout analysis across varied documents. It reports strong benchmark accuracy alongside real-time inference speed.

Introduction

  • DocLayout-YOLO matches YOLOv10 (Wang et al., 2024a) in speed and surpasses DINO-4scale (Zhang et al., 2023a), YOLOv10, LayoutLMv3 (Huang et al., 2022), and DiT-Cascade (Li et al., 2022) in accuracy.The paper frames the work around the speed–accuracy trade-off between visual-only methods and slower multimodal methods.Source: §1 · pp. 1–2
  • Mesh-candidate BestFit synthesizes layouts from text, images, and tables as a 2D bin-packing problem, creating the diverse DocSynth-300K pretraining corpus.The paper presents this pipeline as a response to unimodal models overfitting to datasets dominated by single document types.Source: §1 · p. 2
  • The Global-to-Local Controllable Receptive Module (GL-CRM) is designed to detect document elements spanning scales from single-line titles to full-page content.It adapts YOLO’s network to document layouts and is intended to improve detection accuracy across these scale variations.Source: §1 · p. 2
  • DocLayout-YOLO achieves mAP scores of 70.3%, 79.7%, and 78.8% on D4LA, DocLayNet, and DocStructBench, respectively, at 85.5 FPS.The paper describes these results as state of the art and enabling real-time layout analysis on diverse documents.Source: §1 · p. 2

2 RELATED WORK

Prior work approaches document layout analysis through either visual-only detection or text-visual pre-training. Existing datasets and generation methods leave gaps in diversity, volume, and annotation detail, motivating broader data and synthesis approaches.

2.1 DOCUMENT LAYOUT ANALYSIS APPROACHES

  • Unimodal methods apply generic object detectors to layout analysis, while multimodal methods align text and visual features through pre-training, including LayoutLM, DiT, and VGT.LayoutLM uses varied pre-training objectives, DiT uses self-supervised pre-training on extensive document datasets, and VGT adds grid-based textual encoding.Source: §2.1 · p. 3

2.2 DOCUMENT LAYOUT ANALYSIS DATASETS

  • Existing datasets vary in scale and annotation, but remain limited in diversity, volume, and granularity; most document-generation methods focus on academic papers.Examples range from PubLayNet’s 360,000 journal pages and DocBank’s 500,000 weakly supervised arXiv pages to DocLayNet’s 80,863 diverse pages; M6Doc is not open source due to copyright restrictions.Source: §2.2 · p. 3
  • Figure 2 depicts Mesh-candidate BestFit as preprocessing a category-wise element pool, then iteratively searching for an optimal candidate-grid match during layout generation.Source: Figure 2 · p. 3

3 DIVERSE DOCSYNTH-300K DATASET CONSTRUCTION

DocSynth-300K addresses the homogeneity of existing unimodal pre-training datasets by synthesizing documents with diverse elements and layouts. Its Mesh-candidate BestFit pipeline aims to balance layout variety with organization.

3 DIVERSE DOCSYNTH-300K DATASET CONSTRUCTION

  • Mesh-candidate BestFit synthesizes DocSynth-300K with diverse document elements and layouts, enhancing model performance across real-world document types.Source: §3 · pp. 3–4

3.1 PREPROCESSING: ENSURING ELEMENT DIVERSITY

  • M6Doc test supplies 74 element categories from about 2,800 pages; rare categories with fewer than 100 elements are augmented to diversify the element pool.Source: §3.1 · p. 4

3.2 LAYOUT GENERATION: ENSURING LAYOUT DIVERSITY

  • Mesh-candidate BestFit iteratively matches size-stratified element candidates to valid grids by maximum fill rate, producing diverse, organized layouts that adhere to alignment and density principles.The process balances layout randomness with aesthetics; the generated documents support adaptation to varied real-world document types.Source: §3.2 · pp. 4–5

4 GLOBAL-TO-LOCAL MODEL ARCHITECTURE

DocLayout-YOLO’s GL-CRM addresses document elements’ wide scale variation through multi-granularity feature integration and hierarchical perception from whole-page context to local semantics.

4.1 CONTROLLABLE RECEPTIVE MODULE

  • CRM extracts features at multiple granularities using shared convolution and varying dilation rates, then learns to weight and fuse them with a gated, residual pathway.It is inserted into a CSP bottleneck; kernel size k and dilation rates d control feature granularity and scale.Source: §4.1 · p. 5

4.2 GLOBAL-TO-LOCAL DESIGN

  • The global-to-local design assigns large-kernel CRM to shallow whole-page features, smaller-kernel CRM to intermediate sub-blocks, and a lightweight bottleneck to deep local semantics.The shallow stage uses k = 5, d = 1, 2, 3; the intermediate stage uses k = 3, d = 1, 2, 3.Source: §4.2 · pp. 5–6

5 EXPERIMENTS

Experiments evaluate DocLayout-YOLO on D4LA, DocLayNet, and DocStructBench. The reported results show accuracy advantages over unimodal and multimodal comparators alongside substantial inference-speed advantages.

5.1 EXPERIMENTAL METRICS AND DATASETS

  • Evaluation reports COCO-style mAP for accuracy and FPS for speed on D4LA and DocLayNet, and evaluates DocStructBench’s four subsets separately.D4LA has 11,092 noisy images across 27 categories; DocLayNet has 80,863 pages across 11 categories. DocStructBench has 7,310 training images and 2,645 test images, manually annotated across 10 categories.Source: §5.1 · p. 6

5.2 COMPARISON DLA METHODS & DATASETS

  • Comparisons include multimodal LayoutLMv3, DiT-Cascade, and VGT, unimodal DINO-4scale-R50, and pre-training datasets PubLayNet and DocBank.The DLA datasets are compared with DocSynth-300K as pre-training sources.Source: §5.2 · p. 6

5.3 IMPLEMENTATION DETAILS

  • DocLayout-YOLO is pre-trained at a 1600-pixel long side for 30 epochs and fine-tuned using dataset-specific image sizes and learning rates.Pre-training uses batch size 128 and learning rate 0.02; training uses 8×A100 GPUs with patience of 100 epochs.Source: §5.3 · pp. 6–7

5.4.2 COMPARISON WITH CURRENT DLA METHODS

  • DocLayout-YOLO reaches 70.3 mAP on D4LA versus VGT’s 68.8, beats DINO by 2.0 on DocLayNet, and leads three of four DocStructBench subsets, placing second on Market Analysis.The authors attribute the Market Analysis result to DocSynth-300K pre-training being insufficient for the most complex layouts.Source: Table 2 · p. 7, §5.4.2 · p. 7, Table 3 · p. 8

5.4.1 EFFECTIVENESS OF PROPOSED OPTIMIZATION STRATEGIES

  • DocSynth-300K pre-training improves results by 1.2 on D4LA and 2.6 on DocLayNet, while CRM plus pre-training improves over baseline YOLO-v10 across reported benchmarks.Pre-training also improves results on all four DocStructBench subsets.Source: §5.4.1 · p. 7, Table 1 · p. 7

5.4.2 COMPARISON WITH CURRENT DLA METHODS

  • DocLayout-YOLO is more efficient than current DLA methods: despite a slight speed decrease versus baseline YOLO-v10, it achieves 14.3× faster FPS than best multimodal DIT-Cascade-L and 3.2× faster FPS than best unimodal DINO.Source: §5.4.2 · p. 8

5.5.1 COMPARISONS WITH DIFFERENT DOCUMENT SYNTHETIC METHODS

  • DocSynth-300K pre-training improves results across all four DocStructBench subsets, while random layouts are suboptimal and diffusion layouts offer limited diversity.Diffusion layouts improve on Academic and Textbook, whereas DocSynth-300K shows better generalization across document types; visualizations compare images generated by the methods.Source: §5.5.1 · pp. 8–9, Figure 7 · p. 9

5.5.2 COMPARISONS WITH PUBLIC DOCUMENT PRE-TRAINING DATASETS

  • Compared with public pre-training datasets, DocSynth-300K delivers comprehensive improvements and performs better on most downstream datasets.The paper attributes limitations of PubLayNet and DocBank to narrow element and layout diversity, and reports overfitting on M6Doc due to limited size.Source: Table 4 · p. 8, §5.5.2 · p. 9

6 CONCLUSION

DocLayout-YOLO combines diverse synthetic pre-training data with hierarchical global-to-block-to-local perception, and the paper reports that it outperforms existing document layout analysis methods in speed and accuracy. Ablations support the GL-CRM’s global-to-local design, with different levels benefiting object scales differently.

6 CONCLUSION

  • GL-CRM ablations show that global-level features improve medium- and large-object detection, while block-level features yield the largest improvement for medium objects.The authors report that the results validate GL-CRM’s global-to-local design.Source: §6 · p. 9
  • DocLayout-YOLO significantly outperforms existing document layout analysis methods in both speed and accuracy, combining DocSynth-300K pre-training with GL-CRM model optimization.The Mesh-candidate BestFit methodology creates the diverse DocSynth-300K pre-training dataset; GL-CRM applies hierarchical global-to-block-to-local perception.Source: §6 · p. 9

A APPENDIX

The appendix provides details on DocStructBench and Mesh-candidate BestFit, evaluates DocSynth-300K design principles and pre-training data volume, and shows real-world detection examples from DocLayout-YOLO.

Appendix A.1

  • Appendix A.1 provides detailed information on the in-house evaluation dataset DocStructBench.Source: A APPENDIX · p. 12

Appendix A.2

  • Appendix A.2 details Mesh-candidate BestFit and presents additional visualizations of generated documents.Source: A APPENDIX · p. 12

Appendix A.3

  • Appendices A.3.1 and A.3.2 evaluate DocSynth-300K from a design-principle perspective and ablate pre-training data volume.Source: A APPENDIX · p. 12

Appendix A.4

  • Appendix A.4 demonstrates DocLayout-YOLO detection examples across multiple kinds of real-world documents.Source: A APPENDIX · p. 12

A.1 DOCSTRUCTBENCH DETAILS

DocStructBench is a diverse, complex document-structure dataset with four subsets and details on document sources, splits, and annotated component categories.

A.1 DOCSTRUCTBENCH DETAILS

  • DocStructBench contains 9,082 training images and 2,232 test images across Academic, Textbook, Market Analysis, and Financial subsets.Table 7 details the document sources and train/test split for the benchmark.Source: §A.1 · p. 12, Table 7 · p. 12
  • The benchmark’s annotated document-component categories and instance counts are detailed in Table 8.Source: §A.1 · p. 12, Table 8 · p. 12

A.2 MESH-CANDIDATE BESTFIT · A.2.1 ALGORITHM OF LAYOUT GENERATION

The layout-generation algorithm iteratively matches candidates to document grids, inserts the best match, and repeats until it reaches a set element-count threshold.

A.2.1 ALGORITHM OF LAYOUT GENERATION

  • The algorithm inserts each best-matching candidate and repeats the search until the layout reaches N = 15 elements, using a matching threshold frthr of 10−4.The threshold of 15 elements is set empirically.Source: §A.2.1 · p. 12

A.2.2 DATA AUGMENTATION PIPELINE … A.4 DETECTION EXAMPLES

The appendix details rare-category augmentation and layout-generation constraints, then evaluates synthetic layouts, pre-training data volume, and detection examples across document types.

A.2.2 DATA AUGMENTATION PIPELINE

  • Rare-category augmentation applies geometric, photometric, cropping, edge, and distortion transformations to document elements.The pipeline uses horizontal and vertical flips and brightness/contrast changes at probability 0.5; cropping at 0.7 within area range 0.5–0.9; Sobel edge extraction at 0.2; and slight elastic transformation with Gaussian noise.Source: §A.2.2 · pp. 12–13

A.2.3 OTHER DETAILS

  • Layout generation repeatedly selects the best-matching candidate-grid pair by fill rate, stops when no valid pair remains, and caps small elements at five.The small-element restriction is intended to prevent layouts from violating conventional aesthetic standards.Source: §A.2.3 · p. 13

A.2.4 MORE VISUALIZATION EXAMPLES

  • Generated examples demonstrate category-rich layouts ranging from dense pages with many small elements to sparse pages with a few large elements.The examples are described as similar in layout diversity to those produced by diffusion-based models.Source: §A.2.4 · p. 13, Figure 8 · p. 14

A.3.1 EVALUATION OF SYNTHETIC DOCUMENT FROM DESIGN PRINCIPLE PERSPECTIVE

  • Mesh-candidate BestFit significantly outperforms diffusion and random methods on both alignment and density metrics.Lower Align indicates better alignment, while larger Density indicates a more compact layout.Source: Table 9 · p. 14, §A.3.1 · p. 14

A.3.2 ABLATIONS ON PRETRAINING DATA VOLUME

  • Pre-training performance improves consistently below 100K examples, fluctuates at 200K, and reaches its peak at 300K.The ablation pre-trained YOLO-v10 on 0–500K generated examples and fine-tuned it on D4LA.Source: §A.3.2 · pp. 14–15, Figure 9 · p. 15

A.4 DETECTION EXAMPLES

  • Fine-tuned on DocStructBench, DocLayout-YOLO adapts to multiple downstream document types, including academic, textbook, market-analysis, and financial documents.The detection examples are shown in Figures 10 and 11.Source: §A.4 · p. 15, Figure 10 · p. 15, Figure 11 · p. 15

Related papers

Loading 2410.12628v1…