PaperlayerView exact revision on arXiv

Source-linked AI summary

DocLayNet: A Large Human-Annotated Dataset for Document-Layout Analysis

Birgit Pfitzmann, Christoph Auer, Michele Dolfi, Ahmed S Nassar, Peter W J Staar

arXiv:2206.01062v1cs.CVcs.LG

TL;DR

Document-layout analysis needs datasets that reflect the wide variability of real PDFs, which existing large automated datasets insufficiently capture. The paper introduces DocLayNet, a large human-annotated dataset with diverse pages, detailed labels, and redundant annotations, and evaluates models and dataset-design choices. DocLayNet-trained models provide more robust layout recovery, while model performance remains below human agreement and language coverage is uneven.

  • Problem

    Existing large layout datasets have limited layout variability because they are sourced mainly from scientific articles, reducing their suitability for challenging and diverse documents.

  • Method

    The paper constructs DocLayNet from diverse documents using human bounding-box annotations, 11 layout classes, redundant annotations, and standardized benchmark splits, then evaluates object-detection baselines and dataset effects.

  • Results

    DocLayNet-trained models provide overall more robust layout recovery than models trained on PubLayNet or DocBank, while object-detection baselines are 6–10% below pairwise human-annotation mAP.

  • Takeaways & Limitations

    DocLayNet offers a challenging public benchmark for improving document-layout analysis across varied publication and typesetting styles.

  • Takeaways & Limitations

    Nearly 95% of DocLayNet documents are in English, and language was not controlled, which may challenge methods that exploit textual features.

Abstract

from arXiv · show

Accurate document layout analysis is a key requirement for high-quality PDF document conversion. With the recent availability of public, large ground-truth datasets such as PubLayNet and DocBank, deep-learning models have proven to be very effective at layout detection and segmentation. While these datasets are of adequate size to train such models, they severely lack in layout variability since they are sourced from scientific article repositories such as PubMed and arXiv only. Consequently, the accuracy of the layout segmentation drops significantly when these models are applied on more challenging and diverse layouts. In this paper, we present \textit{DocLayNet}, a new, publicly available, document-layout annotation dataset in COCO format. It contains 80863 manually annotated pages from diverse data sources to represent a wide variability in layouts. For each PDF page, the layout annotations provide labelled bounding-boxes with a choice of 11 distinct classes. DocLayNet also provides a subset of double- and triple-annotated pages to determine the inter-annotator agreement. In multiple experiments, we provide baseline accuracy scores (in mAP) for a set of popular object detection models. We also demonstrate that these models fall approximately 10\% behind the inter-annotator agreement. Furthermore, we provide evidence that DocLayNet is of sufficient size. Lastly, we compare models trained on PubLayNet, DocBank and DocLayNet, showing that layout predictions of the DocLayNet-trained models are more robust and thus the preferred choice for general-purpose document-layout analysis.

1 INTRODUCTION

Document-layout analysis remains difficult because PDF layouts vary widely, while existing large datasets have limited variability. DocLayNet addresses this gap with human annotation, diverse layouts, detailed labels, redundant annotations, fixed splits, and comparative model experiments.

  • 1 INTRODUCTION: PDF layout variability across languages, formats, and document designs makes one high-quality layout-segmentation model difficult to engineer.
  • 1 INTRODUCTION: Existing PubLayNet and DocBank datasets are large but rely on automated ground-truth generation that constrains layout variability.They contain approximately 300K and 500K ground-truth pages, respectively.
  • 1 INTRODUCTION: DocLayNet provides 80863 uniquely annotated pages with 11 bounding-box classes and a fraction of double- or triple-annotated pages.
  • 1 INTRODUCTION: Human annotation, diverse public sources, detailed labels, redundant annotations, and predefined data splits distinguish DocLayNet from earlier datasets.
  • 1 INTRODUCTION: Experiments evaluate Faster R-CNN, Mask R-CNN, and YOLOv5, dataset-size and label-set effects, train/test splitting, and cross-dataset robustness.

2 RELATED WORK

Document-layout analysis has shifted from rules and heuristics toward object detection and newer multimodal models. However, multimodal approaches still lack a broadly accepted format linking geometric and textual features.

  • 2 RELATED WORK: Deep-learning document-layout analysis commonly uses object detection models, whose accuracy and speed have increased substantially.
  • 2 RELATED WORK: Newer models combine page text tokens with image representations, but no broadly accepted data format yet links geometric and textual features.

3 THE DOCLAYNET DATASET

DocLayNet combines broad document coverage with human-generated, multi-annotation layout ground truth in a standardized format. Its design supports varied layouts, controlled benchmarking, and analysis of annotation consistency, while language coverage remains uneven.

  • 3 THE DOCLAYNET DATASET: DocLayNet contains 80863 PDF pages, 91104 annotation instances, and 11 labeled rectangular bounding-box classes.7059 pages have two human annotations and 1591 have three.
  • 3 THE DOCLAYNET DATASET: DocLayNet minimizes scanned documents and focuses on medium-to-large technical documents with dense, complex visual elements.Receipts, invoices, handwritten documents, and photographs showing text are excluded examples.
  • 3 THE DOCLAYNET DATASET: The dataset spans Financial Reports, Manuals, Scientific Articles, Laws & Regulations, Patents, and Government Tenders from varied repositories.Financial Reports and Manuals include substantial free-style layouts to maximize variability.
  • 3 THE DOCLAYNET DATASET: Nearly 95% of documents are in English, with German, French, and Japanese representing 2.5%, 1.0%, and 1.0%, respectively.Language was not controlled during document selection, which may challenge methods using textual features.
  • 3 THE DOCLAYNET DATASET: Predefined train, test, and validation sets preserve label proportions and prevent document pages from being split across subsets.This design supports comparable benchmarks and avoids evaluation advantages from shared-document pages.
  • 3 THE DOCLAYNET DATASET: The dataset is distributed in an augmented COCO format with page images, metadata, original PDF pages, and sidecar files.
  • 3 THE DOCLAYNET DATASET: Human annotation supports documents without available programmatic sources and can represent layouts differently from typesetting structures.For example, visually invisible tables used for alignment may be labeled as Text rather than Table.
  • 3 THE DOCLAYNET DATASET: Multiple annotations estimate a natural upper bound for segmentation accuracy, although maintaining high annotation consistency is challenging.

4 ANNOTATION CAMPAIGN

The annotation campaign combined diverse document sourcing, explicit 11-class label design, extensive annotator training, quality control, and a visual bounding-box interface. Guidelines addressed recurring ambiguities in complex layouts, while some interpretations remained unresolved.

  • Annotation tooling: The CCS interface overlaid PDF text-cells and let annotators draw labelled rectangular boxes, automatically shrinking text-segment boxes to their minimum bounding boxes.The automatic snapping excluded Table and Picture annotations.
  • Campaign design: DocLayNet sources included arXiv, government offices, company websites, and financial-report and patent directories, while scanned documents were generally excluded.Pages were selectively subsampled to balance categories and emphasize titles, figures, and tables.
  • Label design: The dataset used 11 labels selected from recurrent layout elements, considering occurrence, specificity, and recognisability on a single page.The labels include Caption, Footnote, Formula, List-item, Page-footer, Page-header, Picture, Section-header, Table, Text, and Title.
  • Annotation ambiguity: Challenging layouts produced different but plausible annotations, and guideline criteria resolved some alternatives while others remained ambiguous.Examples included grouping subfigures together or separately and grouping list items into one block or separate objects.
  • Annotation guideline: The guideline defined each list-item as an individual object and as a paragraph with hanging indentation, without requiring bullets or enumeration symbols.This differs from PubLayNet and DocBank, which group list-items into one List object.
  • Campaign design: 40 annotators underwent training, practice, exams, and quality controls before production annotation by 32 annotators.The campaign included four phases, with 974 reference-annotated pages used for practice and examination.

5 EXPERIMENTS

The experiments evaluate object-detection models on DocLayNet and examine how dataset and evaluation choices affect layout-analysis performance. They also compare DocLayNet with earlier datasets to assess robustness across layouts.

  • Experimental scope: The study evaluates Faster R-CNN, Mask R-CNN, and YOLOv5 baselines trained on DocLayNet.It also varies dataset size, label-set reduction, and train/test splitting, and compares models trained on PubLayNet, DocBank, and DocLayNet.
  • Evaluation: Prediction quality is measured using COCO mean average precision across overlap thresholds from 0.5 to 0.95.The metric is reported as mAP@0.5-0.95.

Baselines for Object Detection

Baseline object-detection models achieve similar overall mAP on DocLayNet, but remain below human agreement, leaving a measurable challenge for layout-analysis systems. Label-set changes can improve the macro-average without substantially changing remaining-class accuracy.

  • Object-detection baselines: Mask R-CNN and Faster R-CNN produce very comparable mAP scores on DocLayNet.The authors report that pixel-based image segmentation derived from bounding boxes does not improve predictions in this comparison.
  • Label-set analysis: Around 5% improvement in overall macro-average occurs when labels are merged or excluded, especially when Page-footer and Page-header are removed.Accuracy on the remaining class labels does not change significantly under these label-set modifications.

Learning Curve

A data-ablation study tests whether DocLayNet contains enough pages for training. Mask R-CNN performance rises rapidly at first, then flattens between 80% and 100% of the dataset.

  • Learning-curve analysis: The learning curve flattens between 80% and 100% of DocLayNet, with the 80% result within the 100% error bars.Five full-dataset training runs produced a 1% error bar.
  • Learning-curve analysis: Increasing DocLayNet with similar data is unlikely to improve predictions significantly once approximately 80% is used.The authors instead identify improved consistency, data augmentation, or more document categories and styles as potential benefits.

Impact of Class Labels

The paper examines how label-set choices affect layout-model performance, finding that merging or excluding classes generally preserves remaining-label accuracy while some exclusions improve the macro-average.

  • The label set can significantly affect overall model performance, so the paper tests label merging and exclusion.
  • Around 5%: the overall macro-average improves when labels are simplified, particularly by excluding Page-footer and Page-header.
  • Prediction accuracy on remaining class labels does not change significantly when other classes are merged into them.
  • A four-class label set most closely matches PubLayNet when its List label is down-mapped to Text.

Impact of Document Split in Train and Test Set

DocLayNet splits pages by document boundaries to prevent related pages from crossing train, validation, and test sets; naive page-wise splitting can inflate performance.

  • DocLayNet assigns every document’s pages to only one train, validation, or test set to avoid overfitting to a document’s unique style.
  • The split comparison uses a Mask R-CNN R50 model evaluated on a modified dataset version.

Dataset Comparison

The cross-dataset comparison evaluates models on common labels because annotation schemes differ, and finds DocLayNet-trained models more robust across datasets.

  • The comparison uses common label classes because the datasets have different label sets and annotation styles.
  • DocLayNet-trained models show less pronounced performance variation across PubLayNet, DocBank, and DocLayNet test sets.
  • Models trained on PubLayNet and DocBank perform well on their own test sets but substantially worse on foreign datasets.
  • DocLayNet-trained models also vary across datasets, but their performance difference is far less pronounced.
  • For DocBank versus DocLayNet, training and evaluation use only Picture and Table clusters because their Text annotations are incompatible.

Example Predictions

Examples of DocLayNet-trained predictions look generally decent across document categories, while also revealing overlapping clusters and missing boxes caused by low confidence.

  • DocLayNet-trained models produce generally decent layout predictions across document categories without further post-processing.
  • Observed errors include overlapping clusters belonging to different classes.
  • Low confidence can result in entirely missing prediction boxes.
  • The dataset provides challenging, diverse document pages for improving and fine-tuning machine-learning methods.
  • The paper reports human-performance reference metrics, object-detection baselines, data-ablation experiments, and cross-dataset robustness comparisons.
  • A significant gap remains between human and machine-learning accuracy on layout interpretation.
Loading 2206.01062v1…