PaperlayerView exact revision on arXiv

Source-linked AI summary

PubLayNet: largest dataset ever for document layout analysis

Xu Zhong, Jianbin Tang, Antonio Jimeno Yepes

arXiv:1908.07836v1cs.CL

TL;DR

Document layout analysis lacks large annotated datasets, despite the difficulty of processing PDF layouts automatically. PubLayNet matches paired XML and PDF representations from over 1 million PubMed Central articles to create a large layout dataset. Models trained on it accurately recognize scientific-article layouts and provide a more effective base for transfer learning across domains.

  • Problem

    Existing document layout datasets are small because manual annotation is slow and expensive, limiting available training data for varied document templates.

  • Method

    PubLayNet automatically matches XML representations with PDF content from over 1 million PubMed Central articles to annotate document-layout elements.

  • Results

    Deep object detection models achieve MAP > 0.9 on document-layout recognition, and PubLayNet pre-training is a more effective base for transfer learning.

  • Takeaways & Limitations

    PubLayNet supports high-performance layout recognition for scientific articles and transfer learning for documents in other domains.

Abstract

from arXiv · show

Recognizing the layout of unstructured digital documents is an important step when parsing the documents into structured machine-readable format for downstream applications. Deep neural networks that are developed for computer vision have been proven to be an effective method to analyze layout of document images. However, document layout datasets that are currently publicly available are several magnitudes smaller than established computing vision datasets. Models have to be trained by transfer learning from a base model that is pre-trained on a traditional computer vision dataset. In this paper, we develop the PubLayNet dataset for document layout analysis by automatically matching the XML representations and the content of over 1 million PDF articles that are publicly available on PubMed Central. The size of the dataset is comparable to established computer vision datasets, containing over 360 thousand document images, where typical document layout elements are annotated. The experiments demonstrate that deep neural networks trained on PubLayNet accurately recognize the layout of scientific articles. The pre-trained models are also a more effective base mode for transfer learning on a different document domain. We release the dataset (https://github.com/ibm-aur-nlp/PubLayNet) to support development and evaluation of more advanced models for document layout analysis.

I. INTRODUCTION

PDF layout analysis is difficult to automate, while existing manually annotated datasets are small and costly to expand. PubLayNet addresses this gap by automatically annotating over 1 million PubMed Central articles for training and transfer learning.

  • 2.5 trillion PDF documents are available, but their layouts complicate automatic processing and information extraction.
  • Manual annotation is slow and expensive, especially because document-template diversity increases the amount of training data required.
  • PubLayNet automatically annotates over 1 million PubMed Central PDF articles and includes over 360k page samples covering text, title, list, figure, and table elements.
  • Deep object detection networks trained on PubLayNet recognize scientific-article layouts, while PubLayNet pre-training provides a more effective base for transfer learning.
  • Existing document layout datasets contain only several hundred pages, largely because they require manual annotation.

III. AUTOMATIC ANNOTATION OF DOCUMENT LAYOUT

PubLayNet uses paired PDF and XML versions of PubMed Central Open Access articles to automatically identify document layout components. XML categories are aggregated into visually learnable layout classes intended to support downstream document studies.

  • Figure 1 illustrates parsing PDF pages with PDFMiner and matching the parsed layout against XML representations to generate page annotations.
  • PMCOA supplies the same articles in PDF and structured XML formats, enabling automatic alignment of visual pages with document structure.The collection contained 1,162,856 articles with complete XML representations.
  • The annotation method aggregates diverse XML node categories into document-layout categories that visual models can distinguish and learn.
  • The selected categories are commonly found across domains and cover elements important for downstream tasks such as text classification, entity recognition, and figure/table understanding.

B. Annotation algorithms

The annotation algorithm matches parsed PDF elements to XML nodes, assigns layout labels and segmentations, and applies quality control to limit annotation noise. Several paper-specific layout conventions govern ambiguous or nested elements.

  • PDF elements are matched to XML nodes, whose categories determine each bounding-box label and segmentation before quality control is applied.
  • Section titles or figure/table labels inline with leading text or captions are labeled as text rather than separate titles.
  • Nested lists are annotated as one object, so child lists are not separately annotated.
  • Sub-figures are not individually annotated; the entire figure panel is treated as a single object.
  • Annotation quality is controlled using a metric based on the annotated area of textboxes, images, and geometric shapes within the page’s main text box.

1) PMCOA XML pre-processing and parsing:

XML preprocessing removes interfering nodes, standardizes inconsistent placement of lists, tables, and figures, and partitions remaining nodes into groups reflecting document structure and reading order.

  • Nodes such as tex-math, edition, institution-id, and disp-formula are removed because their content may interfere with matching other nodes.
  • List, table, and figure nodes are moved into the floats-group branch to standardize their inconsistent placement across XML articles.
  • Sorted nodes include titles, abstracts, keywords, section titles, and main-text content in PDF reading order.
  • Unsorted nodes include copyright, licensing, author, affiliation, acknowledgment, and abbreviation content whose XML order may differ from PDF reading order.
  • Figures, tables, and lists form separate groups containing their associated captions, bodies, footnotes, or list content.

2) PMCOA PDF parsing:

The PDF parsing stage uses PDFMiner to extract textboxes, images, and geometric shapes, preserving their text and bounding-box information. Unicode normalization makes later PDF–XML matching more robust.

  • 2) PMCOA PDF parsing:: PDFMiner extracts three layout types: textboxes, images, and geometric shapes.Textboxes contain textlines and bounding boxes; images and geometric shapes are each associated with bounding boxes.
  • 2) PMCOA PDF parsing:: Each textbox stores its text, bounding box, and constituent textlines.Each textline likewise stores its text and bounding box.
  • 2) PMCOA PDF parsing:: Unicode strings are normalized to KD normal form before matching PDF and XML content.This replaces compatibility characters with their equivalents to make matching more robust.

4) PDF-XML matching algorithms:

The matching algorithm aligns PDF-derived text with XML nodes using fuzzy string search and adaptive Levenshtein-distance thresholds. It then uses matched text and page margins to identify figure and table bodies.

  • 4) PDF-XML matching algorithms:: Fuzzy matching uses Levenshtein distance to tolerate minor discrepancies between PDF and XML text.The maximum allowed distance adapts to the target-string length.
  • 4) PDF-XML matching algorithms:: The allowed matching distance is 0.2 * ltarget for ltarget ≤20, 0.15 * ltarget for 20 < ltarget ≤40, and 0.1 * ltarget for ltarget >40.These thresholds define the adaptive dmax used for fuzzy matching.
  • 4) PDF-XML matching algorithms:: Textlines are sequentially searched across XML nodes, with textboxes split when a node ends before the textbox content.The procedure continues across text XML groups, captions, and table footnotes.
  • 4) PDF-XML matching algorithms:: Inline section titles and figure or table caption labels are annotated as text when their final line does not cover a whole textline.This rule accounts for journal-specific templates.
  • 4) PDF-XML matching algorithms:: Figure and table bodies are inferred from margins between annotated text elements and their captions.The main article text box and a potential figure box define the region used for body annotation.

5) Generation of instance segmentation:

PubLayNet generates polygonal segmentations from PDF textlines and controls dataset construction across journal-level splits. Quality filtering excludes non-title pages whose annotation quality falls below 99%.

  • 5) Generation of instance segmentation:: Text, title, and list instances receive regular polygon segmentations derived from their textlines.The polygons use horizontal and vertical edges shaped by adjacent textline positions.
  • 5) Generation of instance segmentation:: Non-title pages with annotation quality below 99% are excluded to keep annotation noise low.Annotation quality is the annotated area divided by the area of relevant elements within the page’s main text box.
  • 5) Generation of instance segmentation:: Training, development, and testing pages are partitioned at journal level to evaluate generalization across unseen paper templates.This maximizes template differences between the sets.
  • 5) Generation of instance segmentation:: Journals meeting page, figure, table, and list thresholds supply development and testing pages with adequate instance coverage.The criteria are ≤2000 pages, ≥320 figures, ≥140 tables, and ≥20 lists.
  • 5) Generation of instance segmentation:: Other journals supply diverse training data through per-journal caps for lists, tables, figures, title pages, and plain pages.The sampling limits are at most 200, 50, 50, 50, and 25 pages, respectively.
  • 5) Generation of instance segmentation:: PubLayNet is one to two orders of magnitude larger than existing document layout datasets.Table II reports statistics for its training, development, and testing sets.

IV. RESULTS

The experiments test Faster-RCNN and Mask-RCNN on PubLayNet and evaluate their transfer to table recognition and documents from another domain.

  • IV. RESULTS: The experiments evaluate Faster-RCNN and Mask-RCNN for document-layout recognition on PubLayNet.They also test fine-tuning on ICDAR 2013 table recognition and initialization for a different document domain.
  • IV. RESULTS: The evaluation covers both layout recognition and transfer-learning performance across table recognition and a different document domain.These experiments address the paper’s three stated research questions.
  • IV. RESULTS: The study compares PubLayNet-pre-trained models with ImageNet- and COCO-pre-trained initializations for cross-domain document analysis.This comparison tests whether PubLayNet provides a better transfer-learning starting point.

A. Document layout recognition using deep learning

PubLayNet-trained Faster R-CNN and Mask R-CNN models accurately recognize document layouts, with Mask R-CNN slightly ahead. The evaluation also examines transfer to table detection using limited training data.

  • Model training: 180k-iteration Faster R-CNN and Mask R-CNN models were trained on PubLayNet using ResNeXt-101-64x4d backbones initialized from ImageNet.Training used Detectron, eight GPUs, and an effective mini-batch size of eight.
  • Recognition results: MAP > 0.9 shows that both models accurately recognize document layout, with Mask R-CNN holding a small advantage over Faster R-CNN.The metric is MAP @ IOU [0.50:0.95] of bounding boxes.
  • Recognition results: Tables and figures are detected more accurately than texts, titles, and lists, while titles are the hardest category.The paper attributes this pattern to shape regularity, category distinctiveness, annotation noise, and the small size of titles.
  • Recognition results: Some rare Mask R-CNN errors are attributed to noise in PubLayNet, motivating further dataset-quality improvement.The paper presents representative correct predictions alongside erroneous examples.
  • Table detection: The ICDAR 2013 Table Recognition Competition is addressed by fine-tuning models trained on PubLayNet table pages.The table dataset is created from training pages containing tables, with non-table instances removed.
  • Table detection: State-of-the-art ICDAR 2013 table-recognition performance was obtained using PubLayNet-pretrained models with only 170 training pages.The reported result is based on models pretrained on PubLayNet table pages.

C. Fine tuning for a different domain

The paper evaluates PubLayNet pretraining for document layout analysis in private health-insurance Summary Plan Description documents. Fine-tuned PubLayNet models substantially outperform alternative initializations, although transfer varies by category and architecture.

  • Target domain: Summary Plan Description documents have diverse layouts that differ distinctly from scientific publications.The domain therefore provides a different setting for evaluating transfer learning.
  • Target domain: The domain-specific dataset contains 2,131 pages from 20 manually annotated documents, including texts, tables, and lists.The annotations cover 9,379 text, 2,500 table, and 820 list instances.
  • Evaluation: Five-fold cross-document validation compares different pretrained Faster R-CNN and Mask R-CNN models for fine-tuning.Each fold trains on 16 documents and tests on four documents.
  • Transfer results: Fine-tuning PubLayNet-pretrained models substantially outperforms fine-tuning models initialized from ImageNet or COCO.The comparison includes ImageNet backbone initialization, COCO whole-model initialization, and PubLayNet whole-model initialization.
  • Transfer results: Zero-shot PubLayNet performance is considerably worse than fine-tuned performance because SPD and PubMed Central article layouts differ distinctly.The result indicates that adaptation to the target domain is necessary for strong performance.
  • Transfer results: COCO-pretrained Faster R-CNN detects tables more accurately in fine-tuning, while PubLayNet Mask R-CNN gains relatively less on tables than on text and lists.The paper relates this to differences in table styles between SPD and PubMed Central articles.

V. DISCUSSION

PubLayNet provides a large, high-quality dataset for document layout analysis, with strong performance on scientific articles and demonstrated value for transfer learning. Its main scope boundaries are title identification and domain specificity, while document-element relationships remain future work.

  • V. DISCUSSION: PubLayNet’s automatically generated annotations are high quality, and state-of-the-art object detectors successfully reproduce them.The title category is the weakest because titles appear in varied forms.
  • V. DISCUSSION: Over 6,500 journals contribute enough variation for models trained on PubLayNet to recognize layouts from unseen scientific journals.The dataset remains domain-specific because all documents are scientific literature.
  • V. DISCUSSION: PubLayNet supports transfer learning for document layout analysis in distant domains, including government and SPD documents.The paper reports demonstrated value when adapting models to these domains.
  • V. DISCUSSION: The dataset enables high-performance layout recognition on biomedical articles and helps pre-train detectors for tables and other layout objects in health insurance documents.The authors describe these results as encouraging for annotation in other domains.
  • V. DISCUSSION: PubLayNet does not encode relationships between layout elements, such as paragraphs and section titles.The XML representation could support future datasets for document logical structure.
Loading 1908.07836v1…