Source-linked AI summary
DocBank: A Benchmark Dataset for Document Layout Analysis
Minghao Li, Yiheng Xu, Lei Cui, Shaohan Huang, Furu Wei, Zhoujun Li, Ming Zhou
TL;DR
Document layout analysis needs high-quality labeled data that captures textual as well as visual information. DocBank provides 500K document pages with fine-grained token-level annotations, constructed automatically through weak supervision from LaTeX documents. Experiments with baseline models support text-and-layout integration as a promising direction for layout analysis.
Problem
Existing document layout datasets are mainly image-based, making them difficult to apply to NLP methods, while visual approaches often neglect textual features.
Method
DocBank constructs 500K document pages with token-level annotations using weak supervision from LaTeX semantic structures, supporting both NLP and computer-vision models.
Results
Experiments with BERT, RoBERTa, LayoutLM and Faster R-CNN confirm DocBank's effectiveness for document layout analysis, while LayoutLM achieves the highest scores on most labels.
Takeaways & Limitations
DocBank supports fair comparison across modalities and makes text-and-layout integration a promising research direction for document layout analysis.
Abstract
from arXiv · showhide
Document layout analysis usually relies on computer vision models to understand documents while ignoring textual information that is vital to capture. Meanwhile, high quality labeled datasets with both visual and textual information are still insufficient. In this paper, we present \textbf{DocBank}, a benchmark dataset that contains 500K document pages with fine-grained token-level annotations for document layout analysis. DocBank is constructed using a simple yet effective way with weak supervision from the \LaTeX{} documents available on the arXiv.com. With DocBank, models from different modalities can be compared fairly and multi-modal approaches will be further investigated and boost the performance of document layout analysis. We build several strong baselines and manually split train/dev/test sets for evaluation. Experiment results show that models trained on DocBank accurately recognize the layout information for a variety of documents. The DocBank dataset is publicly available at \url{https://github.com/doc-analysis/DocBank}.
1 Introduction
Document layout analysis supports document understanding but remains difficult across varied layouts, and existing image-based approaches often neglect textual information. DocBank addresses this gap with a large weakly supervised dataset designed for models that use text and layout information.
- Document layout analysis transforms semi-structured documents into structured representations while extracting key information for document understanding applications.
- Existing rule-based and machine-learning techniques often generalize poorly because handcrafted features are not robust to layout variations.
- Recent computer-vision approaches have improved layout analysis but usually leverage visual features while neglecting textual features.
- DocBank is a large-scale benchmark constructed with weak supervision to provide fine-grained annotations for document layout analysis.
- The dataset enables models to integrate textual and layout information and supports experiments with multiple baseline models.
2 Task Definition
The task extracts predefined semantic units from visually rich documents by classifying document tokens according to their content and bounding-box information.
- Document layout analysis classifies tokens in visually rich documents into predefined semantic categories.
- Each document token consists of a word and its bounding box coordinates, while the category set defines the labels used for classification.
- The task seeks a function F that maps the document and category set to a prediction set of semantic structures.
3 DocBank
DocBank is a 500K-page, token-level layout dataset built from arXiv LaTeX sources using weak supervision and designed for multimodal document analysis.
- Dataset construction: DocBank contains 500K document pages with 12 semantic unit types across training, validation, and test sets.The dataset includes 400K training pages and 50K pages in each validation and test set.
- Dataset construction: The dataset focuses on English arXiv papers spanning Physics, Mathematics, Computer Science, and other areas, while preserving the natural year distribution.Documents from different years are randomly sampled without balancing them.
- Semantic Structures Detection: DocBank detects semantic structures by inserting structure-specific font colors into LaTeX source code and recompiling the documents.The annotated structures include Abstract, Author, Caption, Equation, Figure, Footer, List, Paragraph, Reference, Section, Table, and Title.
- Token Annotation: PDFPlumber extracts text lines and non-text elements with bounding boxes, while token labels are assigned from the extracted color-to-structure mapping.Figures and lines without text are represented using PDFMiner class names as special tokens; table words and lines receive the table label.
- Token Annotation: Tokens with the same semantic label are grouped into connected components using x- and y-coordinate thresholds to form object bounding boxes.Each object’s bounding box is determined by its boundary tokens, and the annotations can be converted for object-detection models such as Faster R-CNN.
- Dataset comparison: Compared with prior layout datasets, DocBank is larger, contains more semantic structure types, supports text-based and image-based models, and is automatically extendable.The comparison includes Article Regions, GROTOAP2, PubLayNet, and TableBank.
4 Method
The method evaluates sequence labeling and object detection approaches on token-level DocBank annotations, using BERT, RoBERTa, LayoutLM, and Faster R-CNN.
- Task formulation: Because DocBank is fully annotated at token level, document layout analysis is evaluated as a text-based sequence labeling task.The task assigns semantic structure labels to document tokens.
- Cross-modal evaluation: Faster R-CNN is trained on DocBank’s object-detection format to compare models from different modalities.Its output is unified with the sequence-labeling models for evaluation.
- Models: BERT is a Transformer-based language model that produces contextualized token representations through bidirectional Transformer encoder layers.Its input representation sums token, segment, and position embeddings before attention-based Transformer blocks.
- Models: RoBERTa uses the BERT architecture with a different tokenization algorithm and improved training strategies, including more pretraining data and training steps.The passage describes RoBERTa as achieving better performance on several downstream NLP tasks.
- Models: LayoutLM jointly models document text and layout by adding 2-D position embeddings for token bounding-box coordinates.It accepts tokens with corresponding bounding boxes and combines layout embeddings with BERT-style representations.
5 Experiment
The experiments evaluate token-level layout analysis with a specialized metric and compare language, multimodal, vision, and ensemble models on DocBank.
- 5.1 Evaluation Metrics: The evaluation uses a new metric because semantic units may distribute discontinuously across serialized 2-D document token sequences.Metrics are calculated separately for each document semantic structure.
- 5.2 Baselines: The baselines include BERT, RoBERTa, LayoutLM, Faster R-CNN, and ensembles combining ResNeXt-101 with LayoutLM.The models represent sequence-labeling, multimodal, object-detection, and ensemble approaches.
- 5.3 Results: Table 4 reports performance for BERT, RoBERTa, LayoutLM, and Faster R-CNN on the DocBank test set.The table compares representative models across modalities using the DocBank evaluation setup.
- 5.3 Results: LayoutLM achieves the highest scores on 11 labels, while RoBERTa performs best on reference with only a small gap from LayoutLM.The LayoutLM results cover abstract, author, caption, equation, figure, footer, list, paragraph, section, table, and title.
- 5.3 Results: ResNeXt-101 outputs bounding boxes that are converted into token labels before metrics are computed alongside sequence-labeling outputs.Two ensemble models combine ResNeXt-101 and LayoutLM.
6 Case Study
The case study compares pretrained BERT and LayoutLM outputs on test samples, showing that incorporating layout information improves semantic-unit detection over text alone.
- Case Study: The case study visualizes pretrained BERT and LayoutLM predictions on samples from the DocBank test set.The outputs are shown in Figures 3 and 4.
- Case Study: Pretrained BERT incorrectly detects some tokens, indicating that text information alone is insufficient for document layout analysis.The comparison motivates incorporating visual information into the model.
- Case Study: LayoutLM integrates text and layout information and produces better benchmark performance than pretrained BERT.Its 2-D position embeddings model spatial distance and semantic-structure boundaries in a unified framework.
7 Related Work
Related work spans rule-based, conventional machine-learning, and deep-learning approaches, with earlier methods relying on heuristics or handcrafted features for document layout analysis.
- Overview: Document layout analysis research is categorized into rule-based, conventional machine-learning, and deep-learning approaches.These categories organize the methods discussed in the related-work section.
- Rule-based Approaches: Rule-based methods use bottom-up or top-down strategies to detect regions and combine them into higher-level document structures.Examples include connected components, texture analysis, projection profiles, and X-Y cuts.
- Rule-based Approaches: Rule-based heuristic algorithms are difficult to apply to complex documents and generally target relatively simple document types.The limitation follows the related-work discussion of these approaches.
- Conventional Machine Learning Approaches: Conventional learning-based methods address issues such as data imbalance but often depend on feature extraction and handcrafted representations.Examples include dynamic MLPs, Gradient Shape Features, and Scale Invariant features.
- Deep Learning Approaches: Deep-learning approaches include weighted-loss fully convolutional networks and methods that use pretrained network weights.These approaches received attention for handling more complex layout-analysis settings.
8 Conclusion
DocBank provides 500K high-quality document pages built automatically with weak supervision, enabling models to use textual and visual information for layout analysis. Experiments identify text-and-layout integration as a promising research direction and suggest DocBank can support broader deep-learning research.
- DocBank contains 500K high-quality document pages built automatically with weak supervision for document layout analysis.
- The dataset enables layout-analysis models to use both textual and visual information.
- Experiments show that integrating text and layout information is a promising research direction with DocBank.
- DocBank is expected to help release the power of other deep learning models in document layout analysis.