PaperlayerView exact revision on arXiv

Source-linked AI summary

PubTables-1M: Towards comprehensive table extraction from unstructured documents

Brandon Smock, Rohith Pesala, Robin Abraham

arXiv:2110.00061v3cs.LGcs.CV

TL;DR

Table extraction lacks complete, unambiguous ground truth at scale, limiting learning from document tables. The paper introduces PubTables-1M with detailed annotations and canonicalization, then shows strong transformer-based performance across detection, structure recognition, and functional analysis. The authors identify expansion beyond scientific articles and improved row-header annotation as future directions.

  • Problem

    Table extraction needs complete, reliable, and unambiguous ground truth at scale because document tables often leave logical structure implicit and oversegmented annotations create ambiguity.

  • Method

    The paper builds PubTables-1M from scientific articles, aligns PDF text with HTML structure, adds detailed spatial and header annotations, and canonicalizes oversegmented structures.

  • Results

    DETR-based models address table detection, structure recognition, and functional analysis without special task customization, while improved ground truth increases TSR performance and evaluation reliability.

  • Takeaways & Limitations

    PubTables-1M provides a large, detailed resource for training and evaluating table-extraction models across multiple input modalities.

  • Takeaways & Limitations

    The dataset and canonicalization are initially scoped to scientific articles, and accurately annotating row headers at large scale remains an open challenge.

Abstract

from arXiv · show

Recently, significant progress has been made applying machine learning to the problem of table structure inference and extraction from unstructured documents. However, one of the greatest challenges remains the creation of datasets with complete, unambiguous ground truth at scale. To address this, we develop a new, more comprehensive dataset for table extraction, called PubTables-1M. PubTables-1M contains nearly one million tables from scientific articles, supports multiple input modalities, and contains detailed header and location information for table structures, making it useful for a wide variety of modeling approaches. It also addresses a significant source of ground truth inconsistency observed in prior datasets called oversegmentation, using a novel canonicalization procedure. We demonstrate that these improvements lead to a significant increase in training performance and a more reliable estimate of model performance at evaluation for table structure recognition. Further, we show that transformer-based object detection models trained on PubTables-1M produce excellent results for all three tasks of detection, structure recognition, and functional analysis without the need for any special customization for these tasks. Data and code will be released at https://github.com/microsoft/table-transformer.

1. Introduction

Table extraction requires inferring logical structure that presentation tables often leave implicit, while scalable datasets must provide complete and unambiguous ground truth. PubTables-1M addresses these issues with richer annotations, canonicalization, quality controls, and transformer-based models across all three extraction subtasks.

  • Table extraction infers rows, columns, cells, and functional roles from presentation tables whose logical structure is often implicit.
  • Crowd-sourced markup annotations can omit spatial information and exhibit oversegmentation, producing ambiguous or inconsistent ground truth for training and evaluation.
  • PubTables-1M contains nearly one million scientific-article tables and addresses table detection, structure recognition, and functional analysis.
  • The dataset adds projected row headers, bounding boxes for rows, columns, cells, and blank cells, plus source-document annotations supporting multiple input modalities.
  • Canonicalization merges oversegmented header cells to ensure each table has a unique, unambiguous structure interpretation.
  • Data improvements significantly increase TSR performance, while DETR addresses all three tasks within a transformer-based object-detection framework without special customization.

2. Related Work

Prior table-structure datasets expanded through crowd-sourcing but remained limited in scale, annotation completeness, quality verification, and handling of oversegmentation. Existing modeling approaches often require task-specific adaptations because standard architectures underperform on TSR out of the box.

  • ICDAR-2013 supports all three extraction tasks but contains only 248 tables for TD and TSR and 92 tables for FA.
  • Larger crowd-sourced datasets derive structure and content from HTML, XML, or LaTeX and connect markup with presentation locations.
  • FinTabNet and enhanced PubTabNet add cell locations, but prior work left open how to define bounding boxes for all cells, including blank cells.
  • Prior large-scale datasets generally omit row and column bounding boxes, column-header completeness, or row-header annotations, limiting modeling and end-to-end extraction.
  • Large-scale annotation quality remains difficult to verify, and oversegmentation can harm training data and underestimate evaluation performance.
  • Object detection, image-to-text, and graph-based TSR approaches commonly underperform out of the box or use partial solutions, engineered components, or custom pipelines.

3. PubTables-1M

PubTables-1M derives large-scale table ground truth from aligned PDF and XML sources, then completes, canonicalizes, and verifies annotations. Its design adds detailed spatial and header information while explicitly limiting canonicalization to PMCOA-specific assumptions.

  • Dataset construction: PubTables-1M is built from scientific articles represented as paired PDF and XML documents in the PMCOA corpus.PDFs provide visual presentation, while XML provides semantic structure for each table.
  • Alignment: PDF–XML text alignment assigns spatial locations to HTML-tagged table content using character-level Needleman-Wunsch matching.Text cell bounding boxes are computed as unions of the character bounding boxes for each cell’s text.
  • Spatial completion: Completed annotations define bounding boxes for tables, rows, columns, and blank grid cells, extending spatial supervision beyond text-bearing cells.Grid cells are formed from row and column bounding boxes, including cells without text.
  • Canonicalization: Canonicalization merges adjacent cells under structural conditions to correct oversegmentation, using header-tree assumptions and conventions for blank cells.The method infers projected row headers but treats full row-header inference as outside its scope.
  • Limitations and quality control: Canonicalization is designed specifically for PMCOA annotations and does not guarantee mistake-free ground truth, so automated quality control remains necessary.The quality-control process discards overlapping or otherwise anomalous tables; fewer than 0.1% are removed as outliers.
  • Dataset statistics: 947,642 tables are available for TSR, 52.7% are complex, and canonicalization changes annotations for 34.7% of tables.Only 40.1% were considered complex by the original annotators before canonicalization.

4. Proposed Model

The proposed models frame table detection, structure recognition, and functional analysis as object detection tasks. DETR jointly models TSR and FA through overlapping object classes and is evaluated alongside Faster R-CNN on PubTables-1M.

  • Task formulation: All three table-extraction tasks are modeled as object detection with images as input.Table detection uses table and table-rotated classes.
  • Joint TSR and FA model: TSR and FA are jointly represented with six object classes whose physical overlaps encode table hierarchy and imply a seventh grid-cell class.The classes include tables, rows, columns, column headers, projected row headers, and spanning cells.
  • Bounding-box design: Dilated bounding boxes expand adjacent row and column boundaries halfway into intervening whitespace, eliminating gaps and overlaps.Other object classes are adjusted to match the expanded row and column boundaries.
  • Experiments: DETR is applied to all three tasks using one model for detection and one joint model for TSR and FA, with Faster R-CNN trained for comparison.The experiments use ResNet-18 backbones pretrained on ImageNet, with early layers frozen.
  • Experimental design: The models avoid task-specific custom engineering and use default settings wherever possible so the data drives the result.The supplied passage describes this as an experimental design choice rather than a task-specific architectural modification.

5. Experiments

Experiments evaluate detection, structure recognition, and functional analysis models trained on PubTables-1M, emphasizing DETR and the effects of canonicalized annotations. Canonicalization improves TSR performance and makes evaluation more reliable, especially for complex tables.

  • Detection: DETR slightly outperforms Faster R-CNN on AP50 and significantly outperforms it on AP for table detection.The authors interpret this as evidence that DETR localizes tables more precisely.
  • Evaluation metrics: AccCont measures exact text agreement for every cell, while partial-correctness metrics award credit when only some cells are correct.GriTS compares two-dimensional table substructures and includes topology, content, and location variants.
  • Structure recognition and functional analysis: DETR trained on canonical data produces strong TSR and FA results, outperforming the other models when evaluated on all tables.The comparison includes Faster R-CNN and DETR-NC models.
  • Canonicalization: 0.5360 to 0.6944: canonical data raises table accuracy for complex tables.This comparison concerns DETR-NC trained on non-canonical annotations versus DETR trained on canonical annotations.
  • Canonicalization: 0.9349 versus 0.8678 accuracy: DETR-NC performs better on simple tables when evaluated on canonical rather than non-canonical data.The result isolates the effect of evaluation data and supports a more reliable evaluation with canonical annotations.
  • Canonicalization: Canonical data significantly improves TSR model performance because non-canonical oversegmentation does not correspond to tables’ true logical structures.DETR-NC performs much worse on complex tables, partly because it learns an inconsistent treatment of spanning header cells.

6. Conclusion

The paper introduces PubTables-1M to provide complete, reliable ground truth for table structure recognition at scale. It addresses oversegmentation through canonicalization and shows strong results from DETR across all three table extraction tasks without task-specific customization.

  • Conclusion: PubTables-1M addresses the challenge of creating complete, reliable ground truth at scale for table structure recognition.The dataset is introduced for table extraction in unstructured documents.
  • Conclusion: Canonicalization addresses ambiguous ground truth caused by oversegmentation in markup annotations.The paper reports that improved ground-truth data significantly benefits model performance.
  • Conclusion: DETR achieves state-of-the-art performance within a standard object detection framework for detection, structure recognition, and functional analysis without special customization.The conclusion presents this as the paper’s result across all three table extraction tasks.

7. Future Work

Future work extends PubTables-1M’s methods and canonicalization beyond scientific articles, improves large-scale row-header annotation, and develops end-to-end document-understanding and information-retrieval systems.

  • Domain expansion: The authors hope to extend the methods and canonicalization beyond scientific articles to domains such as financial documents.This is identified as a future expansion of the dataset and approach.
  • Annotation completeness: They identify accurate annotation of row headers in large-scale datasets as an open challenge.Addressing it would support more complete table extraction solutions.
  • End-to-end systems: They propose end-to-end systems for document understanding and information retrieval as an important future direction.Table extraction is described as one stage in larger pipelines with their own challenges.

9. Appendix

The appendix describes a largely standard DETR setup and the inference steps used to convert hierarchical detections into conflict-free logical tables. Evaluation additionally tightens row and column boxes around extracted text for location scoring.

  • Model configuration: Both DETR models use a ResNet-18 backbone, six encoder layers, and six decoder layers.TD uses 15 object queries, while TSR and FA use 125.
  • Training setup: Models are trained for 20 epochs on one Tesla V100 GPU with ImageNet-pretrained weights and mostly default settings.The TSR and FA model additionally uses a 0.00005 initial learning rate and a no-object class weight of 0.4.
  • Training setup: Training uses no custom components, losses, or procedures beyond standard data augmentations.PDF pages are rendered to images and bounding boxes are scaled for the detection tasks.
  • Inference: At inference, hierarchical object detections are converted into a structured table after modeling parent-child relationships through physical containment or overlap.The model is trained as an object detector, but structured-table conversion requires this additional inference step.
  • Inference: A conflict-resolution step suppresses objects or adjusts bounding boxes so same-class objects become conflict-free before conversion to a logical table.The procedure is compared with non-maxima suppression, except conflicts depend on child-object overlap rather than pixels.
  • Evaluation: For location scoring, row and column boxes are tightened after text extraction to wrap their contained text closely.Cell text remains unchanged before and after this adjustment.
Loading 2110.00061v3…