Source-linked AI summary
TableBank: A Benchmark Dataset for Table Detection and Recognition
Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, Ming Zhou, Zhoujun Li
TL;DR
Existing table-analysis datasets are small and difficult to generalize across document layouts and formats. TableBank uses weak supervision from online Word and Latex documents to create a large image-based benchmark with deep-learning baselines. Its experiments show strong domain dependence and indicate that larger mixed-domain training improves generalization, while complex table structures remain challenging.
Problem
Existing image-based table analysis often relies on a few thousand human-labeled examples and out-of-domain fine-tuning, limiting evidence for generalization across varied documents.
Method
The paper automatically constructs TableBank from markup in online Word and Latex documents, then evaluates Faster R-CNN detection and image-to-text structure-recognition baselines.
Results
417,234 high quality labeled tables are provided, while experiments show domain-specific models perform poorly across document types and larger mixed-domain training generalizes better.
Takeaways & Limitations
TableBank provides a large open benchmark for developing image-based table detection and structure-recognition models across Word and Latex document domains.
Takeaways & Limitations
Structure-recognition exact-match accuracy falls from around 50% for sequences under 40 tokens to 8.6% for longer sequences, making big and complex tables difficult.
Abstract
from arXiv · showhide
We present TableBank, a new image-based table detection and recognition dataset built with novel weak supervision from Word and Latex documents on the internet. Existing research for image-based table detection and recognition usually fine-tunes pre-trained models on out-of-domain data with a few thousand human-labeled examples, which is difficult to generalize on real-world applications. With TableBank that contains 417K high quality labeled tables, we build several strong baselines using state-of-the-art models with deep neural networks. We make TableBank publicly available and hope it will empower more deep learning approaches in the table detection and recognition task. The dataset and models are available at \url{https://github.com/doc-analysis/TableBank}.
1. Introduction
TableBank addresses weak generalization in image-based table analysis by using weak supervision from online Word and Latex documents to build a large benchmark dataset. Its experiments show that layout and format variation strongly affect accuracy, while cross-domain training improves generalization.
- Table analysis is difficult because tables vary widely in layout and format, limiting handcrafted document-layout features.
- 417,234 high quality labeled tables are collected from online Word and Latex documents for TableBank.
- TableBank evaluates Faster R-CNN for table detection and an encoder-decoder image-to-text model for table structure recognition.
- Layout and format variation has a great impact on table analysis accuracy, and models trained in one domain do not perform well in the other.
2. Existing Datasets
Existing public table datasets are relatively small and cover different document formats, including digital documents, scanned images, and PDFs.
- ICDAR 2013 contains 128 natively-digital document examples from European Union and US Government sources.
- UNLV contains 427 scanned-image examples from magazines, newspapers, business letters, and annual reports.
- Marmot contains 2,000 PDF pages, with most of the described content continuing beyond the supplied passage.
3. Data Collection
TableBank is constructed from Word and Latex source documents by exploiting markup to create detection boxes and HTML-based structure labels. The resulting resources include 417,234 labeled tables and 145,463 structure-recognition instances.
- Document acquisition: Word and Latex documents are used because both contain table markup tags in their source code.
- Document acquisition: Word documents are crawled as .docx files, while Latex documents and source code are obtained from arXiv.org.
- Table Detection: For Word detection labels, table markup between <w:tbl> and </w:tbl> is modified to add distinguishable colored borders before PDF rendering.
- Table Detection: 417,234 labeled tables are created by comparing annotated and original PDF pages at the pixel level to recover table bounding boxes.
- Table Structure Recognition: 145,463 structure-recognition training instances are produced by transforming Word XML and Latex-derived XML into HTML tag sequences.
4. Baseline
The paper uses established end-to-end deep-learning baselines for both tasks: Faster R-CNN for table detection and image-to-text for table structure recognition.
- Table Detection: Faster R-CNN is used as the table-detection baseline.
- Table Detection: Faster R-CNN combines a Region Proposal Network with Fast R-CNN by sharing convolutional features for end-to-end training.
- Table Structure Recognition: The image-to-text baseline encodes a table image and decodes a text output representing its structure.
5. Experiment
Experiments evaluate TableBank for table detection and structure recognition using deep neural baselines across Word, Latex, mixed, and ICDAR 2013 data. Results show strong within-domain performance but substantial cross-domain and complex-table challenges.
- 5.2. Settings: Faster R-CNN with ResNeXt backbones provides the table-detection baseline, while an encoder-decoder image-to-text model provides the structure-recognition baseline.Detection models use Detectron; structure recognition uses OpenNMT and an image-to-markup approach.
- 5.3. Results: 0.9166 F1 is achieved by Word-trained ResNeXt-152 on Word data, versus 0.8094 on Latex; Latex-trained ResNeXt-152 achieves 0.9810 on Latex versus 0.8863 on Word.These results indicate different visual appearances across document types and weaker cross-domain transfer.
- 5.3. Results: 0.9625 F1 is achieved by the Latex ResNeXt-152 model on ICDAR 2013, while TableBank models outperform conventional Tesseract3 and Camelot approaches.The comparisons also include models trained with ICDAR, UNLV, Marmot, and DeepFigures.
- 5.4. Analysis: 6.5% of ground-truth tables are undetected, 3.5% of predicted tables are mis-detected, and 2.3% are partially detected for Word+Latex ResNeXt-152.The case study counts 164 undetected, 86 mis-detected, and 57 partially detected tables.
- 5.4. Analysis: 8.6% exact match is obtained for HTML sequences longer than 40 tokens, showing that recognition becomes harder as table output length increases.The Word+Latex image-to-text model has around 50% exact match for sequences shorter than 40 tokens and correctly generates 338 tables overall.
6. Related Work
Prior table analysis methods include rule-based, statistical, machine-learning, and deep-learning approaches. TableBank differs by providing large-scale data for end-to-end table detection and structure recognition.
- 6.1. Table Detection: Rule-based table detection uses text-block arrangement, ruled lines, or horizontal and vertical lines, but requires extensive rule engineering and may fail across document sources.These methods perform well on some documents but do not reliably generalize to other sources.
- 6.1. Table Detection: Image-based table detection is designed to handle both scanned pages and natively digital documents while reducing handcrafted feature engineering.Earlier convolutional approaches first select table-like regions using loose rules before refining them with neural networks.
- 6.2. Table Structure Recognition: Table structure recognition identifies the row and column layout of a table through rule-based, machine-learning, or deep-learning methods.Prior work includes graphic-line analysis, text-element overlap, clustering, and object detection.
- 6.2. Table Structure Recognition: TableBank uses end-to-end data-driven models with large-scale training data, whereas existing structure-recognition methods usually use no training data or only small-scale training data.The paper presents TableBank as the first large-scale dataset for both table detection and recognition tasks.
7. Conclusion
TableBank is introduced as an image-based table analysis dataset built from online Word and Latex documents, with baseline evaluations and planned expansion to broader, fine-grained document labels.
- TableBank is an image-based table analysis dataset built using online Word and Latex documents.
- Faster R-CNN and image-to-text models serve as baselines for evaluating table detection and structure recognition on TableBank.
- Testing data from Word and Latex documents evaluates model accuracy across different domains.
- The authors plan to enlarge TableBank across more domains and add labels for figures, headings, subheadings, and text blocks.