Source-linked AI summary
Global Table Extractor (GTE): A Framework for Joint Table Identification and Cell Structure Recognition Using Visual Context
Xinyi Zheng, Doug Burdick, Lucian Popa, Xu Zhong, Nancy Xin Ru Wang
TL;DR
PDF and scanned documents do not preserve table structure, while existing datasets and methods limit cell-structure recognition. GTE combines vision-guided joint table and cell recognition with automatically labeled datasets, outperforming previous systems on ICDAR 2013 and 2019 benchmarks.
Problem
Existing object-detection approaches have limitations in accurately detecting cell structures, while large datasets with detailed cell annotations remain scarce.
Method
GTE jointly detects tables and cell structures using a cell-containment constraint loss, hierarchical style-aware cell detection, and automatic PDF–HTML labeling.
Results
GTE outperforms previous systems on the ICDAR 2013 and 2019 table competition benchmarks in table detection and cell structure recognition.
Takeaways & Limitations
The framework supports full document-to-table-structure extraction and provides enhanced PubTabNet and FinTabNet resources for table-related research.
Takeaways & Limitations
Anchor-based detection can produce boxes that are too short for long text and may merge cells when graphical lines are absent.
Abstract
from arXiv · showhide
Documents are often used for knowledge sharing and preservation in business and science, within which are tables that capture most of the critical data. Unfortunately, most documents are stored and distributed as PDF or scanned images, which fail to preserve logical table structure. Recent vision-based deep learning approaches have been proposed to address this gap, but most still cannot achieve state-of-the-art results. We present Global Table Extractor (GTE), a vision-guided systematic framework for joint table detection and cell structured recognition, which could be built on top of any object detection model. With GTE-Table, we invent a new penalty based on the natural cell containment constraint of tables to train our table network aided by cell location predictions. GTE-Cell is a new hierarchical cell detection network that leverages table styles. Further, we design a method to automatically label table and cell structure in existing documents to cheaply create a large corpus of training and test data. We use this to enhance PubTabNet with cell labels and create FinTabNet, real-world and complex scientific and financial datasets with detailed table structure annotations to help train and test structure recognition. Our framework surpasses previous state-of-the-art results on the ICDAR 2013 and ICDAR 2019 table competition in both table detection and cell structure recognition with a significant 5.8% improvement in the full table extraction system. Further experiments demonstrate a greater than 45% improvement in cell structure recognition when compared to a vanilla RetinaNet object detection model in our new out-of-domain FinTabNet.
1. Introduction
GTE addresses the difficulty of reconstructing table boundaries and cell structure from PDFs or scanned images by combining vision-guided detection, cell-aware training, hierarchical recognition, and automatically generated annotations. The framework reports stronger benchmark performance and introduces datasets intended to support table-related tasks.
- Problem: PDF and scanned documents preserve neither table boundaries nor logical cell relations, limiting downstream uses such as question answering and knowledge-base population.Table detection identifies table boundaries, while cell structure recognition recovers logical row and column relations.
- Problem: Rule-based and statistical extraction methods often fail across heterogeneous table styles because they depend on handcrafted cues such as graphical lines or bold fonts.Graphical ruling lines may be absent or may not delineate every cell.
- Framework: GTE jointly performs table detection and cell structure recognition using a vision-guided framework that can be built on object detection models.Its networks exchange context: GTE-Table uses cell predictions, while GTE-Cell uses table boundaries and table-level style information.
- Results: GTE outperforms previous systems on the ICDAR 2013 and 2019 table competition benchmarks.The cited contribution states the benchmark comparison without specifying the individual metric values.
- Framework: GTE-Table uses cell detection to guide table-network training, including a penalty based on the constraint that tables contain cells.The method also feeds a predicted cell mask into the table network as an image channel.
- Framework: GTE-Cell directly detects cells with a hierarchical network and converts detections into structure using spatial clustering while leveraging table styles.This avoids relying only on separately detected full rows and columns and supports tables whose rows or columns do not span the entire table.
- Data: The authors automatically generate fine-grained table and cell annotations, enhance PubTabNet, and create FinTabNet from real-world data sources.The datasets include logical structure and cell-location information for table-related training and evaluation.
2. Related Work
Table recognition has progressed from handcrafted and statistical methods toward deep learning, but available datasets remain insufficient for detailed cell structure recognition. Existing deep-learning approaches include image-to-sequence and object-detection-based systems, while the paper develops expanded datasets with richer annotations.
- Table Detection: Rule-based table detectors use text-block arrangement, horizontal and vertical lines, and item blocks but require extensive manual rule design and generalize poorly across heterogeneous formats.Statistical methods were introduced to address some of these limitations.
- Table Detection: Deep-learning table-recognition methods comprise end-to-end image-to-sequence models and object-detection-based methods.Object detectors such as Faster-RCNN have been adapted for table detection.
- Datasets: Detailed cell-structure datasets require pixel locations, logical coordinates such as row-span and col-span, and cell text, but existing resources provide limited coverage or incomplete annotation.ICDAR2013 had 254 examples, while TableBank had 145K labelled tables but only logical cell coordinates.
3. PubTabNet, FinTabNet
The paper expands table-structure resources and presents a vision-based framework that combines table detection, cell detection, and structure conversion. Its datasets provide large-scale annotations across scientific, financial, and related document pages.
- Datasets: PubTabNet was enhanced with cell labels, while PubXNet provides pages containing full table and cell information.The labels are generated by matching PDF and HTML documents: HTML supplies logical cell structure, while PDF supplies boundary locations.
- Datasets: FinTabNet contains complex tables from S&P 500 annual reports with full table, structure, and cell bounding-box annotations.It includes more than 70K annotated pages and more than 110K tables with cell bounding boxes.
- Framework: GTE combines GTE-Table and GTE-Cell, with cell structure recognition using table boundaries and table-level style information.The cell network depends on table boundaries produced by the table network for each specific table.
- GTE-Table: During table-network training, a piecewise constraint loss penalizes unrealistic table detections using predicted cell locations.The loss can be added to any detection network, and a cell-prediction mask further aids table-network training.
- GTE-Table: The constraint indicator marks table boxes with insufficient internal cell coverage or cells outside the proposed table boundary.The formulation also includes bottom-interior coverage, while inference ranking uses cells just inside and outside candidate tables.
- GTE-Cell: GTE-Cell converts detected cell boxes into logical rows and columns through text alignment, directional sampling, cell merging, and post-processing.The procedure expands cell edges for missing cells and merges likely over-split cells based on capitalization cues.
5. Experiments
Experiments evaluate GTE across table detection, cell structure recognition, ablations, and additional scientific and financial datasets. Results show strong benchmark performance, benefits from specialized and hierarchical components, and limitations related to boundary and aspect-ratio handling.
- Evaluation Metrics: ICDAR2013 experiments measure table detection with character-level recall, precision, F1, purity, and completeness, and cell structure recognition with adjacency-matrix precision, recall, and F1.The evaluation uses the official ICDAR2013 table competition script for table detection.
- Table Detection: GTE-Table achieves the best character-level F1 measure among the compared ICDAR2013 table detection methods.FineReader slightly outperforms GTE on purity, but GTE obtains higher character-level F1.
- Table Detection: GTE-Table produces no table non-detections and only one mis-detection on the ICDAR2013 test results, while remaining errors are partial under- or over-detections.Most partial detections miss or add one or two extra lines, such as short captions.
- Table Detection Ablation Study: The specialized table and cell networks outperform Detection-base, while cross-network information provides a nearly 3% F1 boost over regular object-detection losses.Detection-base is disadvantaged by missing cell annotations in TableBank and the different scales of tables and cells.
- Cell Structure Recognition: GTE-Cell outperforms previous methods and commercial software across all reported ICDAR2013 cell-structure metrics, with and without ground-truth table borders.The cell evaluation uses outputs from each framework's table detector rather than ground-truth tables.
- Experiments with Additional Datasets: Cell detection remains limited by anchor-based handling of very long text lines, and tables without graphical lines may have multiple cells mistakenly merged.ICDAR2019 evaluation also raises concerns about IOU=0.9 because annotations inconsistently include or exclude whitespace.
- Cell Structure Ablation Study: The hierarchical GTE-Cell model outperforms its individual style-specific submodels, while the attributes network classifies graphical-line table style correctly for 123 of 156 tables (78.84%).The hierarchical design first determines table style and then selects a model trained on similar data; alternate-model selection uses sampling-point variability.
- Experiments with Additional Datasets: GTE transfers to ICDAR2019, PubTabNet, and FinTabNet, including a PubTabNet TEDS score of 93.01 versus the original score of 88.38.The PubTabNet and FinTabNet results use validation PDFs and are not directly comparable with an unavailable test set; styling tags were ignored in evaluation.
6. Conclusion and Future Work
The paper presents a vision-based framework for extracting table structure from documents, reports state-of-the-art results, and releases enhanced datasets for downstream research.
- GTE performs the full pipeline from document images to table structure for downstream analysis.The framework can be combined with OCR to perform table recognition fully from images.
- GTE leverages global table visual context, including table style and relationships between cells and tables.
- The authors enhanced PubTabNet and planned to release FinTabNet to support data-hungry table-related methods.
1 Visualization of different structure format
This section references a comparison of TableBank and ICDAR2013 structure annotations and identifies the paper's 2020 arXiv version.
- Figure 1 compares structure annotations from TableBank and ICDAR2013.
- The referenced document is arXiv version 2, dated 2 December 2020.
2 Experimental Details
The experimental setup modifies RetinaNet-based components for wide tables and small cells, while inference uses table-characteristic heuristics to resolve competing detections.
- GTE-Table adds wide-table anchors with aspect ratios 0.1 and 0.25 and uses a 900 ∗643 input image size.
- GTE-Cell combines a graphical-line classifier with two style-specialized cell detectors in a hierarchical network.The cell detectors use pyramid levels P3 and P5, additional wide-cell anchors, and smaller-scale anchors.
- The graphical-line classifier is pretrained on SD-Tables attributes and fine-tuned on ICDAR training data.Its labels derive from nearby vertical graphical lines detected by a PDF parser.
- Joint-training hyperparameters are selected from ICDAR training-data characteristics, including character height and cell-density estimates.The reported choices include µ1 = 5, µ2 = 5, and α = 1/8.
- Inference prioritizes tables with tabular characteristics, including cells inside the table and limited text outside it.The procedure also handles overlapping tables with differing shapes and similar confidence levels.
4 Additional cell detection examples
The section describes examples and a clustering pipeline that converts detected cell boundaries into rows, columns, and table structure by aligning boxes and assigning text.
- Additional cell detection examples: Algorithm 1 preprocesses cell boxes by deleting boxes without text overlap and expanding boxes to cover intersecting text or cell boxes.
- Additional cell detection examples: The algorithm estimates row and column counts from intersecting cell midpoints and derives x- and y-axis alignments.
- Additional cell detection examples: Detected cells are assigned to columns and rows using KMeans clustering of aligned x- and y-coordinates.
- Additional cell detection examples: Text lines intersecting cell boxes inherit the corresponding cell's row and column assignments.
- Additional cell detection examples: When a row-column position is empty, neighboring text lines are reconsidered and realigned to rows and columns.
- Additional cell detection examples: Figure 2 presents additional examples of converting cell boundaries into structure.
5 Detailed ICDAR13 Results
This section directs readers to Tables 1 and 2, which present ICDAR 2013 results for table detection and cell structure recognition.
- Tables 1 and 2 contain the reported ICDAR 2013 evaluation results.
- Table 1 reports ICDAR 2013 table detection results with additional comparisons.
- Table 2 reports ICDAR 2013 cell structure results with additional comparisons.
6 ICDAR19 evaluation metric ambiguities
The section highlights cell detection errors and an evaluation ambiguity involving high-IoU matching. Visually correct bounding boxes can receive no matches at an IoU threshold of 0.9.
- Figure 3 presents examples of cell detection errors.
- At IoU=0.9, visually correct detected cell boxes may receive zero matches despite containing all ground-truth characters.
- Figure 4 illustrates the discrepancy between visual correctness and matching outcomes at a high IoU threshold.