Source-linked AI summary
TableNet: Deep Learning model for end-to-end Table detection and Tabular data extraction from Scanned Document Images
Shubham Paliwal, Vishwanath D, Rohit Rahul, Monika Sharma, Lovekesh Vig
TL;DR
TableNet targets the challenge of extracting tabular content from varied document images, where table detection and structure recognition are often handled separately. It introduces an end-to-end multi-task model with shared table-column representations, mask-based extraction, and rule-based row processing. Evaluations on ICDAR 2013 and Marmot report state-of-the-art results, semantic-feature gains, and transfer learning across datasets.
Problem
Extracting tables from document images requires accurate table-region detection and fine-grained row and column structure recognition, while prior approaches commonly separate these tasks.
Method
TableNet jointly trains a shared pre-trained VGG-19 encoder with separate table and column decoders, then applies semantic rule-based row extraction to OCR-filtered regions.
Results
TableNet achieves state-of-the-art results on ICDAR 2013 and Marmot, with semantic features improving performance and fine-tuning enabling transfer across datasets.
Takeaways & Limitations
The model jointly addresses table detection and structure recognition, transfers knowledge to related datasets, and provides new Marmot structure annotations for future research.
Abstract
from arXiv · showhide
With the widespread use of mobile phones and scanners to photograph and upload documents, the need for extracting the information trapped in unstructured document images such as retail receipts, insurance claim forms and financial invoices is becoming more acute. A major hurdle to this objective is that these images often contain information in the form of tables and extracting data from tabular sub-images presents a unique set of challenges. This includes accurate detection of the tabular region within an image, and subsequently detecting and extracting information from the rows and columns of the detected table. While some progress has been made in table detection, extracting the table contents is still a challenge since this involves more fine grained table structure(rows & columns) recognition. Prior approaches have attempted to solve the table detection and structure recognition problems independently using two separate models. In this paper, we propose TableNet: a novel end-to-end deep learning model for both table detection and structure recognition. The model exploits the interdependence between the twin tasks of table detection and table structure recognition to segment out the table and column regions. This is followed by semantic rule-based row extraction from the identified tabular sub-regions. The proposed model and extraction approach was evaluated on the publicly available ICDAR 2013 and Marmot Table datasets obtaining state of the art results. Additionally, we demonstrate that feeding additional semantic features further improves model performance and that the model exhibits transfer learning across datasets. Another contribution of this paper is to provide additional table structure annotations for the Marmot data, which currently only has annotations for table detection.
I. INTRODUCTION
TableNet addresses the costly challenge of extracting tabular information from varied document images by jointly modeling table detection and structure recognition. It combines an end-to-end multi-task model with semantic features, transfer learning, and expanded Marmot annotations.
- Document-image information extraction is pressing because manual processing is costly and tables vary substantially in layout and visual appearance.
- Existing approaches commonly separate table detection from table structure recognition, including row and column segmentation.
- TableNet jointly addresses both tasks with a shared encoder and separate decoder branches that segment table and column regions, followed by rule-based row extraction.
- The paper reports marginally better ICDAR-2013 detection and extraction results than other deep and state-of-the-art methods, with transfer across datasets after minimal fine-tuning.
- The contributions include semantic-feature augmentation, transfer learning, and publicly releasing manual Marmot column annotations.
- The paper proceeds from related work through model design, extraction, datasets and training, experiments, and conclusions.
II. RELATED WORK
Prior work addressed table detection and tabular extraction with heuristics, probabilistic models, clustering, geometric analysis, and deep learning. These approaches used varied representations and often treated detection and structure recognition separately.
- Prior studies frequently reported table detection and data extraction as separate problems rather than a unified task.
- Earlier table-detection systems used structural heuristics, hierarchical MXY representations, intersecting lines, low-level features, and SVM classification.
- Probabilistic approaches modeled visual page elements or used table headers and whitespace to detect and decompose tables, including borderless tables.
- DeepDeSRT applied deep learning to table detection and structure recognition, identifying rows, columns, and cell positions in detected tables.
- Other systems combined convolutional networks, graphical models, saliency, text-component rules, clustering, geometric probabilities, HTML formatting, or document-layout object detection.
III. TABLENET: DEEP MODEL FOR TABLE AND COLUMN DETECTION
TableNet uses a shared VGG-19 encoder and separate decoder branches to produce pixel-wise table and column masks. The architecture exploits the column-region subset relationship and combines encoder features through skip connections.
- The model jointly trains shared encoding layers with table and column targets while keeping the two task decoders separate.
- Figure 1 depicts the Marmot training image and the common VGG-19 encoder feeding conv7 table and conv7 column decoder branches.
- TableNet replaces VGG-19 fully connected layers after pool5 with two 1x1 convolution layers, ReLU activations, and dropout before branching.
- The column region is treated as a subset of the table region, allowing shared features from both regions to filter active areas before task-specific decoding.
- The table and column branches upsample feature maps and combine them with pool4 and pool3 encoder features before returning to the original image size.
- Figure 2 shows a sample document image alongside the output masks generated by TableNet.
- Both computational branches output masks for table and column regions, and the model produces separate semantically labeled output images from one input image.
IV. TABLE ROW EXTRACTION
After TableNet generates table and column masks, OCR word positions are filtered to the detected regions and grouped into rows using rule-based spatial cues. The rules handle line demarcations, multiline entries, and fully populated columns.
- TableNet filters OCR-recognized word patches that lie inside the predicted table and column regions.
- A row is defined as words from multiple columns at a similar horizontal level, but it may span multiple lines.
- Horizontal line demarcations are detected between vertically placed words using a Radon transform and segment rows when present.
- For multiline rows, a new row begins when the entities across columns are filled, using the rows with maximum non-blank entries as the starting point.
- When columns are completely filled and no line demarcations exist, each horizontal text level is treated as a unique row.
V. DATASET PREPARATION
Deep-learning table analysis is constrained by limited annotated datasets, especially for structure recognition. The authors address this by adding column annotations to Marmot and releasing the resulting Marmot Extended dataset.
- Dataset constraints: Few public datasets provide sufficient training data for table detection and structure analysis.Marmot and UW3 contain only a few hundred images, while structure-identification datasets are even fewer.
- Dataset constraints: Marmot is the largest publicly available table-detection dataset but lacks annotations for table columns and rows.Its original ground truth covers table detection only.
- Marmot Extended: The authors manually labeled column bounding boxes and publicly released the modified dataset as Marmot Extended.The annotations support table structure recognition.
A. Providing Semantic Information
TableNet receives additional semantic and spatial cues by color-coding document words according to their basic data types. These cues encode regularities such as strings in name columns and numbers in quantity columns.
- Semantic encoding: Words are color-coded by basic data type to provide semantic information about table content.A name column may contain strings, whereas a quantity header may contain numbers.
- Semantic encoding: The color-coded word patches are included in the network input to provide both semantic and spatial information.The modified images are intended to improve model performance.
- Preprocessing: Document images are histogram-equalized and processed with Tesseract OCR before word patches are colored.The resulting modified images are used as TableNet inputs.
B. Training Data Preparation for TableNet
The training-data preparation combines OCR-derived, data-type-colored word regions with evaluation tables for detection and structure recognition. The supplied passages describe the data-processing setup but do not provide table values.
- Training input preparation: Word patches are generated from Tesseract OCR and assigned colors according to data type using regular expressions.Word bounding boxes are filtered to remove spurious detections.
- Evaluation outputs: Table I reports results for table detection.The supplied material gives the table caption but not the underlying values.
- Evaluation outputs: Table II reports results for table structure recognition and data extraction.The supplied material gives the table caption but not the underlying values.
VI. EXPERIMENTS AND RESULTS
The experiments evaluate TableNet for table detection and data extraction using document-level precision, recall, and F1-score. Training uses paired image, table-mask, and column-mask data, with semantic features tested in a separate experiment.
- Evaluation: Performance is evaluated using Recall, Precision, and F1-score averaged across documents.The measures are computed for each document before averaging.
- Table detection: Table detection evaluation uses character-level sub-objects to distinguish major and minor segmentation errors.Completeness and purity motivate the region-based evaluation, while precision and recall are calculated over the sub-objects.
- Data extraction: Table data extraction represents each cell through adjacency relations with its nearest horizontal and vertical neighbors.Normalized neighboring-cell text is organized as a 1D-tuple and compared with ground truth using precision and recall.
- Training setup: The training set contains 509 annotated English Marmot documents drawn from 1016 documents containing tables.Each training sample includes a document image, table mask, and column mask.
- Semantic-feature experiment: Adding spatial semantic information to the input images produced a slight improvement in results.Experiment 2 used modified Marmot images with highlighted words while keeping Experiment 1 parameters identical.
VII. CONCLUSION
TableNet jointly performs table detection and structure recognition end-to-end, exploiting their interdependence. The paper reports transfer learning across related datasets and improved performance from semantic data-type information.
- TableNet jointly addresses table detection and structure recognition end-to-end instead of treating them as independent problems.
- The model exploits knowledge from previously learned tasks to transfer to newer, related tasks, which is particularly useful when training data is sparse.
- Highlighting text to provide data-type information improves the model’s performance.
- A future third branch for row identification would require manual row-based annotations that current datasets do not provide.