Source-linked AI summary
CascadeTabNet: An approach for end to end table detection and structure recognition from image-based documents
Devashish Prasad, Ayan Gadpal, Kshitij Kapadni, Manish Visave, Kavita Sultanpure
TL;DR
Table image understanding requires both detecting tables and recognizing their structure, tasks traditionally handled independently. CascadeTabNet uses a single Cascade Mask R-CNN HRNet model with simultaneous table and cell segmentation, supported by transfer learning and image augmentation. It achieved leading results across the evaluated ICDAR and TableBank benchmarks, while structure evaluation was limited by dataset annotation and metric constraints.
Problem
Table recognition in document images requires jointly addressing table detection and structure recognition, which prior work generally solved with separate approaches.
Method
CascadeTabNet uses a single Cascade Mask R-CNN HRNet model for end-to-end table and cell segmentation, with iterative transfer learning and document-focused image augmentation.
Results
The model ranked third for ICDAR 2019 table detection, achieved highest detection accuracy on ICDAR 2013 and TableBank, and highest post-competition rank for ICDAR 2019 structure recognition.
Takeaways & Limitations
Instance-segmentation CNNs, iterative transfer learning, and image augmentation can support end-to-end table understanding and learning from limited data across datasets.
Takeaways & Limitations
TableBank could not support structure evaluation because it lacks cell or column coordinates, while ICDAR 2013 evaluation depends on OCR-extracted cell text.
Abstract
from arXiv · showhide
An automatic table recognition method for interpretation of tabular data in document images majorly involves solving two problems of table detection and table structure recognition. The prior work involved solving both problems independently using two separate approaches. More recent works signify the use of deep learning-based solutions while also attempting to design an end to end solution. In this paper, we present an improved deep learning-based end to end approach for solving both problems of table detection and structure recognition using a single Convolution Neural Network (CNN) model. We propose CascadeTabNet: a Cascade mask Region-based CNN High-Resolution Network (Cascade mask R-CNN HRNet) based model that detects the regions of tables and recognizes the structural body cells from the detected tables at the same time. We evaluate our results on ICDAR 2013, ICDAR 2019 and TableBank public datasets. We achieved 3rd rank in ICDAR 2019 post-competition results for table detection while attaining the best accuracy results for the ICDAR 2013 and TableBank dataset. We also attain the highest accuracy results on the ICDAR 2019 table structure recognition dataset. Additionally, we demonstrate effective transfer learning and image augmentation techniques that enable CNNs to achieve very accurate table detection results. Code and dataset has been made available at: https://github.com/DevashishPrasad/CascadeTabNet
1. Introduction
CascadeTabNet addresses table detection and structure recognition jointly with one end-to-end deep-learning model, while combining transfer learning and image augmentation for broad dataset performance.
- Table information extraction comprises table detection and table structure recognition, which prior approaches solved independently.Detection identifies the table region, while structure recognition identifies rows, columns, and individual cells.
- CascadeTabNet uses a single model to perform table and cell instance segmentation in one inference.The model predicts table regions and segmented cell regions simultaneously to recognize table structure.
- The model achieved third place for ICDAR 2019 table detection, highest accuracy on ICDAR 2013 and TableBank detection, and the highest rank for ICDAR 2019 structure recognition.
- The Cascade Mask R-CNN HRNet approach combines end-to-end detection and structure recognition within a deep-learning framework.
- Proposed image transformations significantly enhance table detection accuracy during training.
- Iterative transfer learning enables performance across datasets using a small amount of training data.The model is fine-tuned on respective datasets after learning from more general tasks.
- The authors manually annotated some ICDAR 2019 images for cell detection in borderless tables and categorized tables as bordered or borderless.The annotations were intended for release to the community.
2. Related work
Related work progressed from heuristic and object-based methods toward deep-learning approaches for table detection, structure recognition, and joint processing.
- Early table detection methods used character alignment, holes, gaps, junctions, and SVM-based junction information.
- Deep-learning table detection introduced Faster R-CNN, distance-based augmentation, semantic color coding, and page segmentation approaches.
- Earlier table structure recognition methods grouped text into chunks, used column borders, constructed cells from lines and characters, or applied graph-based modeling.
- Some systems combined table detection and structure recognition using separate Faster R-CNN, semantic segmentation, GAN, or SegNet components.
- Paliwal et al. proposed an end-to-end system using a shared VGG-19 encoder, separate decoders for table and column detection, rule-based row extraction, and Tesseract OCR.
3. CascadeTabNet: The presented approach
CascadeTabNet combines table detection and structure recognition in one Cascade mask R-CNN HRNet model, supported by two-stage transfer learning and document-specific image augmentation.
- 3.1. Model architecture: Experiments found the cascaded multi-stage model with the HRNet backbone achieved the best results among the analyzed CNN models.The authors attribute this to both components supporting high-accuracy object segmentation.
- 3.1. Model architecture: CascadeTabNet uses a Cascade mask R-CNN HRNet model for end-to-end table detection and structure recognition.The model performs both tasks with one CNN-based approach.
- 3.1. Model architecture: The HRNetV2p W32 backbone produces feature maps, while proposal, bounding-box, and mask heads predict table and cell segmentations.The mask branch complements detected objects with segmentation outputs during inference.
- 3.2. Iterative transfer learning: Two-stage transfer learning trains the same model first for general table detection, then for more specific table-type and cell-mask prediction.The strategy progressively moves from broader to narrower tasks and uses smaller datasets in later stages.
- 3.3. Image Transformation and data augmentation: The method augments document images with dilation and smudge transformations to thicken or spread black pixel regions and increase relevant training data.Dilation uses a 2x2 kernel for one iteration; smudge uses distance transforms to create smeared regions.
4. Dataset Preparation
The study prepares detection and structure-recognition datasets by combining public sources, selecting subsets, transforming images, and adding manual annotations for table cells.
- General dataset preparation: The general table-detection dataset merges ICDAR 19 Modern, Marmot, and an internet dataset containing borderless tables.ICDAR 19 contributes modern Word and LaTeX documents, while Marmot contributes Chinese and English subsets.
- Image augmentation: For preliminary augmentation experiments, four training sets contain original images, original-plus-dilated images, original-plus-smudged images, or all three variants.These sets compare the effects of dilation and smudge transformations.
- Dataset-specific preparation: The ICDAR 19 Modern evaluation data is dilated before the general model is fine-tuned on it.The transformed images are applied to all images in the Track A Modern dataset.
- Evaluation setup: Testing uses ICDAR 19 Track A Modern because it contains varied document types and table forms.The set includes LaTeX and Word images and multiple table types.
- TableBank preparation: TableBank experiments use Word and LaTeX subsets, with randomly selected training and testing images rather than fine-tuning on the entire dataset.The combined Word-plus-LaTeX split contains 3,000 training images and 2,000 testing images.
- ICDAR 13 preparation: The ICDAR 13 data provides 238 converted document images, with 40 randomly selected for fine-tuning and the remainder for testing.Its 150 tables span EU and US subsets.
- Structure-recognition dataset: For structure recognition, 342 ICDAR 19 Modern training images receive manual annotations for bordered tables, borderless tables, and borderless-table cells.The annotations cover two table classes and one cell class.
5. Results and Analysis
CascadeTabNet’s evaluation combines augmentation and CNN architecture comparisons with benchmark testing for table detection and structure recognition. It achieves strong results across datasets, while structure evaluation remains bounded by dataset label formats and some difficult images.
- Preliminary Analysis: Both image transformation techniques improve learning effectiveness, so subsequent table-detection experiments use them together on the General dataset.
- Preliminary Analysis: The Cascade mask R-CNN HRNet model achieves the highest accuracy among compared models on ICDAR Test Track A Modern.The passage attributes this to multi-stage cascading and high-resolution convolutions for instance segmentation.
- Table detection evaluation: 3rd rank on ICDAR 19 Track A post-competition results was achieved for table detection, with the best accuracy at IoU 0.9.The comparison is based on weighted-average metrics and the reported IoU 0.9 result.
- Table detection evaluation: Best accuracy results were achieved on all three TableBank detection subsets using only 1,500 Word, 1,500 Latex, and 3,000 Word+Latex training images.
- Table structure recognition evaluation: The model attained the highest post-competition accuracy on ICDAR 19 Track B2 table structure recognition.Evaluation compares predicted cell polygons and row or column adjacency information using IoU-thresholded precision, recall, and F1 scores.
- Table structure recognition evaluation: TableBank was unsuitable for structure evaluation because it provides HTML-tag labels without cell or column coordinates, while ICDAR 13 requires OCR-dependent cell-text mapping.
- Table structure recognition evaluation: Some missed borderless-table cell predictions are corrected with line estimation and contour-based text detection, but the model fails badly on some images.
7. Conclusion
The paper presents an end-to-end system for table detection and structure recognition using instance-segmentation CNNs, transfer learning, augmentation, and line information. It reports better performance across public datasets, while noting that improved post-processing could further enhance accuracy.
- CascadeTabNet performs table detection and structure recognition end to end by predicting table and cell masks with an instance-segmentation CNN.
- Iterative transfer learning and image augmentation help the model learn efficiently from small amounts of data and adapt to specific datasets.
- The system performs better on various public datasets for both table detection and structure recognition.
- Improving the post-processing modules could further enhance the end-to-end model’s accuracy.