PaperlayerView exact revision on arXiv

Source-linked AI summary

TableFormer: Table Structure Understanding with Transformers

Ahmed Nassar, Nikolaos Livathinos, Maksym Lysak, Peter Staar

arXiv:2203.01017v2cs.CVcs.LG

TL;DR

Table structure extraction must handle diverse layouts, complex headers, multiline content, and missing entries. TableFormer addresses this with an end-to-end transformer model that predicts structure and cell bounding boxes, using PDF cell content when available, and the paper reports strong benchmark improvements across datasets.

  • Problem

    Table structure is difficult to identify from images because tables contain diverse layouts, complex headers, multiline entries, and missing cells.

  • Method

    TableFormer uses an end-to-end transformer architecture to predict table structure and cell bounding boxes, while SynthTabNet provides varied synthetic training data.

  • Results

    TableFormer significantly outperforms existing state-of-the-art metrics across all benchmark datasets and is more efficient in training and inference.

  • Takeaways & Limitations

    The approach supports reconstructing table structure and extracting cell content from PDF or OCR across varied document types and languages.

  • Takeaways & Limitations

    Existing datasets have limited diversity in table structures, styles, and domains, which can damage generalization to unseen data.

Abstract

from arXiv · show

Tables organize valuable content in a concise and compact representation. This content is extremely valuable for systems such as search engines, Knowledge Graph's, etc, since they enhance their predictive capabilities. Unfortunately, tables come in a large variety of shapes and sizes. Furthermore, they can have complex column/row-header configurations, multiline rows, different variety of separation lines, missing entries, etc. As such, the correct identification of the table-structure from an image is a non-trivial task. In this paper, we present a new table-structure identification model. The latter improves the latest end-to-end deep learning model (i.e. encoder-dual-decoder from PubTabNet) in two significant ways. First, we introduce a new object detection decoder for table-cells. In this way, we can obtain the content of the table-cells from programmatic PDF's directly from the PDF source and avoid the training of the custom OCR decoders. This architectural change leads to more accurate table-content extraction and allows us to tackle non-english tables. Second, we replace the LSTM decoders with transformer based decoders. This upgrade improves significantly the previous state-of-the-art tree-editing-distance-score (TEDS) from 91% to 98.5% on simple tables and from 88.7% to 95% on complex tables.

1. Introduction

Table structure extraction is difficult because tables combine varied layouts, implicit conventions, and incomplete or multiline content. TableFormer addresses these challenges with an end-to-end transformer model that predicts structure and cell bounding boxes, alongside datasets designed for varied table appearances and complexity.

  • Motivation: Table images combine complex headers, multiline entries, empty cells, varied separators, and diverse shapes, making machine parsing difficult.These features are illustrated in the PubTabNet evaluation example.
  • Approach: TableFormer predicts table structure and cell bounding boxes simultaneously in an end-to-end transformer-based approach.The model is intended to link predicted cells directly to their locations in the source image.
  • Results: Across all benchmark datasets, TableFormer significantly outperforms existing state-of-the-art metrics while improving training and inference efficiency.The introduction reports this as the model’s overall benchmark result.
  • Datasets: The work introduces SynthTabNet, a synthetic dataset covering varied appearance styles and table complexity.It also augments PubTabNet, FinTabNet, and TableBank with generated ground truth for reproducibility.

2. Previous work and State of the Art

Prior table-structure methods include image-to-text, graph-neural-network, and hybrid rule-based approaches, but they retain important limitations. In particular, image-to-text systems rely on custom OCR, while hybrid systems require rules for different table types.

  • Challenges: Table-structure identification remains difficult because tables vary widely and large, diverse structure annotations are costly to produce.Complex headers and limited structure datasets are identified as major sources of difficulty.
  • Image-to-Text Networks: Image-to-text networks predict HTML or LaTeX sequences, using either a single decoder or separate tag and content decoders.The dual-decoder design first constructs the empty table and then generates cell text.
  • Image-to-Text Networks: Image-to-text architectures require custom trained OCR decoders, limiting practical use across languages with little non-English training data.Programmatic PDFs can instead provide cell content and positions without OCR.
  • Graph Neural Networks: Graph neural networks represent text-cells as nodes with image, coordinate, and text features to recover table structure.A single table cell may consist of multiple text-cells.
  • Hybrid Approaches: Hybrid deep-learning and rule-based systems detect cells, classify table types, and apply type-specific rules, rather than operating end to end.Encountering new table types requires writing additional rules.

3. Datasets

The paper combines established table datasets with synthetic data to address skewed structure, style, and content distributions. SynthTabNet provides controlled variation in dataset size, structure, style, and content.

  • 3. Datasets: The training and evaluation sources include PubTabNet, FinTabNet, and TableBank, supplemented by the synthetically generated SynthTabNet dataset.The paper also describes a combined synthetic dataset containing 600k examples.
  • 3. Datasets: PubTabNet contains 509k annotated PNG tables with HTML structure, tokenized text, and per-cell bounding boxes.Tables are categorized as simple or complex according to whether they contain row or column spans.
  • 3. Datasets: The datasets are homogenized into a common PNG-based format, with extreme table sizes filtered to 1*1 through 20*10 rows/columns.Missing cell bounding boxes are generated during preprocessing, including for many FinTabNet tables.
  • 3. Datasets: Existing datasets are skewed toward simpler tables with fewer rows and columns, limited styling variance, and domain-restricted content.These limitations can damage model generalization to unseen data.
  • 3. Datasets: SynthTabNet offers control over dataset size, table structure, table style, and content to address limitations in existing datasets.Its structure variation includes header and body size and the proportion of cells covered by row and column spans.

4. The TableFormer model

TableFormer encodes a table image and jointly predicts HTML structure tokens and synchronized cell bounding boxes. Its transformer-based structure decoder and cell-specific bounding-box decoder are trained with a combined multi-task loss.

  • 4. The TableFormer model: TableFormer jointly predicts table-structure tokens and cell bounding boxes from an input image in an end-to-end architecture.The synchronized predictions link each detected cell region to the corresponding structural representation.
  • 4.1. Model architecture: A CNN backbone encodes the table image, and the resulting features are passed to both the Structure Decoder and Cell BBox Decoder.The implementation uses a modified ResNet-18 with adaptive pooling to produce a 28*28 feature map.
  • 4.1. Model architecture: The transformer Structure Decoder generates HTML tags, while a shared feed-forward network converts decoder hidden states into bounding-box coordinates and cell classifications.The decoder uses a transformer encoder and a four-layer transformer decoder with multi-head attention and feed-forward layers.
  • 4.1. Model architecture: The Cell BBox Decoder uses hidden states for data-cell tags as object queries, enabling simultaneous HTML and cell-box prediction without a separate detector.An attention network and MLP predict normalized cell coordinates, followed by classification of cells as empty or non-empty.
  • 4.1. Model architecture: The model is trained with a multi-task loss combining cross-entropy for structure tokens with L1 and IoU losses for cell bounding boxes.The weighting coefficient λ and the bounding-box loss coefficients are hyperparameters.

5. Experimental Results

TableFormer is evaluated for structure prediction, cell detection, and content recovery across multiple datasets, with qualitative tests covering varied styles and unseen language.

  • Structure: TableFormer outperforms state-of-the-art methods across datasets for table-structure prediction.The evaluation includes PubTabNet, FinTabNet, TableBank, and SynthTabNet.
  • Structure: Large tables can produce poor predictions after image resizing because downsampling makes their features indistinguishable.The paper suggests handling such tables with a separate model using a larger input image size.
  • Cell Detection: A separate cell-bounding-box decoder predicts content-cell boxes, with structure-decoder errors directly affecting its predictions.The decoder is evaluated using PASCAL VOC mAP, including pre-processing and post-processing settings.
  • Cell Content: 5.3% higher TEDS is achieved for structure with content retrieved from PDF cells compared with state-of-the-art and commercial solutions.The authors note that discrepancies between HTML ground truth and extracted PDF text may reduce the score.
  • Qualitative Analysis: TableFormer predicts bounding boxes for empty as well as populated cells and extracts Japanese text despite training only on English content.The visualizations also cover diverse table styles and content types.

6. Future Work & Conclusion

The paper presents TableFormer as an end-to-end transformer approach for predicting table structures and cell bounding boxes, alongside SynthTabNet as a synthetic dataset. It reports broad state-of-the-art performance and versatility across document types and languages.

  • Conclusion: TableFormer predicts table structures and cell bounding boxes from images in an end-to-end transformer-based approach.Bounding boxes enable reconstruction of table structure and extraction of cell content from PDF or OCR.
  • Conclusion: The paper introduces SynthTabNet, a synthetically generated dataset designed to reinforce missing characteristics in other datasets.
  • Conclusion: TableFormer outperforms state-of-the-art methods with a wide margin and supports varied PDF documents and languages.

1. Details on the datasets

The dataset preparation analyzes table size, complexity, HTML strictness, and annotation completeness, while synthetic datasets broaden variation in structure, style, content, and appearance.

  • Data preparation: Dataset statistics cover table size, table complexity, HTML strictness, and completeness of cell bounding-box annotations.Table size is measured by rows and columns, while strict tables have equal row lengths after accounting for spans.
  • Data preparation: Missing bounding boxes can be derived from neighboring boxes by constructing a fine-grained grid over the table structure.The method uses geometrical border lines when enough boxes are known for a rectangular table.
  • Synthetic data: Four synthetic dataset types contain 150k examples each, totaling 600k examples split into training, test, and validation sets.The splits are 80%, 10%, and 10%, respectively.
  • Synthetic data: Synthetic tables vary in size, structure, style, and content through templates for styling, structures, content, and rendering.Generated structures include multi-row headers and combinations of row and column spans.

2. Prediction post-processing for PDF documents

For programmatic PDFs, TableFormer predictions are post-processed by matching predicted cells to PDF cells and correcting geometric errors. The procedure uses grid structure, IOU thresholds, alignment statistics, and orphan-cell recovery to improve table reconstruction.

  • TableFormer still does not include table-cell content and can occasionally produce inaccurate bounding boxes, limiting standalone full-table reconstruction.The supplementary examples include complex, empty, multiline, colorful, triangular, and long tables, including post-processing corrections.
  • Post-processing links predicted table cells with PDF cells so PDF content and bounding boxes can correct structural misalignments.The process treats PDF documents as sequences of cells described by content and bounding boxes.
  • The pipeline first builds the minimal predicted grid, matches PDF and predicted bounding boxes with IOU, and separates good from bad matches using a threshold.Columns whose IOU scores all fall below the threshold can have their structure and bounding-box predictions discarded.
  • Column alignment selects among left, centroid, and right x-coordinates, then median column positions and cell sizes provide robust geometric references.Median statistics reduce the influence of unusually wide column-spanning cells.
  • Cells with poor IOU are snapped to median coordinates and sizes, then rematched using PDF-cell-area-normalized intersection scores.When multiple predictions match one PDF cell, the higher-scoring prediction is retained.
  • The procedure handles duplicate columns and orphan PDF cells by using intersection scores and grid-row or grid-column geometry.Orphan content is appended to an existing cell or assigned to a new structural cell depending on whether the identified grid position is occupied.
Loading 2203.01017v2…