PaperlayerView exact revision on arXiv

Source-linked AI summary

Enhancing Table Structure Recognition via Bounding Box Guidance

Lei Hu, Shuangping Huang

arXiv:2609.08705v1cs.CV

TL;DR

TSR methods can overlook explicit bbox information when generating HTML, weakening recognition in complex scenes. BGTR uses predicted bboxes to guide HTML decoding and combines this with progressive training and SNSTab for natural-scene tables. Experiments report state-of-the-art performance on five benchmark datasets.

  • Problem

    Image-to-sequence TSR methods overlook explicit bbox use during HTML prediction, while natural-scene tables lack sufficient data for training bbox-guided generation.

  • Method

    BGTR predicts cell bboxes before decoding HTML, and progressive training uses SNSTab for foundation training before adapting to specific natural-scene datasets.

  • Results

    State-of-the-art performance is achieved on five public benchmark datasets.

  • Takeaways & Limitations

    Explicit bbox guidance improves HTML generation in complex scenes, while progressive training alleviates inadequate training caused by limited natural-scene data.

  • Takeaways & Limitations

    Graph-based TSR methods can remain dependent on additional bbox data or OCR accuracy and require complex post-processing.

Abstract

from arXiv · show

Table Structure Recognition (TSR) aims to extract the bounding boxes of cells and table structure (e.g., HTML) from table images. Although current approaches have made significant progress, the latest image-to-sequence methods overlook the explicit utilization of the bounding box information when predicting HTML sequences, leading to error predictions in complex scenes. In this paper, we introduce a novel framework BGTR (Bounding Box-Guided Table Recognizer). To more effectively utilize bounding box information, we first predict the bounding boxes of cells and then use this information to guide the generation of HTML sequences. While utilizing bounding box information can enhance the accuracy of HTML sequences, for natural scene tables, the data volume is too small to allow for sufficient training of bbox-guided HTML generation. In response, we adopt a progressive training method for natural scene tables and introduce SNSTab, a synthetically generated natural scene table dataset. Our experiments on five benchmark datasets demonstrate SOTA performance.

1 Introduction

BGTR explicitly uses predicted cell bounding boxes to guide HTML generation, targeting errors in complex table scenes. Progressive training with SNSTab addresses limited natural-scene data, and experiments report state-of-the-art performance on five benchmarks.

  • Existing image-to-sequence TSR methods predict HTML sequences while overlooking explicit bbox use, which may cause errors in complex scenes such as spanning cells.
  • BGTR first predicts cell bboxes, then uses image and bbox information in a Bbox-Guided Structure Decoder to generate HTML sequences.
  • Natural-scene tables have limited data and complex structures, so the paper introduces progressive training and SNSTab for foundation training.
  • SNSTab contains 500k synthetic natural-scene table images spanning varied structures and backgrounds for foundation training.
  • Extensive experiments on five public benchmarks demonstrate state-of-the-art performance for BGTR and the progressive training method.

2 Related Work

Prior TSR methods include graph-based, split-and-merge, and image-to-sequence approaches. Natural-scene datasets remain much smaller and harder to annotate than digital-document datasets, motivating synthetic data construction.

  • TSR methods are categorized into graph-based, split-and-merge, and image-to-sequence approaches.
  • Graph-based Methods: Graph-based methods infer row and column relationships from cells or text boxes using graph networks, but may depend on bbox data or OCR accuracy and require complex post-processing.
  • Split-and-merge Methods: Split-and-merge methods detect row and column regions, form grid cells, and determine which adjacent cells should be merged.
  • Image-to-sequence Methods: Image-to-sequence methods convert table images into structured representations such as HTML or LaTeX through encoder-decoder architectures.
  • Existing Datasets: Natural-scene tables are harder to annotate because they cannot be automatically extracted from PDFs and often appear inclined, rotated, or curved.

3 SNSTab

SNSTab is a large synthetic natural-scene table dataset built through table generation, geometric transformation, and background synthesis. It provides annotations and varied scenes for training table-structure recognition.

  • SNSTab contains 500k synthetic natural-scene table images with cell coordinates, cell text, and HTML structure annotations.
  • Table generation: Table generation creates random grids, merges adjacent cells into spanning cells, generates text, and renders the resulting HTML as table images.
  • Table transformation: Table transformation applies thin plate spline transformations to simulate inclined or rotated natural-scene tables and updates cell coordinates accordingly.
  • Background synthesis: Background synthesis selects natural-scene backgrounds and merges table images with randomly cropped regions of matching size.

4 Method

BGTR predicts cell bounding boxes and uses them to guide HTML sequence generation. Its progressive training combines synthetic natural-scene data with dataset-specific training to address limited data for bbox-guided generation.

  • 4.1 Preliminary: BGTR represents table structure as HTML tokens and cell bounding boxes, with a one-to-one correspondence between tokens and boxes.Each cell box is represented by top-left and bottom-right coordinates; box values are non-zero for cell tokens.
  • 4.2 Overall Architecture: The framework extracts image features, uses a shared decoder, predicts cell boxes with a bbox decoder, and generates HTML with a bbox-guided structure decoder.The shared decoder reduces the image-to-sequence gap before its features feed the bbox and structure-decoding branches.
  • 4.3 Bbox-Guided Structure Decoder: The structure decoder integrates bbox features through masked cross-attention, using shared features as queries and bbox features as keys and values.This lets HTML prediction use current and previous bbox outputs while modeling cell positions and relative relationships.
  • 4.4 Progressive Training Method: Progressive training first trains on SNSTab to learn common natural-scene table knowledge, then advances on a specific natural-scene dataset.The shared and structure decoders are initialized from foundation training, while the bbox decoder is trained from scratch in the advancement stage.
  • 4.4 Progressive Training Method: Progressive training significantly alleviates inadequate training of bbox-guided HTML generation caused by insufficient natural-scene data.The foundation stage also supports learning bbox-guided generation for spanning cells and deformed tables.
  • 4.5 Loss Functions: The end-to-end objective combines bbox regression with HTML-token prediction using L = λLhtml + Lbbox.L_bbox uses L1 loss, L_html uses cross-entropy, and λ is a hyperparameter.

5 Experiments

BGTR is evaluated on five public benchmarks spanning natural-scene and digital-document tables, using TEDS-based structure metrics and cell-bounding-box detection. It achieves strong results across datasets, with bbox guidance and progressive training particularly benefiting complex tables.

  • Datasets and Evaluation Metric: The evaluation covers TabRecSet, iFLYTAB, PubTabNet, FinTabNet, and SynthTabNet, encompassing natural-scene and digital-document tables.TabRecSet is split into 80% training and 20% testing because it lacks a predefined split.
  • Datasets and Evaluation Metric: TEDS evaluates table similarity as tree structures, while TEDS-S measures structure without table content to reduce OCR effects.
  • Results on Natural Scene Tables: 98.65% TEDS-S on simple and 92.47% on complex TabRecSet tables exceed TableMaster by 1.45% and 8.36%, respectively, with a 3.07% overall improvement.
  • Results on Natural Scene Tables: 92.00% TEDS-S on iFLYTAB is comparable to SEMv2 and exceeds the other evaluated methods.
  • Results on Digital Document Tables: 97.63% TEDS-S and 96.57% TEDS on PubTabNet outperform other methods, while FinTabNet and SynthTabNet reach 98.89% and 99.11% TEDS-S.
  • Cell Bounding-Box Detection: 5.1% higher PASCAL VOC mAP than TableFormer is achieved for PubTabNet cell-bounding-box detection without post-processing.
  • Ablation Studies: Bbox-guided HTML generation improves complex-table TEDS-S by 2.77%, while progressive training improves complex-table generalization and spanning-cell and deformed-table results.
  • Ablation Studies: Foundation-stage training helps the Shared Decoder and Bbox-Guided Structure Decoder in advancement training, whereas the Bbox Decoder contributes little because of stage data differences.

6 Conclusion

The conclusion presents BGTR as a bounding-box-guided HTML recognizer and pairs it with progressive training and SNSTab to address insufficient natural-scene data. Experiments on five benchmarks report state-of-the-art performance.

  • 6 Conclusion: BGTR explicitly uses cell bounding-box information to guide HTML sequence generation.
  • 6 Conclusion: Progressive training and the large synthetic SNSTab dataset address insufficient data for training bbox-guided HTML generation in natural scenes.
  • 6 Conclusion: Experiments on five benchmark datasets demonstrate state-of-the-art performance for the proposed method.

A.1 Dataset creation

SNSTab samples are created through table generation, geometric transformation, and background synthesis. The process introduces varied table structures and natural-scene appearance while retaining controlled generation settings.

  • A.1 Dataset creation: SNSTab creation comprises table generation, table transformation, and background synthesis.
  • Table transformation: Source points are randomly moved by at most 10% of image width horizontally and height vertically, while 20% of tables remain untransformed.
  • Background synthesis: Background synthesis replaces a selected background mask region with the table image, excluding backgrounds containing text to reduce interference with table text.

A.2 Samples

SNSTab includes diverse table forms designed to represent varied natural-scene appearances and structures.

  • A.2 Samples: SNSTab includes wired, wireless, provincial-line, inclined, curved, and large tables.

A.3 Statistics

SNSTab is characterized through representative samples and statistics covering table appearance, structure, and cell-content complexity.

  • Table statistics: Cell number measures the number of cells contained in each table.This statistic is shown in Fig. 10.
  • Table statistics: Row and column numbers measure the rows and columns contained in each table.These statistics are shown in Figs. 11 and 12.
  • Cell statistics: Length of cell content measures the text length of each cell.This statistic is shown in Fig. 13.
  • Cell statistics: Rowspan and colspan numbers measure how many rows or columns each cell spans.These statistics are shown in Figs. 14 and 15.
  • Dataset samples: SNSTab samples include wired, wireless, provincial line, inclined, curved, and large tables.The background images include paper, walls, and daily-life items.
Loading 2609.08705v1…