PaperlayerView exact revision on arXiv

Source-linked AI summary

Image-based table recognition: data, model, and evaluation

Xu Zhong, Elaheh ShafieiBavani, Antonio Jimeno Yepes

arXiv:1911.10683v5cs.CV

TL;DR

Unstructured PDF and image tables are difficult to convert into machine-readable structure and cell content because their layouts and styles vary. The paper introduces PubTabNet, an attention-based encoder-dual-decoder model, and TEDS for image-based table recognition, with EDD outperforming WYGIWYS by 9.7% absolute TEDS score.

  • Problem

    Image-based table recognition must convert visually diverse PDF and image tables into machine-readable structure and cell content.

  • Method

    The paper builds PubTabNet, trains an attention-based encoder-dual-decoder model with separate structure and cell decoders, and proposes the TEDS metric.

  • Results

    EDD outperformed WYGIWYS by 9.7% absolute TEDS score when trained on PubTabNet.

  • Takeaways & Limitations

    PubTabNet supports training and evaluation for image-based table recognition, while TEDS captures structural and cell-content errors more appropriately than adjacency scoring.

Abstract

from arXiv · show

Important information that relates to a specific topic in a document is often organized in tabular format to assist readers with information retrieval and comparison, which may be difficult to provide in natural language. However, tabular data in unstructured digital documents, e.g., Portable Document Format (PDF) and images, are difficult to parse into structured machine-readable format, due to complexity and diversity in their structure and style. To facilitate image-based table recognition with deep learning, we develop the largest publicly available table recognition dataset PubTabNet (https://github.com/ibm-aur-nlp/PubTabNet), containing 568k table images with corresponding structured HTML representation. PubTabNet is automatically generated by matching the XML and PDF representations of the scientific articles in PubMed Central Open Access Subset (PMCOA). We also propose a novel attention-based encoder-dual-decoder (EDD) architecture that converts images of tables into HTML code. The model has a structure decoder which reconstructs the table structure and helps the cell decoder to recognize cell content. In addition, we propose a new Tree-Edit-Distance-based Similarity (TEDS) metric for table recognition, which more appropriately captures multi-hop cell misalignment and OCR errors than the pre-established metric. The experiments demonstrate that the EDD model can accurately recognize complex tables solely relying on the image representation, outperforming the state-of-the-art by 9.7% absolute TEDS score.

I. INTRODUCTION

Image-based table recognition addresses the challenge of converting visually diverse, unstructured tables into machine-readable structure and cell content. The paper contributes PubTabNet, an attention-based EDD model, and a tree-edit-distance-based evaluation metric.

  • Tables support compact information retrieval and comparison, but much tabular information may be absent from accompanying narrative text.In biomedical articles, only 2% to 8% of information was available in narrative text compared with tables or tabular files.
  • Machine table understanding is difficult because PDF and image tables vary substantially in layout and style.Table recognition encodes both table structure and cell content into a predefined machine-readable format.
  • The paper targets image-based table recognition, reconstructing structured representations solely from table images.
  • PubTabNet contains over 568k heterogeneous table images automatically paired with HTML annotations for structure and cell text.The images come from scientific articles in PMCOA, matched between PDF metadata and XML representations.
  • The attention-based EDD architecture uses an encoder, structure decoder, and cell decoder, with structure reconstruction helping cell-content recognition.The model is trained on PubTabNet and reports superior performance to existing table-recognition methods.
  • The proposed tree-edit-distance-based metric models tables as trees to evaluate image-based table recognition.The paper reports that it is superior to the metric commonly used in the literature and competitions.

A. Data

The paper distinguishes table detection, structure recognition, and full table recognition, then positions PubTabNet and EDD within image-based table recognition. PubTabNet emphasizes diverse publisher-generated tables, header labels, and HTML targets compatible with tree-based evaluation.

  • A. Data: Table analysis separates table detection, structure recognition, and table recognition, which also parses cell content.
  • A. Data: PubTabNet covers image-based table recognition, unlike datasets that target only detection or structural parsing.
  • A. Data: PubTabNet draws tables typeset across over 6,000 PMCOA journals, providing greater table-style diversity than other datasets.
  • A. Data: The dataset categorizes cells as headers or body cells and uses a target format that enables the proposed tree-edit-distance metric.
  • B. Model: EDD improves attention-based encoder-decoder recognition by coupling the cell decoder to structure decoding when new cells are generated.The structure decoder’s hidden state helps the cell decoder attend to the corresponding cell in the image.

C. Evaluation

The evaluation section motivates tree-based similarity by identifying weaknesses in adjacency-relation scoring and describes dataset curation and balanced testing for complex table structures.

  • C. Evaluation: The conventional adjacency-relation metric misses errors involving empty cells or nonlocal cell misalignment and lacks fine-grained content scoring.
  • C. Evaluation: TEDS addresses these limitations by modeling tables as trees and using string-edit distance for node substitution.
  • C. Evaluation: PubTabNet is generated by matching XML and PDF table representations from PMCOA, with automated bounding-box verification using text similarity thresholds.Bounding boxes are accepted when TF-IDF cosine similarity exceeds 90% and text lengths differ by less than 10%.
  • C. Evaluation: HTML annotations are curated by removing unreconstructable variations, standardizing header cells, and retaining only rowspan and colspan attributes.
  • C. Evaluation: The training set contains 548,592 samples, while balanced development and test sets each sample 5,000 tables with spanning cells and 5,000 without.

IV. ENCODER-DUAL-DECODER (EDD) MODEL

The EDD model separates table structure and cell-content recognition while linking the decoders, and TEDS evaluates both structural and content errors more appropriately than adjacency relations.

  • EDD architecture: The EDD architecture uses an encoder, structure decoder, and cell decoder to convert table images into HTML.The structure decoder generates HTML structure, while the cell decoder recognizes content for each generated cell.
  • EDD architecture: The cell decoder is triggered by each new cell and receives the structure decoder’s hidden state to focus on the corresponding image region.This creates a one-to-one correspondence between generated structure cells and cell-content sequences.
  • TEDS metric: TEDS compares HTML table trees using edit operations on structure, spans, and cell content, then averages scores across test samples.Cell-content substitutions use normalized Levenshtein similarity when row and column spans match.
  • TEDS evaluation: At 90% cell-shift perturbation, adjacency F1 remains nearly 80% while TEDS drops by 60%, capturing errors the adjacency metric misses.The perturbation shifts cells and pads leftover space with empty cells, exposing empty-cell and multi-hop misalignment weaknesses.
  • TEDS evaluation: For cell-content perturbations, adjacency F1 falls over 70% at only 10% corruption, whereas TEDS decreases from 90% to 40% across 10%–90% corruption.This indicates that TEDS captures fine-grained content-recognition errors more proportionately.

VI. EXPERIMENTS

The experiments compare EDD with five off-the-shelf tools and the WYGIWYS model using TEDS on outputs converted into a common tree representation.

  • Test comparisons: EDD is compared with Tabula, Traprange, Camelot, PDFPlumber, Adobe Acrobat Pro, and WYGIWYS.The off-the-shelf tools receive cropped PDF tables or high-resolution images according to their input capabilities.
  • Evaluation protocol: Outputs from the comparison methods are parsed into the same tree structure as the HTML tables to compute TEDS scores.This provides a common representation for evaluating structurally different outputs.

A. Implementation details

Implementation experiments train EDD on a memory-constrained subset, test encoder feature resolutions and decoder configurations, and use the best setting for test comparisons.

  • Training setup: To avoid GPU RAM limits, EDD is trained on 399k samples from the PubTabNet training set.Inference uses beam search with beam=3 for both decoders.
  • Preprocessing: Training images are resized to 448 × 448 pixels, and each channel is normalized by z-score.The training vocabulary contains 32 structural tokens and 281 cell tokens.
  • Encoder settings: Five ResNet-18 encoder settings vary final-layer stride and whether structure and cell decoders use independent layers.The settings include EDD-S2, EDD-S1, EDD-S2S2, EDD-S2S1, and EDD-S1S1.
  • Encoder settings: Higher-resolution feature maps and independent CNN layers improve validation performance, making EDD-S1S1 the selected test configuration.EDD-S1S1 uses independent stride-1 final CNN layers for both decoders.
  • Decoder settings: The structure and cell decoders are single-layer LSTMs with hidden-state sizes of 256 and 512, respectively, using soft attention.The attention mechanism includes a hidden layer of size 256.
  • Training setup: Training uses two stages: structural-token pretraining followed by joint generation of structural and cell tokens.The loss weighting changes from λ = 1 during pretraining to λ = 0.5 during joint training.

B. Quantitative analysis

EDD substantially outperforms seven baselines on image-only recognition of both simple and complex tables. Its advantage over WYGIWYS is 9.7% absolute TEDS overall, increasing to 9.9% on complex tables.

  • EDD substantially outperforms all seven baselines on both simple and complex tables using only table images.The comparison includes methods that directly use text extracted from PDF to fill cells.
  • 9.7% absolute TEDS is EDD’s advantage over WYGIWYS after both models are trained on PubTabNet.
  • 9.9% absolute TEDS is EDD’s advantage on complex tables, compared with 9.5% on simple tables.

C. Qualitative analysis

Qualitative examples show EDD handling complex headers and cell content more accurately than the baselines. Its decoders also exhibit complementary attention patterns for structure and cell recognition.

  • EDD perfectly recognizes the complex structure and cell content of an example table with multi-row and multi-column header spans.The baselines struggle particularly with the complex table header.
  • EDD makes no structure-recognition errors and only one OCR error, recognizing “PF” as “PC.”WYGIWYS misses the second header row and makes several cell-content errors.
  • The structure decoder attends around rows and cells, while the cell decoder focuses more narrowly on the content being generated.For closing-cell tags, the structure decoder can rely on its language model rather than image features.

D. Error analysis

EDD outperforms WYGIWYS across all tested table-size groups, while both models’ performance decreases as table size increases.

  • EDD outperforms WYGIWYS in every group defined by width, height, structural-token count, or longest-cell token count.The test set is divided into 15 equal-interval groups for each property.
  • Performance decreases for both EDD and WYGIWYS as table size increases.

E. Generalization

EDD generalizes beyond PubTabNet to a synthetic table-recognition dataset and maintains very high performance across table-style categories. It also outperforms TIES on exact match, including more complex categories.

  • The synthetic dataset contains 500K table images with corresponding HTML representations.It was created to test whether EDD is suitable beyond PubTabNet.
  • 99.7+% TEDS is achieved by EDD on every category of the synthetic dataset.This reflects near-perfect reconstruction of both table structure and cell content from images.
  • EDD outperforms TIES in exact match across all synthetic table categories.TEDS is not computed for TIES because TIES does not generate an HTML representation.
  • EDD shows no significant performance downgrade on synthetic categories 3 or 4 with more complex structure.The authors describe EDD as more robust and generalizable than TIES on difficult examples.

VII. CONCLUSION

The paper studies image-based table recognition through the PubTabNet dataset, the attention-based EDD model, and the TEDS evaluation metric. It reports effective recognition of complex tables while identifying missing cell coordinates and dependence on prelocalized tables as future-work boundaries.

  • VII. CONCLUSION: PubTabNet provides a large-scale resource for training and evaluating deep learning models on image-based table recognition.The dataset contains over 568k heterogeneous table images with HTML annotations for table structure and cell text.
  • VII. CONCLUSION: The attention-based EDD model separates table structure recognition from cell content recognition.Its structure decoder also helps the cell decoder attend to the correct cell content.
  • VII. CONCLUSION: TEDS evaluates both table structure and cell content recognition while capturing multi-hop cell misalignment and OCR errors.The paper presents TEDS as more appropriate than the traditional adjacency-relation metric for these errors.
  • VII. CONCLUSION: The EDD model trained on PubTabNet effectively recognizes complex table structures and extracts cell content from images.The dataset and model are presented as resources for advancing table recognition and pre-training.
  • VII. CONCLUSION: Current PubTabNet lacks table-cell coordinates, and the EDD model assumes that table locations are already provided.Future work proposes adding cell-location prediction and integrating EDD with table-detection networks for end-to-end recognition.
Loading 1911.10683v5…