PaperlayerView exact revision on arXiv

Source-linked AI summary

TabPedia: Towards Comprehensive Visual Table Understanding with Concept Synergy

Weichao Zhao, Hao Feng, Qi Liu, Jingqun Tang, Shu Wei, Binghong Wu, Lei Liao, Yongjie Ye, Hao Liu, Wengang Zhou, Houqiang Li, Can Huang

arXiv:2406.01326v2cs.CV

TL;DR

Visual table understanding must handle both table structure and content, while prior approaches generally use task-specific architectures and objectives. TabPedia unifies four table tasks with meditative tokens and multi-source visual embeddings; experiments report performance across perception and comprehension benchmarks, including a strong result on ComTQA.

  • Problem

    Prior visual table understanding methods generally use task-specific architectures, while table parsing must reconcile two-dimensional structure with one-dimensional sequences and image-resolution demands.

    Source: §1 · p. 2

  • Method

    TabPedia uses dual vision encoders and meditative tokens to integrate table tasks and multi-source visual embeddings in an LLM-based framework.

    Source: §1 · p. 2

  • Results

    TabPedia consistently surpasses OmniParser with 4.96% and 3.56% S-TEDS on both datasets, respectively, and achieves the optimal result with a notable margin on ComTQA.

    Source: §5.2 · p. 8

  • Takeaways & Limitations

    The experiments support the feasibility of using LLMs for visual table understanding when tasks and visual tokens work in synergy.

    Source: §7 · p. 10

    Because TQA datasets are dominated by tables, TabPedia still lacks the capability to answer table questions directly from original document images.

    Source: §6 · p. 10

Abstract

from arXiv · show

Tables contain factual and quantitative data accompanied by various structures and contents that pose challenges for machine comprehension. Previous methods generally design task-specific architectures and objectives for individual tasks, resulting in modal isolation and intricate workflows. In this paper, we present a novel large vision-language model, TabPedia, equipped with a concept synergy mechanism. In this mechanism, all the involved diverse visual table understanding (VTU) tasks and multi-source visual embeddings are abstracted as concepts. This unified framework allows TabPedia to seamlessly integrate VTU tasks, such as table detection, table structure recognition, table querying, and table question answering, by leveraging the capabilities of large language models (LLMs). Moreover, the concept synergy mechanism enables table perception-related and comprehension-related tasks to work in harmony, as they can effectively leverage the needed clues from the corresponding source perception embeddings. Furthermore, to better evaluate the VTU task in real-world scenarios, we establish a new and comprehensive table VQA benchmark, ComTQA, featuring approximately 9,000 QA pairs. Extensive quantitative and qualitative experiments on both table perception and comprehension tasks, conducted across various public benchmarks, validate the effectiveness of our TabPedia. The superior performance further confirms the feasibility of using LLMs for understanding visual tables when all concepts work in synergy. The benchmark ComTQA has been open-sourced at https://huggingface.co/datasets/ByteDance/ComTQA. The source code and model also have been released athttps://github.com/zhaowc-ustc/TabPedia.

1 Introduction

Visual table understanding spans perception and comprehension tasks, but their varied visual-semantic requirements make unified modeling difficult. TabPedia integrates these tasks through concept synergy and evaluates performance across benchmarks, alongside ComTQA for more challenging table question answering.

1 Introduction

  • The central challenge is reconciling tables’ two-dimensional structure with one-dimensional output sequences while handling the image resolutions required for parsing.Markup representations can omit cell coordinates, produce lengthy output, and risk illegal grammars.Source: §1 · p. 2
  • TabPedia uses ViT-L and Swin-B to encode global low-resolution and fine-grained high-resolution information, respectively.The resulting multi-source visual embeddings are combined with task instructions through the concept synergy mechanism.Source: §1 · p. 2
  • Quantitative and qualitative experiments across public benchmarks validate TabPedia on table perception and comprehension tasks.Source: §1 · pp. 2–3
  • TabPedia unifies table detection, structure recognition, querying, and question answering in one large vision-language model.The framework abstracts tasks and multi-source visual embeddings as concepts, using meditative tokens to support their synergy.Source: §1 · p. 2
  • ComTQA provides approximately 9,000 question-answer pairs for more challenging table VQA scenarios, including multiple answers, calculations, and logical reasoning.The benchmark contains about 1,500 images.Source: §1 · pp. 2–3

2 Related Work

Prior work addresses table perception and comprehension largely through distinct approaches, while existing large vision-language models still face challenges in two-dimensional table parsing. TabPedia’s meditative tokens are intended to support visual-table perception and comprehension within a unified framework.

2.1 Table Recognition

  • Table detection and structure recognition have used detection, segmentation, relationship modeling, and markup-generation approaches, but intricate table-content understanding remains limited.The paper’s approach jointly exploits perception and comprehension tasks in a unified framework.Source: §2.1 · p. 3

2.2 Large Vision-Language Models

  • Existing large vision-language models still face challenges in two-dimensional table parsing and understanding, motivating a unified framework for table perception and comprehension.Source: §2.2 · p. 3

2.3 Additional Tokens

  • TabPedia introduces meditative tokens to enhance its perceptive and comprehensive capabilities for visual tables.Figure 2 depicts image features from two encoders combined with instruction-derived tokens before entering the LLM.Source: §2.3 · p. 4, Figure 2 · p. 4

3 Method

TabPedia uses a two-phase training pipeline: visual-feature pre-training followed by table-aware instruction fine-tuning. Its architecture combines visual features at different resolutions with task instructions and meditative tokens for LLM response generation.

3.1 Model Architecture

  • The architecture pairs a high-resolution Swin-B encoder with a low-resolution ViT-L encoder to capture fine-grained visual information and overall layout, respectively.The high-resolution input is resized while maintaining aspect ratio; the low-resolution ViT-L encoder remains frozen during training.Source: §3.1 · p. 4
  • Meditative tokens adaptively enable different visual-token regions in response to task instructions, with the LLM generating the response from the combined sequence.The input sequence concatenates instructions, projected low- and high-resolution visual tokens, and meditative tokens.Source: §3.1 · pp. 4–5

3.2 Pre-training

  • Pre-training aligns visual features to the LLM using text detection, recognition, spotting, long-text reading, and image captioning tasks.The high-resolution encoder and both projectors are optimized while the LLM and low-resolution encoder are frozen.Source: §3.2 · p. 5

3.3 Table-aware Fine-tuning

  • Table-aware fine-tuning jointly trains table detection, structure recognition, querying, and question answering to cultivate perception and comprehension capabilities.The full framework except the low-resolution vision encoder is fine-tuned for instruction following.Source: §3.3 · p. 5

4 Dataset Construction

The paper constructs training data for four visual table understanding tasks from public datasets and synthesizes data for table querying and question answering. It also introduces ComTQA, an expert-annotated benchmark of real-world table images with more challenging questions.

Dataset sources and instructions

  • The instruction-following dataset draws on PubTab1M, FinTabNet, PubTabNet, WikiTableQuestions, and TabFact, with multiple task instructions generated using GPT3.5.PubTab1M-Structure annotations are transferred to original images to create the synthesized PubTab1M-Syn subset for table querying.Source: §4 · p. 5

Table Detection

  • Table detection training uses PubTab1M-Det images collected from PDF documents with tables at different scales and rotations.The task generates table locations as bounding-box coordinates.Source: §4 · pp. 5–6

Table Structure Recognition

  • Table structure recognition uses PubTab1M-Str, FinTabNet, and PubTabNet, converting their HTML annotations into a canonical format of labeled object boxes.The representation models columns, rows, headers, projected row headers, and spanning cells.Source: §4 · p. 6

Table Querying

  • Table querying parses a table from its original document image at a given location, a more challenging setting than parsing cropped table images.PubTab1M-Syn is synthesized from PubTab1M annotations for this task.Source: §4 · p. 6

Table Question Answering

  • For TQA training, the authors augment WTQ and TabFact with question-answer data generated from partial FinTabNet and PubTab1M images using Gemini Pro.The cited public datasets use Wikipedia table images, which the passage describes as limiting generalization to real-world tables.Source: §4 · p. 6

ComTQA

  • ComTQA contains about 9,000 expert-annotated question-answer pairs from about 1,500 real-world table images, with multiple answers, calculations, and logical reasoning.The benchmark is built from test sets of FinTabNet and PubTab1M.Source: §4 · p. 6

5 Experiment

TabPedia is evaluated across table detection, structure recognition, querying, and question answering, alongside qualitative tests and component ablations. Results report performance across these tasks and examine the roles of meditative tokens, dual vision encoders, and freezing the low-resolution encoder.

5.1 Implementation Details

  • Evaluation covers public benchmarks for each task, with accuracy for TQA, detection metrics for TD, and S-TEDS and GriTS metrics for structure tasks.The reported test sets include 9,070 ComTQA QA pairs; training uses a cosine one-cycle schedule and AdamW.Source: §5.1 · pp. 6–7

5.2 Quantitative Results

  • TabPedia outperforms Faster R-CNN by a notable margin and performs competitively with DETR on table detection; combining TD and TQ supports parsing all tables in a document.Directly generated table locations require no NMS.Source: §5.2 · pp. 7–8
  • 4.96% and 3.56% S-TEDS gains over OmniParser on PubTabNet and FinTabNet accompany TabPedia’s optimal result on ComTQA.It also achieves promising performance on WTQ and TabFact.Source: §5.2 · p. 8
  • For table querying, TabPedia reaches 96.04% GriTSTop and 95.07% S-TEDS on raw images, nearly matching results from cropped images despite surrounding document content.The authors also report a combined TD+TQ multi-round dialogue result.Source: §5.2 · p. 8
  • On WTQ and TabFact, TabPedia performs promisingly among open- and closed-source LVLMs, and it achieves the optimal result on ComTQA.The authors attribute the ComTQA result to jointly learning perception and comprehension tasks.Source: §5.2 · p. 8

5.3 Qualitative Results

  • Qualitative examples show formatted TD and TSR outputs, document-wide parsing through multi-round TD+TQ instructions, and table answers supported by cited contents and calculations.Figure 3 presents perception examples in its first row, multi-instruction examples in the second, and table-content answers in the bottom row.Source: §5.3 · p. 8, Figure 3 · p. 9

5.4 Ablation Studies

  • Meditative tokens improve perception and comprehension, contribute the most information to generated answers, and attend more to high-resolution tokens for TSR and TQ.For TD and TQA, attention to high- and low-resolution visual tokens is nearly equal.Source: §5.4 · pp. 8–9
  • Dual encoders combine high-resolution detail with low-resolution layout information, while unfreezing the low-resolution encoder brings no significant performance improvement and lengthens training.The study uses three datasets across three tasks for its ablations.Source: §5.4 · pp. 8, 10

6 Limitation

TabPedia has limitations in parsing distorted tables, answering from original document images, and inference speed.

  • Rectangular-box table representations do not accurately parse structural information in twisted or distorted tables.Source: §6 · p. 10
  • Because TQA dataset images are dominated by tables, TabPedia cannot yet directly answer questions from original document images.Source: §6 · p. 10
  • TabPedia decodes more slowly than parallel algorithms such as DETR and Faster R-CNN, though KV cache, flash attention, and hardware improvements can improve efficiency.Source: §6 · p. 10

7 Conclusion

TabPedia unifies diverse visual table understanding tasks through concept synergy, using an LLM to interpret tasks and visual tokens. Experiments validate the model across public benchmarks, and the paper introduces ComTQA with about 9,000 QA pairs.

  • TabPedia unifies diverse visual table understanding tasks in a large vision-language model.Source: §7 · p. 10
  • Its concept synergy mechanism represents diverse tasks and dual-encoder visual tokens as concepts, integrating them into the LLM with meditative tokens.Source: §7 · p. 10
  • Quantitative and qualitative experiments across public benchmarks validate TabPedia, and ComTQA adds approximately 9,000 table-VQA pairs.Source: §7 · p. 10

A More details about TQA datasets

ComTQA QA pairs are generated from table images using Gemini Pro and filtered into an agreed format. Prompting techniques are used to improve answer reliability.

  • Gemini Pro first recognizes table structure using image OCR results, then generates question-answer pairs from those results.Source: §A.1 · p. 17, Figure A1 · p. 17
  • Chain-of-Thought and few-shot prompting are used to improve the reliability of generated answers.Source: §A.1 · p. 17
  • A regularized matching algorithm and special-character filter extract available pairs from Gemini Pro's raw responses.Source: §A.1 · p. 17

A.2 ComTQA Benchmark

ComTQA contains 9,070 QA pairs across 1,591 images, including complex question types such as multiple answers, calculation, and logical inference. It is intended to address question types absent from previous benchmarks.

  • 9,070 QA pairs span 1,591 images in ComTQA, averaging 5 questions per image.Source: §A.2 · p. 17, Table A1 · p. 17
  • ComTQA includes multiple-answer, mathematical-calculation, and logical-inference questions that the authors say are lacking in previous benchmarks.Source: §A.2 · p. 17, Figure A2 · p. 17
  • ComTQA uses real-world table images with more complex questions than existing TQA benchmarks to assess model robustness.Source: §A.2 · p. 17

B Annotation in TSR task

The TSR annotation scheme uses overlapping boxes for five table elements whose relationships form a hierarchy. The broader-impact discussion notes potential accessibility and labor benefits alongside privacy risks.

B Annotation in TSR task

  • Five TSR elements—columns, rows, spanning cells, column headers, and projected row headers—form a hierarchical structure through physically overlapping boxes.Figure B3 illustrates dilated bounding-box annotations for the object classes.Source: B Annotation in TSR task · p. 18, Figure B3 · p. 18

C Broader Impact

  • Visual table understanding could help people with visual impairments access tabular data and reduce manual data-entry and correction work.The discussion also flags heightened risk of sensitive information being mishandled or exposed, making privacy measures important.Source: C Broader Impact · p. 18

D More Qualitative Results

Qualitative examples show TabPedia producing reasonable outputs on selected in-the-wild images, while visualizations examine response changes with meditative tokens and their attention patterns across tasks.

Results on in-the-wild cases

  • TabPedia generates accurate and reasonable responses for TD, TSR, and TQA on selected in-the-wild document images.The paper presents these examples as evidence of robustness for visual table understanding.Source: D More Qualitative Results · p. 18, Figure D5 · p. 21

Attention map of meditative tokens

  • Attention maps show dense shallow-layer attention and diverse deeper-layer regions for perception, but sparse shallow-layer attention for TQA.The authors interpret these patterns as meditative tokens adaptively capturing task-related information and attending to regions relevant to task questions.Source: D More Qualitative Results · p. 19, Figure D4 · p. 19

Visualization of TabPedia’s responses

  • Meditative tokens mainly improve long-form responses, reducing meaningless or repetitive TD and TSR text and producing more elaborated, reasonable TQA responses.Table D2 presents qualitative response comparisons before and after introducing meditative tokens.Source: D More Qualitative Results · p. 20, Table D2 · p. 20
  • One illustrated output lists table-element locations as bounding-box coordinates.The displayed sample includes several coordinate ranges.Source: D More Qualitative Results · p. 20
Loading 2406.01326v2…