PaperlayerView exact revision on arXiv

Source-linked AI summary

Visual-TableQA: Open-Domain Benchmark for Reasoning over Table Images

Boammani Aser Lompo, Marc Haraoui

arXiv:2509.07966v2cs.CVcs.CL

TL;DR

Existing benchmarks offer limited scale, diversity, or reasoning depth for visual reasoning over rendered tables. Visual-TableQA builds an open-domain dataset through collaborative LLM generation and validation; fine-tuning on it improves generalization across external benchmarks and narrows the gap between open and proprietary models.

  • Problem

    Benchmarks remain limited in scale, diversity, or reasoning depth for visual reasoning over tables, while complex table images demand interpretation of visual structure.

    Source: §1 · pp. 1–2

  • Method

    Visual-TableQA uses a modular, low-cost pipeline to generate LaTeX-rendered tables and reasoning-intensive QA pairs through cross-model inspiration and LLM-jury validation.

    Source: §1 · p. 2, §4.3 · p. 9

  • Results

    Fine-tuning on Visual-TableQA generalizes to external benchmarks and narrows the gap between proprietary and open-source models, with Qwen2.5-VL-7B-Instruct surpassing several proprietary models.

    Source: §3.2 · pp. 7–8

  • Takeaways & Limitations

    Visual-TableQA serves as an effective training signal for improving reasoning performance on complex table images.

    Source: §5 · p. 9

    After fine-tuning, incoherence remains the dominant error class, motivating future work on targeted synthetic supervision for specific error types.

    Source: §4.2 · p. 9

Abstract

from arXiv · show

Visual reasoning over structured data such as tables is a critical capability for modern vision-language models (VLMs), yet current benchmarks remain limited in scale, diversity, or reasoning depth, especially when it comes to rendered table images. Addressing this gap, we introduce Visual-TableQA, a large-scale, open-domain multimodal dataset specifically designed to evaluate and enhance visual reasoning over complex tabular data. Our generation pipeline is modular, scalable, and fully autonomous, involving multiple reasoning LLMs collaborating across distinct roles: generation, validation, and inspiration. Visual-TableQA comprises 2.5k richly structured LaTeX-rendered tables and 6k reasoning-intensive QA pairs, all produced at a cost of under USD 100. To promote diversity and creativity, our pipeline performs multi-model collaborative data generation via cross-model prompting ('inspiration') and LLM-jury filtering. Stronger models seed layouts and topics that weaker models elaborate, collectively distilling diverse reasoning patterns and visual structures into the dataset. Empirical results show that models fine-tuned on Visual-TableQA generalize robustly to external benchmarks, outperforming several proprietary models despite the dataset's synthetic nature. The full pipeline and resources are publicly available at https://github.com/AI-4-Everyone/Visual-TableQA.

1 Introduction

Complex table images require models to interpret visual structure and reason across cells, while existing table-QA resources provide limited visual-layout diversity and reasoning depth. Visual-TableQA addresses this gap with a synthetic, open-domain benchmark of complex table-based questions.

1 Introduction

  • Complex tables require VLMs to interpret layouts and visual components and reason across cells to answer questions.The cited example combines multirow cells, diagrams, and color encoding.Source: §1 · p. 1
  • Existing table-QA datasets often use text-only tables or offer limited visual diversity and reasoning depth, while open-domain chart datasets overlook tables.The cited examples include text-only, domain-specific, standardized-query, and technical datasets.Source: §1 · pp. 1–2
  • Visual-TableQA introduces a synthetic, multimodal, open-domain dataset with 2.5k tables and 9k QA pairs, generated for under $100.Its tasks emphasize visual layout and multi-step reasoning; a sample of 800 QA pairs was manually verified.Source: §1 · p. 2
  • The paper evaluates a broad range of VLMs and reports that most continue to struggle with table understanding.Its stated contributions also include comparisons with existing datasets and performance gains after fine-tuning.Source: §1 · p. 2

2 Visual-TableQA Dataset

Visual-TableQA uses LaTeX-rendered tables and a modular generation pipeline to produce reasoning-intensive QA pairs, with automated and human quality checks. Its construction combines iterative cross-model inspiration with jury-based evaluation.

2 Visual-TableQA Dataset

  • Visual-TableQA uses LaTeX-rendered table images, automated reasoning-task creation, and LLM-based evaluation to keep annotation costs under $100.The pipeline is described as scalable and modular, with table generation, QA creation, and validation separated.Source: §2 · p. 3

2.1 Modality-Swap: Table Representation in LaTeX

  • The modality-swap strategy represents tables in LaTeX so language models can use textual reasoning to create complex visual tables economically.The paper reports that most state-of-the-art models perform better on the same query in textual form than as a visual input.Source: §2.1 · p. 3
  • The authors' LaTeX-generation observations align with Kale & Nadadur (2025), who report accuracy drops as task complexity increases.Table 3 reports successful LaTeX compilations for models, with at least 500 generated samples per accuracy.Source: §2.1 · p. 3, Table 3 · p. 4

2.2 Data Generation Pipeline

  • The pipeline converts seed table images to LaTeX, then uses topic prompts and generated tables iteratively to create diverse layouts.The initial pool includes 20 table images and 5,000 topic prompts; each generation prompt supplies one table and three topics.Source: §2.2 · pp. 4–5, Figure 2 · p. 5
  • Cross-model prompting expands the inspiration pool with selected generated tables, allowing different models' structural and stylistic tendencies to diversify layouts.The authors call this iterative collaborative prompting “inspiration.”Source: §2.2 · p. 5
  • For each generated table, an LLM creates three questions requiring multi-step reasoning, pattern recognition, or symbolic interpretation.Tables are not fact-checked, so some content may be non-factual; the authors say this can encourage reliance on reasoning over prior knowledge.Source: §2.2 · p. 5

2.3 Quality Assurance

  • An independent LLM jury judges table and QA validity against four criteria, discarding items that fail any criterion and accepting by majority vote.Criteria include table relevance and coherence, question grounding, and answer support from the table.Source: §2.3 · p. 5
  • ROSCOE scores assess rationale coherence, logic, and grounding, while two human annotators rated 92% of 800 QA pairs at least four stars.The human ratings were given by both annotators; ROSCOE results were reported as directionally consistent with expectations.Source: §2.3 · p. 6

3 Experiments

The experiments evaluate Visual-TableQA alongside table- and chart-focused benchmarks, then test its value as a visual-reasoning benchmark and training signal. Results indicate that its image-based format is challenging, its model rankings align most closely with balanced ReachQA, and fine-tuning transfers to other benchmarks.

3.1 Benchmark Comparison

  • The evaluation compares reasoning VLMs across four benchmarks and includes supervised fine-tuning of four models on either ReachQA or Visual-TableQA.Models are tested on image-question pairs; inference uses a 5,000-token limit and majority-vote evaluation by an LLM jury, with evaluations run twice.Source: §3.1 · p. 6, §3.2 · p. 6

3.2 Experimentation Results

  • Models perform +6.26% better on Visual-TableQA-CIT than on image-based Visual-TableQA, indicating greater challenge in the image format.Model performance trends on Visual-TableQA also resemble those on human-annotated ChartQA and MATH-Vision.Source: §3.2 · p. 7
  • Visual-TableQA rankings align most closely with balanced ReachQA, rather than recognition-focused ChartQA or reasoning-focused MATH-Vision.The paper reports that the benchmark rewards models capable of both visual recognition and reasoning.Source: §3.2 · p. 7, Figure 3 · p. 7
  • Fine-tuning Qwen2.5-VL-7B-Instruct on Visual-TableQA raises ReachQA accuracy from 49.23% to 60.95% and MATH-Vision accuracy from 25.10% to 49.77%.The reported gains also generalize across LLaVA-Next-Llama3-8B, MiniCPM-V2.5-Llama3, and InternVL2-8B.Source: §3.2 · p. 7
  • Claude 3.5 Sonnet leads on nearly all benchmarks, while Visual-TableQA fine-tuning narrows the proprietary–open-source gap and lifts Qwen2.5-VL-7B-Instruct above several proprietary models.The named models it surpasses include GPT-4o, GPT-4o-mini, and Gemini 2.5 Pro.Source: §3.2 · p. 8

4 Discussion

The discussion contrasts Visual-TableQA fine-tuning with ReachQA and other datasets, and describes how the generation pipeline supports scalable, collaborative data creation. It highlights reasoning gains, transfer to external benchmarks, and an error pattern that remains a target for future work.

4.1 Visual-TableQA vs ReachQA

  • Visual-TableQA fine-tuning yields +9.35 Recognition and +17.68 Reasoning average gains, versus ReachQA’s +10.25 Recognition and +4.4 Reasoning gains.The comparison uses ReachQA’s equally sized Recognition and Reasoning subsets.Source: §4.1 · p. 8
  • Visual-TableQA’s rationales and more complex, diverse visual structures are identified as factors in its larger ReachQA Reasoning gain despite roughly three times fewer samples.The authors suggest these qualities support more effective distillation for symbolic interpretation and multi-step reasoning.Source: §4.1 · p. 8

4.2 Visual-TableQA’s Advantages Compared to Other Datasets

  • Visual-TableQA offers richer layout diversity, broader topic coverage, systematic visual complexity, and high-quality rationales; fine-tuning is reported to improve external benchmark performance.The discussion cites LLaVA-Next-Llama3-8B as an example and points to Table-VQA and TableVQA-Bench.Source: §4.2 · p. 8
  • For Qwen2.5-VL-7B-Instruct, post-fine-tuning errors slightly increase overall but concentrate in incoherence, while all other analyzed error types decrease.The authors identify targeted synthetic supervision for specific error types as future work.Source: §4.2 · pp. 8–9

4.3 Scalability of the Pipeline and Its Benefits for Knowledge Distillation

  • The pipeline separates table synthesis, QA creation, and validation into reusable, upgradable components and automates jury-based quality control.Stronger models seed layouts and topical cues that weaker models elaborate into diverse tables and QA pairs through cross-model inspiration.Source: §4.3 · p. 9, §4.2 · p. 9

5 Conclusion

The paper concludes that Visual-TableQA is a low-cost, open-domain dataset for evaluating and improving reasoning over complex table images, with reported fine-tuning gains across benchmarks. Its related-work discussion positions the dataset against prior resources that trade off rendered-table focus, visual diversity, reasoning depth, and scalability.

5 Conclusion

  • Visual-TableQA combines a fully automated, modular pipeline, LaTeX-rendered tables, reasoning-intensive QA pairs, and LLM-jury verification at a cost under $100.The conclusion reports fine-tuning gains across table-centric and general-purpose benchmarks, including ReachQA and MATH-Vision.Source: §5 · p. 9

A Extended Related Works

  • MultiModalQA is described as an open-domain synthetic resource combining Wikipedia figures, diagrams, and text, but its QA pairs derive from a finite set of templates.The paper attributes its synthetic characterization to linking independent modalities and generating QA pairs through formalized templates.Source: A Extended Related Works · p. 13
  • Table-VQA is described as fully synthetic but lacking visual diversity and complexity, with similar table formats concentrated in technical domains.The listed domains include statistics, physics, and algorithms.Source: A Extended Related Works · p. 13
  • The related-work review identifies a gap in open-domain evaluation of rendered table images, noting that existing benchmarks fall short in visual diversity, reasoning depth, or scalability.It states that, aside from MultiModalQA, no open-domain dataset was designed to evaluate models on rendered table images.Source: A Extended Related Works · pp. 12–13

B Considerations and Limitations

The paper identifies limitations in both table representation and data-quality assessment, and reports that one model does not benefit consistently across downstream tasks.

B Considerations and Limitations

  • LaTeX’s expressiveness is limited for more complex or visually rich images, while robust bidirectional image-to-text encoding remains open.Automatic metrics such as ROSCOE are less reliable than human judgment, so human annotators remain important when scaling synthetic reasoning datasets.Source: B Considerations and Limitations · p. 13
  • Qwen2.5-VL-7B-Instruct did not consistently benefit from Visual-TableQA supervision across downstream tasks, leaving its generalization pattern for further investigation.Source: B Considerations and Limitations · p. 13

C TableQA Layout and Topic Diversity

The section examines topic and layout diversity, generation settings, and common table-generation anomalies. Its evidence describes a balanced distribution across topic clusters and varied layouts produced from limited starting templates.

C TableQA Layout and Topic Diversity

D Table Generation Settings

  • LLM-1 uses one LaTeX table and three topic inspirations to generate three tables; LLM-2 generates QA pairs from a table.The generation-phase temperature is 0.7; Table 6 reports average QA-pair validity rates by model, based on at least 500 pairs per model.Source: D Table Generation Settings · p. 15, Table 6 · p. 15

E Common Anomalies in Generated Tables

Layout and Formatting Issues

  • Layout and formatting issues include tables exceeding page margins, cell text overflowing boundaries, obscured multirow text, and broken horizontal rules.Source: Layout and Formatting Issues · p. 16

Content Relevance and Correctness

Diagram-Specific Issues

  • Diagram-specific problems include missing tables, misaligned nodes, misplaced arrows, detached titles or legends, and other visual inconsistencies.Source: Diagram-Specific Issues · p. 17

F ROSCOE Metric Scores

The paper reports ROSCOE-based reasoning assessment and describes an LLM-jury protocol for validation and benchmarking. It also finds that jury judgments become more reliable when models are prompted to explain their decisions.

F ROSCOE Metric Scores

  • ROSCOE scores across the dataset assess reasoning coherence, including semantic alignment, logical consistency, and contextual grounding.The authors say the scores further support the reliability and quality of generated tables and QA pairs; Table 7 reports dataset-averaged metrics and their preferred directions.Source: F ROSCOE Metric Scores · p. 17, Table 7 · p. 18

G LLM Jury Reliability

  • A high-performing multi-model jury with majority voting evaluates predictions, and the authors report that aggregating strong LLM judgments improves consistency over a single evaluator.The protocol is used for both generated-data quality assurance and benchmark evaluation; the paper also compares jury judgments with human annotators and reports jury agreement analyses.Source: G LLM Jury Reliability · p. 18, §G.1 · p. 18

G.1 LLM Jury Settings

  • Explicitly requesting a rationale before the verdict improved evaluation reliability and depth; avoiding the word “JSON” mitigated Mistral-large’s omission of reasoning.GPT-4.1 remained robust when JSON was mentioned; jury calls used temperatures between 0.0 and 0.1.Source: §G.1 · pp. 18–19

G.2 LLM Jury Limitations and Mitigation Strategies

The section reports jury-to-human alignment, characterizes agreement across jury models, and describes evaluation and fine-tuning choices. It also presents an error taxonomy for analyzing model predictions.

G.2 LLM Jury Limitations and Mitigation Strategies

  • The jury compares predictions with reference answers but cannot verify table grounding without visual context, risking false positives from correct answers based on outside knowledge.Providing rendered tables could mitigate this risk, but increases API cost and inference latency; the authors point to hybrid automatic and human evaluation as a need.Source: §G.2 · p. 19
  • The LLM jury aligned with human judgments within 4.7% on Visual-TableQA and VTabFact, even under a strict protocol that penalizes flawed reasoning despite correct final answers.Table 8 details jury misclassifications as percentages of evaluated examples.Source: §G.2 · p. 20, Table 8 · p. 20

G.3 LLM Jury Agreement Analysis

  • For table and QA-quality assessment, GPT-4.1 and Gemini-2.5-pro aligned most strongly with majority votes, while Deepseek-v3.1 aligned least; benchmark-evaluation agreement was strong across models.The paper attributes stronger benchmark agreement to semantic comparison being simpler than judging table-question-answer reasoning.Source: §G.3 · p. 20, Figure 11 · p. 21, Figure 12 · p. 21

G.4 Evaluator Consistency Compared to the Literature

  • The authors consider their majority-vote jury results broadly comparable with cited studies’ results, while noting that their stricter multi-model vote may yield more demanding evaluations.Their jury includes Mistral-large, Deepseek-v3.1, Gemini-2.5-pro, GPT-4.1, and Deepcogito-v2; cited baselines used GPT-4o or GPT-4 as sole evaluators.Source: §G.4 · p. 20

H Model Finetuning Hyperparameters

  • Qwen2.5-VL-7B-Instruct was fine-tuned in two phases: first on text-side modules and the multimodal projector, then on attention projections in the last four vision blocks.The remaining vision layers were left untouched; Table 9 lists LoRA hyperparameters.Source: H Model Finetuning Hyperparameters · pp. 20–21, Table 9 · p. 22

I Error Taxonomy of Model Predictions

  • The paper’s manual analysis examines Qwen2.5-VL-7B-Instruct predictions before and after fine-tuning on Visual-TableQA and TableVQA-Bench’s VTabFact split.The analysis defines error types and compares errors on VTabFact.Source: I Error Taxonomy of Model Predictions · p. 21

I.1 Error Categories

  • The error categories include partial extraction, hallucination, incoherence, misunderstanding, faulty reasoning, and ambiguity.Source: §I.1 · p. 22
  • The taxonomy also includes dataset mistakes, where the source contains annotation or label errors.Source: §I.1 · p. 22

I.2 Error Analysis on VTabFact

Fine-tuning changes the VTabFact error profile: total errors rise, while reasoning-method errors fall and newly introduced errors concentrate in fewer categories. Arithmetic and answer-template issues are identified as sources of some errors.

I.2 Error Analysis on VTabFact

  • Errors rose from 67 to 88 out of 250 samples, with most newly introduced errors in Incoherence and Hallucination while other error types declined.Faulty Methodology/Reasoning errors were significantly reduced; corrected errors spanned all categories, whereas newly introduced errors were concentrated in two.Source: §I.2 · p. 22
  • Post-finetuning arithmetic and numerical-comparison mistakes could produce faulty deductions despite otherwise correct reasoning, and were less frequent in the pretrained model.Source: §I.2 · p. 22
  • The fine-tuned model sometimes states an incorrect answer before a chain of thought that justifies the correct conclusion; these cases count as errors, unlike the pretrained model’s reasoning-first pattern.The authors associate the pretrained model’s pattern with fewer coherence violations.Source: §I.2 · p. 22

J Visual-TableQA Sample

Table 10 provides more detailed examples of dataset samples.

J Visual-TableQA Sample

K Image-to-LaTeX Dataset

The paper releases Img2TeX, a companion dataset of generated table images and their LaTeX source. It targets visual-to-LaTeX generation, while the accompanying sample QA pairs illustrate interpretive table questions.

K Image-to-LaTeX Dataset

  • Img2TeX contains table images generated during Visual-TableQA construction and their corresponding LaTeX source code.Source: K Image-to-LaTeX Dataset · pp. 23–24
  • Unlike Kale & Nadadur (2025), which evaluates LaTeX generation from textual prompts, Img2TeX targets generation from visual inputs.The paper presents it as a benchmark for VLMs’ visual-to-structured-text generation capabilities.Source: K Image-to-LaTeX Dataset · p. 24
  • The sample QA pairs address multiple visual aspects and include interpretive questions beyond simple information extraction.The second-row example is a “how” question.Source: Table 10 · p. 25
Loading 2509.07966v2…