WildTableBench: Benchmarking Multimodal Foundation Models on Table Understanding In the Wild

Junzhe Huang; Xiaoxiao Sun; Yan Yang; Yuxuan Hou; Ruotian Zhang; Sirui Li; Hehe Fan; Serena Yeung-Levy; Xin Yu

Source-linked AI summary

WildTableBench: Benchmarking Multimodal Foundation Models on Table Understanding In the Wild

Junzhe Huang, Xiaoxiao Sun, Yan Yang, Yuxuan Hou, Ruotian Zhang, Sirui Li, Hehe Fan, Serena Yeung-Levy, Xin Yu

arXiv:2605.01018v2May 1, 2026cs.CV

TL;DR

Real-world table images remain underrepresented in multimodal model evaluations despite their visual and structural complexity. WildTableBench evaluates 21 models on 402 images and 928 questions, finding that only one model exceeds 50% accuracy.

Problem
Existing evaluations largely use structured-text tables or clean rendered images rather than visually complex, naturally occurring table images from real-world settings.
Method
WildTableBench evaluates multimodal models on 402 real-world table images using 928 questions across five categories and 17 subtypes.
Results
Only one evaluated model exceeds 50% accuracy, while all other models score below 50.0%, indicating persistent difficulty with table structure perception and reasoning over cell values.
Takeaways & Limitations
WildTableBench provides a diagnostic benchmark for characterizing current multimodal models’ weaknesses in real-world table image understanding.
Takeaways & Limitations
Because its images come from web sources, the benchmark may inherit domain imbalance and annotation bias from source materials.

Abstract

from arXiv · show

Using multimodal foundation models to analyze table images is a high-value yet challenging application in consumer and enterprise scenarios. Despite its importance, current evaluations rely largely on structured-text tables or clean rendered images, leaving the visual complexity of in-the-wild table images underexplored. Such images feature varied layouts and diverse domains that demand sophisticated structural perception and numerical reasoning. To bridge this gap, we introduce WildTableBench, the first question-answering benchmark for naturally occurring table images from real-world settings. WildTableBench comprises 402 high-information-density table images collected from online forums and websites across diverse domains, together with 928 manually annotated and verified questions spanning 17 subtypes across five categories. We evaluate 21 frontier proprietary and open-source multimodal foundation models on this benchmark. Only one model exceeds 50% accuracy, while all remaining models range from 4.1% to 49.9%. We further conduct diagnostic analyses to characterize model failures and reveal persistent weaknesses in structural perception and reasoning. These results and analyses provide useful insights into current model capabilities and establish WildTableBench as a valuable diagnostic benchmark for table image understanding. Dataset: https://huggingface.co/datasets/jzhuang/WildTableBench Code: https://github.com/hjzhe/WildTableBench Leaderboard: https://hjzhe.github.io/WildTableBench

1 Introduction

WildTableBench addresses the gap in evaluating multimodal models on visually dense, structurally irregular table images from real-world settings. It combines diverse images and annotated questions with broad model evaluations and diagnostics that expose persistent reasoning and visual-grounding weaknesses.

Benchmark: 402 table images from publicly available web sources span diverse visual styles, structural patterns, and domains, including worksheets, transportation, sports, finance, science, engineering, and healthcare.The images include screenshots, scans, and photographs.
Evaluation: 21 proprietary and open-source multimodal foundation models, including instruction-tuned and reasoning variants, were systematically evaluated on naturally occurring table images.The evaluated families include Gemini, GPT, Claude, Seed-series, Kimi, GLM, and Qwen-series models.
Evaluation: 67.9% of questions were answered correctly by the best model, Gemini-3-Pro, while all other models scored below 50.0%, with some below 5%.Performance varied substantially across model families and parameter sizes.
Benchmark: 928 human-crafted and verified questions span 17 subtypes across five categories, covering cell-level understanding, numerical reasoning, fact verification, hypothetical reasoning, and color-based tasks.The taxonomy was developed from practical use cases found in public forums, websites, and user-shared table images.
Diagnostics: Diagnostic experiments quantify the effects of reasoning difficulty and reasoning budget, while identifying performance declines for cells in deeper rows or columns.These failure patterns indicate a need for improved visual grounding and reasoning.

2 Related Work

Prior table-understanding benchmarks primarily use structured text or clean, programmatically rendered images, leaving naturally occurring table images underexplored. WildTableBench addresses this gap by collecting real-world images with diverse domains, irregular layouts, visual noise, and structural complexity.

Text-based table benchmarks: Existing table benchmarks largely represent tables as structured text for tasks such as fact verification and multi-subtype language-model evaluation.Examples include TabFact, which uses Wikipedia tables in CSV format, and TableBench, which evaluates 18 task subtypes.
Rendered table image benchmarks: Visual table benchmarks commonly convert structured data into clean images through webpage rendering or crops from annual-report PDFs.TableVQA-Bench uses CSS-based webpage screenshots and FinTabNet report-PDF crops; M2-TabFact extends TabFact with rendered images.
Summary: No existing benchmark directly collects table images from real-world visual sources, which motivates WildTableBench’s focus on domain diversity, layout irregularity, visual noise, and structural complexity.The benchmark sources images through web search engines and includes both high-fidelity digital screenshots and natural photographs across professional and daily domains.

3 Benchmark Construction

WildTableBench is constructed from naturally occurring table images collected and quality-controlled for visual diversity, structural complexity, and de-identification. Its annotation framework spans five categories and 17 subtypes, yielding 928 human-verified questions across 402 images.

Image Collection: Three-stage collection pipeline searches diverse web platforms, filters candidates, and expands the dataset through Bing and Yandex retrieval.Images are automatically filtered with a GPT-4o-based VLM, manually reviewed, and screened for quality, structural complexity, diversity, and personal information.
Image Collection: 402 table images were collected from publicly available web sources across seven domain categories.The images cover a wide range of topics and scenarios.
Question Taxonomy: The question taxonomy organizes items into five categories and 17 subtypes, covering cell understanding, numerical reasoning, fact verification, hypothetical reasoning, and color-based reasoning.Capabilities range from local cell perception and retrieval to multi-step arithmetic, ranking, conditional aggregation, counterfactual recomputation, and color-selected computations.
Annotation Strategy: 928 questions were annotated across 402 images, averaging 2.3 questions per image.More than 95% were newly written or substantially revised by humans, and all reference answers were human-verified.

4 Experiments

Experiments evaluate 21 proprietary and open-source models on free-form WildTableBench questions using GPT-5.2 as an LLM judge, with 96%±2% agreement with human judgments. Results show substantial performance gaps, scaling effects, category-specific weaknesses, and a dominant visual-grounding bottleneck.

Models: 21 models are evaluated, spanning proprietary API-based and open-source systems.The open-source set includes ten Qwen3-VL variants across five parameter scales, plus Kimi K2.5 and GLM-4.6V.
Evaluation protocol: Free-form answers avoid multiple-choice shortcuts, and GPT-5.2 judges responses using format-insensitive matching across answer types.Four audits of 50 responses each found 96%±2% agreement with human judgments.
Overall results: 67.9% is Gemini-3-Pro’s best overall accuracy, while Kimi-K2.5 leads open-source models at 49.9% and Qwen3-VL-235B-T reaches 34.7%.Gemini-3-Pro exceeds Kimi-K2.5 and Qwen3-VL-235B-T by 18.0% and 33.2%, respectively.
Category analysis: C3 verification is often strongest, whereas C5 color is frequently weakest and becomes substantially harder when color supports reasoning.Kimi-K2.5 declines from 66.7% on C5-I to 27.5% on C5-C and 35.5% on C5-R; GLM-4.6V declines from 66.7% to 17.5% and 19.4%.
Scaling analysis: Qwen3-VL Thinking accuracy rises from 4.1% at 2B to 34.7% at 235B, with a larger Thinking advantage at larger scales.The corresponding Instruct results rise from 5.8% to 25.2%, while Thinking exceeds Instruct by 4.8 points at 235B.
Error analysis: Most failures occur during Locating and Recognition rather than explicit reasoning, indicating that reliable visual grounding is the main benchmark bottleneck.Errors increase as overall accuracy decreases, with locating errors growing especially sharply for weaker models.

5 Conclusion

WildTableBench evaluates multimodal foundation models on real-world table images using a diverse, manually annotated question set, addressing limitations of prior structured-text and programmatically rendered benchmarks.

WildTableBench comprises 402 table images collected directly from real-world web sources.This distinguishes it from benchmarks based on structured text representations or programmatically rendered images.
928 questions span five categories and 17 subtypes derived from these images.
The benchmark evaluates 21 frontier and representative multimodal foundation models.

Ethics Statement

WildTableBench uses publicly available web images while avoiding intentionally private or highly sensitive content, but may inherit source-data bias and domain imbalance. It is intended only for research evaluation and should not be treated as evidence of reliability in high-stakes applications.

Data ethics: The benchmark uses publicly available real-world table images and avoids intentionally collecting private or highly sensitive content.Its sources may still contain inherited bias, annotation bias, and domain imbalance.
Intended use: WildTableBench is intended solely for research evaluation of multimodal foundation models on realistic table image understanding.The authors position it as a tool for diagnosing limitations and improving robustness in visual grounding and reasoning.
Intended use: The benchmark should not be treated as evidence that models are reliable for medical, financial, legal, or safety-critical decision-making.These domains require caution beyond benchmark performance.

Appendices · A Keyword-Based Image Collection · B Image Domain Taxonomy

The appendices describe a broad keyword-based retrieval strategy for collecting real-world table images and a taxonomy for organizing the resulting 402-image dataset. The taxonomy combines seven coarse domains with fine-grained category labels spanning diverse professional, public, scientific, and recreational settings.

A Keyword-Based Image Collection: Figure 6 summarizes the keyword schema used to seed candidate image retrieval.The schema is presented as a retrieval framework for the image-collection process.
A Keyword-Based Image Collection: The retrieval space combines spreadsheet-oriented prompts that foreground tabular structure with scenario-grounded web queries reflecting realistic search intents.This approach avoids relying only on a narrow set of generic spreadsheet prompts.
A Keyword-Based Image Collection: The keyword schema is organized by representative table types and lists corresponding example retrieval queries.The schematic connects table types with example queries used to guide candidate retrieval.
B Image Domain Taxonomy: Table 4 presents the domain taxonomy used to organize 402 table images in WildTableBench.The taxonomy provides the organizational structure for the benchmark’s collected images.
B Image Domain Taxonomy: Each image is assigned to one of seven coarse-grained domains and annotated with a fine-grained category label.Finance & Accounting, for example, includes financial statement, stock table, and price list categories.
B Image Domain Taxonomy: The taxonomy covers professional, public, scientific, and recreational real-world settings.Its breadth provides a structured basis for analyzing model performance across image domains.

C Implementation Details

The evaluation uses unified API access with timeout and retry controls, while locally deployed Qwen3-VL models run on NVIDIA GPUs. Final evaluation questions are guaranteed valid model responses.

API-accessed models use OpenRouter’s unified endpoint, except Seed-2.0-Pro, which uses a separate API.
120-second per-request timeouts and up to five retries with exponential backoff handle transient errors.
Locally deployed Qwen3-VL models run on NVIDIA GPUs, and all final evaluation questions have valid model responses.

D Evaluation Prompts

The evaluation uses a two-step pipeline that separates standardized answer generation from GPT-5.2-based answer verification against annotated ground truth. Verification is format-insensitive for general questions but stricter for Transcription, requiring complete, correctly ordered cell-value recovery.

D Evaluation Prompts: The pipeline first supplies each model with the table image and task prompt, then uses GPT-5.2 to compare its response with the annotated ground-truth answer.The judge follows explicit, format-insensitive decision rules.
D Evaluation Prompts: Generation is standardized across models to reduce prompt-induced variance, while judging prompts specialize to the required answer format.Transcription uses a stricter judging template because it requires exact recovery of multiple table entries.
D.1 Answer Generation Stage: Models are instructed to reason step-by-step and end with a clearly marked final answer for deterministic extraction.The full response remains available when the extracted answer is missing, incomplete, or ambiguous.
D.2 Automatic Answer Verification: General Questions: For general questions, the judge receives the question, ground truth, extracted final answer, and full response, prioritizing the extracted answer and using the full response only as fallback.The judge treats equal values as correct and ignores formatting differences.
D.3 Automatic Answer Verification: Transcription Questions: Transcription verification requires all cell values in the specified order, rejecting omissions, insertions, permutations, and wrong values.If a final answer is partial but contains only correct values, the judge consults the full response for completeness.

E Cell-Position Sensitivity: Full Results

This evaluation measures exact cell-value retrieval across 2,489 needles from 50 real-world spreadsheet images. Accuracy is analyzed on a 10×10 row-by-column position grid, with per-model heatmaps showing min–max-normalized performance for eight additional models.

Position grid: Row depth and column depth are divided into ten equal bands, producing a 10×10 position grid with accuracy computed per bin.The bands span 0–10% through 90–100% for both dimensions.
Heatmap results: Figure 13 presents cell-retrieval accuracy heatmaps for eight additional models, including two proprietary and six open-source models.Each subplot reports per-model min–max-normalized accuracy across the 10×10 row × column grid.

F More Analysis on Reasoning · G Accuracy by Image Type

WildTableBench rewards increased reasoning effort across question categories, while image type affects accuracy differently by category: Hypothetical questions favor Non-Spreadsheet images, whereas Verification favors Spreadsheet images.

F More Analysis on Reasoning: Higher reasoning effort improves performance across all WildTableBench question categories.Gemini-3-Pro shows particularly large gains for Hypothetical questions, with improvements of around one quarter when moving from low to high reasoning effort.
F More Analysis on Reasoning: Around one quarter is the gain for Gemini-3-Pro on Hypothetical questions when reasoning effort increases.Hypothetical questions are among the most improved categories, underscoring their need for careful reasoning rather than direct answers.
G Accuracy by Image Type: 54.6% of images are Spreadsheet and 45.4% are Non-Spreadsheet in the image-type analysis.Table 5 reports accuracy by image type and question category across all 21 models.
G Accuracy by Image Type: −9.1% is the average C4 Hypothetical difference, with Non-Spreadsheet images outperforming Spreadsheet images across nearly all models.The Non-Spreadsheet advantage varies by model and category but is consistent for C4.
G Accuracy by Image Type: +5.1% is the average C3 Verification difference, with Spreadsheet images generally outperforming Non-Spreadsheet images.Verification is the category that tends to favor Spreadsheet images in the per-model breakdown.
G Accuracy by Image Type: Keyword-based retrieval expands the benchmark across spreadsheet-style and scenario-grounded table queries from the open web.The collection schema uses representative table types and query families to cover diverse real-world table scenarios.
G Accuracy by Image Type: The image collection includes Business & Management, Finance & Accounting, Sports & Health, Education & Science, Transportation, and Society & Media domains.These domain labels summarize the benchmark’s covered image categories.