PaperlayerView exact revision on arXiv

Source-linked AI summary

ChartMuseum: Testing Visual Reasoning Capabilities of Large Vision-Language Models

Liyan Tang, Grace Kim, Xinyu Zhao, Thom Lake, Wenxuan Ding, Fangcong Yin, Prasann Singhal, Manya Wadhwa, Zeyu Leo Liu, Zayne Sprague, Ramya Namuduri, Bodun Hu, Juan Diego Rodriguez, Puyuan Peng, Greg Durrett

arXiv:2505.13444v3cs.CLcs.CV

TL;DR

Chart understanding requires both textual and visual reasoning, but current LVLMs fall short on visual tasks that are difficult to solve through text. The paper tests visual reasoning with synthetic charts and introduces ChartMuseum, a human-curated benchmark; models remain behind humans, with a larger performance gap on visual-reasoning questions.

  • Problem

    Existing chart QA benchmarks often prioritize textual reasoning or have limited real-world chart sources, while LVLMs fall short on visual reasoning that cannot be performed through text.

    Source: §1 · pp. 1–2

  • Method

    The authors test models on synthetic charts solvable only through visual reasoning and introduce ChartMuseum, a researcher-curated benchmark of 1,162 question-answer tuples from real-world charts.

    Source: §1 · p. 2

  • Results

    35%-55% worse on complex visual-reasoning questions than on complex textual-reasoning questions; Gemini-2.5-Pro achieves 63.0% accuracy versus 93.0% for humans.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Leading proprietary models over-rely on textual reasoning and struggle with visual comparisons, selecting objects by visual markers, and reasoning about line trajectories.

    Source: §1 · p. 2

    The benchmark covers English short-answer QA and excludes unanswerable questions, limiting its scope across languages and chart-understanding task types.

    Source: §7 · p. 10

Abstract

from arXiv · show

Chart understanding presents a unique challenge for large vision-language models (LVLMs), as it requires the integration of sophisticated textual and visual reasoning capabilities. However, current LVLMs exhibit a notable imbalance between these skills, falling short on visual reasoning that is difficult to perform in text. We conduct a case study using a synthetic dataset solvable only through visual reasoning and show that model performance degrades significantly with increasing visual complexity, while human performance remains robust. We then introduce ChartMuseum, a new Chart Question Answering (QA) benchmark containing 1,162 expert-annotated questions spanning multiple reasoning types, curated from real-world charts across 184 sources, specifically built to evaluate complex visual and textual reasoning. Unlike prior chart understanding benchmarks -- where frontier models perform similarly and near saturation -- our benchmark exposes a substantial gap between model and human performance, while effectively differentiating model capabilities: although humans achieve 93% accuracy, the best-performing model Gemini-2.5-Pro attains only 63.0%, and the leading open-source LVLM Qwen2.5-VL-72B-Instruct achieves only 38.5%. Moreover, on questions requiring primarily visual reasoning, all models experience a 35%-55% performance drop from text-reasoning-heavy question performance. Lastly, our qualitative error analysis reveals specific categories of visual reasoning that are challenging for current LVLMs.

1 Introduction

Chart reasoning requires distinguishing tasks solvable through textual reasoning from those requiring visual reasoning, yet existing chart QA datasets often emphasize text or draw on limited real-world sources. ChartMuseum introduces human-curated questions to evaluate both capabilities and shows persistent model gaps, especially on visual reasoning.

1 Introduction

  • Existing chart QA datasets often prioritize textual reasoning or draw on limited real-world chart sources, leaving visual reasoning under-evaluated.The authors’ synthetic-chart case study found that LVLMs fell short at purely visual reasoning even when humans performed nearly perfectly.Source: §1 · p. 2
  • ChartMuseum contains 1,162 question-answer tuples from 928 real-world images across 184 websites, evaluating complex visual and textual reasoning.The dataset was created by 13 computer science researchers.Source: §1 · p. 2
  • Unlike prior benchmarks whose questions are often model-generated and refined by annotators, ChartMuseum questions were researcher-curated without LLM assistance and manually reviewed for quality and answer objectivity.The authors identify question templates and model-generated questions as potential limits on realism and diversity.Source: §1 · p. 2
  • Humans achieve 93.0% accuracy, compared with 63.0% for Gemini-2.5-Pro and 38.5% for Qwen2.5-VL-72B; visual-reasoning performance is 35%-55% worse than textual-reasoning performance.The analysis also finds that leading proprietary models over-rely on textual reasoning and struggle with visual comparisons, visual markers, and line trajectories.Source: §1 · p. 2
  • The paper contributes an analysis of chart understanding datasets, a synthetic experiment on visual reasoning, the ChartMuseum dataset, and qualitative error analysis of recent LVLMs.The error analysis is intended to identify model shortcomings for future research.Source: §1 · p. 2

2 Background and Motivation

The section distinguishes visual reasoning from textual reasoning and tests whether existing chart QA benchmarks depend heavily on chart text. Its experiments find that ChartQA can largely be answered from extracted text, while ChartMuseum performance degrades as synthetic visual complexity rises.

2 Background and Motivation

  • Prior chart-question datasets often overlook whether answering relies on textual or visual information, motivating the section’s distinction between the two.The paper examines the extent to which existing real-image chart benchmarks rely on textual reasoning.Source: §2 · p. 2, §2.2 · p. 3

2.1 Terminology

  • Visual reasoning is inference that arises more naturally from visual comparison than from natural-language reasoning, including chart relationships such as scatterplot correlation.A bar-height comparison is visual reasoning even though extracting values could also support it; visual extraction is the subclass involving numeric information inferred visually.Source: §2.1 · p. 3
  • Textual reasoning includes reasoning over extracted and verbalized information and directly extracting chart text, such as labels, titles, and annotated values.The authors treat images of text as fundamentally textual, despite the visual process involved in reading them.Source: §2.1 · p. 3

2.2 Prior Chart Understanding Benchmarks Over-represent Textual Reasoning

  • Claude-3.7-Sonnet scores 74.1% using extracted text versus 87.4% using chart images on ChartQA, but 15.2% versus 61.3% on ChartMuseum.The authors interpret the 46% ChartMuseum performance gap as evidence that the benchmark tests information not captured in accessible text.Source: §2.2 · p. 3, Table 1 · p. 3

A Synthetic Testbed for Visual Reasoning

  • The synthetic testbed uses 13 visual-reasoning questions across density, histogram, line, scatter, and violin charts, including overlay and subplot settings with n from 3 to 9.Overlay layers figures in one image, while subplot places each figure in a distinct panel.Source: A Synthetic Testbed for Visual Reasoning · pp. 3–4
  • Model performance steadily declines as the number of plots increases in both overlay and subplot settings, while increased visual complexity does not significantly affect human performance.The synthetic plots contain no written content and cannot be solved through text extraction; Figure 2 shows the histogram results.Source: A Synthetic Testbed for Visual Reasoning · p. 4, Figure 2 · p. 4

3 CHARTMUSEUM

CHARTMUSEUM is a 1,162-example chart QA benchmark built from real-world charts to evaluate LVLM reasoning across textual and visual tasks. Its researchers manually curated objective questions and used a multi-stage review process to check their quality.

3 CHARTMUSEUM

  • 1,162 examples make CHARTMUSEUM a real-world chart QA benchmark for evaluating LVLM reasoning; 13 computer science researchers collectively annotated it.Its charts come from 184 distinct website domains.Source: §3 · p. 4, §3.1 · p. 4

3.2 Annotation and Quality Check

  • ChartMuseum charts span 184 website domains, and Figure 3 organizes the dataset by chart types, sources, and major question topics.Researchers rejected templated questions in favor of questions tailored to individual charts.Source: §3.1 · p. 4, §3.2 · p. 5, Figure 3 · p. 6

3.1 Data Annotation Requirements

  • Questions require at least four answer options and objectively verifiable answers; numeric answers from unannotated chart elements are avoided.Questions about unannotated data focus on comparisons with unique answers, and question creation uses no LLM assistance.Source: §3.1 · pp. 4–5
  • The benchmark excludes subjective why/how questions, merely descriptive questions, and compound questions combining multiple queries.These exclusions were intended to support question quality and evaluability.Source: §3.1 · p. 5
  • Questions are classified as textual, visual, text/visual, or synthesis reasoning, distinguishing which modalities can answer them.Synthesis questions require both textual and visual reasoning.Source: §3.1 · p. 5

3.2 Annotation and Quality Check

  • Annotators selected charts, wrote question-answer pairs, and refined them with an independent reviewer through iterative quality checks.The process included practice and two formal sessions, with the reviewer checking answer correctness and resolving disagreements with annotators.Source: §3.2 · pp. 5–6

4 Experiments

The experiments compare recent proprietary and open-source LVLMs with human performance on CHARTMUSEUM. Models trail humans overall and show a substantial disadvantage on visual-reasoning questions relative to textual-reasoning questions.

4.1 Model Comparison

  • The benchmark includes nine listed open-source models, 10 proprietary models, and the specialized Bespoke-MiniChart-7B; its human study used six annotators to independently label 100 examples, each by three annotators.LLM-as-a-Judge is the main evaluation method for accounting for paraphrases.Source: §4.1 · p. 6

4.2 Results

  • 24.5% accuracy separates the best proprietary model Gemini-2.5-Pro (63.0%) from the best open-source model Qwen2.5-VL-72B-Instruct (38.5%).Humans achieve 93.0% overall accuracy on a randomly sampled 100-example subset.Source: Table 3 · p. 7, §4.2 · p. 7
  • 35% to 55% behind textual reasoning, LVLM performance on visual-reasoning questions remains markedly weaker; humans answer 56/57 sampled visual questions correctly (98.2%).Bespoke-MiniChart-7B and several other models show visual-subset decreases exceeding 50%, while Claude-3.7-Sonnet, o3 (high), and Gemini-2.5-Pro still show approximately 35% absolute accuracy drops.Source: §4.2 · pp. 7–8, Table 3 · p. 7
  • 1.4% improvement: Claude-3.7-Sonnet with extended thinking scores 61.7%, compared with 60.3% for its standard version.The passage reports that reasoning models’ improvements are within 3% of Claude-3.7-Sonnet without extended thinking.Source: §4.2 · p. 8

5 Qualitative Analysis

The analysis classifies visual tasks in ChartMuseum and examines sampled model errors. Most examined errors arise from visual reasoning, with distinct patterns in strategy choice and value identification.

5.1 Visual Task Taxonomy

  • The taxonomy identifies symbol selection, visual comparison, trajectory tracking and judgment, and x/y value identification as visual tasks.Questions can involve multiple categories; the reported category percentages come from a random sample of 50 non-textual-reasoning questions.Source: §5.1 · p. 8, Table 4 · p. 9

5.2 Visual Reasoning Errors

  • Only 6% of Claude-3.7-Sonnet and 2% of Gemini-2.5-Pro examined error instances were text-reasoning errors; most involved visual reasoning.The analysis examined 50 error instances per model, excluding Textual Reasoning questions.Source: §5.2 · p. 9
  • Strategy errors occur when a model misses the intended visual reasoning approach and instead takes a divergent chain of thought, often by extracting explicit x/y-values.This can happen when a desired value is implied relative to other visual elements rather than stated explicitly.Source: §5.2 · p. 9
  • Claude-3.7-Sonnet shows many explicit strategy errors, while Gemini-2.5-Pro has overrepresented x/y identification errors, often misreading the relevant tick label.In 3 of 14 Gemini x/y identification errors, the question could have been answered by comparing visual objects, but the model instead approximated their values.Source: §5.2 · p. 9

6 Related Work

Chart QA benchmarks have progressed toward greater realism and difficulty, but the authors identify limits in source diversity, question generation, and differentiation of frontier models. They position ChartMuseum as a focused evaluation of visual and textual reasoning in charts.

6 Related Work

  • Earlier chart QA benchmarks used synthetic images and templated questions, while ChartQA introduced human-annotated questions on real charts.Later benchmarks include CharXiv and ChartQAPro, but the authors note limitations in source breadth, frontier-model differentiation, and question generation.Source: §6 · p. 9
  • Multimodal benchmarks often target visual math or broad domain knowledge rather than visual reasoning specifically, while many reasoning benchmarks focus on text domains.The authors also note proposed limits involving visual encoders, feature alignment, abstract visual reasoning, and features that are difficult to describe in text.Source: §6 · p. 9
  • ChartMuseum focuses on chart understanding and explicitly separates visual from textual reasoning to investigate where and how models fail.The authors frame this as a focused investigation of a gap previously described more broadly in LVLM visual reasoning.Source: §6 · p. 10

7 Conclusion

ChartMuseum is a human-curated benchmark of real-world charts for evaluating complex visual and textual reasoning, and the authors report that models remain weak on visual reasoning. Its scope is limited to English short-answer QA without unanswerable questions.

7 Conclusion

  • Models perform 35%-55% worse on visual reasoning than text-based reasoning, and the authors attribute the weakness to limitations across visual reasoning tasks.ChartMuseum is based on real-world images and covers diverse chart types; the authors report that model accuracy varies widely but remains below human performance.Source: §7 · p. 10
  • The benchmark covers English charts and short-answer QA, excludes unanswerable questions, and does not evaluate other chart tasks such as summarization or open-ended responses.The authors argue short-answer QA is an effective proxy for identifying model weaknesses, while noting English-only results may not reflect multilingual performance.Source: §7 · p. 10

A CHARTMUSEUM Details

The appendix describes benchmark statistics and comparisons, alongside additional human and model performance reporting and licensing details.

A CHARTMUSEUM Details

  • Figure 5 compares CLIP encodings of 1,000 randomly sampled ChartMuseum charts with encodings from several real-image chart benchmarks.Figure 6 presents statistics of ChartMuseum; GPT-4o is used as the tokenizer for question and answer lengths.Source: A CHARTMUSEUM Details · p. 13, Figure 5 · p. 14, Figure 6 · p. 14
  • In sampled charts, 58% from ChartMuseum and 54% from CharXiv contain subplots.The comparison uses 50 randomly sampled arXiv charts from each dataset.Source: A CHARTMUSEUM Details · p. 14, Table 6 · p. 14
  • Among 50 sampled questions, 25 from CharXiv and 26 from ChartMuseum focus on extreme values, but ChartMuseum questions require image-specific multi-step reasoning.The authors report that sampled CharXiv questions often share similar reasoning, whereas ChartMuseum questions were written independently without a shared prior.Source: A CHARTMUSEUM Details · p. 14
  • In Section 4, humans achieved 93% performance by majority vote, while individual human performance ranged from 76% to 96% across the reported individual results.Source: A CHARTMUSEUM Details · p. 14
  • Development-set results report overall accuracy only because the set contains 162 examples.Source: A CHARTMUSEUM Details · p. 15
  • ChartMuseum is licensed under CC BY-SA 4.0, while copyright for included charts remains with their original authors and sources.Source: A CHARTMUSEUM Details · p. 15

A.1 A Comparison Against Existing Chart QA Benchmarks

The benchmark comparison reports average test-set performance across chart QA benchmarks and finds a larger spread among models on CHARTMUSEUM. The appendix also points readers to examples of CHARTMUSEUM questions.

A.1 A Comparison Against Existing Chart QA Benchmarks

  • The comparison focuses on ChartQA, CharXiv, and ChartQAPro as representative real-image-based benchmarks.For o3 (high) and Gemini-2.5-Pro, evaluation was limited to CHARTMUSEUM because of inference costs.Source: §A.1 · p. 15
  • Model performance varies widely on CHARTMUSEUM, unlike on the other compared datasets.Performance is averaged over each benchmark’s test set; ChartQAPro conversational and unanswerable questions are excluded to align its scope with CHARTMUSEUM.Source: §A.1 · p. 15, Figure 7 · p. 15
  • TinyChart-3B and ChartGemma-3B are excluded because a cited study reports they perform significantly worse than non-state-of-the-art open-source LVLMs.The exclusion is attributed to.Source: §A.1 · p. 15

A.2 Examples

  • Table 7 points readers to CHARTMUSEUM examples across different question and reasoning categories.The appendix describes these examples as image, question, and answer tuples.Source: §A.2 · p. 15

B.1.1 Dataset Creation

The synthetic dataset uses charts with visual properties that questions require readers to identify or compare. Its examples span chart types and question categories.

Dataset creation

  • Density and Histogram questions test shape categorization, while Line, Scatter, and Violin questions test comparisons of characteristics such as cluster density or height.The dataset presents five chart types and thirteen visual-reasoning questions.Source: §B.1.1 · p. 15, Table 8 · p. 16
  • The examples span textual, visual/textual, synthesis, and visual reasoning question categories.The appendix directs readers to figures illustrating each category.Source: §B.1.1 · p. 16
  • The visual-task taxonomy names Symbol Selection, Visual Comparison, Trajectory Tracking, and X/Y Value Identification.The appendix points to figures illustrating these task types.Source: §B.1.1 · p. 16
  • A Python script selects a figure with a target property, such as a left-skewed histogram, then generates the remaining figures with small random variations.The variations make the images appear different from one another.Source: §B.1.1 · p. 16

B.1.2 Human Evaluation and Results

Human annotators generally maintain high accuracy as synthetic-chart complexity increases, while model accuracy tends to decrease or plateau. The appendix also specifies how examples are sampled and categorized.

B.1.2 Human Evaluation and Results

  • Two researchers annotated 1,560 image-question pairs, with exact-match evaluation of their answers.Each annotator received 10 examples per chart type; examples used n ∈ [3, 6, 9], except Histogram used n ∈ [4, 6, 9].Source: §B.1.2 · p. 16
  • Human accuracy generally remains perfect as n increases, while model accuracy tends to decrease or plateau at similar or lower accuracy than human.Across chart types, human performance in Overlay and Subplot setups is consistently higher than or similar to model performance.Source: §B.1.2 · p. 16

C.1 Task Taxonomy Examples

  • The appendix groups visual tasks into Visual Comparison, Symbol Selection, Trajectory Tracking and Judgment, and X/Y-Value Identification.Table 9 breaks down visual comparisons in a random sample of 50 CHARTMUSEUM questions, excluding textual-reasoning questions; a question may involve multiple comparison types.Source: §C.1 · pp. 16–17, Table 9 · p. 16

C.2 Strategy Error Examples

The paper provides examples of strategy errors and identifies a synthesis example as one such error for two models.

C.2 Strategy Error Examples

  • Figures 20, 21, and 22 show strategy-error examples.Source: §C.2 · p. 17
  • The synthesis example in Figure 14 is also a strategy error for Claude-3.7-Sonnet and Gemini-2.5-Pro.Source: §C.2 · p. 17

D Model Details

This section documents model inference, hyperparameter, compute, and answer-evaluation settings, illustrates textual and visual reasoning examples, and summarizes benchmark and evaluation-code release details. It also records the study’s reporting scope, limitations, copyright and release-risk considerations, and statements about human subjects and LLM use.

D Model Details

  • The authors use official APIs for proprietary LVLMs, NVIDIA RTX A6000 GPUs for open-source models, and temperature 1 for o3, o4-mini, Claude-3.7-Sonnet Extended-Thinking, and Gemini-2.5-Pro; other models use temperature 0.Other hyperparameters remain at defaults; Table 10 lists the checkpoints, and open-source inference takes 5–20 minutes per model depending on size.Source: D Model Details · p. 17, Table 10 · p. 17
  • Answer correctness is evaluated with gpt-4.1-mini-2025-04-14, and evaluating a model on the benchmark costs $0.2.The evaluation prompt compares the ground-truth answer with the model’s extracted final answer for equivalence and uses temperature 0.Source: D Model Details · p. 17, Figure 27 · p. 41
  • The meat example computes cost per gram from chart coordinates and selects chicken wings, whose cost is approximately $0.0082/g, to maximize meat bought for $5.The example equates minimizing the product of calories per gram and cost per 100 calories with minimizing cost per gram.Source: D Model Details · p. 20
  • The plastic Sankey example shows competing interpretations: one calculates 0.4 billion tonnes from 5%, another infers 1% or 0.083 billion tonnes using discarded totals, and another says the path is indeterminate.The latter inference uses the 59% total discarded versus 55% discarded after single use to account for 4% of total plastic.Source: D Model Details · p. 23
  • For the Simpsons heatmap, the example identifies the median episode count across Seasons 1–9 as 24 by comparing row lengths rather than counting every episode.It notes that Seasons 6–9 each have 25 episodes and the second-longest row has 24.Source: D Model Details · p. 25
  • Claude-3.7-Sonnet focuses on the flight chart’s average-distance inset, while Gemini-2.5-Pro recognizes the flight arcs but gives up on estimating values; the intended strategy compares colored arcs’ ranges.The example identifies Friday as the answer because its green arcs include both the closest and furthest flights.Source: D Model Details · p. 35

NeurIPS Paper Checklist

  • The NeurIPS checklist says the paper’s main claims appear in the abstract and Section 1 and that limitations are discussed in Section 7.The checklist also says claims should match the theoretical and experimental results and reflect the scope of expected generalization.Source: NeurIPS Paper Checklist · p. 42
  • The authors state that the paper contains no theoretical results.Source: NeurIPS Paper Checklist · p. 43
  • The benchmark evaluation prompt is in Figure 27, with model inference details and hyperparameter setups in Appendix D; the authors also reference data construction and human evaluation procedures.They report releasing the benchmark and evaluation code, and uploading the benchmark and code with documentation.Source: NeurIPS Paper Checklist · pp. 43–44
  • Bootstrap confidence intervals are used for the synthetic-data case study, but benchmark results have no significance tests because the authors make no claim that any model is superior.They state that the benchmark test set has 1000 examples and that moderate gaps are meaningful; compute-resource details are in Appendix D.Source: NeurIPS Paper Checklist · p. 45
  • The authors say broader impacts are not discussed because they consider them ancillary to this benchmark, whose intended role is to advance vision-language model research.They describe the benchmark’s broader impacts as circumscribed by those of vision-language models.Source: NeurIPS Paper Checklist · p. 46
  • The authors do not believe their data poses a high misuse risk and state that they release no models.Source: NeurIPS Paper Checklist · p. 47
  • The authors say the benchmark license is mentioned, chart copyrights remain with their original sources, and benchmark data should not be used to pretrain or fine-tune NLP models.They also report uploading the benchmark and code with documentation.Source: NeurIPS Paper Checklist · p. 47
  • The paper states that it involves neither crowdsourcing nor research with human subjects.Source: NeurIPS Paper Checklist · p. 48
  • The authors state that core method development does not use LLMs as important, original, or non-standard components.Source: NeurIPS Paper Checklist · p. 49
Loading 2505.13444v3…