PaperlayerView exact revision on arXiv

Source-linked AI summary

Chartographer: Counterfactual Chart Generation for Evaluating Vision-Language Models

Yifan Jiang, Dae Yon Hwang, Jesse C. Cresswell, Freda Shi

arXiv:2605.27311v1cs.CLcs.CV

TL;DR

Existing chart QA benchmarks can permit shortcuts or prior chart familiarity, leaving unclear whether VLM behavior generalizes when underlying visual evidence changes. CHARTOGRAPHER reconstructs charts into executable representations, generates controlled counterfactuals, and recomputes answers; across benchmark datasets, these variants reveal failures after original-chart success, especially when updated charts require novel visual reasoning pathways.

  • Problem

    Existing fixed chart-question-answer benchmarks leave unclear whether VLM behavior generalizes when underlying chart data changes rather than relying on shortcuts or prior familiarity.

  • Method

    CHARTOGRAPHER reverse engineers charts into executable data and rendering code, validates reconstructions, generates seed-controlled counterfactual variants, and recomputes answers with executable QA logic.

  • Results

    Counterfactual charts reveal that models often fail to generalize after answering the original chart correctly, exposing stale and noisy updates across ChartQA, CharXiv, and ChartMuseum.

  • Takeaways & Limitations

    Counterfactual evaluation exposes generalizability gaps, memorization, and reliance on parametric knowledge that single-chart performance can miss.

  • Takeaways & Limitations

    The benchmark filters out charts whose data, encodings, or question semantics cannot be recovered reliably and does not test changes in chart type, question style, or visual design.

Abstract

from arXiv · show

Chart question-answering (QA) benchmarks aim to pose questions that require visual reasoning to correctly answer, but models can often reach solutions through shortcuts or prior familiarity with a chart based on their own background knowledge. To strictly evaluate visual reasoning, we propose counterfactual charts where the chart-question task remains fixed, but underlying chart and the corresponding answer are varied. We introduce Chartographer, a framework to reverse engineer charts into executable code, validate reconstruction fidelity, generate seed-controlled counterfactual variants, and derive new answers from executable QA logic. We apply this framework to existing chart QA datasets and evaluate proprietary and open-source vision-language models (VLMs), measuring variation sensitivity and generalizability. Counterfactual charts reveal failures hidden by single-chart performance: VLMs often fail to generalize after answering the original chart correctly. We find failures are most prevalent when updated charts require novel visual reasoning pathways.

1 Introduction

Chart QA benchmarks test visual reasoning but often use fixed chart-question-answer triples that permit shortcuts. CHARTOGRAPHER addresses this gap with counterfactual chart-question families and finds that original-chart success often fails to generalize when visual evidence changes.

  • Chart QA requires reading encodings, comparing values, aggregating information, tracking trends, and sometimes applying domain conventions.
  • Existing benchmarks largely tie each question and answer to one fixed chart, leaving generalization under changed underlying data unclear.
  • CHARTOGRAPHER reconstructs charts into code, validates the reconstruction, generates counterfactual variants, and recomputes answers with executable QA logic.
  • Chart-question families shift evaluation from recovering one fixed answer to testing whether predictions remain grounded when visual evidence changes.
  • Success on original charts often fails to generalize when visual evidence changes.

2 Related Work

Related work establishes chart QA as a test of increasingly complex visual and logical reasoning across synthetic, web-sourced, and scientific charts. CHARTOGRAPHER extends counterfactual evaluation by altering chart data itself while preserving the chart-question task.

  • Chart QA datasets span controlled synthetic figures, scientific plots, web-sourced charts, and realistic scientific visualizations.
  • VLM chart understanding requires parsing encodings, recovering quantities, following labels and legends, and performing numerical operations.
  • Prior multimodal evaluations can reward parametric knowledge and superficial linguistic regularities instead of visual logic.
  • CHARTOGRAPHER regenerates charts with altered data rather than using textual hypotheticals or visually similar replacements, requiring reasoning over changed visual evidence.

3 CHARTOGRAPHER: Counterfactual Chart Framework

CHARTOGRAPHER converts chart QA examples into validated counterfactual families containing an original, a reconstruction, and data-altered variants. Its evaluation separates reconstruction fidelity, variant sensitivity, and conditional generalizability while diagnosing update failures.

  • Family construction: Each family contains the original chart, a base reconstruction, and counterfactual variants whose answers are recomputed from underlying data.
  • Family construction: Chart-to-code reconstruction separates semantic data from rendering code so data values can vary while visual encodings and layout remain controlled.
  • Family construction: Human reviewers approve usable reconstructions, reject low-fidelity outputs, and record assumptions when labels, values, or visual encodings are difficult to recover.
  • Family construction: Each accepted reconstruction generates ten seed-controlled counterfactual charts that preserve schema, rendering constraints, and domain assumptions while changing data meaningfully.
  • Family construction: Executable QA logic computes answers directly from underlying data and rewrites questions only when necessary to preserve coherence.
  • Evaluation metrics: The framework reports original accuracy, reconstruction accuracy, variant accuracy, relative variant change, and conditional variant accuracy.
  • Failure diagnosis: Failed generalization is classified as correct update, stale prediction, or noisy update among variants whose original chart was answered correctly.

4 Experiment Setup

The experiments evaluate instruction-tuned proprietary and open-source VLMs on sampled chart QA tasks from three datasets. The setup spans conventional charts, scientific figures, and human-designed visualizations.

  • Datasets: The study samples 462 chart QA tasks from ChartQA, CharXiv, and ChartMuseum evaluation splits.
  • Datasets: The datasets provide a varied testbed covering conventional charts, scientific figures, and human-designed visualizations.
  • Models: Models include proprietary Claude, Gemini, and GPT families alongside open-source Gemma, InternVL, LLaVA, Pixtral, and Qwen variants.

5 Results

Chartographer evaluates whether VLMs remain grounded when chart evidence changes, separating reconstruction effects from counterfactual generalizability. Across datasets and reasoning types, models often fail after solving the original chart, especially when updated answers require visual grounding.

  • Evaluation framework: Chartographer uses chart-question families with reconstructions and counterfactual variants to measure reconstruction fidelity, variant sensitivity, and generalizability after original success.The framework checks base reconstructions before evaluating changed visual evidence.
  • 5.1 Reconstruction Controls: Reconstructed charts achieve accuracy close to or above original charts across datasets, with the clearest increase on CharXiv.Cleaner, higher-resolution code-generated charts may reduce incidental clutter while preserving task-relevant relations.
  • 5.2 CVA Reveals Failed Generalization After Original-Chart Success: Conditional variant accuracy shows that models do not consistently generalize to altered visual evidence even after answering the original chart correctly.ChartQA generalizes relatively well, whereas CharXiv and ChartMuseum have low CVA; proprietary models outperform open-source models on CVA.
  • 5.3 Failed Generalization Reflects Stale and Misgrounded Answers: Noisy updates are the more common failure type, while stale predictions remain substantial on harder datasets, especially for open-source models.Noisy updates change the answer without grounding it correctly; stale predictions retain the original answer after the evidence changes.
  • 5.4 Generalizability is Weakest for Visually Grounded Questions: Generalizability is highest for text-only questions and weakest for Synthesis and Visual questions that require visual reasoning.The same pattern appears for both proprietary and open-source model groups.
  • 5.5 Failure Case Studies: Case studies show stale predictions, noisy updates, and correct updates after visual evidence changes, including reuse of reasoning that describes the original rather than the variant.One visual-comparison case preserves the original prediction despite a changed answer; another requires inferring a missing month label and only GPT-5.4 updates correctly.

6 Conclusion

CHARTOGRAPHER constructs counterfactual chart-question families to test whether VLM answers remain grounded when chart data changes. Across multiple datasets, the variants expose generalization gaps and failures to re-ground reasoning that single-chart evaluation can hide.

  • CHARTOGRAPHER constructs counterfactual chart-question families by reverse engineering chart code, applying quality controls, changing underlying data, and re-rendering variants.The resulting variants preserve the overall chart-question task while changing the visual evidence.
  • Across ChartQA, CharXiv, and ChartMuseum, counterfactual variants expose failures that single-chart evaluation can hide.Models can answer an original chart correctly yet fail the same task after altered data are rendered.
  • Models show stale predictions and noisy updates when they must re-ground reasoning on updated visual evidence rather than rely on explicit textual cues.These failure modes reveal generalizability gaps, memorization, and reliance on parametric knowledge rather than reasoning ability.
  • Counterfactual evaluation provides a tool for diagnosing VLM chart reasoning beyond single-chart performance.The framework measures whether predictions remain grounded when the chart data change.

Limitations

The benchmark is limited to charts that can be reconstructed and parameterized reliably, and its counterfactuals change data without changing chart type, question style, or visual design.

  • The benchmark filters out charts with ambiguous labels or unrecoverable values because their relevant data, encodings, and question semantics cannot support reliable executable generation.This improves counterfactual label reliability but biases the benchmark toward charts that can be reconstructed and parameterized cleanly.
  • Counterfactual variants preserve the chart-question task while changing underlying data, so they do not test broader shifts in chart type, question style, or visual design.The benchmark isolates sensitivity to changed visual evidence rather than robustness to changes in task posing or chart encoding.

A.1 Source Data, Filtering, and Licenses

The pipeline filters charts for reliable controlled reconstruction, then generates and evaluates chart-question families through executable, seed-controlled components. Metrics separate original performance from reconstruction, variant accuracy, and post-success update behavior.

  • Source Data, Filtering, and Licenses: Charts are excluded when distributions, dense values, or target relations cannot support reliable controlled reconstruction.Filtering removes cases where chart-to-code recovery or visual-detail reconstruction would not permit dependable data edits.
  • Reconstruction Pipeline: Each accepted chart is reconstructed as semantic data plus Python rendering code, with iterative diagnostics, revision, and human review when needed.Review covers layout, marks, scales, labels, legends, panels, styling, encoded values, and answerability.
  • Counterfactual Generation: Accepted families generate ten seed-controlled variants using same-schema data generators with meaningful changes rather than only small noise.Seeds 0–9 produce the counterfactual variants while preserving schema and rendering constraints.
  • Executable QA: Executable question and answer modules preserve the original task when valid and recompute gold answers from current data, including reordered elements and changed values.The answer generator uses schema fields and preserves the question’s answer format.
  • Evaluation: Models receive only the chart image and applicable question, while answer equivalence ignores superficial formatting differences but requires semantic agreement.Gold answers, code, seeds, reconstructed data, and regime labels are withheld from models.
  • Metrics: Conditional variant accuracy isolates generalization after original-chart success, while CU, SP, and NU partition update behavior within model-specific diagnostic sets.Aggregates average per-model quantities, giving each model equal weight despite differing diagnostic-set sizes.

B.1 Reconstruction Accuracy Analysis

Reconstructed charts can match or exceed original-chart performance, yet counterfactual evaluation still reveals model-specific failures after original success. The accompanying tables and figures report conditional behavior rather than only static accuracy.

  • Reconstruction Accuracy: VLMs showed near-equal, and sometimes superior, performance on reconstructed charts compared with originals.GPT-5.4-mini examples answer an identical question correctly on a reconstruction but not the original, despite near-identical content to humans.
  • Conditional Evaluation: Conditional variant accuracy reports performance only for families whose model answered the original chart correctly.This framing complements aggregate metrics and focuses on generalization after original-chart success.
  • Update Outcomes: Per-model update outcomes are computed on each model’s own diagnostic set, so reported rates are model-specific conditional rates rather than a shared variant-pool rate.Table 5 expands group-level CU, NU, and SP patterns into per-model results.

B.5 ChartMuseum Failure Case Studies

ChartMuseum case studies distinguish stale reuse of original-chart reasoning from incorrect updates that fail to ground changed visual evidence. These failures span multiple visual reasoning types and can reflect either memorization-like reuse or local grounding errors.

  • Failure Modes: The case studies define stale prediction as retaining the original answer after the gold answer changes, and noisy update as changing the answer but remaining incorrect.The analysis conditions these outcomes on original-chart success and uses reasoning-type annotations or analogous categories.
  • Case 1: Stale Prediction: Stale responses reused original-chart duration claims nearly word-for-word, supporting possible reliance on parametric knowledge of the original chart or QA pair.GPT-5.4 and Gemini 2.5 Pro preserved explanations about “Watching TV” after the counterfactual changed the longest Time Waster.
  • Reasoning Types: The case-study categories cover trajectory tracking, spatial comparison, symbol-guided comparison, value identification or thresholding, and other visually grounded cases.CharXiv lacks reasoning-type annotations, so selected families use ChartMuseum-inspired categories with more specific labels when needed.

Counterfactual Variant

Counterfactual variants preserve chart questions while changing visual evidence and recomputing answers, exposing whether models update their reasoning appropriately. Case studies show failures across trajectory, spatial, symbol-guided, and thresholding tasks, with noisy and stale updates varying by family.

  • Counterfactual Design: Counterfactual families include an original chart, a reconstruction, and variants whose answers are recomputed after controlled visual changes.The reconstruction checks task preservation, while variants test grounding after altered evidence.
  • Update Outcomes: Update outcomes decompose post-success variant behavior into correct update, noisy update, and stale prediction across model-seed pairs.Table 7 summarizes dominant NU/SP modes by visual task category.
  • Spatial Visual Comparison: In the spatial step-counting case, the counterfactual gold answer changed from 4 to 6, while representative models answered 10, 4, and 5; CU was 0.20 and NU was 0.77.The task requires following a discrete staircase path rather than measuring continuous distance.
  • Spatial Visual Comparison: In line-marker binding, the gold answer remained λL = 0.13 originally and counterfactually, but representative models shifted to λL = 0.30; CU was 0.30 and NU was 0.70.The comparison depends on binding each colored trajectory to its corresponding marker before measuring distance from the WH axis.
  • Symbol-Guided Comparison: In legend-bound bar comparison, the counterfactual gold changed from PyTorch to XLA, while models answered MXNet or PyTorch; CU was 0.34 and NU was 0.60.The question requires comparing CONV bars while preserving library identities and ignoring plausible distractor bars.
  • Symbol-Guided Comparison: Paired-bar matching produced a counterfactual gold answer of Abiotic resources versus Biotic resources originally, with SP at 0.48 and CU at 0.38.The task requires every non-(A) bar to be smaller than its matching (A) counterpart across a subplot.
  • Value Identification and Thresholding: Thresholding cases changed from 2 to 3 bars above 0.25 and from 3 to 6 DJS points below 0.1, with CU/NU rates of 0.45/0.50 and 0.50/0.50.These tasks require panel selection, strict threshold application, and counting individual bars or points near the cutoff.
Loading 2605.27311v1…