PaperlayerView exact revision on arXiv

Source-linked AI summary

Visual Programmability: A Guide for Code-as-Thought in Chart Understanding

Bohao Tang, Yan Ma, Fei Zhang, Jiadi Su, Ethan Chern, Zhulin Hu, Zhixin Wang, Pengfei Liu, Ya Zhang

arXiv:2509.09286v1cs.CV

TL;DR

Chart understanding tests VLMs’ ability to generalize beyond rigid strategies, including fixed code-based reasoning. This paper learns whether each chart-question pair is better handled with code or direct visual analysis, using reinforcement learning with dual rewards. The adaptive model consistently outperforms pure visual baselines and rigid code-based methods across diverse benchmarks.

  • Problem

    VLMs show a stark performance decline on complex real-world charts, while tool-based and single-mode approaches constrain reasoning strategy.

    Source: §1 · pp. 1–2

  • Method

    The framework learns Visual Programmability for each chart-question pair and uses dual-reward reinforcement learning to select code-based or direct visual reasoning.

    Source: §1 · pp. 2–3

  • Results

    Across diverse benchmarks, the adaptive model consistently outperforms pure visual baselines and rigid code-based methods while varying code use with its advantage.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    Robust chart reasoning depends on recognizing when symbolic code is suitable and when holistic visual analysis is needed.

    Source: §6 · p. 12

    Code-based reasoning can be actively harmful on low-programmability charts, such as scientific plots whose meaning relies on holistic patterns.

    Source: §6 · p. 12

Abstract

from arXiv · show

Chart understanding presents a critical test to the reasoning capabilities of Vision-Language Models (VLMs). Prior approaches face critical limitations: some rely on external tools, making them brittle and constrained by a predefined toolkit, while others fine-tune specialist models that often adopt a single reasoning strategy, such as text-based chain-of-thought (CoT). The intermediate steps of text-based reasoning are difficult to verify, which complicates the use of reinforcement-learning signals that reward factual accuracy. To address this, we propose a Code-as-Thought (CaT) approach to represent the visual information of a chart in a verifiable, symbolic format. Our key insight is that this strategy must be adaptive: a fixed, code-only implementation consistently fails on complex charts where symbolic representation is unsuitable. This finding leads us to introduce Visual Programmability: a learnable property that determines if a chart-question pair is better solved with code or direct visual analysis. We implement this concept in an adaptive framework where a VLM learns to choose between the CaT pathway and a direct visual reasoning pathway. The selection policy of the model is trained with reinforcement learning using a novel dual-reward system. This system combines a data-accuracy reward to ground the model in facts and prevent numerical hallucination, with a decision reward that teaches the model when to use each strategy, preventing it from defaulting to a single reasoning mode. Experiments demonstrate strong and robust performance across diverse chart-understanding benchmarks. Our work shows that VLMs can be taught not only to reason but also how to reason, dynamically selecting the optimal reasoning pathway for each task.

1 Introduction

The introduction argues that chart understanding exposes a generalization gap for rigid reasoning strategies and proposes learning when to use code or direct visual analysis. Its adaptive framework combines this choice with dual-reward reinforcement learning and outperforms fixed strategies across diverse benchmarks.

1 Introduction

  • Tool-using models are constrained by predefined toolkits, while specialist models’ single reasoning modes hinder generalization to out-of-distribution visualizations.The introduction argues that no single reasoning style is optimal for every chart type.Source: §1 · pp. 1–2
  • A dual-reward RL system pairs data-accuracy feedback to prevent numerical hallucination with a decision reward that discourages mode collapse.The decision reward teaches the model the boundaries of programmability and promotes strategic flexibility.Source: §1 · pp. 2–3
  • Visual Programmability is a learnable, task-dependent property used to select code-based reasoning or direct visual analysis for a chart-question pair.The approach responds to the observation that some charts are programmable while others require holistic visual analysis.Source: §1 · pp. 2–3
  • Across diverse benchmarks, the adaptive model consistently outperforms pure visual baselines and rigid code-based methods by switching reasoning modes.It uses code-based reasoning more than 60% of the time where advantageous and less than 10% where detrimental.Source: §1 · pp. 2–3

2 Related Work

Related work covers programmatic chart reasoning and adaptive computation, then positions this paper’s contribution as learning which reasoning process to use. It applies reinforcement learning with a dual reward to encourage both task correctness and strategic diversity.

2.1 Programmatic Reasoning for Chart Understanding

  • The paper frames the generalization gap in chart understanding as stemming from strategic rigidity, despite progress in models and benchmarks.Its related-work discussion connects benchmark-specific shortcuts with performance drops on newer, more complex benchmarks.Source: §2.1 · p. 3
  • Prior programmatic chart-understanding methods include external tools, generated symbolic code, specialized modules, and visually grounded reasoning, but still use fixed strategies.This work shifts the focus from augmenting VLMs with tools to teaching them when to deploy code-like reasoning.Source: §2.1 · p. 3

2.2 Adaptive Learning to Strategic Cognition

  • The paper applies adaptive computation to the choice of reasoning process—holistic visual analysis or formal code-based reasoning—rather than computational pathways.It terms this higher-level adaptation strategic cognition.Source: §2.2 · p. 3
  • Accuracy-only reinforcement learning can cause mode collapse, so the proposed dual reward adds a decision signal that encourages strategic diversity.The paper uses RL to optimize verifiable task correctness while teaching the model to manage its reasoning strategies.Source: §2.2 · pp. 3–4

3 Exploring Code-as-Thought as a Universal Strategy

The fixed Code-as-Thought specialist performs very differently on structured and complex charts, showing that code is not a universal strategy. Even reinforcement learning does not resolve the difficulty on complex charts, motivating task-dependent strategy selection.

3 Exploring Code-as-Thought as a Universal Strategy

  • The specialist was trained on structured data and evaluated across four benchmarks; Figure 2 illustrates the trade-off using ChartX and CharXiv.Full results across all four benchmarks are reported in Appendix A.Source: §3 · p. 4, Figure 2 · p. 4
  • 71.6% with SFT on ChartX contrasts with 18.4% accuracy on CharXiv, showing fixed CaT’s sharp performance drop from structured to complex charts.Figure 2 presents the fixed-strategy trade-off on these structured and in-the-wild benchmarks.Source: Figure 2 · p. 4, §3 · p. 4
  • Reinforcement learning and extensive pre-training plus RL leave CharXiv performance critically low, indicating the issue is strategy applicability, not coding competence.The passage identifies this task-dependent trade-off as motivation for learning when to use code.Source: §3 · p. 4

4 Adaptive Code-Based Reasoning Framework

The framework makes reasoning strategy adaptive to a chart-question pair’s Visual Programmability, then trains a VLM to select between code-based and direct visual reasoning. Reinforcement learning supplies outcome-based training signals for strategy selection and answer generation.

4 Adaptive Code-Based Reasoning Framework

  • The framework combines Visual Programmability, adaptive inference, and reinforcement-learning training to select a reasoning path for each chart task.At inference, the VLM selects a pathway based on its perceived Visual Programmability.Source: §4 · p. 5, Figure 3 · p. 5

4.1 Visual Programmability: Understanding the Boundaries of Code

  • Visual Programmability is a task-dependent range of code suitability shaped by chart clarity, visual complexity, and the query.Standard charts with clear data points suit code; charts whose meaning depends on holistic patterns may lose information in symbolic translation. The same chart can be highly programmable for counting series but poorly suited to extracting a value from a logarithmic scale.Source: §4.1 · pp. 5–6

4.2 Adaptive Reasoning Mechanism

  • The model first selects a strategy token, then generates reasoning and an answer through either the Code-as-Thought or direct visual pathway.The code path parses a chart into a structured representation for computation; the direct path uses natural-language reasoning based on holistic visual perception. The framework also requires final answers in \boxed{} for automated evaluation.Source: §4.2 · p. 6

4.3 Training via Reinforcement Learning

  • Because ground-truth strategy labels are unavailable, reinforcement learning trains the model’s selection policy from outcome-based reward signals.Source: §4.3 · p. 7

4.3.1 GRPO Policy Update

  • GRPO updates the policy using group-relative rewards, and the final configuration sets β = 0 to remove the reference-policy KL penalty.Source: §4.3.1 · p. 7

4.3.2 Comprehensive Reward Function

  • The weighted reward combines answer correctness, strategy choice, code-path data fidelity, and output formatting to promote accuracy and strategic adaptation.The decision reward gives partial credit for a wrong answer reached with the correct strategy, while the data-accuracy reward compares the generated DataFrame with a ground-truth table; the format reward enforces \boxed{} output.Source: §4.3.2 · p. 7

5 Experiments

Experiments show that the adaptive framework outperforms fixed-strategy baselines and state-of-the-art models on average, while varying code use across benchmarks. Ablations and qualitative cases support the role of reward components and task-dependent strategy selection.

5.1 Experimental Setup

  • Four benchmarks span high- to low-programmability charts, while training uses ChartMimic-derived chart-question-answer pairs and GRPO with a multi-component reward.The evaluation includes ChartX, ChartBench, ChartQA, and CharXiv; the training set is balanced between code-suitable and direct-visual tasks.Source: §5.1 · pp. 7–8

5.2 Comparison with Fixed-Strategy Baselines

  • 62.8% average accuracy gives the adaptive framework the highest result against fixed-strategy baselines and state-of-the-art models.Its code use is 76.0% on ChartX and 66.6% on ChartBench, but 10.1% on CharXiv, where it achieves 44.3% accuracy.Source: Table 1 · p. 8, §5.2 · p. 9, §5.3 · p. 9

5.3 Comparison with State-of-the-Art Models

  • The adaptive model achieves the highest average accuracy (62.8%) among the compared state-of-the-art models.The reported comparison uses a stringent evaluation protocol unless otherwise noted.Source: §5.3 · p. 9, Table 3 · p. 9

5.4 Analysis on Different Model Scales

  • At 32B, the adaptive framework achieves the highest average accuracy (61.0%) and top performance on ChartX and CharXiv; at 3B, fixed Code CoT leads at 56.5%.After RL, Standard CoT falls from 31.9% to 20.4%, while both code-based strategies gain substantially.Source: §5.4 · pp. 9–10, Table 4 · p. 10

5.5.1 Dissecting the Reward Function

  • Without the decision reward, code use collapses to either 0% or 100%, with a 4.2-point CharXiv drop; without data accuracy reward, ChartX code use falls to 50.4%.The full reward function encourages a balanced policy and yields the best overall performance.Source: §5.5.1 · p. 10, Table 5 · p. 10, Table 6 · p. 10

5.5.2 The Critical Role of Numerical Fidelity

  • 85.6% answer accuracy on high-fidelity extractions supports the reported correlation between extracted-data fidelity and final-answer correctness on ChartX.During training, accuracy rises on high-data-fidelity tasks and trends downward on low-fidelity tasks as the model unlearns guessing.Source: §5.5.2 · p. 11, Table 7 · p. 11, Figure 6 · p. 11

5.6 Qualitative Analysis: Knowing When to Code

  • On ChartX, the adaptive model uses code to compute a stacked-area-chart answer that visual estimation misses; on CharXiv, it chooses direct reasoning for a qualitative comparison.The fixed code-based model hallucinates a data table on the CharXiv case.Source: §5.6 · p. 11

6 Discussion and Conclusion

The discussion frames Visual Programmability as a boundary on when chart information can be faithfully represented in code. The conclusion presents adaptive selection between code and direct visual reasoning as the central lesson.

6 Discussion and Conclusion

  • Code-as-Thought helps when charts are structurally transparent and tasks require numerical precision, but can harm performance when holistic patterns resist symbolic decomposition.The discussion identifies scientific plots as an example of low-programmability charts.Source: §6 · p. 12
  • Visual Programmability describes whether a chart’s essential information can be faithfully translated into a programmatic format.The property marks the boundary between tasks suited to code and those better handled through holistic visual analysis.Source: §6 · p. 12
  • The conclusion argues that robust general-purpose reasoning depends on selecting between Code-as-Thought and direct visual reasoning rather than relying on one monolithic strategy.The framework deploys code for structured tasks and visual reasoning for ambiguous ones.Source: §6 · p. 12

A Detailed Analysis of Fixed-Strategy Experiments

The fixed-strategy experiments test a ChartX-trained specialist across benchmarks with varied chart styles and difficulty. They find strong structured-chart performance but poor CharXiv generalization, including a case where flawed code supports a wrong answer.

A Detailed Analysis of Fixed-Strategy Experiments

  • The evaluation contrasts in-domain ChartX, near-domain ChartBench, and far-domain ChartQA and CharXiv to test generalization from structured training charts.Each test suite contains 500 samples; ChartX training uses approximately 4,800 structured charts.Source: A Detailed Analysis of Fixed-Strategy… · p. 17
  • 71.6% on ChartX falls to 18.4% on CharXiv for the code-based specialist; RL and CPT plus RL do not resolve the fixed-strategy conflict.The specialist is fine-tuned on ChartX validation data, and the passage describes its strategy as brittle when generalized to complex charts.Source: A Detailed Analysis of Fixed-Strategy… · p. 17, Table 8 · p. 17

B Case Study: Failure due to Numerical Hallucination

  • In the CharXiv case, flawed code constructs an incorrect chart, which the model then reasons over to answer 8 instead of the ground truth 26.The failure follows a misperception of the original chart and a logically consistent analysis of the model’s erroneous reconstruction.Source: B Case Study: Failure due to Numerical… · p. 17, Figure 7 · p. 18

C Data Accuracy Reward Implementation

The data accuracy reward evaluates whether generated code faithfully extracts chart data, while Visual Programmability labels are grounded in expert judgments about information preservation and reconstruction. The resulting binary supervision is intended to teach flexible strategy selection.

C Data Accuracy Reward Implementation

  • The data accuracy reward compares the code-generated DataFrame with a ground-truth CSV to assess faithful chart-data extraction.The implementation uses a relative tolerance of 10^-2 for numerical comparisons, normalized case-insensitive text matching, and matching NaN values.Source: C Data Accuracy Reward Implementation · p. 18

D.1 Guiding Principle

  • Annotators assess whether a code representation preserves the essential information needed to answer each chart-question pair.The labeling decision is task-aware because it depends on both chart properties and the specific query.Source: §D.1 · p. 18, D Annotation Framework for Visual… · p. 18

D.2 Assessment Criteria

  • The assessment checks whether chart data can be reliably extracted and whether the code representation retains answer-relevant information; lost visual meaning indicates low programmability.Annotators also mentally compile a reconstruction using a standard plotting library and treat uncaptured key visual elements or context as a low-programmability signal.Source: §D.2 · pp. 18–19

D.3 Annotation Process and Quality Control

  • Instances receive a binary high- or low-programmability label, with ambiguous cases handled conservatively to preserve visual information.The process includes rulebook guidance and regular review of annotation batches.Source: §D.3 · pp. 19–20
  • The human-centered annotations provide supervision intended to help the model learn a flexible policy beyond rigid, rule-based systems.The paper links this supervisory signal to the nuances of human cognition.Source: §D.3 · p. 20

E Detailed Prompt Specifications

Appendix E presents prompts for synthetic question-answer generation, baseline reasoning, and adaptive strategy selection. Its instructions specify when to use code or direct analysis and how to format responses.

E Detailed Prompt Specifications

  • The prompt appendix identifies separate synthetic-QA, direct-CoT, mandatory-code, and adaptive-model prompts.The master prompt requires an explicit <CODE> or <DIRECT> choice, followed by an answer in the prescribed format.Source: E Detailed Prompt Specifications · pp. 20, 22, 23
  • The synthetic-question prompt requests one concise chart question and answer, covering numerical operations, recognition, comparison, and temporal-analysis tasks.It specifies brief answers and a JSON question-answer output format.Source: E Detailed Prompt Specifications · pp. 20–21
  • The code-based baseline requires chart redrawing with runnable Python and a chart_data DataFrame before step-by-step reasoning.The baseline CoT prompt instead asks for visual examination and step-by-step reasoning without requiring code.Source: E Detailed Prompt Specifications · p. 21
  • The adaptive prompt chooses between code and direct analysis, preferring code for extractable data and precision tasks while reserving direct analysis for unsuitable charts or qualitative questions.It asks for a pandas DataFrame named chart_data before coding and requires responses to end with a boxed answer.Source: E Detailed Prompt Specifications · pp. 21–22

F Broader Implications and Future Directions

The discussion frames flexible strategy selection as a broader AI challenge and identifies extensions to the framework, including hybrid reasoning, broader formal reasoning, and self-supervised policy learning. It also describes current boundaries, including binary strategy selection and chart-level programmability assessment.

The Path Forward

  • The discussion presents strategy flexibility as a broader AI challenge and argues that systems should match reasoning style and computational effort to problem structure.It positions the chart-focused framework as a concrete instantiation of this direction.Source: F Broader Implications and Future… · pp. 23–24

From Modality Fusion to Method Fusion

  • The paper presents method fusion as selecting and combining visual-perceptual and symbolic-programmatic strategies within one modality.It contrasts holistic pattern-based analysis with precise, step-by-step decomposition.Source: F Broader Implications and Future… · p. 23

Competence Awareness as a Foundational Capability

  • The authors describe competence awareness as recognizing which available method is most likely to succeed and identify richer capability models as a future need.The discussion connects this ability to more dependable strategy selection on novel tasks.Source: F Broader Implications and Future… · p. 23

Granular and Hybrid Reasoning

  • The current decision process is binary and chart-level, limiting it to code or direct reasoning rather than hybrid or region-specific strategy use.The proposed extension would combine code-based data extraction with visual interpretation and assess programmability by chart region.Source: F Broader Implications and Future… · p. 23

Expanding the Vocabulary of Formal Reasoning

  • Code-as-Thought currently centers on data-analysis logic; future formal reasoning could add signal-processing or graph-theoretic formalisms.These examples target time-series charts and network diagrams, respectively.Source: F Broader Implications and Future… · p. 23

Self-Supervised Policy Learning

  • Reducing reliance on annotated programmability labels is identified as a challenge, with self-supervised learning proposed to connect strategy choice to task success.The proposed direction aims to learn the decision boundary without explicit human-provided labels.Source: F Broader Implications and Future… · pp. 23–24

Towards Dynamic Strategy Composition

  • The framework currently makes discrete selections among predefined strategies; future systems could compose primitive operations into novel approaches.The proposed example combines graph-based structural analysis with time-series forecasting for temporal patterns.Source: F Broader Implications and Future… · p. 24

G Detailed Case Studies

Appendix G supplies visualizations and verbatim baseline and adaptive-model outputs for qualitative case studies. Its two figure captions identify a high-programmability ChartX case and a low-programmability CharXiv case.

G Detailed Case Studies

  • Appendix G presents each qualitative case with a figure, task details, and verbatim baseline and adaptive-model outputs.Source: G Detailed Case Studies · p. 24
  • Figure 8 identifies Case Study 1 as a high-programmability chart from ChartX.Source: Figure 8 · p. 25
  • Figure 9 identifies Case Study 2 as a low-programmability chart from CharXiv.Source: Figure 9 · p. 26

H Implementation and Hyperparameter Details

The implementation used the EasyR1 framework for reinforcement learning and trained Qwen2.5-VL-7B for 200 episodes while keeping its vision tower frozen. Table 9 is identified as the source of configuration details.

H Implementation and Hyperparameter Details

Loading 2509.09286v1…