PaperlayerView exact revision on arXiv

Source-linked AI summary

CURV: Enhancing Chart Understanding Through Curriculum Visual Grounded Reasoning

Xuehang Guo, Pingyue Zhang, Ruiyi Zhang, Zhenhailong Wang, Hanrui Lyu, Heng Ji, Tong Sun, Qingyun Wang, Manling Li

arXiv:2608.02833v1cs.CVcs.AIcs.CL

TL;DR

Existing MLLMs struggle to connect coherent stepwise chart reasoning with accurate visual grounding. CURV trains multi-step reasoning coupled with dynamic chart-region focus and reports improvements across chart understanding and out-of-domain multimodal reasoning, including up to ↑10.20% accuracy improvements in the latter.

  • Problem

    Existing MLLMs struggle to decompose chart questions into coherent reasoning steps, ground each step in chart regions, and integrate grounded reasoning across steps.

    Source: §1 · p. 2

  • Method

    CURV develops intrinsic visual-grounded reasoning through progressively harder multi-step chart question answering, coupling logical reasoning with dynamic chart-region focus and using a three-level curriculum dataset.

    Source: §1 · p. 2, §6 · p. 10

  • Results

    Up to ↑10.20% accuracy improvements in out-of-domain multimodal reasoning accompany consistent improvements across chart understanding and out-of-domain reasoning benchmarks.

    Source: §5.3 · p. 9, §6 · p. 10

  • Takeaways & Limitations

    CURV establishes a foundation for developing self-contained visual reasoning capabilities in MLLMs beyond extrinsic assistance.

    Source: §6 · p. 10

    CCQA covers seven common chart types, and the authors identify less conventional visualization forms such as Sankey charts as a limitation.

    Source: I Limitations & Future Work · p. 52

Abstract

from arXiv · show

Chart question answering (CQA) requires multimodal large language models (MLLMs) to integrate visual comprehension with logical reasoning, yet current models struggle with accurate visual grounding and coherent reasoning chains. While extrinsic chain-of-thought prompting and visual cues significantly improve performance, current MLLMs lack intrinsic visual grounded reasoning capabilities, leading to inaccurate perception and reasoning disconnected from visual evidence. To address these limitations, we propose CURV, a curriculum learning framework that develops intrinsic visual reasoning capabilities by reformulating CQA as multi-step visual grounded reasoning, where each step coordinates logical reasoning with dynamic visual grounding through spatial attention concentration. To assist model learning, we further introduce CCQA, a three-level curriculum dataset with scalable synthetic generation across diverse chart types and reasoning patterns. Our curriculum systematically progresses from basic single-operation reasoning to complex multi-chart compositional tasks. Experiments demonstrate that CURV achieves up to $\uparrow20.50\%$ improvements over baselines and is generalizable to real-world benchmarks (up to $\uparrow12.30\%$) and out-of-domain multimodal reasoning tasks (up to $\uparrow10.20\%$), validating the effectiveness of internalizing visual reasoning with dynamic grounding for enhanced chart understanding capabilities. Code is available at: https://xhguo7.github.io/CURV/.

1 Introduction

CURV targets chart question answering failures by training models to interleave stepwise reasoning with dynamic visual grounding. Its curriculum progresses from single-operation tasks to complex compositions, supported by scalable synthetic CQA data and a three-level curriculum dataset.

1 Introduction

  • Existing MLLMs struggle to decompose chart questions, ground reasoning steps in chart regions, and compose perception and reasoning into coherent chains.These limitations can cause errors even when the necessary information is present in the chart.Source: §1 · p. 2
  • CURV reformulates chart question answering as multi-step reasoning that couples each logical step with dynamic visual grounding, progressively shifting attention from relevant chart regions.Its curriculum advances from single-operation reasoning to complex multi-operation compositions to develop visual perception accuracy and reasoning sophistication.Source: §1 · p. 2
  • The authors introduce scalable synthetic CQA generation and CCQA, a curriculum dataset spanning four generation modes, three visual grounding strategies, and multi-level evaluation metrics.The synthetic generation method increases task complexity through nested reasoning chains.Source: §1 · p. 2
  • The authors report that CoT reasoning with visual grounding provides models step-by-step alignment between visual perception and logical reasoning, yielding notable improvements across different chart types, complexity levels, and domains.Source: §1 · p. 2

2 What Prohibits MLLMs From Chart Understanding Success?

GPT-4o improves with reasoning and visual guidance, and combining them performs best; tests across two models and three reasoning depths show that both supports together further improve chart understanding.

2.1 Preliminary Analysis of CQA Failures

  • GPT-4o’s reasoning-only mode improves scores by 10.00%, while combining reasoning and visual cues yields the best performance, with a 43.33% improvement.The four modes are answer only, vision plus answer, reasoning plus answer, and reasoning plus vision plus answer.Source: Table 1 · p. 3, §2.1 · p. 3

2.2 Enhancing Chart Reasoning Through Dynamic Visual Grounding

  • Across GPT-4o and Qwen2.5-VL-7B, reasoning or visual grounding improves performance, and combining both improves it further across curriculum levels 1–3.The evaluation uses 1,800 CCQA samples evenly distributed across the three reasoning depths.Source: §2.2 · p. 3, Figure 2 · p. 3

3 CURV: Chart Reasoning with Dynamic Visual Grounding

CURV reframes chart question answering as progressive reasoning grounded in visual regions, and trains models to update that grounding as reasoning unfolds. Its curriculum increases difficulty across both reasoning and chart structure.

3.1 Problem Formulation

  • CURV replaces direct image-question-to-answer mapping with a progressive chain linking each natural-language reasoning step to visual regions.The paper identifies the direct mapping as lacking intermediate structure for accurate perception and dynamic grounding.Source: §3.1 · pp. 3–4

3.2 Multi-Step Reasoning With Dynamic Visual Grounding

  • CURV trains grounding first, then interleaves visual feedback with reasoning so the model can shift attention and keep reasoning aligned with evidence across steps.Stage I learns visual focuses for reasoning steps; Stage II feeds the grounded image state into the next step and jointly supervises reasoning, grounding, and the final answer.Source: §3.2 · pp. 4–5

3.3 Curriculum Visual Grounded Reasoning

  • CURV supports applied, boxed, and cropped grounding strategies, as well as SFT, reinforcement learning, or combined training paradigms.The framework uses explicit grounding to develop intrinsic visual grounded reasoning and examines implicit grounding in ablations.Source: §3.3 · p. 5
  • The curriculum distinguishes reasoning-step count from logical depth, then progresses across three data levels and five tiers while increasing reasoning and visual difficulty.It also moves from low-level chart components toward high-level chart structures across diverse chart types.Source: §3.3 · p. 5, Figure 4 · p. 5

4 CCQA: Curriculum Chart Question Answering

CCQA supports CURV with a curriculum dataset designed to progressively develop visual reasoning capabilities. Its construction combines increasing task complexity with step-by-step visual grounding and extends across chart types, domains, and reasoning patterns.

4.1 Dataset Construction Principles

  • CCQA uses three curriculum levels that vary reasoning depth, chart complexity, and operation sophistication.Source: §4.1 · p. 6
  • Each reasoning step is paired with corresponding ground-truth visual evidence and binary masks to align reasoning with visual grounding.Source: §4.1 · p. 6
  • Synthetic templates support data accuracy and systematic coverage of reasoning patterns, helping maintain stable curriculum progression.Source: §4.1 · p. 6

4.2 Curriculum Chart Question Answering

  • CCQA spans seven chart types, 30 domain categories, and atomic or nested QA operations across chart components and subplots.Source: §4.2 · p. 6
  • The meta-learning design derives diverse query-reason-ground-answer instances from only 30 unique charts per chart type to encourage transferable reasoning rather than visual memorization.It systematically instantiates multiple CQA instances for each base image across appearances, data distributions, contexts, domains, and task complexities.Source: §4.2 · p. 6
  • CURV finetuned on CCQA is also applicable to real-world chart understanding and out-of-domain benchmarks.Source: §4.2 · p. 7

5 Experiments

CURV improves chart-question-answering performance and generalizes to chart and out-of-domain tasks. Ablations highlight explicit visual grounding and foundational curriculum learning as useful for stronger chart reasoning.

5.1 Setup

  • CURV fine-tunes Qwen2.5-VL (3B, 7B) and InternVL-3 (1B, 2B, 8B) on CCQA, with evaluation on test sets unseen during fine-tuning.Baselines include close-source and open-source MLLMs.Source: §5.1 · p. 7

5.2 Evaluation

  • Evaluation spans three CCQA curriculum levels, four chart benchmarks, and MathVista and MMMU-Pro, using answer-accuracy and visual-grounding measures.Answer accuracy includes MLLM-as-judge and rule-based range metrics; visual grounding uses CIOU and GIOU.Source: §5.2 · pp. 7–8

5.3 Main Results On Chart & Multimodal Understanding

  • CURV improves by ≥↑1.20% across four chart benchmarks and achieves up to ↑10.20% accuracy improvement on out-of-domain multimodal reasoning tasks.The chart-benchmark evaluation covers ChartMuseum, CharXiv, ChartQA, and ChartQAPro; out-of-domain tasks include MathVista and MMMU-Pro.Source: Table 3 · p. 8, §5.3 · p. 9
  • CURV gains up to 14.85% and 20.92% over baselines with single-stage and two-stage training, respectively; CURV@Applied (Qwen2.5-VL-7B) gains up to 15.65% over its base model and 12.22% over GPT models.The reported gains are across six CCQA metrics.Source: §5.3 · p. 8
  • Despite training only on single-plot charts, CURV@Applied improves multi-plot performance by up to ↑7.10% (Qwen2.5-VL-7B) and ↑6.73% (InternVL-8B) across metrics.CURV also transfers across training paradigms: reinforcement learning yields up to 12.58% improvement and SFT combined with RL yields 17.04%, while RL incurs substantially higher computational overhead than SFT.Source: §5.3 · pp. 8–9

5.4 Ablations on Visual Grounded Reasoning

  • Explicit visual grounded reasoning outperforms implicit reasoning by ↑8.78%; applied masking is more beneficial than boxed grounding, while cropped grounding gains ↑7.93% on acc@MLLM and ↑9.92% on acc@0.0.The crop gains occur despite a sixteen-fold reduction in resolution; zoom-in resolution trades off against computation overhead.Source: §5.4 · p. 9, Figure 7 · p. 9
  • Curriculum learning improves over standard training but remains below CURV; using levels 1+2 yields the most consistent gains, while level 3 alone is less effective.Foundational learning supports adaptive reasoning across difficulty levels.Source: §5.4 · pp. 9–10, Figure 9 · p. 10
  • Bar charts gain most and radar charts least; CURV also narrows the single- versus multi-chart accuracy gap by improving localization and relational understanding.Single-chart levels reach up to 80.43% accuracy, compared with at most 42.54% on multi-chart level 3.Source: §5.4 · pp. 9–10, Figure 8 · p. 9, Figure 10 · p. 10

6 Conclusion … A.2 Preliminary Exploration on CQA Challenges

CURV develops intrinsic visual reasoning through progressive, multi-step reasoning interleaved with visual grounding, supported by the three-level CCQA curriculum. Preliminary analyses identify perception, reasoning consistency, and linking visual attention to logic as persistent CQA challenges.

6 Conclusion

  • CURV uses progressive multi-step visual grounded reasoning, supported by three-level CCQA, and improves performance across curriculum levels while generalizing to chart and out-of-domain multimodal reasoning.The framework interleaves reasoning with visual grounding to develop intrinsic visual reasoning capabilities.Source: §6 · p. 10

A.1 Preliminary Exploration On Motivations

  • 28.33% of GPT-4o’s failures on 60 CharXiv samples were perception errors, compared with 10.00% reasoning errors and 8.33% integration errors.The analysis identifies difficulty grounding reasoning steps in fine-grained chart details as the dominant failure category.Source: §A.1 · p. 18, Figure 11 · p. 18
  • The preliminary design decomposes charts into low-level components and problems into reasoning steps, then interleaves reasoning with shifting visual focus and composes the results into coherent understanding.It draws on the human-inspired sequence of decomposition, interleaved visual reasoning, and composition.Source: §A.1 · p. 18
  • CURV’s curriculum increases reasoning difficulty through more nested functions and chart complexity through more components, chart types, and subplots; meta-learning supports adaptability and generalizability.Source: §A.1 · p. 18

A.2 Preliminary Exploration on CQA Challenges

  • MLLMs can misread fine-grained chart values: GPT-4o estimates the red “fi” point’s WH value at about 0.105 although its true value is below 0.1 and just above 0.0.Source: §A.2 · p. 19
  • For complex reasoning tasks integrating multiple visual regions and reasoning steps, both models perceive accurately initially, but GPT-4o incorrectly includes “Roar” and GPT-4o-mini fails to incorporate “Loki” in subsequent reasoning.Source: §A.2 · p. 19

A.3 Preliminary Validation on CCQA · B Related Work

CCQA case studies show that failures increase as reasoning complexity grows, while related work often leaves visual evidence loosely connected to reasoning. CURV instead treats visual grounding as an iterative part of multimodal reasoning.

A.3 Preliminary Validation on CCQA

  • At Tiers 1–2, Qwen2.5-VL-3B and GPT-4o-mini misperceive chart numbers, Qwen2.5-VL-7B miscalculates, and all but Gemini-2.5-Flash struggle to identify components.The examples cover a Statistics-Mean query at Tier 1 and a Subset query at Tier 2.Source: §A.3 · p. 19, Figure 14 · p. 22, Figure 15 · p. 23
  • At Tier 3, all but GPT-4o struggle to calculate the mean; at Tiers 4–5, all five models' visual reasoning fails on multi-plot tasks involving localization or subplot relations.Tier 4 combines localization with Statistics-Sum; Tier 5 additionally requires attending to relations across subplots.Source: §A.3 · p. 19, Figure 16 · p. 24, Figure 17 · p. 25
  • Across five MLLMs and five CCQA difficulty tiers, failures increase and successes decrease with complexity, showing curriculum progression while exposing persistent chart-understanding challenges.The validation uses the same five MLLMs as the preliminary exploration.Source: §A.3 · p. 19

B Related Work

C Dataset Construction … F.1 Evaluation of Answers

This section describes CCQA’s chart data, progressive reasoning curriculum, quality controls, visual-grounding strategies, and generation modes, along with answer-evaluation methods that apply type-specific matching rules across accuracy thresholds.

C Dataset Construction

  • CCQA spans seven chart types and 30 domains, with chart-specific augmentations diversifying visual layouts and appearances.GPT-4o generates plotting data, while rendering functions create charts; augmentations vary rotation, orientation, axes, colors, legends, labels, and chart-specific features.Source: C Dataset Construction · p. 27, §C.1 · p. 29

C.2 Multi-Level Curriculum

  • The three-level curriculum progresses from single-operation reasoning on one chart to nested operations and complex multi-chart reasoning with step-specific visual grounding.Level 1 covers direct reading, simple arithmetic, and elementary comparisons; Level 2 composes operations sequentially on one chart; Level 3 adds cross-chart dependencies, localization, and relations.Source: §C.2 · pp. 30–31

C.3 Fine-Grained Curriculum Tiers

  • Five tiers refine task difficulty by reasoning depth and chart complexity, culminating in multi-chart localization and relations across identified subplots.Tiers 1–3 use single plots with one, two, or at least three nested-function depths; Tiers 4–5 use multi-plot charts at depth three or more, with Tier 5 adding cross-subplot relations to localization.Source: §C.3 · pp. 31–32

C.4 Meta-Learning Supported Curriculum Learning

  • Meta-learning structures CCQA around domain, chart-type, and operator variation, using task decomposition and controlled complexity to support transfer across domains, chart types, and reasoning depths.Thirty domains and seven chart types yield 210 chart-structure metadata combinations; 12 operators serve as compositional primitives, while visual and reasoning decomposition scaffold learning.Source: §C.4 · p. 32

C.5 Quality Control

  • Quality control combines executable consistency checks, manual validation of visual-grounding and reasoning annotations, and constrained LLM use to protect answer reliability.Reviewers verify 30 meta images per chart type and randomly sample 10 instances per each of 30 categories across seven chart types (10 × 30 × 7); GPT-4o supplies randomized numerical parameters, not ground-truth answers.Source: §C.5 · p. 33

D Visual Grounding Strategies

  • Three dynamic grounding strategies shift visual focus during reasoning: applied highlighting, boxed boundaries, and cropped zooms, each trading off clarity, efficiency, and precision.Applied grounding uses semi-transparent yellow overlays while retaining context; boxed grounding relocates and resizes red outlines; cropped grounding presents focused regions alongside the full chart.Source: D Visual Grounding Strategies · p. 33, §D.1 · p. 33, §D.2 · p. 34

E Generation Modes

  • The four generation modes vary whether the model outputs only an answer or first produces visual grounding, reasoning, or both.Mode A gives the answer directly; VA grounds before answering; RA reasons before answering in one turn; RVA produces reasoning with visual grounding before the final answer.Source: §E.1 · p. 37, §E.2 · p. 37, §E.3 · p. 38

F.1 Evaluation of Answers

  • Answer evaluation averages accuracy across test samples, combining semantic GPT-4.1-mini judging with rule-based checks at acc@0.0, acc@0.05, acc@0.1, and acc@0.2.The judge assesses semantic equivalence to the ground truth, while predefined parsing and judgment rules provide absolute accuracy and three progressively relaxed tolerance thresholds.Source: §F.1 · pp. 40–41
  • Empty ground-truth and prediction pairs receive 1.0 at every threshold; multiple-choice answers are scored by the fraction of ground-truth choices matched.A choice list matching exactly receives 1.0 at every threshold; otherwise each threshold checks whether the match rate reaches that threshold.Source: §F.1 · p. 41
  • Numeric answers receive exact-value scoring at 0.0 tolerance and are checked within ±5%, ±10%, and ±20% of the ground truth.The predicted number is extracted from the response, then tested against the ground-truth-centered interval for each relative tolerance.Source: §F.1 · p. 41
  • Text answers are scored using case-insensitive exact matching or, for ground truths longer than five characters, substring matching.The score is the maximum of the exact-match and substring-match results, applied across all thresholds.Source: §F.1 · p. 41

F.2 Evaluation of Reasoning … G.1 Implementation Details.

The paper evaluates reasoning with semantic-similarity metrics and judge-based quality scores, assesses visual grounding with two IoU variants, and compares four generation modes. Implementation details specify staged supervision and LoRA-based training.

F.2 Evaluation of Reasoning

  • The micro-level final score, acc@mic, averages the five reasoning-similarity metrics.Source: §F.2 · p. 41
  • Five metrics—ROUGE-L, BLEU, METEOR, BERTSCORE, and cosine similarity—measure semantic similarity between predicted and ground-truth reasoning.Source: §F.2 · p. 43
  • GPT-4.1-mini assigns reasoning-quality scores from 0–10, averaged across test samples and based on grounding, logical coherence and multimodal integration, and ground-truth alignment.The grounding criterion also penalizes hallucinated visual details.Source: §F.2 · p. 43

F.3 Evaluation of Visual Grounding

  • Visual grounding is evaluated using CIOU and GIOU.Source: §F.3 · p. 43

F.4 Evaluation Mode

  • Four generation modes compare answer-only, reasoning-then-answer, vision-grounding-then-answer, and reasoning with dynamic grounding before the answer.Source: §F.4 · p. 44

G.1 Implementation Details.

  • Training uses two stages: visual-grounding-only supervision first, then joint reasoning, grounding, and answer supervision.Models train for 3 epochs with an initial learning rate of 1e-4, a cosine scheduler, a 9:1 train-validation split, and LoRA on two 80G H100 GPUs.Source: §G.1 · p. 44

G.2 Grounding Method & Computation Cost … G.6 Curriculum Learning: Effectiveness of CL in Chart Understanding

CURV uses visual grounding as a training signal to build intrinsic chart reasoning, while curriculum learning helps models handle increasingly complex chart tasks. Explicit grounding at inference can introduce errors that outweigh its benefits.

G.2 Grounding Method & Computation Cost

  • At 128 × 128, cropped visual grounding achieves up to 4.72% improvement on CCQA over the baseline and performs competitively with other grounding methods, while limiting computation cost.At the same resolution, cropped grounding requires substantially more memory; lowering its training resolution mitigates the computational overhead despite degradation in resolution.Source: §G.2 · p. 44

G.3 The Role of Visual Grounding: From Extrinsic Assistance To Intrinsic Abilities

  • Visual grounding supervision during training improves intrinsic visual reasoning, but explicit multi-turn grounding at inference can underperform direct answers or reasoning without grounding because errors accumulate and misalign.Finetuned models nevertheless outperform baselines in direct-answer, reasoning-only, visual-answer, and reasoning-with-visual-grounding modes.Source: §G.3 · pp. 44–45

G.4 Visual Grounded Reasoning via Reinforcement Learning

  • The RL extension adapts CURV’s two-stage training to reward accurate visual grounding, faithful reasoning, correct answers, and structured-format compliance.Its component signals include geometric focus alignment, semantic similarity to reference reasoning, binary exact-match answer reward, and binary format compliance.Source: §G.4 · pp. 45–46

G.5 Explicit vs. Implicit Visual Grounded Reasoning

  • CURV explores explicit and implicit visual grounded reasoning, which differ in how they encourage the MLLM to attend to chart regions across reasoning steps.Source: §G.5 · p. 46

G.5.1 Explicit Visual Grounding

  • In explicit grounding, the MLLM predicts a bounding box for each reasoning step, uses focused chart views as visual input during interleaved reasoning, and jointly learns reasoning, grounding, and answer generation.Training uses ground-truth focus regions as intermediate visual augmentation; the model learns to shift attention alongside reasoning.Source: §G.5.1 · pp. 46–47

G.5.2 Implicit Visual Grounding

  • Implicit grounding uses a separate patch-attention grounder trained while the MLLM is frozen, then freezes the grounder while finetuning the MLLM with grounded feedback; its visual augmentation is less effective than explicit grounding.The grounder uses visual patch embeddings and the decoder state at a <GROUND> token to produce attention maps, without requiring coordinate predictions from the MLLM.Source: §G.5.2 · pp. 47–48

G.6 Curriculum Learning: Effectiveness of CL in Chart Understanding

  • Without curriculum, CCQA performance improves over the untrained baseline but remains below CURV with Stage I or Stage I+II, with the gap widening from levels 1 to 3.Non-curriculum gains diminish as task complexity rises; the reported curriculum benefits also extend to out-of-domain multimodal reasoning.Source: §G.6 · p. 48, Figure 30 · p. 48

G.7 Two-Stage Learning: Strength of Training Curriculum … H Multi-Step Reasoning With Dynamic Visual Grounding

CURV’s two-stage curriculum performs best when foundational visual grounding is followed by interleaved reasoning, while progressive data training balances performance across difficulty levels. Broader comparisons and RVA evaluation cover newer backbones, specialist and tool-using systems, and reasoning, grounding, and answer quality.

G.7 Two-Stage Learning: Strength of Training Curriculum

  • Stage I establishes visual grounding, while Stage II operationalizes it in step-by-step reasoning over visual inputs.The two-stage design moves from decoupled visual grounding to interleaved visual reasoning.Source: §G.7 · pp. 48–49
  • CURV@Stage I+II consistently outperforms both single-stage variants, supporting the combination of decoupled alignment and interleaved reasoning.The comparison uses Qwen2.5-VL-7B as the base model.Source: Figure 31 · p. 48, §G.7 · p. 49

G.9 Broader Baselines: Comparison Against Chart Specialist & Tool-Use Agents

  • ↑16.14% is CURV’s reported maximum improvement over Qwen3-VL-4B, and CURV also outperforms chart specialists and tool-augmented agents using the Qwen2.5-VL-7B backbone.The comparison is reported across CCQA’s three curriculum levels.Source: Table 10 · p. 49, §G.9 · p. 50

G.7 Two-Stage Learning: Strength of Training Curriculum

  • Stage I alone improves level 1 simple CQA but degrades harder single- and multi-chart tasks, whereas Stage II alone yields weaker visual-reasoning alignment.The passages attribute these limitations to insufficient compositional grounding in Stage I alone and the absence of well-established grounding in Stage II alone.Source: §G.7 · p. 49

G.8 Foundational Learning: Trade-off between Robustness & Adaptability

  • ↑15.65% on level 1, ↑11.53% on level 2, and ↑7.10% on level 3 are achieved by progressive training on levels 1+2; adding level 3 raises level 3 to ↑12.82% but leaves levels 1 and 2 at ↑5.93% and ↑0.03%.Training on level 1 alone gives ↑35.08% on level 1; the passage describes the level 1+2 results as the best overall performance across difficulty levels.Source: §G.8 · p. 49

G.9 Broader Baselines: Comparison Against Chart Specialist & Tool-Use Agents

H Multi-Step Reasoning With Dynamic Visual Grounding

  • Table 11 evaluates RVA mode inference across model reasoning, visual grounding, and final-answer metrics.The supplied passage identifies the evaluation scope but provides no metric values or outcomes.Source: Table 11 · p. 50

H.1 Challenges In Multi-Step Visual Grounding · H.2 Inference Failure · H.3 Inference Success

RVA training with explicit reasoning and visual grounding improves intrinsic visual reasoning, but inference can falter when early grounding or reasoning errors accumulate. Examples show CURV succeeding on increasingly complex chart questions and RVA inference succeeding with accurate or near-focus grounding.

H.1 Challenges In Multi-Step Visual Grounding

  • Explicit reasoning and visual grounding as intermediate outputs help models decompose complex tasks into coherent, dynamically grounded reasoning chains.The authors describe this stepwise process as strengthening intrinsic visual reasoning and aligning with human visual reasoning.Source: §H.1 · p. 50
  • At inference, an incorrect calculation or inaccurate visual comprehension can propagate through steps, and RVA answering remains below CURV with the same base model and grounding method.The authors therefore distinguish RVA’s effectiveness for learning from its vulnerability during inference.Source: §H.1 · p. 50
  • Up to 10.23% absolute gain in reasoning, 9.38% in grounding, and 14% in answering were reported for finetuned models in RVA inference evaluation.Qualitative inspection also found more systematic reasoning chains and sharper object localization.Source: §H.1 · p. 50
  • GPT-4o and Qwen2.5-VL-7B both give incorrect answers in an RVA example when incorrect visual grounding disrupts their reasoning chains.Figure 32 illustrates this inference challenge.Source: §H.1 · p. 51, Figure 32 · p. 51

H.2 Inference Failure

  • In both Fig. 35 failure examples, poor chart grounding causes inaccurate extraction or misinterpretation that propagates into incorrect answers.The visual comprehension errors arise early in reasoning.Source: §H.2 · p. 51

H.3 Inference Success

  • In mode A examples, CURV answers correctly where baseline Qwen2.5-VL-7B fails on single- and multi-chart questions requiring localization, nested reasoning, or cross-chart relations.The examples span D = 1 to D = 5, with harder cases adding subset and subplot localization, value reading, extrema comparisons, and relations across charts.Source: §H.3 · pp. 51–52
  • RVA inference succeeds on a bar chart with accurate grounding and on a heatmap with grounding close to, but not exactly at, the focus regions.Both examples lead to correct answers.Source: §H.3 · p. 52

I Limitations & Future Work

The authors identify limited chart-type coverage and an English-only focus as limitations of CURV and CCQA, and plan to extend the dataset to more chart types and multilingual settings. They also propose exploring agentic chart understanding to complement the framework’s intrinsic visual reasoning.

I Limitations & Future Work

  • CCQA covers seven common chart types but omits specialized forms such as Sankey diagrams, treemaps, and geographic maps, which may limit specialized training applications.The authors plan to broaden CCQA’s chart-type coverage.Source: I Limitations & Future Work · p. 52
  • CURV and CCQA focus exclusively on English-language charts and questions, potentially restricting generalizability to multilingual contexts.The authors plan to extend CCQA to multilingual settings.Source: I Limitations & Future Work · p. 52
  • The authors plan to explore agentic chart understanding that uses external knowledge, tools, and multi-agent collaboration alongside intrinsic visual reasoning.Source: I Limitations & Future Work · p. 52

Related papers

Loading 2608.02833v1…