PaperlayerView exact revision on arXiv

Source-linked AI summary

CURV: Enhancing Chart Understanding Through Curriculum Visual Grounded Reasoning

Xuehang Guo, Pingyue Zhang, Ruiyi Zhang, Zhenhailong Wang, Hanrui Lyu, Heng Ji, Tong Sun, Qingyun Wang, Manling Li

arXiv:2608.02833v1cs.CVcs.AIcs.CL

TL;DR

Chart question answering remains difficult because MLLMs struggle to ground coherent multi-step reasoning in visual evidence. CURV addresses this with curriculum-based visual grounded reasoning and achieves consistent gains across chart and out-of-domain multimodal reasoning benchmarks, including up to 20.92% on CCQA.

  • Problem

    Existing MLLMs struggle to decompose chart questions, ground reasoning steps accurately, and compose coherent reasoning chains across visual inputs.

  • Method

    CURV reformulates chart question answering as progressive multi-step reasoning that interleaves logical inference with dynamic visual grounding through curriculum learning.

  • Results

    CURV consistently improves chart understanding and out-of-domain multimodal reasoning, with gains up to 20.92% on CCQA over baselines.

  • Takeaways & Limitations

    The results support developing self-contained visual reasoning capabilities in MLLMs beyond reliance on extrinsic assistance.

  • Takeaways & Limitations

    The authors identify CCQA’s coverage of seven common chart types as a limitation regarding less conventional visualizations such as Sankey diagrams.

Abstract

from arXiv · show

Chart question answering (CQA) requires multimodal large language models (MLLMs) to integrate visual comprehension with logical reasoning, yet current models struggle with accurate visual grounding and coherent reasoning chains. While extrinsic chain-of-thought prompting and visual cues significantly improve performance, current MLLMs lack intrinsic visual grounded reasoning capabilities, leading to inaccurate perception and reasoning disconnected from visual evidence. To address these limitations, we propose CURV, a curriculum learning framework that develops intrinsic visual reasoning capabilities by reformulating CQA as multi-step visual grounded reasoning, where each step coordinates logical reasoning with dynamic visual grounding through spatial attention concentration. To assist model learning, we further introduce CCQA, a three-level curriculum dataset with scalable synthetic generation across diverse chart types and reasoning patterns. Our curriculum systematically progresses from basic single-operation reasoning to complex multi-chart compositional tasks. Experiments demonstrate that CURV achieves up to $\uparrow20.50\%$ improvements over baselines and is generalizable to real-world benchmarks (up to $\uparrow12.30\%$) and out-of-domain multimodal reasoning tasks (up to $\uparrow10.20\%$), validating the effectiveness of internalizing visual reasoning with dynamic grounding for enhanced chart understanding capabilities. Code is available at: https://xhguo7.github.io/CURV/.

1 Introduction

Chart question answering requires models to interpret complex visual structures while performing coherent, multi-step reasoning with dynamically shifting visual attention. CURV addresses current MLLM limitations by reformulating CQA as intrinsically visual-grounded, stepwise reasoning that couples each inference with relevant chart regions.

  • Motivation: Humans solve multimodal problems by decomposing them into stepwise reasoning chains, interleaving each step with dynamic visual grounding, and composing grounded steps into coherent solutions.The introduction connects this process to cognitive theories and notes that chain-of-thought reasoning supports stepwise problem decomposition.
  • Motivation: CQA requires faithful interpretation of complex geometric structures, spatial relationships, and quantitative patterns to derive correct answers.These requirements make chart question answering a particularly demanding multimodal setting.
  • Limitations: Existing MLLMs struggle to decompose complex CQA problems into coherent reasoning chains and to interleave reasoning with accurate visual perception.The introduction attributes these limitations to inconsistent or logically flawed intermediate steps and insufficient coordination between reasoning and visual evidence.
  • CURV: CURV develops intrinsic visual grounded reasoning by reformulating CQA as multi-step processes that couple logical reasoning with dynamic grounding at every step.Models learn to focus on relevant chart regions while maintaining coherent reasoning chains without relying solely on extrinsic assistance.

2 What Prohibits MLLMs From Chart Understanding Success?

The section investigates why MLLMs struggle with chart question answering and evaluates whether reasoning prompts and visual guidance can address these limitations. It motivates internalizing these capabilities through visual reasoning with grounded focuses.

  • The section frames MLLMs’ correct understanding of chart images as an unresolved bottleneck.
  • GPT-4o is evaluated on 60 CharXiv CQA samples under four modes: Answer only, Vision + Answer, Reason + Answer, and Reason + Vision + Answer.The modes vary direct answering, human-annotated visual information, and chain-of-thought reasoning.
  • Motivated by extrinsic CoT prompting and visual guidance, the authors aim to internalize these capabilities within MLLMs.
  • The study extends its preliminary exploration to visual reasoning with grounded focuses using CCQA and Qwen2.5-VL-7B and GPT-4o.It examines performance under four modes with ground-truth reasoning and visual grounding.

3 CURV: Chart Reasoning with Dynamic Visual Grounding

CURV reformulates chart question answering as multi-step visual grounded reasoning, pairing each reasoning step with visual regions and dynamically updating visual focus. Its two-stage curriculum trains visual grounding first, then interleaved reasoning that uses grounded visual states to support coherent, evidence-based answers.

  • Problem Definition: Direct image-question-to-answer mapping lacks intermediate structure for accurate visual perception and dynamic grounding of reasoning chains.
  • Our Approach: CURV decomposes CQA into a progressive chain of reasoning steps, each paired with visual regions that ground the step in chart space before producing the answer.The chain consists of {(Rt, Vt)}T, where Rt is a natural-language reasoning step and Vt is its associated visual grounding.
  • Stage I: Visual Grounding (RV): Stage I trains the model to establish multimodal correspondences by predicting the visual focus Vt associated with each reasoning step Rt.Grounding supervision compares each predicted visual focus with its ground-truth focus at step t.
  • Stage II: Interleaved Visual Reasoning (RVA): Stage II feeds each step’s grounded visual state back into reasoning, enabling dynamic region focus, evidence-grounded logic, and cross-step coherence.The objective jointly balances reasoning, grounding, and final-answer supervision through λR, λV, and λA.
  • Curriculum Learning: The curriculum guides models from low-level visual components to high-level chart structures through three difficulty levels spanning reasoning and visual complexity.CURV also supports applied, boxed, and cropped grounding strategies for shifting visual focus during reasoning.

4 CCQA: Curriculum Chart Question Answering

CCQA is a curriculum dataset supporting CURV by progressively developing visual reasoning through controlled complexity, interleaved visual grounding, and template-based accuracy. It spans diverse chart types, domains, and query operations while using meta-learning to promote generalization to real-world and out-of-domain charts.

  • Curriculum Design: CCQA progressively develops visual reasoning through three curriculum levels varying reasoning depth, chart complexity, and operation sophistication.This progression is supported by systematic variation across the three factors.
  • Curriculum Design: Each reasoning step is paired with ground-truth visual regions and binary masks, enabling direct alignment between reasoning and visual grounding.The paired annotations connect every reasoning step with its corresponding visual evidence.
  • Dataset Construction: Synthetic templates ensure data accuracy, systematic reasoning-pattern coverage, and stable curriculum progression.Templates replace chart-specific features with plotting data for controlled generation.
  • Dataset Construction: CCQA covers 7 chart types, 30 domain categories, and atomic or nested operations across chart components and subplots.The chart types are bar, histogram, scatter, line, heatmap, pie, and radar; GPT-4o generates plotting data for chart drawing.
  • Meta-Learning: Meta-learning derives diverse query-reason-ground-answer quadruplets from only 30 unique charts per chart type to reduce visual overfitting and improve transfer.Systematic template instantiation produces varied tasks from each base image across appearances, distributions, contexts, domains, and task complexities.
  • Generalization: CURV finetuned on CCQA applies to real-world chart understanding and out-of-domain benchmarks, supporting the approach’s adaptability and generalizability.CCQA emphasizes fundamental visual components and spatial features for accurate visual reasoning.

5 Experiments

CURV is evaluated across curriculum levels, chart benchmarks, and out-of-domain multimodal reasoning tasks using complementary reasoning, visual-grounding, and answer metrics. Results show consistent gains on CCQA, complex multi-chart understanding, external benchmarks, and training-paradigm and grounding ablations.

  • Evaluation: Evaluation spans three CCQA curriculum-level test sets, four chart benchmarks, and out-of-domain MathVista and MMMU-Pro tasks.Metrics jointly assess reasoning, visual grounding through CIOU and GIOU, and answer accuracy using MLLM- and rule-based judges.
  • CCQA Performance: 20.92% absolute gains are achieved by finetuned models over baselines across six CCQA metrics under two-stage training.CURV@Applied (Qwen2.5-VL-7B) also reaches up to 15.65% over its base model and 12.22% over GPT models.
  • Complex Chart Understanding: 7.10% improvements are achieved by CURV@Applied (Qwen2.5-VL-7B) on multi-plot charts despite training only on single-plot charts.CURV@Applied (InternVL-8B) also shows ↑6.73% across all metrics.
  • Generalization: 1.20% improvements are obtained across four chart benchmarks, while out-of-domain multimodal reasoning achieves up to ↑10.20% accuracy improvements.These results support generalization beyond the curriculum data to real-world chart comprehension and diverse multimodal reasoning categories.
  • Training Paradigms: 17.04% improvement is achieved by combining supervised fine-tuning and reinforcement learning, exceeding the 12.58% gain from reinforcement learning alone.Reinforcement learning has higher computational overhead, whereas supervised fine-tuning offers a more favorable performance-efficiency trade-off.
  • Ablations and Analysis: 8.78% improvement favors explicit over implicit visual grounded reasoning, while applied masking outperforms boxed grounding and cropped grounding reaches up to ↑7.93%.Curriculum learning consistently beats standard training, foundational levels 1+2 provide the most balanced gains, and CURV improves localization and relational understanding on level 3.

6 Conclusion … A.2 Preliminary Exploration on CQA Challenges

CURV develops intrinsic visual reasoning through progressive multi-step visual grounded reasoning and the three-level CCQA curriculum. Preliminary analyses identify perception, reasoning consistency, and visual-reasoning integration as persistent CQA weaknesses that motivate this design.

  • 6 Conclusion: CURV trains intrinsic visual reasoning through progressive multi-step visual grounded reasoning, supported by CCQA’s three progressive curriculum levels.The framework tightly interleaves reasoning with visual grounding and reports consistent performance improvements across curriculum levels, with generalization to chart understanding.
  • A.1 Preliminary Exploration On Motivations: Preliminary studies examine current MLLM failures through decomposition, interleaved visual reasoning, and composition.These cognitive perspectives motivate the analysis of where multimodal models fall short.
  • A.1 Preliminary Exploration On Motivations: Perception errors account for 28.33% of GPT-4o’s failures on 60 CharXiv samples, making them the dominant identified cause.The analysis categorizes failures into reasoning, perception, and integration errors, linking perception failures to fine-grained chart grounding difficulty.
  • A.1 Preliminary Exploration On Motivations: The curriculum decomposes charts into low-level visual components and problems into structured reasoning steps.Visual decomposition guides attention toward informative details, while reasoning decomposition structures the chain of reasoning.
  • A.1 Preliminary Exploration On Motivations: Curriculum progression increases reasoning difficulty through nested functions and visual complexity through components, chart types, and subplots.The design also uses meta-learning to support adaptability and generalizability across varying conditions.
  • A.2 Preliminary Exploration on CQA Challenges: Case-by-case analysis across five MLLMs and different CQA benchmarks reveals recurring failure patterns in chart question answering.The models include GPT-4.1-mini, GPT-4o, Gemini-2.5-Flask, Qwen2.5-VL-3B, and Qwen2.5-VL-7B.
  • A.2 Preliminary Exploration on CQA Challenges: MLLMs show reasoning inconsistency when early visual misperceptions propagate through later reasoning, even when chain-of-thought prompting sometimes helps.Examples include Qwen2.5-VL-3B misaligning line colors with labels and GPT-4o failing to exclude “Loki” after identifying it.
  • A.2 Preliminary Exploration on CQA Challenges: MLLMs struggle to ground precise chart details and connect visual attention with logical reasoning across multiple regions and steps.GPT-4o inaccurately estimates the red “fi” point, while GPT-4o and GPT-4o-mini make different downstream inclusion errors after accurate initial perception.

A.3 Preliminary Validation on CCQA · B Related Work · C Dataset Construction

The paper validates CCQA across progressively harder chart-reasoning tiers, surveys related work in chain-of-thought, visual grounding, and chart understanding, and describes dataset construction through chart metadata and augmentation. The validation exposes persistent visual-perception, calculation, and grounding challenges as task complexity increases.

  • A.3 Preliminary Validation on CCQA: CCQA validation examines five MLLMs across five difficulty tiers spanning three curriculum levels.The tiers progress from single-plot reasoning to multi-plot localization and relation reasoning.
  • A.3 Preliminary Validation on CCQA: Across CCQA tiers, models fail through chart-number perception, component identification, mathematical calculation, or visual grounding.Qwen2.5-VL-3B and GPT-4o-mini misperceive numbers; Qwen2.5-VL-7B makes calculation errors; most models fail on components or multi-plot visual information.
  • A.3 Preliminary Validation on CCQA: Increasing failure cases and decreasing success cases across Figures 14–18 support progressively increased curriculum complexity while revealing persistent chart-understanding challenges.The challenges remain across reasoning depths and task-complexity levels.
  • B Related Work: Related work identifies chain-of-thought reasoning as beneficial for interpretability, performance, and complex multimodal visual reasoning.The cited literature describes gains from prompting language models to reason before answering.
  • B Related Work: Visual grounding aligns language with visual regions for spatially aware localization, using explicit or coordinate-free approaches alongside segmentation and GUI grounding.The passage notes that grounding is typically treated as a standalone objective.
  • B Related Work: Multimodal chart understanding requires structured visual comprehension and complex reasoning over visual and textual elements, motivating diverse CQA benchmarks and prior methods.Examples include ChartQA, ChartQA-Pro, ChartMuseum, and CharXiv.
  • C Dataset Construction: CCQA dataset construction specifies seven chart types, 30 domain categories, and 12 foundational operators for querying chart components.The metadata and operators support multi-layer nested functions and comprehensive understanding of chart elements.
  • C Dataset Construction: Chart-specific rendering augmentations introduce controlled variability in structural layout and visual presentation to improve generalization across diverse chart types.The augmentation design supports curriculum learning with meta-learning insights.

C.1 Data Structure … F.1 Evaluation of Answers

The paper builds CCQA as a controlled, multi-level curriculum dataset and trains visual grounded reasoning through progressively dynamic focus mechanisms. It evaluates generation modes and answer quality using semantic judging alongside tolerance-aware rule-based metrics.

  • C.1 Data Structure: CCQA spans seven chart types and 30 domain categories, using GPT-4o for plotting data while templates generate question-answer pairs, reasoning steps, and grounding coordinates.The template-based process is designed to preserve task diversity and the accuracy of intermediate reasoning, visual grounding, and final answers.
  • C.2 Multi-Level Curriculum: The curriculum progresses from single-operation reasoning (D1 = 1), through nested multi-operation reasoning (D2 > 1), to multi-chart reasoning with localization and relation operations (D3 ≥2).Reasoning depth is defined by nested operations, and higher levels require sequential grounding that builds on previous computations.
  • C.3 Fine-Grained Curriculum Tiers: Five curriculum tiers refine difficulty from one-depth single-plot queries to deeper nested reasoning and multi-plot localization tasks.Tier 1 uses D = 1, Tier 2 uses D = 2, Tier 3 uses D ≥3 on single plots, and Tier 4 introduces multi-plot localization with D ≥3.
  • C.4 Meta-Learning Supported Curriculum Learning: Meta-learning organizes 30 domain categories and 7 chart types into diverse, progressively complex meta-tasks that support transferable visual and reasoning competencies.The design yields 210 unique chart-task combinations from 30 plotting datasets multiplied by 7 chart types.
  • C.5 Quality Control: CCQA quality control combines human-authored meta-functions, manual validation of visual grounding and reasoning annotations, and restricted GPT-4o numerical randomization.Validation includes 10 × 30 × 7 randomly sampled visual reasoning instances in addition to checking 30 meta images for each chart type.
  • D Visual Grounding Strategies; D.1 Applied: Grounding Through Dynamic Visual Focus Highlighting; D.2 Boxed: Grounding Through Dynamic Visual Box Guides; D.3 Cropped: Grounding Through Dynamic Visual Focus Zooming: Three grounding strategies—applied, boxed, and cropped—adapt visual focus across reasoning steps while preserving or selectively presenting chart context.Applied uses shifting yellow overlays, boxed uses relocating red borders, and cropped supplies dynamically generated zoomed regions alongside the full chart.
  • E Generation Mode; E.1 Mode A; E.2 Mode VA; E.3 Mode RA; E.4 Mode RVA: The evaluation examples cover answer-only (A), visual-grounding-then-answer (VA), reasoning-then-answer (RA), and reasoning-with-grounding-then-answer (RVA) generation modes.These modes differ in whether intermediate reasoning and visual grounding precede the final answer.

F.2 Evaluation of Reasoning … G.1 Implementation Details.

The paper evaluates reasoning with complementary micro- and macro-level metrics, visual grounding with CIOU and GIOU, and generation modes that vary the ordering of answers, reasoning, and visual grounding. Implementation uses staged supervision, cosine-scheduled training, and LoRA-based memory efficiency.

  • F.2 Evaluation of Reasoning: F.2 Evaluation of Reasoning: Reasoning is assessed at micro and macro levels, with acc@mic averaging five semantic-similarity metrics and acc@mac using GPT-4.1-mini as judge.Micro-level evaluation uses ROUGE-L, BLEU, METEOR, BERTSCORE, and cosine similarity; macro-level evaluation rates reasoning quality on a 0–10 scale.
  • F.2 Evaluation of Reasoning: F.2 Evaluation of Reasoning: The macro-level judge evaluates visual understanding and grounding, logical coherence and multimodal integration, and alignment with ground-truth reasoning.The criteria assess accurate visual evidence use, coherent multimodal inference, and agreement with the reference reasoning process.
  • F.3 Evaluation of Visual Grounding: F.3 Evaluation of Visual Grounding: Visual grounding is evaluated with two intersection-over-union variants, CIOU and GIOU.These metrics incorporate predicted and ground-truth boxes, including centroid distance, enclosing-box geometry, and union area.
  • F.4 Evaluation Mode: F.4 Evaluation Mode: Four generation modes compare answer-only, reasoning-before-answer, visual-grounding-before-answer, and dynamically grounded reasoning-before-answer generation.The modes are A, RA, VA, and RVA, respectively.
  • G.1 Implementation Details: G.1 Implementation Details: Models are trained for 3 epochs with initial learning rate lr = 1e −4, cosine scheduling, and a 9:1 training-to-validation split.Training uses two NVIDIA 80G H100 GPUs.
  • G.1 Implementation Details: G.1 Implementation Details: LoRA is used for memory-efficient training, while Stage I supervises visual grounding only and Stage II jointly supervises reasoning, grounding, and answers.Stage I sets λV = 1.0, λR = 0.0, and λA = 0.0.

G.2 Grounding Method & Computation Cost … G.6 Curriculum Learning: Effectiveness of CL in Chart Understanding

CURV’s ablations show that visual grounding is most effective as training supervision that internalizes visual reasoning, while curriculum learning and explicit grounding improve chart understanding. The framework also extends to reinforcement learning, using rewards for grounding, reasoning, answers, and output format.

  • G.2 Grounding Method & Computation Cost: Up to 4.72% improvement on CCQA is achieved by cropped visual grounding over the baseline despite reducing training resolution to 128 × 128.The reduced resolution maintains comparable computational overhead while retaining competitive performance with the other grounding methods.
  • G.3 The Role of Visual Grounding: From Extrinsic Assistance To Intrinsic Abilities: Visual grounding significantly improves intrinsic reasoning during training, but explicitly generating multi-step grounding at inference degrades performance through cumulative grounding errors.Direct answer generation performs better than inference modes requiring explicit grounding coordinates or reasoning without grounding.
  • G.4 Visual Grounded Reasoning via Reinforcement Learning: CURV’s reinforcement-learning extension uses specialized rewards to incentivize accurate visual grounding, faithful reasoning, correct answers, and structured outputs.The answer and format rewards are binary exact-match and compliance signals, respectively.
  • G.5 Explicit vs. Implicit Visual Grounded Reasoning: CURV integrates grounding through explicit coordinate prediction or implicit attention modulation, with both designs built on the same two-stage training framework.The designs differ in how the MLLM is encouraged to attend to chart regions across reasoning steps.
  • G.5.1 Explicit Visual Grounding: Explicit grounding jointly trains the MLLM to predict dynamic visual focuses, interleave them with reasoning, and optimize reasoning, grounding, and answer quality.During training, ground-truth focus regions provide intermediate visual augmentation, while inference continues from the model’s predicted focus.
  • G.5.2 Implicit Visual Grounding: Implicit grounding trains a lightweight grounder on internal representations and visual patches, but its indirect feedback produces less effective visual augmentation than explicit grounding.The grounder is trained first with the MLLM frozen, then frozen while the MLLM learns interleaved reasoning with <GROUND> anchors.
  • G.6 Curriculum Learning: Effectiveness of CL in Chart Understanding: Curriculum training consistently outperforms untrained and non-curriculum settings on CCQA, with the gap widening as task complexity increases from levels 1 →3.Non-curriculum training improves CCQA but shows diminishing gains relative to CURV trained with Stage I or Stage I+II.

G.7 Two-Stage Learning: Strength of Training Curriculum

CURV uses two-stage curriculum training to build visual-reasoning capability by first learning visual grounding alignment and then applying it through interleaved, step-by-step reasoning. Combining both stages consistently outperforms either stage alone and supports more robust reasoning across task complexities.

  • Two-Stage Curriculum: Stage I learns vision-reasoning alignment through decoupled visual grounding, explicitly associating visual evidence with reasoning logic.This stage establishes foundational visual grounding ability.
  • Two-Stage Curriculum: Stage II trains interleaved visual reasoning, leveraging Stage I grounding to support step-by-step reasoning over visual inputs.Stage II operationalizes learned grounding during reasoning.
  • Training Strength: CURV@Stage I+II consistently outperforms both single-stage variants, demonstrating the advantage of combining decoupled alignment with interleaved reasoning.The comparison includes Qwen2.5-VL-7B, CURV@Stage I, CURV@Stage II, and CURV@Stage I+II.
  • Training Strength: Stage I improves level 1 simple CQA but degrades harder single-chart and multi-chart tasks, showing that decoupled grounding alone is insufficient for compositional reasoning.Compositional reasoning requires integrating multiple pieces of visual evidence.
  • Training Strength: Together, the stages yield more robust and generalizable visual reasoning capacity across varying task complexities.Stage I provides foundational grounding, while Stage II enables coherent, step-by-step visual reasoning.

G.8 Foundational Learning: Trade-off between Robustness & Adaptability · G.9 Broader Baselines: Comparison Against Chart Specialist & Tool-Use Agents · H Multi-Step Reasoning With Dynamic Visual Grounding

CURV’s curriculum benefits from progressive training through levels 1 and 2, while broader evaluations report transferability to newer backbones and comparisons with specialist and tool-augmented agents. The RVA evaluation assesses reasoning, visual grounding, and final answers.

  • G.8 Foundational Learning: Trade-off between Robustness & Adaptability: Training on level 1 alone yields foundational performance of ↑35.08% on level 1.This establishes solid performance before progressive curriculum training.
  • G.8 Foundational Learning: Trade-off between Robustness & Adaptability: Progressive training on levels 1+2 achieves the best overall performance across difficulty levels: ↑15.65% on level 1, ↑11.53% on level 2, and ↑7.10% on level 3.The passage identifies levels 1+2 as the optimal learning accumulation.
  • G.9 Broader Baselines: Comparison Against Chart Specialist & Tool-Use Agents: The study compares CURV with Qwen3-VL-4B, ChartGemma, Thyme, and DeepEyes on the CCQA test set across three curriculum levels.The comparisons cover a newer backbone, a chart specialist, and tool-augmented agents.
  • G.9 Broader Baselines: Comparison Against Chart Specialist & Tool-Use Agents: CURV also outperforms chart-specialist and tool-augmented agents built on the same Qwen2.5-VL-7B backbone.The cited passage names ChartGemma, Thyme, and DeepEyes among the broader comparisons.
  • G.9 Broader Baselines: Comparison Against Chart Specialist & Tool-Use Agents: CURV improves over its Qwen3-VL-4B base model by up to ↑16.14%, indicating generalizability to newer backbones.The evaluation uses acc@M and absolute accuracy acc@0.0.
  • H Multi-Step Reasoning With Dynamic Visual Grounding: RVA mode inference evaluates model reasoning, visual grounding, and final answers using the evaluation metrics defined in §5.2.Table 11 reports performance evaluation for RVA mode inference.

H.1 Challenges In Multi-Step Visual Grounding · H.2 Inference Failure · H.3 Inference Success

RVA training strengthens intrinsic visual reasoning through explicit intermediate grounding, but RVA inference remains vulnerable to cascading errors from flawed visual comprehension or calculation. CURV succeeds on increasingly complex chart QA tasks, while RVA can also answer correctly with accurate or approximate grounding.

  • H.1 Challenges In Multi-Step Visual Grounding: Explicit reasoning and visual-grounding outputs improve intrinsic visual reasoning by structuring complex tasks into dynamically grounded chains.Intermediate grounding supports more coherent reasoning trajectories.
  • H.1 Challenges In Multi-Step Visual Grounding: Inference exposes step-wise fragility: an incorrect calculation or inaccurate visual comprehension can propagate through the reasoning chain.This disrupts the balance between perception and reasoning.
  • H.1 Challenges In Multi-Step Visual Grounding: 14% absolute gain is the maximum reported RVA fine-tuning improvement for answering, alongside gains of up to 10.23% for reasoning and 9.38% for grounding.Qualitative inspection also finds systematic reasoning chains and sharper object localization.
  • H.1 Challenges In Multi-Step Visual Grounding: Despite improving intrinsic visual reasoning, RVA training yields lower answering performance than CURV with the same base model and grounding method.The comparison indicates that RVA inference magnifies intermediate error accumulation.
  • H.1 Challenges In Multi-Step Visual Grounding: GPT-4o and Qwen2.5-VL-7B both fail on an RVA CQA example because incorrect visual grounding disrupts their reasoning chains.The example illustrates how cumulative intermediate errors compromise correct answers.
  • H.2 Inference Failure: RVA inference failures arise when early visual-comprehension inaccuracies cause inaccurate chart extraction and misinterpretation that propagate into incorrect answers.Figure 35 presents two such failure cases.
  • H.3 Inference Success: CURV with Qwen2.5-VL-7B succeeds where the baseline fails on value reading and nested-function tasks by localizing the queried chart components.The examples span D = 1 and D = 2, while harder cases involve extremia comparison at D = 4 and cross-chart relations at D = 5.
  • H.3 Inference Success: RVA can produce correct answers with accurate grounding in a bar chart and near-region grounding in a heatmap.The heatmap example succeeds despite grounding that is not exact but remains close to the focus regions.

I Limitations & Future Work

This section summarizes CURV’s intrinsic visual grounded reasoning framework and CCQA curriculum dataset, alongside examples of successes and failures across curriculum levels and RVA inference.

  • Contributions: CURV reformulates chart question answering as multi-step visual grounded reasoning with dynamic spatial attention to develop intrinsic capabilities in MLLMs.The framework is paired with CCQA, a three-level curriculum dataset supporting scalable synthetic generation across diverse chart types and reasoning patterns.
  • Curriculum examples: Across curriculum levels 1–2, CURV with applied grounding succeeds on examples where Qwen2.5-VL-7B fails.The examples are drawn from CCQA curriculum levels 1–2.
  • Curriculum examples: At curriculum level 3, CURV with applied grounding succeeds on examples where Qwen2.5-VL-7B fails.These examples illustrate performance on the most advanced CCQA curriculum level described in the supplied passages.
  • RVA inference: RVA inference includes both failures to produce correct answers and cases where CURV with applied grounding succeeds while Qwen2.5-VL-7B fails.The supplied figures show two RVA failure examples and three RVA success examples on CCQA.
Loading 2608.02833v1…