PaperlayerView exact revision on arXiv

Source-linked AI summary

Visually-Guided Policy Optimization for Multimodal Reasoning

Zengbin Wang, Feng Xiong, Liang Lin, Xuecai Hu, Yong Wang, Yanlin Wang, Man Zhang, Xiangxiang Chu

arXiv:2604.09349v2cs.CVcs.AIcs.CL

TL;DR

VLMs’ text-dominated reasoning can produce sparse visual activation and temporal visual forgetting, limiting visual faithfulness. VGPO uses intrinsic visual similarity, Visual Attention Compensation, and dual-grained advantage re-weighting to sustain visual focus, with reported gains across mathematical and vision-dependent reasoning tasks. The authors note that its progressive visual-expectation heuristic may be suboptimal for reasoning steps independent of visual cues, and performance remains bounded by visual representation quality.

  • Problem

    Text-dominated VLM inference causes sparse visual-token activation, while temporal visual forgetting further reduces visual focus during reasoning.

  • Method

    VGPO uses hidden-state similarity to localize visual tokens, compensates visual attention with progressively higher later-step expectations, and re-weights advantages within and across trajectories.

  • Results

    VGPO achieves better visual activation and state-of-the-art performance across mathematical and vision-dependent multimodal reasoning tasks, including 33.2% and 30.0% relative gains over Qwen2.5-VL-7B.

  • Takeaways & Limitations

    The results support incorporating intrinsic visual focus into policy optimization as a strategy for improving sustained visual reasoning across multimodal task types.

  • Takeaways & Limitations

    Progressively elevating visual expectations is a heuristic that may be suboptimal when later reasoning relies strictly on logic or calculation independent of visual cues.

Abstract

from arXiv · show

Reinforcement learning with verifiable rewards (RLVR) has significantly advanced the reasoning ability of vision-language models (VLMs). However, the inherent text-dominated nature of VLMs often leads to insufficient visual faithfulness, characterized by sparse attention activation to visual tokens. More importantly, our empirical analysis reveals that temporal visual forgetting along reasoning steps exacerbates this deficiency. To bridge this gap, we propose Visually-Guided Policy Optimization (VGPO), a novel framework to reinforce visual focus during policy optimization. Specifically, VGPO initially introduces a Visual Attention Compensation mechanism that leverages visual similarity to localize and amplify visual cues, while progressively elevating visual expectations in later steps to counteract visual forgetting. Building on this mechanism, we implement a dual-grained advantage re-weighting strategy: the intra-trajectory level highlights tokens exhibiting relatively high visual activation, while the inter-trajectory level prioritizes trajectories demonstrating superior visual accumulation. Extensive experiments demonstrate that VGPO achieves better visual activation and superior performance in mathematical multimodal reasoning and visual-dependent tasks. The code has been released at https://github.com/wzb-bupt/VGPO.

1 Introduction

VLM reasoning can become text-dominated, causing sparse visual activation and temporal visual forgetting. VGPO addresses this by using intrinsic visual signals and dual-grained policy optimization to sustain visual focus.

  • Motivation: VLMs often assign less attention to image tokens than to query and generated-text tokens, producing sparse visual activation.As reasoning chains extend, attention to visual inputs progressively decays, increasing reliance on textual priors.
  • Motivation: Prior approaches amplify visual responses through external triggers, auxiliary verification models, or contrasts between original and noisy images.These methods introduce external tokens, GPT-5-based verification, or visual-token highlighting through KL-divergence comparisons.
  • Approach: Intrinsic hidden-state similarity between generated tokens and image tokens provides a Visual Focus Score for localizing visual activations without external supervision.The score identifies visually grounded tokens and supports targeted attention modulation.
  • Approach: Visual Attention Compensation re-weights visually grounded tokens and progressively raises visual expectations in later reasoning steps to combat temporal visual forgetting.The mechanism targets both visual amplification and sustained visual focus.
  • Approach: VGPO integrates this compensation into policy optimization through intra-trajectory token re-weighting and inter-trajectory prioritization of visually accumulated trajectories.The strategy explicitly incentivizes sustained visual faithfulness at both local and global trajectory levels.
  • Results: Experiments on mathematical and vision-dependent multimodal reasoning tasks report better visual activation and state-of-the-art performance.The introduction presents these results as evidence that VGPO mitigates visual forgetting across task types.

2 Preliminary

The preliminary analysis describes GRPO as a resource-efficient policy-optimization baseline and identifies sparse, decaying visual attention as a central multimodal reasoning problem. Intrinsic visual similarity motivates VGPO’s visual-focus mechanisms.

  • Group Relative Policy Optimization: GRPO eliminates the value model and estimates advantages through group-based computation, making it a resource-efficient reinforcement-learning variant.The method samples candidate responses, assigns exact-match rewards, and normalizes group-based advantages before policy updates.
  • Group Relative Policy Optimization: The GRPO policy is updated by maximizing a surrogate objective based on response-level advantages and token probability ratios.The formulation uses candidate responses sampled from the current policy and exact-match binary rewards.
  • Key Findings in Multimodal Reasoning: VLMs heavily attend to generated text history and query tokens, while visual attention remains lower and often appears as a brief early image glance.The analysis reports long later periods of visual neglect.
  • Key Findings in Multimodal Reasoning: Visual activation initially increases but progressively decays with generation length, a pattern termed temporal visual forgetting.Correct samples show higher late/early visual accumulation ratios than incorrect samples, with averages of 0.680 versus 0.532.
  • Key Findings in Multimodal Reasoning: When visual tokens activate, their image attention maps are highly accurate and semantically grounded, indicating that the model can locate relevant information despite weak sustained focus.This observation motivates using visual similarity as an intrinsic score to detect and re-weight critical visual tokens without auxiliary models.

3 Methodology

VGPO converts intrinsic visual attention into explicit policy-optimization signals through visual focus scoring, attention compensation, and dual-grained advantage re-weighting. The framework progressively strengthens later visual signals and prioritizes visually salient tokens and trajectories.

  • VGPO comprises Visual Focus Score, Visual Attention Compensation, and Dual-grained Advantage Re-weighting to sustain visual focus during reasoning.These components transform intrinsic visual attention into explicit signals for policy optimization.
  • Visual Focus Score: Visual Focus Score uses hidden-state similarity between generated tokens and a visual prototype derived from input image-token states.A scaling function normalizes the resulting focus scores to [0, 1].
  • Visual Attention Compensation: Visual Attention Compensation linearly increases visual optimization expectations as reasoning progresses, counteracting temporal visual-attention decay.The compensation intensity is controlled by β, while a visual gate selects tokens with higher visual similarity and filters noisy text tokens.
  • Dual-grained Advantage Re-weighting: Dual-grained Advantage Re-weighting adjusts policy advantages using both token-level visual saliency within trajectories and aggregate visual focus across trajectories.Intra-trajectory normalization highlights locally salient tokens, while inter-trajectory scoring aggregates compensated weights across timesteps and normalizes them across the rollout group.
  • Overall Integration: The final visual-focus advantage multiplicatively combines the standard outcome-based advantage with fine- and coarse-grained visual scaling factors.This incorporates visual focus alongside final-answer correctness in the policy update.

4 Experiments

Experiments evaluate VGPO across multimodal reasoning benchmarks, training settings, advantage-shaping strategies, and visual-attention analyses. VGPO improves performance and training dynamics, while ablations support complementary re-weighting and late-stage compensation.

  • Main Results: VGPO delivers relative gains of 33.2% on general mathematical reasoning and 30.0% on vision-dependent multimodal reasoning over base Qwen2.5-VL-7B.It also achieves average accuracies of 66.6% and 63.3% across the corresponding benchmark groups.
  • Training Dynamics: VGPO maintains higher training rewards and validation accuracy than GRPO and DAPO throughout the reported training dynamics.The comparison uses Qwen2.5-VL-7B on MMK12.
  • Ablation Studies: The combined intra- and inter-trajectory re-weighting strategies achieve peak average performance across all reported metrics.The ablation identifies the two mechanisms as complementary and synergistic.
  • Advantage Shaping: VGPO reaches a peak average accuracy of 63.3%, outperforming entropy- and KL-based advantage shaping, especially on vision-centric tasks such as LogicVista and Counting.These comparisons evaluate generic regularization against explicit visual feedback.
  • Sensitivity Analysis: Optimal hyperparameters are β = 0.3, κ = 0.2, and γ = 0.5, with best performance occurring when the Late/Early Ratio stabilizes near 1.The ratio divides late-stage visual attention by early-stage visual attention.
  • Compensation Schedule: Full-trajectory compensation significantly reduces performance because early visual attention is already high, whereas late-trajectory compensation targets later visual decay.The reported rationale is to avoid distracting textual-query parsing or overemphasizing early visual attention.

5 Related Work

Related work extends multimodal reasoning through reinforcement learning, external interventions, and visual-perception methods. VGPO instead applies continuous visual guidance through attention compensation without external models or outcome-based proxy tasks.

  • Multimodal Reasoning Challenges: Current VLMs inherit text-dominated inductive biases from their language-model backbones, leading to insufficient visual faithfulness.This limitation motivates methods that strengthen visual grounding during reasoning.
  • Mainstream Strategies: Existing multimodal-reasoning methods primarily use reinforcement learning with rollout sampling to refine reasoning trajectories.Related improvements include training strategies such as rollout diversity and entropy-based regulation.
  • Visual Perception Methods: Visual-perception methods such as PEARL and ViCrit require external checklists or synthetic hallucination detection, incurring computational or data-construction costs.Other concurrent approaches explore visual guidance within reinforcement learning.
  • Visual Perception Methods: VGPO uses continuous visual guidance through attention compensation to mitigate visual decay and sparse activation without external models or outcome-based proxy tasks.The method is presented as supporting robust reasoning and cross-domain generalizability.

6 Conclusion

The paper addresses insufficient and temporally decaying visual activation in multimodal reasoning with VGPO. Its visual-attention compensation and dual-grained advantage re-weighting improve sustained visual reasoning and reported task performance.

  • Conclusion: VGPO uses intrinsic hidden states to autonomously ground visual focus during multimodal reasoning.The framework couples Visual Attention Compensation with dual-grained advantage re-weighting.
  • Conclusion: Comprehensive experiments show that VGPO enhances visual activation and achieves state-of-the-art performance.The conclusion frames these results across the paper’s multimodal reasoning evaluations.

Limitations

VGPO has two stated limitations: its progressive visual-expectation heuristic may not suit every reasoning pattern, and its ceiling depends on the base model’s visual representations.

  • Heuristic visual expectations: Progressively increasing visual expectations may be suboptimal when later reasoning relies on logic or calculation rather than visual cues.The paper calls for more adaptive mechanisms that adjust visual reliance to each reasoning step.
  • Representation ceiling: VGPO’s performance upper bound is constrained by the representational quality of the visual encoder and projector.The method optimizes use of existing visual cues rather than improving the encoder’s raw perceptual capabilities.

A.1 Training Datasets

The appendix describes ViRL39K as the main training dataset and surveys related multimodal reasoning methods and evaluation benchmarks spanning mathematical, geometric, and vision-dependent tasks.

  • Training dataset: Training experiments mainly use ViRL39K, a roughly 39,000-sample verifiable instruction-tuning dataset for vision-language reasoning.Its samples are synthesized from proprietary collections and multiple upstream datasets.
  • Related methods: The related training literature includes reinforcement-learning methods using replay, forced rethinking, online filtering, noise schedules, perception losses, and visual-dependency modulation.The cited methods include VL-Rethinker, MM-Eureka, NoisyRollout, PAPO, and VPPO.
  • Evaluation benchmarks: Evaluation coverage includes mathematical and visual benchmarks such as MathVista, MathVerse, We-Math, MMK12, GeoMath, and Geometry3K.These benchmarks target visual interpretation, compositional reasoning, multimodal science reasoning, Earth-observation mathematics, and geometry understanding.
  • Evaluation benchmarks: Additional benchmarks assess integrated visual-context reasoning, domain generalization, and intrinsic multimodal reasoning through LogicVista, Super-CLEVR, and MMMU-Pro.Their designs include annotated multiple-choice reasoning, controlled domain shifts, and vision-only questions.

B.1 Detailed Results of Re-weighting Strategy

The ablation results show complementary benefits from intra-trajectory and inter-trajectory re-weighting, with their combination producing the most robust cross-category performance.

  • Intra-trajectory strategy: MathVista improves by 5.4% and MMK12 by 4.7% with intra-trajectory re-weighting.The results attribute these gains to reinforcing self-consistency within individual reasoning paths and refocusing visual activation.
  • Inter-trajectory strategy: Inter-trajectory re-weighting reaches 54.9% on GeoMath and matches the top result on the Counting dataset.The passage associates this strategy with geometry and visual-counting tasks.
  • Combined strategy: The combined strategy achieves 72.5% on WeMath and top results on LogicVista and MMMU-Pro.The reported pattern is a synergistic combination of intra-trajectory logical refinement and inter-trajectory reasoning-scope expansion.

B.2 Detailed Results of Advantage Shaping Methods

VGPO outperforms generic advantage-shaping baselines across mathematical and vision-centric benchmarks, while training-dynamics analyses report stronger rewards, validation accuracy, and optimization stability.

  • General reasoning: VGPO achieves the highest scores on 4 of 6 datasets in general mathematical and geometric reasoning.The passage reports significant leads on MathVista and MathVerse while preserving general problem-solving proficiency.
  • Visual-dependent reasoning: VGPO reaches 95.5% accuracy on Counting, surpassing Entropy and KL baselines, with similar dominance on LogicVista.The comparison highlights the method’s performance on vision-dependent multimodal reasoning.
  • Advantage shaping: The comparison evaluates VGPO against Entropy-based and KL-based advantage-shaping strategies.VGPO explicitly uses visual feedback, unlike these baselines’ implicit policy-divergence constraints.
  • Training dynamics: VGPO maintains a consistently higher training-reward trajectory than DAPO for Qwen2.5-VL-3B.Figure 7 compares training rewards and MMK12 validation accuracy across the two methods.
  • Training dynamics: For Qwen2.5-VL-32B, VGPO consistently outperforms DAPO in validation accuracy and establishes a significant end-of-training margin.The accompanying analysis interprets the dynamics as evidence of reduced variance and more stable optimization.

B.4 Detailed Results of Different Compensation Schedules

The compensation-schedule analysis compares linear, step-function, exponential, and full-trajectory strategies. Linear compensation is selected because it matches progressive visual-attention decay and consistently performs best, while excessive early or late compensation can hurt results.

  • Schedule comparison: Linear compensation consistently achieves the best performance among the tested schedules across four visual-dependent tasks.The comparison includes LogicVista, CLEVR Counting, MMMU-Pro, and MathVerse-V.
  • Schedule comparison: The linear schedule matches the observed progressive, continuous, and nearly linear decay of visual attention across reasoning trajectories.This motivates smoothly increasing compensation rather than abrupt or heavily back-loaded incentives.
  • Schedule comparison: Exponential compensation can over-correct by concentrating bonuses on final tokens that may represent calculation or formatting rather than visual grounding.This imbalance can dilute compensation for genuinely visual-dependent reasoning steps and reduce accuracy relative to the linear schedule.
  • Schedule comparison: Step-function compensation can destabilize training because it jumps from zero to full compensation at one threshold instead of tracking gradual visual forgetting.The linear schedule increases compensation smoothly over positions.
  • Trajectory placement: Full-trajectory compensation significantly reduces performance on most benchmarks and produces a lower Late/Early Visual Activation Ratio than VGPO.The reported ratios are 0.8591 for full-trajectory compensation and 1.0696 for VGPO.

E Failure Case Analysis

The failure-case analysis shows that increasing visual focus does not guarantee correct multimodal reasoning. Errors arise both from flawed visual perception and from incorrect reasoning applied to correctly perceived visual information.

  • Failure boundaries: VGPO can reinforce a base encoder’s visual misinterpretation, producing confidently incorrect answers instead of relying on potentially helpful common-sense priors.The example describes a blurry “6” being misrepresented as an “8” because of low resolution.
  • Failure boundaries: High visual focus is necessary but not sufficient for correctness because models also need appropriate world knowledge and reasoning logic.Some incorrect samples still exhibit high visual focus scores.
  • Visual perception error: In the GeoMath example, the model counts 10 vertical bars instead of the actual 7, then consistently computes an incorrect total of 11.The arithmetic is internally consistent, but the reasoning begins from a flawed visual count.
  • Logical reasoning error: In the MathVerse-V example, the model correctly perceives the geometry and computes ∠C as 40°, but misapplies the parallel-line theorem to derive ∠CED.The failure is attributed to flawed geometric knowledge rather than visual perception.
Loading 2604.09349v2…