PaperlayerView exact revision on arXiv

Source-linked AI summary

Spotlight on Token Perception for Multimodal Reinforcement Learning

Siyuan Huang, Xiaoye Qu, Yafu Li, Yun Luo, Zefeng He, Daizong Liu, Yu Cheng

arXiv:2510.09285v2cs.CV

TL;DR

Multimodal RLVR often neglects visual perception, despite its role in visually grounded reasoning. The paper measures token visual dependency and introduces VPPO, which reshapes trajectory advantages and filters updates to pivotal tokens; across eight benchmarks, it improves performance at both 7B and 32B scales.

  • Problem

    Existing multimodal RLVR methods neglect visual perception, making them unable to distinguish visually grounded reasoning paths or pivotal perceptual tokens.

  • Method

    VPPO measures token visual dependency, reweights trajectory advantages by overall dependency, and restricts policy updates to perceptually pivotal tokens.

  • Results

    VPPO improves average accuracy by 19.2% for 7B models and 7.6% for 32B models across eight multimodal reasoning benchmarks, while outperforming strong open-source competitors.

  • Takeaways & Limitations

    Token-level perceptual analysis and hierarchical signal modulation provide an optimization strategy for improving multimodal reasoning across model scales.

  • Takeaways & Limitations

    Outcome-based RLVR rewards cannot distinguish equally correct trajectories with different visual grounding and applies the same reward indiscriminately to every token.

Abstract

from arXiv · show

While Reinforcement Learning with Verifiable Rewards (RLVR) has advanced the reasoning capabilities of Large Vision-Language Models (LVLMs), most existing methods in multimodal reasoning neglect the critical role of visual perception within the RLVR optimization process. In this paper, we undertake a pioneering exploration of multimodal RLVR through the novel perspective of token perception, which measures the visual dependency of each generated token. With a granular analysis of Chain-of-Thought (CoT) processes, we uncover two key insights: first, token perception in a rollout trajectory is sparsely distributed, where only a small fraction of tokens have high visual dependency for visually-grounded reasoning; second, different trajectories exhibit significant divergence in their overall visual dependency. Based on these observations, we propose Visually-Perceptive Policy Optimization (VPPO), a novel policy gradient algorithm that explicitly leverages token perception to refine the learning signal. Specifically, VPPO achieves this through a dual mechanism: it reweights a trajectory's advantage by its overall visual dependency, and focuses policy updates exclusively on perceptually pivotal tokens. On a comprehensive suite of eight perception and reasoning benchmarks, VPPO demonstrates substantial gains over leading open-source RL-tuned models, with its effectiveness consistently validated across 7B and 32B model scales. Our findings not only establish a new token-level perceptual perspective for analyzing multimodal RLVR but also present a novel and effective optimization strategy to significantly enhance the multimodal reasoning capabilities of LVLMs.

1 INTRODUCTION

Existing multimodal RLVR methods largely neglect visual perception, even though perception grounds multimodal reasoning. The paper analyzes token-level visual dependency and introduces VPPO, which reshapes trajectory advantages and filters token gradients to target perceptually important reasoning.

  • 1 INTRODUCTION: Prevailing LVLM RLVR frameworks neglect visual perception despite its role in grounding logical deduction.The paper motivates this gap with a geometry problem whose solution depends on interpreting visual relationships.
  • 1 INTRODUCTION: Only a minority of generated tokens have high visual dependency, while trajectories differ substantially in their overall reliance on visual evidence.These observations identify limitations in applying uniform text-based RLVR signals to multimodal reasoning.
  • 1 INTRODUCTION: VPPO reweights each trajectory’s advantage using average visual dependency and focuses policy updates on perceptually pivotal tokens.Its two-level design converts uniform learning into a targeted signal for perception-grounded reasoning.
  • 1 INTRODUCTION: VPPO improves average accuracy by 19.2% for 7B models and 7.6% for 32B models over baseline across eight multimodal reasoning benchmarks.The gains also accompany greater training stability and faster convergence.
  • 1 INTRODUCTION: The paper presents token perception analysis and VPPO as three main contributions: perceptual diagnosis, hierarchical optimization, and broad empirical validation.The validation covers eight perception and reasoning benchmarks and both 7B and 32B scales.

2 RELATED WORK

Related work adapts text-only reasoning advances through multimodal data, curricula, rewards, and distillation, while prior pivotal-token methods focus mainly on logical or confidence signals. VPPO instead identifies pivotal tokens through visual dependency for multimodal reasoning.

  • 2 RELATED WORK: LVLMs still exhibit a performance gap on tasks requiring integrated visual and linguistic reasoning compared with text-only LLMs.The related-work discussion frames multimodal adaptation as necessary for bridging this gap.
  • 2 RELATED WORK: Multimodal RL strategies commonly modify data, curricula, distillation, or reward design rather than the core learning algorithm.These directions include visually grounded data curation, Chain-of-Thought distillation, training curricula, and perception-aware rewards.
  • 2 RELATED WORK: Prior RL methods identify pivotal tokens through entropy-based forking points, low-confidence errors, or correct-versus-incorrect model contrasts.These signals do not directly encode visual grounding.
  • 2 RELATED WORK: VPPO introduces a multimodal RL algorithm that formally identifies perceptually pivotal tokens using token visual dependency.This connects token selection to visually grounded reasoning rather than only to logical uncertainty.

3 METHOD

The method measures how much visual input changes each token’s predictive distribution, reveals sparse token dependency and heterogeneous trajectory grounding, then uses these signals to shape advantages and filter gradients. VPPO therefore replaces uniform updates with hierarchical, perception-focused policy optimization.

  • 3 METHOD: Token visual dependency is the KL divergence between output distributions conditioned on the true image and a perturbed, non-informative image.A high value indicates that the image provides critical information for predicting that token.
  • 3 METHOD: Token dependency is highly skewed: only a small fraction of tokens are critical for visually grounded reasoning, with frequency dropping exponentially as dependency increases.The analysis uses Qwen2.5-VL-7B on the vision-dominant MathVerse subset and visualizes the distribution in Figure 3.
  • 3 METHOD: Trajectory dependency is the mean token dependency over a trajectory, and its right-skewed distribution reveals heterogeneous visual grounding across reasoning paths.High-dependency trajectories form a distinct tail, with the reported mean at 0.09.
  • 3 METHOD: Token-level Gradient Filtering updates only the top-k% tokens by visual dependency, using a binary mask to remove generic-token noise.The mask targets tokens that bridge vision and language.
  • 3 METHOD: Trajectory-level Advantage Shaping rescales each GRPO advantage by a dependency-based factor, amplifying visually engaged trajectories and dampening less grounded ones.The shaped advantage is defined as ˆA′(τi) = α(τi) · ˆAGRPO(τi).
  • 3 METHOD: VPPO reduces gradient-estimator variance by filtering low-dependency tokens and regularizing updates for less visually grounded trajectories.The analysis links the reduction to the sparsity ratio k and shaping factor α(τ).

4 EXPERIMENTS

Experiments train VPPO on Qwen2.5-VL models using ViRL39K and evaluate against open-source reasoning LVLMs on eight exact-match multimodal benchmarks. The setup covers both model scales and uses direct RL with specified training controls.

  • 4 EXPERIMENTS: The study applies VPPO to Qwen2.5-VL-7B and Qwen2.5-VL-32B, trains on ViRL39K, and compares against leading open-source reasoning LVLMs.The comparison is conducted at both model scales.
  • 4 EXPERIMENTS: Training runs for two epochs with learning rate 1e-6 and rollout batch size 384, using maximum response lengths of 2048 for 7B and 4096 for 32B models.A 0.06 entropy penalty is applied to VPPO and the baseline; VPPO uses k = 0.4 for gradient filtering.
  • 4 EXPERIMENTS: Table 1 reports average-at-eight accuracy as the main evaluation metric, with exact-match scoring and direct RL without supervised fine-tuning.The table also documents reproduction and training-data caveats for selected baselines.
  • 4 EXPERIMENTS: VPPO is evaluated on eight benchmarks spanning mathematical, geometric, logical, and multidisciplinary multimodal reasoning.Evaluation uses exact-match scoring on verifiable instances rather than LLM-as-a-judge assessment.

5 RESULTS

VPPO outperforms strong open-source competitors across 7B and 32B models, with training and ablation analyses supporting its hierarchical visual-dependency design.

  • 5.1 MAIN RESULTS: 57.5% average accuracy at 7B and 64.6% at 32B let VPPO lead the next-best open-source methods, PAPO and DAPO respectively.The comparison covers both model scales and is reported in Table 1.
  • 5.1 MAIN RESULTS: VPPO converges faster initially than baselines and reaches higher performance more efficiently during training.The authors associate these dynamics with a more efficient and robust path to high performance.
  • 5.2 ABLATION STUDIES: Combining Trajectory-level Advantage Shaping and Token-level Gradient Filtering achieves the best ablation performance, while TGF contributes the largest individual gain.Both components individually outperform the baseline, but their combination provides the optimal result.
  • 5.2 ABLATION STUDIES: Performance peaks around k = 0.4 because smaller ratios provide insufficient signal whereas larger ratios reintroduce non-pivotal-token noise.Figure 6 reports the average score and per-benchmark deviations from the mean.
  • 5.2 ABLATION STUDIES: Visual-dependency token selection outperforms entropy-based selection because it captures confident but indispensable visual facts as well as uncertain reasoning junctions.Entropy-based selection can miss direct observations such as numbers and entities that ground multimodal reasoning.
  • 5.2 ABLATION STUDIES: VPPO improves GRPO accuracy by 1.7%, from 53.1% to 54.8%, supporting gains across different base policy-gradient algorithms.The improvement is consistent with the 2.5% gain reported when VPPO is applied to DAPO.
  • 5.2 ABLATION STUDIES: The binary TGF mask outperforms a soft mask by 1.3%, while advantage shaping exceeds reward shaping by 0.7% on average.These comparisons support decisive filtering and direct advantage modulation as the stronger design choices.
  • 5.3 GENERALIZATION: VPPO achieves the highest performance among evaluated methods on two unseen out-of-domain VQA benchmarks.All evaluated reinforcement-learning fine-tuning methods improve over the Qwen2.5-VL-7B base model by approximately 4% average accuracy.

6 CONCLUSION

The paper presents VPPO as a hierarchical strategy that prioritizes visually grounded trajectories and updates only sparse pivotal perception tokens, improving multimodal reasoning stability and efficiency.

  • 6 CONCLUSION: VPPO prioritizes visually grounded trajectories and restricts policy updates to sparse pivotal perception tokens.The conclusion frames this targeted signal modulation as central to the method.
  • 6 CONCLUSION: The authors report state-of-the-art performance, greater training stability, and improved efficiency across challenging multimodal benchmarks.The conclusion presents learning-signal structure as important alongside the reward itself.
  • 6 CONCLUSION: The paper positions modality-aware signal modulation as a promising path for advancing LVLM reasoning capabilities.This is the stated broader implication within the paper's supported scope.

ETHICS STATEMENT

The study uses public academic multimodal-reasoning benchmarks and reports no personal data collection or human-subject involvement.

  • ETHICS STATEMENT: The research uses publicly available academic benchmarks without personal, sensitive, or private user data, and involves no human subjects.Its stated goal is improving AI reasoning on mathematical and logical problems.

REPRODUCIBILITY STATEMENT

The paper describes its implementation, training setup, VPPO procedure, theoretical assumptions, and variance result to support reproducibility and analysis.

  • REPRODUCIBILITY STATEMENT: The authors provide an anonymous code repository, public datasets, hyperparameter documentation, and supplementary implementation details.The setup is summarized in Table 9 and expanded in the appendices.
  • IMPLEMENTATION DETAILS: Experiments use Qwen2.5-VL-7B and Qwen2.5-VL-32B with two training epochs on ViRL39K and eight generated responses per question.The vision tower is unfrozen, and the reward is binary accuracy.
  • COMPUTATIONAL SETUP: Training was conducted on a cluster of 8 x NVIDIA H800 80GB GPUs using PyTorch 2.6.0 and CUDA 12.4.These details specify the computational environment for the reported experiments.
  • VPPO CONFIGURATION: The reported configuration uses KL divergence for dependency calculation and random patch blackening for image perturbation.The paper states that both choices were selected through appendix ablations.
  • VPPO ALGORITHM: VPPO computes token dependency after image masking, shapes trajectory advantages, filters the top-k dependency tokens, and updates the policy with the resulting signals.Algorithm 1 organizes the procedure into rollout generation, dependency calculation, hierarchical modulation, and policy update phases.
  • HIERARCHICAL SIGNAL MODULATION: The method uses a normalized dependency-based shaping factor and a binary mask over the top k · T token positions.The shaped advantage is α_i · Â_i, while the mask retains indices with the highest dependency scores.
  • THEORETICAL ANALYSIS: Under four stated assumptions, the VPPO gradient-estimator variance is reduced relative to GRPO by approximately k · E[α(τ)^2].The derivation interprets this proportional reduction as producing more stable training.

E THE ROLE OF THE ENTROPY PENALTY IN STABILIZING TRAINING

Entropy regularization is necessary to stabilize online RL training for LVLMs: without it, sparse rewards can drive DAPO into policy collapse, whereas a coefficient of 0.06 supports stable learning and preserves VPPO comparisons.

  • Without regularization, DAPO rapidly collapses into incoherent gibberish after brief exploration.The collapse abandons coherent reasoning and reflects a policy exploiting reward correlations rather than producing meaningful solutions.
  • Sparse binary rewards assign the same signal to every token in a trajectory, allowing nonsensical sequences that happen to receive positive rewards.This coarse outcome-based signal does not distinguish visually grounded reasoning from shortcut solutions.
  • The collapse is accompanied by uncontrolled entropy growth, motivating an entropy penalty as a stabilizing regularizer.The penalty counteracts the chaotic regime associated with reward under-specification.
  • Applying the same λ = 0.06 penalty to DAPO and VPPO prevents immediate baseline collapse and enables a controlled comparison.VPPO remains more robust to noisy uniform rewards because it focuses updates on sparse pivotal tokens.
  • λ = 0.06 achieves the highest and most stable training accuracy, while λ = 0.00 suffers a sharp collapse.The coefficient ablation links the accuracy failure to uncontrolled entropy divergence.

F ABLATION STUDY ON MASKING STRATEGY FOR DEPENDENCY CALCULATION

The masking ablation evaluates how image perturbations affect visual-dependency calculation and selects Random Patch Blackening as the default because it achieves the best average accuracy.

  • The ablation compares the default perturbation with common alternatives to test how the non-informative image used for dependency calculation affects identified tokens.The perturbation choice is treated as a key hyperparameter because it can influence which tokens appear visually dependent.
  • Random Patch Blackening independently drops 14x14 ViT-aligned image patches with probability 0.5, while alternatives perturb pixels, blur the image, or remove visual information entirely.The compared strategies use the same training hyperparameters and are evaluated on final model performance.
  • 57.5% average accuracy makes Random Patch Blackening the strongest masking strategy, ahead of Additive Gaussian Noise, Gaussian Blur, and Complete Masking.The corresponding averages are 56.5% for Additive Gaussian Noise and 56.4% for both Gaussian Blur and Complete Masking.
  • The authors hypothesize that patch-based masking provides a more localized and informative signal by forcing reasoning from incomplete visual evidence.Complete Masking remains competitive, but the consistent edge of patch blackening supports using partial visual context as the default.

G ABLATION STUDY ON METHODS FOR DEPENDENCY CALCULATION

The dependency-calculation ablation compares full-distribution and sampled-token scoring methods, finding KL Divergence strongest and Top-1 Probability Drop substantially weaker.

  • KL Divergence measures total change across the vocabulary distribution, whereas JSD provides a symmetric alternative and Top-1 Probability Drop tracks only the sampled token.All methods are evaluated with the same hyperparameters in the dependency-calculation ablation.
  • Top-1 Probability Drop trails KL Divergence by 7.2% in average accuracy, showing that sampled-token confidence is an insufficient proxy for visual reliance.The heuristic misses distributional changes affecting other likely candidate tokens.
  • JSD performs competitively at only 0.6% below KL Divergence, but KL retains the best result.The authors regard KL’s asymmetric information-gain interpretation as better aligned with measuring information supplied by the visual input.

H ABLATION STUDY ON ROLLOUT GROUP SIZE

The rollout-group ablation shows that increasing group size improves advantage estimation initially but yields diminishing returns, making G = 8 the preferred efficiency–performance balance.

  • Group size G controls a trade-off between advantage-estimation quality and data-generation cost in online RL.Larger groups provide more stable expected-reward estimates but require more computation.
  • Increasing G from 5 to 8 improves average performance by 1.0%, demonstrating the benefit of a more stable advantage estimate.The main experiments therefore use G = 8 rather than the smaller comparison setting.
  • G = 12 decreases average performance by 0.1%, while G = 16 improves it by only 0.2% over G = 8.These results show minimal additional benefit from larger rollout groups.
  • Because rollout computation scales linearly with group size, doubling the rollouts for a marginal gain is inefficient.G = 8 captures most potential performance gains without unnecessary rollout cost.
  • Soft Mask Calibration for Gradient Filtering: The soft-mask calibration matches the average token weight to a target filtering ratio of 0.4 through trajectory-level normalization and offset calibration.This controls the learning-signal magnitude when comparing graded and discrete token updates.

J ANALYSIS OF COMPUTATIONAL OVERHEAD

VPPO adds a modest computational overhead while improving learning efficiency under equal training time, but its validated scope remains limited to models up to 32B and reasoning-intensive benchmarks.

  • Computational Overhead: ~10% overhead remains consistent across 7B and 32B models, attributable to VPPO’s second forward pass for token perception.The additional pass calculates all token probabilities in parallel.
  • Equal-Budget Comparison: 2.5-point average gain under the same 17.0-hour budget shows VPPO outperforming DAPO through more efficient learning.The baseline stagnated despite receiving the same extended training time.
  • Limitation: VPPO’s computational cost is approximately a 10% increase, from 15.5 to 17 hours on the 7B setup.The authors characterize this as a favorable trade-off while noting interest in single-pass approximations.
  • Scope of Generalization: The method’s effectiveness has been demonstrated only up to 32B models, leaving performance at 72B-plus scales unverified.Its benefits were also demonstrated on reasoning-intensive benchmarks, while applicability to subjective or creative tasks remains open.
  • Methodological Assumptions and Hyperparameters: VPPO’s dependency calculation relies on image perturbation choices, and filtering and shaping hyperparameters may require retuning for new datasets or model scales.The paper specifically identifies masking strategy, filtering ratio k, and shaping range [βmin, βmax] as potentially domain-dependent.
Loading 2510.09285v2…