PaperlayerView exact revision on arXiv

Source-linked AI summary

Anchoring What Matters: A Dual-Level Learning Framework for Visually-Grounded Multimodal Reasoning

Xinxin Song, Siyuan Li, Tingxiong Xiao, Jinli Suo

arXiv:2609.18057v1cs.AI

TL;DR

Standard on-policy multimodal RLVR discards valuable visually grounded trajectories and assigns uniform token advantages, weakening reinforcement of critical visual reasoning steps. PIVOT addresses this with self-calibrated experience replay and vision-guided advantage allocation. Across diverse benchmarks, it consistently improves LVLM multimodal reasoning performance and out-of-domain generalization.

  • Problem

    Standard multimodal RLVR fails to preserve informative visually grounded trajectories and dilutes critical perception and reasoning signals through uniform token-level advantage allocation.

  • Method

    PIVOT uses self-calibrated replay of high-quality visually grounded experiences as anchors and allocates fine-grained visual advantages according to local support and downstream reasoning impact.

  • Results

    PIVOT consistently improves multimodal reasoning capabilities and out-of-domain generalization across diverse benchmarks, data scales, and model scales.

  • Takeaways & Limitations

    PIVOT provides a dual-level framework for preserving and reinforcing sparse visual reasoning signals during LVLM reinforcement learning.

  • Takeaways & Limitations

    PIVOT introduces additional computational and storage overhead, and its scalability beyond vision-language models up to 7B parameters remains to be validated.

Abstract

from arXiv · show

Reinforcement learning with verifiable rewards (RLVR) has significantly improved the reasoning capabilities of large vision-language models (LVLMs). However, standard on-policy RLVR algorithms face a critical optimization bottleneck in preserving and reinforcing visually grounded reasoning behaviors: valuable visually-grounded reasoning trajectories are discarded after a single update, while uniform token advantage allocation prevents the model from reinforcing critical perception or reasoning steps. To bridge this gap, we propose PIVOT, a dual-level learning framework that anchors policy optimization around informative visual reasoning signals. Specifically, PIVOT introduces a self-calibrated experience replay mechanism, which selectively collects and replays visually-grounded historical experiences as stable reference anchors for policy optimization. Building upon this, we further design a vision-guided advantage allocation mechanism to allocate additional vision-aware advantages to tokens based on their local visual support and impact on downstream reasoning. Extensive experiments across diverse benchmarks demonstrate that PIVOT achieves highly competitive performance in enhancing the multimodal reasoning capabilities of LVLMs.

1 Introduction

PIVOT addresses multimodal RLVR’s two-level optimization bottleneck by preserving informative visually grounded trajectories and concentrating learning on critical visual reasoning tokens. Its dual-level design consistently improves multimodal reasoning performance over baselines across data and model scales.

  • PIVOT introduces self-calibrated experience replay that selectively replays high-quality visually grounded historical trajectories as retrospective anchors without directly constraining exploration.The dynamic experience pool preserves informative prompts and trajectories for policy optimization.
  • PIVOT further uses vision-guided advantage allocation based on tokens’ local visual support and downstream impact to strengthen valuable perception and reasoning steps.The mechanism obtains fine-grained visual learning signals through counterfactual support estimates and future reasoning impact.
  • Standard multimodal RLVR dilutes valuable visual reasoning signals by discarding informative trajectories and assigning uniform advantages across heterogeneous tokens.High-quality trajectories encode visual evidence into logical steps, while pivotal perception and reasoning tokens require stronger credit assignment.

2 Related Works

Prior multimodal RLVR work improves reasoning through high-quality data, perceptual augmentation, specialized rewards, and decomposition of visual and language reasoning. Related experience-based methods extend replay to language and vision-language models, motivating PIVOT’s focus on visually grounded historical trajectories.

  • Multimodal RLVR studies address reasoning through curated chain-of-thought data, perceptual diversity, task-specific rewards, and decomposed visual-language reasoning.
  • Experience-based reinforcement learning has expanded from classical replay to LLMs and LVLMs through collection-replay frameworks, stable off-policy objectives, and high-value trajectory selection.LVLM examples replay successful trajectories for difficult prompts.

3 Preliminaries

Multimodal RLVR generates reasoning chains but evaluates responses with a binary verifier based on final-answer correctness. Trajectory quality can therefore be assessed using complementary visual-dependency and entropy-based stability signals.

  • 3.1 Multimodal RLVR: An LVLM generates a multimodal response consisting typically of an intermediate multi-step reasoning chain followed by a final answer.
  • 3.1 Multimodal RLVR: RLVR assigns each response a binary reward according to whether its extracted final answer matches the ground truth.
  • 3.1 Multimodal RLVR: DAPO samples candidate responses, computes response advantages, and updates policy parameters with a clipped objective using importance-sampling ratios and clipping hyperparameters.
  • 3.2 Trajectory Quality Estimation: Trajectory quality combines visual dependency and reasoning stability: image corruption-based KL divergence measures visual grounding, while average token entropy reflects stability.Higher visual dependency indicates stronger reliance on visual evidence, and lower-entropy trajectories are associated with more stable, higher-quality reasoning.

4 Method

PIVOT extends RLVR with replayed historical anchors and vision-guided token advantages, preserving informative trajectories while concentrating optimization on visually critical reasoning steps.

  • 4 Method: PIVOT combines self-calibrated experience replay with vision-guided advantage allocation to preserve high-value experiences and prioritize visually critical tokens.The final objective retains the on-policy RLVR signal while anchoring optimization to historical experiences and visually critical tokens.
  • 4.1 Self-Calibrated Experience Replay: The experience pool stores each input with its empirical success probability and successful responses, then prioritizes frontier prompts through reward-variance sampling.This variance-driven selection focuses replay on partially solved problems.
  • 4.1 Self-Calibrated Experience Replay: For selected prompts, PIVOT filters successful trajectories by visual dependency and reasoning stability before choosing a historical experience for replay.The selection uses visual dependency and trajectory entropy in a two-stage filtering process.
  • 4.1 Self-Calibrated Experience Replay: Historical trajectories serve as stop-gradient reference anchors in a contrastive calibration loss rather than direct optimization targets.Correct explorations gain confidence, while incorrect trajectories are penalized until their confidence falls below the proven historical baseline.
  • 4.2 Vision-Guided Advantage Allocation: Token visual utility combines counterfactual image support with discounted future influence, while entropy gating emphasizes likely reasoning decision points.The future summation is truncated to a finite window, and the resulting utility modulates the originally uniform token advantages.
  • 4.3 Final Training Objective: The modified token advantages are integrated with the standard RLVR loss through a coefficient that balances on-policy learning and experience calibration.This preserves the on-policy signal while anchoring optimization to high-quality historical experiences and visually critical tokens.

5 Experiments

PIVOT is evaluated across model scales, datasets, benchmarks, training dynamics, ablations, out-of-domain tasks, hyperparameters, and RLVR algorithms. It consistently improves multimodal reasoning, while its components contribute through complementary trajectory-level anchoring and token-level visual utility allocation.

  • 5.2 Main Results: 51.25 and 57.85 average accuracy are achieved by PIVOT on Geometry3K with Qwen2.5-VL-3B and 7B, respectively, while VIRL39K training reaches 55.68.These correspond to 11.27% and 10.57% relative gains over DAPO for the 3B and 7B models; the VIRL39K result is highest across the compared baselines.
  • 5.3 Quantitative Analysis: PIVOT improves training rewards and validation accuracy on Geometry3K, indicating more efficient optimization and better generalization than the compared baselines.The validation trend is reported on unseen Geometry3K@test data for the Qwen2.5-VL-7B model.
  • 5.3 Quantitative Analysis: SER-only and VAA-only each improve over baseline, but their combination yields the best average performance across evaluation benchmarks.Using experiences as calibration anchors outperforms off-policy gradient or supervised fine-tuning reuse, while entropy-gated future visual utility improves over local utility alone.
  • 5.3 Quantitative Analysis: PIVOT improves out-of-domain performance on ScienceQA, HallusionBench, ChartQAPro, InfographicVQA, and RealWorldQA after training on Geometry3K.The reported results suggest broader multimodal generalization and help mitigate multimodal hallucinations.
  • 5.3 Quantitative Analysis: Moderate values of β and λexp perform best, because weak visual scaling reduces modulation while excessive scaling can destabilize optimization.λexp balances the experience calibration loss with the main RL objective.
  • 5.3 Quantitative Analysis: Integrating PIVOT into GRPO consistently improves multimodal reasoning, indicating that the framework is not confined to one RLVR algorithm.Additional analyses show concentration of optimization on visually grounded perception tokens and reasoning pivots, alongside greater attention to image tokens.

6 Conclusion

The conclusion presents PIVOT as a dual-level framework that preserves high-quality visual reasoning experiences and allocates visual advantages according to token-level support and downstream impact. Experiments show consistent improvements in multimodal reasoning and out-of-domain generalization, with broader multimodal agent tasks identified as future work.

  • 6 Conclusion: PIVOT preserves high-quality experiences as optimization anchors through self-calibration and allocates fine-grained visual advantages using local support and downstream reasoning impact.The framework addresses optimization bottlenecks at both the trajectory and token levels.
  • 6 Conclusion: Extensive experiments show that PIVOT consistently improves LVLM multimodal reasoning capabilities and out-of-domain generalization.The conclusion identifies broader multimodal agent tasks as a possible direction for future work.

Limitations

PIVOT’s promising results are accompanied by limitations involving training overhead, scale coverage, and empirical trajectory-quality selection.

  • Training PIVOT introduces additional computational and storage overhead from experience replay and counterfactual intervention, motivating more resource-efficient designs.The authors describe these overheads as acceptable but identify lightweight perturbation and more efficient designs as future work.
  • Experiments were limited to vision-language models up to 7B parameters, leaving scalability to larger models such as 32B variants unverified.
  • High-quality trajectory selection relies on visual dependency and trajectory entropy, which may not fully capture semantic correctness in complex multi-step reasoning.The authors suggest structured, rubric-based evaluation as a future way to define and filter high-quality trajectories.

Ethical Considerations

The work evaluates multimodal reasoning on public data without private or human-subject data, but deployment remains bounded by possible incorrect or biased outputs.

  • PIVOT is evaluated using public datasets and does not use private user data, personally identifiable information, or human-subject data.
  • Models trained with PIVOT may still produce incorrect or biased outputs under distribution shifts or in high-stakes real-world scenarios.
  • The authors recommend careful validation and human oversight before deployment in medicine, law, or safety-critical inspection.

A Justification of the Counterfactual Visual Support Score

The counterfactual visual support score is justified as a signed, zero-mean probability residual that concentrates optimization on visual evidence rather than uniformly rewarding divergent or position-favored tokens.

  • The score compares token probability under the original and corrupted images, identifying tokens with greater support when correct visual evidence is available.For sampled token y_t, c_t > 0 indicates stronger original-image support, whereas c_t ≈ 0 indicates similar support under both conditions.
  • The residual provides an unbiased estimate of the signed probability-mass difference p_t − q_t under the original-image policy.
  • Unlike a log-likelihood gap, the residual is zero-mean and redistributes optimization strength instead of injecting a uniformly positive reward.The log-likelihood gap has a non-negative expectation and may encourage longer responses or over-reward tokens from highly divergent prefixes.
  • As an additive token-level correction, the score raises logits for tokens favored by the original image and lowers logits for tokens favored by corrupted-image conditions or language priors.This gives the correction an optimization meaning tied specifically to visually contributed probability mass and suppression of visual hallucinations.
  • B Details of the Detrending Operator: The future-aware visual score can exhibit length-related bias, so response-wise position detrending removes location-explained variation while preserving token-specific future visual support.The detrended signal is intended to avoid rewarding tokens merely for appearing in favorable response positions.

C.1 Data and Evaluation

The study trains Qwen2.5-VL models on Geometry3K and ViRL39K, then evaluates multimodal reasoning across diverse mathematical, scientific, logical, and visual benchmarks.

  • Training Datasets: Training uses Geometry3K and the larger ViRL39K dataset to examine data scalability and broader visual-reasoning coverage.Geometry3K contains 3,002 geometry problems, while ViRL39K contains 38,870 verifiable question-answering instances spanning varied visual scenarios.
  • Evaluation Benchmarks: Evaluation spans MathVista, MathVerse, We-Math, and MMK12, covering visual mathematics, modality reliance, hierarchical knowledge, and K12 science reasoning.The benchmarks include diverse visual contexts and assess fine-grained understanding, compositional reasoning, generalization, and multiple school subjects.
  • Evaluation Benchmarks: Additional evaluation data include LogicVista, SuperCLEVR-Counting, MMMU-Pro, and MathVerse-V for logical, object-counting, robust multimodal, and vision-centric reasoning.

D.1 Ablation on Intervention Strategy

PIVOT’s visual intervention study finds random patch masking to be the strongest option among tested strategies, while analyses show that its visual utilities and attention patterns emphasize visually grounded reasoning.

  • D.1 Ablation on Intervention Strategy: All tested visual intervention strategies outperform DAPO, with random patch masking achieving the best average performance and becoming PIVOT’s default.Random patch masking avoids overly coarse whole-image removal while producing stronger corruption than Gaussian noise without severe pixel-level artifacts.
  • D.3 Layer-wise Attention Analysis: PIVOT assigns higher attention mass to image tokens across most layers than DAPO and the original Qwen2.5-VL-7B-Instruct model, especially in middle and later layers.The analysis provides complementary evidence that trajectory anchoring and token-level visual advantage allocation strengthen visual grounding, while attention alone does not fully characterize reasoning.
  • Computational Overhead: On Qwen2.5-VL-7B using four NVIDIA A100 80GB GPUs, PIVOT takes 860.8 seconds per step, a 23.2% overhead over DAPO.VPPO takes 830.4 seconds per step versus 698.8 seconds for DAPO, while PIVOT adds only modest overhead beyond VPPO.
Loading 2609.18057v1…