PaperlayerView exact revision on arXiv

Source-linked AI summary

Uncovering Shortcut Amplification in Video RLVR through the Lens of Perceptual Uncertainty Collapse

Jiafeng Liang, Xianhao Ou, Tao Ren, Rongchuan Mu, Runxuan Liu, Dandan Tu, Ming Liu, See-Kiong Ng, Bing Qin

arXiv:2610.37211v1cs.CVcs.CL

TL;DR

Outcome-level video RLVR can amplify text-prior answers even when video evidence does not support them. The paper analyzes this uncertainty collapse and introduces BSPO, which combines reasoning-sensitive credit assignment with visual perturbation. BSPO improves video reasoning performance and strengthens visual faithfulness.

  • Problem

    The paper asks whether video RLVR’s performance gains reflect stronger visual grounding or increased confidence driven by textual priors.

    Source: §1 · p. 1, §2.1 · p. 2

  • Method

    BSPO combines principles derived from correct and incorrect rollouts with comparisons between predictions under original and visually perturbed videos to regulate policy updates.

    Source: §1 · p. 2, §3 · p. 4, §3.2 · p. 4

  • Results

    BSPO achieves the best performance among compared methods across six video understanding and reasoning benchmarks on Qwen3-VL-4B and Qwen3-VL-8B, while strengthening visual faithfulness.

    Source: §1 · p. 2, §4.2 · p. 6, §6 · p. 7

  • Takeaways & Limitations

    The analyses show that outcome-level RL can increase confidence while making uncertainty reduction less attributable to visual evidence, motivating updates that preserve visual dependence.

    Source: §1 · p. 2, §6 · p. 7

Abstract

from arXiv · show

Reinforcement learning with verifiable rewards (RLVR) has emerged as an effective post-training paradigm for enhancing the video reasoning capability of Large Multimodal Models (LMMs). Despite its empirical effectiveness, we show that outcome-level optimization can introduce an undesirable shortcut amplification effect, whereby models increasingly favor text-prior answers even when they are not supported by the visual evidence. To characterize this behavior, we investigate the training dynamics of video RLVR through the lens of perceptual uncertainty, analyzing how predictive uncertainty evolves under video-conditioned and text-only contexts. Our analysis identifies a perceptual uncertainty collapse phenomenon, in which RL training continuously sharpens the model's answer distribution while the uncertainty reduction attributable to visual evidence progressively diminishes. A further gradient-direction analysis reveals that such confidence enhancement becomes increasingly less aligned with visually grounded uncertainty reduction and is instead dominated by text-prior compression. Motivated by these findings, we propose Bifocal Self-distillation Policy Optimization (BSPO), a self-distillation framework that regulates policy updates with two complementary distributional signals. Specifically, BSPO derives empirical reasoning principles from intra-group correct and incorrect rollouts to identify reasoning-sensitive tokens, and introduces shortcut-aware visual perturbation to penalize predictions that remain invariant when task-relevant visual evidence is weakened. By adaptively reweighting the original RL advantage, BSPO preserves the task-solving benefits of RLVR while promoting updates that are both reasoning-sensitive and visually grounded. Extensive experiments validate the effectiveness of BSPO in improving video reasoning performance and strengthening visual faithfulness.

1 Introduction

The introduction frames video RLVR as useful for reasoning but shows that outcome-level training can reinforce text-prior answers unsupported by video. It analyzes this shortcut amplification through perceptual uncertainty and introduces BSPO to promote reasoning-sensitive, visually grounded updates.

1 Introduction

  • RLVR uses rule-based outcome rewards to improve LMM reasoning without requiring dense human annotations for intermediate reasoning steps.The introduction notes its potential for video reasoning tasks.Source: §1 · p. 1
  • On the faithful set, the original video is retained; on the counterfactual set, key segments are replaced while the question and options stay fixed and the shortcut answer becomes favored by textual priors.The counterfactual set adds “None of the above” as the correct answer, and the performance gap between sets narrows during training.Source: §1 · pp. 1–2
  • Outcome-level RL training can sharpen predictions while reducing uncertainty less through visual evidence, with confidence increasingly dominated by text-prior compression.The introduction calls this perceptual uncertainty collapse and reports that visual-specific compression weakens as training proceeds.Source: §1 · p. 2
  • BSPO combines principle-guided credit assignment from correct and incorrect rollouts with shortcut-aware visual perturbation to encourage reasoning-sensitive, visually grounded updates.Its perturbation component penalizes predictions that remain invariant when task-relevant visual evidence is weakened.Source: §1 · p. 2
  • BSPO preserves the visual-conditioning entropy gap and mitigates the decline in visual-specific compression, indicating more visually faithful reasoning.The method is presented as retaining RLVR’s reasoning benefits while suppressing shortcut-driven optimization.Source: §1 · p. 2

2 Rethinking Video RLVR from a Perceptual Uncertainty Perspective

The section examines whether video RLVR’s performance gains reflect stronger visual grounding or increased confidence from textual priors. Its uncertainty and gradient analyses find that confidence rises as visual-attributable uncertainty reduction declines and updates increasingly reflect text-prior sharpening.

2.1 Background and Motivation

  • The section asks whether video RLVR gains reflect visual grounding or confidence driven by textual priors, in a setting where a binary verifier rewards correct answers and output format.The described RLVR objective uses a group-normalized advantage, while its scalar reward measures correctness and output format.Source: §2.1 · p. 2

2.2 Evidence of Perceptual Uncertainty Collapse

  • The analysis compares video-conditioned and text-only predictive entropy under teacher forcing, using their entropy gap to quantify uncertainty reduction induced by visual input.Ground-truth responses provide a controlled comparison between the two contexts.Source: §2.2 · pp. 2–3
  • Video-conditioned entropy falls while the visual-conditioning gap also declines, so increased confidence is not accompanied by stronger visual evidence utilization.The section calls this mismatch perceptual uncertainty collapse.Source: §2.2 · p. 3, Figure 2 · p. 3

2.3 Mechanistic Analysis of RL-Induced Uncertainty Compression

  • Visual-specific compression diminishes as RL sharpens predictions, with increased confidence becoming less attributable to visual input and increasingly reflecting text-prior sharpening.The analysis links this pattern to the declining visual-conditioning entropy gap.Source: Figure 2 · p. 3, §2.3 · p. 4

3 Bifocal Self-distillation Policy Optimization

BSPO regulates video-RLVR updates with two complementary signals derived from the model’s own distributional variations. Its components target reasoning-sensitive credit assignment and visual dependence while retaining outcome-reward optimization.

3 Bifocal Self-distillation Policy Optimization

  • BSPO combines PCA and SCP, deriving supervision from the model’s distributional variations rather than external annotations or a separate teacher.PCA guides reasoning refinement; SCP penalizes predictions that remain overly stable when visual evidence is weakened.Source: §3 · p. 4

3.1 Principle-guided Credit Assignment

  • PCA contrasts correct and incorrect rollouts to derive empirical reasoning principles, then uses principle-conditioned distribution shifts to identify reasoning-sensitive tokens.The principles extract reusable patterns from successful trajectories and contrast them with failure modes; smaller divergence indicates stronger alignment with the guidance.Source: §3.1 · p. 4

3.2 Shortcut-aware Contrastive Perturbation

  • SCP compares policy distributions on original and perturbed videos, using weakened task-relevant visual evidence as a counterfactual probe of visual dependence.The question and generated prefix remain unchanged during perturbation, isolating the effect of degraded visual evidence on the policy distribution.Source: §3.2 · p. 4, §3 · p. 4

3.3 Bifocal Optimization

  • Bifocal optimization combines normalized PCA and SCP signals to adapt token-level RL advantages while preserving their trajectory-level signs.The signals reallocate update strength according to reasoning sensitivity and visual dependence; PCA favors smaller divergence, whereas SCP favors larger divergence from the perturbed-video reference.Source: §3.3 · pp. 4–5

4 Experiment

Across six video benchmarks and two model scales, BSPO outperforms the compared RLVR and on-policy self-distillation baselines. Its ablations and training analyses indicate that reasoning-sensitive credit assignment and visual perturbation contribute complementary signals for preserving visually grounded reasoning.

4.1 Experimental Setups

  • Experiments use Qwen3-VL-4B and Qwen3-VL-8B across six benchmarks covering long-video, general-video, and complex video reasoning.GRPO, DAPO, SDPO, and VISD are comparison baselines trained under the same backbone, data, reward, and evaluation protocol.Source: §4.1 · p. 5

4.2 Experiment Result

  • BSPO achieves the best performance among compared methods on Qwen3-VL-4B and Qwen3-VL-8B, with consistent gains across benchmark categories.The reported gains on long-video and complex-reasoning benchmarks suggest particular benefit when tasks require temporally distributed evidence and fine-grained visual inference.Source: §4.2 · p. 6

4.3 Ablation Study

  • Removing PCA lowers reasoning-intensive benchmark performance, while removing SCP causes larger degradation in visually grounded evaluation, especially under counterfactual settings.Reasoning-focused credit assignment alone does not prevent shortcut reliance when visual evidence is weakened.Source: §4.3 · p. 6
  • Among SCP perturbations, shuffle is more balanced than crop or replacement because it disrupts temporal order while preserving frame-level semantics.Crop may remove task-relevant content too directly, whereas replacement can introduce distributional shifts.Source: §4.3 · p. 6

4.4 Analysis

  • GRPO lowers video-conditioned entropy while its visual-conditioning entropy gap declines; BSPO lowers video-conditioned entropy while keeping text-only entropy relatively stable and enlarging the gap.Removing SCP weakens gap preservation, and removing PCA also degrades the trend.Source: §4.4 · p. 6
  • GRPO’s visual-specific compression declines rapidly, whereas BSPO maintains higher compression and recovers later; removing SCP brings it close to GRPO, while removing PCA leaves it below full BSPO.The reported measure compares alignment of RL-induced sharpening with video-conditioned entropy reduction versus text-only compression.Source: §4.4 · pp. 6–7
  • During training, BSPO enlarges the accuracy gap between faithful and counterfactual subsets, improving visually valid examples while suppressing shortcut answers when their evidence is invalidated.The passage describes this as mitigating shortcut amplification and strengthening dependence on visual evidence.Source: §4.4 · p. 7

5 Related Work

Prior video-RLVR work emphasizes performance gains, while evidence about whether those gains remain grounded in visual evidence is limited. This paper analyzes that question through perceptual uncertainty and proposes BSPO to preserve visual dependence during training.

RLVR for Video Reasoning

OSPD for Video Reasoning

  • On-policy self-distillation methods use rollout-derived signals, policy differences, privileged guidance, or token-level feedback to provide denser supervision than final-answer rewards (Pan et al. 2026).The cited approaches address coarse credit assignment, but most do not explicitly examine whether confidence remains visually grounded.Source: §5 · p. 7
  • BSPO differs from the described prior approaches by coupling self-distillation with counterfactual visual perturbation to guide updates by reasoning sensitivity and visual dependence.The accompanying figure caption describes accuracy trends on faithful and counterfactual sets during training.Source: §5 · p. 7, Figure 4 · p. 7

6 Conclusion

The paper identifies shortcut amplification in video RLVR and analyzes it through perceptual uncertainty. It proposes BSPO, combining principle-guided credit assignment with shortcut-aware perturbation, and reports improved video reasoning performance and visual faithfulness.

6 Conclusion

  • The paper reports that outcome-level RL can increase confidence while making uncertainty reduction less attributable to visual evidence, and proposes BSPO to promote reasoning-sensitive, visually dependent updates.BSPO combines principle-guided credit assignment with shortcut-aware contrastive perturbation; experiments report improved performance and visual faithfulness during RL training.Source: §6 · p. 7

A Additional Implementation Details

The appendix specifies the default SCP frame perturbation and the controlled setup used for entropy and gradient analyses.

A Additional Implementation Details

  • SCP perturbs 20% of input frames; the default shuffle randomly permutes selected frames, while crop, replacement, and blank-frame replacement are alternatives.Crop retains a random region covering 50% of each selected frame and resizes it; replacement uses blank frames.Source: A Additional Implementation Details · p. 10
  • Entropy and gradient analyses use Video-MME with Qwen3-VL-8B, evaluate checkpoints on the same samples and ground-truth responses under teacher forcing, and average results over three seeds.The passage also notes that the principle-summary prompt is shown in Figure 5.Source: A Additional Implementation Details · p. 10

B.1 Interpretation of the Visual-conditioning Entropy Gap

The visual-conditioning entropy gap measures how much video conditioning reduces next-token predictive uncertainty under the same textual context. Its decline alongside decreasing video-conditioned entropy indicates confidence is increasing while a smaller share of that confidence is attributable to video.

B.1 Interpretation of the Visual-conditioning Entropy Gap

  • With identical question and response prefix, the difference between video-conditioned and text-only entropy is induced by the video condition.Source: §B.1 · p. 10
  • A declining visual-conditioning entropy gap alongside declining video-conditioned entropy means predictions become more confident while a smaller share of that confidence is attributable to video.The gap is an output-level measure of video’s contribution to confidence formation, not a measure of visual information in hidden representations.Source: §B.1 · p. 10
  • Perceptual uncertainty collapse denotes predictive uncertainty compression without a corresponding increase in uncertainty reduction attributable to visual evidence.It does not refer to an observed collapse of an explicit visual latent variable.Source: §B.1 · p. 10

B.2 Teacher-forced Gradient Analysis as a Controlled Update Probe

The teacher-forced gradient analysis is a controlled probe of how a policy-gradient direction locally changes predictive uncertainty. Fixing the ground-truth token and prefix removes trajectory-level variation, but the probe does not exactly reconstruct stochastic RLVR updates.

B.2 Teacher-forced Gradient Analysis as a Controlled Update Probe

  • The gradient analysis uses a fixed ground-truth trajectory as a controlled probe, not an exact reconstruction of each stochastic RLVR update.Source: §B.2 · p. 10
  • Teacher forcing fixes the ground-truth token and prefix so measured gradient-geometry changes reflect policy parameters or conditioning modality rather than different sampled reasoning paths.Source: §B.2 · p. 10
  • The local update-response probe measures whether increasing the ground-truth continuation’s likelihood would compress or expand predictive uncertainty at the current checkpoint.It enables comparison of video-conditioned and text-only uncertainty-reducing directions under identical semantic supervision.Source: §B.2 · p. 10

Related papers

Loading 2610.37211v1…