PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Entropy: Self-Diagnostic Multi-Role Token Optimization for Video Reasoning

Yudong Han, Yong Wang, Zaiquan Yang, Liang Lin, Chongyang Tao, Xiangxiang Chu, Liyuan Pan

arXiv:2610.03400v1cs.CV

TL;DR

Video RL still faces unresolved token-level credit assignment, while single-role selection and fixed counterfactuals provide limited ways to identify useful signals. DyCPO combines multi-role token selection with adaptive, rollout-derived counterfactuals. It reports strong performance across reasoning and general video-understanding benchmarks, including a Video-Holmes result above Video-KTR.

  • Problem

    RL with verifiable rewards still has ambiguous token-level credit assignment, and the applicability of high-entropy token selection to video reasoning remains uncertain.

    Source: §1 · p. 1, §3.1 · p. 3

  • Method

    DyCPO combines a multi-role dependency score for token selection with counterfactual signals derived adaptively from policy-linked rollout trajectories.

    Source: §1 · pp. 2–3, §3.3 · p. 6

  • Results

    DyCPO reports state-of-the-art results across three reasoning and two general video-understanding benchmarks, including 44.8% on Video-Holmes, 3.2% above Video-KTR.

    Source: §4.2 · p. 7

  • Takeaways & Limitations

    The paper concludes that joint multi-role credit assignment and self-diagnostic counterfactual optimization provide a robust RL foundation for multimodal video reasoning.

    Source: §1 · p. 3, §5 · p. 10

Abstract

from arXiv · show

Reinforcement learning with verifiable rewards has substantially advanced multimodal reasoning, yet it remains fundamentally limited by ambiguous token-level credit assignment. While high-entropy token heuristics encourage possibility exploration, naively extending them to video reasoning tends to induce lengthy reasoning, as the model becomes overly reliant on high-entropy visual activations. Alternative approaches that rely on counterfactual-based visual token localization for credit assignment also tend to over-prioritize visual exploration at the expense of decisive reasoning cues for answer derivation, thereby exacerbating the interference from spurious visual nuances. Moreover, these methods employ static counterfactual strategies that fail to co-evolve with the policy during training. In this paper, we introduce DyCPO, a co-evolutionary framework that jointly optimizes reliable token selection and adaptive counterfactual intervention. It constructs a multi-role dependence metric to balance visual exploration and answer-relevance mining in token-wise contrastive learning, while suppressing exploration-only filler tokens and spurious visual noise. Rather than relying on static counterfactual priors, DyCPO dynamically derives counterfactual signals from the model's own successful and failed rollouts, enabling self-diagnostic analysis and co-evolution of the optimization objective with the policy. Extensive experiments on complex video reasoning and general video understanding benchmarks demonstrate consistent performance improvements, establishing DyCPO as a robust token-level credit assignment paradigm for multimodal reinforcement learning.

1 INTRODUCTION

Video reasoning leaves token-level credit assignment unresolved: entropy-based selection and visual-only sensitivity each miss important signals. DyCPO combines visual and answer dependence with counterfactuals that adapt through policy-linked rollouts, and reports strong results across video benchmarks.

1 INTRODUCTION

  • GRPO's uniform trajectory-level reward leaves token-level credit assignment unresolved, while the transfer of high-entropy token heuristics to video reasoning remains uncertain.Prior multimodal RL can risk amplifying visual noise, and on-policy distillation relies on privileged annotations and has limited scalability.Source: §1 · p. 1
  • Naively rewarding entropy deteriorates video reasoning: high-entropy visual cues dominate throughout decoding, diverting updates from decisive content and contributing to over-reasoning.In ordinary decoding, low-entropy visual activations later provide stable object and spatio-temporal scaffolds that help maintain consistency and mitigate hallucination.Source: §1 · pp. 1–2
  • Single-rule selection has blind spots: top-20% entropy tokens miss nearly half of video-grounded tokens, while top-20% vision-sensitive tokens capture only 20% of answer signals.These mismatches motivate measuring visual dependence and answer dependence separately.Source: §1 · p. 2, Figure 1 · p. 2
  • Visual-sensitive tokens concentrate in early decoding, whereas answer-sensitive tokens appear in middle and late reasoning, marking complementary perceptual and logical roles.Visual counterfactual sensitivity decays across positions 0–40; answer-sensitive tokens localize around positions 60–80.Source: §1 · p. 2
  • DyCPO selects tokens sensitive to both visual and answer counterfactuals while retaining high entropy, and uses rollout signals to adapt counterfactual generation with the evolving policy.Its proposal network updates in lockstep with the policy model at each training step.Source: §1 · pp. 2–3
  • 44.8% accuracy on Video-Holmes outperforms GPT-4o at 42.0%, while DyCPO reports state-of-the-art or highly competitive performance across diverse video tasks.The reported evaluations span knowledge reasoning, general understanding, spatial perception, long-video benchmarks, and video hallucination.Source: §1 · p. 3

2 RELATED WORK

Video-centric RL methods address temporal dynamics through contrastive learning, tool-based pipelines, or scalable reward functions, but fine-grained reasoning remains difficult to model reliably. Related work also turns to token attribution and teacher-informed on-policy distillation.

2 RELATED WORK

  • Video-centric RL adds temporal contrastive learning, tool-based agent pipelines, or difficulty-aware regression, but these approaches largely retain coarse sequence rewards or simple heuristics.The cited examples are Feng et al. (2025) and Wang et al. (2025d), Zeng et al. (2026), and Park et al. (2025), respectively.Source: §2 · p. 3
  • Token-level attribution modeling and teacher-informed on-policy distillation are recent approaches to finer-grained credit assignment.The passage cites Wang et al. (2026b) for attribution and Wang et al. (2026a) for on-policy distillation.Source: §2 · p. 3
  • On-policy distillation can provide fine-grained supervision using a teacher's privileged information.Source: §2 · p. 3

3 METHODS

DyCPO diagnoses token behavior using visual and answer counterfactuals, then uses multi-role token selection and adaptive rollout-derived counterfactuals to shape policy optimization. Its state-aware strategy maintains higher token-to-answer correlation than fixed random masking, particularly later in training.

3.1 PRELIMINARIES

  • GRPO group-normalizes verifier rewards across sampled trajectories, but uniform token weighting leaves fine-grained credit assignment unresolved.Visual and answer counterfactual perturbations provide sensitivity signals for diagnosing token behavior alongside entropy and low-entropy visual-reliance measures.Source: §3.1 · p. 3

3.2 MULTI-ROLE CONTRASTIVE CREDIT ASSIGNMENT

  • Multi-role scores normalize and fuse visual and answer sensitivities, reweight them by token entropy, and select the top-k tokens for optimization.The score is designed to retain tokens informative for visual grounding and answer derivation while filtering low-informativeness noise-prone tokens.Source: §3.2 · p. 4
  • The token-level contrastive loss aligns selected-token predictions with an information-preserving view and separates them from an information-removing view.It is applied only to successful samples with positive advantages and added to the GRPO objective.Source: §3.2 · pp. 4–5

3.3 SELF-DIAGNOSTIC EVOLVING OPTIMIZATION

  • State-aware counterfactuals achieve consistently higher token-to-answer correlation than fixed random masking, particularly in later training stages.The paper presents this pattern as evidence that the state-aware strategy can co-evolve with the policy and prioritize answer-relevant visual clues.Source: §3.3 · p. 5, Figure 3 · p. 5
  • A slowly updated EMA proposal model supplies rollouts for contrastive localization using attention discrepancies between correct and incorrect trajectories.The method also contrasts their correct and incorrect answer texts, and anneals the keyframe count from fine-grained grounding toward broader temporal spans.Source: §3.3 · pp. 5–6

4 EXPERIMENTS

Experiments evaluate DyCPO across video-reasoning and understanding benchmarks, then ablate scoring and analyze curriculum, token patterns, visual rollouts, and reward-linked activation.

4.1 EXPERIMENTAL SETUP

  • Evaluation uses difficulty-filtered 3K RL subsets and seven benchmarks spanning reasoning, general understanding, long-video comprehension, spatial perception, and hallucination diagnosis.The benchmark set includes Video-Holmes, VideoMMMU, MMVU, TempCompass, VideoMME, LVBench, VSIBench, and VidHalluc.Source: §4.1 · pp. 6–7

4.2 MAIN RESULTS

  • DyCPO sets state-of-the-art results across three reasoning and two general benchmarks; versus PAPO it gains +6.5 on VideoMMMU (Qwen3-VL-4B) and +4.2 on MMVU (Qwen2.5-VL-7B).The reported comparison uses the same backbone for fairness.Source: Table 3 · p. 7, §4.2 · pp. 7–8

4.3 ABLATION STUDY

  • Ablations find complementary answer-aware and vision-aware signals improve performance, while entropy-only credit allocation degrades reasoning-oriented benchmarks.Removing Ai also degrades performance across benchmarks, with largest drops of 4.0% on VideoMME and 3.7% on Video-Holmes.Source: §4.3 · p. 8

4.4 ANALYSIS

  • On VidHalluc, DyCPO improves all hallucination metrics, including +1.1 on ACH MCQ versus w/o Jans/∆Hans and +5.4 on STH versus Only Entropy.The reported STH scores are 93.5 versus 88.1.Source: §4.4 · p. 8, Table 5 · p. 8
  • Curriculum learning raises TA from 28.5% to 31.6% while perception remains stable, as the visual receptive field expands from local keyframes toward global dependencies.TA denotes Timeline Analysis; TCI denotes Temporal Causal Inference.Source: §4.4 · p. 9, Figure 4 · p. 9
  • Vision-aware tokens favor semantic and color attributes, whereas answer-aware tokens also include conclusion-bearing function words such as “suggests” and “followed.”Figure 4(b) visualizes the two token distributions.Source: §4.4 · p. 9, Figure 4 · p. 9
  • For a Video-Holmes query, correct rollouts attend to frames 0–4 while incorrect rollouts neglect frames 14–15, where switch-related evidence is decisive.The passage identifies frames 9, 14, and 15 as critical visual cues.Source: §4.4 · p. 9, Figure 4 · p. 9
  • Token activation follows an M-shaped pattern as reward rises: very easy and very hard prompts suppress activation through vanishing advantages or sparse positive-advantage rollouts.The analysis identifies an intermediate reward band as important for avoiding these two failure modes.Source: §4.4 · pp. 9–10, Figure 5 · p. 9

5 CONCLUSION

The paper concludes that DyCPO combines multi-role token selection with self-diagnostic rollout-contrastive intervention, reporting robust gains across challenging video reasoning settings.

  • DyCPO combines multi-role token selection and self-diagnostic rollout-contrastive intervention in a token-level proposal-policy collaborative learning framework.The conclusion reports consistent effectiveness across backbone comparisons, ablations, and on-policy distillation comparisons, and state-of-the-art results on key video reasoning benchmarks.Source: §5 · p. 10

A MORE IMPLEMENTATION DETAILS

Implementation uses Qwen vision-language backbones, GRPO configurations, keyframe-driven contrastive perception, and a one-epoch video-QA RL training setup.

  • Ablative analysis uses Qwen2.5-VL-7B, Qwen3-VL-2B, and Qwen3-VL-4B, with Qwen2.5-VL-7B as the default benchmark-comparison base.Qwen2.5-VL models start from a VideoRFT SFT checkpoint; Qwen3-VL models start from official weights.Source: A MORE IMPLEMENTATION DETAILS · p. 14
  • The GRPO baseline uses symmetric clipping at 0.2, no reference KL penalty, no per-group advantage standard-deviation normalization, and zero entropy bonus.Source: A MORE IMPLEMENTATION DETAILS · p. 14
  • The Ours configuration computes keyframe-driven LMRC from four equally weighted visual and answer divergences and entropy shifts, then adds it to policy loss under advantage gating.It uses hard occlusion negatives and introduces no EMA self-distillation teacher.Source: A MORE IMPLEMENTATION DETAILS · p. 14
  • Keyframe counts follow linear curricula, and training runs one epoch on a 3K video-QA set with learning rate 1e−6 and eight responses per prompt.The keyframe schedule scales with frame budget: 4→8 for 16-frame inputs and 8→16 for 32-frame inputs.Source: A MORE IMPLEMENTATION DETAILS · p. 14, Table 6 · p. 14

B ADDITIONAL EXPERIMENTS

The additional experiments report gains on extended-horizon video understanding and test how frame density, feedback sources, token-selection weights, localization settings, and negative-view perturbations affect performance. Across these analyses, the results favor balanced multi-role selection and adaptive localization, with random region occlusion the strongest perturbation strategy overall.

Extended temporal horizons

  • 40.67% average accuracy on LVBench establishes a new state of the art, exceeding GRPO (40.03%) and entropy-only (39.70%).The advantage is particularly pronounced in Sport (+0.83% over GRPO), Self-Media (+2.84%), and Live (+1.15%), while GRPO slightly leads in Documentary and TV.Source: Table 7 · p. 15, B ADDITIONAL EXPERIMENTS · p. 15

Training with more frames

  • With f=32, Video-Holmes rises from 42.4% to 45.2% and VSI from 36.7% to 40.2% over GRPO; increasing f from 24 to 32 also raises VideoMME from 63.9% to 64.8% and VideoMMMU from 51.3% to 52.8%.The frame-count comparison reports modest but consistent gains from denser training inputs.Source: Table 8 · p. 15, B ADDITIONAL EXPERIMENTS · p. 15

Answer-aware feedback

  • 55.6% versus 53.4% average accuracy favors ground-truth answers as positive references over peer full-length rollouts, while direct wrong-answer negatives outperform privileged full-rollout negatives.Sampling random incorrect choices instead of erroneous model rollouts also degrades performance.Source: B ADDITIONAL EXPERIMENTS · pp. 15–16, Table 9 · p. 16

Parametric sensitivity to α

  • Balanced weights αvis=αans=0.25 perform best across benchmarks; skewing the ratio to 0.35:0.15 lowers accuracy universally, especially on reasoning-intensive tasks.The authors attribute this decline to answer-dominated tokens bypassing diverse visual exploration in intermediate reasoning.Source: Table 10 · p. 16, B ADDITIONAL EXPERIMENTS · p. 16

Sensitivity analysis on K1 and K2

High-attention frames from different layers

  • Pruning the final five layers consistently yields optimal performance, whereas relying exclusively on terminal layers generally degrades it.The authors use this result to support localizing key frames from rollouts using earlier layers.Source: B ADDITIONAL EXPERIMENTS · p. 16, Table 12 · p. 17

Perturbation strategies

  • Random region occlusion works best overall: scattered small rectangles create harder, better-calibrated negatives, while hotspot masking leaves contextual clues and weakens appearance-heavy task performance.Negative views corrupt rollout-selected target frames; positive views apply mild photometric jitter.Source: B ADDITIONAL EXPERIMENTS · pp. 17–18, Table 13 · p. 18

C TRAINING DYNAMICS

Training dynamics show that the perception objective converges and separates positive from negative views while remaining aligned with reward. The method also shortens responses, with added step time concentrated in actor updates.

Reward, response length, and loss analysis

  • The MRC loss briefly spikes during roughly the first 50 steps, then decreases monotonically and stabilizes near 0.03.The authors report that this convergence does not destructively interfere with policy-gradient updates.Source: C TRAINING DYNAMICS · p. 18, Figure 6 · p. 19
  • The positive-view distance remains about 0.015 below the negative-view distance throughout training, separating answer-consistent views from perturbed negatives.The reported distances are measured from the anchor representation.Source: C TRAINING DYNAMICS · p. 18, Figure 6 · p. 19
  • Reward tracks GRPO closely, while response length converges near 120 tokens versus GRPO’s 140–160 tokens.The authors attribute the shorter responses to discarding redundant reasoning while retaining evidence-grounded content.Source: C TRAINING DYNAMICS · p. 18, Figure 6 · p. 19

Training efficiency and computational overhead

  • Total step time increases by 22.3% (+8.5s on Qwen3-VL-2B) to 39.9% (+18.3s on Qwen3-VL-4B), with added latency concentrated in actor updates.Rollout generation times remain virtually unchanged; on 7B, the update increment is 18.1s versus 17.1s for the teacher forward pass.Source: C TRAINING DYNAMICS · pp. 18–19, Table 14 · p. 19

D ADDITIONAL ANALYSIS ON RELATED WORK

The comparison with Video-KTR distinguishes token selection from reward reweighting and highlights multi-dimensional token analysis and self-diagnostic masking. The authors report that their method outperforms Video-KTR across evaluation metrics.

Token selection vs. reward reweighting

  • Unlike Video-KTR’s score-based GRPO reweighting, the method uses scores to select salient tokens without explicitly changing their relative importance.The authors report that reweighting can amplify visual noise and bias the model toward the base policy.Source: D ADDITIONAL ANALYSIS ON RELATED WORK · p. 19

Multi-dimensional token analysis

  • The analysis quantifies token types and positional preferences, adds answer-aware perturbation, and traces high entropy to low-entropy visual activations.The authors also describe self-diagnostic curriculum masking as stabilizing training.Source: D ADDITIONAL ANALYSIS ON RELATED WORK · p. 19

Superior performance

Related papers

Loading 2610.03400v1…