Source-linked AI summary
Video-OPSD: Exploiting Privileged Visual Evidence for On-Policy Self-Distillation in Video Large Language Models
Ziyue Wang, Shiqi Huang, Weiwen Xu, Bihan Wen, Xudong Jiang
TL;DR
Video-OPSD addresses the limited exploration of on-policy self-distillation for Video-LLMs by using privileged visual evidence within long videos. It builds an evidence-grounded teacher and evidence-guided token optimization, consistently outperforming Standard OPSD across backbones and matching GRPO-level performance with less training time.
Problem
OPSD remains largely underexplored for Video-LLMs, despite providing dense token-level supervision through a privileged self-teacher.
Method
Video-OPSD conditions its self-teacher on temporally localized evidence frames and weights distillation according to each reasoning token’s reliance on visual evidence.
Results
Video-OPSD consistently improves over Standard OPSD across three Qwen-VL backbones and all 15 backbone–benchmark comparisons, while achieving performance comparable to GRPO.
Takeaways & Limitations
Video-OPSD provides an effective and efficient post-training approach for Video-LLMs, using dense supervision from one student rollout and teacher replay.
Takeaways & Limitations
The teacher’s visual privilege depends on removing non-evidence context while retaining signals accessible to the student.
Abstract
from arXiv · showhide
On-policy self-distillation (OPSD) has recently emerged as an effective post-training paradigm that improves policy optimization through dense token-level supervision from a privileged self-teacher. Despite its promise, OPSD remains largely underexplored for Video Large Language Models (Video-LLMs). Existing methods typically construct privileged teachers by augmenting their context with additional information while keeping the primary input unchanged for both teacher and student. Video reasoning, however, offers a distinct source of privileged supervision within the primary input itself: long videos contain substantial temporal redundancy, and only a small subset of frames provides the evidence necessary to answer a question. Building on this observation, we present $\textbf{Video-OPSD}$, an OPSD framework that exploits privileged visual evidence for both self-teacher construction and knowledge transfer. First, our Evidence-Grounded Self-Teacher conditions the teacher exclusively on annotated evidence frames while the student continues to reason over the complete video. This focused visual input enables the teacher to provide more informative supervision. Second, our Evidence-Guided Token Optimization adaptively weights token-level distillation according to each reasoning token's reliance on privileged visual evidence, thereby emphasizing perceptually grounded reasoning. Experiments across video understanding and reasoning benchmarks show that $\textbf{Video-OPSD}$ consistently improves upon Standard OPSD across multiple backbones and achieves performance comparable to GRPO while requiring substantially less training time, establishing an effective and efficient post-training approach for Video-LLMs.
Introduction
Figure 1 contrasts conventional OPSD with Video-OPSD’s evidence-driven design. The comparison highlights shared inputs and uniform distillation versus an evidence-grounded teacher and evidence-guided token optimization.
- Conventional OPSD gives the teacher and student the same primary input.
- Conventional OPSD applies token-level distillation uniformly.
- Video-OPSD combines an Evidence-Grounded Self-Teacher with Evidence-Guided Token Optimization.
Related Work
Related work has applied reinforcement learning and on-policy distillation to improve Video-LLM reasoning. OPSD removes the external teacher by constructing a privileged self-teacher from the same policy, typically through additional contextual information.
- Reinforcement Learning for Video-LLMs: Reinforcement-learning methods for Video-LLMs include temporal, spatio-temporal, and fine-grained perception reasoning strategies.
- Reinforcement Learning for Video-LLMs: Recent studies explore on-policy distillation for temporal video grounding and video reasoning, complementing sparse rewards with dense token-level learning.
- OPSD constructs a privileged self-teacher from the same policy, eliminating the need for an external teacher.
- Existing OPSD approaches typically create privileged supervision by enriching the teacher with additional contextual information.
Method
Video-OPSD constructs a privileged self-teacher from annotated evidence frames while the student reasons over the broader video, then transfers evidence-grounded supervision through adaptively weighted token distillation. A correctness gate filters unreliable teacher predictions before the student-only optimization objective is applied.
- Framework Overview: The student and frozen self-teacher share initialization, but the teacher receives privileged context containing evidence-focused visual input and the gold answer.The student is conditioned on cS=(V^S,q), while the teacher uses cT=(V^T,q,a).
- Framework Overview: The student generates one on-policy trajectory, which the teacher replays under privileged context using the same student-generated prefixes.This produces dense next-token supervision without autoregressively generating a separate teacher response.
- Evidence-Grounded Self-Teacher: Evidence-Grounded Self-Teacher restricts the teacher to annotated evidence frames while retaining those frames unchanged in the student’s broader video input.The student additionally receives frames sampled from the remaining video, so teacher-visible visual signals remain accessible to the student.
- Evidence-Guided Token Distillation: Evidence-Guided Token Optimization estimates each token’s visual-evidence reliance from teacher attention and uses normalized scores to weight token-level distribution matching.Attention is summed over visual tokens and averaged across heads in the last three layers; stronger reliance receives greater weight while all tokens remain included.
- Correctness Filtering: A correctness gate retains examples only when the privileged teacher’s greedy response passes rule-based answer extraction, excluding incorrect or unparsable responses.The gate addresses the fact that privileged conditioning does not guarantee reliable teacher predictions.
- Evidence-Guided Token Distillation: The objective computes generalized JSD over the teacher’s Top-K vocabulary support, treating teacher distributions and evidence-guided weights as fixed targets.Gradients propagate only through the student branch, while Top-K truncation reduces computation with negligible approximation error according to the passage.
Experiments
Across five video benchmarks and three Qwen-VL backbones, Video-OPSD consistently outperforms Standard OPSD through evidence-grounded teacher construction and evidence-guided token weighting. It also approaches GRPO performance with substantially lower training cost.
- Main Results: Video-OPSD improves Standard OPSD by 2.1, 2.0, and 1.3 average points on Qwen2.5-VL-7B, Qwen3-VL-8B, and Qwen3-VL-4B, respectively.It achieves higher performance in all 15 backbone–benchmark comparisons.
- Main Results: Video-OPSD is comparable to GRPO, slightly outperforming it on Qwen2.5-VL-7B and Qwen3-VL-8B while matching Qwen3-VL-4B.The comparison is based on average benchmark scores.
- Efficiency: 2.2 hours versus 5.5 hours reduces Video-OPSD training time to 40% of GRPO’s under Qwen2.5-VL-7B.The framework uses one student rollout followed by teacher replay for dense token-level supervision.
- Ablation Studies: 70.4% inference accuracy with evidence-only frames falls to 64.9% with added context and 49.8% with uniform sampling.Evidence frames improve teacher accuracy by 20.6 percentage points over uniform sampling, even with gold-answer conditioning.
- Ablation Studies: Soft window-attention weighting reaches 51.0 average score and outperforms full updating, entropy weighting, and hard top-30% selection by 1.0, 0.7, and 0.9 points.The strategy weights all generated positions according to their attention to the ground-truth temporal evidence window.
Conclusion
Video-OPSD exploits privileged visual evidence through grounded teacher construction and evidence-guided token optimization. Across five benchmarks, it improves over supervised fine-tuning and Standard OPSD, matches GRPO comparably, and uses one student rollout.
- Video-OPSD uses temporally localized evidence frames for teacher construction and weights supervision by each reasoning token’s visual-evidence reliance.
- Across five video understanding and reasoning benchmarks, Video-OPSD consistently improves over supervised fine-tuning and Standard OPSD.
- Video-OPSD achieves performance comparable to GRPO using only a single student rollout per training example.
- Ablations validate that evidence-grounded teacher construction improves privileged supervision quality and evidence-guided weighting enables more effective knowledge transfer.