Source-linked AI summary
Where to Look Matters: On-Policy Self-Distillation for Long-Video Understanding
Kaishen Wang, Dongdi Zhao, Yijun Liang, Dingqiang Ye, Ruibo Chen, Heng Huang, Di Fu
TL;DR
Long-video VLMs can be distracted by question-irrelevant content in full-video inputs. Clue-OPSD trains a full-video student against a clue-conditioned self-teacher through on-policy next-token distribution matching, and consistently improves performance across benchmarks and model scales without answer labels or inference-time additions.
Problem
Full-video contexts for long-video question answering contain increasingly more question-irrelevant content, which can obscure evidence needed for specific questions.
Method
Clue-OPSD uses an annotated clue-conditioned EMA self-teacher to supervise a full-video student by aligning next-token distributions along student-generated trajectories.
Results
Clue-OPSD consistently improves over corresponding backbones across long-video benchmarks and Qwen3.5 scales, while achieving strong performance against supervised post-training baselines.
Takeaways & Limitations
Temporal clue intervals provide effective privileged visual supervision for long-video understanding without requiring ground-truth answer labels or additional inference-time modules.
Abstract
from arXiv · showhide
Vision-language models (VLMs) have made substantial progress in long-video understanding, with standard backbone models typically answering questions from frames sampled across the full video. However, as videos become longer, the full-video context inevitably contains more question-irrelevant temporal content, which can distract the model from the evidence needed to answer a specific question. We empirically find that focusing the visual input on short annotated clue intervals containing question-relevant evidence consistently improves prediction accuracy across model scales compared with using the corresponding full videos, while requiring fewer input frames. Based on this finding, we introduce Clue-OPSD, a clue-privileged on-policy self-distillation framework for long-video understanding. During training, a full-video student learns from a self-teacher conditioned on the corresponding clue interval by aligning their next-token distributions along student-generated trajectories. Clue-OPSD thus uses clue intervals as privileged supervision without relying on ground-truth answer labels, while requiring no clue annotations or additional modules at inference time. Extensive experiments across multiple long-video understanding benchmarks and Qwen3.5 model scales demonstrate consistent improvements over the corresponding backbone models and strong performance against supervised post-training baselines.
1 INTRODUCTION
Long-video question answering can be hindered by irrelevant temporal content in full-video inputs, while short clue intervals improve evidence presentation and motivate Clue-OPSD.
- 1 INTRODUCTION: Full-video inputs can obscure question-relevant events among unrelated temporal content, causing incorrect answers.In the representative example, the model fails on frames from a 3,409-second video but succeeds using the corresponding 10-second clue interval.
- 1 INTRODUCTION: Annotated clue intervals consistently improve prediction accuracy over corresponding full videos while requiring fewer input frames.The comparison uses matched FPS and maximum-frame constraints across Qwen3.5 model scales.
- 1 INTRODUCTION: Clue-OPSD trains a full-video student to match next-token distributions from an EMA self-teacher conditioned on the question’s annotated clue interval.Both branches are evaluated along student-generated prefixes, making the supervision on-policy.
- 1 INTRODUCTION: Across multiple benchmarks and Qwen3.5 scales, Clue-OPSD consistently improves over backbone models and performs strongly against supervised post-training baselines.The paper reports this pattern across model sizes and evaluation settings.
- 1 INTRODUCTION: The framework uses clue intervals as privileged visual supervision without ground-truth answer labels or an external teacher model.Inference retains the standard full-video VLM pipeline without additional modules.
2 RELATED WORK
Related work covers full-video VLM processing, active temporal search during inference, on-policy distillation, and multimodal extensions using asymmetric visual supervision.
- 2 RELATED WORK: Standard long-video VLMs sample frames across the full video and process them jointly with the textual query.Related work also explores efficient representations and compression strategies for long-video inputs.
- 2 RELATED WORK: Thinking-with-videos methods search for, retrieve, or temporally ground relevant segments during iterative reasoning.These methods typically involve multiple rounds of segment inspection or tool interaction, increasing pipeline complexity.
- 2 RELATED WORK: On-policy distillation supervises student-generated trajectories by matching teacher and student token-level predictive distributions.This reduces mismatch between training supervision and states encountered during autoregressive inference.
- 2 RELATED WORK: Multimodal extensions apply asymmetric or privileged visual supervision to images and video, including evidence-centered crops and temporal video grounding.Clue-OPSD instead targets long-video understanding.
3 METHOD
Clue-OPSD trains a full-video student with clue-conditioned supervision from an EMA self-teacher, aligning next-token distributions along student-generated trajectories. Clue intervals are used during training, while inference retains the standard full-video VLM pipeline.
- Inference: At inference, the clue interval and teacher are removed, and only the student operates with the standard full-video pipeline.Thus, clue annotations are used as training-time privileged supervision rather than as an inference-time input.
- Empirical Observation: Clue-interval input consistently achieves higher accuracy than full-video input across model scales while using fewer frames.The comparison uses identical FPS and maximum-frame constraints on sampled CG-Bench questions.
- Asymmetric Visual Contexts: Clue-OPSD compares full-video and clue-interval visual conditions while keeping the question and answer options identical.The student receives full-video frames, whereas the teacher receives frames from the corresponding annotated clue interval.
- On-Policy Distillation: The student first samples an on-policy response, which becomes the shared autoregressive trajectory for both branches.At each decoding step, both models evaluate next-token distributions on the same student-generated prefix.
- On-Policy Distribution Alignment: The student and teacher distributions are aligned at every decoding step using generalized Jensen–Shannon divergence.The teacher distribution is fixed during optimization, and gradients propagate only through the student branch.
- EMA Self-Teacher: The teacher is maintained as an exponential moving average of the student rather than introduced as a separately pretrained model.This creates a self-teacher while preserving the clue-conditioned visual context during training.
4 EXPERIMENTS
Experiments evaluate Clue-OPSD across Qwen3.5 scales, five long-video benchmarks, video durations, subtitle settings, and controlled training baselines. The method consistently improves backbone performance, with evidence that directly using question-relevant visual clues is more effective than answer-only, textual, or random privileged supervision.
- 4.1 EXPERIMENTAL SETTING: Training uses 5,000 CG-Bench multiple-choice instances covering 1,206 long videos, with valid annotated temporal clues.
- 4.1 EXPERIMENTAL SETTING: Experiments compare Qwen3.5-2B, 4B, and 9B using vanilla backbones, SFT, GRPO, and standard OPSD baselines across five benchmarks.The benchmarks are Video-MME, LVBench, LongVideoBench, MLVU, and MMVU.
- 4.2 EXPERIMENTAL RESULTS: 6.94, 3.45, 9.81, and 3.41 points are gained by Qwen3.5-2B on MLVU, LVBench, LongVideoBench, and MMVU over vanilla Qwen3.5.For Qwen3.5-4B, gains include 8.23 points on MLVU and 5.42 points on LVBench.
- 4.2 EXPERIMENTAL RESULTS: 1.16, 0.85, and 0.49 points separate Clue-OPSD from the strongest competing post-training baseline for Qwen3.5-2B, 4B, and 9B on average Video-MME.Clue-OPSD achieves the highest average Video-MME accuracy at all three scales.
- 4.4 DISCUSSION: Directly conditioning the teacher on annotated clue intervals outperforms textual-clue, random-interval, and other privileged teacher constructions.The discussion attributes the advantage to supervision generated from visual contexts emphasizing evidence relevant to the question.
5 CONCLUSION
The paper shows that short, question-relevant temporal clues can provide more effective visual context than full videos, then uses those clues to train a full-video student through clue-conditioned self-distillation. Across benchmarks and Qwen3.5 scales, Clue-OPSD consistently improves the original backbones and performs strongly against supervised post-training baselines.
- Short question-relevant clue intervals can outperform corresponding full-video inputs even with fewer frames.
- Clue-OPSD uses a clue-conditioned self-teacher to supervise a full-video student without ground-truth answer labels.
- Experiments across multiple benchmarks and Qwen3.5 scales show consistent backbone improvements and strong performance against supervised post-training baselines.
A.1 PROMPTS
The appendix provides benchmark evaluation prompt templates, including separate Video-MME prompts with and without subtitles and templates for LVBench, LongVideoBench, MLVU, and MMVU.
- The appendix presents prompt templates used for benchmark evaluation for reproducibility.
- Video-MME uses separate prompt templates for evaluations without subtitles and with subtitles.
- Additional templates are provided for LVBench, LongVideoBench, MLVU, and MMVU.
A.2 ALGORITHM
Clue-OPSD trains a full-video student using a clue-conditioned EMA teacher, aligning their next-token distributions along student-generated trajectories. The procedure updates only the student by distillation, then refreshes the teacher and rollout model iteratively.
- On-policy rollout: The full-video student first generates an on-policy response, whose prefixes are reused to evaluate both student and teacher distributions.Both models predict next tokens conditioned on the same student-generated prefixes.
- Distribution alignment: JSD distillation aligns the student and clue-conditioned teacher next-token distributions, with gradients applied only to the student.The algorithm constructs top-K compressed distributions before accumulating the instance-level distillation loss.
- Model updates: After each update, the teacher parameters follow an exponential moving average of the student, and the rollout model synchronizes with the updated student.The procedure returns the final student parameters after completing the optimization loop.
- Training inputs: Each instance uses full-video frames for the student and frames from an annotated clue interval for the teacher.The clue interval supplies privileged visual context during training.
- Optimization: The loss is averaged over the mini-batch before back-propagating the update to the student parameters.This produces the optimization step used in each training iteration.