Source-linked AI summary
VISD: Enhancing Video Reasoning via Structured Self-Distillation
Hao Lin, Kunyang Lv, Xu Jiang, Jingqi Tian, Zhongjing Du, Jiayu Ding, Qiaoman Zhang, Hongbo Jin
TL;DR
VideoLLM training struggles to assign fine-grained credit across long, temporally grounded reasoning trajectories: RLVR is sparse, while existing self-distillation lacks structure and can interact unstably with RL. VISD uses a video-aware judge and structured privileged feedback to guide token-level supervision, decoupling update direction from magnitude; it consistently improves performance and reaches these gains with nearly 2× faster convergence.
Problem
VideoLLM training lacks fine-grained credit assignment for long reasoning trajectories because RLVR is sequence-level, while self-distillation is insufficiently structured and diagnostically specific.
Method
VISD uses a video-aware judge to generate multidimensional privileged feedback for a teacher policy, while direction–magnitude decoupling integrates token-level supervision with reinforcement learning.
Results
VISD consistently outperforms strong baselines, improving answer accuracy and spatio-temporal grounding quality while achieving nearly 2× faster convergence in optimization steps.
Takeaways & Limitations
Structured self-distillation provides semantically aligned credit assignment that improves reasoning faithfulness and training efficiency for complex video reasoning.
Takeaways & Limitations
The current method focuses mainly on video reasoning with explicit temporal and spatial grounding signals, leaving richer structured outputs and open-ended interactive tasks for future work.
Abstract
from arXiv · showhide
Training VideoLLMs for complex reasoning remains challenging due to sparse sequence level rewards and the lack of fine grained credit assignment over long, temporally grounded reasoning trajectories. While reinforcement learning with verifiable rewards (RLVR) provides reliable supervision, it fails to capture token level contributions, leading to inefficient learning. Conversely, existing self distillation methods offer dense supervision but lack structure and diagnostic specificity, and often interact unstably with reinforcement learning. In this work, we propose VISD, a structured self distillation framework that introduces diagnostically meaningful privileged information for video reasoning. VISD employs a video aware judge model to decompose reasoning quality into multiple dimensions, including answer correctness, logical consistency, and spatio-temporal grounding, and uses this structured feedback to guide a teacher policy for token level supervision. To stably integrate dense supervision with RL, we introduce a direction magnitude decoupling mechanism, where rollout level advantages computed from rewards determine update direction, while structured privileged signals modulate token level update magnitudes. This design enables semantically aligned and fine grained credit assignment, improving both reasoning faithfulness and training efficiency. Additionally, VISD incorporates curriculum scheduling and EMA based teacher stabilization to support robust optimization over long video sequences. Experiments on diverse benchmarks show that VISD consistently outperforms strong baselines, improving answer accuracy and spatio temporal grounding quality. Notably, VISD reaches these gains with nearly 2x faster convergence in optimization steps, highlighting the effectiveness of structured self supervision in improving both performance and sample efficiency for VideoLLMs.
1 Introduction
VISD addresses the difficulty of assigning fine-grained credit in long-horizon video reasoning by combining structured diagnostic supervision with reinforcement learning. Its design targets more faithful reasoning and faster, more stable optimization.
- Motivation: VideoLLM training must assess both final-answer correctness and the faithfulness of intermediate reasoning against temporally evolving visual evidence.
- Limitations of Existing Methods: RLVR provides reliable sequence-level optimization signals but cannot distinguish heterogeneous token contributions or subtle temporal and grounding errors.
- Limitations of Existing Methods: Existing self-distillation methods offer dense supervision but often lack diagnostic specificity and interact unstably with reinforcement learning.
- VISD: VISD uses a video-aware judge to evaluate answer correctness, reasoning consistency, and spatio-temporal grounding, then conditions a teacher policy for token-level guidance.
- VISD: Rollout-level rewards determine update direction, while structured privileged information modulates token-level update magnitude for semantically aligned credit assignment.
- Optimization: Curriculum scheduling and EMA teacher stabilization support robust optimization over complex, long video sequences.
- Results: Nearly 2× faster convergence in optimization steps accompanies consistent gains over strong baselines in answer accuracy and spatio-temporal grounding quality.
2 Related Work
Related work places VISD alongside VideoLLMs, reinforcement learning with verifiable rewards, and on-policy or self-distillation methods for fine-grained credit assignment.
- VideoLLMs: VideoLLMs extend vision-language models to dynamic visual understanding through frame sampling, temporal aggregation, and video-specific instruction tuning.
3 Method
VISD augments on-policy reinforcement learning with structured, video-aware privileged feedback to assign semantically aligned credit at token level. It separates reward-driven update direction from teacher-guided update magnitude, while curriculum scheduling and EMA stabilization support long-video optimization.
- Rollout-level reward advantages determine policy-update direction, while teacher-student discrepancies modulate token-level update magnitudes.
- The framework adds a privileged-information pathway and a feedback-conditioned teacher that reassesses the student’s sampled trajectory token by token.
- VISD uses a video-aware judge to generate structured feedback that captures answer correctness, reasoning consistency, and spatio-temporal grounding.
- Structured top-K teacher-student discrepancies encode temporal, spatial, and logical corrections within a locally relevant candidate set.
- A curriculum gradually shifts training from structured self-distillation toward reinforcement learning, interpolating efficient early learning with stable long-term convergence.
- EMA teacher parameters reduce high-frequency fluctuations, while the clipped surrogate objective preserves policy-optimization stability without a separate distillation loss.
4 Experiments
VISD is evaluated across diverse video reasoning and grounding benchmarks, where it consistently improves accuracy, grounding quality, and general video performance over strong baselines. Ablations attribute these gains to structured feedback, EMA stabilization, and local top-K token support.
- Main Results: VISD improves answer accuracy over Qwen2.5-VL-7B by +28.4 points and achieves the best overall V-STAR scores.Its V-STAR mAM/mLGM improve over VisionCoach from 34.3/47.5 to 35.1/48.9.
- Main Results: VISD outperforms Qwen2.5-VL-7B by +7.4 average points and VisionCoach by +1.8 average points across broader video benchmarks.The strongest gains occur on long-video reasoning and temporal grounding metrics, while Video-MME-v2 non-linear score improves from 8.6 to 9.4 over VisionCoach.
- Main Results: VISD improves R@0.3, R@0.5, and mIoU over Open-o3-Video and VisionCoach on zero-shot Charades-STA temporal grounding.This evaluates moment localization without task-specific fine-tuning on the target benchmark.
- Ablation Studies: Structured judge feedback improves final reward from 2.280 to 2.318 and answer accuracy from 0.565 to 0.571 over verified answer-side information alone.The ablation indicates that diagnostic feedback changes token-level update magnitudes while reinforcement advantages retain update direction.
- Ablation Studies: EMA provides the best reward, answer accuracy, VSTAR overall score, and general-video average, while Sync-10 shows a transient drop and larger final gradient norm.The results characterize EMA as a stabilizing parameterization of the feedback-conditioned teacher rather than the primary source of VISD’s gains.
- Ablation Studies: VISD’s top-K local support maintains stronger reward trajectories and better grounding scores than sampled-token credit, including VSTAR mAM/mLGM and TVGBench.The top-K formulation compares distributions around the prefix instead of evaluating only the realized token.
5 Conclusion
VISD addresses fine-grained credit assignment in long-horizon video reasoning by combining structured judge feedback with reinforcement learning. Its experiments report improved accuracy, grounding, interpretability, and convergence over strong baselines.
- Method: VISD uses a video-aware judge to generate token-level supervision for answer correctness, reasoning consistency, temporal alignment, and spatial grounding.The framework targets structured errors in grounded video reasoning rather than treating privileged information as a generic signal.
- Method: Direction–magnitude decoupling preserves reward-driven policy stability while enabling semantically aligned credit assignment.Rollout-level rewards determine update direction, while structured privileged information supports finer token-level corrections.
- Results: VISD improves accuracy, grounding, and interpretability while significantly accelerating convergence over strong baselines.The conclusion attributes robustness on complex video sequences to curriculum scheduling and EMA-based teacher stabilization.
- Motivation: VideoLLM training remains difficult because long reasoning trajectories must align with temporally evolving visual evidence and preserve faithful intermediate reasoning.This challenge distinguishes video reasoning from static image or text reasoning.
B Limitations and Future Works
The evaluation is bounded by public video reasoning and grounding benchmarks and by tasks with explicit temporal and spatial grounding. These settings do not fully represent noisy, open-ended real-world video use or the broader range of structured outputs VideoLLMs may require.
- Benchmark scope: Public benchmarks cover several video reasoning and grounding settings but cannot fully capture the diversity and noise of real-world video use cases.The stated boundary motivates evaluation on more open-ended, noisy, and complex real-world tasks.
- Task scope: VISD currently focuses on video reasoning with explicit temporal and spatial grounding signals.The paper identifies richer event descriptions, compositional grounded reasoning, and interactive video tasks as future extensions.
C Implementation and Evaluation Details
VISD combines verifier-based rollout rewards with feedback-conditioned teacher reweighting to convert structured judge feedback into token-level credit. Its objective preserves reward-determined update direction while modifying token-level update magnitudes.
- Feedback and scoring: An external judge reads each student rollout with the question, verified answer, and grounding evidence to produce structured process feedback.The feedback covers answer correctness, reasoning consistency, temporal alignment, and spatial grounding.
- Feedback and scoring: The verifier reward covers answer, format, final-answer grounding, and reasoning-trace grounding rather than introducing a new reward source.VISD changes how rollout rewards become token-level credit through feedback-conditioned teacher reweighting.
- Feedback and scoring: The reward masks unsupported components and computes answer, temporal, and spatial grounding terms from the available supervision.Answer scoring uses exact correctness or ROUGE-based similarity; grounding uses temporal or visual IoU where applicable.
- Optimization objective: When no valid prediction or supervision exists, the relevant component is zero or masked, and judge feedback does not alter the reward-determined advantage direction.This separates reward computation from the teacher-replay mechanism in the main experiments.
- Optimization objective: VISD computes group-relative rollout advantages, derives token modifiers from top-K teacher–student log-ratios, and inserts token-reweighted advantages into the clipped policy objective.The sequence-level advantage supplies the update direction, while token-level reweighting changes credit magnitude.
- Training procedure: The training procedure samples grouped rollouts, scores them, computes token weights, updates the policy, and then updates the EMA teacher.The procedure explicitly ends with a direction-preserving policy update and EMA teacher update.
C.2 Training Details
VISD uses the Open-o3-Video SFT initialization and trains the RL stage with specified rollout, optimization, visual-input, and EMA-teacher settings. The configuration also defines a scheduled transition from VISD guidance toward the standard objective.
- Initialization: VISD initializes from STGR-CoT-30k grounded reasoning data spanning temporal, spatial, spatio-temporal, and general video reasoning samples.The same SFT initialization recipe as Open-o3-Video is used before the RL stage.
- Optimization: Training uses two nodes with eight NVIDIA A100 GPUs per node, one epoch per stage, learning rate 1 × 10^-6, and four on-policy completions per prompt.The setup also uses DeepSpeed ZeRO-3, FlashAttention-2, and a 16,384-token prompt limit with 768-token completions.
- Optimization: The main result evaluates the EMA-teacher student at optimization step 1200 under a 401,408-pixel visual-input budget.Per-device batch size is 1, with gradient clipping at maximum norm 5.
- Teacher guidance: VISD maintains an EMA teacher with update rate 0.01, uses top-K support size K = 16, clips token weights to [0.8, 1.2], and anneals mixing from 0.5 to 0 over 600 steps.The main setting omits a reference model, teacher KL loss, and separate distillation loss.
- Configuration overview: The configuration table summarizes the backbone, teacher and evaluator setup, rollout configuration, token-credit reweighting, and optimization hyperparameters.It serves as an overview of the main model and training configuration.
C.3 Evaluation Details
The evaluation covers grounded reasoning, general video understanding, and temporal grounding using benchmark-specific metrics and official protocols. VSTAR combines answer accuracy with temporal and spatial localization, while other benchmarks use their prescribed scores.
- VSTAR metrics: VSTAR measures what, when, and where through answer accuracy, temporal IoU, and spatial IoU.It also reports mAM and mLGM to aggregate answer correctness, temporal localization, and spatial grounding across two reasoning chains.
- VSTAR metrics: VSTAR reports two reasoning chains: what–when–where and what–where–when.The benchmark computes aggregate scores across both chain orderings.
- General video understanding metrics: WorldSense, VideoMMMU, Video-MME-v2, and LongVideo-Reason use their official benchmark-level accuracy or score protocols.WorldSense and VideoMMMU additionally report category-level scores when available.
- Temporal grounding metrics: TVGBench and Charades-STA evaluate temporal grounding with temporal IoU between predicted and ground-truth intervals.Charades-STA reports Recall at IoU thresholds 0.3, 0.5, and 0.7 plus mean IoU, while TVGBench reports mean IoU.
- Evaluation protocol: Evaluations run on 8 NVIDIA A100 GPUs, using 16 frames for VSTAR and 64 frames without subtitles for Video-MME-v2.VSTAR answers are judged by Qwen2.5-72B-Instruct under the benchmark protocol.
D.1 Benchmark-Level Ablation Results
Benchmark-level ablations extend the analysis beyond training curves to Video-MME-v2 and temporal-grounding tasks. Teacher/feedback variants and top-K variants are reported as separate design axes.
- Benchmark-level ablations: Tables 9 and 10 report additional ablations on Video-MME-v2, Charades-STA, and TVGBench.These results test whether design-choice effects remain visible under downstream benchmark evaluation.
- Video-MME-v2: Table 9 separates teacher/feedback ablations from top-K analysis for Video-MME-v2.All evaluated ablation models use 64 sampled frames without subtitles.
- Temporal grounding: Table 10 reports detailed Charades-STA and TVGBench temporal-grounding metrics in separate teacher/feedback and top-K blocks.The layout isolates the two design axes rather than combining them into one ablation block.
D.2 Additional Training Curves
Additional training curves examine top-K support, judge feedback, and teacher-update parameterizations. They show stronger stability for top-K support, component-level feedback effects, and smoother optimization from EMA teacher updates.
- Top-K support versus sampled token: Top-K support gives a slightly stronger and more stable trend than sampled tokens on total reward and several grounding-related components.The sampled-token variant remains competitive on several signals, and the curve differences match benchmark-level differences.
- Feedback conditioning: Feedback and no-feedback variants are compared separately across total reward, accuracy, grounding, temporal, spatial, and format components.The component curves are noisier than total reward, particularly for sparse grounding sub-rewards.
- Teacher update stability: EMA smoothing provides a more stable teacher signal than Sync-10 synchronization or instantaneous current-policy parameters.Sync-10 exhibits a substantially larger gradient norm and a sharper disturbance in the teacher diagnostics.
- Feedback conditioning: The feedback ablation uses the same reinforcement-learning and teacher-replay pipeline while removing only judge-generated process feedback.This controls the comparison for the role of feedback itself.
E Mechanistic Analysis and Qualitative Diagnostics
The mechanistic analysis shows how judge feedback changes token credit without replacing reward-based policy direction. VISD compares training signals, rollout-dependent diagnoses, local-support ratios, and feedback-conditioned replay to explain this separation.
- Training-curve diagnostics: Figure 4 compares top-K support and sampled-token training curves across reward, accuracy, grounding, temporal, spatial, and format metrics.The panels cover total and group mean reward, answer accuracy, temporal and spatial IoU, temporal point and segment, spatial grounding, and format reward.
- Trajectory-dependent feedback: Judge feedback distinguishes failed, partially correct, and correct rollouts of the same question by diagnosing answer, temporal, spatial, and consistency errors.This makes feedback rollout-dependent rather than a generic additional prompt field.
- Direction–magnitude decoupling: The rollout-level advantage determines whether a trajectory is reinforced or suppressed, while teacher-student ratios modulate token-level update magnitude.The teacher observes answer-side information and, in the feedback-conditioned case, judge diagnosis.
- Feedback-conditioned credit: Feedback-conditioned replay changes teacher token evidence while keeping the student rollout, reward, and token sequence fixed.Additional changes are sparse and tend to occur around diagnosed process errors such as spurious actions, incomplete temporal coverage, or answer-critical object descriptions.
- Local-support calibration: VISD uses a local-support ratio based on teacher and student distributions for the realized token rather than introducing gradients over all supported tokens.The realized rollout token is added to the teacher top-K support, and the ratio is used as a stop-gradient scalar for policy-gradient reweighting.
- Direction preservation: The positive token reweighting cannot flip the rollout-level update direction and reduces to the standard rollout advantage as λ anneals to zero.Teacher-conditioned information redistributes token-level magnitudes within a bounded range.
- Feedback as an information layer: Equation 31 isolates judge feedback as an information layer that changes teacher scoring of the same student completion rather than creating a separate generation target.The feedback-conditioned log-ratio is compared with answer-only replay on a common local support.
- Qualitative token credit: Figure 8 contrasts answer-only and feedback-conditioned token credit under the same fixed student rollout, with color indicating positive or negative token evidence.The answer-only teacher uses verified answer-side information, whereas the feedback-conditioned teacher additionally receives judge diagnosis.
F Qualitative Case Studies
VISD’s qualitative case studies show precise spatial and temporal grounding before answer generation, yielding more faithful alignment between visual evidence, reasoning, and responses than related models.
- Case-study overview: VISD grounds relevant objects, actors, actions, and temporal intervals before producing answers across diverse video understanding scenarios.The cases span spatial relations, temporal actions, object disappearance, and temporal localization.
- Spatial reasoning: In spatial relation reasoning, VISD localizes the queried child and identifies the object above him while avoiding nearby-object confusion.The comparison contrasts VISD’s precise grounding with incorrect answers or incomplete spatial evidence from related models.
- Action and interaction reasoning: VISD correctly identifies the panda’s bucket-related action and the baby’s interaction with the walker using grounded temporal and spatial evidence.The corresponding cases contrast VISD’s interpretations with competing models’ action-transition or subject-confusion errors.
- Temporal localization: VISD captures sustained temporal windows and action transitions, including window-looking and turning off the light, while related models focus on incomplete movements.The cases emphasize matching the relevant interval rather than merely recognizing an earlier partial movement.
- Grounding-response alignment: VISD aligns visual grounding with final responses when detecting object disappearance and temporal actions, whereas related models may answer inconsistently or rely on inaccurate evidence.The examples include apple removal and the orange kitten’s kick, where competing models either produce inconsistent answers or inaccurate temporal evidence.