PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Thought Alignment: Trajectory-Level Latent Distillation for Video Reasoning

Ao Shen, Yongheng Zhang, Yinghui Li, Manning Wang, Di Yin, Xing Sun

arXiv:2608.16316v1cs.CVcs.AIcs.CL

TL;DR

Video reasoning distillation must capture latent spatiotemporal evidence, not only output-token preferences. Latent-OPD aligns trajectory-tail states with progressively deeper teacher layers and consistently outperforms vanilla OPD across six benchmarks, especially with limited frames.

  • Problem

    Output-level supervision improves token preferences but underconstrains latent spatiotemporal representations formed during video reasoning.

  • Method

    Latent-OPD augments output distillation with trajectory-tail hidden-state alignment and progressive teacher-lookahead mapping.

  • Results

    64.4% with 16 frames surpasses vanilla OPD’s 63.0% with 32 frames, while Latent-OPD consistently outperforms vanilla OPD across six benchmarks.

  • Takeaways & Limitations

    The results support transferring video reasoning beyond final-answer imitation by distilling latent integration of events, relations, and temporal structure.

  • Takeaways & Limitations

    A case study is not intended as standalone evidence and only illustrates behavior suggested by quantitative analysis.

Abstract

from arXiv · show

Large Multimodal Models (LMMs) for video reasoning have long been hindered by the high computational cost of processing vast amounts of visual information. This dilemma motivates the transfer of the reasoning capabilities of large models to smaller, more efficient ones. On-Policy Distillation (OPD) offers a promising solution by matching output-token distributions along student-generated trajectories. However, video reasoning often depends on evidence accumulated across multiple frames. In this context, output-level supervision only captures information expressed through token predictions and does not directly constrain the latent representations formed during reasoning. To address this limitation, we propose Latent-OPD, which augments OPD with trajectory-level latent distillation. Specifically, our method focuses on the position at the end of each trajectory, where hidden states effectively summarize the accumulated visual evidence and reasoning context. Furthermore, we introduce a progressive teacher-lookahead strategy, which aligns middle-to-late student layers with increasingly deeper teacher layers. Experiments on six video reasoning benchmarks show that Latent-OPD consistently outperforms output-only OPD. Notably, the improvements are particularly pronounced in scenarios with limited frames, long videos, or tasks requiring complex evidence aggregation. These results establish Latent-OPD as a highly effective approach to frame-efficient video reasoning.

1 Introduction

Video reasoning requires aggregating sparse, decisive evidence across extensive spatiotemporal information, but vanilla OPD supervises only output-token distributions and underuses latent representations. Latent-OPD addresses this bottleneck through trajectory-level latent distillation with sparse tail anchors and progressive teacher lookahead, improving video reasoning across six benchmarks, especially when frames are limited or evidence aggregation is difficult.

  • Video reasoning must process vast spatiotemporal information and distill sparse evidence for multi-step decisions.
  • Vanilla OPD minimizes output-distribution KL divergence, leaving teacher hidden-state knowledge and latent spatiotemporal representations underused.
  • Latent-OPD adds trajectory-level latent distillation using sparse trajectory-tail hidden states as compact global anchors.
  • Across six video reasoning benchmarks, Latent-OPD consistently improves over vanilla OPD, with stronger gains in low- and mid-frame, long-video, and cross-frame evidence-aggregation settings.It can match or surpass higher-frame vanilla OPD while using fewer input frames.
  • Progressive teacher lookahead enables deep thought alignment and efficient latent transfer across heterogeneous architectures.

2 Method

Latent-OPD transfers video-reasoning capability through both student-trajectory token supervision and trajectory-level latent-state alignment. It uses reliable teacher rollouts, tail-state matching, and progressive teacher lookahead while adding no inference-time overhead.

  • Framework: Latent-OPD complements output-token preference transfer with latent-state supervision from a frozen teacher for complex video reasoning.The framework is explicitly designed for deep thought alignment in large multimodal models.
  • Two-stream distillation: The output stream evaluates teacher and student token distributions on the same student-generated prefixes, while the latent stream uses teacher-generated trajectories as anchors.Teacher trajectories are filtered for reliability before latent supervision.
  • Trajectory filtering: Teacher trajectories are retained using final-answer correctness, providing a lightweight reliability mask for latent supervision.The correctness indicator is cT = 1[Acc(yT, y⋆) = 1].
  • Trajectory-level latent alignment: Latent alignment occurs only at the final valid trajectory token, whose causal hidden state summarizes the full video-question context and preceding reasoning.This avoids dense hidden-state matching across all video and response tokens.
  • Layer mapping: Progressive teacher-lookahead pairs student middle-to-late layers with increasingly deeper teacher layers, while leaving output-facing student layers unconstrained by the latent loss.The asymmetric mapping exposes student representations to more abstract teacher representations.
  • Unified objective and inference: The output and latent streams are jointly optimized, and the deployed student requires neither the teacher nor projection heads during inference.Consequently, latent distillation introduces no additional inference-time computational overhead.

3 Experiments

Experiments across six video reasoning benchmarks show that Latent-OPD consistently improves accuracy, convergence, long-horizon temporal integration, and frame efficiency over output-only OPD. Ablations and smaller-student results further support trajectory-level latent alignment, progressive teacher lookahead, correctness-filtered anchors, and aligned teacher-student reasoning paths.

  • Main results: Latent-OPD converges faster than vanilla OPD, with training curves separating early and remaining stable, indicating informative optimization signals for extended visual contexts.The method outperforms vanilla OPD across all 18 benchmark/frame-budget pairs.
  • Long-horizon temporal integration: Up to +7.0 on Video-MMMU and up to +3.6 on Video-MME: Latent-OPD delivers its largest gains on cross-frame reasoning tasks.In the 16-frame Video-MME breakdown, gains reach +3.22 on long videos, +4.44 in Artistic Performance, and +3.89 in Film & Television.
  • Frame efficiency: 64.4% versus 63.0%: Latent-OPD’s 16-frame average surpasses vanilla OPD’s 32-frame performance, demonstrating improved frame utilization efficiency.At 32 frames, Latent-OPD matches or beats the 64-frame baseline on Video-MMMU, MVBench, and TempCompass.
  • Smaller student: 61.1% to 62.5%: Latent-OPD improves the 16-frame average for a Qwen3.5-4B-Base student, with especially visible gains on Video-MMMU, MMVU, and Video-MME.This result extends the evaluation beyond the 9B student and supports transfer of complex reasoning capabilities to a smaller model.
  • Ablations: The default progressive teacher-lookahead mapping uniformly surpasses alternative mappings across 16, 32, and 64 frames, while dense token-level controls and same-depth SFT variants consistently underperform.The results support aligning mid-to-late student depths with deeper teacher states, using latent rather than merely token-level supervision.

4 Related Work

Related work extends on-policy distillation from language models to multimodal and video settings, while representation-level studies motivate supervising hidden states beyond output logits. Latent-OPD adapts this direction to video by aligning compact trajectory-tail states after cross-frame integration and progressively matching student layers to deeper teacher layers.

  • Visual and Video On-Policy Distillation: On-policy distillation has progressed from language-model alignment and compression to multimodal methods using fine-grained visual supervision, token reweighting, text-reasoning transfer, and visual-grounding control.The passage identifies Vision-OPD and VA-OPD among these multimodal extensions.
  • Latent and Representation-Level Reasoning: Output logits can discard structural reasoning dynamics formed before verbalization, and intermediate hidden states can encode knowledge relevant to multimodal distillation.These findings motivate latent or representation-level reasoning supervision.
  • Latent-OPD for Video Reasoning: Directly applying OPRD to video is sub-optimal because dense same-depth matching suffers from redundant frames and mismatched reasoning.Latent-OPD addresses this limitation by adapting representation-level OPD to spatiotemporal reasoning.
  • Latent-OPD for Video Reasoning: Latent-OPD aligns compact trajectory-tail states after cross-frame integration and maps middle-to-late student layers to deeper teacher layers.This design targets representations formed after integrating evidence across frames.

5 Conclusion · Appendix

The paper concludes that video reasoning benefits from aligning latent spatiotemporal states alongside output-token preferences, motivating Latent-OPD with trajectory-tail hidden-state alignment and progressive teacher lookahead. The appendix documents implementation details and additional 4B-scale experiments for reproducibility.

  • 5 Conclusion: Latent-OPD augments on-policy distillation with compact trajectory-tail hidden-state alignment and progressive teacher-lookahead mapping.The method targets latent spatiotemporal states that organize sparse visual evidence before verbalization.
  • 5 Conclusion: Latent-OPD outperforms vanilla OPD across six benchmarks, with especially strong gains under limited frame budgets.The conclusion attributes video-reasoning effectiveness to both output-token preference matching and latent spatiotemporal states.
  • Appendix: Together, the conclusion and appendix connect Latent-OPD’s reported effectiveness with documented training, evaluation, and experimental procedures.This synthesis is based on the method summary and the appendix’s stated reproducibility scope.
  • Appendix: The appendix provides implementation details omitted from the main paper to support reproducibility.It covers training and evaluation data, public baselines, model initialization, post-training, objectives, teacher trajectories, optimization, and preprocessing.
  • Appendix: Section A describes Latent-OPD hyperparameters and training setup, including output and latent objectives and teacher trajectory generation.The training recipe also includes data, baselines, initialization, a two-stage post-training process, optimization schedule, and video preprocessing.
  • Appendix: Section B reports additional 4B-scale experimental results across all three frame buThe supplied passage truncates the remainder of this sentence.

A Hyperparameters and Training Setup … Video preprocessing

The paper uses a two-stage Video-R1-based training setup with Qwen3.5 student–teacher models, on-policy distillation, and trajectory-level latent alignment. Training uses controlled video preprocessing, frozen teacher trajectories, and a short staged optimization schedule.

  • A Hyperparameters and Training Setup; Training and evaluation data: Stage I performs SFT on Video-R1-CoT-165k, while Stage II samples from Video-R1-260k for RL/OPD post-training.The video data targets temporal understanding, including event order, frame dependencies, motion, and causal dynamics.
  • Baselines: Baselines include seven public models and four Qwen3.5-9B variants evaluated under the same frame-budget protocol.The implemented variants are C0T, SFT+GRPO, Vanilla OPD, and Latent-OPD.
  • Models: The student is Qwen3.5-9B-Base and the teacher is a Qwen3.5-27B video-CoT SFT checkpoint trained for 2,000 steps.The student has 4096 hidden dimensions and 32 text layers; the teacher has 5120 hidden dimensions and 64 text layers.
  • Two-stage post-training: Latent-OPD adds teacher-trajectory hidden-state alignment to OPD, using separate student-rollout and retained-teacher trajectories with separately accumulated gradients.The latent loss is applied as an auxiliary backpropagation term rather than a policy-gradient term.
  • Output objective: The output objective uses JSD-style distillation with α = 0.5, reference KL weight β = 0.04, format weight λfmt = 0.05, and four sampled completions per prompt.Distillation uses the student-selected top-100 token support plus an aggregated tail bucket, with generation batch size 8.
  • Latent objective: Correct-only filtering uses threshold 1.0, while teacher lookahead aligns (s16, t48), (s20, t56), and (s24, t64) at progressively deeper relative depths.The pairs correspond to (50%, 75%), (62.5%, 87.5%), and (75%, 100%).
  • Latent objective: The latent weight is λg = 0.01, with disabled SFT and prompt-end branches, cosine alignment, full-rank projectors, a 15% loss cap, and 5% warm-up.Each projector has approximately 20.97M parameters, totaling approximately 62.9M across three pairs.
  • Teacher trajectory generation; Optimization and schedule; Video preprocessing: Teacher trajectories are generated on the fly with up to 512 new tokens, while Stage II uses AdamW settings, 1.0 gradient clipping, 300 training steps, and 1-FPS sampling capped at 16 frames.Generation uses temperature 0.7 and top-p 0.9; videos use 28 × 28 patches, and tokenized prompts/videos are reused across four rollouts.

B Additional 4B-Scale Results

Additional Qwen3.5-4B experiments show that Latent-OPD consistently improves average accuracy over Vanilla OPD across 16-, 32-, and 64-frame budgets. The gains on Video-MMMU, MMVU, and Video-MME indicate that trajectory-level latent supervision remains effective at 4B scale.

  • 4B-scale comparison: Latent-OPD raises average accuracy from 61.1% to 62.5% at 16 frames, from 62.6% to 64.1% at 32 frames, and from 64.1% to 65.3% at 64 frames.These results use the same evaluation protocol as the 9B experiments.
  • 4B-scale comparison: Improvements on Video-MMMU, MMVU, and Video-MME show that the trajectory-level latent signal remains effective for the 4B student.The reported gains extend beyond the average score across the evaluated frame budgets.
  • 4B-scale comparison: The Qwen3.5-4B comparison covers six benchmarks under three frame budgets, with bold entries marking the better result between Vanilla OPD and Latent-OPD.Table 3 reports accuracy (%) for each benchmark–budget pair.

C Teacher Model Results

The fine-tuned Qwen3.5-27B teacher remains stronger than the 9B Latent-OPD student on most benchmarks, but their average performance gap is moderate across frame budgets.

  • Teacher setup: The Qwen3.5-27B teacher is fine-tuned from Qwen3.5-27B-Base for 2,000 steps on video CoT SFT data.Evaluation uses the same six-benchmark, three-frame protocol.
  • Average performance: 66.6%, 68.0%, and 68.8% are the teacher’s six-benchmark averages at 16, 32, and 64 frames, respectively.The corresponding 9B Latent-OPD student averages are approximately 64.4%, 65.6%, and 66.5%.
  • Average performance: The teacher–student average gaps are about 2.2, 2.4, and 2.3 points at 16, 32, and 64 frames, respectively.The teacher remains stronger on most benchmarks, with the largest residual gap on Video-MMMU.

D Video-MME Fine-Grained Results

On the 2700-question Video-MME validation split, the study compares vanilla OPD and Latent-OPD across 16, 32, and 64 input frames. Latent-OPD is most beneficial when reasoning requires compressing sparse or temporally distributed evidence, while gains diminish with dense visual input.

  • Evaluation setup: The fine-grained evaluation uses the 2700-question validation split and compares vanilla OPD with Latent-OPD under 16, 32, and 64 input frames.The checkpoints and decoding protocol match those used in the main experiments.
  • When latent distillation helps: Latent-OPD most helps on long videos with 16 frames, where students must compress sparse or temporally distributed visual evidence into a global reasoning state.The section identifies this evidence-compression setting as the clearest source of improvement.
  • When latent distillation helps: Latent-OPD strengthens performance in process-oriented domains such as Film & Television and Artistic Performance.These domains are highlighted as examples of tasks benefiting from latent trajectory supervision.
  • Boundary condition: At 64 frames, gains are smaller or mixed because dense visual evidence allows vanilla OPD to recover some missing information by observing more frames.This frames higher frame budgets as a boundary condition for the marginal value of latent trajectory anchors.

E Ablation Setting Details

The ablation setting fixes a shared training recipe and isolates layer mapping, representation/token-level supervision, trajectory source, and anchor-position choices. Controls distinguish teacher lookahead from projector count, supervision density, trajectory reliability, and positional correspondence.

  • Shared recipe: All variants share 9B SFT initialization, Video-R1 data, 300 training steps, four sampled rollouts, 16 training frames, and Video-MME evaluation at 16/32/64 frames.The output objective is the same JSD-style OPD loss over student-selected top-100 support plus an aggregated tail.
  • Layer mapping: The default layer mapping pairs student depths 50%, 62.5%, and 75% with progressively deeper teacher depths 75%, 87.5%, and 100%.Same-depth, fixed-offset, and reverse-lookahead variants isolate the effect of teacher depth while holding aligned-state count, projector sizes, latent weight, correctness filtering, and loss cap constant.
  • Layer mapping: Pair-count controls test whether gains arise from lookahead rather than simply adding projectors or supervising the final student layer.They compare a single-tail mapping, an added final-student pair, and an added early pair against the default mapping.
  • Supervision controls: Representation and token-level controls compare dense response-token hidden matching, normalized-MSE latent OPD, low-rank projectors, and teacher-trajectory SFT.These variants test supervision density, projector capacity, representation-level adaptation, and whether teacher trajectories improve training.
  • Trajectory and anchor controls: Trajectory and anchor controls test whether latent targets are reliable and position-matched by varying teacher-answer filtering, trajectory sharing, and the single tail anchor.Independent trajectories summarize different reasoning paths; shared student trajectories restore positional correspondence but condition the teacher on an unchosen path, whereas Latent-OPD uses the same correctness-filtered teacher trajectory.

F Additional CKA Analysis

The CKA analysis compares trajectory-tail student and teacher representations at matched relative depths without learned projection or teacher-layer max-pooling. Latent-OPD improves alignment mainly in deeper student layers, while shallow layers remain nearly unchanged.

  • CKA Protocol: CKA compares centered trajectory-tail hidden states through sample Gram matrices across vanilla OPD, Latent-OPD, and teacher layers.The student and teacher states are evaluated on the same N examples.
  • Depth Alignment: Each student layer is matched to the teacher layer at the same relative depth rather than using max-pooling over teacher layers.Directly supervised student layers are s ∈{16, 20, 24}.
  • Results: +0.108 average CKA gain occurs at or beyond 50% student depth under Latent-OPD.The peak gain is about +0.137, while shallow layers remain nearly unchanged.
Loading 2608.16316v1…