Source-linked AI summary
Context-Matched Distillation: Teacher Causality for Autoregressive Video Distillation
Hmrishav Bandyopadhyay, Xuanchi Ren, Zijian Huang, Jay Zhangjie Wu, Tianshi Cao, Ruilong Li, Bryan Chu, Sanja Fidler, Yi-Zhe Song, Zian Wang
TL;DR
Causal few-step video students can receive mismatched supervision when bidirectional teachers score targets using future frames or controls. CMD uses causal teacher scoring, realized student prefixes, and prefix corruption to align supervision with online context, achieving strong short- and long-video quality and substantially lower camera-pose errors.
Problem
Bidirectional DMD teachers can supervise causal students with future frames and controls unavailable when each target is generated, creating a teacher–student context mismatch.
Method
CMD trains and uses a causal teacher, initializes the few-step student from it, scores targets under cached student-generated prefixes, and corrupts unreliable prefixes during training.
Results
CMD achieves strong short- and long-video quality and substantially lower camera-pose errors across autoregressive evaluations.
Takeaways & Limitations
Matching teacher supervision to the temporal information available during online generation supports frame-wise, chunk-wise, long-video, and camera-conditioned distillation.
Abstract
from arXiv · showhide
Interactive autoregressive video generation demands both low-latency rollouts and precise online control. Few-step distillation accelerates generation by reducing denoising steps, while online control imposes a causal constraint: frames and blocks should depend on history and controls available during generation. Existing video distribution matching distillation (DMD) pipelines, however, often supervise causal few-step students using bidirectional teachers that score complete clips. The score for a target can therefore depend on future frames and controls that were unavailable when the student generated it, misaligning teacher supervision with the student's causal information set. We introduce Context-Matched Distillation (CMD), a causal DMD framework that aligns teacher supervision with the information available when each target is generated. CMD replaces bidirectional full-clip scoring with a causal teacher that evaluates each target without access to future frames or controls. The same causal teacher initializes the few-step student, establishing a consistent causal formulation across teacher training, student distillation, and inference. Beyond aligning the temporal information boundary, Prefix Scoring matches supervision to the student's realized rollout context by evaluating each target under the cached student-generated prefix that produced it. Prefix Corruption further stabilizes training by perturbing unreliable prefixes produced early in training while preserving this target-context alignment. With a simple causal formulation, CMD naturally extends to frame-wise and chunk-wise generation, long video distillation, and camera-conditioned distillation. Experiments demonstrate state-of-the-art aggregate performance among autoregressive methods on both short- and long-video benchmarks, together with substantially improved adherence to time-varying camera controls.
1. Introduction
Interactive autoregressive video generation needs efficient rollouts and precise online control, but bidirectional teachers can supervise causal students with unavailable future information. CMD aligns teacher scoring with causal generation through matched teachers, realized prefixes, and prefix corruption.
- Motivation: Bidirectional DMD teachers can score causal student targets using future frames and controls unavailable during generation.This teacher–student context mismatch is especially consequential for camera control and long-video generation.
- Context-Matched Distillation: CMD replaces full-clip bidirectional scoring with a causal teacher that respects the information boundary available to each target.The same causal formulation is used for teacher training, student distillation, and inference.
- Context-Matched Distillation: Prefix Scoring evaluates each target under the cached student-generated prefix that produced it, rather than only preceding noised DMD targets.This matches supervision to the realized rollout history.
- Context-Matched Distillation: Prefix Corruption perturbs unreliable early prefixes to stabilize training while preserving target–context alignment.The approach is designed for rollout artifacts that differ from independent teacher-training corruptions.
- Scope and Results: CMD extends the causal formulation to frame-wise and chunk-wise generation, long-video rollout, and camera control.Experiments report improved quality over bidirectional-scoring baselines and substantially reduced camera-pose errors.
2. Related Work
Related work develops autoregressive video generation, long-video rollout mechanisms, and camera-conditioned synthesis as alternatives or extensions to full-clip diffusion. These directions support streaming, interactive control, extended horizons, and viewpoint-conditioned generation.
- Block-causal autoregressive video generation: Causal and block-causal video models generate frames or short blocks sequentially while reusing cached history, supporting streaming and interactive generation.Bidirectional full-video attention instead scales quadratically with token count.
- Synthesis: The related directions collectively address the scalability and controllability limits of jointly generated video clips.Their common setting is sequential or trajectory-conditioned generation rather than unconstrained full-clip synthesis.
- Long video generation: Long-video methods extend generation beyond training horizons through noise rescheduling, memory mechanisms, rolling windows, attention sinks, and positional-horizon extensions.These approaches target real-time or effectively extended rollouts.
- Camera-controlled video generation: Camera-conditioned video generation injects motion through ray maps, pose adapters, and geometry-aware attention transformations.These methods aim to separate viewpoint changes from object motion and improve consistency across wider viewpoint ranges.
3. Preliminaries
The preliminaries define autoregressive video as sequential latent generation conditioned on history and per-frame controls, and describe DMD and Diffusion Forcing as foundations for few-step students and causal teachers.
- Distribution Matching Distillation: DMD distills a few-step student by matching its distribution to a multi-step teacher through the difference between fake and real score functions.The score is evaluated on a noised sample at a diffusion timestep.
- Distribution Matching Distillation: Video DMD applications commonly let the student generate a full sample while the teacher scores the corresponding full video clip.This follows the text-to-image distillation setup even for causal and frame-wise generators.
- Autoregressive Video Generation: An autoregressive video latent sequence contains a fixed input-image latent followed by generated video latents and per-frame conditioning signals.Conditioning can represent camera motion or user controls.
- Autoregressive Video Generation: Autoregressive generation decomposes the video distribution over time, with causality enforced across generated blocks and possible bidirectional interaction within a block.Setting K = 1 yields frame-wise generation.
- Diffusion Forcing: Diffusion Forcing independently corrupts frames and trains each target under noisy history, improving robustness when generated frames are reused as context.CMD uses this objective to pre-train its causal teacher.
4. Methodology
CMD distills a few-step causal student using teacher scores matched to the temporal and realized rollout contexts available when each target is generated. It combines causal teacher training, Prefix Scoring, Prefix Corruption, and bounded-context extensions for long-video and camera-conditioned generation.
- Context-matched causal scoring: CMD replaces bidirectional full-clip supervision with causal teacher scores conditioned only on past frames and available controls.The causal teacher is fine-tuned with independently noised histories and then used to initialize the few-step student.
- Causal student distillation: The student is initialized from the causal teacher and rolled out as during inference, preserving the causal inductive bias throughout distillation.This formulation avoids requiring ODE- or Consistency-Distillation-based student initialization.
- Prefix Scoring: Prefix Scoring evaluates each noised student target under the cached generated prefix and controls that produced it, rather than preceding noised DMD targets.A block-causal mask allows all target queries to be scored in parallel while restricting each query to its causal prefix.
- Prefix Corruption: Prefix Corruption perturbs unreliable on-policy prefixes with a dedicated noise level to stabilize distillation without changing target-side DMD noising.The corruption is applied to the generated context, independently of randomized DMD timesteps.
- Long-video extension: For long videos, bounded Prefix Scoring preserves preceding rollout context and prevents targets from attending to later frames within a local scoring window.A bidirectional local-clip teacher instead omits pre-clip context and permits future-frame access, which mismatches the student’s causal factorization.
- Camera-conditioned generation: CMD uses frame-relative camera increments and restricts teacher scores to visual and camera information available up to each frame.The same context matching supports frame-wise, chunk-wise, long-video, and time-varying camera-conditioned generation.
5. Experiments
CMD is evaluated across short-video, long-video, and camera-controlled settings, with ablations testing causal scoring, matched prefixes, and prefix corruption. It improves aggregate quality over bidirectional scoring and reduces camera-control errors.
- Evaluation setup: CMD is evaluated on VBench-I2V, SANA-WM long-video, and camera-controlled benchmarks using chunk-1 and chunk-4 autoregressive models.The experiments also compare camera representations and ablate Prefix Scoring and Prefix Corruption.
- Short-video quality: 88.47, 96.54, and 76.12 are the chunk-4 model’s best VBench-I2V Total, I2V, and Camera Motion scores.These exceed the strongest baseline by 0.84, 1.18, and 33.58 percentage points, respectively.
- Long-video quality: 0.77 points is CMD chunk-1’s improvement over Context Forcing on long-video Total, while it achieves the best aggregate score in every main-comparison metric.The chunk-1 model also improves Q by 0.67 points over Context Forcing and S by 0.31 points over LingBot-World.
- Camera control: CMD chunk-1 achieves the best Semantic and Total scores on both camera-controlled splits, while chunk-4 has the lowest rotation, translation, and CamMC errors.Rotation, translation, and CamMC are lower-is-better camera-error metrics.
- Ablations: Base CMD raises short-video Total by 5.68 points over bidirectional scoring, while default Prefix Scoring further improves Total by 0.13 points.Prefix Scoring also improves Camera Motion by 2.98 points and Dynamic Degree by 5.85 points; prefix corruption with t_prefix = 256 gives the best aggregate result.
- Ablations: Full CMD improves long-video Total by 4.04 points over bidirectional scoring and improves Semantic by 0.58 points.Base CMD alone improves Total by 3.39 points, while Full CMD adds the bounded cross-clip prefix used by the student.
- Ablations: Camera-control ablations show that cached prefixes further improve Total by 0.0313/0.0319 on Simple/Hard after the ray-map Base CMD model.The full ray-map model also improves all SANA-WM VBench metrics, translation, and CamMC over PRoPE on both splits.
- Qualitative analysis: Qualitative rollouts show CMD preserving object and scene continuity while producing clear progression across short examples and 32-second long videos.Baselines exhibit larger structural discontinuities, abrupt object changes, or limited scene progression.
6. Conclusion
CMD is a causal DMD framework that aligns teacher scoring with the frames, controls, and student-generated history available at each target. Across short and long rollouts, it improves quality over bidirectional scoring and reduces camera-pose errors.
- Method: CMD uses a causal teacher, Prefix Scoring, and Prefix Corruption to align and stabilize few-step autoregressive video distillation.The same causal model family is used for teacher training, student initialization, and inference.
- Scope: CMD supports frame-wise and chunk-wise generation, long-video distillation, and camera-controlled generation with frame-relative conditioning.The formulation evaluates targets using the student-generated history and controls available when they were produced.
- Findings: Across short and long rollouts, CMD improves generation quality over bidirectional scoring and substantially reduces camera-pose errors.The paper’s ablations isolate the benefits of causal scoring, matched rollout context, and prefix corruption.