Source-linked AI summary
Rollout-Marginal Distillation for Long-Horizon Autoregressive Video Generation
Chenjian Gao, Zhihao Hu, Jianqi Ma, Jun Zhang, Weidong Zhang, Tianfan Xue
TL;DR
Autoregressive video generation faces compounding errors over long rollouts, while joint video-level distillation couples chunk corrections to surrounding context. RMD scores chunks independently and then applies video-level DMD; experiments report higher visual quality than video-level DMD baselines beyond the training horizon.
Problem
In autoregressive video generation, prediction errors compound over long rollouts, and joint video-level DMD couples a chunk's correction to imperfect history and future, potentially compromising local visual realism.
Source: §1 · p. 2, §3.1 · p. 4Method
RMD retains generated history for autoregressive prediction, scores each chunk independently against a chunk teacher, and subsequently applies video-level DMD to restore temporal coherence.
Source: §1 · pp. 2–3Results
Across approximately 60-second rollouts, 12× the 5-second training horizon, RMD achieves significantly higher visual quality than video-level DMD baselines.
Source: §1 · p. 3Takeaways & Limitations
The results support decoupling chunk-level visual realism from temporal coherence as a direction for long-horizon, streamable video generation.
Source: §5 · pp. 9–10
Abstract
from arXiv · showhide
Autoregressive (AR) video diffusion enables low-latency, streamable video generation, but prediction errors often accumulate over long rollouts. Training the generator on its own rollouts exposes it to these imperfect histories. However, existing video-level distribution matching distillation (DMD) scores the whole rollout jointly. Because a chunk is evaluated together with its past and future, its correction can favor matching artifacts in the surrounding context merely to preserve temporal consistency. To provide a clearer visual-quality signal, we introduce Rollout-Marginal Distillation (RMD). RMD retains the generated history for AR prediction but scores each chunk independently against a chunk teacher, ensuring its quality correction is not compromised by an imperfect temporal context. To compensate for the lack of temporal context in independent chunk scoring, RMD subsequently applies video-level DMD to restore temporal coherence. Extensive experiments demonstrate that RMD maintains high visual quality far beyond its training horizon and outperforms video-level DMD baselines. Code and video results are available at https://cjeen.github.io/RMD
1 INTRODUCTION
Autoregressive video generation can extend videos chunk by chunk, but errors accumulate over long rollouts and joint video-level scoring can compromise local corrections to preserve temporal consistency. RMD scores chunks independently for visual quality, then applies video-level DMD to promote temporal coherence.
1 INTRODUCTION
- Autoregressive generation extends videos chunk by chunk with bounded cost, yet small prediction errors can compound and severely degrade visual quality over long rollouts.A sliding context window limits attention to recent history; self-rollout training exposes the generator to its own imperfect predictions when paired ground-truth continuations are unavailable.Source: §1 · p. 2
- Video-level DMD uses a bidirectional teacher to score the whole sequence, but accumulated artifacts in history can make local corrections conflict with temporal consistency.The teacher cannot modify past frames, so its real-score prediction may retain errors to avoid an abrupt visual shift.Source: §1 · p. 2
- RMD retains generated history for autoregressive prediction but evaluates each chunk independently, then uses video-level DMD to smooth transitions and promote temporal coherence.Its chunk teacher is adapted from a video diffusion model using individual chunks extracted from videos generated by the original bidirectional model.Source: §1 · pp. 2–3
- RMD reports significantly higher visual quality than video-level DMD baselines beyond the training horizon, including quantitative evaluation at approximately 60 seconds and a qualitative example at 500 seconds.Both durations use a fully sliding context window, without fixed anchors, additional explicit memory mechanisms, or changes to the generator architecture.Source: §1 · p. 3
2 RELATED WORK
Prior work develops causal generators and on-policy training, improves distribution matching objectives, and extends generation beyond the training horizon. RMD retains DMD's score-difference update while changing its target from a joint rollout to an autoregressively induced chunk marginal.
2 RELATED WORK
- Causal video methods establish autoregressive students and on-policy training pipelines, while RMD adopts Solaris's replay mechanism and focuses on the distribution matched during on-policy DMD.CausVid [Yin et al., 2025], Self-Forcing [Huang et al., 2026a], Causal Forcing [Zhu et al., 2026], Causal-rCM [Zheng et al., 2026a], and Solaris [Savva et al., 2026] are described as complementary approaches.Source: §2 · p. 3
- DMD-related work modifies divergence, regularization, or optimization, whereas RMD keeps the score-difference update and changes its output space to the chunk marginal induced by autoregressive sampling.The passage describes DMD [Yin et al., 2024b], DMD2 [Yin et al., 2024a], and subsequent variants.Source: §2 · p. 3
- Long-rollout methods address extended generation through streaming tuning, cache maintenance, or alternative teacher contexts; RMD scores rollout chunks independently and later restores joint temporal structure.Unlike CMD [Bandyopadhyay et al., 2026], RMD does not construct a teacher conditional for each self-rollout history and requires neither paired continuations nor real long-video context.Source: §2 · p. 3
3 ROLLOUT-MARGINAL DISTILLATION
RMD separates chunk-quality supervision from temporal context while retaining causal generation, then uses video-level DMD to refine cross-chunk coherence. The section motivates this decoupling, defines the rollout-marginal objective, and describes its training and refinement stages.
3.1 REVISITING VIDEO-LEVEL DISTRIBUTION MATCHING
- Joint video-level scores couple each chunk’s correction to its past and future, potentially sacrificing local realism to accommodate artifacts in imperfect context.A sudden correction toward natural colors can conflict with temporal consistency when the history is corrupted.Source: §3.1 · p. 4
3.2 ROLLOUT-MARGINAL OBJECTIVE
- RMD preserves on-policy causal prediction but scores generated chunks independently, separating visual-quality supervision from compatibility with surrounding content.The rollout-marginal objective penalizes chunks unlikely under the teacher’s chunk distribution, without conditioning their supervision on imperfect history.Source: §3.2 · pp. 4–5
- The objective treats initial and continuation chunks separately because their latent statistics differ, pooling continuation chunks across rollout histories and positions.The initial latent frame lacks temporal compression, unlike subsequent frames; the two groups receive teacher-defined targets weighted by their rollout frequency.Source: §3.2 · pp. 4–5
3.3 ROLLOUT-MARGINAL TRAINING
- Training supervises the initial chunk at a random intermediate denoising exit and continuation chunks at the final, lowest-noise exit.The asymmetric exits retain early-step temporal-dynamics information while applying continuation supervision where the model refines local visual details.Source: Figure 3 · p. 5, §3.3 · p. 6
- An adapted video teacher scores isolated continuations, and normalized teacher–fake prediction differences are propagated through differentiable replay.The continuation teacher is adapted on chunks from videos generated by the original model because an isolated continuation has different latent statistics from a video’s first frame.Source: §3.3 · p. 6
- During later replay steps, attention to the initial block is masked because losing that distinctive frame can create a condition absent from short rollouts and cause flicker.The mask is applied in the latter half of the training window.Source: §3.3 · p. 6
3.4 REFINEMENT WITH VIDEO-LEVEL DMD
- Because realistic chunks can still have inconsistent appearance or motion, video-level DMD jointly refines full rollouts with smaller learning rates.The refinement stage supervises motion and consistency across chunks through differentiable replay.Source: §3.4 · p. 6
4 EXPERIMENTS
On 1-minute VBench-Long comparisons, RMD outperforms video-level distillation baselines; extrapolation and ablations examine quality retention and temporal coherence.
4.1 EXPERIMENTAL SETUP
- Evaluation uses VBench-Long at approximately 60 seconds across 944 prompts with five generations each; the 500-second rollout is qualitative, and training uses a fixed 81-frame horizon.Videos are generated at 832 × 480 resolution and 16 FPS, with no longer rollouts during training.Source: §4.1 · pp. 6–7
4.2 COMPARISON WITH BASELINES
- Beyond the 5-second training horizon, baselines show color saturation and structural collapse, while RMD preserves subject detail and scene structure through 60 seconds.Figure 4 presents the qualitative comparison over extended rollouts.Source: §4.2 · p. 7, Figure 4 · p. 7
- With chunk size 1, RMD improves total score by over 10 absolute points versus the baselines (81.26 vs. 70.94 and 69.88); Table 1 reports advantages across all metrics.With chunk size 3, RMD achieves a total score of 81.48, with quality and semantic scores of 85.03 and 67.24, respectively.Source: Table 1 · p. 8, §4.2 · p. 8
4.3 LONG-HORIZON EXTRAPOLATION
- At 10–60 seconds, RMD’s quality score changes only from 84.68 to 84.52, while Self Forcing’s total score drops from 79.12 to 70.94.At 60 seconds, RMD retains a semantic score of 68.24; Figure 5 tracks performance over rollout durations.Source: §4.3 · p. 8, Figure 5 · p. 8
4.4 ABLATION STUDIES
- Video-level DMD alone scores 72.44 and produces severe color artifacts; marginal-only scoring reaches 84.90 quality but lacks temporal coherence, which video-level refinement restores.The full RMD pipeline achieves total and semantic scores of 81.26 and 68.24, respectively, with smoother trajectories; Table 2 reports ablation results.Source: §4.4 · pp. 8–9, Table 2 · p. 9, Figure 6 · p. 9
- Uniform random-step exits cause severe temporal jitter and score 78.29 total and 82.48 quality, supporting the design of random-step supervision for the initial chunk and final-exit supervision for continuations.The authors state that this asymmetric design preserves the pretrained temporal prior while refining continuation-chunk details.Source: §4.4 · p. 9, Figure 6 · p. 9
5 CONCLUSION
Autoregressive video rollouts accumulate artifacts when joint temporal scoring makes teachers trade local visual quality for sequence-level consistency. RMD separates chunk-level realism from temporal coherence, with reported gains beyond the training horizon.
5 CONCLUSION
- Joint temporal scoring can force the teacher to compromise local visual quality for sequence-level consistency as autoregressive errors accumulate.RMD is presented as resolving this coupling.Source: §5 · pp. 9–10
- RMD independently matches chunk marginals and then refines global dynamics to separate spatial realism from temporal coherence.This provides a clean, context-free supervision signal.Source: §5 · pp. 9–10
- Quantitative evaluations over 60-second rollouts show RMD prevents artifact accumulation beyond training, while a 500-second qualitative example illustrates longer-duration potential.The conclusion describes the decoupled paradigm as a foundation for future long-horizon, streamable video generation.Source: §5 · p. 10