Source-linked AI summary
Uncertainty DMD: Restoring Diversity in Few-Step Autoregressive Video Distillation
Zixuan Duan, Xunzhi Xiang, Yabo Chen, Xin Zhang, Changhan Liu, Haibin Huang, Chi Zhang, Qi Fan, Xuelong Li
TL;DR
Few-step DMD makes autoregressive video generation efficient but can collapse diversity and motion because similar first chunks are propagated by the deterministic cache. Uncertainty DMD injects stochasticity through first-chunk timestep perturbation and stochastic cache writing, improving diversity and motion dynamics while maintaining comparable visual quality.
Problem
Few-step DMD can produce highly similar videos from different noise samples under one prompt, reducing diversity and motion dynamics despite preserving per-sample quality.
Method
Uncertainty DMD perturbs the first-chunk timestep and stochastically writes the autoregressive cache during both training and inference without architectural changes.
Results
Uncertainty DMD improves sample diversity and motion dynamics while maintaining comparable visual quality.
Takeaways & Limitations
Restoring uncertainty at the rollout root and in autoregressive conditioning mitigates diversity collapse in DMD-based autoregressive video generation.
Abstract
from arXiv · showhide
Few-step distillation improves the efficiency of autoregressive (AR) video generation, but often causes diversity collapse: under the same prompt, different noise samples tend to produce highly similar videos with weakened motion dynamics. We analyze this degradation in Distribution Matching Distillation (DMD)-distilled AR video generators and find that, in the autoregressive setting, it takes the form of a structured uncertainty collapse: the mode-seeking bias of DMD maps different noise samples to nearly identical first chunks, and the deterministic AR cache then propagates this collapsed state to all subsequent chunks, turning a local loss of stochasticity at the rollout root into a global suppression of temporal variation. Based on this analysis, we propose Uncertainty DMD, a simple uncertainty-injection framework that restores stochasticity at two key stages of AR generation: a timestep perturbation for the first chunk to increase first-chunk diversity, and a stochastic cache-writing mechanism for later chunks to preserve uncertainty in autoregressive conditioning. The method requires no architectural changes and introduces only lightweight perturbation operations. The same perturbation mechanisms are used during both training and inference. Experiments show that Uncertainty DMD consistently improves diversity and motion dynamics while maintaining comparable per-sample visual quality.
1 Introduction
The introduction identifies uncertainty collapse in DMD-distilled autoregressive video generation and proposes structured uncertainty injection to restore diversity and motion while preserving visual quality.
- DMD distillation can preserve individual-video quality while substantially reducing per-prompt diversity and motion dynamics.
- Mode-seeking DMD collapses different noise samples into similar first chunks, while the deterministic cache propagates this collapse through later chunks.
- Uncertainty DMD perturbs the first-chunk timestep and stochastically writes the autoregressive cache to create and preserve diverse rollout branches.
- The method applies the same uncertainty injection during training and inference, avoiding train–test mismatch without architectural modification.
- Experiments on two DMD-based autoregressive video generators show improved diversity and motion dynamics while preserving visual quality.
2 Related Work
Related work situates autoregressive generation as a sequential alternative to costly full-sequence denoising and reviews diversity-preservation interventions in distilled models.
- Autoregressive video generation synthesizes frames or chunks sequentially, whereas many video diffusion models use costly bidirectional or full-sequence denoising.
- Prior diversity methods intervene at early denoising steps, preserve first-step diversity, or use prompt-space policies, but some interventions may trade off sample quality.
3 Diagnosing Mode Collapse in Few-Step AR Distillation
Few-step DMD collapses diversity at the first chunk, while deterministic cache conditioning propagates that collapse through later chunks. Variance decomposition and sampler-swap studies identify the cache as the dominant source of later-chunk diversity and motivate targeting the first-chunk sampler and cache.
- 3.2 DMD Mode Collapse and First Chunk Collapse: DMD’s mode-seeking score update concentrates samples in high-density teacher regions, reducing diversity under different random noise.This mechanism is associated with reduced sensitivity to noise in few-step students.
- 3.2 DMD Mode Collapse and First Chunk Collapse: Different noise samples can map to similar high-probability first chunks, producing first-chunk collapse in layouts, poses, viewpoints, and initial motion.The first chunk must determine global scene and motion factors from the prompt and random noise because it has no historical cache.
- 3.3 Deterministic Cache Propagates Diversity Collapse: Later-chunk diversity separates into innovation diversity under a fixed cache and cache-inherited diversity caused by differing historical caches.The first term measures new semantic or motion variation from the current sampler; the second measures variation from different cache states.
- 3.3 Deterministic Cache Propagates Diversity Collapse: 0.94–0.97 of later-chunk variance is cache-inherited across history lengths, while innovation contributes only a small fraction.The experiment uses 100 prompts, history lengths 1–4, 8 history caches, and 8 continuations per cache.
- 3.3 Deterministic Cache Propagates Diversity Collapse: Cache transplantation shows that teacher-derived first-chunk caches yield more diverse later generations than DMD-derived caches, even with identical later noise.The deterministic cache therefore propagates first-chunk collapse through the rollout rather than merely reflecting a local sampling failure.
- 3.4 Testable Prediction and Motivation: Replacing only the first-chunk sampler with the teacher recovers most diversity, whereas replacing only later samplers provides much smaller gains.Table 1 compares Teacher / DMD, DMD / DMD, and DMD / Teacher to isolate where diversity is lost.
4 Methodology
Uncertainty DMD restores stochasticity by perturbing two autoregressive state variables: the first-chunk timestep and the cache written for future chunks.
- 4 Methodology: Uncertainty DMD injects stochasticity into the first-chunk timestep condition and the autoregressive cache used by later chunks.These two structured perturbations target the AR state variables that steer rollout diversity.
- 4.1 Module A: Timestep Perturbation: The first chunk keeps its initial noisy latent but replaces timestep T with a clipped perturbed timestep T−Δ, changing its denoising trajectory.The timestep offset is implemented with Δ sampled uniformly from 0 to 100.
- 4.1 Module A: Timestep Perturbation: Timestep mismatch induces different first-chunk branches without directly perturbing the latent, diversifying subsequent autoregressive rollouts.During training, root perturbation is applied according to a curriculum-controlled probability.
- 4.2 Module B: Stochastic Cache Writing: For later chunks, the method optionally perturbs each predicted clean chunk once before inserting it into the autoregressive cache.The perturbation uses a forward noising operator controlled by p_cache and τ_cache, after which the cached entry remains fixed.
- 4.3 Curriculum and Algorithm: The same stochastic cache-writing rule is used during training and inference, with each perturbed cache entry fixed for subsequent autoregressive steps.A warm-up and ramp schedule gradually increases the perturbation probabilities; inference uses p_root = 1 and p_cache = p_max.
5 Experiments
Experiments evaluate Uncertainty DMD on autoregressive video generators using diversity and quality benchmarks, ablations, generalization tests, and training-dynamics analyses.
- 5 Experiments: Uncertainty DMD consistently improves diversity while maintaining comparable VBench quality, and user studies report substantially higher diversity with comparable perceptual quality.The method also outperforms first-step alternatives with comparable quality and is evaluated on 2,000 generated videos per method.
- Ablation study: TE-CLIP drops from 19.5 to 16.7 and VENDI-DINO from 3.3 to 2.6 when timestep perturbation is removed, showing its importance for seed-dependent diversity.Removing stochastic cache writing mainly lowers Dynamic Degree from 42.4 to 37.0.
- Ablation study: Without curriculum, TE-CLIP decreases from 19.5 to 18.4, VENDI-DINO from 3.3 to 3.1, and Dynamic Degree from 42.4 to 37.4.The full method achieves the best balance between diversity and dynamic generation quality.
- Generalization study: The method generalizes across Wan2.1 1.3B frame-wise and 14B chunk-wise models, supporting use across model sizes and temporal granularities.The generalization results are reported in Table 4.
- Training dynamics: Across training steps, Uncertainty DMD maintains competitive VBench quality, higher later-stage Dynamic Degree, and consistently higher diversity than DMD and DMD + GAN.The training comparison indicates slower diversity degradation during distillation.
- Timestep perturbation: Varying only the first-step timestep produces visibly different predictions from fixed noise and prompt, demonstrating added stochasticity in generation trajectories.This experiment isolates the effect of timestep perturbation.
6 Conclusion
Uncertainty DMD combines timestep perturbation with stochastic cache writing to mitigate diversity collapse in autoregressive video distillation. It improves sample diversity and motion dynamics while maintaining comparable visual quality, though complex motions may remain unnatural.
- Uncertainty DMD combines timestep perturbation with stochastic cache writing to mitigate diversity collapse in DMD-based autoregressive video distillation.
- The first-chunk timestep effect is visualized across four evenly spaced timesteps using the same noise sample and prompt.
- Uncertainty DMD improves sample diversity and motion dynamics while maintaining comparable visual quality, but complex motions may remain unnatural.