PaperlayerView exact revision on arXiv

Source-linked AI summary

Salt: Self-Consistent Distribution Matching with Cache-Aware Training for Fast Video Generation

Xingtong Ge, Yi Zhang, Yushi Huang, Dailan He, Xiahong Wang, Bingqi Ma, Guanglu Song, Yu Liu, Jun Zhang

arXiv:2604.03118v2cs.CVeess.IV

TL;DR

Extremely low-NFE video generation remains difficult because trajectory distillation can oversmooth complex motion, while DMD lacks explicit multi-step compositional regularization. Salt combines SC-DMD with cache-aware mixed-step training and feature alignment, consistently improving low-NFE quality and autoregressive rollout stability across diverse settings.

  • Problem

    Extremely low-NFE video generation remains challenging because trajectory-style distillation can produce over-smoothed appearance and weak motion, while DMD lacks explicit multi-step compositional constraints.

  • Method

    Salt combines DMD with SC-DMD’s endpoint-consistency regularizer and cache-aware mixed-step training with cache-conditioned feature alignment for autoregressive generation.

  • Results

    Salt consistently improves low-NFE video generation quality and long-horizon autoregressive stability across non-autoregressive and autoregressive paradigms while remaining compatible with diverse KV-cache mechanisms.

  • Takeaways & Limitations

    Combining distribution matching with compositional and cache-conditioned supervision addresses both multi-step denoising coherence and variable-cache-quality challenges in few-step video generation.

Abstract

from arXiv · show

Distilling video generation models to extremely low inference budgets (e.g., 2--4 NFEs) is crucial for real-time deployment, yet remains challenging. Trajectory-style consistency distillation often becomes conservative under complex video dynamics, yielding an over-smoothed appearance and weak motion. Distribution matching distillation (DMD) can recover sharp, mode-seeking samples, but its local training signals do not explicitly regularize how denoising updates compose across timesteps, making composed rollouts prone to drift. To overcome this challenge, we propose Self-Consistent Distribution Matching Distillation (SC-DMD), which explicitly regularizes the endpoint-consistent composition of consecutive denoising updates. For real-time autoregressive video generation, we further treat the KV cache as a quality parameterized condition and propose Cache-Distribution-Aware training. This training scheme applies SC-DMD over multi-step rollouts and introduces a cache-conditioned feature alignment objective that steers low-quality outputs toward high-quality references. Across extensive experiments on both non-autoregressive backbones (e.g., Wan~2.1) and autoregressive real-time paradigms (e.g., Self Forcing), our method, dubbed \textbf{Salt}, consistently improves low-NFE video generation quality while remaining compatible with diverse KV-cache memory mechanisms. Project page: https://xingtongge.github.io/Salt

1 Introduction

Salt targets the difficulty of distilling video generation to 2–4 NFEs by combining distribution matching with compositional regularization and cache-aware training. The method improves low-NFE quality and autoregressive rollout stability across non-autoregressive and autoregressive settings.

  • 2–4 NFEs remain challenging for video generation because complex dynamics can admit multiple plausible futures.
  • Trajectory-style objectives can average over plausible futures, producing over-smoothed appearance and conservative motion.
  • DMD produces sharper, more expressive low-NFE samples by matching the student output distribution to a strong reference.
  • DMD’s local supervision lacks structural constraints on composed denoising updates, allowing per-step errors to accumulate into multi-step drift.
  • SC-DMD augments DMD with a semigroup-defect regularizer that encourages endpoint-consistent composition while preserving distribution-matching sharpness.
  • Cache-aware mixed-step training uses K ∈{2, 4, 8} rollouts and cross-step feature alignment to improve low-step generation and rollout stability under variable KV-cache quality.
  • Salt consistently improves extremely low-NFE quality and long-horizon autoregressive stability across Wan 2.1, Self Forcing, and LongLive while adding no inference overhead.

2 Related Work

Video diffusion distillation methods primarily follow trajectory-based or distribution-based approaches. Related work also distinguishes autoregressive video generation by whether streaming models use training-based or training-free methods.

  • Trajectory-based distillation compresses sampling by approximating the teacher’s denoising dynamics.
  • Distribution-based methods target the generative distribution, including adversarial post-training and phased equilibrium procedures for video models.
  • Autoregressive video generation supports streaming and interactivity through causal factorization and includes both training-based and training-free approaches.

3 Method

Salt combines distribution matching with self-consistency regularization to make denoising updates compose coherently, then extends this design to autoregressive generation through cache-quality-aware training and feature alignment.

  • Preliminaries and motivation: DMD distills few-step generators by matching student and teacher distributions at sampled noise levels, but its local supervision does not constrain multi-step composition.Repeatedly applying learned updates can accumulate errors and cause drift as the number of denoising steps increases.
  • SC-DMD: Increasing DMD inference steps can degrade quality: in Wan 2.1 experiments, longer rollouts exhibit over-exposure and semantic degradation rather than improvement.The 16-step visualization loses helmet texture, corrupts motorcycle structure, and produces incoherent television details.
  • SC-DMD: SC-DMD retains DMD for distribution alignment and sharp samples while adding a semigroup-defect regularizer that aligns direct and composed student updates.The self-consistency loss compares a direct endpoint with a two-step endpoint through an intermediate timestep.
  • SC-DMD: The SC constraint anchors its endpoint to the inference grid while sampling intermediate times from a finer training grid, exposing richer shortcut structure.This design directly supervises test-time endpoints while retaining dense intermediate-time DMD supervision.

4 Experimental Results

Experiments across non-autoregressive and autoregressive video generation show that Salt improves low-step quality, compositional consistency, and long-horizon stability. Ablations attribute these gains to explicit self-consistency regularization and cache-aware training rather than denser timestep grids alone.

  • Non-autoregressive distillation: SC-DMD achieves the best I2V Score (93.90) and Imaging Quality (72.16) on Wan 2.1 14B at 4 NFEs.Background Consistency increases from 92.79 to 95.97, Motion Smoothness from 97.99 to 98.37, and Temporal Flickering from 95.21 to 97.41.
  • Non-autoregressive distillation: SC-DMD improves Wan 2.1 T2V over DMD at 4 NFEs, reaching Total Score 83.36 versus 82.78 and Quality Score 84.76 versus 84.39.An adversarial variant, Ours-α, further raises Quality Score to 81.71 and Dynamic Degree to 68.13.
  • Autoregressive generation: At 4 NFEs, Salt consistently improves Total Score and Quality Score across Self Forcing, LongLive, and Causal Forcing.On LongLive, Total Score rises from 84.40 to 84.93 and Quality Score from 85.12 to 85.41; on Self Forcing, Total Score rises from 84.20 to 84.47.
  • Autoregressive generation: 84.80 Total and 85.63 Quality are achieved by Salt at 2 steps in Causal Forcing, surpassing the original 4-step baseline’s 84.62 Total and 85.41 Quality.The result demonstrates effectiveness under a more aggressive inference budget.
  • Long-horizon generation: On 30-second rollouts, LongLive Total Score improves from 79.03 to 79.27 and Semantic Score from 63.88 to 64.74, while Causal Forcing Semantic Score rises from 60.25 to 62.77.The reported gains extend beyond short-horizon generation, especially in overall quality and semantic consistency.
  • Ablation study: SC-DMD reaches Total 83.36, Quality 84.76, and Semantic 77.77, whereas denser-grid DMD-8 declines relative to DMD-4 on Quality and Total.This supports explicit compositional regularization as the source of improvement rather than simply increasing training-grid density.
  • Ablation study: SC-DMD produces more consistent outputs across 2-, 4-, and 8-step sampling and is less prone to structural corruption than vanilla DMD.The comparison uses the same prompt and random seed; vanilla DMD shows step-dependent variation and artifacts such as duplicated heads.
  • Ablation study: Reference alignment improves the 2-NFE mixed-step+LSC variant from Semantic 80.65 to 81.49 and Total 84.63 to 84.80.The effect is consistent with guiding weaker low-step rollouts toward stronger high-step references.

5 Conclusion

Salt addresses compositionality deficits in DMD and cache-quality variability in autoregressive generation. Across Wan 2.1 and several autoregressive backbones, it improves few-step quality, cross-step consistency, and long-horizon semantic stability.

  • Conclusion: Salt combines SC-DMD’s semigroup-defect regularization with cache-conditioned mixed-step training and reference alignment.SC-DMD encourages endpoint-consistent denoising composition, while cache-aware training exposes the model to diverse cache regimes and strengthens weak-cache outputs.
  • Conclusion: Across non-autoregressive and autoregressive settings, Salt delivers consistent gains in few-step quality, cross-step compositional consistency, and long-horizon semantic stability.

A.1 Measuring Semigroup Defect on the Test-Time Inference Path

The appendix measures whether SC-DMD’s learned denoising operators compose coherently along the test-time 4-step path. SC-DMD yields a lower local semigroup defect than DMD, although this diagnostic is not a substitute for rollout-level benchmarks.

  • Diagnostic definition: The local semigroup defect directly diagnoses compositional behavior by comparing direct and composed endpoints for adjacent denoising intervals.The discrepancy is the quantity regularized by SC-DMD’s self-consistency loss.
  • Metric interpretation: Displacement normalization measures composition error relative to the magnitude of direct denoising transport rather than endpoint latent magnitude.This avoids misleadingly small errors when both trajectories move very little.
  • Results: 0.0111 vs. 0.0135: SC-DMD achieves a lower displacement-normalized local semigroup defect than the DMD baseline.The improvement is strongest on early and middle intervals, while the final low-noise interval shows a smaller gap.
  • Interpretation: The defect is a local compositionality diagnostic, not a replacement for rollout-level generation benchmarks.Its trend is nevertheless consistent with improved few-step quality and reduced degradation from multi-step composition.

A.2 Comparison with Shortcut-Style Consistency

SC-DMD outperforms shortcut-style consistency alternatives by retaining DMD as the primary distribution-matching objective and using self-consistency as a regularizer. The appendix attributes the naive hybrid’s degradation partly to mismatched training grids.

  • Comparison: SCFM substantially underperforms DMD-4 on Wan 2.1 T2V under the same 4-NFE evaluation setting.A naive DMD+SCFM hybrid further degrades performance, whereas SC-DMD obtains the best overall result.
  • Design: SC-DMD keeps DMD as the main distribution-matching objective and adds self-consistency as a regularizer.This preserves DMD’s role while addressing compositional behavior rather than replacing distribution matching with trajectory supervision.
  • Failure analysis: The naive hybrid combines shortcut supervision on a dense grid with DMD supervision on a coarse distillation grid.This leaves some intermediate operators jointly optimized and others trained only with shortcut-style supervision.

A.3 Additional Results on Standard VBench Prompts

Additional evaluations on the original short-prompt VBench set test whether Salt’s gains transfer beyond the prompt-extended protocol. SC-DMD improves all three non-autoregressive metrics and yields consistent autoregressive gains across three backbones.

  • Prompt transfer: The non-autoregressive gains remain effective on the original short-prompt benchmark rather than only under prompt expansion.The standard VBench evaluation was added specifically to test transfer beyond the main-paper prompt protocol.
  • Autoregressive results: On standard VBench prompts, Ours-Self Forcing improves Total from 82.16 to 82.43 and Quality from 84.89 to 85.29.These are reported for the Self Forcing autoregressive backbone.
  • Autoregressive results: On standard VBench prompts, Ours-Causal Forcing improves Total from 82.28 to 82.52 and Semantic from 70.35 to 70.89.The gains are reported for the Causal Forcing backbone.
  • Autoregressive results: On standard VBench prompts, Ours-LongLive improves Total from 82.31 to 82.60 and Semantic from 71.16 to 71.30.The results extend the evaluation across the LongLive backbone as well.

A.4 Additional VideoAlign Evaluation and Motion Analysis

VideoAlign provides an additional human-feedback-oriented assessment of autoregressive generation, while the motion analysis cautions against interpreting optical-flow magnitude as motion quality by itself. Salt improves VideoAlign motion quality across backbones despite a Dynamic Degree decrease in one family.

  • VideoAlign results: VideoAlign scores increase by +0.73, +0.54, and +1.07 on Self Forcing, LongLive, and Causal Forcing, respectively.The evaluation uses the same prompt-extended 5-second setting as the main autoregressive evaluation.
  • Motion quality: VideoAlign’s MQ dimension also improves across all three autoregressive backbones.MQ denotes the motion-quality dimension from VideoAlign.
  • Metric interpretation: Dynamic Degree can rise because of flicker, abrupt scene changes, or noisy transitions even when perceptual motion quality worsens.It is based on optical-flow magnitude, so larger values do not always indicate better motion.
  • Causal Forcing analysis: In Causal Forcing, Dynamic Degree decreases from 76.95 to 54.02 while Semantic Score, Subject Consistency, Temporal Flickering, and Motion Smoothness improve.Independent VideoAlign MQ gains across all three backbones support interpreting the decrease as compatible with better motion quality.

B Implementation Details

The non-autoregressive experiments evaluate SC-DMD in Wan 2.1 text-to-video and image-to-video settings, while autoregressive training uses specified optimizer, rollout, and update configurations.

  • Non-autoregressive distillation: SC-DMD is evaluated for text-to-video with Wan 2.1 T2V 1.3B and image-to-video with Wan 2.1 I2V 14B teacher models.Non-autoregressive experiments use 16 H200 GPUs, total batch size 16, and AdamW.
  • Non-autoregressive distillation: Wan 2.1 I2V 14B training runs for 800 iterations, while Wan 2.1 T2V 1.3B training runs for 2400 iterations.
  • Autoregressive training: Autoregressive training uses AdamW for both generator and critic, with learning rates of 2 × 10−6 and 4 × 10−7, respectively.Backward simulation is enabled by default.
  • Autoregressive training: Mixed-step autoregressive rollouts sample K ∈{2, 4, 8} with probabilities {0.2, 0.4, 0.4}, activating the SC loss only when K = 8.Standard DMD updates apply for all sampled step counts.

C More visualization results

Additional visualizations show that the method improves visual quality, semantic consistency, structural stability, and motion progression relative to the Causal Forcing baseline.

  • Overall comparison: The method shows clearer advantages than Causal Forcing in both visual quality and semantic consistency.These qualitative comparisons are reported as consistent with the paper’s quantitative improvements.
  • Subject identity and scene semantics: The reading-girl example shows that the method preserves character identity, clothing, and scene context while the baseline exhibits semantic and identity drift.
  • Visual and structural consistency: The trombone example shows cleaner object structure and more stable subject-object relations than the baseline.The comparison concerns instrument geometry, body pose, and stage composition across frames.
  • Visual and structural consistency: The grape example shows better preservation of spatial layout, object count, and overall composition, producing more visually coherent and realistic videos.
  • Motion progression: The method produces motion that evolves more naturally over time, whereas baseline frames often appear loosely related.The passage connects cache-aware training with improved autoregressive rollout stability, frame quality, and cross-frame semantic consistency.
Loading 2604.03118v2…