PaperlayerView exact revision on arXiv

Source-linked AI summary

Phased DMD: Few-step Distribution Matching Distillation via Score Matching within Subintervals

Xiangyu Fan, Zesong Qiu, Zhuguanyu Wu, Fanzhou Wang, Zhiqian Lin, Tianxiang Ren, Dahua Lin, Ruihao Gong, Lei Yang

arXiv:2510.27684v3cs.CV

TL;DR

One-step distilled models have limited capacity that diminishes output diversity and undermines performance on complex tasks, while direct few-step distillation raises computational depth and memory demands. Phased DMD is a few-step distillation framework that uses gradient truncation to manage memory constraints while avoiding one-step degeneration and preserving generative diversity and motion dynamics. Phased DMD improves motion dynamics and visual fidelity in video generation and increases output diversity in image generation.

  • Problem

    One-step distilled models have limited capacity that diminishes output diversity and undermines performance on complex tasks, while direct few-step distillation raises computational depth and memory demands.

    Source: §1 · p. 2

  • Method

    Phased DMD is a few-step distillation framework that uses gradient truncation to manage memory constraints while avoiding one-step degeneration and preserving generative diversity and motion dynamics.

    Source: §1 · p. 2

  • Results

    Phased DMD improves motion dynamics and visual fidelity in video generation and increases output diversity in image generation.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The data-free few-step framework has higher performance ceilings while keeping memory usage similar to single-step distillation.

    Source: §1 · p. 3

    The improvement is less pronounced for Qwen-Image, whose outputs are inherently less diverse.

    Source: §5 · p. 11

Abstract

from arXiv · show

Distribution Matching Distillation (DMD) distills score-based generative models into efficient one-step generators, without requiring a one-to-one correspondence with the sampling trajectories of their teachers. Yet, the limited capacity of one-step distilled models compromises generative diversity and degrades performance in complex generative tasks, e.g., generating intricate object motions in text-to-video task. Directly extending DMD to multi-step distillation increases memory usage and computational depth, leading to instability and reduced efficiency. While prior works propose stochastic gradient truncation as a potential solution, we observe that it substantially reduces the generative diversity in text-to-image generation and slows motion dynamics in video generation, reducing performance to the level of one-step models. To address these limitations, we propose Phased DMD, a multi-step distillation framework that bridges the idea of phase-wise distillation with Mixture-of-Experts (MoE), reducing learning difficulty while enhancing model capacity. Phased DMD incorporates two key ideas: progressive distribution matching and score matching within subintervals. First, our model divides the SNR range into subintervals, progressively refining the model to higher SNR levels, to better capture complex distributions. Next, to ensure accurate training within each subinterval, we derive rigorous mathematical formulations for the objective. We validate Phased DMD by distilling state-of-the-art image and video generation models, including Qwen-Image-20B and Wan2.2-28B. Experiments demonstrate that Phased DMD enhances motion dynamics, improves visual fidelity in video generation, and increases output diversity in image generation. Our code and models are available at https://x-niper.github.io/projects/Phased-DMD/.

1 Introduction

Phased DMD targets the capacity limits of one-step models and the memory and stability costs of direct few-step distillation. It progressively matches distributions across SNR subintervals and reports improved diversity, motion dynamics, and video fidelity.

1 Introduction

  • One-step distillation can reduce diversity and impair complex generation, while direct few-step training raises memory use and graph depth and can destabilize optimization.SGTS improves convergence and memory efficiency, but iterations that terminate after one step can leave diversity and video motion dynamics comparable to one-step generators.Source: §1 · p. 2
  • Because low-SNR stages model visual structure and dynamics while high-SNR stages refine details, assigning experts to SNR levels can raise capacity without increasing inference cost.The cited MoE work particularly reports strong low-SNR experts for video dynamics.Source: §1 · p. 2
  • Phased DMD distills one expert per SNR subinterval, advancing toward higher SNR and ending backward simulation at that phase’s SNR rather than the clean state.The paper distinguishes this progression from progressive distillation, which halves sampling steps in each phase.Source: §1 · p. 2
  • Phased DMD derives an unbiased score-estimation objective for subinterval training when clean samples are unavailable, a condition in all but the final phase.The paper identifies objective correctness as necessary for DMD distillation.Source: §1 · p. 3
  • Phased DMD achieves SOTA results on text-to-image and text-to-video, increasing image diversity and enhancing video motion dynamics and visual fidelity.The reported evaluations distill Qwen-Image and Wan models.Source: §1 · p. 3
  • The framework combines DMD and MoE ideas in a data-free few-step distillation method, with memory usage similar to single-step distillation.It can also be combined with SGTS to train a four-step generator within two phases.Source: §1 · p. 3

2 Method

Phased DMD builds few-step distillation around phased SNR progression and subinterval score matching. Its method connects diffusion score estimation and DMD to intermediate-timestep training, with objectives adapted for phases where clean samples are unavailable.

2.1.2 Distribution Matching Distillation

  • DMD minimizes reverse KL divergence using a generator, a trainable fake score estimator, and a frozen pretrained teacher score estimator.The generator update is guided by the discrepancy between teacher and fake scores.Source: §2.1.2 · p. 4

2.1.3 Few-step Generator

  • N-step distillation generates samples iteratively with a scheduler, but its computational graph grows linearly with step count and raises memory use.Stochastic gradient truncation can improve memory efficiency and stability, but compromises diversity, prompt-following, and video-motion speed.Source: §2.1.3 · p. 5

2.2 Phased DMD

  • Each phase optimizes an expert at an intermediate timestep while matching distributions and score-matching over a diffusion subinterval.This avoids relying on clean samples in phases before the final one.Source: §2.2 · p. 5

2.2.1 Distribution Matching at Intermediate Timesteps

  • For intermediate phases, the generator produces x_tk, which is diffused over t sampled from (t_k, 1) for its distribution-matching objective.The authors report that sampling from (t_k, 1) aligns better with the progressive architecture and performs better than sampling from (t_k, t_{k-1}).Source: §2.2.1 · p. 5
  • Phased DMD progressively distills toward higher SNR levels, with each expert mapping one phase's input distribution to the next.The fake diffusion model for each phase is initialized from the pretrained teacher.Source: §2.2.1 · p. 5

2.2.2 Score Matching Within Subintervals

  • When clean samples are unavailable, Phased DMD trains the fake diffusion model with an unbiased score-matching objective over a subinterval.The phase-specific model is optimized over (t_k, 1), with five fake-model updates for each generator update.Source: §2.2.2 · pp. 5–6
  • Clamping the final loss addresses the singularity and numerical instability that arise as σ_t|s approaches zero when t approaches s.The unclamped formulation is reported to encounter that singularity.Source: §2.2.2 · p. 6
  • In a four-value 1D toy experiment, trajectories from the correct subinterval objective overlap with those from the full-interval objective, unlike trajectories from an incorrect target.The incorrect objective induces biased score estimation and violates the A2 assumption.Source: §2.2.2 · p. 6, Figure 2 · p. 6

3 Experiments and Results

Across text-to-video, image-to-video, and text-to-image evaluations, Phased DMD is compared with base and distilled models. It outperforms DMD2 in video motion and quality, improves diversity over other distilled methods, and preserves Qwen-Image text rendering.

3 Experiments and Results

  • Experiments cover T2I, T2V, and I2V with Wan2.1-T2V-14B, Wan2.2-T2V-A14B, Wan2.2-I2V-A14B, and Qwen-Image-20B; vanilla few-step DMD is tested only on the smallest configuration.The Wan2.2 distillations use two experts under equivalent parameter budgets for standard DMD and Phased DMD.Source: §3 · pp. 6–7

3.1 Phased DMD in Video Generation

  • Phased DMD outperforms DMD2 on OF, DD, FID, and FVD for both T2V and I2V.The evaluation reports stronger motion dynamics and video quality on both tasks.Source: Table 2 · p. 7, §3.1 · p. 9
  • In video comparisons, DMD2 produces slower motion than the base model and Phased DMD, and tends toward close-ups rather than prompt-specified camera views.The evaluation uses 220 text prompts for T2V and 220 image-prompt pairs for I2V, with one video per prompt.Source: §3.1 · p. 7

3.2 Phased DMD in Image Generation

  • Phased DMD preserves image diversity better than vanilla few-step DMD and DMD2, producing more varied compositions and lighting across generated samples.The comparison uses 21 prompts and eight random-seed images per prompt.Source: §3.2 · p. 9
  • Phased DMD retains Qwen-Image’s high-quality image generation and accurate text rendering.The examples use prompts from Qwen-Image’s official website.Source: §3.2 · p. 9, Figure 7 · p. 10
  • Phased DMD outperforms vanilla DMD and DMD2 on both diversity metrics, although its diversity improvement on Qwen-Image is marginal.The metrics are average pairwise DINOv3 cosine similarity and LPIPS distance.Source: §3.2 · p. 9, Table 3 · p. 10

3.3 Merit of MoE

  • Low-SNR phases establish composition; freezing that expert lets high-SNR training refine lighting and texture while preserving the layout.The reported examples show that extended high-SNR expert training primarily changes lighting and textural detail.Source: §3.3 · p. 10, Figure 8 · p. 10

4 Related Work

The section situates Phased DMD in VSD-based distillation and contrasts DMD's computational graph with SID's. It also distinguishes Phased DMD from TDM by its theoretical grounding, MoE generation, and reverse-nested SNR intervals.

VSD framework

  • VSD trains a generator to approximate real data, a fake score estimator to model generated outputs, and uses the discrepancy between fake and teacher scores to update the generator.The pretrained diffusion model can initialize all three components, which the passage says enhances training stability.Source: §4 · pp. 10–11

DMD and SID

  • DMD minimizes reverse KL and avoids backpropagating through fake and teacher score estimators during generator optimization, unlike SID's Fisher-divergence approach.The passage says this computational-graph difference makes DMD more amenable to engineering implementation and scaling to large base models.Source: §4 · p. 11

TDM comparison

  • Phased DMD differs from TDM [27] through its theoretical grounding, inherent MoE generation, and reverse-nested rather than disjoint SNR intervals.The passage characterizes TDM's fake-flow formulation as suboptimal due to insufficient theoretical grounding.Source: §4 · p. 11

5 Conclusion and Discussion

Phased DMD primarily improves structural generation, including composition diversity, motion dynamics, and camera control. Its gains are less pronounced for Qwen-Image, and the paper prioritizes the data-free paradigm over trajectory-data integration.

Scope of improvements

  • Phased DMD primarily enhances structural generation, including composition diversity, motion dynamics, and camera control.The paper notes that improvements are less pronounced for Qwen-Image because its outputs are inherently less diverse.Source: §5 · p. 11

Model-specific boundary

  • For Qwen-Image, inherently lower output diversity corresponds to less pronounced improvement.This qualifies the reported improvement scope for that base model.Source: §5 · p. 11

Future directions

  • Integrating trajectories pre-generated by the base model is a possible direction, but would compromise DMD's data-free advantage, which this work prioritizes.Extending phased distillation to Fisher divergence in SiD [51] is also left for future exploration.Source: §5 · p. 11

A Detailed Derivation

The appendix outlines derivations for the paper's objectives, including the relationship between x-prediction and score matching. It identifies the conditional score and DSM–ESM equivalence used in one derivation, then states the subinterval objective for x-prediction models.

Eq. 5 derivation

  • The derivation of Eq. 5 uses the score of p(x_t|x_0) and the equivalence between DSM and ESM.The displayed conditional-score expression is given as ∇x_t log(p(x_t|x_0)) = −1/σ_t ε.Source: A Detailed Derivation · p. 15

Eq. 11 derivation

Prediction and score matching

Subinterval objective

B Experimental Details

The experimental details specify task-specific training resolutions and timestep shifts for Wan and Qwen-Image models. Wan2.2 training also uses phase-dependent models and timestep ranges, while Figure 9 illustrates how steps and shift determine intervals.

Wan resolutions

  • Wan2.2 text-to-video and image-to-video training samples four 81-frame resolutions with probabilities 0.1, 0.1, 0.4, and 0.4, while Wan text-to-image uses one 1280×720 frame.The four video resolutions are (81, 720, 1280), (81, 1280, 720), (81, 480, 832), and (81, 832, 480).Source: B Experimental Details · p. 16

Timestep intervals

  • Wan2.x uses timestep shift 5 and Qwen-Image shift 3; Figure 9 shows intervals determined by the number of steps and the shift.The figure describes re-noising ranges as (min step, max step), with t sampled uniformly before applying the shift.Source: B Experimental Details · p. 16, Figure 9 · p. 16

Wan2.2 phases

  • In Wan2.2's first phase, only the high-noise model is used and t is restricted to (0.875, 1) for text-to-video or (0.9, 1) for image-to-video; the second phase uses both noise models.During phase two, the low-noise generator and both high- and low-noise fake models are trainable, with teacher and fake-model selection determined by the re-noising timestep.Source: B Experimental Details · pp. 16–17

C.1 Additional Quantitative Evaluation on Video Generation

Additional video evaluations compare distilled models with their Wan2.2 base models, while the authors caution that quantitative rankings do not match human preferences. A four-step, four-phase configuration improves motion dynamics and visual quality over the four-step, two-phase configuration, with greater system complexity.

C.1 Additional Quantitative Evaluation on Video Generation

  • Table 4 covers text-to-video comparisons for Wan2.2-T2V-A14B, and Table 5 covers image-to-video comparisons for Wan2.2-I2V-A14B.Source: Table 4 · p. 17, Table 5 · p. 17
  • Human preferences favor the base model over both distilled variants, while VBench metrics rank DMD2 best and the base model worst; Phased DMD is reported to preserve base-model performance better than DMD2.The authors say the gap between distilled models and the base model is more closely aligned for Phased DMD, which they interpret as better preservation of the base model’s generative distribution.Source: §C.1 · p. 17
  • The four-step, four-phase configuration performs better in motion dynamics and visual quality than the four-step, two-phase configuration, but increases system complexity.The authors attribute the improvement to avoiding SGTS and including more trainable parameters.Source: §C.1 · p. 17

C.2 Ablation on Diffusion Timestep Subintervals

The timestep ablations favor reverse nested noise-injection intervals over disjoint intervals and show that low-noise-only injection can fail to converge. The authors connect this pattern to the need to train across high-noise levels in each phase.

C.2 Ablation on Diffusion Timestep Subintervals

  • Reverse nested intervals yield normal color tones and accurate structures, whereas disjoint intervals produce low-contrast tones and degraded facial structures in Wan2.2 text-to-video distillation.The figure caption describes the reverse nested intervals as ending at timestep 1.0, unlike the non-overlapping disjoint intervals.Source: §C.2 · p. 17, Figure 10 · p. 18
  • At the start of each phase, generated samples differ substantially from real samples, especially at high SNR, while low-SNR diffused distributions overlap more and permit more accurate teacher score estimates.This motivates high-noise injection during DMD training.Source: §C.2 · pp. 17–18
  • In vanilla DMD on Wan2.1 text-to-image, fixing t at 0.357 fails to converge, while t = 0.882 produces correct results, supporting the importance of including high-noise levels.Both settings are described as theoretically valid under the cited KL-divergence equivalence.Source: §C.2 · p. 18
  • The figure reports that DMD training does not converge correctly when noise is injected exclusively at a low noise level.Source: Figure 11 · p. 19

C.3 Additional Discussion on MoE

In diffusion MoE, dense experts specialize in denoising ranges and are applied sequentially as SNR rises. Phased DMD is compatible with this setup, and its phases align with the base model’s expert count.

C.3 Additional Discussion on MoE

  • Diffusion MoE assigns dense experts to denoising ranges: low-SNR experts model global structure and dynamics, while high-SNR experts refine local details.The figure compares a high-noise expert over the first two denoising steps with the combined pipeline over all four steps.Source: §C.3 · p. 19, Figure 12 · p. 20
  • MoE can increase capacity and generative quality without a proportional increase in inference cost, with the cited performance gain particularly pronounced in video generation.Source: §C.3 · p. 19
  • For a base model with N experts, Phased DMD’s stated optimal practice is N-phase training, with one trainable generator and k trainable fake models in phase k.Source: §C.3 · p. 19

Related papers

Loading 2510.27684v3…