PaperlayerView exact revision on arXiv

Source-linked AI summary

SGMD: Score Gradient Matching Distillation for Few-Step Video Diffusion Distillation

Zhuguanyu Wu, Ruihao Gong, Yang Yong, Yushi Huang, Xiangyu Fan, Lei Yang, Dahua Lin, Xianglong Liu

arXiv:2605.30116v2cs.CVcs.LG

TL;DR

DMD-style few-step video distillation faces costly fake-score tracking and potentially conservative motion matching. SGMD combines teacher stop-gradient Fisher with dual-potential tracking updates; it reports motion and temporal-consistency gains, faster training, and human preference for motion quality and overall preference.

  • Problem

    DMD-style few-step video distillation faces costly fake-score tracking and reverse-KL-style matching that can conservatively suppress motion in motion-rich video.

    Source: §1 · pp. 1–2

  • Method

    SGMD directly aligns the fake score toward the teacher under teacher stop-gradient Fisher, using NR for outer-loop correction and RC for inner-loop residual contraction.

    Source: §1 · p. 2, §3.4 · p. 4

  • Results

    Approximately ∼3× training speedup accompanies improved motion dynamics and temporal consistency under 4-step distillation, with human-preference gains in motion quality and overall preference and comparable visual quality and text alignment.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    SGMD makes the Fisher-style signal usable in practice with fewer fake-score updates through a lightweight two-step bilevel update.

    Source: §1 · p. 2

    Using L(1)(θ) with L(2)(θ) as in SIM can fail to converge for large-scale models, producing blurry and less sharp outputs.

    Source: Then the effective outer-loop descent… · p. 3

Abstract

from arXiv · show

Distribution Matching Distillation (DMD) is a widely used paradigm for accelerating inference in few-step video diffusion models. However, DMD-style video distillation faces two coupled challenges: the fake score must track a continuously evolving generator, making training costly when frequent updates are required, while reverse-KL-style matching can be mode-seeking and conservative for preserving strong motion dynamics. To address these issues, we propose \textbf{Score Gradient Matching Distillation (SGMD)}. SGMD adopts a fake-score perspective by directly optimizing the fake score toward the teacher, while using teacher stop-gradient Fisher as a stable distribution-matching objective. We provide a gradient analysis that motivates this objective choice under ideal tracking. Building on this, SGMD introduces a pair of dual potentials: negative-residual (NR) for outer-loop correction and residual-contraction (RC) for inner-loop tracking. Empirically, compared to DMD2, SGMD achieves an approximately $\sim 3\times$ training speedup and substantially improves motion dynamics for 4-step distilled models while preserving temporal consistency. A human study confirms that SGMD is preferred in motion quality and overall preference, while visual quality and text alignment remain comparable. Code is available at https://github.com/ModelTC/LightX2V.

1. Introduction

Few-step DMD-style video distillation is challenged by costly fake-score tracking and conservative motion under reverse-KL-style matching. SGMD pairs teacher stop-gradient Fisher with cooperative fake-score tracking and dual potentials to address these challenges.

1. Introduction

  • DMD-style distillation faces costly fake-score updates to track a rapidly evolving generator, while reverse-KL-style matching can conservatively suppress motion in motion-rich video.These are coupled practical challenges in aggressive few-step regimes.Source: §1 · pp. 1–2
  • Teacher stop-gradient Fisher is motivated as a stable distribution-matching objective whose ideal-tracking outer update is consistent with DMD-style matching without teacher input gradients.The passage also identifies its stable gradient structure.Source: §1 · p. 2
  • SGMD aligns the fake score toward the teacher and uses the generator as a score-consistent tracker, with NR correcting the outer update and RC contracting inner-loop residuals.This yields a lightweight two-step bilevel update with fewer fake-score updates.Source: §1 · p. 2
  • SGMD reports improved motion dynamics and temporal consistency under 4-step distillation, an approximately ∼3× training speedup, and human-preference gains in motion quality and overall preference.Visual quality and text alignment remain comparable.Source: §1 · p. 2

2. Preliminaries

The preliminaries describe DMD's generator, teacher, and auxiliary fake score, along with the alternating updates that create a tracking-cost bottleneck. They then introduce SIM's Fisher Divergence objective and its explicit and implicit gradient components.

2.1. DMD

  • DMD trains a generator against a fixed diffusion teacher using an auxiliary fake score that approximates the generator distribution's score.The teacher is kept frozen.Source: §2.1 · p. 2
  • DMD alternates inner fake-score regression with outer generator updates, but tracking error can require multiple fake-score updates and reducing them risks lag, instability, and degraded sample consistency.The outer update assumes the fake score is sufficiently up-to-date.Source: §2.1 · p. 2

2.2. SIM

  • SIM takes Fisher Divergence as its training objective and expresses its gradient through explicit and implicit contributions.The implicit contribution arises because the optimal inner solution depends on the generator parameter.Source: §2.2 · p. 2
  • SIM obtains the implicit-gradient contribution by minimizing a loss, and presents the implicit and explicit terms in x-prediction form before giving the overall loss.The exposition specifies stop-gradient notation for the loss formulation.Source: §2.2 · p. 3

3. Method

The method uses teacher stop-gradient Fisher matching and a fake-score perspective to address unreliable teacher input gradients and fake-score tracking lag. SGMD combines outer-loop direction correction with inner-loop residual contraction in a lightweight two-step update.

3.1. Teacher Stop-Gradient Fisher Divergence

  • Teacher stop-gradient Fisher provides a stable alternative after teacher input gradients made training extremely unstable, which the authors attribute to unreliable generated out-of-distribution states.Backpropagation can amplify those teacher gradients.Source: §3.1 · p. 3

Then the effective outer-loop descent direction induced

  • Under ideal tracking, the teacher stop-gradient Fisher objective induces the same descent direction as DMD’s reverse-KL-style matching, aligned with the score difference.This direction equivalence is stated for the ideal-tracking condition.Source: Then the effective outer-loop descent… · p. 3, §3.1 · p. 3

3.2. Fake-Score Perspective

  • SGMD treats the fake score as the primary target to move toward the teacher, while the generator tracks it to maintain score-consistency.The fake score depends on both its model parameters and the generator-induced distribution.Source: §3.2 · p. 3

3.3. Gradient Analysis

  • Tracking lag can bend Fisher’s net update direction, while systematic deviation from the desired direction leads to conservative updates and less vivid motion and details.Figure 2 contrasts Fisher, SIM, and SGMD under coupled one-iteration updates.Source: §3.3 · p. 4, Figure 2 · p. 4

3.3.1. FISHER SCORE MATCHING

  • With the fake score fixed, Fisher matching yields a clean outer direction, but refitting the fake score to the updated generator changes the coupled iteration’s net behavior.The generator update alone does not capture the subsequent tracking step.Source: §3.3.1 · p. 4

3.4. SGMD

  • SGMD’s two complementary potentials correct the generator’s outer-loop direction and contract the fake-score tracking residual in the inner-loop update.The residual measures the tracking gap between the generator output and the fake score’s x0-prediction.Source: §3.4 · p. 4, §3.4.1 · p. 5

3.4.2. OVERALL OBJECTIVE AND TWO-STEP UPDATE

  • Each iteration updates the generator with Fisher plus NR, then the fake score with RC, using two backward passes without second-order implicit-gradient terms.The weight λ balances distribution matching and tracking correction.Source: §3.4.2 · p. 5

3.3.2. SIM

  • SIM can improve tracking, but its generator update is not the unbiased Fisher outer direction, and its relative strength between behaviors cannot be tuned.The paper reports that SIM can fail to converge effectively for large-scale models, producing blurry, less-sharp outputs.Source: Then the effective outer-loop descent… · p. 3, §3.4.3 · p. 5

4. Experiments

Under 4-step distillation, SGMD strengthens motion dynamics while remaining competitive on visual quality and semantics, and human evaluations favor its motion and overall quality. Its dual potentials support stable training with fewer fake-score updates, yielding an approximately 3× speedup under the reported settings.

4.1. Experimental setup

  • Experiments distill the Wan2.1-T2V-14B teacher using prompt-only training and compare SGMD with DMD2, TSG-Fisher, and TSG-SIM under shared evaluation settings.Evaluation includes VBench, FVD, and optical-flow motion intensity; checkpoints are generally evaluated after 300 iterations.Source: §4 · p. 5, §4.1 · p. 6

4.2. Results

  • SGMD delivers stronger OptFlow and DynDeg than DMD2, remains competitive on VBench quality and semantic scores, and achieves the best FVD among distilled models.Table 1 reports results on VBench-T2V, FVD, and optical-flow motion metrics under 4-step distillation.Source: §4.2 · p. 6, Table 1 · p. 6
  • SGMD shows clearer temporal progression and larger motion changes than DMD2 under comparable sharpness and visual quality, while maintaining good temporal consistency.The qualitative comparison previews frames 0, 16, 32, 48, 64, and 80 from each 81-frame video.Source: Figure 3 · p. 7, §4.2 · p. 8
  • In pairwise human evaluation of DMD2 versus SGMD, SGMD is preferred for overall preference (65%) and motion quality (71%); text-video alignment (90%) and visual quality (74%) are mostly ties.VideoAlign shows a consistent trend: SGMD improves overall score and motion quality, with small trade-offs in visual quality and text alignment.Source: §4.2 · pp. 7–8
  • The authors interpret reverse-KL-style matching as more conservative, potentially preserving finer details while suppressing motion, whereas Fisher-style objectives can encourage stronger dynamics at a possible cost to overall VBench score.This is offered as an intuition for the dynamics-quality trade-off, not as a separate measured result.Source: §4.2 · p. 8

4.3. Analysis

  • Compared with DMD2, teacher stop-gradient Fisher improves motion dynamics but lowers VBench quality and semantic scores; SGMD’s NR/RC potentials reduce Fake-R from 5 to 1 while improving those scores and largely preserving dynamics.The ablation attributes complementary effects to the Fisher objective and SGMD’s dual potentials.Source: §4.3 · p. 8
  • λ = 0.1 gives the highest VBench total score, while λ = 0.2 gives the best DynDeg but a lower total score and mild blur; λ ≥ 0.5 often impedes convergence and leaves blurry videos with poor details.λ = 0.05 preserves strong dynamics but tends to incur larger tracking lag and a lower total score.Source: §4.3 · p. 8
  • Reducing fake-score updates per iteration from 5 to 1 yields an approximate 3× training speedup over the DMD2-style baseline under the reported settings.The authors provide the detailed counting and wall-clock estimate in Appendix C.Source: §4.3 · p. 8

5. Related Works

SGMD builds on DMD and differs from SIM and SiD in its fake-score analysis and tunable dual potentials. The paper positions its focus on video fake-score tracking as complementary to image-side work.

Related works

  • SGMD analyzes the net one-iteration update direction from a fake-score perspective, identifying bias induced by SIM-style coupling.The related works include DMD (Yin et al., 2024b), DMD2 (Yin et al., 2024a), Phased-DMD (Fan et al., 2025), SIM (Luo et al., 2024), and SiD (Zhou et al., 2024).Source: §5 · p. 8
  • Unlike SIM’s fixed relative weighting and SiD’s explicit reweighting coefficient, SGMD uses tunable dual potentials to decouple outer-loop correction from inner-loop contraction.The dual potentials operate under the teacher stop-gradient Fisher objective.Source: §5 · p. 8
  • Flash-DMD studies efficient DMD-style image distillation through timestep-aware training and joint reinforcement learning, which the paper describes as complementary to SGMD’s video tracking focus.Flash-DMD is cited as Chen et al. (2025).Source: §5 · p. 8

6. Conclusion

SGMD combines a fake-score perspective and teacher stop-gradient Fisher with NR/RC dual potentials for few-step video diffusion distillation. The paper reports stronger motion dynamics, temporal consistency, and lower training overhead under 4-step distillation.

Conclusion

  • SGMD directly improves the fake score toward the teacher under a stable teacher stop-gradient Fisher objective, while the generator tracks it to maintain score-consistency.A pair of dual potentials separates outer-loop correction (NR) from inner-loop contraction (RC).Source: §6 · p. 8
  • Under 4-step distillation, SGMD improves motion dynamics and temporal consistency while reducing the fake-score update ratio from 5 to 1 for an approximately 3× speedup under the reported settings.The update design yields a lightweight two-step update per iteration.Source: §6 · p. 8
  • A human study favors SGMD in motion quality and overall preference, while visual quality and text alignment remain comparable.Source: §6 · p. 8

Impact Statement

The statement notes potential benefits and risks associated with generative ML, and calls for responsible release practices and evaluative transparency.

Impact Statement

A. Additional Proofs

This appendix formalizes the fake-score perspective by constraining the learned fake score to match the generator-induced distribution’s score. Teacher alignment is then optimized subject to that score-consistency constraint.

A. Additional Proofs

  • The score-consistency set M contains generator and fake-score parameters for which the learned fake score equals the score of the generator-induced noisy-state distribution.It is defined as a constraint manifold.Source: §A.1 · p. 12
  • The abstract functional F(ψ) measures teacher alignment, and minimizing it over M represents improving the fake score toward the teacher while preserving score consistency.A teacher stop-gradient Fisher score-matching divergence is one example of such a functional.Source: §A.1 · p. 12
  • If M is locally feasible as assumed, a stationary point of the constrained objective has a compatible generator parameter, separating teacher-alignment updates from generator updates that restore score consistency.The generator stays on or near M so the fake score remains compatible with the current generator-induced distribution.Source: §A.1 · p. 12

A.2. Why λ should be moderate

This appendix analyzes how the tracking weight λ balances fake-score residual contraction against staleness as the generator changes. Its bounds favor a moderate λ rather than either extreme.

A.2. Why λ should be moderate

  • The coupled iteration updates the generator first, then contracts the fake-score residual toward a stop-gradient target from the previous generator state.The tracking residual is r_k = x0,k − xfake,k, and its recursion includes the generator change Δx0,k.Source: §A.2 · p. 12
  • Smaller λ loosens the asymptotic tracking-residual bound as A/λ, while larger λ increases the staleness bound as λB when A and B are positive.The proposition assumes 0 < ηψλ < 1.Source: §A.2 · p. 12
  • The proxy bound on coupled-iteration inaccuracy is minimized at a moderate λ.The bound combines tracking residual and staleness terms derived from the coupled update.Source: §A.2 · p. 13

A.3. SiD vs. SGMD: a fake-score perspective

The appendix contrasts SiD’s tunable reweighting of an implicit term with SGMD’s fake-score perspective, then gives pseudocode for SGMD’s stop-gradient updates. The pseudocode uses separate generator and fake-score optimization steps.

A.3. SiD vs. SGMD: a fake-score perspective

  • SiD reweights SIM’s implicit term with a tunable α, whereas SIM fixes the ratio between its explicit and implicit terms.At α = 1, the SiD expression reduces to SIM; other α values introduce an additional fake-score bias proportional to (1 − α)(xfake − xreal).Source: §A.3 · p. 13

B. PyTorch-style pseudocode

  • The pseudocode presents SGMD as two backward updates per iteration, applying L_fisher + λL_NR to the generator and λL_RC to the fake score.The generator and fake-score optimizers take their respective steps after these backward passes.Source: B. PyTorch-style pseudocode · pp. 13–14
  • The pseudocode samples a noise level and runs shared forward passes before the separate updates, with the residual using stop-gradient on x0.The generator forward pass may include text or image conditioning.Source: B. PyTorch-style pseudocode · pp. 13–14

C. Training-time breakdown

The training-time estimate compares SGMD with a DMD2-style baseline under a 4-step setting, while a separate 1D mixture-fitting example contrasts reverse-KL and Fisher matching. Table 5 maps the prompts to the four videos shown in Figure 3.

C. Training-time breakdown

  • 3.3× is the estimated SGMD training speedup over the DMD2-style baseline under the stated settings.The estimate is 77.5 seconds for SGMD versus 255 seconds for the baseline, with the baseline using one generator update and five fake-score updates per iteration.Source: C. Training-time breakdown · p. 14

D. 1D mixture fitting details: reverse-KL vs. Fisher

  • The 1D mixture-fitting example contrasts reverse-KL’s conservative avoidance of low-density target regions with Fisher’s smoother score-matching signal.The example uses an asymmetric two-component Gaussian target and a symmetric two-component model, optimized by numerical quadrature and gradient-based updates.Source: D. 1D mixture fitting details:… · pp. 14–15

E. Qualitative prompts

  • Table 5 lists the prompts corresponding in order to the four videos in Figure 3.The supplied passages identify the prompt-to-video mapping, not the videos’ comparative outcomes.Source: Table 5 · p. 15

Related papers

Loading 2605.30116v2…