PaperlayerView exact revision on arXiv

Source-linked AI summary

Consistent Distribution Matching for Data-Free Diffusion Distillation

Yuxiang Fu, Qi Yan, Zike Wu, Yongxing Zhang, Purang Abolmaesumi, Lele Wang, Renjie Liao

arXiv:2610.09221v1cs.LGcs.AIcs.CV

TL;DR

Diffusion and flow models are slow at inference, and existing distillation can rely on external data, teacher rollouts, or auxiliary score models. CDMD uses one student for generation and score estimation, training from student-induced states without data or teacher simulation. It reports state-of-the-art data-free distillation results on ImageNet 256×256 and proves a distribution-matching result under assumptions.

  • Problem

    Diffusion and flow models require costly iterative inference, while distillation pipelines can depend on external data, teacher rollouts, or auxiliary networks.

    Source: §1 · pp. 1–2

  • Method

    CDMD trains one student to provide both sample generation and score estimation, using student-induced states without external data or teacher rollouts.

    Source: §1 · pp. 2–3, §3.1 · p. 5

  • Results

    1-NFE FID 2.04 and 4-NFE FID 1.37 on ImageNet 256×256 achieve state-of-the-art performance among data-free distillation methods.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Under regularity assumptions, minimizing the CDMD objective implies convergence of student flow-map pushforwards to teacher marginals in averaged Wasserstein distance.

    Source: §1 · p. 3

    The appendix notes that private training data and inaccessible teacher weights prevent initializing the student from the teacher weights.

    Source: H LIMITATIONS AND BROADER IMPACT · p. 33

Abstract

from arXiv · show

Flow and diffusion models suffer from slow inference due to computationally expensive numerical integration. Distillation provides a promising way for a student model to learn from a teacher's dynamics, enabling one-step or few-step generation. However, existing methods often depend on curated distillation datasets, costly teacher rollouts, or auxiliary proxy networks, which complicate model training and scaling. In this work, we propose Consistent Distribution Matching, a simulation-free and data-free distillation method for accelerating diffusion and flow models while preserving strong generative capacity. Our key insight is to unify sample generation and score estimation with one student network. Thus, our framework uses only two models, a frozen teacher and a trainable student, and optimizes one objective. We prove that minimizing our objective indicates Wasserstein convergence of the student flow-map pushforwards to the teacher marginals. On ImageNet 256$\times$256, our method attains an FID of 2.04 with a single function evaluation (1-NFE) and a 4-NFE FID of 1.37 within 40 epochs of training, surpassing the state-of-the-art distillation baselines without data. Our code code and model are available at https://consistentdmd.github.io/.

1 INTRODUCTION

Diffusion and flow models generate high-quality samples but require costly iterative inference, while distillation can add data, teacher rollouts, or auxiliary networks. CDMD unifies sample generation and score estimation in one student, and its objective connects local consistency to matching teacher distributions.

1 INTRODUCTION

  • Hundreds of neural function evaluations are often needed to numerically solve the generative dynamics for a single sample.This iterative denoising procedure is the primary inference-cost bottleneck described for diffusion and flow models.Source: §1 · p. 1
  • Existing distillation pipelines may require teacher backward simulation, auxiliary fake-score networks, adversarial objectives, or multi-stage training.These components add memory and optimization burden and can hinder scaling.Source: §1 · p. 2
  • External distillation data can produce supervision states that differ from the teacher’s probability path, while teacher rollouts to obtain matching states add computational burden.The mismatch can be more pronounced after teacher fine-tuning or with classifier-free or autoguidance, and data-anchored paths may be infeasible for proprietary-data teachers.Source: §1 · p. 2
  • CDMD distills diffusion and flow models using a frozen teacher and trainable student, without external data or teacher simulation.The student unifies generation and score estimation; intermediate states come from student predictions and analytic forward noising.Source: §1 · pp. 2–3
  • Under regularity assumptions, minimizing the CDMD objective narrows the student–teacher distribution gap, with the teacher’s average velocity as the objective’s unique global minimizer.The analysis does not require supervision states to come from the teacher probability path.Source: §1 · p. 3
  • 1-NFE FID 2.04 and 4-NFE FID 1.37 on ImageNet 256×256 achieve state-of-the-art performance among data-free distillation methods.In text-to-image comparison when training data is inaccessible, CDMD leads most preference and alignment metrics and FID.Source: §1 · p. 3

2 BACKGROUND

Flow Matching describes a continuous path from data to a prior and learns a velocity field to represent its dynamics. Score distillation instead matches diffused student and teacher distributions, but common implementations add an auxiliary fake-score model and coupled training.

2 BACKGROUND

  • Flow Matching defines an affine path between data and prior samples using prescribed schedules, with a linear path yielding constant conditional velocity.Conditioning over data–prior pairs induces a marginal probability path and target velocity field.Source: §2 · p. 3
  • Because the marginal velocity field is intractable, Flow Matching learns a neural velocity field through a conditional objective that recovers the true marginal velocity at optimum.The learned field is used to solve the generative ODE at inference.Source: §2 · pp. 3–4
  • Distribution Matching Distillation applies reverse KL to diffused student and teacher distributions, using the teacher for the real score and an auxiliary network for the fake score.The fake-score model is trained on student-generated samples, creating a coupled optimization that consumes substantial compute and is discarded at inference.Source: §2 · p. 4

3 CONSISTENT DISTRIBUTION MATCHING DISTILLATION

CDMD constructs supervision states from the student and analytic forward noising, then uses one flow-map student to generate samples and provide a velocity surrogate. Its consistency objective links student-path training to teacher distribution matching, while the JVP correction’s tangent choice affects its stochasticity and bias.

3.1 MINIMAL DESIGN ON DATA-FREE DISTILLATION

  • A flow-map model represents average velocity between arbitrary trajectory times, from which the student derives both one-step generation and an instantaneous-velocity surrogate.The same student parameterization replaces the auxiliary proxy model.Source: §3.1 · pp. 5–6
  • CDMD forms each intermediate state with one student evaluation followed by analytic forward noising, avoiding external data and teacher rollouts.The balancing time s ranges from the student-path endpoint at s=t to a noised one-step student sample at s=0.Source: §3.1 · pp. 5–6
  • Vanishing population CDMD loss implies convergence of student flow-map pushforwards to teacher marginals in averaged squared Wasserstein-2 distance under the theorem’s assumptions.When an admissible student can represent the teacher average velocity, every global minimizer recovers it almost surely under the training distribution.Source: §3.1 · p. 6
  • Applying the learned flow map over consecutive intervals provides few-step sampling and inference-time scaling without retraining.The full interval gives one-step generation.Source: §3.1 · pp. 6–7
  • CDMD combines data-free and simulation-free distillation with two models and one optimizer, avoiding adversarial training and the external real data required by GAN objectives.The comparison is reported in Table 4.Source: §3.1 · p. 7

3.2 VARIANCE OF THE JVP CORRECTION

  • The cross-centered projected JVP second moment captures both correction fluctuations on student-generated states and the distributional mean shift from the target path.Intrinsic JVP variance is its special case when both distributions in the moment are the student path.Source: §3.2 · p. 7
  • Using the teacher velocity as the JVP tangent works best among three tested choices on the synthetic dataset.The alternatives are the student limiting velocity and the conditional velocity field; the tangent choice affects correction stochasticity and bias under data-free supervision.Source: §3.2 · p. 7

4 EXPERIMENTS

Experiments test CDMD on a synthetic 2D spiral, class-conditional ImageNet 256×256, and SANA-1.6B text-to-image generation, then examine guidance, training-state distribution, and sampling steps.

Synthetic benchmark

  • CDMD and FreeFlow reduce endpoint W2 distance at the same rate, but only CDMD continues reducing teacher average-velocity MSE; tied marginal scores therefore do not imply matched transport.The figure also compares tangent choices for JVP correction; the teacher direction gives the most stable behavior, including on student-bootstrapped states.Source: §4 · p. 8, Figure 5 · p. 8

ImageNet 256×256

  • 1-NFE FID is 3.47 with SiT-XL/2, 2.93 with SiT-XL/2+VA-REPA, and 2.04 with LightningDiT-XL/2; CDMD is best across all three teachers.Without retraining, the VA-REPA student reaches FID 2.60 at 2 NFEs; LightningDiT reaches 1.62 at 2 NFEs and 1.37 at 4 NFEs, close to the teacher’s 1.35.Source: §4 · p. 8

Inference-time scaling

  • With guidance w = 1.5, FID improves as sampling steps increase, while with w = 2.0 it worsens; CDMD’s multi-step benefit depends on guidance and sampler.FreeFlow is anchored at noise and cannot run over sub-intervals, whereas CDMD supports arbitrary intervals.Source: §4 · p. 9

Training-state construction

  • Intermediate training-state mixing gives the lowest FID at m = 0.5 and highest IS at m = 0.875; 10% t = r improves 1-NFE FID, while 50% yields more stable few-step performance.The table reports 1-NFE results at each mixing ratio’s best checkpoint after 50K iterations.Source: Table 2 · p. 8, §4 · p. 9

Text-to-image generation

  • At 4 NFEs, CDMD leads learned methods on Aesthetic Score, PickScore, ImageReward, and FID (27.83 vs MFD’s 28.77), but trails on HPSv2 and CLIPScore.It beats the 4-NFE teacher on all six metrics and the 20-NFE teacher on Aesthetic Score, ImageReward, and FID; the results do not establish statistical superiority.Source: §4 · p. 9, Table 3 · p. 9

5 RELATED WORK

Related work distills trajectories through direct mappings, repeated step halving, shared clean-sample consistency, and flow maps between arbitrary trajectory times. CDMD avoids teacher integration and supports arbitrary input states and times.

Trajectory Distillation

6 CONCLUSION

CDMD distills pretrained diffusion and flow teachers into one-step or few-step flow-map students using a data-free, simulation-free two-model pipeline. Its theory and experiments support distribution matching and strong generation quality within the reported settings.

6 CONCLUSION

  • CDMD removes external distillation data, teacher rollouts, backward simulation, and auxiliary proxy networks by training a student flow map on a student-induced distribution with one objective.Source: §6 · p. 10
  • Under regularity assumptions, minimizing the CDMD objective yields time-averaged Wasserstein convergence of student sampling distributions to teacher distributions, and every global minimizer recovers teacher average velocity on sampled states.Source: §6 · p. 10
  • CDMD achieves state-of-the-art ImageNet 256×256 quality among data-free distillation methods and performs strongly in the reported text-to-image setting; future work includes JVP stability, initialization without model weights, and larger-scale tasks with data scarcity.Source: §6 · p. 10

A PROOFS

The proofs connect the CDMD residual to Wasserstein convergence of student pushforwards under regularity and coverage assumptions, and show that zero residual identifies the teacher average velocity. The accompanying sections compare target-variance diagnostics and distillation methods, highlighting CDMD’s combination of data-free, simulation-free training with two models and one optimizer.

A PROOFS

  • Under the stated coverage and regularity assumptions, vanishing CDMD loss yields Wasserstein convergence of student flow-map pushforwards to teacher marginals.The proof couples student and teacher outputs from the same prior sample and shows the resulting transport error vanishes.Source: A PROOFS · pp. 17–18
  • Zero residual and the characteristic boundary condition identify the teacher average-velocity field along teacher trajectories; under coverage, every global minimizer realizes it.The theorem also requires an admissible student capable of representing the teacher field.Source: A PROOFS · pp. 17, 19

B MORE ON THE VARIANCE OF THE TARGET

  • The variance analysis defines projected variance as a stability measure for the detached correction and projected error as deviation from the teacher-consistent tangent.Its cross-centered projected JVP quantity also captures both student-state stochasticity and mismatch between student-generated and target-path mean corrections.Source: B MORE ON THE VARIANCE OF THE TARGET · p. 20

C RELATIONSHIP WITH PRIOR METHODS

  • CDMD is the only compared method combining data-free and simulation-free training with only the teacher, student, and one optimizer.The comparison also notes that several methods train a third auxiliary network, while BOOT simulates trajectories and flow-map methods use real data.Source: C RELATIONSHIP WITH PRIOR METHODS · p. 21, Table 4 · p. 21

C.1 FREEFLOW

  • FreeFlow avoids teacher-data mismatch by sampling from the prior, but its prior-anchored training confines it to transitions initialized at t=1 and limits natural multi-step scaling.The comparison also notes that FreeFlow uses a proxy flow-matching model and pseudo-gradients through student-generated trajectories; CDMD instead supervises arbitrary local intervals without explicit student rollouts.Source: §C.1 · pp. 21–22

C.2 FLOW MAPS

  • A backward flow map sends a state at time t to its earlier-time state at r by following the velocity-field ODE, with sampling performed by numerical integration.The related discussion describes average-velocity parameterization and a corrected prediction matched to instantaneous velocity in an objective analogous to standard conditional flow matching.Source: §C.2 · pp. 22–23

C.3 EQUIVALENCE WITH ORIGINAL DMD UPDATES

The section shows that CDMD’s velocity-form update is equivalent to the original DMD update up to a time-dependent weight. It also details data-free training-state construction and class-conditional ImageNet implementation choices.

C.3 EQUIVALENCE WITH ORIGINAL DMD UPDATES

  • The velocity-form update optimizes the same distribution-matching objective as DMD, with score and velocity differences related by a time-dependent scalar weight.For the linear schedule, the scalar factor is absorbed into the time-dependent weighting function.Source: §C.3 · pp. 23–24

D IMPLEMENTATION DETAILS

  • CDMD constructs training states either by jumping from the prior to an intermediate time or by re-noising a student-generated clean sample, without teacher rollouts.Each construction costs one student evaluation; the resulting state is detached before the teacher supplies velocity supervision.Source: §D.1 · p. 25, Figure 7 · p. 25

D.1 CLASS-CONDITIONAL IMAGENET

  • The ImageNet student adds r and guidance-scale embeddings through adaLN-Zero, while retaining the original timestep conditioning throughout the transformer blocks.The time-gap signal is introduced only in later blocks; block 10 is used for the full LightningDiT-XL/2 run.Source: §D.1 · pp. 25–26
  • Evaluation uses a common PyTorch pipeline and sweeps guidance strength from 1 to 2 for the SiT-XL/2+VA-REPA teacher student.Representative data-based and other reference methods use real training data and different training budgets, so they are not direct data-free baselines.Source: §D.1 · p. 26
  • The SiT-XL/2 teacher setting reports only t and r sampling distributions; other settings and evaluation results await stable training outcomes.Source: §D.1 · p. 27

D.2 SYNTHETIC 2D BENCHMARKS

The analytic 2D benchmarks assess both terminal distributions and transport behavior, while the text-to-image adaptation anchors training states to a coarse teacher trajectory. The benchmarks show that matching endpoint marginals alone does not capture transport quality.

D.2 SYNTHETIC 2D BENCHMARKS

  • CDMD reduces spiral average-velocity error from 0.317 to 0.070 and coupled endpoint error from 0.0316 to 0.0040, while both methods reach W2 = 0.190.The comparison is against FreeFlow and uses the converged-state evaluation.Source: §D.2 · p. 27
  • Across checkerboard and spiral benchmarks, the teacher-tangent JVP has the smallest, most stable correction statistics; the global one-step displacement has the greatest variability.The projected variance and cross-centered second moment show nearly identical trends and rankings.Source: §D.2 · p. 27, Figure 9 · p. 28

D.3 TEXT-TO-IMAGE DATASET

  • For SANA-1.6B text-to-image, intermediate states are re-noised from the nearest cleaner cached teacher-anchored state, keeping them near the four-step inference trajectory.The reported training cost is approximately 7 GPU hours and 18 GB peak memory; it is higher than CD but lower than the reported costs of Diff-Instruct, DMD2, and MFD.Source: §D.3 · p. 28, Figure 10 · p. 29, Table 7 · p. 29

E ABLATION ON ENDPOINT CONDITIONING

The ablation studies where to inject the endpoint gap t − r in the transformer. Delaying that signal preserves early blocks for teacher conditioning, but injecting it too late also hurts performance.

E ABLATION ON ENDPOINT CONDITIONING

  • Conditioning every block on t − r gives FID 17.60, while delaying the signal by five blocks lowers FID to 2.80; delaying it to D = 25 raises FID to 3.84.All depths from 5 to 20 reach FID at most 2.80 and IS above 273.Source: E ABLATION ON ENDPOINT CONDITIONING · p. 29
  • Early blocks can reproduce teacher features for instantaneous velocity before later blocks use the endpoint signal to form interval-specific average velocities.With D = 0, every block must process both time variables from the start, spending capacity on the endpoint before reproducing the teacher.Source: E ABLATION ON ENDPOINT CONDITIONING · p. 29
  • With D = 8, the 28-block student reaches FID 2.21 and IS 284.46 under a 25K-iteration budget.The table varies D, the number of initial blocks that receive only teacher conditioning.Source: E ABLATION ON ENDPOINT CONDITIONING · p. 29, Table 8 · p. 30

F.1 SAMPLER COMPARISON WITH THE SIT-XL/2+VA-REPA TEACHER

Inference-time scaling depends on the sampler, guidance, and training schedule. The LightningDiT student improves with additional Euler steps, while the SiT+VA-REPA student favors stable low-step DDIM sampling; guidance settings also affect FID.

F.2 INFERENCE-TIME SCALING WITH THE LIGHTNINGDIT-XL/2 TEACHER

  • Using t = r for 50% of training pairs preserves multi-step behavior across teachers; the SiT+VA-REPA student trained with 10% shows greater degradation as steps increase.For the LightningDiT student, the t = r objective directly regresses student velocity to teacher instantaneous velocity.Source: §F.2 · p. 31

G CDMD WITH GUIDANCE

  • The guided objective matches guided teacher dynamics, and a training-time student guidance signal can absorb part of the conditional guidance before inference.Standard classifier-free guidance evaluates conditional and unconditional branches at each solver step, doubling NFE.Source: G CDMD WITH GUIDANCE · pp. 31–32

F.1 SAMPLER COMPARISON WITH THE SIT-XL/2+VA-REPA TEACHER

  • For the SiT-XL/2+VA-REPA student, DDIM is most stable from one to four steps, while Heun spikes at two steps and Euler becomes less favorable as steps increase.The sampler comparison covers guidance scales w = 1.5 and w = 2.0.Source: §F.1 · p. 30, Figure 11 · p. 32

F.2 INFERENCE-TIME SCALING WITH THE LIGHTNINGDIT-XL/2 TEACHER

  • With the LightningDiT-XL/2 teacher, Euler FID falls from 2.04 at 1 step to 1.62 at 2 steps and 1.37 at 4 steps, then stays close to the teacher through 64 steps.Euler reaches its minimum FID at 8 steps; IS rises from about 294 at 1 step to about 320 at 16 steps and then saturates.Source: §F.2 · p. 31, Figure 12 · p. 32

G CDMD WITH GUIDANCE

  • For SiT-XL/2+VA-REPA, all three EMA settings reach their lowest FID at w ∈ [1.8, 1.9], and the no-EMA student reaches about 2.93 FID-50K.The best w depends on the teacher checkpoint and EMA rate; with the SiT-XL/2 teacher, the no-EMA minimum is about 3.47 at w ≈ 2.0.Source: G CDMD WITH GUIDANCE · p. 32, Figure 14 · p. 33

H LIMITATIONS AND BROADER IMPACT

The limitations discussion identifies unresolved analysis of the total derivative term and training challenges when teacher weights are inaccessible. The appendices also list prompts and prompt examples for generated samples.

H LIMITATIONS AND BROADER IMPACT

  • The total derivative term lacks meaningful constraints and rigorous analysis, while its norm is large and fluctuates during training.The authors suggest regularizing it using a well-trained reference velocity field, but describe this as a plausible direction.Source: H LIMITATIONS AND BROADER IMPACT · p. 33
  • Private data and inaccessible teacher weights prevent teacher-weight initialization, making training more challenging and less stable.The authors also speculate that the choice of how to bootstrap x_t can substantially affect learning dynamics and distillation effectiveness across tasks.Source: H LIMITATIONS AND BROADER IMPACT · p. 33

I IMAGE PROMPTS

2. Portrait: portrait photo of a girl, photograph, highly detailed face, depth of field

  • One listed prompt describes a neon bear riding a skateboard beside a sign reading “CDMD.”Source: §2 · p. 34
  • Another listed prompt is for an Assassin’s Creed IV Black Flag concept-art warrior.Source: §2 · p. 34

5. Shiba Inu: a shiba inu wearing a beret and black turtleneck

  • The listed prompts include a llama wearing sunglasses on a spaceship and a pirate ship in a cosmic nebula.Source: §5 · p. 34

8. Astronaut: Astronaut in a jungle, cold color palette, muted colors, detailed, 4k

  • Another listed prompt describes an autumn forest scene with a river and foliage.Source: §8 · p. 34

J QUALITATIVE SAMPLES

The qualitative figures present 1-NFE samples on ImageNet 256×256 and 512×512, with class indices sampled randomly. Additional figures show uncurated CDMD samples for several specified ImageNet classes.

J QUALITATIVE SAMPLES

Loading 2610.09221v1…