PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Bandwidth Distribution Matching Distillation: On the Equivalence of Distribution Matching Distillation and Drifting Models

Jialin Zhu, Xing Liu, Feixiang He, He Wang

arXiv:2610.10989v1cs.LGcs.CV

TL;DR

DMD and Drifting Models had not been precisely connected, despite their similar optimization objectives. This paper establishes their exact finite-sample correspondence under DFSGM distillation and uses it to develop MBDMD. The authors report improved convergence and generation quality, while noting constraints on high-dimensional repulsion estimation.

  • Problem

    Prior connections between DFSGMs and Drifting Models existed, but a precise correspondence between Drifting Models and DMD had not been established.

    Source: §1 · p. 2

  • Method

    The paper derives an exact finite-sample equivalence between DMD and Drifting Models under DFSGM distillation, then transfers multi-bandwidth training to DMD as MBDMD.

    Source: §1 · p. 2, §6 · p. 8

  • Results

    Across two image-generation datasets, MBDMD improves convergence and generation quality; on ImageNet 64x64, it achieves FID 2.4697 at 410k steps.

    Source: §5 · p. 7, §6 · p. 8

  • Takeaways & Limitations

    The derivation reveals a shared theoretical basis for DMD and Drifting Models and provides an alternative interpretation of several DMD variants.

    Source: §6 · p. 8

    Direct high-dimensional kernel computation for the repulsion field faces the curse of dimensionality, and the available unbiased fake-score estimator has a narrow bandwidth range.

    Source: §3.1.2 · p. 4, §3.4 · p. 7

Abstract

from arXiv · show

Researchers are exploring effective one-step generative model continuously, and, Drifting Models (Deng et al., 2026), demonstrate great potential in one-step generation recently. There are works that reveal the connection between Diffusion & Flow Style Generative Models (DFSGMs) (Ho et al., 2020; Song et al., 2020a;b; Lipman et al., 2022; Liu et al., 2022) and Drifting Models (Li & Zhu, 2026; Lai et al., 2026; Turan et al., 2026). But no one has yet established a precise correspondence between the Drifting Model and the widely used distillation method- Distribution Matching Distillation (DMD/DMD2) (Yin et al., 2024b;a) to the best of our knowledge, even though their optimization objective formulas are virtually identical. In this paper, we prove that by converting the velocity-field / noise-field from the pre-trained DFSGMs into the attraction force field in Drifting Models and estimating the repulsion force field from the generative distribution, training the Drifting Model is naturally equivalent to the Distribution Matching Distillation. With this equivalent concept, we propose an improved method based on DMD from the Drifting Model's perspective- Multi-Bandwidth Distribution Matching Distillation (MBDMD).

1. Introduction

The introduction asks how Drifting Models relate to DMD and establishes an exact finite-sample correspondence under DFSGM distillation. It then uses that correspondence to introduce MBDMD, which the authors report improves DMD training efficiency and generation quality.

1. Introduction

  • The paper establishes an exact finite-sample correspondence between DMD and Drifting Models under DFSGM distillation.The derivation reformulates DMD training from the Drifting Model perspective.Source: §1 · p. 2
  • MBDMD applies the Drifting Models' multi-bandwidth concept to DMD training to improve training efficiency and generation quality.The authors also say the derivation offers theoretical explanations for several existing DMD variants.Source: §1 · p. 2
  • Earlier work connected DFSGMs and Drifting Models through score-based views or second-order velocity approximation, whereas this paper claims a precise finite-sample correspondence with DMD.The cited connections include Lai et al. (2026), Turan et al. (2026), and Li & Zhu (2026).Source: §1 · p. 2

2. Preliminary

The preliminary section outlines diffusion and flow-based generation, DMD distillation, and Drifting Models' attraction–repulsion formulation. These approaches differ in whether generation follows a learned trajectory or directly maps noise toward the data distribution.

2. Preliminary

  • Diffusion Models learn a reverse evolution across noise levels, using estimated scores to generate samples from a noise prior.The reverse process is supported by time-reversal theory of diffusion processes (Song et al., 2020b).Source: §2 · p. 2
  • Flow Matching constructs a deterministic trajectory from prior noise to data and learns a velocity field through a conditional trajectory target.The conditional target is the velocity conditioned on a target data point.Source: §2 · p. 2
  • DMD trains a deterministic generator to map noise toward the data distribution in few or one steps by minimizing KL divergence between generated and target distributions.Its parameter-update gradient corresponds to the difference between the distributions' score functions.Source: §2 · pp. 2–3
  • In DMD's implementation, a frozen pretrained diffusion teacher supplies the target-distribution attraction score, while a jointly trained dummy denoiser estimates the generated-distribution repulsion score.Their score difference guides the generator and bypasses multi-step sampling.Source: §2 · p. 3
  • Drifting Models represent generation as particles moving under attraction from the target distribution and repulsion from the generated distribution.The attraction and repulsion fields are kernel-weighted mean shifts; their linear superposition supplies the drift field used as a pseudo-label for single-step training.Source: §2 · p. 3

3. Equivalence of Distribution Matching Distillation and Drifting Models

The section establishes a finite-sample correspondence between DMD and Drifting Model training, then uses the kernel-bandwidth view to interpret DMD design choices and fake-score issues.

3. Equivalence of Distribution Matching Distillation and Drifting Models

  • Finite-sample analysis establishes an accurate transformation between Drifting Model training and prior DFSGM formulations that relied on infinite-value approximation.The derivation is presented as a theoretical bridge for DMD-related work.Source: §3 · p. 3

3.1.1. VELOCITY TO ATTRACTION FORCE

  • With the conditional-distribution exponential term as kernel, the attraction field is the pretrained Flow Matching velocity multiplied by remaining time (1 − t).The section derives this transfer from the conditional expectation and score relationship.Source: §3.1.1 · p. 4

3.1.2. REPULSION FORCE AND FULL DRIFTING FIELD

  • Directly computing repulsion in the high-dimensional velocity space encounters the dimensionality curse, so a Fake Flow Matching Model estimates the student distribution’s repulsion field.The fake model is trained on the student Drifting Model’s generative distribution.Source: §3.1.2 · p. 4

3.2. Connection with DMD

  • Finite-sample comparison identifies DMD as Drifting Model training with a specific kernel, where real and fake scores play attraction and repulsion roles.The section notes noise injection and score scaling as differences between the formulations.Source: §3.2 · p. 5

3.3. Unified View deduction

  • The unified formulation extends the attraction-field derivation across Flow Matching and diffusion VP and VE schedules using the conditional-distribution exponential term as kernel.The paper states that the framework can incorporate models whose trajectories interpolate between Gaussian noise and real data.Source: §3.3 · pp. 5–6

3.4. Insight and Connection with DMD-related Studies

  • DMD’s time parameter can be read as kernel bandwidth: larger bandwidth captures coarse information from more samples, while smaller bandwidth captures finer detail.The paper uses this view to interpret prior time-scheduling and multi-expert strategies.Source: §3.4 · pp. 6–7
  • The repulsion-field view identifies fake-score distribution alignment as a DMD issue, but direct kernel computation remains intractable in high dimensions.The reported feasibility demonstration uses low-dimensional 2D checkerboard data.Source: §3.4 · p. 7

4. Multi-Bandwidth Distribution Matching Distillation

MBDMD adapts the Drifting Model’s multi-bandwidth design to DMD, using bandwidth variation to cover receptive fields from global structure to sample detail.

4. Multi-Bandwidth Distribution Matching Distillation

  • Large bandwidths support learning global structure, while bandwidths approaching zero focus learning on specific sample details.At the zero-bandwidth limit, the target becomes discrete delta spikes on the manifold.Source: §4 · p. 7
  • MBDMD is motivated by the vanilla Drifting Model’s use of three bandwidths to balance global-structure learning and sample-detail learning.The Uni kernel’s bandwidth σ_t typically decreases as t increases.Source: §4 · p. 7
  • MBDMD divides the DMD timestep range into regions and randomly samples a bandwidth from each region during student training.This provides varied bandwidths at each training step and covers most of the bandwidth range.Source: §4 · p. 7

5. Experiments and Results

MBDMD improves generation quality and training efficiency over vanilla DMD in the reported experiments. Results cover ImageNet 64×64, bandwidth and sampling ablations, GAN-loss training, and SDXL distillation.

Experiments and results

  • MBDMD improves result quality and training efficiency in the reported experiments.On ImageNet 64×64 the main experiments use pure DMD, without GAN loss, to isolate GAN loss’s influence.Source: §5 · p. 7

Training settings

  • The method divides the bandwidth range into regions and samples a bandwidth from each region during student training.The base setting uses three bandwidths, with their values depending on the pretrained teacher; the authors align the sampling distribution with the teacher’s inference sampler.Source: §5 · p. 7

ImageNet 64×64 experiments

  • At 328k optimization steps, MBDMD achieves better FID than vanilla DMD’s 410k-step result on ImageNet 64×64.At 410k steps, MBDMD’s FID is 2.4697; each step has GPU power consumption equivalent to 1.15 vanilla DMD steps.Source: §5 · p. 7, Table 2 · p. 8

Bandwidth and sampling ablations

  • Across most iso-flop segments, K = 3 is a better choice than K = 5 or K = 7, so the authors use K = 3 as the basic setting.With K = 3, i.i.d. sampling improves over vanilla DMD but remains below MBDMD; K = 5’s improvement is negligible, and its best FID is below K = 3 while its iso-flop best FID loses by around 0.05.Source: §5 · p. 8, Table 3 · p. 8

GAN-loss experiments

  • On ImageNet 64×64, adding GAN loss does not significantly improve MBDMD’s results.The authors speculate that GAN loss mitigates DMD loss, citing the DMD2 ablation where GAN loss alone without distribution matching yielded better FID.Source: §5 · p. 8, Table 5 · p. 8

SDXL experiments

  • At 8,000 steps, MBDMD outperforms vanilla DMD2’s 24,000-step FID on SDXL, with a per-step cost 1.03 times vanilla.The reported improvement includes overhead introduced by GAN loss.Source: §5 · p. 8, Table 6 · p. 8

6. Conclusion

The paper establishes an exact finite-sample correspondence between DMD and Drifting Models under DFSGM distillation. Its derivation motivates MBDMD, which improves convergence and generation quality on two image-generation datasets.

6. Conclusion

  • The paper establishes an exact finite-sample correspondence between DMD and Drifting Models under DFSGM distillation.The correspondence expresses DMD’s distribution-matching computation in an equivalent attraction–repulsion formulation.Source: §6 · p. 8
  • The derivation enables MBDMD, which improves convergence and generation quality on two image-generation datasets.The method transfers multi-bandwidth training to DMD.Source: §6 · p. 8
  • The theoretical framework also provides alternative interpretations of several existing DMD variants.Source: §6 · p. 8

A.1. Kernel Attraction to Probability-based form

The appendix presents transformations connecting kernel-based attraction to probability-based and expectation-based forms, then relates the attraction expression to the score function.

A.1. Kernel Attraction to Probability-based form

  • The appendix changes the kernel function used to calculate the attraction force field into a probability-based computational form.The resulting expression is identified as equation 10.Source: §A.1 · p. 11

A.2. Probability-based Attraction form to Expectation-based form

  • The next step invokes Bayesian theory to move from the probability-based attraction form toward an expectation-based form.Source: §A.2 · p. 11

A.3. Expectation-based Attraction form to score function form

  • For Flow Matching, the appendix introduces the conditional distribution of x_t.Source: §A.3 · p. 11
  • The appendix identifies the gradient of the log-probability density of a Gaussian distribution as the conditional score.Source: §A.3 · p. 12
  • Using differentiation, Leibniz’s integral rule, and Bayes’ theorem, the derivation expresses a conditional expectation of the log-likelihood gradient.Source: §A.3 · p. 12
  • The derivation uses linearity of expectation to obtain an expression for s_t(x_t) in terms of x_t and E[x_1 | x_t].Source: §A.3 · p. 12
  • Given x_t = x, the derivation rearranges an equation to solve for E[x_1 | x_t = x].Source: §A.3 · p. 12

A.4. Score function to Velocity Field

This appendix passage derives a velocity-field expression for Flow Matching from the linear forward process and relates it to the score function.

A.4. Score function to Velocity Field

  • The derivation begins with the trajectory equation for Flow Matching’s linear forward process.Source: §A.4 · p. 12
  • It expresses the base noise x_0 in terms of x_t and x_1 using the reverse relation.Source: §A.4 · p. 12
  • The derivation substitutes x_0 into the marginal velocity field and uses the score-function relation from the previous subsection to obtain the velocity field.Source: §A.4 · pp. 12–13

B. Using Kernel calculation to replace Fake Score Model

A 2D checkerboard experiment compares kernel-based forces with Flow Matching velocity and DMD formulations. All three approaches generally yield excellent one-step results, though the proposed approach is not practical in high dimensions due to the curse of dimensionality.

B. Using Kernel calculation to replace Fake Score Model

Loading 2610.10989v1…