PaperlayerView exact revision on arXiv

Source-linked AI summary

Distribution Matching Distillation without Fake Score Network

Youngjoong Kim, Deokyeong Lee, Jaesik Park

arXiv:2605.19256v1cs.CV

TL;DR

DMD provides distribution-level correction but relies on an auxiliary fake-score tracker with memory and update overhead. FSF-DMD replaces that tracker with a flow-map generator-induced pseudo-velocity surrogate, and experiments report improved flow-map baselines and lower FID than listed DMD2 comparisons under flow-map initialization.

  • Problem

    DMD’s auxiliary fake-score tracker adds memory overhead and can require additional update heuristics, motivating the question of whether it is necessary for flow-map generators [5] [29].

    Source: §1 · pp. 1–2

  • Method

    FSF-DMD replaces DMD’s auxiliary fake-score estimator with an endpoint pseudo-velocity surrogate from the flow-map generator, extending the formulation to backward simulation and a self-teacher variant.

    Source: §1 · p. 2

  • Results

    On ImageNet-1K 256×256 few-step generation, FSF-DMD improves the consistency distillation baseline and reaches lower FID than listed DMD2 comparisons with flow-map initialization, while remaining effective under flow-matching initialization and from scratch.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    In the studied settings, a flow-map generator can supply a useful fake-score correction for DMD without an explicit fake-score network.

    Source: §5 · p. 9

    FSF-DMD becomes unstable when LCD is omitted, suggesting that a meaningful endpoint pseudo-velocity depends on the generator’s flow-map structure.

    Source: §3.3 · p. 6

Abstract

from arXiv · show

Distribution Matching Distillation (DMD) provides an effective distribution-level correction for few-step generation, while relying on an auxiliary fake-score network to track the evolving generative distribution. Recent work combines DMD-style objectives with flow-map generators to exploit both forward-divergence training and reverse-divergence correction. The fake-score estimator remains an additional component with memory and update overhead. In this work, we study whether this explicit tracker can be avoided when the generator itself has a flow-map structure. We propose Fake-Score-network-Free DMD (FSF-DMD), a DMD formulation for flow-map generators that replaces the auxiliary fake-score estimator with a generator-induced pseudo-velocity surrogate. The key observation is that the endpoint pseudo-velocity of a flow-map generator provides a tractable proxy for fake-velocity estimation, allowing the generator itself to supply the reverse-divergence signal. Building on this observation, we derive a practical objective, extend it with flow-map-consistent backward simulation, and introduce a self-teacher variant for training from scratch. In our ImageNet-1K $256 \times 256$ experiments, FSF-DMD improves flow-map baselines, reaches lower FID than the listed DMD2 comparisons in the flow-map-initialized setting, and remains effective under flow-matching initialization and training from scratch.

1 Introduction

FSF-DMD replaces DMD’s auxiliary fake-score tracker with a surrogate supplied by the flow-map generator. On ImageNet-1K 256×256 few-step generation, it improves flow-map baselines and beats the listed DMD2 comparisons in FID under flow-map initialization.

1 Introduction

  • Flow-map training uses forward divergence and tends toward mode coverage, while DMD uses reverse divergence and can provide a mode-seeking correction [31].Recent methods combine both objective types [2] [5] [31].Source: §1 · p. 1
  • DMD’s auxiliary tracker adds memory overhead and can require extra update heuristics for stability [5] [29].The introduction asks whether an explicit fake-score network is necessary for DMD-style objectives with flow-map generators.Source: §1 · pp. 1–2
  • FSF-DMD uses the endpoint pseudo-velocity of a flow-map generator as a tractable surrogate for fake-velocity estimation, supplying the reverse-divergence signal without a separate fake-side network.The formulation also extends to flow-map-consistent backward simulation and a self-teacher variant for training from scratch.Source: §1 · p. 2
  • Because the surrogate comes from the current generator, it tracks the evolving generator without a separately trained tracker and leaves generator capacity focused on generation.The semigroup property can support reasonable performance even without backward simulation for distribution matching.Source: §1 · p. 2
  • On ImageNet-1K 256×256 few-step generation, FSF-DMD improves the consistency distillation baseline and reaches lower FID than the listed DMD2 comparisons with flow-map initialization.It also remains effective with flow-matching initialization and in the studied training-from-scratch setting.Source: §1 · p. 2

2 Related Work

Related work contrasts efficient flow-map generation with DMD’s distribution-level correction, and describes approaches combining the two. These approaches differ in how they estimate the fake-side signal and in their associated trade-offs.

Flow-map models

  • Flow-map models generalize endpoint consistency to mappings between arbitrary timesteps, while MeanFlow predicts the ODE integral as a flow map [1] [6] [7] [12].Their forward-divergence objectives encourage mode coverage but may spread density into low-likelihood regions.Source: §2 · p. 3

Distribution Matching Distillation (DMD)

  • DMD [29] [30] matches generated and data distributions by comparing denoising scores on noised samples, typically using auxiliary networks that increase memory pressure.It exhibits mode-seeking behavior and can reduce sample diversity relative to forward divergence [19].Source: §2 · p. 3

Combined objectives

  • Combined forward- and backward-divergence methods [2] [5] [31] aim to cover both mode-covering and mode-seeking behavior.The comparison in Figure 1 contrasts DMD2’s separate fake-score network with FSF-DMD’s generator-induced surrogate for flow-map generators.Source: §2 · p. 3, Figure 1 · p. 3

Self-contained approximations

  • TwinFlow [2] jointly trains the generator to estimate the fake score, potentially adding expressivity demands, while Moment Matching Distillation [22] avoids explicit tracking but incurs first-order truncation error.FSF-DMD instead uses an endpoint pseudo-velocity surrogate and Euler-based backward simulation for flow-map generators.Source: §2 · p. 3

3 Method

FSF-DMD replaces DMD’s auxiliary fake-score tracker with a flow-map generator’s endpoint pseudo-velocity. The method combines this distributional correction with flow-map consistency training, flow-map-consistent backward simulation, and a self-teacher variant for training from scratch.

3.1 Preliminary

  • Standard DMD tracks the changing fake score with an auxiliary network, while flow-map consistency distillation alone lacks this fake-side correction.The proposed parameterization uses average velocity between timepoints, with the instantaneous velocity recovered in the s → t limit.Source: Figure 2 · p. 4, §3.1 · pp. 4–5
  • The flow-map parameterization represents mappings between timepoints, and its average-velocity prediction provides the form later used for fake-velocity estimation.The flow-map consistency objective anchors the generator to that structure.Source: §3.1 · p. 4

3.2 Our Approach

  • Under PF-ODE coupling, the endpoint pseudo-velocity replaces the fake-velocity tracker as a tractable signal, not as a pointwise estimate under independent coupling.The surrogate is computed from the current generator without a separate network or training loop.Source: §3.2 · p. 5

3.3 Implementation

  • The implemented objective uses independent perturbations for teacher evaluation and combines the generator-induced surrogate with consistency distillation to preserve flow-map structure.The method reports instability when consistency distillation is omitted.Source: §3.3 · p. 6
  • Backward simulation is made consistent with the flow-map generator because the existing simulation does not fully reflect its generation process.Few-step training can otherwise mismatch inference, where later inputs come from previously generated samples.Source: §3.3 · p. 6
  • For training from scratch, the network’s instantaneous-velocity approximation substitutes for the teacher, and consistency training replaces consistency distillation.This supplies distributional correction without a teacher network or an additional fake-velocity objective.Source: §3.3 · p. 7

4 Experiments

On ImageNet-1K 256×256, experiments evaluate FSF-DMD with flow-map initialization, flow-matching initialization, and random initialization. Across these settings, results show improved comparisons, ablation effects, and effective training from scratch.

4 Experiments

  • The experiments use LightningDiT-B/1 on ImageNet-1K 256×256; scaling to larger architectures and broader settings, including text-to-image, is left for future work.The study covers flow-map initialization, flow-matching initialization, and training from random initialization.Source: §4 · p. 7, §4.4 · p. 9

4.1 Proof of Concept: Flow-Map-Initialized Comparison

  • FSF-DMD lowers consistency distillation from 6.06 to 3.85 FID and reaches its best FID within 4K steps, earlier than listed distillation variants.The flow-map-initialized comparison is on ImageNet-1K 256, with an 8-step teacher.Source: §4.1 · p. 7, Table 2 · p. 7
  • In the local setup, FSF-DMD uses 49GB VRAM and 0.31 sec/step, versus DMD2’s 60GB and 1.32 sec/step, respectively.With TTUR=1:1, DMD2 reaches 0.44 sec/step while still requiring 60GB VRAM.Source: §4.1 · p. 8

4.2 Ablation Study

  • With M=2 backward simulation, FSF-DMD improves; its flow-map simulation beats DMD2-style simulation, while M=1 shows relatively small degradation versus DMD2.The authors attribute the flow-map simulation gain to alignment with the generation process; its semigroup property supports approximating backward simulation with single-step generation.Source: §4.2 · p. 8, Table 3 · p. 8
  • Tuning changes the baseline’s 5.34 FID to 4.45, then random masking yields 3.85; masked consistency distillation reaches 4.98 FID.The tuning adjusts λ, the LFSF-DMD weight, and timestep shifting; the remaining gap supports a role for distributional correction.Source: §4.2 · p. 8

4.3 Relaxing to Flow-Matching-Initialization

  • With flow-matching initialization, FSF-DMD improves over the consistency baseline and listed comparisons, with higher precision and recall on par with listed methods.The comparison uses a pretrained LightningDiT-B/1 flow-matching model trained for 400K steps; the teacher uses 250-step Euler sampling.Source: §4.3 · p. 8, Table 4 · p. 8
  • Flow-map initialization gives better FSF-DMD results than flow-matching initialization, possibly because early samples stay closer to the teacher or because iMF uses more parameters and computation.The authors cite both factors as possible contributors to the difference.Source: §4.3 · p. 9

4.4 From-Scratch Training

  • From random initialization, FSF-DMD trains flow-map models from scratch; compared with other methods, it is faster and reaches a better result under the same budget, without an external teacher.Figure 3 plots FID10K over training steps on a log scale; Table 5 reports FID50K and IS for 400K-step ImageNet-1K 256 runs.Source: §4.4 · p. 9, Figure 3 · p. 9, Table 5 · p. 9

5 Conclusion

FSF-DMD replaces the auxiliary fake-score estimator with a generator-induced surrogate for flow-map generators. The conclusion reports effectiveness in the studied settings, and Figure 4 shows 2-step samples on ImageNet-1K 256 with flow-map initialization.

5 Conclusion

  • Figure 4 presents 2-step FSF-DMD samples on ImageNet-1K 256 with flow-map initialization.Source: Figure 4 · p. 9, §5 · p. 9
  • FSF-DMD replaces the auxiliary fake-score estimator with a generator-induced surrogate while retaining DMD’s teacher-fake distributional correction for flow-map generators.The paper frames flow-map structure as providing both forward-divergence training and a reverse-divergence signal.Source: §5 · p. 9
  • The objective extends to flow-map-consistent backward simulation and a self-teacher variant for training from scratch.Source: §5 · p. 9

A Theoretical analysis

The appendix sets up a stochastic interpolant and relates its marginal velocity and score through a shared posterior-mean term. It then describes MeanFlow’s flow-map realization and its consistency interpretation.

A.1 Score-Velocity Connection

  • The appendix assumes a stochastic interpolant with monotonically decreasing α_t and increasing σ_t, and uses α_t = 1 − t, σ_t = t, T = 1, and a standard Gaussian prior by default.Source: §A.1 · p. 12
  • For the stated interpolant, the conditional velocity is given and the marginal velocity is identified as a posterior mean.The passages state that marginal velocity and marginal score share a posterior-mean term, from which their connection is derived.Source: §A.1 · p. 12

A.2 MeanFlow as a Flow Map

  • A flow-map network fθ approximates the ground-truth map and, in the MeanFlow-style realization, uses Fθ as its pseudo-velocity network.Source: §A.2 · p. 12
  • MeanFlow can be interpreted as a consistency model and, further, as a flow-map model under the stated loss-weight relation.Source: §A.2 · p. 13

A.3 Deriving FSF-DMD

This derivation seeks an analytically accessible fake-velocity surrogate for DMD. Under PF-ODE-based coupling and a flow-map generator, it uses the endpoint pseudo-velocity to form a practical objective without an explicit fake-velocity network.

A.3 Deriving FSF-DMD

  • Reusing the generator input in the perturbation expresses fake velocity as a conditional expectation of the generator’s pseudo-velocity, but that expectation remains inaccessible when p(z|ˆx_t) is nontrivial.Source: §A.3 · p. 13
  • With independent coupling, the fake velocity depends on a posterior mean requiring an expectation over the prior support, which is nontrivial.Source: §A.3 · p. 13
  • For PF-ODE coupling, Lipschitz marginal velocity gives an invertible flow map, making the posterior collapse to a point mass and its mean accessible.Source: §A.3 · p. 13
  • Under PF-ODE-based coupling, the fake velocity reduces to the endpoint pseudo-velocity Fθ(ˆx_t; t, 0), yielding an objective without an explicit fake-velocity network.The derivation assumes Lipschitz continuity of the marginal velocity and a flow-map-based generator.Source: §A.3 · p. 14
  • The practical objective uses ˆx_t = (1 − t)ˆx + tz′ and a simplified gradient path, with weights chosen to obtain the stated objective.Source: §A.3 · p. 14

A.4 Related Work

The related-work discussion describes alternative ways to handle fake-side tracking and distribution matching, including tracker-alignment designs and methods that avoid an additional tracker. It also outlines comparison choices for training from scratch.

A.4 Related Work

  • The appendix describes rCM as distribution matching that regularizes flow-map models toward a self-generated correction under trigonometric interpolation and tangent normalization.Source: §A.4 · p. 14
  • SenseFlow introduces IDA to make the fake-score network track the generator more closely and ISG to regularize segments between DMD2 timesteps.Source: §A.4 · pp. 15–16
  • TwinFlow avoids an additional fake-side tracker by training the pseudo-velocity network to estimate fake velocity under an augmented condition.For training from scratch, it approximates teacher velocity using Fθ(ˆx_t; t, t).Source: §A.4 · p. 16
  • The comparison uses improved MeanFlow as the flow-map baseline, while TwinFlow uses RCGM, and evaluates TwinFlow with both base flow-map objectives from scratch.Source: §A.4 · p. 16

B.1 Implementation

The implementation uses a shared ImageNet-1K latent-generation setup and specifies separate training configurations for flow-map, flow-matching-initialized, and DMD2 comparisons. It also reports a CFG-related FID change and documents compute resources and training details.

B.1 Implementation

  • Each reported distillation run uses one B200 GPU, while each training-from-scratch run uses one A100 GPU.The two compute setups list 72-core and 128-core CPUs, respectively; wall-clock seconds per step are reported in Table 10 for distillation.Source: §B.1 · p. 16
  • ImageNet-1K experiments use 256×256 images encoded as 16×16×32 latents, with LightningDiT and a fixed B/1 transformer backbone.The setup enables RMSNorm, QK-normalization, and RoPE.Source: §B.1 · p. 16
  • The iMF-compatible augmented network embeds and repeats conditioning inputs, with Nrepeat = 4 and Nicl = 4 by default.The augmented setup samples CFG scale and timestep ranges for consistency training and distillation.Source: §B.1 · p. 16
  • Flow-map consistency training samples and orders timesteps toward higher-SNR regions, drops 10% of class labels, and drops 50% of s to t to anchor training to flow matching.The implementation uses wt,s = cos(t) by default and reports improved performance when combining cosine similarity loss with pseudo-Huber loss.Source: §B.1 · p. 17
  • For flow-matching-initialized generators, t conditions the first eight transformer layers and s conditions the remaining layers.This architecture passes target timesteps to a network that otherwise takes only xt and t as inputs.Source: §B.1 · p. 17
  • FID 4.96 to 4.70 for FSF-DMD and 7.92 to 5.51 for DMD when using 3-channel CFG with the original LightningDiT-B/1 setting.The paper reports that this CFG setting degrades FID to 14.99 for the augmented backbone setting.Source: §B.1 · p. 17

B.2 Additional Experiments

The additional experiments examine timestep and loss-weight choices, compare TTUR and GAN-loss variants, and report supplemental ImageNet metrics. They identify settings that improve FID and describe FSF-DMD as an efficient alternative without extra network components.

B.2 Additional Experiments

  • FID 4.82 to 4.70 in the flow-matching-initialized setting when adding the cosine factor to wt.For the flow-map-initialized setting, timestep shifting improves distribution matching over the unshifted setting, and dynamic scaling is retained over the clipped fixed-scale alternative with FID 3.98.Source: §B.2 · p. 18
  • Adaptive scaling gives better FID for λ ≥ 0.05 in the flow-map-initialized ablation.The reported setting uses λ × wt = 0.05/| ˜F −vΦ(ˆxt; t)|.Source: §B.2 · p. 18, Table 9 · p. 18
  • FSF-DMD obtains lower FID than the listed explicit-network variants at TTUR=1:1 and at higher TTUR ratios.The paper reports its best FID in the main comparison and describes FSF-DMD as an alternative without an additional network, separate fake-velocity training loop, or GAN.Source: §B.2 · pp. 18–19
  • Table 10 reports FID50K, Inception Score, and precision/recall at each method’s best-FID checkpoint.The supplied table caption identifies the scope as additional ImageNet-1K 256 comparisons.Source: Table 10 · p. 19

C Impact Statement

The impact statement frames the work as an efficiency-focused reformulation of distribution matching and notes that broader societal risks are not directly addressed. The licenses section identifies the code, models, and dataset used in the experiments and their licensing terms.

C Impact Statement

  • The work focuses on reformulating DMD’s fake-side term with a generator-induced pseudo-velocity surrogate, without adding a fake-score network.The impact statement says improved training stability and reduced need for auxiliary networks may improve research efficiency and reduce computational cost.Source: C Impact Statement · p. 19
  • Broader societal risks are not directly addressed; the statement flags dataset bias, misuse of generated content, and downstream applications for future consideration.Source: C Impact Statement · p. 19

D Licenses

  • The experiments use LightningDiT and VA-VAE under the MIT License, ImageNet-1K for noncommercial research and educational purposes, and UCGM under Apache-2.0.Source: D Licenses · p. 19
  • The licenses section also lists DMD2 under CC BY-NC-SA 4.0 and SenseFlow and rCM under Apache-2.0.Source: D Licenses · p. 19

Related papers

Loading 2605.19256v1…