PaperlayerView exact revision on arXiv

Source-linked AI summary

Continuous-Time Distribution Matching for Few-Step Diffusion Distillation

Tao Liu, Hao Yan, Mengting Chen, Taihang Hu, Zhengrong Yue, Zihao Pan, Jinsong Lan, Xiaoyong Zhu, Ming-Ming Cheng, Bo Zheng, Yaxing Wang

arXiv:2605.06376v1cs.CVcs.AI

TL;DR

Few-step diffusion distillation is limited by fixed discrete supervision, which can accumulate discretization errors and degrade detail. CDM replaces this with continuous-time scheduling and off-trajectory alignment, achieving state-of-the-art 4-step generation across evaluated models without complex auxiliary modules.

  • Problem

    Existing discrete-time distillation methods supervise fixed timesteps, leaving accumulated discretization errors and detail degradation during few-step inference.

  • Method

    CDM combines a dynamic continuous schedule with velocity-driven off-trajectory distribution matching to regularize the student across continuous time.

  • Results

    CDM achieves state-of-the-art 4-step generation across SD3-Medium and Longcat-Image, recovering sharp textures and semantic adherence without complex auxiliary modules.

  • Takeaways & Limitations

    Continuous-time distribution matching provides a high-fidelity route to few-step image generation within the evaluated text-to-image models.

  • Takeaways & Limitations

    CDM increases per-iteration training cost through longer backward simulations and an additional teacher forward pass.

Abstract

from arXiv · show

Step distillation has become a leading technique for accelerating diffusion models, among which Distribution Matching Distillation (DMD) and Consistency Distillation are two representative paradigms. While consistency methods enforce self-consistency along the full PF-ODE trajectory to steer it toward the clean data manifold, vanilla DMD relies on sparse supervision at a few predefined discrete timesteps. This restricted discrete-time formulation and mode-seeking nature of the reverse KL divergence tends to exhibit visual artifacts and over-smoothed outputs, often necessitating complex auxiliary modules -- such as GANs or reward models -- to restore visual fidelity. In this work, we introduce Continuous-Time Distribution Matching (CDM), migrating the DMD framework from discrete anchoring to continuous optimization for the first time. CDM achieves this through two continuous-time designs. First, we replace the fixed discrete schedule with a dynamic continuous schedule of random length, so that distribution matching is enforced at arbitrary points along sampling trajectories rather than only at a few fixed anchors. Second, we propose a continuous-time alignment objective that performs active off-trajectory matching on latents extrapolated via the student's velocity field, improving generalization and preserving fine visual details. Extensive experiments on different architectures, including SD3-Medium and Longcat-Image, demonstrate that CDM provides highly competitive visual fidelity for few-step image generation without relying on complex auxiliary objectives. Code is available at https://github.com/byliutao/cdm.

1 Introduction

The paper introduces Continuous-Time Distribution Matching (CDM), migrating DMD from fixed discrete schedules to continuous optimization with dynamic scheduling and velocity-driven off-trajectory matching. Experiments across SD3-Medium and Longcat-Image show strong few-step generation without complex auxiliary modules.

  • Motivation: Diffusion and flow-matching models achieve high-fidelity synthesis but require tens to hundreds of sequential network evaluations, limiting real-world deployment.This iterative sampling process creates a severe computational bottleneck.
  • Limitations: Existing DMD methods restrict backward-simulation training to fixed discrete anchors matching the inference schedule, unlike consistency distillation’s continuous trajectory optimization.The paper identifies this sparse discrete formulation as a structural limitation of DMD.
  • Key Insights: Randomly sampling continuous timesteps t ∈(0, 1] with random length decouples training from the fixed inference schedule and improves details, artifacts, and HPSv3.The dynamic schedule optimizes at arbitrary continuous points rather than predefined discrete inference anchors.
  • Key Insights: The DM loss drives students toward the teacher’s CFG-free distribution rather than serving merely as a training-stability regularizer.This conclusion is supported by comparisons of teacher samples with and without CFG and students distilled using DM alone.
  • CDM Framework: CDM unifies dynamic continuous scheduling with an off-trajectory loss using velocity-driven extrapolation to mitigate numerical integration errors in few-step sampling.The method addresses trajectory drift caused by large integration steps and truncation errors.
  • Results: Experiments across SD3-Medium and Longcat-Image establish state-of-the-art few-step image-generation results without complex auxiliary modules.The reported results demonstrate gains from the continuous paradigm across different model architectures.

2 Related Work

Related work accelerates diffusion inference through progressive step compression, consistency-based approaches, and score-based distribution matching. Score-based distillation evolved from text-to-3D score optimization into KL-based few-step image-generation frameworks.

  • Diffusion Distillation: Diffusion distillation addresses the computational bottleneck caused by iterative sampling in diffusion models.
  • Diffusion Distillation: Progressive distillation trains a student to compress two teacher steps into one, progressively halving required function evaluations.
  • Score-based Distillation: Score-based distillation began in text-to-3D generation, where SDS and VSD used pretrained diffusion scores to optimize 3D representations.
  • Score-based Distillation: Diff-Instruct and DMD extended score-based distillation to 2D image generation through KL-based distribution matching for few-step generators.

3 Method

CDM extends DMD into a continuous-time distillation framework for high-fidelity few-step generation. It combines a dynamic random-length schedule with off-trajectory supervision to regularize the student flow between sampled anchors.

  • Framework overview: CDM lifts DMD into a fully continuous-time framework for training a student flow model to generate high-quality samples in N steps from a teacher requiring T ≫ N steps.The framework uses a student prediction Dθ and a pretrained teacher prediction Dϕ conditioned on text.
  • Framework overview: The objective decouples into CFG Augmentation, which improves text-image alignment, and Distribution Matching, which aligns the student marginal distribution with the real data manifold.CA uses the frozen teacher’s conditional and unconditional predictions, while DM contrasts a frozen real teacher with an online-updated fake teacher.
  • Dynamic continuous schedule: CDM replaces fixed inference anchors with a random-length schedule N ∼ U{1, Nmax} and a strictly decreasing continuous sequence from pure noise at 1 to clean images at 0.The schedule samples intermediate anchors continuously, allowing supervision at arbitrary points in (0, 1] given enough iterations.
  • Dynamic continuous schedule: Independent continuous sampling of the student and teacher timesteps regularizes CA and DM gradients across the time domain rather than sparse discrete anchors.The score-matching analysis connects DM optimization to minimizing KL divergence between student and real-data distributions at the sampled perturbation time.
  • Continuous-time alignment objective: The CDM loss Euler-extrapolates an on-trajectory latent using the student velocity, re-noises the off-trajectory prediction, and applies teacher-based matching to penalize invalid velocity fields.This supervision targets intermediate or beyond-anchor latents, enforcing self-consistency across continuous intervals and reducing inter-anchor inconsistency.

4 Experiments

Experiments on SD3-Medium and Longcat-Image show that CDM delivers competitive 4-NFE generation quality, with sharper details and stronger prompt adherence than representative baselines. Ablations further show that combining all loss components and retaining CDM’s continuous-time design choices is essential for balancing alignment, aesthetics, and fidelity.

  • Experiment Setting: Experiments evaluate 1024 × 1024 SD3-Medium generation using AES, PickScore, HPS v3, CLIP Score, and DPG-Bench prompt adherence.The evaluation uses 2K PickScore test prompts and 1K DPG-Bench prompts.
  • Quantitative Results: With only 4 NFE, CDM achieves the best Aesthetic, DPGBench, PickScore, and HPSv3 results on SD3-Medium while maintaining a highly competitive CLIPScore.CDM improves HPSv3 over image-free D-DMD from 9.176 to 9.561.
  • Qualitative Comparison: CDM produces sharper textures, richer fine-grained details, and stronger multi-entity prompt adherence, whereas competing few-step methods often blur high-frequency content or omit attributes.The comparison includes representative baselines and the 100-NFE teacher.
  • Loss Components: Using LCA alone causes structural collapse, while LDM or LCDM alone improve visual quality but weaken prompt adherence; combining LCA with either distribution-matching loss improves both.The full LCA + LDM + LCDM objective achieves the best performance across all metrics, with HPSv3 peaking at 9.5.
  • Core Mechanism Design: Replacing the dynamic schedule with a fixed schedule or removing velocity-based perturbations reduces generation fidelity and overall performance.Gaussian re-noising fails to capture meaningful off-trajectory states, whereas velocity extrapolation simulates the truncation drift encountered during large-step inference.
  • Core Mechanism Design: Using the full-trajectory final generation as the teacher input is suboptimal compared with supervising localized intermediate targets.Localized predictions provide a more direct error-correction signal than extended backward simulation.

5 Conclusion

The paper presents Continuous-Time Distribution Matching (CDM), a distillation framework for high-quality few-step diffusion generation. CDM addresses discretization errors and detail degradation by combining continuous-time optimization, dynamic scheduling, and off-trajectory alignment.

  • 5 Conclusion: CDM is introduced as a novel framework for high-quality few-step diffusion generation.The framework is designed for diffusion distillation.
  • 5 Conclusion: CDM shifts optimization from fixed discrete timesteps into continuous time to address accumulated discretization errors and detail degradation during few-step inference.Existing discrete-time methods optimize on fixed timesteps, which the paper identifies as the source of these problems.
  • 5 Conclusion: CDM uses dynamic scheduling and the off-trajectory alignment objective LCDM to explicitly simulate and correct the distillation process.These are the framework's two stated mechanisms for bridging the gap between discrete optimization and few-step generation.
  • 5 Conclusion: The authors hope CDM enables more accessible diffusion distillation and motivates future extensions to complex visual synthesis.This is presented as the work's broader intended impact and future direction.

A Limitations · B Broader Impact

CDM’s continuous schedule and alignment loss improve few-step generation but increase training cost and remain bounded by teacher quality and text-to-image evaluation scope. Its efficiency may broaden access to high-quality generation, while associated misuse risks are largely inherited from the distilled teacher.

  • A Limitations: CDM increases per-iteration training cost despite adding no inference-time cost.The variable simulation length prolongs average backward simulation, and the CDM loss requires an extra forward pass through real and fake teachers.
  • A Limitations: The dynamic schedule samples a variable simulation length N ∼ U{1, Nmax}, which prolongs average backward simulation.
  • A Limitations: The CDM loss requires an extra forward pass of the real and fake teachers on an extrapolated off-trajectory latent.
  • A Limitations: CDM is fundamentally upper-bounded by its teacher because DM and CDM losses use the teacher’s score as supervision.Concepts or compositions that the teacher handles poorly are unlikely to be recovered through distillation alone.
  • A Limitations: The empirical study is restricted to text-to-image generation, including SD3-Medium and Longcat-Image.
  • B Broader Impact: CDM reduces inference cost for high-quality text-to-image generation by an order of magnitude, improving accessibility on commodity hardware.
  • B Broader Impact: Because CDM distills a pre-trained teacher without introducing new generative capabilities or training data, risks such as misinformation and copyright infringement are largely inherited.

C Experiment Details … D.3 Score-Matching View of the DM Gradient

The paper details training and baseline implementations for SD3-Medium and LongCat distillation, then provides a score-matching interpretation of the CA and DM gradients. Tweedie’s formula connects denoiser predictions to scores, while continuous-time sampling applies supervision across the full noise-time domain.

  • C.1.1 SD3-Medium: SD3-Medium is distilled into a 4-step student using equally weighted CFG Augmentation, Distribution Matching, and Continuous-Time Distribution Matching objectives.The training data comprises 200K prompts sampled from T2I-2M, LAION, and related datasets.
  • C.1.2 LongCat: LongCat uses the same basic training settings as SD3-Medium, with rank-64, alpha-128 LoRA fine-tuning, batch size 64, and 2K iterations on 16 A100 GPUs.Training takes approximately 24 hours to reach optimal convergence.
  • C.2 Baseline Implementation Details: Baselines share SD3-Medium as the teacher backbone, while NFE and inference-time hyperparameters follow each method’s official configuration.TDM, Hyper-SD, and Flash-Diffusion use their official checkpoints and recommended generation settings.
  • C.2 Baseline Implementation Details: DMD2 and D-DMD are re-implemented within the unified framework under CDM’s exact setting and without GAN-based adversarial loss.No official SD3-Medium checkpoints are publicly available for these two baselines.
  • D.1 Tweedie’s Formula under the Flow-Matching Interpolation: Tweedie’s formula under flow-matching interpolation relates the denoiser’s posterior-mean prediction to the underlying score function.A mean-squared-error denoiser converges to E[x0|zτ], yielding the score-prediction identity used in the paper.
  • D.3 Score-Matching View of the DM Gradient: Because student and teacher timesteps are independently sampled from the same continuous distribution on (0, 1], CA and DM supervision covers the entire time domain rather than sparse anchors.The two gradients share the prefactor τ 2 1−τ (∂Dθ/∂θ)⊤ and act on score-valued targets.
  • D.2 Score-Matching View of the CA Gradient: The CA gradient reduces to the implicit-classifier gradient ∇zτ log preal(c|zτ), thereby steering optimization toward text–image alignment.This follows by applying Tweedie’s formula to conditional and unconditional teacher predictions and using Bayes’ rule.
  • D.3 Score-Matching View of the DM Gradient: The DM gradient compares conditional scores from the real-data and fake distributions, driving pfake toward preal at any continuous noise level ˜τ.The fake teacher Dψ tracks pfake online and estimates its score during optimization.

E Local and Global Truncation Error of Euler Sampling

The appendix shows that explicit Euler sampling has global error O(¯h) controlled by M2, the supremum of the velocity field’s material derivative. CDM constrains M2 through off-trajectory supervision, transferring the teacher’s smoothness to the student and suppressing high-frequency oscillations.

  • Error derivation: The analysis derives local and global truncation-error bounds for explicit Euler sampling of the probability-flow ODE.The setup assumes a sufficiently smooth velocity field that is L-Lipschitz in x and discretizes τ ∈[ϵ, 1] with a decreasing schedule.
  • Global error: The global Euler error is O(¯h), because O(1/¯h) steps accumulate O(¯h2) local error and lose one order.For a fixed step budget N, M2 is the only factor available for optimization through training.
  • CDM regularization: CDM supervises the student at an extrapolated off-trajectory latent, where a first-order expansion links the velocity difference to the material derivative dvθ/dτ.The material derivative is functionally approximated by the finite difference across the Euler leap.
  • CDM regularization: By matching the teacher both at the anchor and at the extrapolated point, CDM transfers the teacher’s bounded velocity variation to the student.This implicitly regularizes M2 and prevents the sporadic high-frequency oscillations observed in discretely trained models.

F Training Algorithm

CDM trains the student with a random-length continuous schedule and combines backward-simulation, fake-teacher, CA, DM, and off-trajectory CDM objectives within each iteration. A shared intermediate latent supports the student losses, while the fake teacher is optimized separately and the student receives one combined update.

  • Algorithm 1 Training Procedure of CDM: All three student-side losses—CA, DM, and off-trajectory CDM—share the same intermediate latent xti within a single backward simulation.This shared latent is used across objectives to improve training efficiency.
  • Algorithm 1 Training Procedure of CDM: The fake teacher is updated separately with the standard flow-matching objective, using the same anchor i as the distillation losses.The fake-teacher update operates on the re-noised one-step student prediction.
  • Algorithm 1 Training Procedure of CDM: Each iteration samples a prompt, a random length N from 1 to Nmax, and a descending schedule 1 = t1 > t2 > ··· > tN > 0.The schedule is sampled uniformly over integer lengths from 1 through Nmax.
  • Algorithm 1 Training Procedure of CDM: The method performs one no-gradient backward simulation from xt1 sampled from N(0, I), running the student for N Euler steps to obtain trajectory latents.The intermediate latent xti is extracted from this trajectory at a uniformly sampled anchor ti.
  • Algorithm 1 Training Procedure of CDM: The student parameters receive one combined gradient step using LCA + LDM + LCDM, repeated until convergence.The update is θ ← θ − η ∇θ(LCA + LDM + LCDM).

G More Quantitative Comparisons

CDM is compared with D-DMD using complementary quality and efficiency evaluations. It ranks first on six of seven quality metrics, while adding training overhead but preserving comparable memory and inference latency.

  • Evaluation protocol: The evaluation measures OCR accuracy on 1K FlowGRPO prompts and FID on 10K COCO 2014 validation prompts.Relative training time is normalized to D-DMD, and inference latency is measured at 1024 × 1024 resolution with 4 NFE on one GPU.
  • Quality comparison: CDM ranks first on six of seven quality metrics: Aesthetic, DPGBench, PickScore, HPSv3, CLIPScore, and FID.The evaluation adds OCR accuracy and FID to the five main quality metrics.
  • Efficiency comparison: 1.8× relative training time is required by CDM versus D-DMD, with comparable peak memory of 62.5 versus 62.2 GB.The overhead is attributed to longer average backward simulations and an extra forward pass on the extrapolated off-trajectory latent.
  • Efficiency comparison: 246 ms/img inference latency is shared by CDM and D-DMD because all configurations use the same backbone and number of function evaluations.The additional computational cost is confined to training.

H Quantitative Evaluation of the DM Loss · I CDM under Varying Inference Steps

Quantitative evaluation shows that DM-only students closely reproduce the semantic and visual degradation of CFG-free teachers on SD3-Medium and Longcat-Image. CDM’s continuous-time training supports coherent, prompt-aligned generation across varying inference-step counts despite a 4-NFE training target.

  • H Quantitative Evaluation of the DM Loss: DM-only students closely track CFG-free teachers’ deterioration in semantic alignment and visual fidelity on SD3-Medium and Longcat-Image.The comparison evaluates students distilled exclusively with the DM loss against teachers with and without CFG.
  • H Quantitative Evaluation of the DM Loss: Disabling CFG significantly degrades both semantic alignment metrics and visual fidelity for the teacher.The passage specifically cites DPGBench and HPSv3 as examples of affected semantic alignment metrics.
  • H Quantitative Evaluation of the DM Loss: The DM-only student mirrors the CFG-free teacher across all reported metrics, quantitatively confirming the visual degradation observed in Figure 3.Table 4 summarizes this alignment with CFG-free distributions.
  • I CDM under Varying Inference Steps: CDM remains coherent and prompt-aligned at NFE values of 3, 4, 6, and 8 using identical prompts and random seeds.Finer details emerge as more inference steps are used.
  • I CDM under Varying Inference Steps: Although trained with a target of 4 NFE, CDM is not bound to a specific inference schedule.Its continuous-time training paradigm enables evaluation under varying inference-step counts.
  • I CDM under Varying Inference Steps: The dynamic continuous schedule randomizes the backward simulation length as N ∼ U{1, Nmax} at every training iteration.This exposes the student to trajectories of varying lengths rather than a single fixed grid.

J More Qualitative Results

Additional qualitative samples show that CDM produces diverse 1024 × 1024 images with 4 NFE on both SD3-Medium and Longcat-Image, across varied prompt categories. The results indicate consistent generalization of the continuous-time distribution matching framework across the two backbones.

  • Longcat-Image: Additional Longcat-Image results at 1024 × 1024 resolution with 4 NFE show that CDM generalizes across a different backbone and diverse prompt categories.The samples provide corresponding qualitative results for Longcat-Image.
  • SD3-Medium: CDM generates additional SD3-Medium samples at 1024 × 1024 resolution with 4 NFE across photorealistic portraits, complex scenes, stylized illustrations, and text-rich compositions.These results complement the main qualitative comparison.
Loading 2605.06376v1…