PaperlayerView exact revision on arXiv

Source-linked AI summary

Adversarial Distribution Matching for Diffusion Distillation Towards Efficient Image and Video Synthesis

Yanzuo Lu, Yuxi Ren, Xin Xia, Shanchuan Lin, Xing Wang, Xuefeng Xiao, Andy J. Ma, Xiaohua Xie, Jian-Huang Lai

arXiv:2507.18569v1cs.CV

TL;DR

DMD’s reverse-KL objective can promote mode collapse, and predefined divergences may limit distribution matching. The paper proposes ADM and adversarial distillation pre-training, combining them in DMDX for efficient image and video generation. It reports competitive one-step SDXL fidelity with 50× acceleration and new multi-step benchmarks.

  • Problem

    DMD’s reverse-KL minimization can promote mode collapse, while reliance on predefined divergence metrics limits distribution matching.

    Source: §1 · p. 2

  • Method

    ADM adversarially aligns latent predictions from real and fake score estimators, while adversarial distillation pre-training uses hybrid latent- and pixel-space discriminators before fine-tuning.

    Source: §1 · p. 2

  • Results

    DMDX achieves competitive fidelity to baseline SDXL at 50× acceleration in function evaluations, and multi-step ADM distillation sets new benchmarks on SD3-Medium, SD3.5-Large, and CogVideoX.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The reported results extend efficient diffusion distillation to one-step SDXL generation and multi-step image and video synthesis.

    Source: §1 · p. 2

    The method’s application is limited to guidance-distilled models because the teacher may require classifier-free guidance for accurate score prediction.

    Source: §6 · p. 8

Abstract

from arXiv · show

Distribution Matching Distillation (DMD) is a promising score distillation technique that compresses pre-trained teacher diffusion models into efficient one-step or multi-step student generators. Nevertheless, its reliance on the reverse Kullback-Leibler (KL) divergence minimization potentially induces mode collapse (or mode-seeking) in certain applications. To circumvent this inherent drawback, we propose Adversarial Distribution Matching (ADM), a novel framework that leverages diffusion-based discriminators to align the latent predictions between real and fake score estimators for score distillation in an adversarial manner. In the context of extremely challenging one-step distillation, we further improve the pre-trained generator by adversarial distillation with hybrid discriminators in both latent and pixel spaces. Different from the mean squared error used in DMD2 pre-training, our method incorporates the distributional loss on ODE pairs collected from the teacher model, and thus providing a better initialization for score distillation fine-tuning in the next stage. By combining the adversarial distillation pre-training with ADM fine-tuning into a unified pipeline termed DMDX, our proposed method achieves superior one-step performance on SDXL compared to DMD2 while consuming less GPU time. Additional experiments that apply multi-step ADM distillation on SD3-Medium, SD3.5-Large, and CogVideoX set a new benchmark towards efficient image and video synthesis.

1. Introduction

DMD’s reverse-KL objective can promote mode collapse, while predefined divergence measures may limit distribution matching. The paper proposes ADM and adversarial distillation pre-training, combining them as DMDX for efficient image and video synthesis.

1. Introduction

  • DMD’s reverse-KL minimization can drive low-probability regions to zero, focusing generation on dominant modes and potentially causing mode collapse.The authors also identify dependence on predefined divergence metrics as a limit on distribution matching.Source: §1 · p. 2
  • ADM replaces DMD’s loss with adversarial distribution matching that aligns latent predictions from real and fake score estimators.Its discriminator learns an implicit, data-driven discrepancy rather than using a predefined divergence.Source: §1 · p. 2
  • Adversarial distillation pre-training aims to improve initialization for score-based fine-tuning, using distribution-level loss and hybrid latent- and pixel-space discriminators.The authors also use a cubic timestep schedule to bias the generator toward higher noise levels.Source: §1 · p. 2
  • DMDX combines adversarial pre-training with ADM fine-tuning; its one-step SDXL generator has competitive fidelity to baseline SDXL at 50× fewer function evaluations.The paper also reports new benchmarks from multi-step ADM distillation on SD3-Medium, SD3.5-Large, and CogVideoX.Source: Figure 1 · p. 1, §1 · p. 2

2. Related Work

Related work spans several approaches to accelerating diffusion models, including progressive and consistency distillation, rectified flow, score distillation, and adversarial distillation. The paper positions its approach as learning distribution discrepancy implicitly rather than optimizing a predefined divergence.

2. Related Work

  • Progressive distillation halves sampling steps through repeated distillation, while consistency methods enforce agreement between predictions along a trajectory.The paper describes the progressive process as potentially cumbersome because it repeatedly distills from the preceding model.Source: §2 · p. 2
  • Rectified Flow learns straighter trajectories through reflow, whereas the authors report achieving straightness adversarially without repeatedly collecting large amounts of synthetic data.The authors contrast implicit adversarial discrepancy learning with SMD’s explicitly designed α-skew Jensen–Shannon divergences [16].Source: §2 · p. 3
  • Adversarial distillation methods differ in their target: some align to real data, others to teacher predictions, while this paper’s pre-training aligns to teacher-generated ODE data.The passage identifies UFOGen [79], DMD2 [85], and APT [24] as real-data examples, and SDXL-Lightning [23] and Hyper-SD [55] as intermediate-teacher-prediction examples.Source: §2 · p. 3

3. Preliminaries

The preliminaries describe diffusion and flow-based generation as processes that transform noisy samples toward data. They then introduce DMD as a reverse-KL method for distilling a teacher into a more efficient generator.

3.1. Diffusion Model

  • Diffusion models progressively add noise to data and generate samples by reversing that process, with noise schedules determining the noise level at each timestep.The passage describes DDPM and DDIM as discrete-time formulations and gives typical T = 1000 timesteps.Source: §3.1 · p. 3
  • DDIM sampling solves a probability-flow ODE from noise xT to data x0; Flow Matching instead uses velocity prediction and continuous-time coefficients.For Flow Matching, the passage gives typical coefficients αt = 1 − t and σt = t over t ∈ [0, 1].Source: §3.1 · p. 3

3.2. Distribution Matching Distillation

  • DMD distills a pretrained diffusion teacher into a one- or multi-step generator by minimizing reverse KL between the target and generator output distributions.Its gradient uses real and fake score functions, and the fake score estimator is initialized from the teacher then learned to describe the generator distribution.Source: §3.2 · p. 3

4. Methodology

ADM replaces a predefined divergence with an adaptive adversarial discrepancy over score predictions, while adversarial distillation pre-training (ADP) initializes one-step generators using teacher ODE pairs and hybrid discriminators. The paper motivates these choices by reverse-KL mode-seeking and instability when student and teacher supports overlap poorly.

4.1. Adversarial Distribution Matching

  • ADM uses a diffusion-based discriminator with trainable heads to adaptively compare fake and real score predictions after a fixed PF-ODE step of Δt.The discriminator combines frozen diffusion-backbone features with dynamically weighted heads and is trained alternately with the generator using Hinge loss.Source: §4.1 · p. 4

4.1.1. Motivation of Discriminator Timestep (t −∆t)

  • Because score functions vary with noise level, ADM preserves timestep information by comparing predictions after a small PF-ODE step rather than only at the endpoint.The default step interval is T/64.Source: §4.1.1 · p. 4, §4.1 · p. 4

4.1.2. Data-driven Effect

  • ADM's discrepancy can vary across training phases, and the recorded CogVideoX DMD loss trends steadily downward despite not being directly optimized.The paper describes global evaluation as useful early in training and finer-grained optimization later, as discrepancies change with the data encountered.Source: §4.1.2 · pp. 4–5, §4.1.3 · p. 5, Figure 3 · p. 5

4.1.3. Relationship with DMD and DMD2

  • Unlike DMD and DMD2, which add ODE- or GAN-based regularizers to counterbalance reverse-KL mode-seeking, ADM uses adversarial loss as the distillation objective itself.ADM therefore does not require those additional regularizers.Source: §4.1.3 · p. 5

4.2. Adversarial Distillation Pre-training

  • ADP initializes one-step distillation by training on offline teacher ODE pairs with latent- and pixel-space discriminators, aiming to bring generator outputs closer to synthetic teacher samples.The generator predicts the ODE endpoint, while the latent discriminator receives a randomly diffused prediction and the pixel discriminator receives its VAE-decoded representation through a vision encoder; λ1 = 0.85 and λ2 = 0.15 produced visually coherent results.Source: §4.2 · p. 5

4.2.1. Cubic Generator Timestep Schedule

  • ADP samples generator timesteps through a cubic schedule concentrated near T, while its discriminator timestep is uniform over (0, T].The cubic schedule is intended to encourage exploration of new modes under higher noise; the uniform discriminator schedule is intended to capture both high-frequency details at lower timesteps and low-frequency structure at higher timesteps.Source: §4.2.1 · p. 5, §4.2.2 · p. 5

4.2.3. Relationship with LADD

  • Compared with LADD, ADP uses ODE-pair noisy samples, a cubic schedule for deterministic Euler sampling, and an added pixel-space encoder; unlike ADP, ADM supervises probability flows across noise levels.ADP focuses on the clean-data distribution at t = 0, whereas ADM additionally compares noisy samples along both models' PF-ODEs.Source: §4.2.3 · p. 5, §4.3.1 · p. 6

4.3.3. Theoretical Objective

  • Poor student–teacher support overlap can make reverse-KL training yield vanishing or exploding gradients; when Hinge GAN's discriminator is sufficiently rich and trained, its optimum minimizes symmetric, bounded TVD.The paper states that TVD remains bounded within [0,1] and can retain optimization directions where reverse KL has vanishing gradients.Source: §4.3.2 · p. 6, §4.3.3 · p. 6

5. Experiments

Experiments evaluate one-step SDXL distillation, multi-step image and video distillation, and ablations of the proposed training choices. The reported results include strong image-text alignment and human preference, comparable video quality to the base model with 92–96% acceleration, and improved diversity.

5. Experiments

  • The experiments use 10K COCO 2014 prompts for image evaluation and report CLIP, PickScore, HPSv2, and MPS; video evaluation uses VBench.The authors state that neither ADP nor ADM requires visual training data and that most runs use 8K iterations.Source: §5 · p. 7

5.1. Efficient Image Synthesis

  • The two-stage ADP plus single-step ADM approach performs strongly on image-text alignment and human-preference benchmarks for fully fine-tuned SDXL-Base.Qualitative comparisons report better portrait aesthetics, animal-hair details, subject-background separation, and physical structure.Source: §5.1 · pp. 7–8
  • Multi-step ADM is evaluated as a standalone score-distillation method, and the reported quantitative results show superior performance across full and LoRA fine-tuning configurations.Training uses SD3-Medium and SD3.5-Large for images, and CogVideoX-2b and CogVideoX-5b for video; image and video generators use different prompt datasets and batch sizes.Source: §5 · p. 7, §5.1 · p. 8

5.2. Efficient Video Synthesis

  • ADM generators achieve comparable VBench performance to the CogVideoX base models with 92–96% acceleration.The study also evaluates CFG-integrated video generation and reports that the scale offset must grow as the targeted sampling steps decrease.Source: §5.2 · p. 8

5.3. Ablation Studies

  • The ablations report that ADP is important for one-step distillation, ADM outperforms DMD loss with or without ADP, and the proposed method significantly outperforms others in diversity.A single reflow process has limited effect, while multiple processes require considerable computation; SAM offers more imaging fidelity than DINOv2, and pixel-space weighting that is too high or low degrades structure or causes blurriness. Increasing TTUR yields marginal performance gains while nearly doubling training time.Source: §5.3 · p. 8, Table 4 · p. 8, Table 5 · p. 8

6. Limitations

The teacher model might require classifier-free guidance (CFG) to produce accurate score predictions, limiting application to guidance-distilled models such as FLUX.1-dev.

6. Limitations

  • The teacher may require CFG for accurate score prediction, limiting application to guidance-distilled models such as FLUX.1-dev.The authors say this appears common to score-distillation methods generally, rather than unique to their approach.Source: §6 · p. 8

A. Adversarial Distribution Matching

ADM alternates updates to the fake score estimator, generator, and discriminator, using teacher initialization and PF-ODE predictions in its training procedure.

A. Adversarial Distribution Matching

  • ADM alternates updates to the fake score estimator, generator, and discriminator; TTUR is set to 1 because ablations found minimal impact on final performance.The fake model and generator are therefore updated at the same frequency.Source: A. Adversarial Distribution Matching · p. 13
  • The procedure initializes the fake estimator, generator, and latent discriminator from the pretrained teacher, then updates the fake estimator using generated samples and a pretraining loss.The fake estimator update samples noise and a timestep after solving the generator PF-ODE over its schedule.Source: A. Adversarial Distribution Matching · p. 13
  • For generator updates, ADM diffuses a partially denoised prediction and obtains fake and real predictions by solving the respective score estimators’ PF-ODEs.The algorithm then updates the discriminator using the resulting fake and real predictions.Source: A. Adversarial Distribution Matching · p. 13

B. Implementation Details

The implementation uses multi-head discriminators over selected image-model features and video-model features, with 3D convolutions for video.

B.1. 2D Discriminator Design

  • The 2D discriminator uses convolutional heads with group normalization and SiLU, appended at multiple layers and fed reshaped backbone features.The implementation uses seven heads for SDXL, 24 for SD3-Medium, 38 for SD3.5-Large, and four for SAM and DINOv2; the cited efficiency table compares A100 GPU efficiency with DMD2, with ADP elapsed time including ODE-pair collection.Source: §B.1 · p. 13, Table 7 · p. 13

B.2. 3D Discriminator Design

  • The video discriminator adds 3D convolutions to extract time-dependent features, with heads taking selected video DiT block outputs.Features are extracted every three DiT blocks, yielding 10 heads for 2B models and 14 for 5B models.Source: §B.2 · p. 13

B.3. GPU efficiency.

The method reports better performance than DMD2 with less GPU time and manageable GPU memory. Implementation choices address memory use, while the listed hyperparameters specify optimizer and guidance settings across experiments.

B.3. GPU efficiency.

  • DMDX achieves better performance than DMD2 with less GPU time, without excessive GPU-memory demands.The implementation maintains a manageable memory footprint despite training multiple networks.Source: §B.3 · p. 13

B.4. Memory efficiency.

  • CPU offloading and gradient checkpointing make GPU memory use nearly that of a single network, at the cost of more CPU memory and longer iterations.The reported peak GPU memory is determined by the maximum activation of each block.Source: §B.4 · p. 14

B.5. Hyperparameters.

  • Training uses AdamW without weight decay, with beta parameters (0.0, 0.99), and discriminator and fake-model learning rates of 5e-6 and 1e-6.Generator learning rates vary by experiment; CFG is used for the real model, while fake-model training omits CFG and dropout.Source: §B.5 · p. 14

C. Main Results

On SD3-series models, the method is qualitatively competitive with the original model and outperforms the compared distillation methods in image quality and text-image alignment.

C.1. Efficient Image Synthesis

  • On SD3-series models, our method is competitive with the original model and outperforms TSCD, PCM [69], Flash [3], and LADD [60] in color, detail, structure, and image-text alignment.The comparison is qualitative, as reported for Fig. 7.Source: §C.1 · p. 14

C.2. Efficient Video Synthesis

The video experiments report VBench scores for CogVideoX base and few-step generators and qualitative comparisons with the base model. Eight-step generators are generally semantically comparable, while CFG affects visual detail and texture.

C.2. Efficient Video Synthesis

  • Figures 11–16 present qualitative comparisons between the authors’ CogVideoX generators and a baseline model; Tables 8 and 9 report VBench results for the base model and few-step generators.Figure 7 caption identifies qualitative results for LoRA fine-tuning SD3-Medium and fully fine-tuning SD3.5-Large.Source: §C.2 · p. 14, Figure 7 · p. 15
  • VBench results cover overall and quality-dimension scores in Table 8 and separate semantic-dimension scores in Table 9.The supplied passages identify the table scopes but do not report their score values.Source: §C.2 · p. 14, Table 8 · p. 15, Table 9 · p. 15
  • Eight-step CogVideoX generators are generally semantically comparable to the original model, with semantic enhancements noted in some examples.The examples include changing light in Fig. 11 and sheep movement in Fig. 14.Source: §C.2 · p. 15
  • With CFG, generators generally produce more detailed images and more delicate textures; without CFG, examples show rough hands and incorrect finger counts.The CFG-free output also has higher color contrast that can look too vibrant to be sufficiently realistic.Source: §C.2 · p. 15

C.3. Ablation Studies

The ablations compare adversarial-distillation and score-distillation settings, highlighting structural and visual differences across losses and configurations. The paper also identifies possible downstream uses and lists prompts for its generated examples.

C.3. Ablation Studies

  • MSE with a single reflow process struggles to generate a structurally visible image, while changing SAM to DINOv2 or increasing pixel-space λ2 also presents structural problems.Lowering λ2 causes noticeable blurring; the authors suggest λ1 = 0.85 and λ2 = 0.15 as a reasonable configuration.Source: §C.3 · p. 17
  • ADM fine-tuning after ADP improves color, detail, and especially structure, while standalone ADM retains one-step noise artifacts that ADP can address.Using DMD loss without ADP integration produces more severe noise artifacts; with ADP, DMD remains visually inferior to ADM in fidelity and structural integrity.Source: §C.3 · p. 17
  • Multi-seed comparisons with DMD2 show richer variation in texture, color, brightness, contrast, and structural composition.The comparison is shown in Figure 10.Source: §C.3 · p. 17, Figure 10 · p. 17

D. Broader Impact

  • The authors identify generated-data acceleration as potentially useful for recognition, detection, retrieval, domain adaptation, and customized image or video-generation LoRA plugins.These are proposed broader-impact applications.Source: D. Broader Impact · p. 17

E. Prompt List

  • The prompt list documents prompts used for generated examples, and notes that SDXL-Base’s CLIP encoder supports at most 77 tokens.The authors caution that response and text-image alignment may be insufficient for some long prompts given the encoder’s limited capacity.Source: E. Prompt List · pp. 17–18

Related papers

Loading 2507.18569v1…