PaperlayerView exact revision on arXiv

Source-linked AI summary

Spectral Amplitude Purification in Distribution Matching for Diffusion Distillation

Zhenyu Zhou, Can Wang, Chun Chen, Zeyu Zheng, Defang Chen

arXiv:2609.29116v1cs.CV

TL;DR

DMD enables few-step diffusion sampling, but its directional error is concentrated at low frequencies, while weaker signals contribute less to fine-detail recovery. SAP-DMD adaptively modulates the directional field’s amplitude spectrum while preserving phase. Across multiple text-to-image backbones, the paper reports faster convergence and improved generation quality for 2-step and 4-step sampling.

  • Problem

    DMD’s directional error concentrates spectral amplitude at low frequencies, while weaker mid- and high-frequency signals contribute less to the update and fine-detail recovery is delayed.

    Source: §3.1 · p. 4

  • Method

    SAP-DMD adaptively suppresses dominant amplitudes in the DMD directional field while preserving phase and the distribution-matching fixed point.

    Source: §1 · pp. 1–2

  • Results

    Across multiple text-to-image backbones, SAP-DMD demonstrates faster convergence and improved generation quality under both 2-step and 4-step sampling.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    Under ideal score estimation, SAP-DMD preserves the distribution-matching fixed point and maintains KL descent.

    Source: §5 · p. 10

    Preserving complex semantic details remains challenging under limited sampling budgets.

    Source: §5 · p. 10

Abstract

from arXiv · show

Distribution Matching Distillation (DMD) enables high-quality diffusion sampling in only a few steps, but its optimization dynamics remain dominated by coarse, low-frequency signals, delaying the recovery of fine-grained details. We identify a pronounced concentration of spectral amplitudes at low frequencies in the DMD directional error, where dominant low-frequency components overwhelm weaker mid- and high-frequency signals. To address this issue, we propose Spectral Amplitude Purification for Distribution Matching Distillation (SAP-DMD), a plug-and-play approach that adaptively modulates the amplitude spectrum of the DMD directional field. By suppressing the dominant tail of the amplitude spectrum, SAP-DMD reduces low-frequency dominance and promotes more effective recovery of fine structures and textures. Experiments on PixArt-$\alpha$, SD3, and SD3.5 demonstrate that SAP-DMD accelerates training convergence and improves generation quality under both 2-step and 4-step sampling.

1 INTRODUCTION

DMD’s directional error concentrates spectral amplitude at low frequencies, which can bias optimization toward coarse structures and delay fine-detail recovery. SAP-DMD adaptively modulates that spectrum, and experiments report faster convergence and improved generation quality across models and sampling budgets.

1 INTRODUCTION

  • DMD’s directional error has a pronounced low-frequency amplitude concentration, with weaker mid- and high-frequency components contributing less to the update.This concentration can bias optimization toward coarse structures and delay recovery of textures and sharp boundaries.Source: §1 · p. 1
  • SAP-DMD adaptively suppresses large upper-tail amplitudes while preserving phase, modulating the DMD directional field without changing the underlying framework.The paper presents it as a plug-and-play approach for promoting faster recovery of fine structures.Source: §1 · pp. 1–2
  • On SD3.5, SAP-DMD demonstrates clear improvements in training convergence and generation quality under both 2-step and 4-step sampling; across multiple text-to-image models, it consistently improves few-step generation quality over competitive baselines.Figure 1 reports that SAP-DMD recovers fine details earlier and improves DrawBench evaluation metrics faster on all 200 prompts using 2-step generation without GAN loss.Source: §1 · p. 2, Figure 1 · p. 2

2 PRELIMINARIES

The preliminaries introduce diffusion processes and score-based and flow-based model formulations, then describe Distribution Matching Distillation and the Fourier representation used to separate spectral amplitude from phase.

2.1 DIFFUSION MODELS

  • Diffusion models connect the data distribution qreal to a predefined noise distribution qnoise through a stochastic forward process, with a corresponding probability flow ODE sharing its marginal distributions.The learned score function is related to other prediction parameterizations; flow-based models directly parameterize a deterministic velocity field.Source: §2.1 · pp. 2–3

2.2 DISTRIBUTION MATCHING DISTILLATION

  • DMD trains a few-step generator by minimizing a time-weighted reverse KL divergence and jointly optimizes it with an auxiliary fake model.The generator gradient uses real and fake scores, while only the generator is retained for few-step inference.Source: §2.2 · p. 3

2.3 PHASE-AMPLITUDE DECOUPLING

  • The 2D DFT represents a spatial tensor through spectral amplitudes and phases, which can be modulated separately and reconstructed by the inverse DFT.Amplitude denotes each frequency component’s magnitude, while phase encodes its relative spatial arrangement.Source: §2.3 · p. 3

3 METHOD

DMD’s directional error is concentrated in low-frequency spectral amplitudes, while SAP-DMD adaptively suppresses dominant amplitudes and preserves phase. Under idealized conditions, the resulting spectral preconditioning reweights KL dissipation without changing the distribution-matching fixed point.

3.1 FREQUENCY-DOMAIN DIAGNOSTICS OF DMD

  • Low-frequency components dominate the DMD directional error, and SAP-DMD reduces their energy concentration while increasing the relative contributions of mid- and high-frequency components.The analysis links this concentration to delayed fine-detail recovery and possible bias toward coarse-scale corrections.Source: §3.1 · p. 4

3.2 SPECTRAL AMPLITUDE PURIFICATION

  • SAP adaptively suppresses dominant upper-tail amplitudes while preserving weaker spectral signals and phase, directly preconditioning the DMD directional field without changing the model architecture.A channel-wise threshold sets the boundary: weak components remain nearly unchanged, while large amplitudes approach the threshold as the kernel becomes sharper.Source: §3.2 · pp. 4–5

3.3 THEORETICAL ANALYSIS

  • Under ideal score estimation and regularity conditions, SAP-DMD reweights frequency contributions to KL dissipation without changing the distribution-matching fixed point.The theorem treats SAP-DMD as a positive spectral preconditioner of the idealized DMD distributional flow.Source: §3.3 · p. 6

4 EXPERIMENTS

Experiments compare SAP-DMD with few-step distillation baselines across text-to-image models, evaluating generation quality, training efficiency, ablations, and spectral behavior. Results cover both 2-step and 4-step sampling.

4.1 EXPERIMENTAL SETUP

  • The evaluation compares three text-to-image backbones and a broad set of distribution-matching, consistency-based, and adversarial distillation methods; it also includes FLUX.1 models in the 2-step setting.Quality is evaluated on 5K MS-COCO prompts and 3K HPSv2.1 prompts across four categories, using ImageReward, PickScore, and CLIP Score.Source: §4.1 · p. 6
  • Training uses LoRA for 4K iterations without GAN loss followed by 2K iterations with GAN supervision, with re-implemented baselines trained under the same recipe.Experiments use 4 NVIDIA A100 GPUs.Source: §4.1 · p. 6

4.2 MAIN RESULTS

  • In 4-step generation, SAP-DMD matches or surpasses competitive distribution-matching baselines without GAN supervision; its GAN-enhanced variant has the best overall preference scores on SD3.On SD3.5, SAP-DMD also outperforms TDM, SenseFlow, and Decoupled DMD on the main preference metrics; Figure 4 shows improved fine-detail quality.Source: §4.2 · pp. 6–7, Table 1 · p. 7, Figure 4 · p. 9
  • In 2-step generation, SAP-DMD without GAN achieves the best average HPSv2.1 score among compared methods and outperforms DMD2 on ImageReward, CLIP Score, and HPSv2.1.With GAN supervision, it achieves the highest ImageReward among evaluated 2-step models; Figure 4 shows sharp structures and fine visual details.Source: Table 2 · p. 7, §4.2 · pp. 7–8, Figure 4 · p. 9
  • SAP-DMD substantially accelerates early fine-detail recovery and improves evaluation metrics faster than DMD2, reaching higher generation quality within fewer training iterations.Training takes 10.56 A100 GPU hours for SAP-DMD versus 10.53 hours for DMD2 under identical hardware and settings.Source: §4.2 · p. 8
  • All tested SAP-DMD configurations outperform DMD2; adaptive amplitude purification improves all evaluated metrics, unlike phase-only modulation, spatial clipping, random masking, or fixed low-frequency suppression.The default p = ∞, k = 1 setting has the highest CLIP Score while remaining competitive on other metrics; p = 3, k = 3 yields the best preference-oriented scores.Source: §4.2 · p. 8, Table 3 · p. 8, Table 4 · p. 8
  • Across training, SAP-DMD consistently reduces low-frequency dominance, with stronger attenuation at low frequencies in its frequency-resolved suppression map.Figure 3 reports statistics averaged over 256 samples at t = 0.5, with checkpoints every 100 steps.Source: Figure 3 · p. 8, §4.2 · p. 9

5 CONCLUSION

The conclusion presents SAP-DMD as a spectral preconditioning method for DMD and reports faster convergence and improved few-step quality. It also notes that complex semantic details remain challenging under limited sampling budgets.

5 CONCLUSION

  • DMD’s directional field concentrates spectral amplitude at low frequencies, with dominant components overwhelming weaker signals and delaying fine-detail recovery.SAP-DMD modulates amplitudes while preserving phase and weaker spectral components.Source: §5 · p. 9
  • Experiments across multiple text-to-image backbones show faster convergence and improved generation quality with both 2-step and 4-step sampling.The method adaptively suppresses dominant amplitudes while preserving phase and weaker spectral components.Source: §5 · pp. 9–10
  • Complex semantic details remain challenging under limited sampling budgets.Source: §5 · p. 10

AI USE STATEMENT

The paper reports its use of generative AI and situates SAP-DMD among few-step distillation and frequency-domain research. It identifies optimization signals in diffusion distillation as an underexplored spectral focus.

AI USE STATEMENT

  • Generative AI assisted with language editing, experimental-design feedback, code refinement, ablations, and theoretical analysis and proof writing; the authors report reviewing and verifying the work.Source: AI USE STATEMENT · p. 10

A RELATED WORK

  • Few-step distillation includes trajectory, consistency, and distribution-matching approaches; SAP-DMD studies DMD’s directional-field spectrum without changing its architecture or training objective.Source: A RELATED WORK · p. 15
  • Prior frequency-domain diffusion work focuses on generation dynamics, representations, or reconstruction, while distillation optimization signals remain largely underexplored.SAP-DMD studies the Fourier structure of the DMD directional field during distillation.Source: A RELATED WORK · p. 15

B IMPLEMENTATION DETAILS

Experiments use a shared two-phase training setup and specify method-specific settings for baselines, SAP-DMD, and design ablations.

B IMPLEMENTATION DETAILS

  • Training uses LoRA with AdamW, first for 4,000 generator iterations without GAN loss on JourneyDB prompts, then 2,000 iterations with GAN loss on MJHQ-30K.The global batch size is 4, and the same common setup is used for re-implemented distribution-matching baselines.Source: B IMPLEMENTATION DETAILS · p. 15
  • DMD2 experiments disable TTUR, use learning rates of 5×10^-5 for the fake model and 1×10^-5 for the generator, and set second-stage GAN loss weight to 0.1.Source: B IMPLEMENTATION DETAILS · pp. 15–16
  • SAP-DMD uses hard clipping with p = ∞ and sets k = 3 for PixArt-α and k = 1 for SD3 and SD3.5.Source: B IMPLEMENTATION DETAILS · p. 16
  • TDM enables interval re-noising, fake-model importance sampling, and Pseudo-Huber generator loss; SenseFlow uses parameter shifting and intra-segment guidance, while Decoupled DMD independently re-noises its DM and CA terms.SenseFlow’s shift is ϕ ← 0.98 · ϕ + 0.02 · θ with guidance weight 0.5.Source: B IMPLEMENTATION DETAILS · p. 16
  • Design ablations compare phase clipping, spatial-magnitude clipping, random masking of conjugate-symmetric frequency pairs, and low-frequency scaling against the same DMD directional error.LF-Sup scales normalized radial frequencies r ≤ 0.25 by 0.5; other frequencies are unchanged.Source: B IMPLEMENTATION DETAILS · p. 16

C THEORETICAL PROOF

The proof analyzes the distributional evolution induced by SAP-DMD under regularity assumptions and ideal score estimation. It relates the directional error to a Fourier-domain inner product and characterizes when the distribution-matching error is stationary.

C THEORETICAL PROOF

  • The analysis assumes smooth, positive densities on a common connected support with suitable boundary decay, enabling integration by parts.Source: C THEORETICAL PROOF · p. 16
  • Under ideal score estimation, the directional error is related to the difference between the densities’ scores, up to a positive time-dependent factor.The factor depends only on diffusion time and can be absorbed into the weighting term.Source: C THEORETICAL PROOF · p. 16
  • Parseval’s identity expresses the spatial inner product through Fourier-domain amplitudes, using a unitary DFT convention whose normalization affects only a positive factor.Source: C THEORETICAL PROOF · p. 17
  • Equality in the derived condition requires zero directional error almost everywhere, which implies p = q; conversely, p = q gives zero error and dE/dτ = 0.Source: C THEORETICAL PROOF · p. 17

D.1 SPECTRAL DIAGNOSTICS ACROSS TIMESTEPS

Across representative timesteps, SAP-DMD suppresses low-frequency spectral dominance, while log amplitudes provide a more symmetric basis for its adaptive boundary. The frequency redistribution remains evident in the generator’s parameter update.

D.1 SPECTRAL DIAGNOSTICS ACROSS TIMESTEPS

  • SAP-DMD suppresses dominant low-frequency amplitudes across timesteps, with weaker modulation toward higher frequencies.The raw DMD field consistently concentrates amplitude at low frequencies despite variation in absolute spectral scale.Source: Figure 5 · p. 18, §D.1 · p. 18
  • Low-frequency energy dominance strengthens at larger t, while SAP-DMD reduces its relative share across the tested timesteps.The relative energy share of mid- and high-frequency components increases after purification.Source: Figure 6 · p. 18, §D.1 · p. 18
  • Log-amplitude distributions are more symmetric and bell-shaped across timesteps, supporting the adaptive boundary, though heavier tails remain at larger t.Across 128 samples and 16 channels, median absolute skewness decreases after the log transform at each tested timestep.Source: §D.1 · pp. 18–19, Figure 7 · p. 19

D.2 PROPAGATION THROUGH THE GENERATOR JACOBIAN

  • Figure 8 shows SAP-DMD’s frequency redistribution remains evident after multiplication by the generator Jacobian across all tested noise levels, confirming an effect on parameter-space updates beyond modifying output-space ∆t.Results use the same 4,000-step DMD2 checkpoint and Jacobian for raw and SAP-transformed signals and are averaged over 128 prompts.Source: §D.2 · p. 19

D.3 SENSITIVITY TO TRAINING RECIPE

The training-recipe analysis finds modest, metric-dependent gains from larger batches but a substantially worse performance–compute trade-off. Increasing fake-model updates likewise adds cost without consistent quality improvement.

D.3 SENSITIVITY TO TRAINING RECIPE

  • Increasing the effective batch size generally improves preference metrics modestly, but raises training cost by more than eightfold when increased from 4 to 32.Adversarial fine-tuning benefits more clearly from a larger batch but does not consistently improve every metric; the main experiments use batch size 4.Source: §D.3 · pp. 19–20
  • Increasing TTUR from 1 to 5 leaves MS-COCO scores nearly unchanged and lowers average HPSv2.1 from 32.67 to 32.46, while cost rises from 10.56 to 28.82 A100 hours.The authors adopt TTUR = 1, defined as the number of fake-model updates per generator update.Source: §D.3 · pp. 19–20
  • Across three matched training seeds on SD3.5 Medium, SAP-DMD improves all metrics on average, with relatively small standard deviations.The authors report that the gains are reproducible across seeds rather than driven by a single run.Source: §D.3 · p. 20

D.4 SAMPLE DIVERSITY

SAP-DMD does not reduce measured sample diversity on SD3 or SD3.5 and is statistically indistinguishable from DMD2 on PixArt-α. Across three architectures, it also accelerates early high-frequency recovery and develops fine visual details sooner.

D.4 SAMPLE DIVERSITY

  • SAP-DMD improves pairwise LPIPS diversity on SD3 and SD3.5 and is statistically indistinguishable from DMD2 on PixArt-α.The comparison uses 128 prompts and four generated images per prompt; absolute LPIPS values are comparable only within each backbone.Source: §D.4 · pp. 20–21, Table 7 · p. 21

D.5 EARLY-STAGE HIGH-FREQUENCY RECOVERY

  • 10.0%, 13.5%, and 33.1% lower AUCHF for SAP-DMD on SD3.5, SD3, and PixArt-α, respectively, indicates faster early high-frequency recovery.AUCHF summarizes normalized area under the teacher-relative discrepancy curve over the first 25% of the 4,000-step training budget; lower is faster convergence.Source: §D.5 · p. 21, Figure 9 · p. 21
  • With four-step generation, SAP-DMD develops fine-grained content sooner across all three architectures while coarse structure and global composition remain comparable.The reported details include fur, hair, facial features, feathers, and flower petals, with the difference visible at intermediate checkpoints.Source: §D.5 · p. 21, Figure 10 · p. 22, Figure 11 · p. 22
Loading 2609.29116v1…