PaperlayerView exact revision on arXiv

Source-linked AI summary

DDT: Decoupled Diffusion Transformer

Shuai Wang, Zhi Tian, Weilin Huang, Limin Wang

arXiv:2504.05741v2cs.CVcs.AI

TL;DR

Diffusion transformers achieve strong generation quality but face slow convergence and an optimization dilemma between low-frequency semantic encoding and high-frequency detail decoding. DDT decouples these roles with a condition encoder and velocity decoder, achieving state-of-the-art ImageNet results while enabling faster inference through optimized self-condition sharing.

  • Problem

    Diffusion transformers require long training and face an optimization dilemma because encoding low-frequency semantics attenuates information needed for high-frequency decoding.

  • Method

    DDT uses a dedicated condition encoder for semantic self-condition features and a velocity decoder for high-frequency velocity regression, with alignment-based self-condition sharing.

  • Results

    1.31 FID: DDT-XL/2 achieves state-of-the-art ImageNet 256 × 256 performance with approximately 4× training acceleration, while reaching 1.28 FID on ImageNet 512 × 512.

  • Takeaways & Limitations

    Larger encoders increasingly benefit performance as model scale grows, and statistical dynamic programming enables faster inference with minimal quality degradation from self-condition sharing.

Abstract

from arXiv · show

Diffusion transformers have demonstrated remarkable generation quality, albeit requiring longer training iterations and numerous inference steps. In each denoising step, diffusion transformers encode the noisy inputs to extract the lower-frequency semantic component and then decode the higher frequency with identical modules. This scheme creates an inherent optimization dilemma: encoding low-frequency semantics necessitates reducing high-frequency components, creating tension between semantic encoding and high-frequency decoding. To resolve this challenge, we propose a new \textbf{\color{ddt}D}ecoupled \textbf{\color{ddt}D}iffusion \textbf{\color{ddt}T}ransformer~(\textbf{\color{ddt}DDT}), with a decoupled design of a dedicated condition encoder for semantic extraction alongside a specialized velocity decoder. Our experiments reveal that a more substantial encoder yields performance improvements as model size increases. For ImageNet $256\times256$, Our DDT-XL/2 achieves a new state-of-the-art performance of {1.31 FID}~(nearly $4\times$ faster training convergence compared to previous diffusion transformers). For ImageNet $512\times512$, Our DDT-XL/2 achieves a new state-of-the-art FID of 1.28. Additionally, as a beneficial by-product, our decoupled architecture enhances inference speed by enabling the sharing self-condition between adjacent denoising steps. To minimize performance degradation, we propose a novel statistical dynamic programming approach to identify optimal sharing strategies.

1. Introduction

Diffusion transformers offer strong image generation quality but converge slowly, motivating investigation of decoupled encoder-decoder designs. DDT separates low-frequency semantic encoding from high-frequency decoding and reports strong ImageNet results alongside faster training and inference.

  • Diffusion transformers can outperform conventional approaches with sufficient training, but their slow convergence raises development costs.
  • DDT addresses the optimization dilemma by pairing a condition encoder for semantic self-condition features with a velocity decoder for noisy-latent velocity regression.
  • DDT uses representation alignment and decoder velocity supervision to maintain local consistency in self-condition features across adjacent denoising steps.
  • 1.31 FID: DDT-XL/2 reaches state-of-the-art ImageNet 256 × 256 performance after 256 epochs, with approximately 4× training acceleration over REPA.
  • DDT’s locally consistent self-condition enables sharing between adjacent steps, while statistical dynamic programming selects sharing strategies that minimize performance drop.

2. Related Work

Related work positions diffusion transformers as scalable alternatives to UNets and surveys efforts to accelerate training through attention, sampling, and representation strategies. The paper also provides visual examples and generation-process observations for DDT and SiT.

  • Diffusion transformers replace UNets and can outperform conventional approaches when trained for sufficient iterations.
  • Fast-training research targets attention complexity, sampling dynamics, and representation learning to address computational, sampling, and representational bottlenecks.
  • Figure 2 presents selected 256 × 256 and 512 × 512 samples generated by DDT-XL/2 trained at the corresponding ImageNet resolutions with CFG = 4.0.
  • Figure 3 depicts SiT-XL/2 reverse-SDE generation progressing from low-frequency to high-frequency components, with most time spent generating high-frequency details from t = 0.4 to t = 1.0.

3. Preliminary Analysis

The preliminary analysis views diffusion generation as low-to-high-frequency refinement and identifies semantic encoding capacity as a central constraint. Experiments show that allocating more computation to noisy timesteps improves final performance.

  • Linear-based flow matching is used as the primary analytical framework because of its simplicity and efficiency, with diffusion and flow matching treated interchangeably in some discussions.
  • Figure 4 shows that allocating more computation to noisy steps significantly improves SiT-XL/2 FID50K across timeshift values using a second-order Adams-like solver.
  • Diffusion transformers refine spectral components from low frequency to high frequency, but semantic encoding attenuates high-frequency information and creates an optimization dilemma.
  • As denoising proceeds toward less noisy timesteps, semantic encoding becomes easier while decoding residual frequencies becomes more complex.
  • The analysis concludes that current diffusion transformers are fundamentally constrained by low-frequency semantic encoding capacity.

4. Method

DDT separates semantic extraction from velocity prediction using a condition encoder and velocity decoder, then accelerates sampling by selectively reusing locally consistent self-condition features. A statistical dynamic programming method chooses the encoder-recomputation schedule under a computation budget.

  • Decoupled architecture: DDT uses a condition encoder to extract low-frequency self-condition features and a velocity decoder to estimate high-frequency velocity from the noisy latent.The encoder receives x_t, t, and y; the decoder receives x_t, t, and z_t, with class information embedded in z_t.
  • Self-condition consistency: Representation alignment regularizes encoder features to maintain local consistency of self-condition z_t across adjacent denoising steps.Intermediate encoder features are aligned with DINOv2 representations, while the decoder injects z_t through AdaLN-Zero.
  • Sampling acceleration: Sharing self-condition z_t between adjacent steps reduces computation in the self-mapping encoder with negligible performance degradation.The encoder recomputes z_t only at selected timesteps and reuses the previous feature otherwise.
  • Sampling acceleration: The sharing ratio is 1 − K/N when z_t is recalculated at K of N inference steps.The set Φ contains the timesteps where the encoder recomputes self-condition; other steps reuse the previously computed feature.
  • Sampling acceleration: Statistical dynamic programming selects the encoder-sharing set Φ by minimizing the total cosine-similarity cost as a minimal sum path problem.The method constructs a similarity matrix across timesteps, computes costs and paths, and recovers the optimal schedule by backtracking.

5. Experiment

Experiments show that DDT improves convergence and generation metrics across model sizes and resolutions, while encoder sharing accelerates inference with marginal quality loss. Larger models benefit from increasingly substantial encoders, and dynamic programming improves sharing decisions over uniform schedules.

  • Metric comparison with baselines: DDT models consistently outperform decoder-only counterparts across model sizes at 400K training steps.The DDT-B/2 and DDT-XL/2 models exceed Improved-REPA-B/2 and REPA-XL/2 by 2.8 and 1.3 FID gains, respectively.
  • System level comparison: 1.31 FID: DDT-XL/2 reaches state-of-the-art ImageNet 256×256 performance after 256 epochs, with approximately 4× faster training acceleration than REPA.At 80 epochs, DDT-L/2 and DDT-XL/2 achieve 1.64 and 1.52 FID, respectively.
  • System level comparison: 1.28 FID: DDT-XL/2 achieves this ImageNet 512×512 result after 500K fine-tuning steps with CFG3.0.At 100K steps, the model achieves 1.90 FID and outperforms REPA by a 0.28 performance margin.
  • Acceleration by Encoder sharing: Dynamic programming produces less FID drop than naive uniform sharing, while sharing every fewer than six steps gives significant speedup with nearly no visual quality loss.Uniform sharing recalculates self-condition every K steps; dynamic programming solves a minimal-sum-path problem on the similarity matrix.
  • Ablations: Larger models benefit from more substantial encoders: the optimal ratios are 8En4De for Base, 20En4De for Large, and 22En6De for XL/2 exploration.The Large model shows an unexpectedly aggressive encoder-decoder ratio, motivating further scaling of encoder depth.
  • Ablations: Attention with MLP decoder blocks perform best, although naive convolution blocks achieve comparable results under the encoder-decoder design.The study compares attention-MLP, 3×3 convolution, and naive MLP decoder configurations.

6. Conclusion

DDT decouples low-frequency encoding from high-frequency decoding to address the optimization dilemma in diffusion transformers. Its architecture also supports inference optimization through encoder-result sharing while maintaining performance.

  • DDT decouples low-frequency encoding and high-frequency decoding into dedicated components.The model uses a condition encoder and velocity decoder.
  • The default Attention + MLP decoder configuration achieves the best performance among evaluated decoder block types.Results are reported after 400K training steps on DDT-B/2(8En4De) without classifier-free guidance.
  • Increasing encoder capacity relative to the decoder yields increasingly beneficial results as overall model scale grows.The DDT-XL/2 configuration uses an aggressive 22En6De encoder-decoder layer ratio.
  • Encoder-result sharing enables faster inference while minimizing quality degradation.The approach uses statistical dynamic programming to determine optimal sharing strategies.

A. Model Specs

The model-specification passage presents a configuration table organized by layer count, hidden dimension, and attention-head count.

  • The configuration table reports the number of layers as a model specification field.
  • The configuration table reports hidden dimension as a model specification field.
  • The configuration table reports the number of attention heads as a model specification field.

B. Hyper-parameters

This section relates diffusion and flow-matching processes through forward and reverse stochastic or deterministic dynamics. It derives scheduler relationships involving α(t), σ(t), f_t, and g_t.

  • The section begins from the SDE forward and reverse processes.
  • A deterministic process exists with trajectories sharing the reverse SDE’s marginal probability densities.
  • Given x_t = α_t x_data + σϵ, the section contrasts what traditional diffusion models and flow matching learn.
  • The derivation demonstrates that flow matching’s velocity prediction is equivalent to the reverse ODE.
  • The derivation establishes relationships between f_t, α(t), and α̇(t), then derives α̇(t) and σ̇(t) under the stated conditions.The text notes that Gaussian noise has additive properties and that σ̇(t) has a complex expression.
  • The text concludes that Eq. (20) is correct.

D. Proof of Spectrum Autoregressive

The spectrum analysis defines the noisy latent from clean data, a noise scheduler, and Gaussian noise, then examines its DCT spectrum. It uses noise’s uniform spectrum to characterize retained data frequencies.

  • The noisy latent x_t is defined from the noise scheduler, clean data x_data, and Gaussian noise ϵ.The clean data’s maximum frequency is denoted K_freq.
  • The spectrum magnitude of x_t is analyzed on DCT bases u_i.
  • Gaussian noise has a uniformly distributed spectrum magnitude.
  • The analysis derives the maximum remaining frequency f_max(t) of the original data in x_t after spectral cancellation.
  • For a simplified analysis, α is replaced by t and σ by 1 − t after assuming the relevant dataset-dependent quantity is constant at 1.

E. Linear multisteps method

The section describes numerical solvers for the reverse diffusion ODE, progressing from Euler’s single-step estimate to Adams–Bashforth-style multistep estimation using previous predictions.

  • A targeted SiT-XL/2 experiment uses an Adams–Bashforth-like linear multistep solver, which is not used for DDT models in the main-paper tables.
  • The reverse diffusion process is formulated as an integral over the velocity field.
  • Euler’s method uses vθ(x_i, t_i) as the velocity estimate throughout [t_i, t_i+1].
  • Adams–Bashforth improves estimation accuracy by incorporating previous predictions through a Lagrange polynomial.
  • Pre-integrating the Lagrange polynomial yields constant coefficients, reducing ODE solving to naive summation.

F. Classifier free guidance.

The section examines classifier-free guidance strength and interval selection for DDT-XL/2, identifying a configuration that gives its best reported performance.

  • Classifier-free guidance affects diffusion-model performance, while conventional guidance improves performance at the cost of reduced diversity.
  • Interval guidance applies classifier-free guidance only during the high-frequency generation phase to preserve diversity.
  • DDT-XL/2 performs best with interval [0.3, 1] and classifier-free guidance strength 2.
Loading 2504.05741v2…