Source-linked AI summary
$Δ$-DiT: A Training-Free Acceleration Method Tailored for Diffusion Transformers
Pengtao Chen, Mingzhu Shen, Peng Ye, Jianjian Cao, Chongjun Tu, Christos-Savvas Bouganis, Yiren Zhao, Tao Chen
TL;DR
Diffusion models offer high-quality, diverse image generation but have poor real-time performance, while existing acceleration methods mainly target UNet architectures and DiT-specific structure remains underexplored. The paper investigates DiT block roles and introduces training-free Δ-DiT with Δ-Cache, achieving 1.6× speedup in 20-step generation and strong results in additional acceleration settings.
Problem
Diffusion models have poor real-time performance, existing acceleration methods mainly target UNets, and DiT structure and DiT-specific acceleration remain insufficiently explored.
Method
Δ-DiT uses Δ-Cache to reuse feature offsets while caching rear DiT blocks early and front blocks later, matching block roles with diffusion-stage characteristics.
Results
1.6× speedup is achieved in 20-step generation with improved generation performance in most cases, while the method significantly outperforms existing methods for 4-step consistent generation and 1.12× acceleration.
Takeaways & Limitations
The results support a training-free, stage-adaptive acceleration strategy tailored to DiT and indicate that cache placement aligned with block roles matters for generation quality.
Abstract
from arXiv · showhide
Diffusion models are widely recognized for generating high-quality and diverse images, but their poor real-time performance has led to numerous acceleration works, primarily focusing on UNet-based structures. With the more successful results achieved by diffusion transformers (DiT), there is still a lack of exploration regarding the impact of DiT structure on generation, as well as the absence of an acceleration framework tailored to the DiT architecture. To tackle these challenges, we conduct an investigation into the correlation between DiT blocks and image generation. Our findings reveal that the front blocks of DiT are associated with the outline of the generated images, while the rear blocks are linked to the details. Based on this insight, we propose an overall training-free inference acceleration framework $Δ$-DiT: using a designed cache mechanism to accelerate the rear DiT blocks in the early sampling stages and the front DiT blocks in the later stages. Specifically, a DiT-specific cache mechanism called $Δ$-Cache is proposed, which considers the inputs of the previous sampling image and reduces the bias in the inference. Extensive experiments on PIXART-$α$ and DiT-XL demonstrate that the $Δ$-DiT can achieve a $1.6\times$ speedup on the 20-step generation and even improves performance in most cases. In the scenario of 4-step consistent model generation and the more challenging $1.12\times$ acceleration, our method significantly outperforms existing methods. Our code will be publicly available.
1 Introduction
Diffusion models generate high-quality, diverse images but suffer from poor real-time performance because denoising is iterative. Existing acceleration methods largely target UNets, motivating DiT-specific structural analysis and acceleration.
- Diffusion models generate high-quality and diverse images, but their iterative denoising causes poor real-time performance.
- Most existing acceleration techniques target UNet-based architectures, while DiT models have achieved major successes.
- DiT lacks UNet-style encoders, decoders, and skip connections, so existing feature-reuse methods may lose previous-step sampling information.
- Δ-Cache stores feature offsets rather than feature maps to avoid losing input information.
- Δ-DiT caches rear DiT blocks during early sampling and front blocks during later sampling, following their associations with details and outlines, respectively.
- 1.6× speedup is achieved for 20-step generation, with better generation performance; the method also significantly outperforms existing methods in 4-step and 1.12× acceleration settings.
2 Related Work
Diffusion acceleration methods reduce computation through network compression, sampling optimization, or combined strategies. Cache-based methods reuse intermediate features, but mechanisms designed for UNet are unsuitable for DiT’s isotropic architecture.
- Diffusion acceleration approaches include pruning, quantization, and distillation of noise-estimation networks.
- Other approaches optimize sampling solvers or time-step skips, while combined methods optimize both networks and sampling steps.
- DeepCache, Faster Diffusion, and TGATE accelerate diffusion generation by caching feature maps in UNet or cross-attention modules.
- Feature-map caching from existing methods is unsuitable for DiT because its isotropic architecture lacks skip connections and therefore loses previous-step sampled-image information.
3 Preliminary
Diffusion generation comprises noise diffusion during training and iterative denoising during inference. DiT implements noise estimation as a composition of transformer blocks, while sampling solvers repeatedly update noisy images using estimated noise.
- The diffusion process consists of a noise-diffusion stage and a denoising stage that transforms Gaussian noise into a generated image.
- During noise diffusion, an image x_0 is perturbed at timestep t by combining it with Gaussian noise, and a network learns to estimate that noise.
- In isotropic DiT architectures, the noise-estimation network is composed sequentially from mappings of individual DiT blocks.
- The described DiT framework uses 28 transformer blocks.
- During denoising, Gaussian noise is repeatedly passed through the noise-estimation network, and a sampling solver produces the next less-noisy sample.
- The noise-estimation network may additionally receive class or text embeddings as conditional control information.
4 Stage-adaptive Inference Acceleration for Diffusion Transformers
This section introduces a training-free, stage-adaptive acceleration method for DiT that addresses information loss in feature caching and aligns cached block locations with outline- and detail-generation stages. The method combines Δ-Cache with experiments showing distinct effects for front, middle, and back DiT blocks.
- Tailored Cache Method for DiT: Δ-Cache caches feature deviations rather than feature maps, preserving the previous sampling result while skipping selected DiT blocks.This design addresses the information loss that occurs when conventional feature-map caching is transferred from UNet to DiT.
- Tailored Cache Method for DiT: Front, middle, or back DiT blocks can be targeted by Δ-Cache, enabling more flexible reuse than approaches that only skip initial blocks.The method is designed specifically for DiT's isotropic architecture and does not depend on UNet skip connections.
- Effect of DiT Blocks on Generation: Caching front blocks weakens outline generation, caching back blocks preserves outlines but reduces details, and caching middle blocks provides a compromise.The qualitative study uses different cache positions and compares their generated images against uncached generation.
- Effect of DiT Blocks on Generation: The quantitative evaluation measures outline ability with Sobel gradients and Fourier high-frequency loss, while PIQE assesses detail generation on 500 MS-COCO2017 samples.These metrics operationalize the observed distinction between outline and detail generation across block locations.
- Stage-adaptive Acceleration Method for DiT: Δ-DiT applies Δ-Cache to back blocks during early outline generation and front blocks during later detail generation.The stage boundary is controlled by hyperparameter b, with back-block caching when t ≤ b and front-block caching when t > b.
- Stage-adaptive Acceleration Method for DiT: The training-free inference schedule fixes the cache interval and cached-block count to approximate the desired computation cost without additional training.The method determines cache parameters from the target computation budget, block cost, number of blocks, and sampling schedule.
5 Experiment
Experiments evaluate Δ-DiT across DiT architectures, datasets, sampling regimes, solvers, and ablations using efficiency and image-quality metrics. The method delivers strong speedups and quality, including under four-step generation, while stage order and cache settings materially affect results.
- Experimental Settings: Experiments cover DiT-XL, PIXART-α, and PIXART-α-LCM using image-quality and efficiency metrics.The evaluation uses MACs and latency for efficiency, and FID, IS, and CLIP-Score for generation quality.
- Comparison with the Baseline Model: 1.60× speedup lets Δ-DiT outperform baseline PIXART-α on all reported metrics except a small MS-COCO2017 CLIP-Score gap.At aligned inference costs, Δ-DiT improves all metrics by a large margin, including FID 39.989 →35.882.
- Comparison with the Baseline Model: Δ-DiT outperforms Faster Diffusion and baseline DiT-XL in FID and IS with similar or lower inference overhead.The comparison supports effectiveness across the DiT-XL architecture as well as PIXART-α.
- Compatibility with Advanced Solvers: Performance conclusions remain consistent across several advanced solvers, demonstrating compatibility beyond the default DPMSolver++ setting.The classic DDIM solver was excluded because it performs poorly for 20-step PIXART-α.
- Ablation Study: Reversing stage adaptation significantly deteriorates results, supporting back-block caching early and front-block caching later.The original order follows the observed progression from outline generation to detail generation.
- Ablation Study: At b = 12, the three generation metrics are best overall, while cached-block optima differ across FID, IS, and CLIP-Score.For cached blocks, FID is optimal around Nc = 14, whereas IS and CLIP-Score peak around Nc = 21.
6 Conclusion and Limitation
The paper concludes that stage-adaptive Δ-DiT is effective for accelerating DiT generation. Its analysis of block roles remains preliminary and coarse-grained, leaving finer-grained exploration for future work.
- Conclusion: Δ-DiT applies Δ-Cache to different DiT block regions at different sampling stages based on block roles in outline and detail generation.The method is supported by the observed association between front blocks and outlines and rear blocks and details.
- Limitation: The block-to-image relationship is preliminary and coarse-grained, motivating more refined search or learning strategies.The authors identify finer-grained exploration as future work.