Source-linked AI summary
DDiT: Dynamic Patch Scheduling for Efficient Diffusion Transformers
Dahye Kim, Deepti Ghadiyaram, Raghudeep Gadde
TL;DR
Diffusion Transformers are computationally expensive, and fixed patch sizes do not adapt computation to changing denoising detail. DDiT dynamically schedules patch sizes at test time, reporting up to 3.52× and 3.2× speedups while preserving generation quality and prompt alignment.
Problem
Fixed patch sizes and rigid reduction strategies impose high computation without adapting granularity to denoising detail or input complexity.
Method
DDiT dynamically selects latent patch sizes at each timestep using latent-manifold evolution as a proxy for transitions between coarse and fine generation.
Results
3.52× speedup on FLUX-1.Dev and 3.2× on Wan 2.1 are reported while maintaining perceptual quality, photorealism, and prompt alignment.
Takeaways & Limitations
Dynamic patch scheduling provides a low-cost test-time strategy for reducing diffusion inference computation across image and video generation.
Abstract
from arXiv · showhide
Diffusion Transformers (DiTs) have achieved state-of-the-art performance in image and video generation, but their success comes at the cost of heavy computation. This inefficiency is largely due to the fixed tokenization process, which uses constant-sized patches throughout the entire denoising phase, regardless of the content's complexity. We propose dynamic tokenization, an efficient test-time strategy that varies patch sizes based on content complexity and the denoising timestep. Our key insight is that early timesteps only require coarser patches to model global structure, while later iterations demand finer (smaller-sized) patches to refine local details. During inference, our method dynamically reallocates patch sizes across denoising steps for image and video generation and substantially reduces cost while preserving perceptual generation quality. Extensive experiments demonstrate the effectiveness of our approach: it achieves up to $3.52\times$ and $3.2\times$ speedup on FLUX-1.Dev and Wan $2.1$, respectively, without compromising the generation quality and prompt adherence.
1. Introduction
DDiT targets the computational cost and rigidity of fixed reduction strategies by adapting latent patch granularity to denoising complexity. It uses coarser patches when less detail is needed and finer patches when high fidelity matters, achieving substantial speedups while maintaining generation quality.
- Motivation: Generating a 5-second 720p video with Wan-2.1 takes 30 minutes on an RTX 4090, motivating more efficient diffusion generation.Existing acceleration methods include feature caching, feature pruning, vector quantization, and model distillation.
- Motivation: Static reduction methods can permanently discard computations critical to a specific output, causing quality degradation.Many existing approaches remove fixed amounts of weights, operations, or tokens.
- Approach: DDiT dynamically adjusts latent patch size at each denoising timestep, using larger patches for coarse structure and smaller patches for high-fidelity detail.This reallocates computation rather than permanently discarding weights or operations.
- Approach: The method estimates generation complexity from latent-manifold change and uses slower evolution to justify coarser patches and faster evolution to motivate finer patches.The scheduler adapts computation to both timestep and prompt.
- Results: 3.52× speedup on FLUX-1.Dev and 3.2× on Wan 2.1 are reported while maintaining perceptual quality, photorealism, and prompt alignment.The strategy is presented as a low-cost test-time method requiring minimal architectural changes and generalizing across image and video diffusion transformers.
2. Related work
Prior diffusion-transformer efficiency methods commonly use predefined, non-adaptive reductions. Figure 2 contrasts those fixed patch sizes with DDiT's timestep-dependent token allocation.
- Related efficiency strategies: Existing efficiency methods include fast sampling, caching, quantization, and knowledge distillation.These methods target sampling steps, operations, numerical precision, or model size through different strategies.
- Dynamic tokenization: Figure 2 contrasts constant patch sizes across denoising steps with DDiT's adaptive allocation of fewer or more tokens according to latent complexity.The figure uses a real pixel-space image for illustration, although DiT operates on VAE latents.
- Limitations of prior work: Many prior approaches rely on hard, predefined reduction rules that lack adaptivity to content or denoising state.This limitation motivates methods that vary computation according to generation conditions.
3. Approach
DDiT adapts diffusion transformers to process latent patches at different resolutions and dynamically schedules patch sizes during denoising. Larger patches reduce token count and computation, while latent-evolution and spatial-variance signals guide finer processing where detail is needed.
- Dynamic patching and tokenization: DDiT extends pretrained DiTs to support patch sizes that are integer multiples of the base size, including p, 2p, and 4p.Patch-specific embedding layers, LoRA branches, interpolated positional embeddings, and patch-size identifiers enable varied-resolution processing.
- Dynamic patching and tokenization: Larger patches reduce the number of tokens quadratically, lowering attention cost during iterative denoising.For a 1024 × 1024 FLUX-1.Dev image, token counts decrease from 4096 to 1024 to 256 as patch size grows from p to 2p to 4p.
- Dynamic patching and tokenization: 3× computational gain results from increasing the patch size from p to 2p in FLUX-1.Dev inference.The comparison uses 50 denoising steps with a fixed patch size at every timestep.
- Dynamic patch scheduling: The method preserves the model’s perceptual output quality while dynamically reallocating computation across denoising timesteps.The approach is presented as extensible to diffusion-based image and video models.
- Dynamic patch scheduling: DDiT selects patch sizes at each timestep using a training-free scheduler based on latent evolution and spatial variation.The scheduler uses finite differences to estimate temporal changes and within-patch standard deviation to identify detail-sensitive regions.
- Dynamic patch scheduling: Higher latent acceleration indicates structural change, while higher within-patch standard deviation indicates finer-grained detail generation.These signals help distinguish when coarse patches are sufficient from when smaller patches are needed.
4. Experiments
Experiments show that DDiT dynamically allocates computation across image and video generation while preserving perceptual quality. Across quantitative, qualitative, and user-study evaluations, the method delivers substantial speedups with competitive or comparable generation quality.
- Text-to-Image Generation: DDiT preserves fine-grained details, pose, spatial layout, and overall color distribution under similar speedups on DrawBench.The qualitative comparison includes the base model, TeaCache, and TaylorSeer.
- Text-to-Image Generation: DDiT remains robust on complex prompts requiring deeper semantic understanding in qualitative DrawBench comparisons.
- Text-to-Video Generation: DDiT reduces inference time while maintaining competitive VBench video quality, preserving motion consistency and fine-grained frame details at accelerated speeds.The evaluation uses Wan-2.1 for text-to-video generation.
- Analysis: Higher-order latent differences improve FID and CLIP scores, with the third-order term achieving the best overall performance.The third-order term also produces the highest ImageReward score according to the analysis passage.
- Analysis: A user study found DDiT generations as pleasing and photorealistic as DiT 61% of the time, while DiT was preferred 22% of the time.DDiT was preferred over the DiT baseline 17% of the time, although this was not the primary goal.
- Analysis: Higher threshold values yield faster inference with a very mild dip in generation quality.The threshold controls switching between coarse and fine patch sizes during denoising.
5. Conclusion and Future Work
DDiT dynamically adapts diffusion-transformer patch sizes across denoising timesteps, preserving visual quality while reducing inference computation. Its plug-and-play design supports fast inference for DiT-based image and long-video generation.
- DDiT dynamically selects patch sizes at each denoising timestep while maintaining visual quality.The method recognizes that different timesteps do not require equally fine-grained latent representations.
- A plug-and-play LoRA adapter makes patch-embedding and de-embedding blocks support varied input patch sizes with minimal architectural changes.
- DDiT enables fast inference for DiT-based models and can generate longer videos using the same amount of compute.
- The current design uses one fixed patch size per timestep, while varying patch sizes across timesteps; within-timestep variation remains future work.