Source-linked AI summary
Efficient Text-to-Image Generation: An Adaptive Step Schedule Controller for Diffusion Models
Kuluhan Binici, Cihan Acar, Shivam Aggarwal, Siying Liu, Tulika Mitra
TL;DR
Fixed denoising schedules do not adapt to the varying needs of individual text prompts. ADSC monitors convergence to adjust schedules, and experiments report that it reduces average timestep requirements by more than 50% in some cases while maintaining visual quality.
Problem
Fixed-step denoising does not account for the different step requirements of individual prompts and can involve redundant computation or immature results.
Source: §1 · p. 1Method
ADSC monitors cosine distance between update directions as a convergence heuristic and transitions among schedules with different step sizes without additional model training.
Source: §3 · p. 2, §3.2 · p. 3Results
In some cases, ADSC reduces the average number of timesteps by more than 50% while maintaining visual quality.
Source: §1 · pp. 1–2Takeaways & Limitations
ADSC offers a way to improve time efficiency while preserving visual fidelity without additional model training.
Source: §5 · p. 5The schedule-transition rationale assumes that larger-step estimation errors have less impact on final image quality after diffusion converges and much content is generated.
Source: §3.1 · p. 2
Abstract
from arXiv · showhide
Text-to-image diffusion models often use a fixed number of denoising steps, balancing time costs and image quality. However, the optimal number of steps depends on the complexity of the input text prompt. We propose an adaptive diffusion controller that dynamically adjusts the number of steps to generate high-quality images efficiently, without additional model training. By leveraging a mixture of step schedules with varying step sizes and evaluating the error term discrepancy at each timestep, our method transitions between schedules to optimize performance. Experiments on COCO and DiffusionDB show that our approach reduces inference time while maintaining visual fidelity, offering a more efficient alternative for text-to-image diffusion models.
1. INTRODUCTION
The optimal denoising-step count varies with the text prompt and random seed, while fixed-step methods do not adapt to each generation task. ADSC monitors convergence and changes schedules, and experiments report fewer average timesteps while maintaining visual quality.
1. INTRODUCTION
- The optimal number of diffusion steps depends on the text prompt and random seed, making a fixed denoising schedule a one-size-fits-all approach.Figure 1 illustrates prompt and seed combinations; its step counts are examples, and optimal counts may vary.Source: §1 · p. 1, Figure 1 · p. 1
- ADSC monitors generation convergence at runtime, using cosine distance between conditional and unconditional update directions to adjust the schedule without additional model training.It uses smaller steps when the distance is large and larger steps as the distance narrows.Source: §1 · p. 1
- In some cases, ADSC reduces the average number of timesteps by more than 50% while maintaining visual quality.Experiments used COCO and DiffusionDB and evaluated CLIP-I and DINO metrics.Source: §1 · pp. 1–2
2. RELATED WORK
Prior work addresses diffusion guidance and inference costs through classifiers, conditioning inputs, faster solvers, compression, distillation, and stage-specific networks. The described efficiency approaches can require extra training or architecture search and still use fixed step counts.
2. RELATED WORK
- Knowledge distillation and architectural compression can reduce required diffusion steps, while Denoising Diffusion Step-aware Models use networks of varying sizes for different generation stages.These approaches are described alongside quantization and the search for effective model architectures.Source: §2 · p. 2
- DDIM, PNDM, and LMS are introduced as efficient solvers to accelerate sampling, addressing the inference cost of repeated forward passes.The passage also describes architectural compression and quantization as efficiency techniques.Source: §2 · p. 2
- Classifier-free guidance embeds guidance in the diffusion model, while internal representations and spatial inputs can control attributes and object placement.The cited spatial inputs include segmentation maps, keypoints, and bounding boxes.Source: §2 · p. 2
- The described methods can require supplementary training or architecture search, and their fixed step counts do not adapt to prompt-specific challenges.The passage identifies these as drawbacks of the efficiency techniques discussed.Source: §2 · p. 2
3. ADAPTIVE DIFFUSION STEP CONTROLLER
ADSC combines a space of schedules with different step lengths and a cosine-distance convergence criterion to adapt denoising to each generation task. As convergence is detected, it transitions from longer schedules to shorter ones with larger steps.
3.1. Heterogenous schedule control space
- ADSC uses schedules with distinct step lengths and transitions to schedules with fewer steps when convergence is flagged.The controller begins with the longest schedule; the example transitions from 40 to 20 to 10 steps. The approach presumes larger-step errors matter less after much of the image content has been generated.Source: §3.1 · p. 2, Figure 2 · p. 2
3.2. Convergence criterion
- The convergence signal is the cosine distance between unconditional and total noise estimates, which declines as diffusion progresses.The elbow, where the distance stops changing significantly, indicates that the denoised image has sufficiently converged to the text condition.Source: §3.2 · p. 3, Figure 3 · p. 3
- Exponential averaging smooths fluctuations in the cosine-distance curve before its derivatives are used to detect the elbow.Convergence is signaled when the absolute first and second derivatives of the smoothed curve fall below their respective thresholds.Source: §3.2 · p. 3
4. EXPERIMENTS
Experiments on COCO and DiffusionDB compare ADSC-controlled DDIM and PNDM with fixed-step schedules. ADSC improves reported image-quality scores in several comparisons, with results varying by scheduler and dataset.
4. EXPERIMENTS
- Experiments used Stable Diffusion with PNDM and DDIM, four schedules with halving step counts, and guidance scale 7.5; the study also compared ADSC-DDIM with DDPM and LMS.The supplied passages identify Table 3 as the comparison with other schedulers but do not report its outcomes.Source: §4 · p. 3, §4.2 · p. 5, Table 3 · p. 5
4.1. COCO results
- On COCO, ADSC-controlled DDIM averaged 18.89 steps per prompt versus 24.48 for PNDM; their step distributions differed, and DDIM used fewer steps.The distributions for both schedulers were described as similar to Gaussian distributions.Source: Figure 4 · p. 4, §4.1 · p. 4
- On COCO, ADSC-DDIM improved CLIP-I 3.3% (95.70 vs 92.40) and DINO 8.30% (92.42 vs 84.12) over fixed 19-step DDIM.The cited passage also reports gains for ADSC-controlled PNDM over its fixed-step counterpart.Source: Table 1 · p. 4, §4.1 · p. 4
- ADSC latency was nearly identical to fixed schedulers: DDIM took 0.79 seconds at 18.89 steps, matching fixed 19-step DDIM.COCO examples also suggest that prompts describing intricate details required more steps than prompts describing plainer objects.Source: §4.1 · p. 4, Figure 5 · p. 4
4.2. DiffusionDB results
- On DiffusionDB, ADSC and fixed schedules both had higher CLIP-T scores than in COCO; ADSC-controlled DDIM retained significant CLIP-I and DINO gains.The reported PNDM results were mixed across reference-image metrics.Source: §4.2 · p. 5
- On DiffusionDB, ADSC-DDIM improved CLIP-I 3.2% (95.10 vs. 91.90) and DINO 6.4% (89.36 vs. 82.96), while PNDM showed no noticeable CLIP-I gain and a 1.59% DINO drop.The authors report that ADSC did not significantly degrade PNDM performance and characterize it as more consistently beneficial with DDIM.Source: Table 2 · p. 5, §4.2 · p. 5
- In DiffusionDB examples, ADSC images were described as high quality, while fixed schedules showed prompt misalignment and visual defects.The examples include missing prompt details or color information in some fixed-step images.Source: Figure 7 · p. 5, §4.2 · p. 5
5. CONCLUSION
ADSC adjusts denoising steps to prompt characteristics to reduce computation while preserving visual fidelity, without additional model training. Manual hyperparameter tuning remains a limitation.
5. CONCLUSION
- ADSC dynamically adjusts denoising steps to each text prompt’s characteristics to improve efficiency without compromising significant visual quality.The method requires no additional model training.Source: §5 · p. 5
- Experiments show that ADSC reduces the average number of diffusion steps while preserving visual fidelity.Source: §5 · p. 5
- Manual tuning of hyperparameters such as gradient thresholds can be time-consuming; future work could explore adaptive thresholds and learned convergence criteria.Source: §5 · p. 5