PaperlayerView exact revision on arXiv

Source-linked AI summary

AdaDiff: Adaptive Step Selection for Fast Diffusion Models

Hui Zhang, Zuxuan Wu, Zhen Xing, Jie Shao, Yu-Gang Jiang

arXiv:2311.14768v2cs.CVcs.AI

TL;DR

Fixed-step acceleration overlooks variation in prompt richness, although prompt complexity affects how many denoising steps may be needed. AdaDiff learns instance-specific step policies using policy gradients and a reward balancing quality with compute. Across image and video benchmarks, it reduces inference time while maintaining similar visual quality to a fixed-step baseline.

  • Problem

    Fixed-step acceleration applies the same denoising-step count without accounting for prompt richness, although simpler and more complex prompts may need different counts.

    Source: Introduction · pp. 1–2

  • Method

    AdaDiff trains a lightweight prompt-conditioned step-selection network with policy gradients and a reward balancing image quality against computational cost.

    Source: Introduction · p. 2

  • Results

    Across image and video benchmarks, AdaDiff saves 33% to 40% of inference time versus a fixed-step baseline while maintaining similar visual quality.

    Source: Introduction · p. 2

  • Takeaways & Limitations

    AdaDiff can be combined with existing acceleration methods and allocates fewer steps to less informative prompts than to informative prompts.

    Source: Introduction · p. 2, Conclusion · p. 7

Abstract

from arXiv · show

Diffusion models, as a type of generative model, have achieved impressive results in generating images and videos conditioned on textual conditions. However, the generation process of diffusion models involves denoising dozens of steps to produce photorealistic images/videos, which is computationally expensive. Unlike previous methods that design ``one-size-fits-all'' approaches for speed up, we argue denoising steps should be sample-specific conditioned on the richness of input texts. To this end, we introduce AdaDiff, a lightweight framework designed to learn instance-specific step usage policies, which are then used by the diffusion model for generation. AdaDiff is optimized using a policy gradient method to maximize a carefully designed reward function, balancing inference time and generation quality. We conduct experiments on three image generation and two video generation benchmarks and demonstrate that our approach achieves similar visual quality compared to the baseline using a fixed 50 denoising steps while reducing inference time by at least 33%, going as high as 40%. Furthermore, our method can be used on top of other acceleration methods to provide further speed benefits. Lastly, qualitative analysis shows that AdaDiff allocates more steps to more informative prompts and fewer steps to simpler prompts.

Introduction

Fixed-step denoising does not account for how prompt complexity varies, even though simpler and more descriptive prompts may need different numbers of steps. AdaDiff learns prompt-conditioned step policies to balance generation quality and computational cost, reducing inference time while maintaining similar visual quality.

Introduction

  • Prompt richness varies: simple prompts may need fewer steps, while prompts with many objects, detailed descriptions, and intricate interactions may need more.Existing acceleration methods typically use the same number of steps regardless of prompt complexity.Source: Introduction · pp. 1–2
  • A lightweight step-selection network learns a step-usage policy with policy gradients, then guides dynamic sampling by a pretrained diffusion model.The reward balances high-quality generation against computational resources.Source: Introduction · p. 2
  • AdaDiff saves 33% to 40% of inference time versus a fixed-step baseline while maintaining similar visual quality across image and video benchmarks.The learned policy also allocates fewer steps to less informative prompts and more to informative prompts.Source: Introduction · p. 2
  • AdaDiff also combines with existing acceleration methods, transfers policies across datasets, and assigns steps according to prompt informativeness.These behaviors are reported alongside the benchmark comparison.Source: Introduction · p. 2

Related Work

Prior work addresses diffusion-model acceleration by reducing sampling steps or computation per step, while reinforcement learning has been used to align generated outputs with human preferences. AdaDiff applies reinforcement learning to instance-specific step policies and is complementary to existing acceleration approaches.

Related Work

  • Diffusion generation spans images, video, and restoration, but its iterative process demands substantial computation and inference time.The cited discussion also notes conditioning on text, semantic maps, representations, and images.Source: Related Work · p. 2
  • Prior reinforcement-learning work fine-tunes generative models for human preferences, whereas AdaDiff uses it to learn instance-specific step policies for acceleration.The paper identifies this as the first use of reinforcement learning for image- and video-generation acceleration through such policies.Source: Related Work · p. 2
  • Existing acceleration methods reduce sampling steps or computation per step, but retain manually set step counts; AdaDiff dynamically selects steps per input and complements them.The two paradigms include fewer-step samplers or distillation, and pruning, distillation, or reducing redundant computation.Source: Related Work · p. 2

Methodology

AdaDiff adapts diffusion sampling steps to each prompt, using a learned policy and a reward that balances image quality against inference time. The image and video variants use this adaptive selection during generation.

Background on diffusion models

  • Latent diffusion models add noise in a forward process and learn to reverse it; DDIM sampling uses a predetermined step count, with Stable Diffusion and ModelScopeT2V defaulting to 50 steps.AdaDiff applies dynamic step selection to latent diffusion models.Source: Background on diffusion models · p. 3

Adaptive step selection for image generation

  • For image generation, a text encoder extracts prompt features, and a self-attention selection network maps them to probabilities over ten DDIM step choices from 5 to 50.A one-hot categorical policy represents the selected step count.Source: Adaptive step selection for image… · p. 3
  • The image-generation reward balances image quality with normalized steps saved, and penalizes outputs whose quality is low.Quality is assessed for each prompt across the ten step choices; the top three scores count as high quality.Source: Adaptive step selection for image… · p. 4
  • A policy-gradient method trains the selection network to maximize expected reward; at inference, the highest-probability choice determines each prompt’s step count.The gradient is approximated using mini-batches, and the network is trained with Adam.Source: Adaptive step selection for image… · p. 4

Adaptive step selection for video generation

  • For video generation, AdaDiff scores each frame with the image-quality model, averages the frame scores for the quality reward, and reuses the image method’s step reward.The overall reward is obtained from the image-generation reward formulation.Source: Adaptive step selection for video… · p. 4

Experiments

Experiments test AdaDiff across image and video generation, other acceleration methods, and transferred policies. They find that prompt-dependent step allocation can reduce generation time while maintaining or improving quality, with reward settings controlling the speed–quality trade-off.

Experimental Details

  • Evaluation covers three image and two video datasets; the 25.71M-parameter, 1.93-GFLOP step selector was trained for 16 to 80 A100 GPU hours.Image generation uses SD-v2.1-base, SDXL-Turbo, and SDXL-v1.0; video generation uses ModelScopeT2V.Source: Experimental Details · pp. 4–5

Main Results

  • 39.7% speed increase with similar quality across five image metrics versus fixed 50-step SD; at similar speeds, AdaDiff beats fixed 28-step SD and random or manually crafted policies.Results are averaged over five COCO-2017 runs.Source: Table 1 · p. 5, Main Results · p. 5
  • 35.8% lower generation time per video (13.6 vs 21.2) and enhanced quality versus fixed 50-step ModelScope; at comparable resources, AdaDiff beats fixed 31-step and random or manual policies across all metrics.Source: Table 2 · p. 5, Main Results · p. 5
  • At similar speed, AdaDiff outperforms DPMSolver in image quality and achieves better performance than SDXL-Lightning with 80 vs 1000+ A100 GPU hours.Against pruning and computation-saving methods including SSD-1B and DeepCache, it also achieves better image quality at similar speeds.Source: Table 3 · p. 5, Main Results · p. 5
  • 52.7% speed up versus fixed 5-step SDXL-Turbo with comparable performance; on SSD-1B, 41.1% speed up versus fixed 50 steps while outperforming fixed 30-step and other policies at similar speeds.Source: Main Results · pp. 5–6, Table 4 · p. 6, Table 5 · p. 6
  • 39.2% and 33.8% less generation time on Laion-COCO and MSR-VTT, respectively, with comparable quality when policies transfer from COCO-2017 and InternVid.Source: Table 6 · p. 6, Main Results · p. 6
  • Policies assign more steps as prompt word and object counts grow; examples use 10–20 steps for easy prompts, about 30 for medium, and 40–50 for hard prompts.Source: Main Results · p. 6, Figure 3 · p. 6

Discussion

  • IQS improves policy effectiveness over CLIP-only or NIQE-only rewards; relative top-k selection is more efficient than an absolute threshold, while smaller k and larger λ increase quality and average steps.Source: Discussion · pp. 6–7, Table 7 · p. 7, Figure 5 · p. 7

Conclusion

AdaDiff derives prompt-specific step-usage policies for efficient image and video generation, balancing generation quality against computational cost. Experiments support the learned policies, and the method can be used with other acceleration methods.

  • AdaDiff derives adaptive step-usage policies tailored to each prompt for efficient image and video generation.Source: Conclusion · p. 7
  • A step-selection network trained with policy-gradient methods generates policies that balance generation quality and computational cost.Source: Conclusion · p. 7
  • Experiments support strong per-input step-usage policies, and AdaDiff can be used on top of other acceleration methods for further speed benefits.Source: Conclusion · p. 7

More Experimental Details

The evaluation discusses limitations of CLIP Score and NIQE Score for assessing generated-image quality, and presents IQS Score as considering both text-image alignment and perceptual fidelity.

More Results

On Laion-COCO and DiffusionDB, AdaDiff reports similar performance to fixed 50-step Stable Diffusion with faster generation. At comparable speeds, it outperforms shorter fixed-step and alternative policies, while qualitative examples show step allocation varying with prompt richness.

More Results

  • 33.9% faster on Laion-COCO and 33.3% faster on DiffusionDB than fixed 50-step SD, with comparable performance across five image-quality metrics on each benchmark.AdaDiff uses an average of 31.34 sampling steps on Laion-COCO and 32.38 on DiffusionDB.Source: Table 8 · p. 8, Table 9 · p. 8, More Results · p. 8
  • At comparable speeds, AdaDiff significantly surpasses fixed 31-step SD on all five image-quality metrics; on DiffusionDB, it averages 32.38 steps and outperforms fixed 32-step SD, with 33.3% faster sampling than fixed 50-step SD.Source: More Results · p. 8
  • On DiffusionDB, AdaDiff also outperforms random, heuristic, and perplexity policies in image quality; on Laion-COCO, it has clear advantages over those policies.Source: More Results · p. 8
  • Figure 7 examples show AdaDiff assigning sample-specific denoising steps according to prompt richness while maintaining high-quality images and generating faster than fixed 50-step Stable Diffusion.Source: More Results · p. 8

Related papers

Loading 2311.14768v2…