Source-linked AI summary
Cache Me if You Can: Accelerating Diffusion Models through Block Caching
Felix Wimbauer, Bichen Wu, Edgar Schoenfeld, Xiaoliang Dai, Ji Hou, Zijian He, Artsiom Sanakoyeu, Peizhao Zhang, Sam Tsai, Jonas Kohler, Christian Rupprecht, Daniel Cremers, Peter Vajda, Jialiang Wang
TL;DR
Diffusion inference is expensive, and existing acceleration methods often treat the denoising network as a black box. The paper analyzes layer behavior, reuses slowly changing block outputs with automatically derived schedules, and reports better image quality at the same computational cost. Across LDM and EMU models with DDIM and DPM solvers, block caching improves FID and human preference while enabling more steps at fixed latency.
Problem
Diffusion generation repeatedly evaluates a large denoising network, while many acceleration methods reduce steps without analyzing the network’s internal computations.
Method
Block caching reuses previous outputs from slowly changing U-Net blocks, uses scale-shift alignment, and automatically derives schedules from block changes over timesteps.
Results
Across LDM and EMU with DDIM and DPM, caching enables more steps at fixed latency, improves FID, and is preferred in independent human evaluations.
Takeaways & Limitations
Analyzing internal denoising-network behavior exposes reusable computations that can improve image quality under a fixed computational budget.
Takeaways & Limitations
Scale-shift adjustment can sometimes change image identity more than naive caching or reducing steps, and automatic threshold selection is sensitive to schedule changes.
Abstract
from arXiv · showhide
Diffusion models have recently revolutionized the field of image synthesis due to their ability to generate photorealistic images. However, one of the major drawbacks of diffusion models is that the image generation process is costly. A large image-to-image network has to be applied many times to iteratively refine an image from random noise. While many recent works propose techniques to reduce the number of required steps, they generally treat the underlying denoising network as a black box. In this work, we investigate the behavior of the layers within the network and find that 1) the layers' output changes smoothly over time, 2) the layers show distinct patterns of change, and 3) the change from step to step is often very small. We hypothesize that many layer computations in the denoising network are redundant. Leveraging this, we introduce block caching, in which we reuse outputs from layer blocks of previous steps to speed up inference. Furthermore, we propose a technique to automatically determine caching schedules based on each block's changes over timesteps. In our experiments, we show through FID, human evaluation and qualitative analysis that Block Caching allows to generate images with higher visual quality at the same computational cost. We demonstrate this for different state-of-the-art models (LDM and EMU) and solvers (DDIM and DPM).
1. Introduction
Diffusion models offer high-quality image synthesis but remain costly because large denoising networks are repeatedly evaluated. This paper examines internal block behavior and introduces caching to reduce redundant computation while improving quality at fixed latency.
- Motivation: Repeated evaluation of large denoising networks creates high latency and computational cost, limiting applications requiring fast and frequent inference.Faster inference is described as important for making large-scale image generation economically and technically viable.
- Motivation: Existing acceleration methods mainly reduce denoising steps or distill models while treating the U-Net itself as a black box.These approaches primarily change how the network output is used rather than analyzing internal computations.
- Analysis: Attention blocks change smoothly, exhibit position-specific patterns consistent across text inputs, and usually change only slightly between steps.Because attention blocks are among the most computationally expensive components, their temporal redundancy is a promising latency-reduction target.
- Contribution: Block caching reuses slowly changing layer-block outputs, adds scale-shift alignment to prevent feature-misalignment artifacts, and automatically derives caching schedules.The method targets redundant computation while adapting reuse decisions to block changes over timesteps.
- Results: Across LDM and EMU models with DDIM and DPM solvers, caching enables more steps at fixed latency, improves FID, and receives preference in independent human evaluations.The experiments cover both model families and both solver settings described in the introduction.
2. Related Work
Prior work accelerates diffusion generation through model architectures, improved solvers, and distillation. The paper situates block caching among these approaches while emphasizing internal reuse within the denoising network.
- Text-to-Image Models: Text-to-image research spans GAN-based methods, self-attention, large architectures, autoencoder-based methods, and generative transformers.These approaches target image synthesis quality and capability through different model families and architectural choices.
- Improved Solvers: Diffusion solvers determine the sampling strategy and strongly influence how many denoising steps are needed for high-quality output.DDIM combines DDPM steps without retraining, while DPM-Solver models denoising as an ordinary differential equation.
- Distillation: Distillation creates student models or procedures requiring fewer neural function evaluations than a pretrained teacher.Examples include guidance distillation, progressive distillation, consistency models, and one-step student architectures.
- Distillation: Distillation can require retraining and may not support negative or composite prompts, motivating alternative lightweight acceleration methods.The cited discussion contrasts these costs and restrictions with methods that preserve the original model parameters.
3. Method
The method analyzes how denoising-network blocks change across diffusion steps and reuses block outputs when their changes are small. It combines automatically scheduled block caching with scale-shift adjustment to reduce redundant computation while preserving image quality.
- Block Analysis: The analysis finds that block outputs change smoothly, follow position-dependent temporal patterns, and often differ only slightly between consecutive steps.Higher-resolution blocks change most near the end of denoising, while deeper lower-resolution blocks change more near the beginning.
- Block Caching: Block caching reuses outputs from previous steps instead of recomputing every U-Net block at every step.Residual connections allow caching at the block level without otherwise disrupting information flow.
- Block Caching: Block caching saves computation selectively while preserving steps that require greater precision.This provides finer-grained control over computation than approaches that reduce the total number of denoising steps.
- Automatic Cache Schedule: Caching schedules retain a block’s cached value until accumulated change exceeds threshold δ, then recompute its output.A higher threshold extends cache lifetimes and increases speed but can affect image appearance more.
- Scale-Shift Adjustment: Scale-shift adjustment adds timestep-dependent scale and shift parameters to cached-input layers to address artifacts from feature-map misalignment.The parameters are optimized while the original model parameters remain frozen.
- Scale-Shift Adjustment: With scale-shift adjustment, the additional multiplication and addition have no noticeable speed effect while improving image quality.Qualitative results report richer colors and finer details for LDM-512, and artifacts visible with naive caching are avoided.
4. Experiments
Experiments across LDM-512 and EMU-768, using DDIM and DPM, show that block caching can trade redundant computation for either faster inference or better quality at comparable latency. The results also identify threshold and block-type choices that affect this trade-off.
- Experimental setup: The experiments use LDM-512 and EMU-768 with controlled inference settings, including bfloat16 inference and latency measurements on a single Nvidia A100 GPU.LDM-512 has 900M parameters, while EMU-768 has 2.7B parameters.
- Overall results: Across LDM and EMU models with DDIM and DPM solvers, caching enables more denoising steps at fixed computational cost and improves image quality.The paper evaluates both FID and human preferences across these model–solver combinations.
- LDM-512: Caching 20 steps improves FID over the 14-step baseline while being slightly faster for both DPM and DDIM; caching 50 steps outperforms the 30-step baseline at comparable latency.Scale-shift adjustment further enhances the results.
- Qualitative results: The full caching and scale-shift configuration produces crisper, more vibrant, and more detailed images than the 14-step baseline.Scale-shift adjustment mitigates local artifacts observed with caching alone.
- Cache threshold: A threshold of δ = 0.5 provides a 1.5× speedup while improving image quality, whereas more aggressive caching eventually deteriorates quality.Higher δ values lengthen cache lifetimes and reduce recomputation frequency.
- Block selection: Caching ResBlocks causes significant quality deterioration for only a 5% speedup, motivating the choice to cache SpatialTransformer blocks instead.ResBlocks change less smoothly and are important for local image details.
5. Conclusion
The paper analyzes diffusion U-Nets rather than treating them as black boxes and uses those observations to introduce Block Caching. Across models, solvers, and step settings, the method speeds inference while retaining or improving image quality relative to matching-compute baselines.
- Conclusion: Block Caching reduces redundant diffusion-model computations and speeds image generation by 1.5×–1.8× with minimal image-quality loss.The method is evaluated on LDM and EMU models ranging from 900M to 2.7B parameters.
- Conclusion: Compared with naively reducing solver steps to match compute, Block Caching produces more vibrant images with more fine-grained details.The findings are supported by FID measurements and human evaluation.
Supplementary Material
The supplementary material organizes additional discussion of future work, limitations, ethical considerations, safety, and qualitative and analytical figures.
- Supplementary Material: The supplementary material contains future-work discussion, method limitations, ethical and safety considerations, and additional figures.The figures include qualitative results, change-metric plots, and caching schedules.
A. Future Work
The authors identify step-to-step change metrics and scale-shift adjustment as potential directions beyond the current caching method.
- Future Work: Step-to-step change metrics may also support the design of better network architectures or noise schedules.The passage presents this as a future-work direction rather than an evaluated result.
- Future Work: The significant effect of scale-shift adjustment on image structure and visual appeal suggests possible use in human-in-the-loop finetuning.The proposed extension is intended to improve adherence to text prompts.
B. Limitations
The method has two noteworthy weaknesses: scale-shift adjustment can alter image identity, and automatic schedule configuration is sensitive to the caching threshold.
- Scale-shift adjustment can sometimes change the image identity more than reducing denoising steps or using naive caching.
- Finding the optimal auto-configuration threshold can take time because the model is sensitive to caching-schedule changes.The authors recommend testing small variations of the desired threshold.
C. Ethical Considerations & Safety
The authors report no new ethical or legal challenges beyond those of the underlying model, but note that scale-shift adjustment can slightly change image identities and may require additional safety checks.
- The technique introduces no new image data, and its scale-shift optimization requires only prompts.The authors therefore believe it introduces no ethical or legal challenges beyond the model to which it is applied.
- Scale-shift adjustment can slightly change image identities while still following the prompt.
- Slight identity changes may make an additional safety check necessary when deploying models with block caching.
D. Additional Figures
The supplementary material provides qualitative results, layer-change plots, automatically derived caching schedules, and an overview across the evaluated configurations.
- Qualitative Results: Additional qualitative figures compare caching with and without scale-shift adjustment against slower and same-latency baselines.The same-latency baseline reduces the number of denoising steps.
- Change Plots: Additional change plots show step-to-step changes for each layer block, averaged over 32 forward passes and two random seeds using L1rel.
- Caching Schedules: Additional caching-schedule figures show schedules automatically derived from the measured layer-block changes.
- Figure Overview: The overview categorizes supplementary figures as qualitative results, change metric plots, or caching schedules.
- Configurations: The final schedule figures complete the supplementary coverage for LDM-512 under DPM and DDIM with 20 and 50 steps.