PaperlayerView exact revision on arXiv

Source-linked AI summary

DeepCache: Accelerating Diffusion Models for Free

Xinyin Ma, Gongfan Fang, Xinchao Wang

arXiv:2312.00858v2cs.CVcs.AI

TL;DR

Diffusion models are computationally expensive because denoising is sequential, while conventional compression often requires costly retraining. DeepCache is a training-free caching approach that reuses temporally consistent high-level U-Net features and updates low-level features cheaply, achieving reported acceleration with limited quality degradation and favorable comparisons with retraining-based methods.

  • Problem

    Diffusion inference is slow because reverse denoising is sequential, while prior compression methods often require substantial retraining or additional pretrained subnetworks.

  • Method

    DeepCache caches temporally consistent high-level features across denoising steps and uses U-Net structure to update low-level features with partial computation.

  • Results

    DeepCache demonstrates superior efficacy to retraining-based pruning and distillation under the same throughput, with comparable or slightly superior generation capabilities reported across tested settings.

  • Takeaways & Limitations

    The method provides a training-free perspective for accelerating diffusion models while remaining compatible with existing fast samplers.

  • Takeaways & Limitations

    Speedup is constrained when the shallowest skip branch contains substantial computation, and larger caching intervals such as N=20 can cause non-negligible performance degradation.

Abstract

from arXiv · show

Diffusion models have recently gained unprecedented attention in the field of image synthesis due to their remarkable generative capabilities. Notwithstanding their prowess, these models often incur substantial computational costs, primarily attributed to the sequential denoising process and cumbersome model size. Traditional methods for compressing diffusion models typically involve extensive retraining, presenting cost and feasibility challenges. In this paper, we introduce DeepCache, a novel training-free paradigm that accelerates diffusion models from the perspective of model architecture. DeepCache capitalizes on the inherent temporal redundancy observed in the sequential denoising steps of diffusion models, which caches and retrieves features across adjacent denoising stages, thereby curtailing redundant computations. Utilizing the property of the U-Net, we reuse the high-level features while updating the low-level features in a very cheap way. This innovative strategy, in turn, enables a speedup factor of 2.3$\times$ for Stable Diffusion v1.5 with only a 0.05 decline in CLIP Score, and 4.1$\times$ for LDM-4-G with a slight decrease of 0.22 in FID on ImageNet. Our experiments also demonstrate DeepCache's superiority over existing pruning and distillation methods that necessitate retraining and its compatibility with current sampling techniques. Furthermore, we find that under the same throughput, DeepCache effectively achieves comparable or even marginally improved results with DDIM or PLMS. The code is available at https://github.com/horseee/DeepCache

1. Introduction

Diffusion models are powerful but slow because reverse denoising is sequential and existing compression approaches often require retraining. DeepCache addresses this by caching temporally consistent high-level features for training-free acceleration.

  • Sequential reverse denoising limits diffusion-model inference speed and parallel decoding.
  • Prior compression methods reduce computation through architectural redesign, pruning, adaptive models, or other strategies, but commonly require retraining or pretrained model collections.
  • DeepCache targets per-step computational overhead without additional training by exploiting temporal consistency in high-level features.
  • The method caches high-level features while cheaply updating low-level features using U-Net structure.
  • DeepCache dynamically compresses diffusion models at runtime and reduces redundant calculations through cached features, including a non-uniform 1:N strategy.
  • Experiments across datasets, models, and samplers report stronger efficacy than retraining-based pruning and distillation methods under the same throughput.

2. Related Work

Related work accelerates diffusion inference either by reducing sampling steps or by reducing computation within each step. These approaches include optimized samplers, distillation, pruning, quantization, adaptive model selection, and architectural redesign.

  • Optimized Sampling Efficiency: Sampling-efficiency methods reduce the number of denoising steps through non-Markovian processes, fast SDE or ODE solvers, distillation, consistency models, or parallel sampling.
  • Optimized Structural Efficiency: Structural-efficiency methods reduce inference time per sampling step through pruning, evolving architectures, frequency-aware designs, early stopping, or quantization.
  • Optimized Structural Efficiency: Some methods select different models at different steps from a diffusion model zoo instead of using one uniform model throughout sampling.

3. Methodology

DeepCache exploits temporal similarity in high-level U-Net features across denoising steps, reusing cached features while retaining cheap low-level updates. It extends one-step caching to 1:N inference and adapts sampling density when similarity varies.

  • 3.1. Preliminary: Reverse diffusion starts from noise and iteratively denoises through timestep-conditioned network evaluations.
  • 3.1. Preliminary: U-Net combines high-level main-branch features with lower-level skip-branch features through downsampling and upsampling blocks.
  • 3.2. Feature Redundancy in Sequential Denoising: Adjacent denoising steps show significant temporal similarity in high-level features, with at least 10% of adjacent timesteps exceeding 0.95 similarity across tested models.
  • 3.3. Deep Cache For Diffusion Models: DeepCache stores slowly evolving features and retrieves them in later steps to eliminate redundant computations.
  • 3.3. Deep Cache For Diffusion Models: The method exploits U-Net skip connections by recomputing shallow low-level paths while retrieving expensive main-branch features.
  • 3.3. Deep Cache For Diffusion Models: At timestep t−1, DeepCache reuses high-level features computed at t and performs partial inference instead of executing the entire U-Net.
  • 3.3. Deep Cache For Diffusion Models: Cached features can be reused across N−1 consecutive steps after one full inference step in a 1:N schedule.
  • 3.3. Deep Cache For Diffusion Models: When feature similarity declines, non-uniform inference samples more densely around steps with lower similarity; its index frequency varies quadratically around a chosen center timestep.

4. Experiment

DeepCache is evaluated across DDPM, LDM, and Stable Diffusion using established datasets, sampling methods, and compression baselines. It accelerates inference while preserving generation quality and remains competitive with fast samplers under comparable throughput.

  • Experimental Settings: Experiments cover DDPM, LDM, and Stable Diffusion across CIFAR10, LSUN, ImageNet, MS-COCO 2017, and PartiPrompts.The evaluations use 100-step DDIM for DDPM, 250-step DDIM for LDM, and 50-step PLMS for Stable Diffusion.
  • Complexity Analysis: DeepCache partitions the U-Net by skip connections, removing layers during selected denoising steps to reduce inference computation.The reported MACs measure the average computation across all steps on a single RTX2080 GPU.
  • Comparison with Compression Methods: 4.1× acceleration on LDM-4-G for ImageNet changes FID from 3.39 to 3.59, while outperforming pruning and distillation methods on FID and sFID.The comparison uses 250 DDIM steps, and non-uniform caching improves quality as the caching interval increases.
  • Comparison with Compression Methods: DeepCache surpasses retraining-based methods on CIFAR-10 and LSUN under similar MACs constraints, with a more significant acceleration ratio attributed to hardware-friendly layer pruning.Table 2 evaluates CIFAR-10, LSUN-Bedroom, and LSUN-Churches using 100 DDIM steps.
  • Comparison with Compression Methods: DeepCache outperforms all three BK-SDM variants at faster denoising speeds, while its generated images remain more consistent with the original model and textual prompts.Stable Diffusion comparisons use 512-resolution images and 50-step PLMS.
  • Comparison with Fast Sampler: DeepCache achieves slightly better results than 25-step PLMS on Stable Diffusion and comparable results to DDIM on LDM-4-G at similar throughput.The method is additive to fast samplers rather than replacing them.
  • Analysis: Without retraining, cached features support effective denoising with a shallow U-Net, while shallow-network inference improves 50-step DDIM by 0.32 and 10-step DDIM by 2.98.The ablation also examines increasing caching intervals; with N < 5, image-quality reduction is slight.

5. Limitations

DeepCache is constrained by the pre-defined structure of the pre-trained diffusion model, and larger caching intervals can degrade performance while limiting maximum acceleration.

  • A shallowest skip branch containing a substantial share of computation constrains the achievable speedup.The paper gives a branch encompassing 50% of the model as an example.
  • Larger caching steps, such as N=20, cause non-negligible performance degradation and may constrain the upper acceleration limit.

6. Conclusion

DeepCache accelerates diffusion models by exploiting temporal similarity in high-level features and U-Net structure to reduce redundant computation. Evaluations report faster inference, stronger results than parameter-reduction methods, and quality comparable to or slightly better than DDIM and PLMS.

  • DeepCache accelerates diffusion models by reducing redundant high-level feature computation across adjacent denoising steps.It updates low-level features through U-Net structure.
  • DeepCache outperforms compression methods focused on reducing parameter size across evaluations on several datasets and diffusion models.
  • DeepCache achieves comparable or slightly superior generation quality to DDIM and PLMS.

Supplementary Material

Algorithm 1 generates samples over N steps using one full model inference followed by N-1 partial inferences. It caches U-Net features during a cache step and retrieves selected upsampling features in later steps.

  • The algorithm performs one complete model inference and N-1 partial model inferences over N generation steps.DDPM sampling is used as the example, and the algorithm is adaptable to other fast samplers.
  • The inputs include a U-Net with down-sampling and middle blocks, a caching interval N, a branch index m, and the current output x_t.
  • During the cache step, the method computes the model prediction and next output while storing down-sampling features h_i and up-sampling features u_i.
  • For later partial inferences, the algorithm retrieves cached u_i+1 features while iterating through the up-sampling blocks from branch m toward 1.

B. Varying Hyper-parameters in Non-uniform 1:N Strategy

The non-uniform 1:N strategy uses center c and power p to select timesteps for full inference. Image quality initially improves as these parameters increase, then declines, with effects depending on caching interval and dataset.

  • B. Varying Hyper-parameters in Non-uniform 1:N Strategy: The non-uniform 1:N strategy determines full-inference timesteps using center c and power p.
  • B. Varying Hyper-parameters in Non-uniform 1:N Strategy: Increasing c and p initially improves generated image quality before a subsequent decline.The experiments on LDM-4-G report this trend with caching interval 20.
  • B. Varying Hyper-parameters in Non-uniform 1:N Strategy: Optimal parameter values vary slightly across datasets, but most center around timestep 15 with power approximately 1.4.
  • C. Non-uniform 1:N v.s. Uniform 1:N: For LDM-4-G on ImageNet, non-uniform caching improves FID from 8.23 to 7.36 at N=20.
  • C. Non-uniform 1:N v.s. Uniform 1:N: For N<5, non-uniform caching does not improve image quality and can slightly degrade it, with FID increasing from 3.39 to 3.46 at N=2.

D. Varying Skip Branches

Varying DeepCache’s skip branch exposes a model-size trade-off between inference speed and image fidelity. The experiments use a uniform 1:5 strategy and evaluate generated samples across datasets and prompt sets.

  • Skip-branch trade-offs: The choice of skip branch produces significant performance variation, particularly for the 6th branch.The results show that branch selection changes both speed and image quality.
  • Experimental setup: Table 12 evaluates skip-branch effects under a uniform 1:5 DeepCache strategy.The sampling process uses DDIM with 100 steps.
  • Evaluation coverage: Stable Diffusion v1.5 results are reported on PartiPrompt and the MS-COCO 2017 validation set, alongside generated-sample comparisons for LSUN-Churches.The figures and tables cover both quantitative and visual evaluations.

F. Detailed Results for Stable Diffusion

Detailed Stable Diffusion experiments examine how DeepCache behaves across cache intervals and sampling configurations. For N < 5, image content changes minimally and CLIP Score varies only slightly.

  • Experimental setup: The experiments evaluate PLMS across different steps because no definitive N aligns DeepCache throughput with PLMS.The authors therefore compare multiple N values and sampling-step settings.
  • Cache interval: For N < 5, image content shows minimal variation while CLIP Score changes only slightly.This supports evaluating several N values when matching DeepCache throughput with PLMS.

G. More Samples for Each Dataset

The paper supplements quantitative evaluations with generated samples across models and datasets. The visual comparisons pair baseline sampling with DeepCache at specified cache intervals and report the resulting speedups.

  • Stable Diffusion v1.5: 2.15× speedup is reported for Stable Diffusion v1.5 using 50 PLMS steps with DeepCache at N=5.The prompts are drawn from the MS-COCO 2017 validation set.
  • LDM-4-G: 6.96× speedup is reported for LDM-4-G on ImageNet using DDIM-250 with DeepCache at N=10.The figure compares DDIM-250 samples with and without DeepCache.
  • DDPM datasets: 1.48× speedup is reported for DDPM on LSUN-Bedroom and LSUN-Churches using DDIM-100 with DeepCache at N=5.The figures compare baseline DDIM-100 samples with DeepCache samples.
Loading 2312.00858v2…