PaperlayerView exact revision on arXiv

Source-linked AI summary

SenCache: Accelerating Diffusion Model Inference via Sensitivity-Aware Caching

Yasaman Haghighi, Alexandre Alahi

arXiv:2602.24208v1cs.CVcs.LG

TL;DR

Diffusion inference is expensive because it repeatedly evaluates large networks, and existing training-free caching methods rely on heuristic reuse rules. SenCache analyzes sensitivity to noisy-latent and timestep perturbations, then makes dynamic per-sample cache decisions using a first-order approximation. Across Wan 2.1, CogVideoX, and LTX-Video, it achieves better visual quality than existing caching strategies under similar computational budgets.

  • Problem

    Diffusion inference requires many costly denoising iterations, while existing caching methods use heuristic reuse criteria that require extensive tuning.

  • Method

    SenCache predicts denoiser output change from local sensitivity to noisy-latent and timestep perturbations and reuses cached outputs only below a target tolerance.

  • Results

    SenCache achieves better visual quality than existing caching strategies under similar computational budgets on Wan 2.1, CogVideoX, and LTX-Video.

  • Takeaways & Limitations

    SenCache provides a theoretically motivated, dynamic, per-sample caching rule with an explicit tolerance for the speed–quality trade-off and no additional training or model modification.

  • Takeaways & Limitations

    The implementation uses a first-order sensitivity surrogate and a fixed threshold ε; richer estimators and dynamically scheduled thresholds remain future work.

Abstract

from arXiv · show

Diffusion models achieve state-of-the-art video generation quality, but their inference remains expensive due to the large number of sequential denoising steps. This has motivated a growing line of research on accelerating diffusion inference. Among training-free acceleration methods, caching reduces computation by reusing previously computed model outputs across timesteps. Existing caching methods rely on heuristic criteria to choose cache/reuse timesteps and require extensive tuning. We address this limitation with a principled sensitivity-aware caching framework. Specifically, we formalize the caching error through an analysis of the model output sensitivity to perturbations in the denoising inputs, i.e., the noisy latent and the timestep, and show that this sensitivity is a key predictor of caching error. Based on this analysis, we propose Sensitivity-Aware Caching (SenCache), a dynamic caching policy that adaptively selects caching timesteps on a per-sample basis. Our framework provides a theoretical basis for adaptive caching, explains why prior empirical heuristics can be partially effective, and extends them to a dynamic, sample-specific approach. Experiments on Wan 2.1, CogVideoX, and LTX-Video show that SenCache achieves better visual quality than existing caching methods under similar computational budgets.

1. Introduction

Diffusion inference is costly because video models repeatedly run large networks, while existing caching methods use heuristic reuse rules. SenCache instead grounds dynamic, per-sample cache decisions in denoiser sensitivity and achieves better visual quality than prior caching methods at similar compute.

  • Video diffusion inference is expensive because generation requires many denoising iterations, each involving a full forward pass of a large network.The cost is particularly severe for video diffusion transformers with billions of parameters.
  • Caching reduces inference cost by reusing denoiser outputs, but prior methods choose reuse timesteps with empirical heuristics.These methods require extensive tuning and lack an explicit sensitivity-based decision rule.
  • Local sensitivity to noisy-latent and timestep perturbations predicts how much the denoiser output changes between neighboring steps.The sensitivities quantify the effects of latent drift and timestep spacing on cache-reuse error.
  • Both latent and timestep sensitivities contribute significantly to caching error, exposing a limitation of heuristics that do not model both variation sources.
  • SenCache uses a first-order sensitivity approximation to make dynamic, per-sample reuse decisions under a target output-deviation tolerance.It reuses cached outputs only when the predicted change is below that tolerance.
  • Experiments on Wan 2.1, CogVideoX, and LTX-Video show better visual quality than existing caching strategies under similar computational budgets.
  • SenCache requires no additional training or model modification and is designed to be agnostic to architecture and sampler.The framework also provides a sensitivity-based interpretation of where prior heuristics succeed or fail.

2. Related Work

Prior diffusion acceleration methods reduce per-step computation, reduce the number of sampling steps, or reuse computations through caching. SenCache distinguishes itself from heuristic full-forward caching by grounding reuse in local denoiser sensitivity.

  • Reducing Per-Step Cost: Quantization, pruning, and neural architecture search reduce per-step computation but may require calibration, optimization, data-dependent criteria, or workload-specific tuning.
  • Reducing the Number of Sampling Steps: Distillation methods reduce sampling steps by learning few-step generators, but typically require additional training and can be sensitive to domain and guidance settings.
  • Caching-Based Acceleration: Caching methods reuse computations across timesteps, including intermediate features, residuals, or attention and MLP outputs, without necessarily retraining.
  • Full-Forward Caching: Full-forward caching stores denoiser outputs at selected timesteps, with TeaCache and MagCache using residual-based or magnitude-based heuristics.
  • Full-Forward Caching: Heuristic full-forward caching methods lack theoretical guarantees and often require substantial tuning to balance speed and quality.
  • Differences with Previous Methods: SenCache replaces ad-hoc triggers with a first-order local-sensitivity estimate of output change, making its criterion modality-, architecture-, and sampler-agnostic.

3. Background

The background describes diffusion or flow-matching generation as repeated evaluation of a learned velocity field during numerical integration. Because these evaluations dominate latency, caching can skip steps when local output change is sufficiently small.

  • Flow Matching and the Probability Flow ODE: Flow matching continuously transforms a data sample x0 into a noisy variable xt over t ∈ [0, T], using scalar schedules αt and σt.The schedules satisfy α0 = 1, σ0 = 0, αT = 0, and σT = 1.
  • Flow Matching and the Probability Flow ODE: The marginal distribution evolves under a velocity field v(x, t), whose conditional target velocity is defined from the interpolation derivatives.
  • Flow Matching and the Probability Flow ODE: A neural network vθ(xt, t) is trained with a velocity-matching objective and then used during sampling.
  • Flow Matching and the Probability Flow ODE: Inference numerically integrates the learned field backward from xT ∼ N(0, I) to t = 0.
  • Inference Cost: Repeated network evaluations dominate inference cost, so reducing function evaluations can reduce latency when the update direction changes minimally.
  • Sensitivity-Based Caching: SenCache reuses the cached denoiser output when changes in xt and t yield a sensitivity score below ε; otherwise, it refreshes the cache.

4. Sensitivity-Aware Caching

SenCache uses local sensitivity to noisy latents and timesteps to predict denoiser-output changes and decide when cached outputs can be safely reused. Its adaptive rule combines both sources of variation, controls reuse with an error tolerance, and uses finite-difference estimates with practical safeguards.

  • Motivation: Caching avoids redundant denoiser evaluations by reusing outputs when predictions change only marginally across timesteps.The central decision is whether the current latent, timestep, and condition imply negligible output change.
  • Model Sensitivity: Local sensitivity measures how network outputs respond to small input perturbations, with low Jacobian norms indicating locally smooth regions suitable for caching.Large norms indicate highly responsive or nonlinear regimes, while small norms identify regions where outputs vary little.
  • Sensitivity Analysis: Both latent and timestep sensitivities contribute substantially, so caching must account jointly for latent drift ∥∆x_t∥ and timestep gap |∆t|.Timestep sensitivity can remain large even when latent changes are small, making latent-only criteria vulnerable to noticeable error.
  • Adaptive Caching: SenCache estimates output variation with a first-order sensitivity approximation and reuses the cache when the resulting score remains below tolerance ε.The score combines latent and timestep terms, while the cache rule refreshes the stored output when the tolerance condition is not met.
  • Practical Implementation: SenCache computes sensitivities once on a small calibration set and uses directional finite differences instead of exact derivatives during practical estimation.The implementation uses 8 videos with varied motion dynamics and scene content for model-level sensitivity estimation.
  • Practical Implementation: The maximum consecutive reuse limit n controls the speed–accuracy trade-off because first-order estimates are only locally accurate.After n reuses, the cache is refreshed; smaller n is more conservative, whereas larger n can increase speed while reducing precision.

5. Experiment

Experiments evaluate SenCache across three video diffusion models using visual-quality and efficiency metrics. The results show stronger quality at matched compute, parameter trade-offs, and stable sensitivity estimates from small calibration sets.

  • Settings: Evaluations compare SenCache with TeaCache and MagCache on Wan 2.1, CogVideoX, and LTX-Video using LPIPS, SSIM, PSNR, NFE, and cache ratio.The evaluation covers the full VBench prompt set and uses quantitative measures of both visual quality and computational efficiency.
  • Main Results: At matched NFE in the aggressive Wan 2.1 regime, SenCache consistently achieves better visual quality than MagCache.In the conservative regime, visual quality is comparable, while SenCache and MagCache provide greater compute reduction than TeaCache.
  • Main Results: CogVideoX and LTX-Video show clearer quality degradation under aggressive reuse, indicating lower tolerance to approximation than Wan 2.1.Matching prior methods’ low NFE requires larger tolerance values, including ε = 0.5 and 0.6.
  • Ablation on n: Increasing the cache lifetime n lowers NFE until n = 4, after which efficiency saturates while visual quality degrades.Longer reuse chains make the first-order approximation less accurate as the denoising trajectory drifts from the reference point.
  • Ablation on ε: Increasing ε from 0.04 to 0.13 lowers NFE from 25 to 21 while LPIPS rises from 0.0455 to 0.0513, PSNR falls from 29.01 dB to 28.72 dB, and SSIM declines from 0.930 to 0.924.The ablation demonstrates a gradual accuracy–efficiency trade-off as caching becomes more permissive.
  • Calibration Set Size: Sensitivity estimates from 8 diverse calibration videos closely match those from much larger sets, suggesting that large calibration batches are unnecessary in practice.The calibration study compares sets ranging from 8 to 4096 videos.

6. Discussions and Future Work

The discussion identifies limitations of the current first-order sensitivity surrogate and fixed tolerance design, while proposing richer estimators, timestep scheduling, and extension beyond video.

  • Limitations: The implementation relies on a first-order sensitivity surrogate that may incur larger error in nonlinear regimes.Higher-order or learned estimators are proposed as future directions.
  • Future Work: The paper uses a fixed ε threshold, leaving dynamically scheduled tolerances across denoising timesteps for future work.The proposed motivation is that different timesteps contribute unequally to final fidelity.
  • Future Work: Validation focuses on video diffusion models, while applying sensitivity-aware caching to text, audio, or multimodal systems remains future work.The paper describes the underlying principle as not limited to the visual domain.

7. Conclusion

The conclusion presents sensitivity-aware caching as a principled way to reduce diffusion inference cost while maintaining generation quality. It emphasizes sensitivity to both noisy latents and timesteps, efficient estimation, and broader applicability.

  • Conclusion: SenCache quantifies denoiser sensitivity to both the noisy latent and timestep to decide when cached outputs can be safely reused.The framework uses a combined metric reflecting the network’s local behavior.
  • Conclusion: Finite-difference sensitivity estimates require only a small calibration set and a single precomputation per model.This supports practical inference-time use without repeatedly recomputing sensitivity statistics.
  • Conclusion: Experiments on video diffusion models demonstrate reduced inference cost while maintaining generation quality.The conclusion frames sensitivity-aware caching as a foundation for adaptive acceleration across diffusion architectures and modalities.

Supplementary Material

Sensitivity and consecutive-step output variation differ across models, affecting caching error and tolerance requirements. SenCache also delivers substantial latency and compute reductions on Wan 2.1.

  • Higher consecutive-step MAE in CogVideoX and LTX-Video indicates greater per-step variation where caching is frequently applied.The diagnostic averages MAE across 100 videos in the mid-range timesteps approximately 800–200.
  • Global timestep optimization can allocate error budgets across timesteps, whereas SenCache uses a fixed per-step tolerance ε for local decisions.
  • 41.1% speedup reduces Wan 2.1 wall-clock latency from 182.3 s to 107.3 s, compared with MagCache’s 39.3% speedup and 110.6 s.Both methods reduce total compute from 8,244,043.09 to 3,482,412.58 GFLOPs, or 57.8% fewer.
  • Sensitivity patterns vary markedly across models, with both timestep and noisy-latent variation relevant to effective caching.A batch of 8 diverse samples is reported as sufficient for reliable sensitivity estimates; large batches are unnecessary.
Loading 2602.24208v1…