PaperlayerView exact revision on arXiv

Source-linked AI summary

SeaCache: Spectral-Evolution-Aware Cache for Accelerating Diffusion Models

Jiwoo Chung, Sangeek Hyun, MinKyu Lee, Byeongju Han, Geonho Cha, Dongyoon Wee, Youngjun Hong, Jae-Pil Heo

arXiv:2602.18993v2cs.CV

TL;DR

Diffusion inference is slow because sequential denoising requires many steps, while raw-feature cache metrics conflate content and noise and overlook spectral evolution. SeaCache derives a training-free, timestep-dependent SEA filter and applies it to input features for content-aware reuse decisions. Across experiments, it improves the latency–quality trade-off over prior caching schemes, although its closed-form filter relies on simplifying spectral assumptions and may only approximate latent-space responses.

  • Problem

    Sequential denoising makes diffusion inference latency- and computation-intensive, while prior cache metrics overlook spectral evolution and fail to separate signal from noise.

  • Method

    SeaCache is a training-free cache policy that filters features with a timestep-dependent SEA response, emphasizing signal before measuring distances for reuse decisions.

  • Results

    SeaCache consistently improves the latency–quality trade-off over prior caching schemes across experiments and preserves perceptual fidelity while reducing forward passes.

  • Takeaways & Limitations

    Spectrally aligned cache scheduling adapts reuse decisions to content while respecting the spectral priors of diffusion and rectified-flow trajectories.

  • Takeaways & Limitations

    The analytically derived filter assumes a power-law signal spectrum, wide-sense stationarity, and signal–noise independence; violations can make it non-optimal and introduce bias, while latent-space responses may only be approximated.

Abstract

from arXiv · show

Diffusion models are a strong backbone for visual generation, but their inherently sequential denoising process leads to slow inference. Previous methods accelerate sampling by caching and reusing intermediate outputs based on feature distances between adjacent timesteps. However, existing caching strategies typically rely on raw feature differences that entangle content and noise. This design overlooks spectral evolution, where low-frequency structure appears early and high-frequency detail is refined later. We introduce Spectral-Evolution-Aware Cache (SeaCache), a training-free cache schedule that bases reuse decisions on a spectrally aligned representation. Through theoretical and empirical analysis, we derive a Spectral-Evolution-Aware (SEA) filter that preserves content-relevant components while suppressing noise. Employing SEA-filtered input features to estimate redundancy leads to dynamic schedules that adapt to content while respecting the spectral priors underlying the diffusion model. Extensive experiments on diverse visual generative models and the baselines show that SeaCache achieves state-of-the-art latency-quality trade-offs.

1. Introduction

Diffusion sampling remains latency-bound because denoising is sequential, while existing cache metrics overlook spectral evolution and entangle content with noise. SeaCache addresses this by filtering features spectrally before distance-based reuse decisions, improving latency–quality trade-offs without retraining.

  • Motivation: Sequential denoising requires tens to hundreds of steps, making diffusion and rectified-flow sampling latency-bound for user-facing applications.Distillation, quantization, and efficient attention reduce cost but add training overhead or task- and data-specific tuning.
  • Motivation: Dynamic caching reduces forward passes by reusing intermediate features, but prior distance metrics operate in raw feature space and overlook spectral evolution.Early denoising establishes low-frequency structure, while later steps refine high-frequency detail; signal is mainly low-frequency and noise mainly high-frequency.
  • SeaCache: SeaCache inserts a timestep-dependent Spectral-Evolution-Aware filter before distance computation, emphasizing signal components and downweighting noise.The filter provides a plug-and-play, network-agnostic, sampler-agnostic scheduling policy without architectural modification or retraining.
  • Evidence: At matched refresh ratios, SEA-filtered output distances achieve higher PSNR than raw output distances relative to the full-compute trajectory on FLUX and Wan2.1 1.3B.The oracle experiment isolates the effect of spectral filtering by comparing consecutive output features rather than input-to-output approximations.
  • Evidence: SeaCache substantially reduces forward passes while preserving perceptual fidelity and consistently improves latency–quality trade-offs over prior caching schemes across experiments.The paper reports extensive evaluation across multiple visual generative models.

2. Related Work

Caching accelerates iterative generation by reusing computations, but static schedules lack input adaptivity and earlier dynamic methods do not explicitly model spectral evolution. SeaCache adds a timestep-conditioned frequency prior while remaining compatible with existing caching approaches.

  • Caching-based Acceleration: Caching reuses intermediate computations across adjacent timesteps without retraining, reducing sequential inference cost.Prior approaches include static schedules, architecture-specific methods, and attention-block-specific video schedules.
  • Caching-based Acceleration: Static schedules cannot adapt to input diversity, motivating dynamic policies that respond to the generated signal.Examples include motion-aware video caching and methods estimating output changes from distances near input features.
  • SeaCache: Unlike prior dynamic caching, SeaCache measures redundancy in a timestep-aligned spectral space using a timestep-conditioned SEA filter and gain normalization.Its design explicitly models spectral evolution and injects a frequency prior into reuse decisions.
  • SeaCache: SeaCache focuses on when to reuse features, complementing methods that instead vary how cached features are utilized across frequency bands.The policy is designed to plug into existing caching baselines.

3. Preliminary

Diffusion and rectified-flow models iteratively remove noise, creating adjacent-timestep redundancy that caching can exploit. SeaCache preserves the accumulated-distance refresh logic while replacing raw feature change with a frequency-aware filtered representation.

  • Denoising Generative Models: Diffusion and rectified-flow models iteratively reconstruct clean latent representations from noisy inputs across solver timesteps.The noise-mixture coefficients depend on the schedule; diffusion models predict the added noise during training.
  • Denoising Generative Models: Reverse denoising induces redundancy between adjacent outputs, enabling cache-based acceleration through intermediate-prediction reuse.Caching reduces forward passes by reusing predictions from previous timesteps.
  • Timestep-Aware Dynamic Caching: TeaCache estimates timestep change with a relative ℓ1 distance between timestep-modulated input features, using a small constant for numerical stability.The input feature is formed by injecting a timestep embedding into the noisy latent and is treated as an inexpensive proxy for denoiser output change.
  • Timestep-Aware Dynamic Caching: Cached outputs are reused until accumulated change exceeds threshold δ, where smaller δ favors fidelity and larger δ favors speed with greater artifact risk.A threshold crossing triggers a refresh and resets the accumulator.
  • Timestep-Aware Dynamic Caching: SeaCache applies FFT-based timestep-dependent SEA filters and iFFT to consecutive input features before measuring filtered relative distance for reuse decisions.The filtered-distance accumulator either reuses cached output or refreshes the denoiser, while the underlying network remains unchanged.

4. Method: SeaCache

SeaCache derives a timestep-dependent spectral filter from denoising evolution, then uses filtered input distances to schedule cache reuse while preserving comparability across timesteps.

  • 4.1. Spectral-Evolution-Aware Filter: SeaCache models spectral evolution through a timestep-dependent frequency response derived from an optimal linear denoiser.The analysis describes low-frequency structure emerging earlier and high-frequency detail being recovered later.
  • 4.1. Spectral-Evolution-Aware Filter: The filter’s linearity assumption is restrictive, although the authors retain it as a useful source of insight into spectral evolution.The theoretical analysis is presented for an optimal linear denoiser, with implementation extending filtering across spatial or spatiotemporal axes.
  • 4.1. Spectral-Evolution-Aware Filter: The SEA mapping applies an FFT, multiplies features by the timestep-dependent filter, and uses an inverse FFT to return filtered features.This produces a timestep-dependent passband that emphasizes signal components while suppressing noise.
  • 4.1. Spectral-Evolution-Aware Filter: Density normalization enforces constant mean gain over radial frequencies so filtered distances remain comparable across timesteps.The normalized response is used as the SEA filter for cache scheduling.
  • 4.2. Spectrum-Aware Dynamic Caching: SEA-filtered input distances closely follow SEA-filtered output distances across the generation trajectory, outperforming raw and polynomial-fitted input alignment.This result is reported from averages over ten samples on FLUX and Wan2.1 1.3B, with weaker alignment especially at early timesteps for raw alternatives.
  • 4.2. Spectrum-Aware Dynamic Caching: SeaCache computes a per-step filtered-feature distance and accumulates it until threshold δ is exceeded, then refreshes the denoiser cache.The cached output is reused between refreshes; the accumulated-distance rule remains unchanged.

5. Experiments

Across image and video generation experiments, SeaCache consistently improves the latency–quality trade-off over caching baselines. Its spectrum-aware schedule preserves quality under both moderate and aggressive refresh budgets while adapting refresh timing to the denoising trajectory.

  • Quantitative Comparison: SeaCache achieves the strongest latency–quality trade-off on FLUX at approximately 50% and 30% refresh ratios.It reduces latency and FLOPs while improving PSNR, LPIPS, and SSIM, whereas baselines show clearer quality degradation under stronger acceleration.
  • Quantitative Comparison: SeaCache consistently provides better latency–quality trade-offs on HunyuanVideo and Wan2.1 1.3B than TeaCache and TaylorSeer.Across cache budgets, it reduces latency or TFLOPs while maintaining or improving reconstruction metrics.
  • Qualitative Comparison: At approximately 30% refresh, SeaCache preserves prompted text and fine details that baselines frequently lose on FLUX.At approximately 50% refresh, it also preserves the two orange plates, while baselines alter their color or remove one.
  • Additional Analysis: The SEA filter yields the best PSNR–refresh ratio trade-off across FLUX and HunyuanVideo ablations.Normalization matters, and a static 30% low-pass filter performs worse, indicating that timestep-dependent spectral evolution is important.
  • Additional Analysis: Applying SEA filtering to DiCache improves PSNR at the same refresh ratio while keeping latency and FLOPs comparable.This plug-and-play adaptation operates on DiCache’s intermediate block features without modifying its policy or network.
  • Additional Analysis: At a 30% refresh budget, SeaCache concentrates refreshes on early FLUX timesteps, unlike TeaCache’s more uniform schedule.The schedule is generated automatically rather than relying on manually fixed early-step refreshes.

6. Conclusion

The conclusion frames SeaCache as a training-free cache policy that integrates spectral evolution into reuse decisions. SEA-filtered distances more faithfully track the full-compute trajectory and support content-adaptive schedules.

  • 6. Conclusion: SeaCache replaces raw feature-space reuse decisions with decisions made in a spectrally aligned space.The approach is training-free and uses the SEA filter to separate content-relevant signal from noise.
  • 6. Conclusion: SEA-filtered distances follow the full-compute trajectory more faithfully than unfiltered metrics.The resulting schedules adapt to content while respecting the spectral priors of the diffusion model.

Supplementary Material

The supplementary material adds a derivation analysis for the linear diffusion process and reports additional experiments validating SeaCache.

  • Supplementary Material: The supplementary material presents the derivation analysis of the linear diffusion process and additional experiments validating SeaCache.

7. Derivation of Optimal Linear Response

The derivation models a timestep-dependent noisy linear mixture and derives a frequency response for reconstructing the clean signal. A power-law spectrum then yields a passband that widens along the sampling trajectory.

  • Linear reconstruction: The filter h_t estimates the clean signal x_0 from the noisy timestep signal x_t in the frequency domain.H_t(f) is the filter’s frequency response, and the derivation minimizes the signal reconstruction MSE.
  • Frequency-domain MSE: Parseval’s theorem decomposes reconstruction MSE into independent frequency components, enabling frequency-by-frequency minimization.The derivation minimizes J_t(f) for each frequency because H_t(f) acts independently across frequencies.
  • Optimality by differentiation: Differentiating the frequency-domain objective yields the unique optimal linear response under the linear MMSE criterion.The solution follows by applying Wirtinger differentiation and substituting the expectation terms derived from the signal and noise assumptions.
  • Power-law prior: The power-law prior S_x(f) ≃ A|f|^-β uses β = 2 for images and β = 3 for videos in the experiments.The prior supplies the spectrum used to obtain the timestep-dependent SEA response.
  • Spectral evolution: The effective passband widens as a_t increases, capturing spectral evolution; SeaCache uses a normalized variant of this response.The normalized filter is denoted G_norm,t(f), while the unnormalized response is G_t(f).

8. Runtime Overhead of SEA Filtering

SEA filtering adds little runtime overhead relative to end-to-end generation, including for a large text-to-video model operating in compressed latent space.

  • 0.058 s per sample out of 9.4 s means SEA filtering occupies about 0.6% of FLUX generation time.This profile uses a 50% caching ratio with all other settings held identical to the main experiments.
  • Below 1% overhead is observed for LightX2V on Wan2.1-14B-T2V under a 50% caching ratio.The model samples in compressed latent space, where the filtering pass occupies only a tiny fraction of runtime.

9. Compatibility with Fast Inference Works

SeaCache remains compatible with two orthogonal fast-inference settings for Wan2.1-T2V: a distilled sampler and an efficient-attention variant. Under comparable refresh budgets, it improves quality over TeaCache and vanilla step reduction.

  • Compatibility with Fast Inference Works: SeaCache consistently improves quality over TeaCache and vanilla step reduction under comparable refresh-ratio budgets.The evaluation covers LightX2V’s 16-step distilled sampler and Jenga’s 50-step efficient-attention configuration.

10. Additional Evaluation

Additional quantitative and qualitative evaluations show that SeaCache preserves visual quality more faithfully than caching baselines across models, refresh budgets, and matched-refresh comparisons. The qualitative cases emphasize fewer artifacts, better content and layout fidelity, and closer agreement with full-compute references.

  • Quantitative Comparison: SeaCache ranks first on HunyuanVideo VBench at both budgets, with average ranks of 1.91 versus 2.03/2.06 at approximately 50% and 1.75 versus 2.16/2.09 at approximately 30%.The comparison aggregates performance across VBench dimensions.
  • Quantitative Comparison: Across HunyuanVideo and Wan2.1 1.3B at approximately 50% and 30% refresh ratios, SeaCache achieves the highest single-scale and multi-scale scores among caching baselines.These results indicate stronger preservation of the original trajectory while reducing refresh operations.
  • Qualitative Comparison in T2I Generation: At approximately 50% refresh, SeaCache avoids water-surface artifacts and better reproduces bottle geometry and fluid dynamics than baselines.These examples compare SeaCache with baselines against full-compute references on FLUX.
  • Qualitative Comparison in T2I Generation: At approximately 30% refresh, SeaCache alone reconstructs seven well-formed stars and five consistent chopsticks, while baselines miss or deform these objects.The comparison emphasizes object count, geometry, and appearance under an aggressive refresh budget.
  • Qualitative Comparisons in T2I and T2V Generation: SeaCache more faithfully preserves text, object design, pose, contact relationships, and video dynamics across additional FLUX, HunyuanVideo, and Wan2.1 examples.It renders the intended word “CUBE,” preserves Taichi hand structure and skateboard contact, and avoids distortions around truck wheels, bicycles, food, and pandas.
  • Comparison with MagCache: At matched refresh ratios, SeaCache shows higher PSNR and lower LPIPS than MagCache on FLUX.1-dev and Wan2.1 1.3B-T2V.The comparison uses full-reference quality against uncached outputs.

11. Limitation

SeaCache’s analytical filter relies on simplifying signal and noise assumptions that may not hold for generated samples or learned latent representations. The paper proposes future extensions to better model timestep- and content-dependent spectra.

  • Analytical Assumptions: The closed-form linear filter can lose strict optimality and introduce bias when power-law spectra, stationarity, or signal-noise independence assumptions are violated.Generated samples may deviate from the assumed signal spectrum, especially at later timesteps or in synthetic backgrounds without salient objects.
  • Latent-Space Scope: Because most modern models operate in learned latent spaces, the pixel-domain spectral model may only approximate the optimal latent-space response.The encoder can reshape the spectrum, causing latent distributions to differ from the assumed power-law model.
  • Future Extensions: Future work could estimate per-timestep spectra, design latent-space content-aware filters, and add lightweight nonlinear corrections.These extensions aim to reduce the gap between assumed and actual signal models while preserving plug-and-play caching.
Loading 2602.18993v2…