SeaCache: Spectral-Evolution-Aware Cache for Accelerating Diffusion Models

Introduction

• Diffusion and rectified-flow models [14] [48] [56] [58] [16] [38] require tens to hundreds of denoising steps, while distillation, quantization, and efficient attention add training overhead or task-specific tuning.The paper frames sampling latency as a barrier for user-facing applications.Source: §1 · p. 1
• Prior dynamic caching methods [1] [6] [33] decide reuse from feature distances but measure error in raw feature space, overlooking spectral evolution.Earlier static schedules [36] cache at fixed trajectory intervals.Source: §1 · p. 1
• Spectral evolution means early timesteps establish low-frequency structure while later ones refine high-frequency detail; signal carries content, whereas noise reflects stochastic variation.The paper describes the evolution as a changing signal-to-noise ratio.Source: §1 · pp. 1–2
• SeaCache applies a timestep-dependent SEA filter before feature-distance computation, emphasizing content-relevant signal and downweighting noise in cache decisions.The filter modulates frequency response along the sampling trajectory and is motivated by spectral evolution.Source: §1 · p. 2
• SeaCache requires no architectural modification or retraining and is network- and sampler-agnostic, integrating with existing caching policies through one filtering step.The paper also reports substantially fewer forward passes while preserving perceptual fidelity.Source: §1 · p. 2
• In an oracle comparison on FLUX [28, 29] and Wan2.1 1.3B [63], SEA-filtered output distances produce higher PSNR than raw output distances at matched refresh ratios.The oracle compares consecutive outputs to isolate spectral filtering’s effect, and the figure reports PSNR against corresponding full-compute samples.Source: Figure 2 · p. 2, §1 · p. 2
• Across multiple visual generative models, SeaCache consistently improves the latency–quality trade-off over prior caching schemes and baselines.The introduction reports this as the outcome of extensive experiments.Source: §1 · p. 2

2.1. Generative Model Acceleration

• Step reduction and per-step efficiency methods lower inference cost, but require extra training, model changes, resources, or engineering effort.Step reduction uses improved solvers and distillation; per-step methods include quantization, efficient attention, and token reduction.Source: §2.1 · pp. 2–3
• Caching reuses intermediate computations without retraining, motivating it as an alternative that exploits redundancy across successive denoising timesteps.Source: §2.1 · p. 3

2.2. Caching-based Acceleration

• Early caching methods [42] [66] target U-Nets, later methods [9] [34] [53] adapt caching to DiTs, and PAB selects video timestep intervals by attention block.Source: §2.2 · p. 3
• Static schedules cannot adapt to input diversity; dynamic methods include motion-aware AdaCache [24] and TeaCache [33] and DiCache [6], which estimate redundancy from distances near input features.Source: §2.2 · p. 3
• SeaCache decides when to reuse via timestep-conditioned spectral filtering and gain normalization, adding a frequency prior unlike prior dynamic caching policies.It measures redundancy in timestep-aligned spectral space to emphasize content-carrying components; prior studies [35] [37] instead explore how to use cached features across frequency bands, while SeaCache leverages early low-frequency structure and later high-frequency refinement in a policy designed to plug into existing caching baselines.Source: §2.2 · p. 3

3.1. Denoising Generative Models

• DPMs [21] and RF models [38] iteratively denoise encoded image or video latents, with their noising mixtures determined by a timestep-dependent schedule.DPMs are trained to predict added noise, and sampling reverses the process from noise toward the clean latent.Source: §3.1 · p. 3
• Adjacent denoising steps have redundant outputs, enabling cache-based acceleration by reusing intermediate predictions.Sampling starts from noise and iteratively reconstructs the clean latent.Source: §3.1 · p. 3

3.2. Timestep-Aware Dynamic Caching

• TeaCache [33] measures relative change in timestep-modulated input features and reuses a cached output until accumulated change exceeds threshold δ.Smaller δ triggers more frequent refreshes and higher fidelity; larger δ increases speed but risks artifacts.Source: §3.2 · p. 3
• SeaCache replaces the distance metric with change between spectrally filtered features, accumulating distance to choose between reusing cached output and refreshing the unchanged denoiser.Its overview applies FFT, timestep-dependent SEA filters, then inverse FFT; the model itself remains unchanged.Source: §3.2 · p. 3, Figure 3 · p. 4

4.1. Spectral-Evolution-Aware Filter

• Early diffusion steps emphasize low frequencies while later steps recover high-frequency detail [3] [22] [30]; SeaCache derives timestep-dependent filters from the optimal linear denoiser response.The filter is applied per channel over spatial axes for images and spatiotemporal axes for videos; FFT filtering emphasizes content-carrying signal while suppressing stochastic noise.Source: §4 · p. 4, §4.1 · pp. 4–5

4.2. Spectrum-Aware Dynamic Caching

• SeaCache normalizes the filter to constant mean gain across radial frequencies, then uses filtered feature distance while retaining the accumulated-distance refresh rule.The next refresh occurs at the smallest timestep whose accumulated distance exceeds threshold δ; the resulting gate is training-free, architecture-agnostic, and depends on sampler schedule coefficients (a_t, b_t).Source: §4.1 · p. 5, §4.2 · p. 6
• SEA-filtered input distances closely track SEA-filtered output distances across FLUX and Wan2.1 1.3B trajectories, unlike raw or polynomial-fitted inputs, especially at early steps.Distances are averaged over ten samples; filtered input distances are larger early in the trajectory, consistent with recomputing early steps.Source: §4.2 · p. 5, Figure 5 · p. 5

5.1. Experimental Settings

• Experiments use 50-step sampling, shared initial seeds, and approximately 50% and 30% cache budgets across three generative models.Evaluation uses 200 Draw-Bench prompts for 1024 × 1024 images and 944 VBench prompts for 480p, 65-frame videos; cached samples are compared with full-timestep references using PSNR, LPIPS, and SSIM.Source: §5.1 · p. 6

5.2. Quantitative Comparison

• At both approximately 50% and 30% budgets, SeaCache achieves the lowest average CycleReward rank among TeaCache and TaylorSeer.This indicates a better latency-quality trade-off for reconstruction fidelity and preference.Source: §5.2 · p. 6
• SeaCache improves the latency-quality trade-off across FLUX.1-dev, HunyuanVideo, and Wan2.1; on HunyuanVideo, PSNR rises by roughly 8 dB over the strongest baseline.The comparisons cover moderate and more aggressive cache settings; SeaCache reduces latency and TFLOPs while improving image-quality metrics.Source: §5.2 · pp. 6–7, Table 3 · p. 6, Table 4 · p. 7

5.3. Qualitative Comparison

• In qualitative comparisons, SeaCache preserves image semantics and details and produces clearer video text and objects than the baselines.Examples include retaining “quantum” in FLUX, a legible “STOP” sign in HunyuanVideo, and clearer pandas and coffee cups in Wan2.1.Source: §5.3 · p. 7, Figure 6 · p. 7, Figure 7 · p. 7

5.4. Additional Analysis

• Across FLUX and HunyuanVideo, the SEA filter yields the best PSNR-refresh ratio trade-off; normalization and timestep-dependent spectral evolution are important to its performance.The complementary 1−SEA filter is consistently lower, while removing normalization or using a static low-pass filter performs worse.Source: §5.4 · pp. 7–8, Figure 8 · p. 8
• Applying SEA filtering to DiCache’s block-level features produces consistently higher PSNR at the same refresh ratio, with comparable latency and FLOPs.The adaptation retains DiCache’s original accumulation rule and does not modify its policy or network.Source: §5.4 · p. 8, Figure 9 · p. 8
• At a 30% FLUX refresh budget, SeaCache concentrates refreshes on early timesteps, unlike TeaCache’s more uniform pattern, without manually setting how many early steps to compute.The schedule is measured across 200 Draw-Bench prompts and aligns refresh allocation with the spectral-evolution prior.Source: §5.4 · p. 8, Figure 10 · p. 8

• SeaCache applies the Spectral-Evolution-Aware (SEA) filter to input features to create content-adaptive cache schedules that respect the diffusion model’s spectral priors.The policy is training-free.Source: §6 · p. 8
• Raw feature distances in prior cache-based approaches fail to separate signal from noise, while SEA-filtered distances follow the full-compute trajectory more faithfully.Source: §6 · p. 8
• Incorporating spectral evolution into cache design can be combined with future acceleration methods.Source: §6 · p. 8

• The supplement includes a derivation analysis of the linear diffusion process and additional experiments validating SeaCache.Source: Supplementary Material · p. 13

Setup and assumptions

• The derivation assumes a wide-sense stationary clean signal independent of zero-mean white Gaussian noise with flat power spectral density Sε(f) = 1.It frames the timestep input as a linear mixture in iterative denoising generative models.Source: §7 · p. 13

Frequency-domain MSE expansion

• Parseval’s theorem decomposes reconstruction MSE into a frequency integral, so minimizing total MSE reduces to minimizing Jt(f) independently at each frequency.The reconstruction MSE objective is equivalent to the diffusion-model denoising objective.Source: §7 · p. 13

Optimality by differentiation

• Differentiating the frequency-wise objective with a Wirtinger derivative and setting it to zero yields a unique linear-MMSE minimizer, which defines the optimal frequency response.The derivation uses the expectation terms established under the signal and noise assumptions.Source: §7 · pp. 13–14

Power-law prior

• Under the adopted power-law spectrum prior, the optimal response’s effective passband widens as a_t increases; β is set to 2 for images and 3 for videos.The method’s SEA filter Gnorm_t(f) is a normalized variant of Gt(f).Source: §7 · p. 14

8. Runtime Overhead of SEA Filtering

• At 50% caching, SEA filtering uses 0.058 s/sample (0.6%) for FLUX and 0.362 s/sample (0.4%) for HunyuanVideo, out of 9.4 s and 90.8 s total, respectively.The measurements use settings identical to the main experiments.Source: §8 · p. 14
• For LightX2V on Wan2.1-14B-T2V at 50% caching, SEA filtering overhead is 0.41% at 480p and 0.27% at 720p.Profiling used a single Blackwell Pro 6000 GPU; sampling runs in compressed latent space.Source: §8 · p. 14

9. Compatibility with Fast Inference Works

• Evaluation uses LightX2V at 16 steps and Jenga at 50 steps, with default configurations, on VBench [23] at 480p and 41 frames across 50 prompts.The settings are a distilled sampler and an efficient-attention variant, respectively.Source: §9 · p. 14

10.1. Quantitative Comparison in T2V Generation

• On HunyuanVideo VBench, SeaCache ranks first at both budgets: 1.91 vs. 2.03/2.06 at ≈50%, and 1.75 vs. 2.16/2.09 at ≈30%.The comparisons are against TeaCache [33] and TaylorSeer [19], aggregated by average rank across VBench [23] dimensions.Source: §10.1 · p. 14, Table 9 · p. 15
• On Wan2.1 1.3B VBench, SeaCache ranks second at both budgets: 1.97 vs. the best 1.91 at ≈50%, and 2.13 vs. the best 1.53 at ≈30%.The results aggregate average rank across VBench dimensions.Source: §10.1 · p. 14, Table 10 · p. 15
• On CompressedVQA, at refresh ratios of approximately 50% and 30%, SeaCache consistently achieves the highest single-scale and multi-scale scores among caching baselines across HunyuanVideo and Wan2.1 1.3B.Source: Table 11 · p. 16, §10.1 · p. 16

10.3. Qualitative Comparison in T2I Generation

• In FLUX examples, SeaCache preserves water and bottle details at 50% refresh and reconstructs seven stars and five consistent chopsticks at 30% better than baselines.At 50%, it avoids blocky water artifacts and captures the bottle and orange liquid; at 30%, it stays closest to the full-compute reference.Source: §10.3 · p. 16
• Across cache budgets, SeaCache alone renders the intended “CUBE” text and most closely matches the full-compute wooden ice-cream designs.Baselines repeatedly hallucinate cube-like background patterns and produce ice creams with different designs or colors.Source: §10.3 · p. 16

10.2. Comparison with MagCache

• At matched refresh ratios, SeaCache improves quality over MagCache [43] on FLUX.1-dev and Wan2.1 1.3B T2V.The comparison reports higher PSNR and lower LPIPS for SeaCache; its input-adaptive redundancy estimate contrasts with MagCache’s fixed magnitude threshold.Source: §10.2 · p. 16, Table 13 · p. 17

10.4. Qualitative Comparison in T2V Generation

• In video examples, at 30% refresh SeaCache preserves a plausible Taichi pose despite baseline hand artifacts; at 50%, it matches the original video with skateboard-surfboard contact, unlike the baselines’ floating skateboard.On Wan2.1 1.3B, SeaCache avoids truck-wheel and bicycle distortions at 30% refresh and follows the full-compute trajectory without food-item disappearance or panda artifacts at 50%.Source: §10.4 · p. 17

11. Limitation

• The closed-form filter may be biased and no longer optimal when the assumed power-law radial spectrum, wide-sense stationarity, or signal-noise independence does not hold.Generated samples can deviate from the power-law model, particularly at later timesteps or in synthetic backgrounds without salient objects.Source: §11 · p. 17
• Because the analysis is formulated in the image or video domain while models operate in learned latent spaces, the filter only approximates the optimal latent-space response.The encoder can reshape the spectrum, making the latent distribution differ from the assumed pixel-domain power law.Source: §11 · pp. 17–18
• Future work could estimate per-timestep spectra, design content-aware latent-space filters, and add lightweight nonlinear corrections while preserving the cache policy’s plug-and-play nature.These extensions aim to reduce the gap between assumed and actual signal models and improve fidelity under real-world deviations.Source: §11 · p. 18