SpectralCache: Accelerating Diffusion-Based World Models via Spectral Feature Caching

1 INTRODUCTION

• Repeated evaluations of large diffusion transformers make world-model generation costly and limit responsiveness in interactive applications.World models predict future observations conditioned on scenes and actions, supporting applications such as visual planning and explorable 3D scene generation.Source: §1 · pp. 1–2
• Existing caching methods mainly target conventional image and video diffusion models, and direct adaptation to world models can degrade quality.Prior work emphasizes step-wise intermediate feature changes; SpectralCache instead examines their underlying mathematical structure using SVD.Source: §1 · p. 2
• SpectralCache is a training-free framework that reuses stable singular subspaces and estimates singular values to skip redundant full computations while preserving generation quality.Its estimates use linear extrapolation across consecutive denoising steps and singular-value scaling at selected full-computation steps.Source: §1 · p. 2
• 5.22× speedup with WorldScores of 65.90 and 46.13 for static and dynamic scenes, respectively, versus WorldCache’s 3.65× speedup and scores of 64.89 and 45.43.SpectralCache also shows consistent quality and efficiency gains across HunyuanWorld-Voyager-13B and Aether-5B.Source: §1 · p. 2

Generation-driven Diffusion World Models

• Policy-centric world models learn dynamics through agent-environment interaction, whereas generative world models learn from visual or video data and simulate future scenes with diffusion methods.The passage cites Hafner et al. (2019; 2023); Robine et al. (2023); Wu et al. (2023) for policy-centric models, and Agarwal et al. (2025); Bar et al. (2025); Russell et al. (2025); Zhu et al. (2025) for generative models.Source: §2 · pp. 2–3

Model-level Diffusion Acceleration

• Diffusion acceleration includes sampling and ODE-trajectory optimization, model distillation, quantization, and pruning, with some optimization-based approaches requiring additional training.The cited examples include Lu et al. (2022); Song et al. (2020) for solvers and Salimans & Ho (2022) for training requirements.Source: §2 · p. 3

Diffusion Acceleration from Representation Cache

• Training-free representation caching reuses or predicts intermediate representations, but methods designed for conventional image or video generation can degrade quality on world models.The passage distinguishes adaptive token caching from forecasting-based methods and cites Kahatapitiya et al. (2025); Ma et al. (2024); Song et al. (2025).Source: §2 · p. 3

Low-rank spectral state of visual features

• The spectral state at step t comprises leading singular subspaces U_t and V_t, diagonal Σ_t, and the truncated residual R_t.The analysis first reshapes each denoising-step visual feature into a matrix X_t for SVD.Source: §3 · p. 3
• The top 128 singular values capture 96% of total feature energy on average, indicating a strong low-rank structure.The analysis applies SVD to matrixified visual features from the DiT backbone.Source: §3 · pp. 3–4

Singular subspaces remain highly stable with similar singular vectors

• Leading U and V subspaces remain highly aligned across gaps of up to eight denoising steps, with cosine similarity mostly above 0.95.Figure 2(a) depicts dominant singular-subspace stability across consecutive denoising steps.Source: Figure 2 · p. 3, §3 · p. 4

Singular values exhibit a linear extrapolation pattern

• Singular values follow an approximately linear extrapolation pattern, and extrapolating them empirically introduces only 0.71% error relative to fully computed features.The observed slope is close to 1/(n−m), supporting estimates from two preceding steps.Source: Figure 2 · p. 3, §3 · p. 4

Singular values can be approximated with a constant scaling factor

• Across nearby full-computation steps, singular-value norms have an approximately constant ratio, motivating singular-value scaling to approximate the current feature.Figure 3 depicts consistency of dominant subspaces and spectral magnitude ratios across nearby full-computation steps.Source: §3 · p. 4, Figure 3 · p. 4

4 METHOD

• SpectralCache reuses cached singular subspaces and estimates singular values by extrapolation or scaling to bypass diffusion computations.Full computation steps cache singular components; approximation steps reuse them.Source: §4 · p. 5

4.1 FULL COMPUTATION STEP

• Full steps reshape final-layer features, compute rank-r spectral states, and retain the two most recent; r=128 captures 96% of feature energy.Source: §4.1 · p. 5

4.2 APPROXIMATION STEPS

• Approximation steps bypass the DiT backbone, reusing saved singular subspaces and residuals while updating only the dominant singular values.Source: §4.2 · pp. 5–6

4.2.1 SINGULAR VALUE CACHE THROUGH LINEAR EXTRAPOLATION

• For consecutive cache steps, singular values are extrapolated from two prior full steps; the resulting feature error is 0.71% relative to fully computed features.Source: §4.2.1 · p. 6

4.2.2 SINGULAR VALUE ESTIMATION THROUGH SCALING

• Scaling estimates singular values at a triggered step from the most recent full step, using γ=1.1 in all main experiments.Source: §4.2.2 · p. 6

4.3 APPROXIMATION ERROR ACCUMULATION TO TRIGGER FULL COMPUTATION

• Accumulated approximation error or reaching the maximum cache interval triggers a full computation to limit error growth.Source: §4.3 · p. 6

4.4 OVERALL PIPELINE

• After two initial full steps, the pipeline uses extrapolation between triggers and alternates triggered full computations with scaling approximations based on the previous trigger type.Source: §4.4 · pp. 6–7

5.1 EXPERIMENTAL SETTINGS

• Experiments compare against training-free caching baselines and follow prior work's settings for model inputs, implementation, and evaluation.Both evaluated models condition on images, text, and camera trajectories to generate RGB video and depth maps.Source: §5.1 · p. 7

5.2 MAIN RESULTS

• On HunyuanVoyager-13B, SpectralCache achieves the highest WorldScores, 65.94 for static and 46.16 for dynamic scenes, while delivering a 5.22× end-to-end speedup.It also improves PSNR from 23.49 to 25.06 and SSIM from 0.770 to 0.831; on Aether-5B it achieves the best WorldScores, 65.44 and 45.82, and a 2.20× speedup versus WorldCache's 1.68×.Source: Table 1 · p. 7, §5.2 · p. 8
• On Aether-5B 3D reconstruction, SpectralCache achieves a 3.45× end-to-end speedup and improves depth and camera-pose metrics over WorldCache.Abs Rel falls from 0.341 to 0.331, ATE from 0.184 to 0.179, and RPE trans. from 0.068 to 0.060; RPE rot. remains comparable at 0.800 versus 0.796.Source: §5.2 · p. 8, Table 2 · p. 8

5.3 ABLATION STUDIES

• Rank 128 provides the reported trade-off of PSNR 25.06, SSIM 0.818, and 5.22× speedup; increasing rank to 256 yields marginal quality gains and slightly less speedup.The evaluation covers HunyuanVoyager-13B and Aether-5B and uses WorldScore, perceptual metrics, 3D reconstruction metrics, speedup, and peak memory.Source: §5.1 · p. 7, §5.3 · p. 8
• Increasing the maximum cache interval improves acceleration but gradually degrades quality; N = 4 is the reported trade-off at PSNR 25.06, SSIM 0.818, and 5.22× speedup.Intervals of 5 or 6 further improve speedup but cause noticeable quality degradation.Source: Table 4 · p. 8, §5.3 · p. 9
• Singular-value scaling at triggered full-computation steps yields 5.22× speedup, compared with 3.84× when scaling is disabled, which gives stronger quality.The tested scaling factor γ = 1.1 achieves PSNR 25.06 and SSIM 0.818, consistent with the observed approximately constant scaling factor.Source: §5.3 · p. 9, Table 5 · p. 9, Figure 5 · p. 9

5.4 COST ANALYSIS

• The pipeline retains low-rank SVD states from the two most recent full-computation steps, adding about 0.1 GB of cache memory, approximately 0.2% of the 50.44 GB model.SVD, spectral approximation, and error-estimation costs are minor compared with a full DiT computation.Source: §5.4 · p. 9, Table 6 · p. 9

• World-model features have stable singular subspaces across nearby denoising steps, while their singular values follow predictable evolution patterns.Source: §6 · p. 9
• SpectralCache reuses these subspaces, estimates singular values by linear extrapolation or scaling, and achieves up to 5.22× acceleration while preserving high generation quality.Source: §6 · p. 10

• LLM use was limited to grammar correction, typo fixes, and language polishing for clarity and readability.Source: AI USE STATEMENT · p. 10

• A prompt describes a virtual collaborative meeting with team-member avatars exchanging ideas and expressing emotions.Source: A APPENDIX · p. 13
• A figure presents a further visual comparison of SpectralCache and existing methods on HunyuanVoyager-13B.Source: Figure 6 · p. 13

A.1 MORE VISUALIZATION RESULTS.

• One comparison prompt describes sunlight illuminating a bed in the Cozy Caravan Bedroom.Source: §A.1 · p. 14
• A second prompt depicts a Modern Living Space with natural light, a couch, and a dining table.Source: §A.1 · p. 14
• Figure 7 presents additional visual comparisons between SpectralCache and existing methods on Aether-5B.The appendix also identifies Figure 6 as a visual comparison.Source: Figure 7 · p. 14, §A.1 · p. 14

A.2 OVERALL FRAMEWORK

• The output head projects the feature to denoised y_t, and the scheduler uses it to update the latent for the next denoising step.The Transformer computation G_θ produces final-layer visual features F_t.Source: §A.2 · p. 14
• Two initial full backbone computations populate a buffer with each feature’s singular subspaces, singular values, and residual.The buffer retains only the two most recent spectral states.Source: §A.2 · p. 15
• When accumulated error is below threshold τ and the cache interval is not reached, SpectralCache extrapolates singular values within the latest full-computation subspace.It adds the stored residual to reconstruct the feature and accumulates chaotic-token error after approximation.Source: §A.2 · p. 15
• At a refresh, singular-value scaling can skip a backbone evaluation after a true full-computation refresh; otherwise, the method performs a full computation.The decision follows the accumulated approximation error or cache-interval limit.Source: §A.2 · pp. 15–16

Detailed Experimental Settings

• Evaluation uses HunyuanVoyager-13B and Aether-5B, with 50-step video generation and 30-step Aether-5B 3D reconstruction.The video protocols generate 49 frames at 512 × 768 for HunyuanVoyager-13B and 41 frames at 480 × 720 for Aether-5B; quality measures include WorldScore, PSNR, and SSIM.Source: §A.3 · p. 16

Effect of Cache Error Threshold

• A smaller error threshold raises quality but reduces acceleration: τ = 0.5 yields PSNR 25.39 and 4.42× speedup.At τ = 1.5, speedup rises to 5.36× while PSNR decreases to 24.71.Source: §A.3 · p. 16
• 5.22× speedup with PSNR 25.06 at τ = 1.0, the inherited default, provides a favorable balance between generation fidelity and inference efficiency.The default is used without additional SpectralCache-specific threshold tuning.Source: Table 7 · p. 16, §A.3 · p. 16

Token Grouping Percentiles

• PSNR 25.06 and SSIM 0.818 are achieved with token-grouping percentiles {p_s, p_c} = {0.3, 0.7}.The percentiles partition tokens into static, regular, and chaotic groups; changing either threshold modestly degrades quality.Source: §A.3 · pp. 16–17, Table 8 · p. 17