PaperlayerView exact revision on arXiv

Source-linked AI summary

WorldCache: Accelerating World Models for Free via Heterogeneous Token Caching

Weilun Feng, Guoxin Fan, Haotong Qin, Mingqiang Wu, Yuqi Li, Xiangqi Li, Zhulin An, Libo Huang, Dingrui Wang, Longlong Liao, Michele Magno, Yongjun Xu, Chuanguang Yang

arXiv:2603.06331v2cs.CV

TL;DR

Diffusion world models remain costly because repeated denoising evaluations hinder interactive use and long-horizon rollouts. WorldCache introduces training-free heterogeneous token caching and achieves up to 3.7× acceleration while preserving 98% of generative quality.

  • Problem

    Existing caching methods assume relatively homogeneous single-modal feature dynamics, limiting their fit for heterogeneous multi-modal tokens and non-uniform world-model dynamics.

  • Method

    WorldCache combines curvature-guided heterogeneous token prediction with chaotic-prioritized adaptive skipping for training-free, token-aware caching.

  • Results

    Up to 3.7× acceleration is achieved while preserving 98% of generative quality across diffusion world-model experiments.

  • Takeaways & Limitations

    WorldCache offers a partial solution for efficient, interactive multi-modal world simulation under resource constraints.

Abstract

from arXiv · show

Diffusion-based world models have shown strong potential for unified world simulation, but the iterative denoising remains too costly for interactive use and long-horizon rollouts. While feature caching can accelerate inference without training, we find that policies designed for single-modal diffusion transfer poorly to world models due to two world-model-specific obstacles: \emph{token heterogeneity} from multi-modal coupling and spatial variation, and \emph{non-uniform temporal dynamics} where a small set of hard tokens drives error growth, making uniform skipping either unstable or overly conservative. We propose \textbf{WorldCache}, a caching framework tailored to diffusion world models. We introduce \textit{Curvature-guided Heterogeneous Token Prediction}, which uses a physics-grounded curvature score to estimate token predictability and applies a Hermite-guided damped predictor for chaotic tokens with abrupt direction changes. We also design \textit{Chaotic-prioritized Adaptive Skipping}, which accumulates a curvature-normalized, dimensionless drift signal and recomputes only when bottleneck tokens begin to drift. Experiments on diffusion world models show that WorldCache delivers up to \textbf{3.7$\times$} end-to-end speedups while maintaining \textbf{98\%} rollout quality, demonstrating the vast advantages and practicality of WorldCache in resource-constrained scenarios. Our code is released in https://github.com/FofGofx/WorldCache.

1. Introduction

Diffusion-based world models support long-horizon environmental imagination but remain expensive to run and are poorly served by caching policies designed for single-modal diffusion. WorldCache addresses their heterogeneous token dynamics and non-stationary failure regimes with token-adaptive prediction and chaotic-prioritized skipping.

  • World models capture spatiotemporal environmental dynamics for long-horizon imagination, planning, decision making, and interactive agents.
  • Diffusion world models require repeated backbone evaluations across many denoising steps, making inference costly.Feature caching is an established training-free approach for efficient diffusion inference.
  • Existing single-modal caching policies cause rapid error accumulation and unstable rollouts because world models combine long-tailed token difficulty with non-stationary regimes dominated by bottleneck tokens.Most tokens evolve smoothly, while a small physically critical subset changes sharply and triggers caching failures during difficult temporal intervals.
  • WorldCache introduces curvature-guided heterogeneous token prediction and chaotic-prioritized adaptive skipping as a training-free acceleration framework for diffusion world models.Prediction rules vary with trajectory nonlinearity, while skipping uses a curvature-induced dimensionless drift score for unified caching thresholds across token scales and timesteps.

2. Related Works

Diffusion-based world models unify coupled modalities for predictive simulation, but this substantially increases inference cost. Existing feature-caching methods largely target single-modal diffusion and rely on homogeneous feature dynamics, motivating token-heterogeneous caching tailored to world models.

  • World Models: Diffusion-based world models use DiT backbones to jointly model coupled modalities such as RGB, geometry/depth, and optionally actions.They extend data-driven world models that learn predictive internal representations for simulating futures in control and planning.
  • World Models: Unified multi-modal generation substantially amplifies inference cost, motivating acceleration techniques tailored to world-model dynamics.
  • Feature Caching: Feature caching accelerates diffusion sampling without training by exploiting temporal redundancy across denoising steps.Prior methods include reuse-based caching across nearby steps and token-adaptive caching over selected token subsets.
  • Limitations of Prior Caching: Most prior caching strategies target single-modal image or video diffusion and implicitly assume relatively homogeneous feature dynamics.That assumption becomes fragile when coupled multi-modal world-model tokens follow distinct physical evolution patterns.
  • Motivation: These limitations motivate a token-heterogeneous caching mechanism tailored to world models.

3. Preliminaries

The paper formulates a transformer-based diffusion world model that jointly represents RGB video and depth latents, predicts denoising directions over tokens, and updates states through a scheduler. It then defines feature caching as intermittent backbone evaluation with cached reuse or lightweight prediction across denoising steps.

  • Diffusion World Models with Transformer Backbones: The world model generates multi-modal world states through T denoising steps using a transformer diffusion backbone.Voyager and Aether are cited as recent transformer-based diffusion world models, with Voyager used for illustration.
  • Diffusion World Models with Transformer Backbones: RGB latents represent 2D video, while corresponding depth latents support 3D estimation; their dimensions use frame, height, and width axes.Both latent tensors have shape R^c×f×h×w, and the multi-modal latent is formed by spatial concatenation.
  • Diffusion World Models with Transformer Backbones: The transformer Fθ tokenizes multi-modal inputs and predicts the denoising direction as yt = Fθ(zt, t) ∈ R^N×c, with N = f × 2h × w.The prediction is expressed in token space.
  • Diffusion World Models with Transformer Backbones: A scheduler S performs the reverse diffusion update after the backbone predicts the denoising direction.The reverse update follows the cited scheduler formulation.
  • Feature Caching for Diffusion Models: Feature caching accelerates diffusion sampling by reusing or cheaply predicting model outputs across denoising steps.A generic scheme replaces expensive backbone evaluations with cached surrogates based on previous full evaluations and a specified cached-computation rule.
  • Feature Caching for Diffusion Models: The expensive forward Fθ is invoked only intermittently, while cached computation may use direct reuse, interpolation, or lightweight prediction.Ht stores information from previous FULL evaluations, and Ct specifies how cached computation is formed.

4. WorldCache

WorldCache accelerates diffusion world models by adapting caching to heterogeneous token predictability and non-uniform temporal dynamics. It combines curvature-guided token prediction with chaotic-token-prioritized skipping, achieving up to 3.7× end-to-end speedups while retaining 98% rollout quality.

  • Motivation: World models mix modalities and spatial regions with distinct token trajectories, making uniform caching either waste computation on stable tokens or fail on chaotic ones.WorldCache therefore allocates computation according to token-level predictability.
  • Curvature-guided Heterogeneous Token Prediction: Curvature measures each token’s normalized local trajectory nonlinearity and serves as a physics-grounded cue for first-order cache difficulty.It captures how rapidly feature drift changes rather than relying on raw displacement magnitude.
  • Curvature-guided Heterogeneous Token Prediction: WorldCache groups tokens by curvature and applies reuse to stable tokens, linear extrapolation to predictable tokens, and damped Hermite-guided prediction to chaotic tokens.The damped update blends recent velocities and becomes more conservative as the cached streak grows, mitigating drift under abrupt direction changes.
  • Chaotic-prioritized Adaptive Skipping: WorldCache accumulates curvature-normalized drift on chaotic tokens and triggers a FULL backbone evaluation when the uncertainty exceeds one threshold η.The normalization makes drift comparable across heterogeneous token statistics and timesteps, while focusing monitoring on the hardest tokens.

5. Experiments

WorldCache is evaluated on two multimodal diffusion world models for image-to-world generation and 3D reconstruction, using perceptual, WorldScore, depth, and pose metrics against training-free caching baselines. It achieves near-lossless quality with substantial acceleration, while ablations support heterogeneous token grouping and curvature-normalized adaptive skipping as key design choices.

  • Experimental setup: Experiments use HunyuanVoyager-13B and Aether-5B, which condition on image, text, and camera trajectory to generate coupled RGB video and depth.All experiments run on a single NVIDIA-A800 GPU.
  • World generation: 23.49 PSNR is achieved on Voyager-13B versus 21.76 for EasyCache, while WorldScore remains near baseline at 45.43 versus 46.40.On Aether-5B, WorldCache reaches WorldScore 44.72 versus 44.02 for EasyCache and a 1.68× speedup with near-zero memory overhead.
  • 3D reconstruction: 2.61× acceleration is obtained for Aether 3D reconstruction while preserving near-lossless geometry-aware performance.WorldCache matches the best depth Abs Rel at 0.341 versus baseline 0.340, and achieves pose RPE trans 0.068 with rotation error 0.796 versus 0.861 for HERO.
  • Qualitative results: WorldCache produces outputs closest to the Original in appearance and geometry, avoiding the drift, color noise, blurring, boundary bleeding, and inconsistent depth seen in baselines.These artifacts are especially evident around textured regions and boundaries.
  • Ablation studies: All nine tested grouping-threshold pairs outperform EasyCache, with PSNR 22.77–23.52, SSIM 0.758–0.770, and LPIPS 0.176–0.188.The broad performance plateau indicates that gains are primarily associated with heterogeneous grouping rather than narrow threshold tuning.
  • Ablation studies: CAS outperforms alternative skipping triggers by combining token difficulty with actual displacement in a dimensionless score while monitoring only chaotic tokens.Smaller η increases fidelity and latency through more frequent FULL evaluations, whereas larger η increases speed at the cost of visible drift; CAS consistently improves over TeaCache.

6. Conclusion … B.3. Acceleration & Memory Metrics

WorldCache is presented as a training-free framework for accelerating multimodal diffusion world models by addressing token heterogeneity and hard-token-driven error growth. The supplementary material formalizes its caching rationale and evaluation protocols, covering curvature-based prediction, adaptive monitoring, world-generation quality, perceptual fidelity, and efficiency.

  • 6. Conclusion: WorldCache combines curvature-guided heterogeneous token prediction with chaotic-prioritized adaptive skipping to accelerate multimodal diffusion world models without training.The framework assigns prediction strategies according to feature non-linearity and shifts updates toward bottleneck tokens.
  • A. Additional Theoretical Justification for Curvature-guided Caching: Reuse and first-order linear extrapolation provide local skipped-step predictors, while curvature bounds first-order cache difficulty under smooth token trajectories.The theoretical analysis uses continuous local token trajectories and establishes approximation properties for reuse, extrapolation, and curvature scoring.
  • Also define the smooth-limit quantity: Global monitoring can dilute hard-token errors, so a trigger based on aggregate error may delay recomputation even when failure is governed by a small hard-token subset.The proposition analyzes this dilution when complement-token error remains bounded while hard-token error drives failure.
  • Also define the smooth-limit quantity: The curvature-normalized product κ_i · ∥∆y_t,i∥2 is dimensionless, with leading dependence on global feature rescaling canceled except for dimensionless numerical or regularization terms.This normalization supports scale-consistent monitoring of feature deviations.
  • B.1. WorldScore Metrics (Static & Dynamic): WorldScore-Static averages normalized controllability and quality scores, while WorldScore-Dynamic additionally incorporates motion accuracy, magnitude, and smoothness.Individual metrics are mapped to [0, 100], with higher values always better, before aggregation.
  • B.2. Perceptual Fidelity Metrics: Perceptual fidelity compares cache-accelerated videos against original model outputs using PSNR (↑), SSIM (↑), and LPIPS (↓).Metrics are computed per frame and averaged across frames and prompts; lower LPIPS indicates closer perceptual similarity.
  • B.3. Acceleration & Memory Metrics: Efficiency evaluation reports FLOPs (T) (↓), Latency (s) (↓), and Memory Overhead (GB) (↓) under identical generation settings across methods.Latency uses warm-up runs and multiple trials, while memory includes cache-introduced intermediate features or states.

B.4. 3D Reconstruction Metrics · C. Detailed Experimental Settings · C.1. Models and Inference Protocols

The evaluation tests whether cache acceleration preserves 3D reconstruction capability while specifying standardized inference protocols and experimental settings for HunyuanVoyager-13B and Aether-5B. Depth and camera-pose predictions are compared with ground truth using frame-averaged depth metrics and trajectory or relative-pose errors.

  • B.4. 3D Reconstruction Metrics: Cache-accelerated predictions are evaluated for depth estimation and camera pose estimation against corresponding ground truth to assess preserved 3D capability.The evaluation follows Aether’s reconstruction protocol and targets geometry-critical tokens and temporal drift.
  • B.4. 3D Reconstruction Metrics: Abs Rel (↓) measures per-frame absolute relative depth error averaged across all frames and samples.
  • B.4. 3D Reconstruction Metrics: δ < 1.25 (↑) measures the fraction of pixels whose predicted depth is within 1.25× of ground truth, averaged per frame.
  • B.4. 3D Reconstruction Metrics: δ < 1.252 (↑) is defined similarly to δ < 1.25.
  • B.4. 3D Reconstruction Metrics: ATE (↓) measures overall camera-trajectory consistency after global Sim(3) alignment, while RPE Trans (↓) and RPE Rot (↓) measure frame-to-frame translation and rotational drift.
  • C. Detailed Experimental Settings: HunyuanVoyager-13B generates 512 × 768p content with 49 frames using 50 denoising steps under the standard inference protocol.Unless otherwise specified, acceleration methods use the official implementation’s scheduler and conditioning inputs.
  • C.1. Models and Inference Protocols: Aether-5B uses 50 steps to generate 480 × 720p, 41-frame content and 30 steps for reconstruction, with identical input conditions and schedulers across methods.
  • C.1. Models and Inference Protocols: Both models use ps = 0.3, pc = 0.7, and nmax = 6; η is 1.0 for HunyuanVoyager and 0.2 for Aether, with experiments run on one NVIDIA-A800 GPU.

C.2. Evaluation Protocols and Metrics … D.2. Different Denoising Schedules and Longer Rollouts

The evaluation combines WorldScore, perceptual consistency, and 3D reconstruction protocols with comparisons against layer-wise, model-wise, and world-model-specific caching baselines. Across additional models, resolutions, denoising schedules, and rollout lengths, WorldCache preserves the strongest quality-efficiency trade-off, including quantified gains on LingBot-14B.

  • C.2. Evaluation Protocols and Metrics: WorldScore evaluates controllability and quality using 40 single-scene and 10 three-scene prompts sampled across indoor/outdoor scenario categories.Inputs include realistic and stylized conditions.
  • C.2. Evaluation Protocols and Metrics: Cached outputs are compared with corresponding no-cache outputs using averaged frame-level PSNR, SSIM, and LPIPS metrics.The same prompts used for WorldScore generate all samples.
  • C.2. Evaluation Protocols and Metrics: 3D reconstruction evaluation follows Aether and HERO settings on the Sintel dataset, using HERO’s experimental configuration.
  • C.3. Baselines and Categorization: Layer-wise baselines cache intermediate Transformer representations, while model-wise baselines cache model outputs with minimal extra memory; WorldCache belongs to the latter category.Compared methods include DuCa, ToCa, TaylorSeer, HiCache, TeaCache, and EasyCache.
  • C.3. Baselines and Categorization: HERO provides a world-model-specific comparison by combining caching with token merging.
  • D. Broader Evaluation Beyond Main-text Settings: Across an additional model, higher resolutions, denoising schedules, and longer rollouts, WorldCache preserves the best quality-efficiency trade-off over model-wise and layer-wise baselines.
  • D.1. Additional World Model and Resolutions: +2.96 PSNR is achieved over EasyCache at 464 × 832 on LingBot-14B, while speed increases from 2.16× to 2.41×.At 720 × 1280, WorldCache again achieves the best fidelity and raises speed from 2.25× to 2.51×.
  • D.2. Different Denoising Schedules and Longer Rollouts: WorldCache remains strongest under both 70-step and 30-step denoising, and continues outperforming baselines at 81 and 161 rollout frames.Speedup remains stable or slightly increases as rollout context grows.

D.3. Runtime Overhead and Memory Behavior

WorldCache adds negligible control-path latency and nearly constant memory overhead as context length grows, because it stores compact output history and token masks rather than per-layer activations. Its main degradation risk arises in highly dynamic regimes, where chaotic tokens can force frequent recomputation or cause local drift.

  • Runtime overhead: 0.064–0.189s of curvature estimation, percentile grouping, and trigger evaluation represents about 0.05% of total latency across models.The reported speedups therefore come from cheaper cached prediction rather than shifted cost in control logic.
  • Memory behavior: WorldCache memory overhead stays at 0.02–0.03 GB from 49 to 161 frames.The model-level cache stores only fixed-depth output history and token masks, not per-layer activations.
  • Limitations: Degradation is most likely under rapid camera motion, complex textures, or abrupt geometric changes.In these regimes, a small chaotic-token subset dominates error, making aggressive caching either trigger more FULL recomputation or introduce visible local drift when thresholds are too loose.
  • Limitations: CAS monitors the hard token subset directly, alleviating but not eliminating the trade-off between recomputation frequency and local drift.The remaining limitation is that tighter thresholds can shrink speedups, while looser thresholds can permit visible drift.

E. More Analysis of Curvature-guided Heterogeneous Token Prediction · E.1. Visualization of Token Heterogeneity

The section empirically supports Curvature-guided Heterogeneous Token Prediction by visualizing token heterogeneity across prompts and modalities and introducing a quantitative ablation of its grouping strategy. Visualizations show distinct modality-specific and spatially varying curvature patterns, motivating spatially adaptive caching decisions.

  • E. More Analysis of Curvature-guided Heterogeneous Token Prediction: The analysis provides further empirical evidence for the design choices of Curvature-guided Heterogeneous Token Prediction.It includes visualization and quantitative ablation components.
  • E. More Analysis of Curvature-guided Heterogeneous Token Prediction: The study visualizes token heterogeneity across different prompts and modalities.The visualizations examine how curvature varies across world-model inputs.
  • E. More Analysis of Curvature-guided Heterogeneous Token Prediction: A quantitative ablation study evaluates the effectiveness of the proposed grouping strategy.The ablation complements the visualization-based analysis of CHTP.
  • E.1. Visualization of Token Heterogeneity: Across diverse prompts, Figure 7 maps RGB and Depth token curvature over denoising steps t = 2 to t = 49 using HunyuanVoyager.This extends visualization beyond an isolated example to demonstrate widespread token heterogeneity in world models.
  • E.1. Visualization of Token Heterogeneity: RGB and Depth modalities exhibit distinct curvature landscapes, so one caching decision cannot simultaneously satisfy both.In Prompt 2 at Step 37, Depth has high-curvature regions associated with large geometric structures, while RGB follows a different texture-dependent distribution.
  • E.1. Visualization of Token Heterogeneity: Curvature is spatially non-uniform, with chaotic high-curvature regions near object boundaries or rapid motion alongside stable low-curvature background regions.This spatial variance persists across prompts and supports spatially adaptive prediction.

E.2. Effectiveness of Curvature-guided Grouping … F.2. Necessity of Dimensionless Indicators

WorldCache’s curvature-guided grouping outperforms uniform and random token-processing strategies by assigning operators according to token dynamics. Its adaptive skipping focuses on chaotic-token drift and uses curvature-normalized indicators to remain reliable across highly variable denoising scales.

  • E.2. Effectiveness of Curvature-guided Grouping: Uniform prediction is suboptimal: reuse achieves PSNR 22.74, while uniform linear extrapolation falls to the worst PSNR of 18.01.Linear extrapolation overshoots in high-curvature chaotic regions, whereas reuse cannot capture evolving features.
  • E.2. Effectiveness of Curvature-guided Grouping: Uniform damping prevents divergence versus linear prediction, improving PSNR from 18.01 to 23.76, but lowers SSIM to 0.665 versus reuse’s 0.714.Applying damping to stable or linear tokens introduces unnecessary historical bias and smoothing.
  • E.2. Effectiveness of Curvature-guided Grouping: Random grouping yields SSIM 0.710 versus 0.791 for curvature-guided grouping, showing that gains depend on correctly matching operators to tokens.The comparison preserves the same Stable/Linear/Chaotic ratios while changing only the assignment strategy.
  • E.2. Effectiveness of Curvature-guided Grouping: CHTP reaches PSNR 25.76 and LPIPS 0.227 with negligible latency overhead by assigning reuse, linear extrapolation, and damped prediction to stable, smooth-motion, and chaotic tokens.This heterogeneous assignment resolves the stability–responsiveness trade-off.
  • F.1. Dominance of Chaotic Tokens in Temporal Dynamics: Figure 8 shows that temporal error is non-uniform: lower token percentiles remain flat and low, while the top percentile exhibits sharp spikes that dominate global variance.These dynamics motivate monitoring the Chaotic group rather than the global mean.
  • F. More Analysis of Chaotic-prioritized Adaptive Skipping: Tracking the Chaotic group captures critical failure points that average-error thresholds can miss because stable tokens dilute the global metric.Chaotic-prioritized Adaptive Skipping is designed around this concentration of error.
  • F.2. Necessity of Dimensionless Indicators: Feature and error scales vary from hundreds at Step 1 to 0–10 at Step 37, making fixed thresholds on raw metrics unreliable across denoising timesteps.The Dimensionless Drift Indicator E = κ · ∥∆y∥ normalizes displacement with curvature to measure relative instability across phases.

F.3. Ablation on Skipping Strategies · H. More Visual Comparison

The ablation compares CAS with fixed-interval, difference/norm-guided, and curvature-guided skipping, finding CAS achieves the strongest reported quality trade-off. Additional visual comparisons in Figures 10–13 demonstrate WorldCache against existing methods.

  • F.3. Ablation on Skipping Strategies: Fixed-interval skipping yields baseline PSNR 26.18 but cannot adapt to steps with variable difficulty.Its rigid “skip-k-compute-1” schedule limits responsiveness to changing step difficulty.
  • F.3. Ablation on Skipping Strategies: Difference/norm-guided skipping improves to PSNR 26.79 but remains suboptimal because scale variance makes global threshold tuning difficult.The method uses raw feature differences as triggers.
  • F.3. Ablation on Skipping Strategies: Curvature-guided skipping alone performs poorly at PSNR 25.87 because high curvature can coexist with small token displacement.Recomputing in that situation is wasteful, since curvature indicates potential difficulty rather than actual movement.
  • F.3. Ablation on Skipping Strategies: CAS allocates computation when and where the model is most likely to fail by combining potential difficulty with actual drift.This design addresses the limitations of using curvature or raw feature differences alone.
  • F.3. Ablation on Skipping Strategies: CAS achieves the best reported trade-off with PSNR 27.10 and SSIM 0.881.It combines curvature and displacement into a dimensionless metric and prioritizes the chaotic group.
  • H. More Visual Comparison: The paper provides additional visual comparisons to demonstrate WorldCache’s effectiveness against existing methods.These results are presented in Figures 10, 11, 12, and 13.
  • H. More Visual Comparison: Figures 10 and 11 show further visual comparisons between WorldCache and existing methods.Both figures are identified as additional comparisons in the paper.
  • H. More Visual Comparison: Figures 12 and 13 also present visual comparisons between WorldCache and existing methods.The supplied figure captions identify both as more visual comparisons.
Loading 2603.06331v2…