PaperlayerView exact revision on arXiv

Source-linked AI summary

WorldKV: Efficient World Memory with World Retrieval and Compression

Jung Yi, Minjae Kim, Paul Hyunbin Cho, Wooseok Jang, Sangdoo Yun, Seungryong Kim

arXiv:2605.22718v1cs.CV

TL;DR

Persistent world models must preserve scene consistency across revisits without the growing memory and attention costs of full KV-cache history. WorldKV retrieves and compresses evicted KV-cache chunks, matching or exceeding full-KV and memory-trained baselines on revisit fidelity while maintaining real-time inference without fine-tuning.

  • Problem

    Persistent world models must reconstruct the same scene content during revisits, but sliding-window inference evicts the KV-cache memory needed for long-term consistency while full-history attention is costly.

  • Method

    WorldKV is a training-free framework that retrieves scene-relevant evicted KV-cache chunks and compresses each chunk through key-similarity pruning for efficient long-term memory.

  • Results

    WorldKV matches or exceeds full KV-cache attention and memory-trained baselines on revisit fidelity across Matrix-Game-2.0 and LingBot-World-Fast while maintaining real-time inference.

  • Takeaways & Limitations

    WorldKV provides competitive long-term world memory through inference-time KV-cache management without fine-tuning or distillation and at lower memory and attention cost.

  • Takeaways & Limitations

    Because WorldKV operates on a frozen backbone, fidelity remains bounded by pretrained generation quality, and substantially longer rollouts may accumulate visual artifacts.

Abstract

from arXiv · show

Autoregressive video diffusion models have enabled real-time, action-conditioned world generation. However, sustaining a persistent world, where revisiting a previously seen viewpoint yields consistent content, remains an open problem. Full KV-cache attention preserves this consistency but breaks real-time constraints: memory footprint and attention cost grow linearly with rollout length. Sliding window inference restores throughput but discards long-term consistency. We propose WorldKV, a training-free framework with two components: World Retrieval and World Compression. World Retrieval stores evicted KV-cache chunks in GPU/CPU memory and selectively retrieves scene-relevant chunks via camera/ action correspondence, inserting them back into the native attention window without re-encoding. World Compression prunes redundant tokens within each chunk via key-key similarity to an anchor frame, halving per-chunk storage to fit 2x more history under a fixed budget. On Matrix-Game-2.0 and LingBot- World-Fast, WorldKV matches or exceeds full-KV memory fidelity at roughly 2x the throughput, and is competitive with memory-trained baselines without any fine-tuning. Project Page: https://cvlab-kaist.github.io/WorldKV/

1 Introduction

Persistent world generation requires retaining and recalling scene content across time and revisits, but sliding-window inference loses this consistency while full-history KV attention is too costly for real-time use. WorldKV addresses this gap with training-free retrieval and compression of the model’s existing KV cache, achieving efficient long-term memory without architectural changes or fine-tuning.

  • Persistent world memory: Sliding-window inference can hallucinate new content or drift when reconstructing previously explored areas, whereas full KV-cache history reproduces revisited viewpoints.This establishes the central persistence-versus-efficiency problem in autoregressive video world models.
  • Persistent world memory: 8.87 to 3.61 FPS: full-history KV attention slows LingBot-World-Fast over a one-minute rollout as the cache accumulates hundreds of thousands of tokens.Each frame produces 880 to 1,560 tokens, and the cache can exceed GPU VRAM capacity.
  • WorldKV overview: WorldKV is a training-free framework with World Retrieval and World Compression that uses the model’s own KV cache as world memory.The approach avoids building external memory on top of the model and targets efficient long-term memory in autoregressive video world models.
  • World Retrieval: World Retrieval stores evicted KV-cache chunks in GPU/CPU memory and selectively reinserts scene-relevant caches into the native attention window without re-encoding or architectural changes.Camera/action-based and attention-based retrieval strategies are interchangeable components.
  • World Compression: 2× more history: World Compression prunes redundant tokens using Key-Key similarity, reducing each chunk to approximately half its original size while preserving or improving revisit fidelity.The resulting memory fidelity is comparable to full KV-cache attention and can sometimes surpass it.
  • Empirical results: WorldKV matches or exceeds full KV-cache attention and memory-trained baselines on revisit fidelity across Matrix-Game-2.0 and LingBot-World-Fast while maintaining real-time inference.The result is demonstrated across two autoregressive video world models with different scales and requires no fine-tuning.

2 Related Work

Prior work combines autoregressive diffusion with interactive, action-conditioned world generation, while KV-cache management addresses long-context inference costs. However, existing cache methods target language-model decoding rather than dense spatiotemporal video generation.

  • Autoregressive Video Diffusion: Autoregressive video diffusion combines diffusion modeling with autoregressive prediction for long-horizon and streaming video generation.CausVid uses causal autoregressive distillation, Self Forcing trains on self-generated rollouts with KV caching, and Rolling Forcing jointly denoises frames at increasing noise levels.
  • Interactive World Model: Interactive world models predict action-conditioned future frames, progressing from keyboard-and-mouse control toward camera actions, text-controlled events, and diverse long-horizon domains.The cited systems include Matrix-Game-2.0, Hunyuan-GameCraft, Yume-1.5, and LingBot-World.
  • KV Cache Management: KV caches grow linearly with sequence length, motivating fixed-budget management through positional heuristics, attention scores, observation-window estimates, and query-aware retrieval.These approaches reduce language-model decoding costs but are not designed for dense spatiotemporal generation.

3 Preliminaries

This section defines interactive world models as action-conditioned visual generators and introduces autoregressive video diffusion with KV-cache conditioning. These preliminaries establish the transition formulation, causal DiT implementations, and sequential cache updates used later.

  • Interactive World Models: Interactive world models predict future visual observations from a current visual state and an action via a transition distribution.The state belongs to S, the action to A, and the transition maps S × A to distributions over S.
  • Autoregressive Video Diffusion with KV Cache: Attention maps show that the model emphasizes KV caches whose viewpoints overlap with the current action, motivating camera/action-based retrieval of relevant past caches.The illustrated sequence includes Right, Stop, Left, Stop, and Right actions across chunks C0–C10.
  • Interactive World Models: Recent world models implement action-conditioned visual transitions with autoregressive video diffusion built on causal DiT architectures.They condition on either discrete keyboard actions or continuous camera trajectories.
  • Autoregressive Video Diffusion with KV Cache: Autoregressive video diffusion synthesizes long videos sequentially by generating frames or chunks, with each conditional represented by a diffusion process.The examples specify chunks of three frames.
  • Autoregressive Video Diffusion with KV Cache: KV caches store key-value projections of previously generated frames or chunks and condition each new denoising step on prior cached entries.New key-value pairs are appended to the cache for subsequent generation steps.

4 Method

WorldKV combines viewpoint-aware retrieval with key-similarity-based compression on top of sliding-window inference, preserving relevant long-term context while bounding attention and storage costs. It stores evicted chunks, selectively reinserts relevant history, and retains distinctive tokens relative to an anchor frame.

  • Framework Overview: WorldKV addresses full-KV inference’s attention and storage bottlenecks through World Retrieval and World Compression layered onto sliding-window inference.Retrieval bounds the active attention window, while compression reduces stored chunk size.
  • World Retrieval: At generation time, World Retrieval selects the top-k history chunks most relevant to the current camera/action state and inserts them into the attention window.The window also contains sink caches, recent caches, and the currently generated denoising chunk.
  • World Compression: World Compression designates the first frame of each F-frame chunk as an anchor and compares non-anchor keys against its anchor-frame keys.Key-Key cosine similarity identifies overlap, while divergent keys indicate newly revealed or dynamic content.
  • World Compression: With F = 3 and P = 25% retention, each chunk shrinks from 3T to approximately 1.5T tokens, achieving 2× storage efficiency.The method retains all anchor tokens plus the least-similar non-anchor tokens, which preserve content not captured by the anchor.

5 Experiments

WorldKV is evaluated on a 60-pair long-horizon benchmark across two autoregressive video world models, using revisit consistency, image-quality, and throughput metrics. It preserves or improves memory fidelity over native and memory-trained baselines while retaining substantially higher throughput, and ablations show moderate compression and broader historical coverage are most effective.

  • Benchmark: The benchmark contains 60 diverse scene-trajectory pairs with repetitive revisits, traversals, and at least one loop-closure event per scene.Scenes use real-world videos, game recordings, and AI-generated images.
  • Base Models: Experiments cover LingBot-World-Fast, a 14B-parameter long-video model, and Matrix-Game-2.0, a 1.3B-parameter short-context model.Native inference uses full KV-cache attention for LingBot-World-Fast and a 6-frame sliding window for Matrix-Game-2.0.
  • Main Results: WorldKV closely approaches Full KV across all metrics on LingBot-World-Fast at roughly 2 × the throughput.Full KV already provides strong memory for this model, which was distilled from a long-video teacher.
  • Main Results: WorldKV outperforms sliding-window and full KV-cache attention across all metrics on Matrix-Game-2.0 by retrieving only scene-relevant KV caches.Full KV degrades because this short-sequence model accumulates degraded caches during out-of-distribution long-horizon generation.
  • Baseline Comparison: WorldKV outperforms WorldPlay and Yume-1.5 on LPIPS, PSNR, and FID with comparable SSIM on LingBot-World-Fast, without memory-specific training.On Matrix-Game-2.0, WorldKV achieves competitive performance against memory-trained baselines.
  • Compression Ablation: Moderate intra-chunk compression, 3 →1.5 or 3 →2.0, performs strongly, while 6 chunks →3 chunks outperforms uncompressed retrieval under a fixed 3-chunk budget.Anchor-only compression loses distinctive non-anchor information, whereas aggressive 9 →3 compression degrades performance.

6 Conclusion

WorldKV is a training-free framework that combines KV-cache retrieval and compression to enable consistent scene revisits during real-time autoregressive video world-model inference. Across two differently scaled world models, it achieves memory fidelity competitive with full KV-cache attention and memory-trained baselines without fine-tuning or distillation, while reducing memory and attention cost.

  • Conclusion: WorldKV combines KV-cache retrieval and compression for efficient world memory in autoregressive video world models.The framework is training-free.
  • Conclusion: WorldKV enables consistent scene revisits while maintaining real-time inference.
  • Conclusion: Across two world models of different scales, WorldKV achieves memory fidelity competitive with full KV-cache attention and memory-trained baselines without fine-tuning or distillation.It also operates at lower memory and attention cost.

7 Limitations & Future Work · Appendix · A Selective Retrieval Outperforms Full KV Cache Attention

WorldKV’s fidelity remains bounded by its frozen backbone, and long rollouts can accumulate artifacts; CPU offloading reduces VRAM use but currently prevents real-time generation. In some revisited-viewpoint cases, selective retrieval and compression outperform full KV-cache attention, likely by avoiding attention dilution.

  • 7 Limitations & Future Work: WorldKV’s visual fidelity is bounded by the generation quality of its frozen pretrained world-model backbone.The method operates only on the backbone’s KV cache at inference time.
  • 7 Limitations & Future Work: Longer-than-training rollouts may still accumulate visual artifacts from autoregressive error accumulation.This limitation arises despite WorldKV’s efficiency and fidelity improvements for long-horizon inference.
  • 7 Limitations & Future Work: CPU offloading stores all KV caches in host memory and loads only attention-relevant chunks onto the GPU, bounding VRAM regardless of rollout length.This provides a complementary route for reducing VRAM consumption.
  • 7 Limitations & Future Work: Host-device transfer latency during retrieval currently prevents real-time generation with CPU offloading.Reducing this latency is left for future work toward bounded-memory, real-time multi-minute world generation.
  • A Selective Retrieval Outperforms Full KV Cache Attention: WorldKV preserves scene-specific details more faithfully than full KV-cache attention across repeated viewpoint visits in the illustrated long-horizon rollout.The comparison is shown in Figure 6 across viewpoints revisited at different times.
  • A Selective Retrieval Outperforms Full KV Cache Attention: In some LingBot-World-Fast cases, WorldKV reconstructs revisited viewpoints more faithfully than attending to the full history.The paper hypothesizes that growing histories dilute attention across redundant or viewpoint-irrelevant entries.
  • A Selective Retrieval Outperforms Full KV Cache Attention: WorldKV restricts attention to retrieved viewpoint-relevant chunks and prunes near-duplicate entries within each dense spatiotemporal memory chunk.The paper presents this as the first empirical observation of selective KV retrieval and compression outperforming full KV-cache attention in autoregressive video world models.

B Key-Key Similarity Visualization

Key-Key similarity identifies redundant visual content relative to an anchor frame while preserving newly revealed spatial regions and dynamic object changes. Visualization across Matrix-Game-2.0 and LingBot-World-Fast supports its use as a cue for World Compression.

  • Key-Key Similarity Visualization: The bottom 12.5% least-similar tokens in the 2nd and 3rd frames are selected by cosine similarity to the first frame’s anchor keys.Figure 7 visualizes this selection across Matrix-Game-2.0 and LingBot-World-Fast.
  • Key-Key Similarity Visualization: In camera-rotation cases, low-similarity tokens concentrate on newly revealed regions at image boundaries that are absent from the anchor frame.These tokens correspond to information not present in the anchor and are retained by World Compression.
  • Key-Key Similarity Visualization: Key-Key similarity distinguishes redundant visual content from newly visible information, which should be preserved during pruning.High similarity corresponds to redundancy, while low similarity identifies distinctive content.
  • Key-Key Similarity Visualization: Low-similarity tokens also appear on rotating windmill blades, indicating that pruning can preserve dynamic temporal information as well as static spatial differences.This occurs in the backward camera-motion case shown in Fig. 7(d).

C Retrieval Algorithm Ablations

This section evaluates World Retrieval independently of its retrieval algorithm by comparing camera/action-based and query-based chunk selection against sliding-window inference. The main strategy uses camera pose or accumulated discrete-action similarity, while the alternative ranks chunks by attention to the current denoising query.

  • Retrieval strategies: World Retrieval is retrieval-algorithm agnostic, and Table 3 compares two retrieval strategies against the sliding-window baseline.The evaluated strategies are camera/action-based retrieval and query-based retrieval.
  • Retrieval strategies: The main-experiment strategy selects chunks using camera pose or accumulated discrete-action similarity.This camera/action-based method is described as the strategy used in Sec. 5.1.
  • Retrieval strategies: The alternative query-based strategy ranks stored chunks by their attention scores with respect to the current denoising query.This approach is inspired by Deep Forcing [31].

D Increasing Retrieval Chunk Size · E WorldKV in Inspatio-World

Retrieving more historical KV-cache chunks generally improves memory fidelity on Matrix-Game-2.0 and LingBot-World-Fast, while World Compression expands retrieval coverage under a fixed attention-window budget. WorldKV is also evaluated on Inspatio-World to test generalization beyond the two base models.

  • D Increasing Retrieval Chunk Size: D Increasing Retrieval Chunk Size: More retrieved chunks generally improve LPIPS, PSNR, and SSIM on both Matrix-Game-2.0 and LingBot-World-Fast.The improvement indicates that broader access to historical KV caches enhances revisit consistency.
  • D Increasing Retrieval Chunk Size: D Increasing Retrieval Chunk Size: World Compression reduces GPU/CPU storage costs and allows more historical chunks to fit within a fixed attention-window budget.This expands the amount of history available for retrieval.
  • D Increasing Retrieval Chunk Size: D Increasing Retrieval Chunk Size: Broader historical-cache access improves memory fidelity as retrieval coverage increases.Figure 8 reports this trend across both evaluated models.
  • E WorldKV in Inspatio-World: E WorldKV in Inspatio-World: WorldKV is applied to Inspatio-World, a video-to-video 4D world model that generates novel-view sequences conditioned on an input video.This experiment tests whether WorldKV generalizes beyond the two base models used in the main evaluation.
  • E WorldKV in Inspatio-World: E WorldKV in Inspatio-World: Inspatio-World stores the input video in the sink region of its attention window but lacks memory preservation for newly generated scenes.WorldKV addresses a memory gap that the native model does not cover.
  • E WorldKV in Inspatio-World: E WorldKV in Inspatio-World: Like Matrix-Game-2.0, Inspatio-World was not trained on long video sequences.The passage identifies this shared training condition when describing the generalization experiment.

F Additional Qualitative Results · G Broader Impact

WorldKV maintains consistent scene revisits across long-horizon trajectories with qualitative results comparable to full KV-cache attention and memory-trained baselines. Its reduced long-term-memory cost may broaden practical use while retaining risks associated with generative video and interactive world models.

  • F Additional Qualitative Results: WorldKV maintains consistent scene revisits across long-horizon trajectories with efficient inference and qualitative results comparable to full KV-cache attention and memory-trained baselines.Additional examples are shown in Figure 10.
  • G Broader Impact: WorldKV reduces the cost of maintaining and retrieving long-term visual memory for autoregressive video world models.The method may reduce the memory and computation required by full-history attention.
  • G Broader Impact: WorldKV may improve the practicality of interactive simulation, gaming, embodied AI, and robotic training environments.
  • G Broader Impact: With WorldKV applied to Inspatio-World, the same scene content is preserved consistently across views without fine-tuning, unlike the unmodified model's loss of scene memory upon revisit.
  • G Broader Impact: Potential risks include misleading synthetic content, unlabeled simulated media, and more realistic persistent virtual environments.Responsible deployment should include provenance, disclosure, and appropriate access controls in public-facing applications.
  • F Additional Qualitative Results: Figure 10 compares methods frame by frame across two trajectories.
Loading 2605.22718v1…