PaperlayerView exact revision on arXiv

Source-linked AI summary

Addressable Memory for Video World Models

Xindi Wu, Sven Elflein, James Lucas, Olga Russakovsky, Laura Leal-Taixé, Despoina Paschalidou, Jonathan Lorraine, Aljoša Ošep

arXiv:2608.07408v1cs.CVcs.LG

TL;DR

Long-horizon video world-model memory fails when temporal RoPE offsets move beyond the training range and compressed rotated keys lose information through phase cancellation. WorldTrace addresses both issues with distinct in-distribution virtual positions and complementary memory writers, improving temporal consistency by +15.5% and episodic recall by +19.5% without retraining.

  • Problem

    Long-horizon generation suffers because cached content becomes hard to address outside the training horizon, while naive RoPE-space compression can make summaries uninformative.

  • Method

    WorldTrace is a training-free memory framework that assigns summary slots distinct in-distribution virtual positions and uses Field and Landmark writers for different visual traces.

  • Results

    +15.5% Temporal Consistency at N=48 and +19.5% Scene Consistency on LoopBench ABA loops are reported without retraining.

  • Takeaways & Limitations

    Addressable memory extends visually persistent generation by keeping distant traces retrievable while supporting either temporal coherence or episodic recall.

  • Takeaways & Limitations

    WorldTrace is designed for temporal-RoPE autoregressive models with a fixed KV-cache budget, while Field can blur scene details and Landmark depends on detecting and retaining the right scene entries.

Abstract

from arXiv · show

We study visual persistence in interactive video world models. These models rely on a Key-Value (KV) cache as a growing visual memory to carry forward previously generated frames. However, we find that models can no longer reliably address stored content once rollouts extend beyond the training horizon, because temporal Rotary Positional Embeddings (RoPE) offsets then fall outside the range seen during training and the model struggles to retrieve the relevant visual information through attention. Moreover, naively compressing the cache in the RoPE-rotated space corrupts memory by averaging together incompatible positional phases. To address this, we propose WorldTrace, a training-free memory framework for long-horizon visual persistence. WorldTrace keeps compressed memory addressable by assigning each summary slot a distinct, in-distribution virtual position. Within this addressable cache, we study two memory compression approaches: WorldTrace-Field compresses history for temporal coherence, while WorldTrace-Landmark stores verbatim scene traces at detected transitions for episodic recall. We further introduce LoopBench, a benchmark evaluating whether a compressed cache can reconstruct a previously visited scene after a long detour. WorldTrace-Field improves temporal consistency by +15.5%, and WorldTrace-Landmark improves episodic recall by +19.5% on LoopBench, extending visually persistent generation without retraining.

1 Introduction

Long-horizon visual persistence fails when cached content becomes difficult to address beyond the training horizon. WorldTrace assigns compressed memories in-distribution positions and uses complementary writers for coherence and episodic recall.

  • Motivation: Interactive video world models use growing KV caches to preserve previously generated visual context during exploration.This supports applications such as interactive game engines and closed-loop robot simulators.
  • Problem: Beyond the training context length, out-of-distribution temporal offsets degrade attention-based retrieval from stored memories.Past content can remain in the cache yet become difficult for the model to access reliably.
  • Approach: WorldTrace is a training-free approach that assigns each memory entry a fixed, in-distribution temporal position.The cache combines a recent window with summary slots so distant observations remain retrievable.
  • Approach: WorldTrace-Field preserves coarse history for temporal coherence, whereas WorldTrace-Landmark preserves sparse scene-entry traces for episodic recall.The variants address different choices about what visual information should remain in the compressed cache.
  • Results: +15.5% temporal consistency and +19.5% episodic recall demonstrate gains over the default sliding-window cache without retraining.LoopBench evaluates whether compressed memory can recover a previously visited scene after a long detour.

2 Rethinking Memory in Video World Models

Long-horizon memory has two coupled bottlenecks: cached tokens must remain addressable, and compressed summaries must remain informative. Temporal RoPE extrapolation harms retrieval, while averaging rotated keys can cancel their content signals.

  • Two bottlenecks: Long-horizon memory depends both on retaining relevant context and on allowing future queries to attend to it.The paper separates these concerns into addressability and informativeness of compressed summaries.
  • Addressability: Cached tokens become unreadable when their temporal RoPE offsets exceed the range encountered during training.The fastest temporal components degrade into positional noise first, while slower components may retain usable signal.
  • Design implication: WorldTrace targets an N-slot compressed cache by assigning every slot a distinct, in-distribution slot-rank position.This avoids the positional collisions that arise when block-relative positions cap distant summaries at the same minimum position.
  • Content compression: Naive averaging of RoPE-rotated keys can cancel signals because frames from distant timestamps carry incompatible rotation phases.Nearby timestamps have closer angles and are less vulnerable to this cancellation.
  • Design implication: Addressable positions cannot compensate for corrupted summaries, and informative summaries cannot compensate for out-of-distribution positions.The two bottlenecks therefore need to be addressed together.

3 WorldTrace

WorldTrace maintains a bounded cache with recent verbatim frames and addressable summary slots. It assigns distinct virtual positions and uses structured projections, including canonical-space averaging for temporal coherence and trace preservation for recall.

  • Cache structure: WorldTrace partitions the attention window into N_r recent verbatim frames and N_s summary slots, with N_s + N_r = L_attn.The summary cache stores compressed distant history while the recent window retains the latest frames exactly.
  • Virtual position assignment: Each summary slot receives a distinct, in-distribution, horizon-stable virtual position determined by its rank relative to the current query.The design keeps positions distinguishable regardless of the absolute generation horizon.
  • Canonical key domain: WorldTrace removes source RoPE phases before storing compressed keys and re-rotates them at the target virtual slot position.Canonical storage avoids mixing incompatible source phases with the phase expected at the summary slot.
  • Structured projection: The compression design can be viewed as structured sparse attention, where a projection matrix determines which historical frames contribute to each compressed slot.This formulation motivates the Field and Landmark variants as different attention patterns.
  • WorldTrace-Field: WorldTrace-Field uniformly averages contiguous temporal groups in canonical space to produce rotation-invariant traces for temporal coherence.Its slot values are means of the source-frame values, and groups are recomputed as the rollout grows.
  • Compression behavior: Canonical-space averaging avoids the phase-cancellation failure that can shrink compressed keys and suppress attention scores.The stated guarantee concerns pre-softmax scores, not the post-softmax attention weights.

4 Experiments

The experiments test whether addressable position assignment and content compression improve long-horizon coherence and episodic recall. WorldTrace-Field preserves coherence, while WorldTrace-Landmark improves scene reconstruction across LoopBench revisits.

  • Setup: The evaluation tests position assignment, compression, temporal coherence, and episodic recall using MG2 and LoopBench.Metrics include TempSSIM, Local Scene Drift, and Position-Aligned CLIP (PAC).
  • Setup: LoopBench varies waypoint count, rollout length, camera orientation, and multi-revisit depth while scoring regenerated return frames against matched original scenes.Its geometries include ABA, ABCA, and ABCDA, with each path returning to scene A.
  • Coherence: WorldTrace-Field: Canonical averaging with slot-rank positions keeps summary slots individually addressable, whereas horizon-dependent or capped positions cause retrieval failures as rollouts grow.WorldTrace outperforms Block-relative by +5.9% and +2.8% TempSSIM and Centroid-linear by +9.5% and +13.8% at N=8 and N=16.
  • Coherence: WorldTrace-Field: At N=32, WorldTrace-Field achieves TempSSIM 0.613 versus 0.571, while at N=48 it is best on both TempSSIM and Scene Drift.In qualitative rollouts, baselines drift by n=18 while WorldTrace-Field remains coherent.
  • Episodic Recall: WorldTrace-Landmark: WorldTrace-Landmark improves PAC over sliding-window caching across reported conditions, with larger gains for longer KV gaps and multiple separating waypoints.At extended horizons, verbatim landmark recall sustains PAC≈0.99 at N=256 versus 0.610 for canonical-K anchoring.
  • Ablation: Canonical key averaging reduces LatentDiff from 0.312 to 0.233 at N_s=4, showing that averaging in unrotated space avoids RoPE phase cancellation.Naive averaging worsens as the number of summary slots increases, while canonical averaging remains comparatively stable.

5 Conclusion

WorldTrace frames long-horizon visual memory as an addressability problem and uses fixed in-distribution virtual positions with complementary coherence and episodic-recall writers. Its results support designing memory around both how past content is addressed and what visual traces are stored.

  • Long-horizon failure arises because compressed memories become effectively unreadable when temporal RoPE offsets move beyond the training horizon.
  • WorldTrace keeps distant traces addressable through fixed in-distribution virtual positions without retraining.
  • WorldTrace-Field provides coherence-oriented summaries, whereas WorldTrace-Landmark preserves verbatim traces for episodic recall.
  • WorldTrace is limited to temporal-RoPE autoregressive models with fixed KV-cache budgets and two simple history projections with distinct trade-offs.Field can blur specific scene details, while Landmark depends on detecting and retaining the right scene entries.

Appendices

The appendices formalize WorldTrace as projected attention over a growing history, with structured projection matrices defining cache variants and virtual-position re-encoding separating addressability from compression. The variants span sparse landmark selection to dense field averaging, while the analysis treats repositioning and compression as distinct error sources.

  • WorldTrace compresses a T-frame history into L_attn ≪ T effective tokens using a projection matrix P ∈ R^(L_attn×T).
  • Projected attention replaces the full sequence with prototype tokens whose rows in P are distributions over past frames.The compressed output is computed from attention over projected keys and projected values.
  • WorldTrace applies projection training-free at inference, merges keys in the canonical domain to avoid RoPE phase cancellation, and re-encodes slots at rank-based virtual positions.
  • Sliding-window P selects recent frames, optionally retaining the first frame as an attention sink while selecting the remaining recent frames verbatim.
  • WorldTrace-Field uses identity rows for recent slots and uniform-average rows for groups of earlier frames.
  • WorldTrace-Landmark uses one-hot rows at detected scene-entry frames, overwriting and evicting landmarks as generation proceeds.
  • The cache projection changes at each generation step as the recent window slides and summary rows are filled or overwritten.
  • Compression error separates key repositioning from key compression, with slot-rank re-encoding trading uncontrolled OOD failure for bounded in-distribution position shifts.

B.5 When Are WorldTrace Projections Close to Optimal?

WorldTrace projections are close to optimal only for query families that match their stated coherence or recall conditions. The analysis also identifies limits from compression capacity, relaxed objectives, and long-horizon RoPE behavior.

  • A compressed cache cannot reproduce every single-frame query when L_attn/T is small, because each frame receives at most an L_attn/T attention fraction.
  • WorldTrace-Field: WorldTrace-Field exactly preserves recent-frame attention and bounds old-history error by ‖α_q^old‖_1 ε_q for ε_q-coherent queries.
  • WorldTrace-Field: WorldTrace-Field is within ε_q of the best projection for coherent query families because canonical averaging creates RoPE-invariant group means.
  • WorldTrace-Landmark: WorldTrace-Landmark is near-optimal for δ_q-covered recall queries when detected landmarks copy the targeted frames, with reliability requiring at most N_s simultaneously relevant scenes.
  • Scope: The theoretical bounds are conditional characterizations for defined query families, not unconditional optimality claims, and their premises are not measured on the pretrained model.
  • RoPE and projection design: Mean-attention preservation separates canonical content averages from virtual positions, avoiding source-phase contamination while keeping summaries addressable.

D.5 Memory and KV Cache for Video World Models

WorldTrace jointly assigns positions and writes compressed content inside a bounded KV cache, unlike methods that address only one side or use external memory. Its unified cache can allocate slots between coherence and episodic recall without retraining.

  • WorldTrace jointly designs in-distribution, distinct slot positions and canonical-domain summary writes in a fixed O(1) cache.
  • Training-time extensions retrain generators or add long-context architectures, whereas WorldTrace operates at inference without retraining.
  • Geometry-conditioned methods select memory using pose, actions, or scene geometry, while WorldTrace uses geometry-free slot-rank positions.
  • External-memory systems retrieve frames outside the KV cache, while WorldTrace retains recent and summary memory under one O(1) budget.
  • Unified allocation: The shared slot-rank interface lets WorldTrace-Field and WorldTrace-Landmark occupy the same cache, with non-landmark slots remaining Field averages.
  • Unified allocation: At four summary slots and two recent slots, allocating more slots to landmarks raises recall, while remaining Field slots maintain coherence at or above the sliding-window baseline.

E.2 Training-Free Positional Baselines

Training-free positional baselines show that keeping offsets in distribution helps, but WorldTrace-Landmark combines positional control with verbatim content and bounded memory. This combination substantially outperforms alternatives on long ABA revisits.

  • MemRoPE is surpassed by Landmark+Block-relative because verbatim canonical keys preserve stronger query-key similarity than smoothed averages.
  • Replacing MemRoPE’s Block-relative positions with WorldTrace positions while retaining EMA content degrades performance because write and read offsets disagree.
  • +22% over the sliding-window baseline at N=32 is achieved by YaRN, but its cache grows as O(N) and reaches OOM around N=100.
  • WorldTrace-Landmark reaches 0.964 versus 0.490 at N=32 and remains near-constant through N=512 in O(1) memory.
  • Within compression-only methods, WorldTrace-Field beats the sliding-window baseline at N=32 to 64, while verbatim recall forms the strongest retained-content tier.

E.4 Cross-Architecture Experiments

Cross-architecture experiments indicate that WorldTrace-Landmark’s recall benefit extends across model scale and architecture, while Field’s benefit depends on whether other conditioning already supplies recall cues. Memory and writer studies preserve constant cache behavior.

  • Cross-architecture results: WorldTrace-Landmark improves over sliding-window retention by +8.9%, +14.1%, and +7.3% at 4×, 6×, and 8× horizons on LingBot-World.
  • Cross-architecture results: WorldTrace-Landmark shows no significant gain at 2×, with −0.006 absolute and p>0.1, but improves from 4× onward.
  • Cross-architecture results: WorldTrace-Field stays close to sliding-window performance on LingBot-World because Plücker camera conditioning supplies the primary recall signal.
  • Slot allocation: With six total slots, N_s≤2 collapses toward the sliding-window baseline because scene-A landmarks are overwritten or evicted during the B-side traversal.
  • Memory and writers: WorldTrace attends over the same N_s+N_r entries at every step, keeping attention cost constant and peak GPU memory nearly flat with rollout length.
  • Memory and writers: WorldTrace-Field matches the sliding-window baseline’s GPU memory, while Landmark adds only a constant scene-anchor cost.
  • Memory and writers: The streaming Field writer folds each evicted frame into a per-slot accumulator once, retains most coherence gains, and keeps state constant in rollout length.

F LoopBench: Loop Memory Benchmark

LoopBench evaluates whether autoregressive world models can reconstruct previously visited scenes after detours by varying path structure, rollout length, camera orientation, and revisit depth. It uses generated first visits as references and tests return-frame similarity without external videos.

  • Benchmark design: LoopBench measures how closely return frames reproduce frames from the initial visit to the same location.The rollout itself supplies both the reference and prediction, so no external reference video is required.
  • Benchmark design: Longer rollouts push revisited scenes deeper into the compressed cache, testing retrieval under extended temporal separation.The benchmark varies rollout length while holding other properties unchanged for controlled comparisons.
  • Benchmark design: Four difficulty axes vary waypoint count, rollout length, camera orientation, and multi-revisit depth.These axes respectively control intervening locations, cache depth, view changes, and repeated visits.
  • Benchmark design: Camera-orientation tiers keep the agent at one location while panning away and returning, testing recall of the initial view.Qualitative results compare sliding-window outputs with WorldTrace-Landmark on 90° pan-and-return sequences.

G.3 Hyperparameters, Compute, and Evaluation Protocol

The section specifies WorldTrace’s cache-update mechanics, inference settings, evaluation protocol, and runtime overhead. Experiments use fixed-capacity summary and recent-window slots, with canonical-space compression and virtual-position RoPE rereading.

  • Qualitative evaluation: The qualitative Fig. 4 comparison follows one continuous Matrix-Game 2 camera trajectory over N=48 AR chunks, with marked waypoints shared by all methods.The trajectory runs from the initial scene to a distinct end pose.
  • Cache update: WorldTrace stores canonical keys and applies each summary slot’s RoPE rotation only when attention scores are computed at its virtual position.This lets slot positions recompute automatically as the query position advances.
  • Hyperparameters: N_s=2 and N_r=4 are used for short-horizon experiments, while N_s=4 and N_r=2 are used for longer-horizon, LoopBench, and ablation experiments, keeping N_s+N_r=L_attn=6.Both configurations preserve the same total attention capacity.
  • Compression: WorldTrace-Field averages each uniformly grouped history bucket after unrotating its keys to canonical space, then re-rotates the summary at its virtual position.The recent window remains verbatim while evicted entries update the corresponding summary slot.
  • Evaluation protocol: Each method is evaluated on 100 videos generated from distinct initial frames, with multiseed experiments using seeds {0, 42, 123, 456, 789}.Experiments run on a single NVIDIA A100 80 GB GPU.
  • Compute: WorldTrace-Field generates at approximately 0.9 s per AR chunk, VAE decoding adds approximately 2.5 s, and all methods remain within 6% of the sliding-window baseline.The reported full experiment set required approximately 100 GPU-hours on single A100 80 GB GPUs.

H.1 Limitations

The section frames WorldTrace as a fixed-budget, inference-time cache mechanism whose extensions target geometry, policy learning, budget allocation, and benchmark scaling. Its scope remains bounded by the pretrained video model and the attention-window budget.

  • Limitations: WorldTrace changes only retained cache information and virtual RoPE positions at inference time, while pretrained weights, visual quality, motion dynamics, and action following remain unchanged.The method assumes an autoregressive pretrained video model.
  • Future directions: Geometry-aware canonical keys could align observations to a shared scene coordinate system before averaging, extending the current unrotate-and-rerotate primitive.The passage connects this direction to camera-pose warping methods such as MosaicMem and UCM.
  • Future directions: Learned scene-entry policies could adapt landmark commits, landmark selection, and the fixed split between summary and recent slots across backbones or horizons.Candidate training signals include action discontinuities, agent-pose changes, and scene-segmentation logits.
  • Limitations: WorldTrace remains within the attention window, with N_s+N_r=L_attn keeping every summary slot in-distribution.Context-extension and position-aware fine-tuning are proposed to relax this fixed-budget constraint.
  • Benchmark scope: LoopBench can scale through longer paths, more waypoints, and richer camera trajectories, while evaluating other world models would test their revisit-memory mechanisms.The benchmark is intended to complement WorldScore, MIND, and VBench-2.0.
  • Limitations: WorldTrace does not expand the underlying video model’s generative capabilities, although its constant memory budget lowers the compute cost of building on pretrained generators.The relevant considerations therefore remain those attached to pretrained video world models.
Loading 2608.07408v1…