PaperlayerView exact revision on arXiv

Source-linked AI summary

LOCI: Spatial Linear Memory for Streaming World Models

Ji Xia, Tingting Liao, Xuezhi Liang, Hao Li, Guangyi Liu

arXiv:2609.40222v1cs.CV

TL;DR

Video world models need both to retain past visual evidence and retrieve observations relevant to a camera revisit, but explicit KV storage grows with history while recurrent memory compresses it. LOCI combines camera-conditioned recurrent memory with retained historical KV, using recurrent readouts to shape later retrieval. It improves revisit fidelity in the reported evaluations and streams with bounded memory at constant memory use.

  • Problem

    At a revisit, a world model must preserve scene appearance and retrieve spatially relevant evidence, although KV history grows with trajectory length and fixed-size recurrent memory does not preserve directly accessible individual observations.

    Source: §1 · p. 2

  • Method

    LOCI combines projectively conditioned recurrent memory with retained historical KV, letting recurrent readouts shape the queries used by later historical-attention blocks.

    Source: §1 · pp. 2–3

  • Results

    On MIND, LOCI has the best MSE, PSNR, and SSIM among evaluated world models and improves over same-recipe full softmax under the same bounded KV budget, raising full-segment PSNR by 0.89 dB.

    Source: §1 · p. 3, §6 · p. 10

  • Takeaways & Limitations

    A bounded bank of retained observations lets LOCI stream at constant memory while remaining more faithful than same-recipe full softmax under the same KV budget.

    Source: §1 · p. 3, §6 · p. 10

    Evaluation uses Unreal Engine held-out trajectories rather than real-world captures, absolute fidelity is low for all models, and the study uses one 5B backbone with 5,000 fine-tuning updates.

    Source: §6 · p. 10

Abstract

from arXiv · show

When a camera revisits a previously observed region, a video world model should reproduce what was there before. This requires both remembering past observations and retrieving the right one for the current viewpoint. Key-value caches preserve visual detail but grow with video length; recurrent memory is compact but compresses history into a fixed-size state, so individual past observations are no longer directly accessible. We introduce LOCI, a hybrid spatial-memory architecture that keeps both representations. In half of the transformer blocks, main attention keeps a key-value cache of past observations; in the other half, it is restricted to the current chunk and complemented by a recurrent linear-attention memory whose reads and writes are conditioned on projective camera geometry, so viewpoint enters both memory addressing and stored content. Recurrent readouts flow into subsequent cache-backed blocks and supply their queries with accumulated scene context. On the public MIND memory benchmark and on held-out recorded trajectories, LOCI reproduces revisited content more faithfully than representative world models and a same-recipe full-softmax model; with full history, it lowers peak memory at equal length by about 30% relative to full softmax. With a bounded bank of retained observations, it streams long videos at constant memory and remains more faithful than full softmax under the same budget.

1 INTRODUCTION

LOCI pairs camera-conditioned recurrent context with retained historical key–value evidence, allowing accumulated scene context to inform retrieval of observations relevant to a revisit. It improves revisit fidelity over evaluated world models and a same-recipe full-softmax model, including under a bounded memory budget.

1 INTRODUCTION

  • The central challenge is that spatial relevance need not follow temporal recency: a revisit may require an observation outside the recent context.Historical KV preserves individual evidence but does not ensure that later queries retrieve relevant evidence; recurrent compression can lose scene-specific detail.Source: §1 · p. 2
  • LOCI combines a camera-conditioned recurrent summary of the full history with retained KV observations, and lets recurrent readouts shape queries in later historical-attention blocks.The hybrid uses channel-wise retention once per chunk, so explicit forgetting follows elapsed video time rather than token order.Source: §1 · pp. 2–3
  • LOCI improves reference PSNR at revisits by 0.62 dB on held-out Unreal Engine trajectories and by 0.99 dB on set A with the same bounded retained observations.On all MIND prediction segments under the same bounded budget, full-segment PSNR improves by 0.89 dB (95% interval [+0.64, +1.15]), with lower LPIPS and MSE.Source: §1 · pp. 2–3
  • On MIND, LOCI beats same-recipe full softmax in all three tested settings; under the same bounded KV budget, it raises PSNR by 0.89 dB and wins 44 of 50 segments.The MIND comparison also reports lower MSE in each setting; full-history LOCI is zero-shot and outperforms the reported MSE, PSNR, and SSIM values for MIND-trained GIM-World.Source: §1 · p. 3
  • With full-history access, LOCI is more locally consistent than full softmax at short-horizon revisits on MIND.In an extended MIND comparison, chunk-level rather than per-token retention lowers local error at revisits more than 20 s apart.Source: §1 · p. 3

2 RELATED WORK

Prior work retrieves explicit historical observations through geometric, pose-indexed, or content-based selection, while other models rely on compressed recurrent memory. LOCI combines retained observation-level KV with a recurrent state whose readout shapes downstream queries.

2 RELATED WORK

3 PRELIMINARIES

The preliminaries describe Kimi Delta Attention as a fixed-size key-by-value state with retention and delta updates, then introduce relative positional encoding and its projective camera extension. LOCI applies projective conditioning to recurrent memory.

3 PRELIMINARIES

  • Kimi Delta Attention maintains a fixed-size state S ∈ R^(d_k×d_v) using channel-wise retention and token-level delta corrections.In the stated parameterization, the write strength is β_t = 2σ(b_t) ∈ (0, 2).Source: §3 · p. 4
  • Retention controls which key channels persist, while delta residuals correct the value predicted at a key; queries read the state as S^⊤q.The chunk-wise read-before-write schedule is specified in Sec. 4.2.Source: §3 · p. 4
  • RoPE makes query–key interaction depend on relative position, and PRoPE extends positional conditioning to camera geometry.The stated relative projective transforms make camera dependence invariant to a common change of world coordinates; the paper applies projective conditioning to recurrent memory.Source: §3 · p. 4

4 LOCI

LOCI interleaves recurrent-memory blocks with blocks that directly attend to retained history. Camera geometry conditions recurrent memory, chunk-synchronous reads and writes govern its updates, and dense or bounded sparse attention controls explicit historical access.

4 LOCI

  • The 30-block architecture uses 15 hybrid blocks with intra-chunk attention and recurrent KDA, and 15 blocks with explicit historical softmax attention.All blocks also keep a UCPE camera-attention branch; at equal retained-frame budget, main-attention KV is stored in 15 rather than 30 layers. Recurrent readouts update token features that later historical-attention blocks use to form queries.Source: §4 · p. 4

4.1 PROJECTIVE CAMERA ENCODING FOR RECURRENT MEMORY

  • PRoPE conditions recurrent queries, keys, and values on each token’s camera ray, so memory addressing and stored associations depend on viewpoint.The projection uses camera pose and normalized intrinsics; projective transforms provide a cue distinct from temporal proximity when a previously observed surface becomes relevant again.Source: §4.1 · pp. 4–5

4.2 CHUNK-SYNCHRONOUS MEMORY DYNAMICS

  • Each chunk reads the preceding recurrent state while local softmax handles within-chunk interactions; its tokens then update the state sequentially with the delta rule.A learned gate combines recurrent readout with local attention, and the scan’s final state becomes the next chunk’s state.Source: §4.2 · p. 5
  • Retention is applied once per chunk, while delta corrections remain token-wise, so explicit forgetting advances by chunk rather than raster position.Training scans noised chunks from zero state; sampling reads a committed state during denoising and commits the generated chunk afterward.Source: §4.2 · p. 5

4.3 DENSE AND BOUNDED SPARSE HISTORICAL ACCESS

  • Dense mode attends over the evaluated prefix; bounded sparse mode keeps a recent window and diverse panorama bank alongside the recurrent state.The sparse set includes a conditioning frame, up to 20 panorama frames, up to 8 recent frames, and the 5-frame current chunk; discarded observations are not archived, while both history stores remain bounded.Source: §4.3 · p. 6

5 EXPERIMENTS

The experiments use matched 5B models and evaluate MIND plus held-out Unreal Engine trajectories against recorded-video ground truth. LOCI outperforms external models and full softmax at revisits while extending long-video generation under bounded memory.

5.2 COMPARISON WITH SOTA

  • On MIND, LOCI is 1.82 dB PSNR above HY-WorldPlay among all-segment models; on held-out revisits, it significantly beats six of seven external models in LPIPS and five in PSNR.With bounded sparse history on MIND, LOCI also has the lowest MSE and highest PSNR and SSIM in the compared group.Source: §5.2 · p. 6

5.3 ABLATION STUDY

  • Under an identical bounded KV budget, LOCI raises PSNR by 0.89 dB and is better in 44 of 50 segments; with full history, it completes seven 100–138 s prediction segments that exhaust full softmax GPU memory.Table 2 compares the models under the same backbone, data, recipe, and update count.Source: §5.3 · pp. 7–8, Table 2 · p. 8
  • On four held-out trajectories, LOCI beats full softmax at returns to places first seen at least 60 s earlier, with revisit PSNR gains of +0.68 to +1.93 dB and lower LPIPS on each.Both models receive 68–245 s of recorded history under the same bounded access and generate the remaining 45–60 s.Source: §5.3 · p. 8, Figure 5 · p. 8
  • Converting a trained full-softmax model to the hybrid layout does not match directly training the hybrid, and neither converted model exceeds its own teacher.The comparison includes conversions using the ARL2 recipe.Source: §5.3 · p. 8
  • At 8–20 s revisits on MIND, LOCI has significantly lower worst-5%-patch DINOv2 feature-distance error than full softmax (−0.073), while whole-frame LPIPS does not separate them.Resetting LOCI’s recurrent state each chunk raises local error (+0.021 [+0.013, +0.028]) and lowers sharpness relative to ground truth from 0.18 to 0.12.Source: §5.3 · pp. 8–9

5.4 ANALYSIS OF THE RECURRENT STATE

  • The recurrent state linearly encodes camera yaw: held-out R2 is 0.946 [0.929, 0.960], and PRoPE raises R2 by +0.646 [+0.544, +0.749] over recurrent memory without camera encoding.The R2 comparison uses models trained on the uniform mixture.Source: §5.4 · p. 9, Figure 6 · p. 9
  • Co-visible tokens are about 0.1% of history, yet LOCI’s historical softmax layers place 11.1% of attention on them versus 10.45% for full softmax; zeroing the recurrent readout lowers downstream co-visible enrichment.On their own rollouts, LOCI also has lower masked LPIPS for co-visible regions (0.597 vs. 0.654).Source: §5.4 · p. 9

5.5 LONG-HORIZON GENERATION AND COST

  • With full history, LOCI extends reachable generation from 157 s to 225 s and lowers memory at equal length by about 30%; with bounded sparse access, it generates 300 s at constant 23.6 GiB.Under that bounded access, it uses 15% less memory than full softmax (27.6 GiB) at the same speed.Source: §5.5 · pp. 9–10

6 DISCUSSION AND LIMITATIONS

LOCI combines compact spatial context with direct access to retained visual detail, reducing the number of main-attention layers storing historical KV and supporting bounded-memory streaming. Its results are relative, with real-world captures unevaluated and scope limited to one 5B backbone and a short fine-tuning budget.

Limitations

  • Real-world captures are unevaluated; held-out trajectories use Unreal Engine, and all models have benchmark-average PSNR below 15 dB, so gains are relative.The study uses one 5B backbone and 5,000 fine-tuning updates.Source: §6 · p. 10
  • In bounded sparse mode, speed parity with full softmax depends on the optimized implementation described in Sec. 5.5.Source: §6 · p. 10

Conclusion

  • LOCI has the best MSE, PSNR, and SSIM among evaluated world models on MIND and improves over identically trained full softmax under the same bounded KV budget.With a bounded bank of retained observations, it streams at constant memory.Source: §6 · p. 10
  • LOCI halves the number of main-attention layers that store historical KV while combining compact spatial context with direct access to retained visual detail.The architecture supports dense and bounded sparse historical access.Source: §6 · p. 10

A IMPLEMENTATION DETAILS

The appendix specifies LOCI’s backbone layout, recurrent-state execution, and bounded panorama-bank policy. It also distinguishes fixed recurrent-state storage from retained history and other device-memory costs.

A.1 BACKBONE AND FEATURE LAYOUT

  • Fifteen of the 30 transformer blocks are hybrid recurrent blocks, while the remaining blocks retain historical softmax attention; recurrent ray transforms use camera geometry and normalized intrinsics.At 512 × 768 resolution, each latent frame produces 384 tokens, and each hybrid block carries 24 states of size 128 × 128.Source: §A.1 · pp. 14–15

A.2 GATES AND STATE EXECUTION

  • A token-and-head write gate and chunk-level log-retention control recurrent updates, while persistent state is committed after sampling rather than during denoising.Temporary denoising updates are discarded; generated chunks are committed in a separate forward pass.Source: §A.2 · pp. 15–16

A.3 EXECUTION SCHEDULE

  • Each hybrid layer reads its incoming recurrent state for current queries before scanning token-level writes to form the next state, and downstream blocks receive context through ordinary transformer features.Training uses a forward-local state that is discarded; inference commits generated chunks when another chunk follows.Source: §A.3 · p. 16

A.4 PANORAMA-BANK SELECTION

  • The panorama bank retains departing frames that add at least 30% new viewing directions and, above 20 frames, evicts the most redundant bank view.Coverage uses camera field-of-view masks and excludes occlusion and scene depth.Source: §A.4 · p. 16

A.5 SCOPE OF THE STREAMING COUNTS

  • Dense main-KV storage grows with the retained prefix, while recurrent layers hold fixed-size states; sparse mode caps explicit history at 34 latent-frame slots across both branches.These architecture counts do not include weights, activations, current-chunk storage, or transfer buffers in total device memory.Source: §A.1 · p. 15, §A.5 · p. 17

A.6 METHOD DETAILS: PROJECTIVE ENCODING, RETENTION, AND STREAMING STORAGE

LOCI’s recurrent path uses camera-conditioned projective encoding and chunk-level retention, while explicit history remains available through attention. Sparse history bounds retained-frame support, whereas dense prefix access continues to grow with history.

A.6 METHOD DETAILS: PROJECTIVE ENCODING, RETENTION, AND STREAMING STORAGE

  • Projective transforms condition recurrent queries, keys, and values on camera geometry while retaining spatial rotary encoding; gated recurrent readouts are added to local-attention output.Each hybrid block has its own state, and the explicit bank stores observation features.Source: §A.6 · p. 17
  • Applying retention once per chunk gives all tokens of an observation the same explicit forgetting factor, while delta-rule corrections remain order-dependent within the chunk.This avoids making explicit forgetting frequency depend on spatial token count.Source: §A.6 · p. 17
  • Sparse access bounds both KV supports to at most 34 latent frames, with recurrent read/write work per chunk independent of elapsed history; dense prefix attention reads growing history.Across chunks, sparse attention and recurrent work accumulate linearly under fixed settings, while dense prefix history-access work accumulates quadratically.Source: §A.6 · pp. 17–18
  • Table 3 reports logical history storage as layer count multiplied by retained latent-frame support, with sparse entries shown as upper bounds rather than measured device-memory ratios.Source: Table 3 · p. 18

B EVALUATION PROTOCOL

The evaluation compares LOCI and baselines under specified training, trajectory, and history-access protocols. It assesses camera following as a validity check and measures how historical attention relates to geometrically co-visible content.

B EVALUATION PROTOCOL

  • Training windows contain 81 latent frames sampled at stride two from 24-fps recordings, and evaluation durations generally use a 16-fps clock except for MIND.Source: B EVALUATION PROTOCOL · p. 18
  • LOCI and full softmax share the Wan2.2-TI2V-5B initialization and 5,000-update recipe, while LOCI adds a 90.7M-parameter recurrent branch.Both use the same UE-weighted data mixture from update 2,000 onward; training also includes rendered Unreal Engine and CARLA scenes and real walking videos.Source: B EVALUATION PROTOCOL · p. 18
  • Held-out tests exclude source episodes used in training; sets A and B contain repeated-viewpoint trajectories, while set C uses designed camera paths without ground truth.Set A has 18 clips from six scenes, and set B adds five clips selected for frequent returns.Source: B EVALUATION PROTOCOL · p. 18
  • Camera following is a validity check on whether generated videos execute commanded trajectories, assessed on designed paths using median rotation and translation-direction errors and motion magnitude ratio.The reported medians are summarized in Table 4.Source: B EVALUATION PROTOCOL · p. 19, Table 4 · p. 19
  • Sparse evaluation retains a conditioning observation, recent context, and a bounded older-view bank, with at most 34 latent-frame slots and shared selected source identities across camera-attention layers.The primary comparison uses identical 40-second trajectories; 300-second sparse rollouts are reported separately.Source: B EVALUATION PROTOCOL · p. 19
  • The co-visible attention analysis compares historical attention allocated to geometrically co-visible keys with their share of historical tokens, using ground-truth depth only for measurement when available.The reconstruction analysis covers 8 recorded clips and 154 revisit instants, with 95% bootstrap intervals over clips.Source: B EVALUATION PROTOCOL · p. 19, Table 5 · p. 20

C COMPARISON ON HELD-OUT RECORDED TRAJECTORIES

On held-out recorded trajectories, LOCI produces more faithful revisit frames than the compared models, including a same-recipe full-softmax model. The section also notes that performance differs between bounded access and full history depending on the trajectory’s history.

C COMPARISON ON HELD-OUT RECORDED TRAJECTORIES

  • LOCI’s revisit PSNR is significantly higher than the same-recipe full-softmax model by 0.62 dB, and its identity consistency rises from 0.659 to 0.796.Against external models, revisit LPIPS is significantly lower than six of seven and revisit PSNR significantly higher than five of seven; across the unfiltered set, PSNR is highest and stable across seeds (±0.14 dB).Source: C COMPARISON ON HELD-OUT RECORDED… · p. 20, Table 6 · p. 20, Table 7 · p. 21
  • Bounded access scores higher than full history at revisits on single-frame-generated histories, while full history scores higher on MIND’s long recorded histories.The authors suggest the 40-second self-generated rollout is further from the training distribution than bounded access, while recorded memory supplies clean evidence.Source: C COMPARISON ON HELD-OUT RECORDED… · pp. 20–21
  • Table 7 pairs LOCI with other methods on the same 20 clips; its 95% bootstrap intervals and LPIPS differences indicate when comparisons favor LOCI.Table 6 summarizes revisit, whole-clip, identity, camera-following, and cost comparisons.Source: Table 6 · p. 20, Table 7 · p. 21

D MIND OVER THE FIRST 20 SECONDS

This appendix compares models over the first 20 seconds of MIND prediction and documents how external baselines receive the memory segment. The reference rows and adapted runs use different evaluation procedures.

D MIND OVER THE FIRST 20 SECONDS

D.1 HOW EACH EXTERNAL BASELINE RECEIVES THE MIND MEMORY SEGMENT

  • Adapted external baselines receive the same raw MIND memory frames and target camera path as LOCI through their unmodified official inference pipelines.The SSM, FramePack, Context-as-Memory, and GIM-World reference rows are copied from the GIM-World paper and are not adapter runs.Source: §D.1 · p. 21

E RELATION TO HYBRID RECURRENT VIDEO MODELS

LOCI differs from recurrent video models by retaining direct softmax access to historical observations alongside a camera-conditioned recurrent path. Comparisons report stronger revisit fidelity than SANA-WM on recorded trajectories and Video SSM on MIND, with different conditioning and inference costs.

E RELATION TO HYBRID RECURRENT VIDEO MODELS

  • On recorded trajectories, LOCI beats SANA-WM at revisits by 1.02 dB PSNR and 0.210 lower LPIPS, with lower LPIPS in 19 of 20 clips.LOCI also has higher identity consistency (DINO 0.796 vs. 0.449); SANA-WM’s released pipeline accepts one conditioning image, so it receives the memory segment’s last frame on MIND.Source: E RELATION TO HYBRID RECURRENT VIDEO… · pp. 21–22
  • Table 10 organizes hybrid recurrent video models by conditioning, softmax history access, recurrent path, and how memory is trained and evaluated.Table 9 describes external baselines’ MIND memory inputs and inference settings, with adapters wrapping unmodified official pipelines.Source: Table 10 · p. 22, Table 9 · p. 22
  • On MIND, LOCI reports 0.0455 / 14.36 / 0.464 / 0.643 versus Video SSM’s 0.0796 / 11.96 / 0.395 / 0.744 for MSE / PSNR / SSIM / LPIPS.Video SSM was retrained on MIND by the GIM-World authors; LOCI was not trained on MIND.Source: E RELATION TO HYBRID RECURRENT VIDEO… · p. 23

F ADDITIONAL RESULTS

Additional results characterize LOCI’s localized revisit fidelity, retention choices, training, and inference costs. They also describe the evaluation scope for camera-return consistency and comparisons with external models.

F ADDITIONAL RESULTS

  • Table 11 evaluates WBench-Navi gated camera-return consistency, with higher scores better and LOCI evaluated using the official code.External-model values come from the WBench paper’s official leaderboard.Source: Table 11 · p. 23
  • On MIND revisits, LOCI’s local error is significantly lower than full softmax (−0.073 [−0.095, −0.050]), while whole-frame LPIPS does not separate them.Across recorded sets, local error is lower in 12 of 18 clips (−0.020 [−0.047, −0.000]); the MIND advantage is concentrated in localized short-horizon fidelity.Source: F ADDITIONAL RESULTS · pp. 23–24
  • Across all MIND prediction-segment revisits, chunk-level retention has lower local error than per-token retention, with the difference growing as revisit gaps increase.This exploratory extension covers 2,928 instants across all 50 segments; the short-range comparison alone did not distinguish the variants.Source: F ADDITIONAL RESULTS · p. 24, Table 14 · p. 24
  • After 5,000 updates, 6.4–11.1% of channels in layers 11, 13, and 23 have mean retention above 0.9, versus at most 2.1% after 1,000 updates.The reported retention corresponds to a half-life beyond six chunks.Source: F ADDITIONAL RESULTS · pp. 24–25
  • Directly trained LOCI outperforms converted full-softmax models on recorded set A revisit PSNR and LPIPS; neither converted model exceeds its own teacher.The authors conclude recurrent memory needs joint training with camera-conditioned generation.Source: F ADDITIONAL RESULTS · p. 25
  • With full history over the first 40 seconds, LOCI uses 33.6 GiB and 7.2 s per video second, versus 44.2 GiB and 8.1 s for full softmax.With bounded sparse access, memory and per-chunk time remain flat through 300 seconds; the bounded setup retains the first frame, 20 latents, and 8 most recent latents while LOCI carries recurrent state over full history.Source: Table 12 · p. 23, Table 13 · p. 23, F ADDITIONAL RESULTS · p. 25
  • Among compared external models, CaR requires 31.4 s per video second and exceeds GPU memory on long trajectories; HyDRA and SPMem require 175 and 97 s per video second.The larger compared models require 71–138 GiB.Source: F ADDITIONAL RESULTS · p. 25

Related papers

Loading 2609.40222v1…