PaperlayerView exact revision on arXiv

Source-linked AI summary

Spatial Memory Intelligence: Endowing World Models with Understanding-Driven Long-Term Memory

Ying Yang, Guiyu Zhang, Lianghua Huang, Chang Nie, Chenyang Si, Haofan Wang, Shaoshuai Shi, Li Jiang

arXiv:2610.02521v1cs.CV

TL;DR

Long-video world models face growing challenges in managing spatial context as observation histories expand. SMI uses an understanding model to organize, sparsify, retrieve, and filter memory; experiments report improvements in memory sparsity, generation stability, and spatial consistency across multiple baselines, benchmarks, and backbones.

  • Problem

    Growing observation histories make it challenging to manage spatial context while balancing memory sparsity, spatial consistency, and generation stability.

    Source: §1 · p. 2, §2.1 · p. 4

  • Method

    SMI uses an MLLM to coordinate spatial clustering, within-cluster sparsification, action-aware retrieval, and reliability-aware filtering of long-term spatial memory.

    Source: §1 · p. 2, §3.3 · p. 5

  • Results

    Across multiple baselines, benchmarks, and world-model backbones, SMI improves memory sparsity, long-horizon generation stability, and visual consistency.

    Source: §1 · p. 3, §4.4.1 · p. 9, §5 · p. 11

  • Takeaways & Limitations

    SMI reduces memory overhead by sparsifying redundant observations while preserving useful spatial evidence.

    Source: §5 · p. 10

    SMI is not trained end to end with the world model, and its performance remains bounded by the base generator’s visual quality, controllability, and stability.

    Source: §6 · p. 11

Abstract

from arXiv · show

Long-video generation and world models have shown strong potential for interactive entertainment and embodied simulation by predicting future observations conditioned on user actions and historical memory. However, as memory sequences grow longer and their structures become increasingly complex, managing long-range spatial context becomes increasingly challenging, calling for a more intelligent and systematic memory-management strategy. Building on the advancing spatial reasoning capabilities of multimodal large language models (MLLMs) and the broader vision of unified models, we propose Spatial Memory Intelligence (SMI), the first framework to systematically employ an understanding model for spatial-memory management in long-video world models. SMI introduces four coordinated atomic operations: spatial clustering, within-cluster sparsification, action-aware retrieval, and reliability-aware filtering. Extensive experiments across multiple baselines, benchmarks, and world-model backbones demonstrate the effectiveness and generalizability of SMI, achieving comprehensive improvements in memory sparsity, generation stability, and spatial consistency.

1 INTRODUCTION

Long-horizon world models need to manage growing spatial memory while balancing storage, consistency, and generation stability. SMI uses an understanding model to coordinate memory organization, sparsification, retrieval, and reliability filtering.

1 INTRODUCTION

  • Growing observation histories create linked challenges in storage cost, retrieving evidence for revisited regions, and preventing generated errors from compounding through reuse.Unretrieved evidence can yield geometric inconsistencies or changes in object identity or state.Source: §1 · p. 2
  • Existing compression, sparse-computation, and selective-retrieval approaches leave gaps in preserving fine-grained evidence or jointly reasoning over spatial and semantic context.Geometry-based retrieval is sensitive to occlusion and geometry errors, while semantic retrieval can confuse similar content across locations or viewpoints.Source: §1 · p. 2
  • SMI coordinates spatial clustering, within-cluster sparsification, action-aware retrieval, and reliability-aware filtering to manage long-term spatial memory.The operations organize spatially proximate chunks, remove redundancy, retrieve evidence using recent context and current action, and filter visually drifting chunks.Source: §1 · pp. 2–3
  • Experiments across baselines, benchmarks, and world-model backbones show SMI consistently improves memory sparsity, long-horizon generation stability, and visual consistency.The framework also introduces operation-oriented supervision datasets for its four memory-management operations.Source: §1 · p. 3

2 RELATED WORK

Video world models have advanced toward persistent, interactive simulation, but long horizons intensify memory and consistency challenges. Prior work addresses pieces of memory management, while multimodal understanding and unified models offer a basis for a more integrated approach.

2.1 VIDEO WORLD MODELS

  • Persistent video world models extend interactive simulation toward long-horizon environments, where growing histories must be sparsified without losing scene consistency or limiting error accumulation.Related work also extends these models to multi-agent interaction and embodied simulation.Source: §2.1 · pp. 3–4

2.2 MEMORY IN VIDEO GENERATION AND WORLD MODELS

  • Memory-management approaches use compression, sparse computation, or selective retrieval, but lack a unified framework combining memory efficiency with spatially grounded management.Compression may discard fine-grained visual and geometric information, while geometry- and semantic-based retrieval have different limitations.Source: §2.2 · p. 4

2.3 MULTIMODAL INTELLIGENCE

  • MLLM and VLM advances in long-video reasoning and spatial understanding, alongside unified multimodal models, motivate integrating understanding into generative processes.The cited capabilities include reasoning over 3D and multi-view environments and geometry-aware representations.Source: §2.3 · p. 4

3 METHOD

SMI uses an MLLM to manage long-term spatial memory in video world models through four coordinated operations. The method covers action-conditioned generation, spatial-memory management, and operation-oriented data construction and training.

3 METHOD

  • SMI organizes spatial memory through clustering, within-cluster sparsification, action-aware retrieval, and reliability-aware filtering.Source: §3 · p. 4, §3.3 · p. 5

3.1 PRELIMINARIES

  • Action-conditioned world models generate frames autoregressively from selected historical frames and current actions, and SMI manages the historical observations used.Source: §3.1 · p. 5

3.2 WHY USE AN MLLM?

  • The method motivates an MLLM memory manager by its contextual, semantic, and spatial reasoning capabilities for organizing, retaining, and retrieving memory.Source: §3.2 · p. 5

3.3 ATOMIC SPATIAL OPERATIONS

  • Camera position and orientation shortlist spatially close cluster prototypes, and the MLLM assigns each new chunk to a matching cluster or creates one.Source: §3.3 · p. 6
  • When a cluster reaches the trigger threshold, the MLLM removes redundant items while retaining chunks that add spatial structure, object states, or occlusion relations.Source: §3.3 · p. 6
  • Camera state and current action shortlist memories by spatial overlap, after which the MLLM selects useful historical chunks using recent context and the action.Source: §3.3 · p. 7
  • For each new chunk, the MLLM detects evident visual drift and either admits it to memory or discards it to reduce accumulated generation errors.Source: §3.3 · p. 7

3.4 OPERATION-ORIENTED DATA CONSTRUCTION

  • The operation-oriented dataset combines teacher-MLLM annotations with human review, and the MLLM is trained on annotated targets using cross-entropy.The pipeline begins with roughly one thousand inference trajectories and human annotations of more than one hundred examples.Source: §3.4 · p. 7

4 EXPERIMENTS

Experiments compare SMI with memory-management baselines on two world-model backbones using complementary quality and efficiency measures. Reported results show broad improvements, while ablations examine component removal and operation-oriented fine-tuning.

4.1 IMPLEMENTATION DETAILS

  • Experiments use HY1.5-8B and Wan2.2-5B world-model backbones, with Qwen3.5-4B as the multimodal understanding model.Qwen3.5-4B is fine-tuned with AdamW at a learning rate of 2 × 10−5 and global batch size 64.Source: §4.1 · p. 7

4.2 EVALUATION METRICS

  • Four protocols assess generation quality, scene consistency, reconstruction consistency, and user judgments; efficiency measures memory sparsity and latency relative to Base.Evaluations use randomly sampled one-minute rollouts, separate revisit-trajectory rollouts for reconstruction, and held-out test scenes.Source: §4.2 · p. 8

4.3 BASELINES

  • Five memory-management baselines cover compression, sparse-routing retrieval, geometry-based retrieval, and semantic retrieval.Comparisons use identical initial observations, action sequences, random seeds, context budget, and generation configuration.Source: §4.3 · p. 8

4.4.1 QUANTITATIVE RESULTS

  • On HY1.5, SMI raises Aesthetic Quality from 0.445389 to 0.469357, GPT-evaluated Scene Consistency from 51.6053 to 67.0200, and PSNR from 12.4785 to 13.3093 dB over Base, with 83.6842% memory sparsity.The authors report a similar trend on Wan2.2 and broader improvements than other methods.Source: §4.4.1 · pp. 8–9, Table 1 · p. 8, Table 2 · p. 8

4.4.2 QUALITATIVE RESULTS

  • On both HY1.5 and Wan2.2, SMI demonstrates the best spatial consistency and generation stability among the compared methods.Figures 5 and 6 show corresponding initial and final-frame regions in red boxes.Source: §4.4.2 · p. 9, Figure 5 · p. 9, Figure 6 · p. 9

4.5.1 COMPONENT ABLATION

  • Removing components degrades generation stability and consistency to varying degrees; without spatial clustering, sparsification can discard important spatial evidence.The reconstruction scores and Figure 7(1) visual inconsistencies support this observation.Source: §4.5.1 · p. 10, Table 3 · p. 10, Figure 7 · p. 10

4.5.2 EFFECT OF OPERATION-ORIENTED TRAINING

  • Without operation-oriented fine-tuning, Qwen3.5-4B struggles to perform the atomic operations reliably, with substantially lower generation quality and consistency.The authors report that fine-tuning on high-quality data enables the model to adapt to the operations and manage complex spatial memory.Source: §4.5.2 · p. 10, Table 4 · p. 10, Figure 7 · p. 10

5 CONCLUSION

The conclusion presents SMI as a unified understanding-model framework for managing spatial memory in long-horizon video world models. It summarizes the framework's operations and reported benefits, with experiments across baselines, benchmarks, and backbones validating its effectiveness.

5 CONCLUSION

  • SMI manages spatial memory through spatial clustering, within-cluster sparsification, action-aware retrieval, and reliability-aware filtering.The conclusion describes these as four coordinated atomic operations.Source: §5 · p. 10
  • By sparsifying redundant observations while preserving useful spatial evidence, SMI reduces memory overhead and improves long-horizon generation stability and spatial consistency.Source: §5 · p. 10
  • Experiments across multiple baselines, benchmarks, and world-model backbones validate SMI's effectiveness.Source: §5 · p. 11

6 LIMITATIONS AND FUTURE WORK

SMI’s limitations include dependence on the base generator, early-rollout computation, and an unaddressed class of long-horizon tasks; proposed future work targets shared learning, reinforcement learning, and broader datasets. Qualitative examples illustrate its runtime memory-management operations.

6 LIMITATIONS AND FUTURE WORK

  • SMI is not trained end to end with the world model, remains bounded by its generator, adds early-rollout computation, and does not explicitly address streaming understanding or complex dependencies.The stated scope boundary includes tasks involving complex long-term dependencies or logical reasoning.Source: §6 · p. 11
  • Future directions include jointly learning generation, understanding, and memory management, adding reinforcement learning, and constructing broader, higher-quality datasets.Joint learning may reduce computational overhead and enable end-to-end optimization, while reinforcement learning may strengthen long-term memory management.Source: §6 · p. 11

A QUALITATIVE EXAMPLES OF RUNTIME MEMORY MANAGEMENT

B ADDITIONAL EXPERIMENTAL DETAILS

The implementation uses MLLM judgments to cluster spatial regions, sparsify redundant memories, retrieve action-relevant context, and filter unreliable generated chunks.

B ADDITIONAL EXPERIMENTAL DETAILS

  • Spatial clustering compares a new chunk’s middle frame with candidate cluster prototypes, assigning it to a matching region or creating a new cluster.The MLLM determines whether the frames depict the same spatial region.Source: B ADDITIONAL EXPERIMENTAL DETAILS · p. 21
  • Within-cluster sparsification begins at a predefined chunk threshold and removes an intermediate chunk when it adds no spatial structure, object states, or occlusion information.The MLLM compares two sampled frames from each chunk with neighboring memories.Source: B ADDITIONAL EXPERIMENTAL DETAILS · p. 21
  • Action-aware retrieval shortlists seven chunks by camera-position proximity, then uses recent observations and the current action to select relevant spatial context.Both backbones retain the two most recent chunks; no additional chunk is retrieved when none is suitable.Source: B ADDITIONAL EXPERIMENTAL DETAILS · p. 21
  • Reliability-aware filtering scores each new chunk from 1–10 and discards chunks scoring below 6 before they enter the memory bank.Higher scores indicate greater visual reliability.Source: B ADDITIONAL EXPERIMENTAL DETAILS · p. 21

C.1 EVALUATION PROTOCOL

The protocol constructs tractable long-horizon evaluation sequences from generated chunks and uses blinded independent scoring across nine dimensions. The main paper reports camera-constrained scene consistency and overall visual quality, with complete scores in Tables 5 and 6.

C.1 EVALUATION PROTOCOL

  • Evaluation samples one representative frame per generated chunk in temporal order, providing action sequences when judging camera-motion adherence.Chunk-level sampling retains scene evolution while keeping the evaluation input tractable.Source: §C.1 · p. 21
  • Three independent GPT-5.6-sol evaluators score each complete sequence with method identities hidden and video order randomized.Their predictions are aggregated only after all three evaluations, reducing ordering and individual-evaluator variation.Source: §C.1 · p. 21
  • The protocol assesses nine dimensions, while the main paper reports Camera-Gated Scene and Overall Visual Quality; Tables 5 and 6 provide all nine scores for HY1.5 and Wan2.2.The table captions specify a 0–100 scale, with higher scores better.Source: §C.1 · p. 21, Table 5 · p. 22, Table 6 · p. 22

D USER STUDY

In a comparison across 15 scenes, SMI received the largest best-choice preference shares for both scene consistency and generation stability. The study used matched initial observations and action sequences for videos lasting 45–60 seconds.

D USER STUDY

  • Twenty participants evaluated 45–60-second videos across 15 scenes under matched initial observations and action sequences.The compared methods were Base, VMem, FramePack, and SMI.Source: D USER STUDY · p. 22
  • 52.50% for scene consistency and 57.83% for generation stability were SMI’s best-choice preference shares, leading Base, VMem, and FramePack in both dimensions.Table 7 reports best-choice preference shares across 15 scenes.Source: Table 7 · p. 22, D USER STUDY · p. 22

E BASELINE IMPLEMENTATION DETAILS

The evaluation holds backbone weights and generation conditions constant across baselines on HY1.5 and Wan2.2. Baselines differ in how they retain or retrieve historical context.

E BASELINE IMPLEMENTATION DETAILS

  • All methods share initial observations, actions, seeds, context budgets, and generation settings; two recent chunks remain local, with capped historical retrieval.HY1.5 allows up to two additional historical chunks and Wan2.2 up to three.Source: E BASELINE IMPLEMENTATION DETAILS · p. 22
  • Base retrieves earlier chunks by field-of-view overlap, while FramePack selects older observations by temporal distance across 4×, 2×, and 1× tiers.Base uses no auxiliary memory manager; FramePack retains one initial anchor and adds no FramePack-specific encoder or weights.Source: E BASELINE IMPLEMENTATION DETAILS · p. 22
  • MoC routes remote chunks by query-key similarity using compact block-0 attention descriptors, then combines top-ranked chunks with local context.Its retrieval is capped according to the backbone.Source: E BASELINE IMPLEMENTATION DETAILS · pp. 22–23
  • Deep Forcing retains a 10-frame deep-sink prefix and 4-frame recent query window, selecting non-sink historical tokens by attention when full.The implementation compensates temporal RoPE positions after reorganization and excludes separate random-selection or compression-free variants.Source: E BASELINE IMPLEMENTATION DETAILS · p. 23
  • VMem ranks historical chunks by contribution to target-view rendering from a LoGeR-estimated surfel map, then re-encodes selected chunks through the native context path.It adds at most two chunks on HY1.5 or three on Wan2.2.Source: E BASELINE IMPLEMENTATION DETAILS · p. 23
  • MemFlow ranks candidate frames by CLIP image-image cosine similarity, using the preceding chunk’s last frame as query and excluding local chunks.It retrieves at most two remote chunks on HY1.5 or three on Wan2.2.Source: E BASELINE IMPLEMENTATION DETAILS · p. 23

F OPERATION-ORIENTED DATA CONSTRUCTION

SMI’s operation-oriented supervision is built from manually reviewed world-model videos and expanded with teacher-model annotations. The section describes training-data composition, operation labels and prompts, and additional qualitative comparisons on both backbones.

F OPERATION-ORIENTED DATA CONSTRUCTION

  • The data pipeline starts with about 1,000 videos longer than one minute, manually annotates 354 samples, and constructs separate supervision datasets for HY1.5 and Wan2.2.The HY1.5 set has 63,911 training samples, including 24,623 action-aware retrieval samples (38.53%).Source: F OPERATION-ORIENTED DATA CONSTRUCTION · p. 23, Table 8 · p. 23
  • Teacher MLLMs annotate remaining examples from manual demonstrations; a third model flags anomalies, which are then reviewed and cleaned.GPT-5.6-sol and Gemini-3.1 Pro annotate, and Claude Fable 5 checks their outputs.Source: F OPERATION-ORIENTED DATA CONSTRUCTION · p. 23

F.1 OPERATION-WISE LABEL COMPOSITION

  • Training labels cover spatial membership, keep/remove decisions, reliability filtering, and action-aware retrieval exposure.HY1.5 has 9,041 belongs and 3,611 does-not-belong clustering samples; Wan2.2 has 8,972 and 3,566. Keep/remove counts are 2,045/5,111 for HY1.5 and 2,051/5,084 for Wan2.2. Filtering labels are 24.55% for HY1.5 and 38.79% for Wan2.2; these are supervision proportions, not runtime filtering rates. Action-aware motion composition covers 24,623 HY1.5 and 6,852 Wan2.2 occurrences.Source: §F.1 · p. 24, Table 9 · p. 24, Table 10 · p. 25

G PROMPTS FOR THE FOUR ATOMIC OPERATIONS

  • The four operations receive dedicated prompts defining chunk reliability, spatial matching, within-cluster redundancy, and action-conditioned retrieval decisions.The prompts specify a 1–10 reliability score, spatial-match criteria, redundancy rules, and motion-aware selection of useful references.Source: G PROMPTS FOR THE FOUR ATOMIC OPERATIONS · pp. 24–26

H ADDITIONAL QUALITATIVE COMPARISONS

  • Additional qualitative comparisons on HY1.5 and Wan2.2 show comprehensive improvements in generation stability and spatial consistency with SMI.The HY1.5 comparisons span Figures 13–16, and the Wan2.2 comparisons span Figures 17–18.Source: H ADDITIONAL QUALITATIVE COMPARISONS · p. 27

H.1 HY1.5

  • Figure 13 provides additional HY1.5 qualitative results.The caption identifies the figure as additional qualitative results on HY1.5.Source: Figure 13 · p. 27

MoC

H.2 WAN2.2

Loading 2610.02521v1…