Source-linked AI summary
WorldMem: Long-term Consistent World Simulation with Memory
Zeqi Xiao, Yushi Lan, Yifan Zhou, Wenqi Ouyang, Shuai Yang, Yanhong Zeng, Xingang Pan
TL;DR
Limited temporal context causes long-term world inconsistency, including divergent scenes after viewpoint changes. WORLDMEM uses state-aware memory attention over past frames to reconstruct scenes across spatial-temporal gaps and model dynamic changes, with experiments validating its effectiveness in virtual and real settings.
Problem
Limited temporal context prevents video generative models from conditioning on the full history, causing inconsistencies in long-term world simulation and 3D spatial coherence.
Method
WORLDMEM augments a memory bank of past frames with spatial, viewpoint, and timestamp states, using state-aware memory attention and confidence-based retrieval for conditional world generation.
Results
WORLDMEM significantly improves 3D spatial consistency and achieves superior performance across all metrics in virtual and real evaluations.
Takeaways & Limitations
WORLDMEM reconstructs previously observed scenes across large viewpoint or temporal gaps while tracking evolving events for perception and interaction.
Takeaways & Limitations
Memory usage increases linearly with sequence length, and view-overlap retrieval may fail when obstacles block views.
Abstract
from arXiv · showhide
World simulation has gained increasing popularity due to its ability to model virtual environments and predict the consequences of actions. However, the limited temporal context window often leads to failures in maintaining long-term consistency, particularly in preserving 3D spatial consistency. In this work, we present WorldMem, a framework that enhances scene generation with a memory bank consisting of memory units that store memory frames and states (e.g., poses and timestamps). By employing a memory attention mechanism that effectively extracts relevant information from these memory frames based on their states, our method is capable of accurately reconstructing previously observed scenes, even under significant viewpoint or temporal gaps. Furthermore, by incorporating timestamps into the states, our framework not only models a static world but also captures its dynamic evolution over time, enabling both perception and interaction within the simulated world. Extensive experiments in both virtual and real scenarios validate the effectiveness of our approach.
1 Introduction
World simulation is limited by finite temporal context, which can break long-term 3D consistency. WORLDMEM addresses this with state-aware memory retrieval for coherent generation across viewpoints and time.
- Finite context windows prevent models from conditioning on all past generations, causing previously generated content to be discarded and worlds to become inconsistent.
- Explicit 3D reconstruction can preserve geometry but is inflexible in dynamic environments and may lose detail in large, unbounded scenes.
- WORLDMEM augments memory units with spatial location, viewpoint, and timestamp cues for reasoning across large spatiotemporal gaps.
- Built on CDiT and Diffusion Forcing, WORLDMEM uses a memory bank, memory attention, and confidence-based retrieval based on field-of-view overlap and temporal proximity.
- Experiments on customized Minecraft and RealEstate10K show improved 3D spatial consistency, viewpoint reasoning, scene generation, and tracking of dynamic environmental changes.
2 Related Work
Prior interactive world simulators and consistency methods are constrained by limited context windows and trade-offs between geometric fidelity and flexibility. WORLDMEM uses memory-based retrieval to guide long-term consistent simulation.
- Interactive world simulation predicts the next environment state from the current state and an action, supporting agent learning and controllable video generation.
- Context window limitations cause existing methods to discard previously generated content, producing inconsistencies in simulated worlds, especially in 3D spatial coherence.
- Consistency methods include geometric-based approaches that reconstruct generated worlds into explicit 3D or 4D representations.
- WORLDMEM retrieves memory units from a dedicated bank and injects them through memory blocks to guide conditional diffusion-transformer generation.
3 WORLDMEM
WORLDMEM extends an autoregressive interactive video simulator with external memory for retrieving historical frames and states beyond the limited context window. State-aware retrieval and attention support reconstruction across viewpoint and temporal gaps, with evaluations and ablations examining consistency and retrieval design.
- Interactive World Simulator: WORLDMEM combines a conditional Diffusion Transformer with Diffusion Forcing for autoregressive first-person video generation controlled by external actions.The simulator uses spatial-temporal reasoning modules and injects action embeddings into temporal blocks.
- Memory Representation and Retrieval: The memory bank stores token-level historical frames with poses and timestamps, forming memory units that retain reconstruction-relevant visual detail.Each pose contains x, y, z, pitch, and yaw coordinates, while corresponding states support retrieval and conditioning.
- Memory Representation and Retrieval: Memory retrieval greedily selects units using field-of-view overlap and timestamp differences, then removes redundant units through similarity filtering.The strategy is designed to select relevant information when the memory-conditioning length is limited.
- Evaluation: Qualitative and evaluation figures examine within-window consistency, beyond-window scene recovery, and dynamic changes such as rain and plant growth.The experiments compare WORLDMEM with ground truth and Diffusion-Forcing, while ablations study embedding designs and memory retrieval strategies.
- State-aware Memory Condition: State-aware memory attention enriches query and memory features with pose and timestamp embeddings before cross-attention retrieves historical content.Relative state encoding resets the query state and normalizes key states to improve alignment under viewpoint changes.
4 Experiments
Experiments evaluate WORLDMEM on Minecraft and RealEstate10K using consistency, fidelity, perceptual similarity, and realism metrics. Across long-horizon generation, real-world trajectories, and ablations, memory retrieval and state-aware designs improve consistency and quality.
- Experimental Setup: WORLDMEM is evaluated on diverse Minecraft environments and RealEstate10K scenes with camera pose annotations.The evaluation covers terrains, actions, interactions, and real-world camera trajectories.
- Minecraft Benchmark: WORLDMEM achieves the best performance in Minecraft within-context scenarios, where the full-sequence baseline remains inconsistent and Diffusion Forcing improves information exchange.All methods use a context window of 16, while WORLDMEM additionally maintains an 8-frame memory window.
- Minecraft Benchmark: Beyond the context window, WORLDMEM consistently outperforms other methods across PSNR, LPIPS, and rFID while preserving long-term consistency and quality.The setting uses 8 context frames, 100 future frames, and a 600-frame memory bank; full-sequence methods cannot roll out that long.
- Real Scenarios: WORLDMEM achieves superior performance across all metrics on RealEstate10K trajectories that return to their starting pose.The trajectories span 37–60 frames, exceeding every baseline’s maximum training length of 25 frames.
- Ablation Studies: Dense relative pose embeddings improve relationship reasoning and retrieval, maintaining stability beyond 300 frames compared with error accumulation from absolute embeddings.Dense pose embeddings improve all metrics, while relative encoding particularly improves LPIPS and rFID.
- Ablation Studies: Progressive sampling significantly improves all metrics over fixed small-range or large-range sampling during Minecraft training.The strategy gradually expands memory-conditioning range from 2m to 8m, increasing training difficulty over time.
- Ablation Studies: Timestamp conditioning significantly improves PSNR and LPIPS for reproducing event progression, while unpredictable plant growth is evaluated qualitatively.The time-condition ablation uses 100 video samples featuring placing events.
- Ablation Studies: Confidence-based retrieval and redundancy filtering improve consistency and generation quality over random memory sampling.Random sampling causes severe quality degradation, including sharp rFID drops and rapid divergence from ground truth.
5 Limitations and Future works
WORLDMEM has unresolved limitations in memory retrieval, interaction realism, and scalability for extremely long sequences.
- Retrieval: View-overlap retrieval may miss necessary information when obstacles block views.The authors identify this as a corner case where the current retrieval criterion is insufficient.
- Interaction: The current environment interaction lacks diversity and realism, motivating future evaluation in more varied real-world scenarios.The authors plan to extend the models to realistic and diverse interactions.
- Scalability: Memory usage increases linearly with sequence length, which may limit handling of extremely long sequences.This remains a limitation of the current memory design.
6 Conclusion
WORLDMEM addresses long-term world-simulation consistency with a memory bank of past frames and associated states. Its memory attention reconstructs scenes across large viewpoint or temporal gaps while modeling dynamic changes over time.
- Core Contribution: WORLDMEM uses a memory bank containing past frames and associated states to address long-term consistency in world simulation.The conclusion identifies this memory-based design as the paper’s central approach.
- Core Contribution: Memory attention enables accurate reconstruction of previously observed scenes across large viewpoint or temporal gaps.The conclusion reports this capability as part of the framework’s contribution.
- Dynamic Simulation: Timestamp-aware states allow WORLDMEM to model dynamic changes over time in simulated worlds.The framework is presented as supporting both static scene consistency and temporal evolution.
- Overall Outcome: Experiments in virtual and real settings confirm WORLDMEM’s capacity for robust, immersive world simulation.The conclusion presents this as the overall experimental outcome.
7 Supplementary Materials
The supplementary materials detail WORLDMEM’s embeddings, pose prediction, retrieval computation, scalability, and experimental visualizations. They report that context length 8 provides the best trade-off, while memory adds limited inference overhead and can reach approximately 10 FPS with acceleration.
- Memory context length: Context length 8 achieves the best trade-off, with the highest PSNR and lowest LPIPS and rFID.Increasing context from 1 to 8 improves performance, whereas extending it to 16 deteriorates results.
- Pose prediction: The pose predictor estimates the next pose from the previous image, previous pose, and upcoming action, removing the need for ground-truth poses at inference.This supports interactive play using actions alone.
- Memory usage: Storing 600 visual memory tokens with shape [600, 16, 18, 32] in float32 requires approximately 21MB.The memory bank is described as lightweight.
- Retrieval and generation efficiency: Retrieval accounts for only 10–20% of total inference time even with 1000 candidates, while generation takes approximately 0.9 seconds per frame.The reported generation cost uses 20 denoising steps.
- Baseline comparison: Adding memory causes moderate training overhead but only a small inference-memory increase and slight speed decrease.The comparison uses consistent settings with 8 context frames, 8 memory frames, and 20 denoising steps on a single H200.
- Inference optimization: With timestep distillation, early exit, and sparse attention, inference speed can reach approximately 10 FPS.These acceleration techniques are presented as making the method practical for deployment.