Source-linked AI summary
Video World Models with Long-term Spatial Memory
Tong Wu, Shuai Yang, Ryan Po, Yinghao Xu, Ziwei Liu, Dahua Lin, Gordon Wetzstein
TL;DR
Video world models struggle to preserve previously generated environments because limited temporal context causes forgetting during revisits. This paper introduces geometry-grounded spatial, working, and episodic memories with storage and retrieval mechanisms, and reports improved quality, spatial consistency, and context length over relevant baselines. The authors position the framework as a step toward long-term consistent world generation, while noting limitations in its memory storage and broader consistency coverage.
Problem
Limited temporal context windows make video world models forget previously generated scenes during revisits, while image-based context representations lack persistent 3D understanding for spatial consistency.
Method
The framework combines recent-frame working memory with a geometry-grounded 3D spatial memory and sparse episodic historical frames, using mechanisms to store and retrieve generated information.
Results
The approach improves quality, spatial consistency, and context length compared to relevant baselines, with higher PSNR, SSIM, and LPIPS reported in quantitative comparisons.
Takeaways & Limitations
Long-term spatial memory can support more consistent video world generation for applications including computer graphics, robotics, content creation, and agent or robot training data.
Takeaways & Limitations
TSDF-Fusion introduces artifacts from substantially different camera poses, and the mechanism primarily targets spatial rather than character consistency; video-model drift remains another challenge.
Abstract
from arXiv · showhide
Emerging world models autoregressively generate video frames in response to actions, such as camera movements and text prompts, among other control signals. Due to limited temporal context window sizes, these models often struggle to maintain scene consistency during revisits, leading to severe forgetting of previously generated environments. Inspired by the mechanisms of human memory, we introduce a novel framework to enhancing long-term consistency of video world models through a geometry-grounded long-term spatial memory. Our framework includes mechanisms to store and retrieve information from the long-term spatial memory and we curate custom datasets to train and evaluate world models with explicitly stored 3D memory mechanisms. Our evaluations show improved quality, consistency, and context length compared to relevant baselines, paving the way towards long-term consistent world generation.
1 Introduction
Video world models can generate interactive environments from actions, but limited context windows cause scene forgetting and weak spatial consistency during revisits. The paper addresses this with a geometry-grounded memory framework combining recent context, persistent 3D structure, and episodic references.
- Problem: Limited temporal context windows cause video world models to forget previously generated scenes during revisits.The computational cost of processing more context frames restricts how much past information models can consider.
- Problem: Existing approaches keep context short or downsample distant frames, but their image-based representations lack persistent 3D understanding.This limits spatial consistency when environments are revisited.
- Approach: The proposed framework combines short-term working memory with long-term spatial and sparse episodic memory.Working memory stores recent pixel frames, while spatial memory uses an explicit 3D representation and episodic memory stores historical keyframes.
- Approach: The spatial memory uses a geometry-grounded point cloud and filters dynamic content before storing newly generated information.This prioritizes static scene structure for long-term spatial recall.
- Contributions: The authors develop memory storage and retrieval mechanisms, curate a custom dataset, and report higher quality and 3D consistency than relevant baselines.The dataset supports training and evaluation of a proof-of-principle implementation.
2 Related work
Related work frames world models as action- or camera-conditioned generative systems, with diffusion and autoregressive methods enabling controllable, longer video generation. Persistent spatial memory complements these approaches by addressing the limited effective memory of sliding context windows.
- World Models: World models autoregressively generate image or video frames conditioned on actions or camera pose.This formulation supports interactive environment simulation.
- Image and Video Generation: Diffusion models have extended from image generation to video generation, producing clips spanning tens to hundreds of frames.Video diffusion architectures incorporate temporal modeling into iterative denoising.
- Autoregressive Video Generation: Autoregressive video methods generate future spatio-temporal tokens or frames from previously generated context, supporting longer online inference.Conditional diffusion models can autoregressively generate videos by passing clean generated frames forward as context.
- Controlled Video Generation: Controlled video generation uses camera, structural, action-based, or scene-level conditioning to guide views, trajectories, motion, and progression.Point clouds and other 3D-aware priors improve spatial consistency and trajectory alignment.
- Long-Context Video Generation: Sliding context windows limit effective memory in long-context video generation, while longer training contexts remain computationally demanding.The paper adopts recent frames as working memory and adds global spatial and episodic memories for long-term generation.
3 Method
The framework combines working, spatial, and episodic memory to condition autoregressive video generation, using persistent geometry to preserve static scene structure while recent frames maintain dynamics. It stores and retrieves memory during generation and trains on structured video samples paired with explicit 3D spatial memory and future observations.
- Memory mechanisms: The model stores and retrieves three dedicated memory types—recent frames, persistent spatial geometry, and sparse historical references—to condition new video frames.Recent frames provide nearby dynamic context; spatial memory captures static structure; episodic memory preserves detailed visual cues from selected past views.
- Spatial memory: A persistent static point map is built with TSDF fusion, whose weighted voxel updates suppress inconsistent dynamic elements.For voxel v, D(v) and W(v) are updated using new observations with truncated signed distances d_i(v) and confidence weights w_i.
- Temporal generation: Spatial memory is updated online from newly reconstructed static maps, while recent-frame conditioning provides motion continuity for autoregressive generation.The model conditions each step on the most recent k + 1 latent frames and can repeat this procedure for open-ended generation.
- Episodic memory: Sparse episodic memory adds representative historical frames when newly revealed unknown regions exceed a predefined visibility threshold.These references compensate for the fused point cloud’s limited ability to preserve detailed visual cues from the past.
- Conditional guidance: Static point-cloud renderings are encoded into condition latents and added to the video diffusion model to guide camera movement and preserve static-area consistency.The rendering follows the current spatial memory along the input trajectory, with regions lacking points set to black.
- Dataset and evaluation: The custom dataset contains 90K structured video samples paired with explicit 3D spatial memory and future observations.The construction pipeline extracts camera poses and dynamic point maps, removes dynamic regions through TSDF fusion, renders geometry along target trajectories, and annotates future actions.
4 Experiments
The experiments evaluate camera control, view-recall consistency, video quality, and memory components against relevant baselines. Across these evaluations, the proposed memory-based approach generally performs better, while each memory component contributes to specific capabilities.
- Quantitative Evaluation: The evaluation uses point-map-conditioned baselines, VBench metrics, view-recall reconstruction metrics, and a user study across 500 unseen video sequences.View recall uses PSNR, SSIM, and LPIPS on paired frames from reversed camera trajectories.
- Quantitative Evaluation: The model achieves top overall performance among relevant baselines on VBench metrics.It improves aesthetic quality, temporal flickering, motion smoothness, and subject consistency, although Wan2.1 exceeds it in imaging quality and background consistency.
- Quantitative Evaluation: The method achieves significantly improved PSNR, SSIM, and LPIPS compared with all baselines for revisited camera poses.Even its PSNR remains far from perfect, showing that retaining every visual detail in complex scenes remains difficult.
- Qualitative Evaluation: Qualitatively, the method better follows camera trajectories, preserves static regions during view revisits, and follows instructed actions without the baselines’ drifting or character disappearance.The comparisons cover significant camera motion, scene revisits, and integration of static and dynamic elements.
- User Study: The user study ranks the method above baselines by a large margin in camera accuracy, static consistency, and dynamic plausibility.The study covers 14 representative use cases and 20 subjects.
- Memory Ablations: Using short-term working, long-term spatial, and long-term episodic memory together produces the best VBench results.Working memory supports smooth dynamic motion, while episodic memory helps retain visual details such as previously seen characters or objects.
5 Discussion
The discussion reports that geometry-grounded memory improves quality and consistency, while identifying reconstruction failures and unresolved consistency challenges. It also positions the mechanism as useful for video-world-model applications and notes societal risks of video generation.
- Discussion: The geometry-grounded long-term spatial memory mechanism improves quality, spatial consistency, and context length compared with relevant baselines.The framework combines spatial, working, and episodic memory mechanisms.
- Limitations and Future Work: The framework primarily targets spatial consistency, while frame-packing strategies primarily target character consistency; combining them is proposed as future work.The paper also leaves drift from accumulated errors unaddressed.
- Limitations and Future Work: Large camera-pose distances and abrupt trajectories can cause 4D reconstruction failure, ghosting, sparse spatial memory, and loss of critical information.The example of rapidly swinging between skyscrapers illustrates this failure case.
- Societal Impacts: Video generation models can be adapted for DeepFake generation, creating significant societal risks.The paper strongly opposes deceptive content intended to mislead or spread misinformation.
- Conclusion: Long-term consistency through memory mechanisms is presented as making video world models more effective for content creation and training data for agents or robots.This consequence is stated in the conclusion.
A Additional Implementation details
The implementation updates spatial memory during autoregressive video generation while addressing alignment and memory constraints in reconstruction and TSDF fusion.
- Each autoregressive step produces a new static points map that is updated into spatial memory.
- Different-stage Mega-SAM reconstructions require alignment before merging because they use the NDC coordinate system.
- Mega-SAM dataset extraction resizes videos to 384×672, refines camera motion with optical flow, and applies Covariance-based Variable Decomposition.
- TSDF fusion proportionally scales voxel size when the maximum grid dimension exceeds 1200, controlling resolution to prevent memory overflow.
B Additional discussion on related works.
The discussion contrasts traditional SfM with learning-based reconstruction and explains the complementary use of Mega-SAM and CUT3R across dataset construction and iterative generation.
- COLMAP-based SfM offers high reconstruction accuracy but scales poorly because incremental feature matching and bundle adjustment are costly for large-scale or real-time applications.
- Mega-SAM provides stable, accurate point clouds and camera poses for dataset construction, while CUT3R is used during iterative generation for time efficiency and global alignment.
- CUT3R jointly estimates per-frame point clouds and uses dynamic-static disentanglement to preserve long-term static scene memory.