Source-linked AI summary
Spatia: Video Generation with Updatable Spatial Memory
Jinjing Zhao, Fangyun Wei, Zhening Liu, Hongyang Zhang, Chang Xu, Yan Lu
TL;DR
Video generation models struggle to preserve long-term spatial and temporal consistency because video histories are dense and high-dimensional. Spatia addresses this with an explicitly maintained 3D scene point cloud updated through visual SLAM, improving spatial consistency while supporting dynamic content, camera control, and interactive editing.
Problem
Dense, high-dimensional video signals make it impractical for existing video generation models to directly encode long-term historical context.
Method
Spatia iteratively generates clips conditioned on a persistent 3D scene point cloud and updates that memory with visual SLAM.
Results
Spatia significantly enhances spatial consistency without compromising dynamic entities or visual quality, while enabling long-horizon generation and spatial editing.
Takeaways & Limitations
Persistent spatial memory provides a geometrically grounded basis for consistent revisits, explicit camera control, and 3D-aware interactive editing.
Abstract
from arXiv · showhide
Existing video generation models struggle to maintain long-term spatial and temporal consistency due to the dense, high-dimensional nature of video signals. To overcome this limitation, we propose Spatia, a spatial memory-aware video generation framework that explicitly preserves a 3D scene point cloud as persistent spatial memory. Spatia iteratively generates video clips conditioned on this spatial memory and continuously updates it through visual SLAM. This dynamic-static disentanglement design enhances spatial consistency throughout the generation process while preserving the model's ability to produce realistic dynamic entities. Furthermore, Spatia enables applications such as explicit camera control and 3D-aware interactive editing, providing a geometrically grounded framework for scalable, memory-driven video generation.
1. Introduction
Spatia addresses the difficulty of preserving long-term video context by maintaining an explicit 3D scene point cloud as persistent spatial memory and updating it during iterative generation. This design supports spatially consistent revisits, dynamic entities, explicit camera control, and 3D-aware editing.
- Motivation: 36,000 spatio-temporal tokens represent only a 5-second 480P video, making direct modeling of minute- or hour-scale histories impractical.An additional 5-second clip would substantially increase computational and memory demands.
- Core approach: Spatia maintains an explicit 3D scene point cloud as persistent spatial memory, generates clips conditioned on it and prior clips, and updates it with visual SLAM.The iterative process incorporates new content while preserving existing scene information.
- Dynamic–Static Disentanglement: Dynamic–static disentanglement preserves a scene point cloud while generating dynamic entities that interact coherently with the scene.This differs from prior memory-oriented methods described as limited to static scenes.
- Spatially Consistent Generation: Retrieving spatial memory allows diverse sequences from different viewpoints while preserving a consistent spatial structure.The capability targets repeated views of the same location.
- Explicit Camera Control: Spatia applies a desired camera path directly to the 3D point cloud and renders a 2D point-cloud sequence to guide generation.This provides geometrically grounded camera conditioning rather than encoding trajectories only into latent features.
- 3D-Aware Interactive Editing: Users can remove or modify objects in the 3D scene before generation, and those edits are reflected in the resulting videos.The paper presents this as fine-grained control over scene composition and content.
- Results: Spatia experimentally enhances spatial consistency without compromising dynamic entities or visual quality, while enabling long-horizon generation and spatial editing.The paper reports these outcomes as benefits of the spatial memory mechanism.
2. Related Works
Related work frames long-term video consistency as a memory problem that is difficult to solve with conventional attention and caching. Recent approaches therefore introduce memory-based architectures for preserving spatial coherence.
- Video generation models: Video generation has progressed from UNet-based latent diffusion models to large-scale Diffusion Transformers.The passage presents this as a rapid architectural evolution of the field.
- Long-term context: Modern language models extend long-term memory through million- or ten-million-token context windows and techniques such as KV-cache compression.The passage contrasts these capabilities with video-generation context limitations.
- Video generation memory: Bidirectional spatiotemporal attention in most diffusion models prevents standard KV caching, limiting context windows and access to previously generated content.This restriction motivates memory-based approaches for long-term spatial consistency.
3. Method
Spatia conditions video generation on persistent 3D scene geometry and iteratively updates that spatial memory during user-guided generation. Its training pipeline estimates view-specific point clouds, retrieves spatially relevant references, and integrates these signals with video, text, and temporal context.
- Problem Formulation: Spatia maintains a static 3D scene point cloud as explicit geometric memory for multimodal video generation.The point cloud anchors generated content within a coherent spatial layout.
- Training Data: Training decomposes each video into target, preceding, and candidate-frame sets, using candidate frames as potential spatial and geometric references.The preceding clip provides temporal context, while candidate frames support spatial consistency.
- Scene Point Cloud Estimation: Spatia estimates a scene point cloud from a candidate frame, removes dynamic entities when needed, and renders view-specific point-cloud sequences using camera poses.These sequences are produced for target, preceding, and candidate viewpoints.
- Reference Frame Retrieval: The method retrieves up to K candidate frames with spatial overlap to the target clip, providing additional geometric cues during generation.Spatial correspondence is computed between target and candidate scene point clouds.
- Architecture: The generation network combines text, reference-frame, preceding-video, and scene-projection tokens, with ControlNet blocks injecting spatial conditions into the main architecture.The model is trained with flow matching to guide noisy tokens toward target video tokens.
- Inference: During inference, users specify text and camera trajectories while newly generated and prior clips update the scene point cloud for subsequent iterations.This supports interactive generation conditioned on the current 3D scene.
4. Experiment
Spatia is evaluated for visual quality and spatial-memory effectiveness across WorldScore, RealEstate, closed-loop, ablation, long-horizon, and point-cloud-density settings. The experiments compare its visual and memory behavior against baselines and measure how design choices affect consistency and quality.
- Main Results: WorldScore compares static-scene models, foundation video models, and Spatia using aggregated static, dynamic, and average world scores.The comparison reflects differing abilities to handle dynamic entities and persistent memory.
- Evaluation: The evaluation measures visual quality and memory-mechanism effectiveness using WorldScore and the RealEstate test set.WorldScore covers static and dynamic visual quality, while RealEstate enables PSNR, SSIM, and LPIPS comparisons against ground truth.
- Main Results: RealEstate evaluations reproduce baseline methods under default configurations on the same test samples for a fair comparison.The benchmark reports PSNR, SSIM, and LPIPS against corresponding ground-truth videos.
- Memory Mechanism Evaluation: Closed-loop memory evaluation compares final frames with initial viewpoints using PSNR, SSIM, LPIPS, and Match Accuracy.The setup tests whether generated videos preserve spatial memory when the camera returns to its starting viewpoint.
- Ablation Studies: Ablations test scene projection videos, reference frames, and different numbers of retrieved references for spatial-memory modeling and camera control.The experiments isolate contributions of spatial conditions and reference-frame count.
- Ablation Studies: Long-horizon experiments use 2, 4, and 6 autoregressive clips with camera paths that return to the original viewpoint, while density studies vary voxel size.Larger voxel cubes reduce memory storage but degrade visual quality by removing fine-grained spatial guidance.
5. Conclusion
Spatia enables consistent, long-horizon video synthesis by maintaining an explicit 3D scene point cloud as persistent memory and updating it during generation.
- Spatia maintains an explicit 3D scene point cloud as persistent memory and iteratively updates it during generation.The framework captures long-term geometric structure that conventional video models cannot preserve.
6. More Implementation Details
Spatia retrieves spatially relevant reference frames and conditions future-frame generation on preceding frames to support long-horizon video synthesis.
- Spatia selects up to K spatially relevant frames, with K = 7 by default, from a candidate-frame set.The complete retrieval procedure is specified in Algorithm 1.
7. Visualization
The visualizations examine spatial memory through variant comparisons, closed-loop consistency, dynamic–static disentanglement, and 3D-aware editing.
- Figure 5 compares Spatia with variants using only scene videos or only reference frames for long-horizon video generation.The comparison evaluates the separate and combined roles of reference frames and scene videos.
- Figure 6 evaluates spatial memory by comparing the first and final frames after the camera follows a trajectory returning to its initial viewpoint.This closed-loop setup examines visual and geometric consistency through final-frame alignment with the first frame.
- Figure 7 shows that Spatia stores only the static scene point cloud while generating videos containing dynamic entities acting within the same scenes.Dynamic entities are removed from the estimated memory while videos containing them remain training targets.
- Figure 8 demonstrates geometrically precise edits by modifying the scene point cloud to remove objects, add objects, or alter object attributes.The edits produce corresponding changes in generated videos.