Source-linked AI summary
MosaicMem: Hybrid Spatial Memory for Controllable Video World Models
Wei Yu, Runjia Qian, Yumeng Li, Liquan Wang, Songheng Yin, Sri Siddarth Chakaravarthy P, Dennis Anthony, Yang Ye, Yidi Li, Weiwei Wan, Animesh Garg
TL;DR
Long-horizon video world models need spatial memory that remains consistent across camera motion, revisits, and intervention, but explicit and implicit paradigms have complementary limitations. MosaicMem combines 3D patch localization and targeted retrieval with native attention conditioning, and reports stronger pose adherence than implicit memory, stronger dynamic modeling than explicit memory, plus long navigation, editing, and autoregressive rollout capabilities.
Problem
Spatial memory remains a bottleneck for long-horizon, physically consistent video interaction because explicit memory struggles with rich dynamics while implicit memory can accumulate spatial drift and redundancy.
Method
MosaicMem lifts video patches into 3D for precise localization and retrieval, then composes aligned patches in the queried view while retaining attention-based conditioning for dynamic evolution.
Results
MosaicMem outperforms both explicit and implicit paradigms in the reported comparisons, supporting precise egomotion, prompt-driven dynamics, long navigation rollouts, scene editing, and autoregressive generation.
Takeaways & Limitations
The hybrid memory design provides a practical route to persistent, explorable video world simulators with camera control and evolving scene content.
Takeaways & Limitations
Implicit and explicit baselines retain distinct limitations: implicit memory can drift and remain redundant, while explicit memory restricts generative flexibility and makes long-horizon updates brittle.
Abstract
from arXiv · showhide
Video diffusion models are moving beyond short, plausible clips toward world simulators that must remain consistent under camera motion, revisits, and intervention. Yet spatial memory remains a key bottleneck: explicit 3D structures can improve reprojection-based consistency but struggle to depict moving objects, while implicit memory often produces inaccurate camera motion even with correct poses. We propose Mosaic Memory (MosaicMem), a hybrid spatial memory that lifts patches into 3D for reliable localization and targeted retrieval, while exploiting the model's native conditioning to preserve prompt-following generation. MosaicMem composes spatially aligned patches in the queried view via a patch-and-compose interface, preserving what should persist while allowing the model to inpaint what should evolve. With PRoPE camera conditioning and two new memory alignment methods, experiments show improved pose adherence compared to implicit memory and stronger dynamic modeling than explicit baselines. MosaicMem further enables minute-level navigation, memory-based scene editing, and autoregressive rollout.
1 Introduction
Long-horizon video world simulators require spatial memory that preserves scene structure across time and revisits while supporting camera control and evolving content. MosaicMem combines explicit 3D localization with implicit, prompt-following generation to improve consistency and enable persistent, controllable rollouts.
- Motivation: Video diffusion world simulators aim to support plausible futures, interaction, and decision-making beyond passive video synthesis.High-fidelity controllable rollouts make learned world simulators increasingly practical for visualizing multiple possible futures from observations.
- Motivation: Spatial memory remains unresolved for long-horizon, physically consistent interaction because existing designs work well in some regimes but break in others.Persistence requires object permanence, viewpoint consistency, and stable cause-and-effect under intervention across time and revisits.
- Existing paradigms: Explicit memory improves geometric and view consistency through projected 3D caches, but it has limited flexibility for rich dynamics and can accumulate reconstruction misalignments.Explicit systems build point clouds or 3D Gaussian caches for viewpoint-conditioned generation, while global reconstruction can make long-horizon updates brittle.
- Existing paradigms: Implicit memory accommodates dynamics and non-rigid changes through posed-frame representations, but it can suffer spatial drift, redundancy, and finite-context limitations.Its frame-based representation is flexible and differentiable, yet viewpoint transfer is not projection-enforced and direct geometric manipulation is unavailable.
- MosaicMem: MosaicMem lifts patches into 3D for precise localization and targeted retrieval while retaining native attention conditioning for dynamic, prompt-following generation.Its patch-and-compose interface places spatially aligned memory patches in the queried view, preserving persistent evidence while leaving evolving content for inpainting.
- Results and capabilities: MosaicMem shows more precise motion and pose adherence than implicit memory, more robust dynamic-object handling than explicit memory, and supports long navigation, editing, and autoregressive generation.The system maintains long-term memory for extremely long navigation video generation, directly copies or relocates patches for scene editing, and supports autoregressive rollout.
2 Methodology
MosaicMem combines explicit 3D patch localization with implicit attention-based conditioning to support camera-controlled, long-horizon video generation. It retrieves and aligns localized memory while leaving the generator flexibility to synthesize unseen content and dynamics.
- Task Definition: The task is to generate long-horizon rollouts that follow camera trajectories, retrieve spatial memory, and render prompt-consistent dynamics and unseen content.The formulation conditions generation on an input image, text prompts, camera poses, and spatial memory.
- Explicit vs. Implicit Memory: Explicit memory improves geometric consistency but restricts generative flexibility and can accumulate cross-view misalignment during updates.Its external 3D cache anchors projection-based conditioning, while global reconstruction makes long-horizon updates brittle.
- Explicit vs. Implicit Memory: Implicit memory accommodates dynamic and non-rigid changes through native conditioning, but pose errors can cause spatial drift and frame storage stresses efficiency and context limits.Because viewpoint transfer is not enforced by projection, revisits may drift even when retrieved frames remain prompt-compatible.
- Mosaic Memory: MosaicMem lifts patches into 3D for localized retrieval, then supplies retrieved patches to the video generator as conditioning.The method uses patches as an intermediate memory unit between explicit 3D structures and implicit video-frame memory.
- Mosaic Memory: MosaicMem’s native conditioning lets the generator choose between reconstructing from memory and synthesizing unseen content or new prompt-driven dynamics.This preserves implicit-memory flexibility while using explicit-style lifting for spatial alignment.
- Mosaic Memory: The memory space supports sparse or dense retrieval, direct patch manipulation, and more stable long-horizon updates than globally reconstructed structures.Localized patches can be displaced, duplicated, or removed, while independent storage avoids accumulated cross-view misalignment.
- Memory Alignment Through Warping: High spatiotemporal compression in 3D VAEs reduces latent coordinate resolution, so retrieved patches may miss the exact center of generated regions.This ambiguity motivates alignment mechanisms such as warped RoPE and complementary warping strategies.
- Memory Alignment Through Warping: Warped RoPE back-projects retrieved patches into 3D and re-projects them into the target camera to align memory across time and viewpoint.The method uses depth and source and target camera information, with differentiable sampling for latent patch warping.
3 Data Curation
MosaicMem-World is designed to evaluate spatial memory under viewpoint changes, especially revisitation and long-range re-localization. Its curation combines complementary video sources with geometric supervision and segment-wise static-plus-dynamic descriptions.
- Benchmark focus: MosaicMem-World targets spatial memory under viewpoint changes, including stable scene retention, re-localization after camera motion, and reuse of stored geometry and semantics.The benchmark intentionally collects trajectories that periodically revisit earlier checkpoints and regions within the same episode.
- Data sources: The benchmark aggregates four complementary data sources, each contributing on the order of tens of hours.One source consists of curated Unreal Engine 5 scenes with single and mixed actions and explicit revisited segments.
- Preprocessing: Videos receive reconstructed depth and camera motion through a unified preprocessing and annotation pipeline.Depth and camera motion are reconstructed using Depth Anything V3 or VIPE to provide a geometric scaffold for view-consistent representations and re-call evaluation.
- Annotations: Each sequence is partitioned into fixed-length segments of 32 frames with separate static-scene and temporal-dynamics descriptions.The descriptions cover first-frame layout, objects, and spatial relations, then camera motion, interactions, and state changes.
4 Evaluation
MosaicMem is evaluated across spatial memory, camera control, long-horizon generation, scene manipulation, and autoregressive rollout. It combines strong camera adherence and retrieval with dynamic modeling, while ablations identify complementary roles for PRoPE and two warping mechanisms.
- Spatial memory evaluation: MosaicMem achieves the best performance across camera control, visual quality, retrieval consistency, and motion dynamics.
- Spatial memory evaluation: Compared with explicit-memory baselines, MosaicMem produces substantially stronger motion dynamics and higher visual quality by modeling dynamic scene evolution.
- Spatial memory evaluation: Compared with implicit-memory baselines, MosaicMem significantly improves camera-motion accuracy and memory retrieval, with negligible camera drift over long horizons.
- Ablation study: PRoPE and MosaicMem progressively improve camera motion accuracy and memory retrieval, but the full model is more robust than MosaicMem alone under fine-grained motion and large camera movements.
- Ablation study: Warped Latent improves camera-motion accuracy, whereas Warped RoPE provides better visual quality and retrieval; combining both yields more robust memory conditioning.
- Long-horizon capabilities: MosaicMem recalls observed regions, renders moving objects, supports scene concatenation and surreal spatial connections, and enables autoregressive generation with strong quality and temporal consistency.
5 Related work
Prior spatial-memory methods fall into explicit and implicit paradigms, trading geometric consistency and interpretability against flexibility for dynamics.
- Explicit methods build persistent geometric structures such as point clouds or surfels, improving geometric consistency but struggling with complex dynamics.
- Implicit methods store past frames in learned representations, offering flexible scene modeling without explicit 3D structures.
6 Conclusion
MosaicMem bridges explicit and implicit spatial memory for long-horizon, camera-controlled video generation. It supports accurate retrieval, egomotion, prompt-driven dynamics, navigation, scene editing, and autoregressive rollout.
- MosaicMem lifts video patches for precise localization and retrieval while retaining attention-based conditioning for dynamic evolution.
- MosaicMem outperforms explicit and implicit paradigms and supports long navigation rollouts, direct memory manipulation, and autoregressive video generation.