Source-linked AI summary
VMem: Consistent Interactive Video Scene Generation with Surfel-Indexed View Memory
Runjia Li, Philip Torr, Andrea Vedaldi, Tomas Jakab
TL;DR
Long-term interactive video generation must follow user-specified camera paths while preserving scene consistency across revisited locations, but existing approaches face accumulated geometric errors or limited context windows. VMem indexes past views by the 3D surfels they observe, retrieves relevant views for conditioning, and achieves coherent scene synthesis with fewer context views and faster generation.
Problem
Long-term interactive video generation must preserve scene consistency while following user-specified camera paths, despite limitations in geometric reconstruction and short context windows.
Method
VMem anchors past views to observed 3D surfels and retrieves views covering surfaces relevant to a new viewpoint for conditioning novel-view generation.
Results
VMem outperforms existing methods in scene coherence while using fewer context views; with K = 4, it achieves a 12× speed improvement over SEVA and generates frames in 4.2 seconds versus 50 seconds.
Takeaways & Limitations
VMem supports scalable, long-term autoregressive scene generation with reduced computational costs for interactive exploration.
Takeaways & Limitations
Short-context video generators can develop severe inconsistencies when sequences extend beyond their small context windows.
Abstract
from arXiv · showhide
We propose a novel memory module for building video generators capable of interactively exploring environments. Previous approaches have achieved similar results either by out-painting 2D views of a scene while incrementally reconstructing its 3D geometry-which quickly accumulates errors-or by using video generators with a short context window, which struggle to maintain scene coherence over the long term. To address these limitations, we introduce Surfel-Indexed View Memory (VMem), a memory module that remembers past views by indexing them geometrically based on the 3D surface elements (surfels) they have observed. VMem enables efficient retrieval of the most relevant past views when generating new ones. By focusing only on these relevant views, our method produces consistent explorations of imagined environments at a fraction of the computational cost required to use all past views as context. We evaluate our approach on challenging long-term scene synthesis benchmarks and demonstrate superior performance compared to existing methods in maintaining scene coherence and camera control.
1. Introduction
Interactive long-video generation requires following user-specified camera paths while preserving scene identity over extended exploration. VMem addresses the limitations of accumulating reconstruction errors and short context windows by retrieving geometrically relevant past views.
- Problem: Interactive scene generation must follow user-directed camera paths while keeping revisited locations visually consistent.The paper motivates this requirement with explorations that may return to previously visited rooms.
- Existing limitations: Outpainting methods alternate 2D view generation with 3D reconstruction, but errors accumulate across outpainting, reconstruction, and stitching.These accumulated errors can severely degrade generated content after a short while.
- Existing limitations: Geometry-free video methods avoid reconstructed-scene errors but use small recent-frame context windows, limiting long-term consistency.Their high computational cost constrains how many conditioning views can be included.
- VMem: VMem anchors past views to the 3D surfels they observed and retrieves views associated with surfaces visible from a new viewpoint.Surfel attributes store indices of past viewpoints, while estimated geometry constructs the memory rather than serving as the final scene representation.
- VMem: VMem renders surfel viewpoint attributes from the novel camera, selects the top K frequently represented indices, and retrieves the corresponding RGB views and camera parameters.This relevance-based retrieval does not require highly accurate scene geometry as long as the relevant views are retrieved.
- Results: 4× fewer context views and a 12× speedup are reported while maintaining comparable performance and improving long-term consistency.The contribution summary presents these efficiency gains alongside long-term coherent video generation and benchmark improvements over existing methods.
2. Related work
The related work spans geometric inpainting and geometry-free multi-view extrapolation, each with distinct consistency or scalability limitations. VMem instead uses surfel-indexed retrieval to connect past views to approximate scene surfaces and support autoregressive generation.
- Novel view synthesis: Novel view synthesis includes interpolation near input views and extrapolation that extends substantially beyond the original scene with new content.The paper identifies extrapolation from single images as particularly difficult.
- Inpainting-based view extrapolation: Inpainting-based methods reproject reconstructed meshes, point clouds, or Gaussian splats into novel views and fill missing regions with 2D inpainting.These methods use pretrained 3D reconstruction models before applying inpainting to projected views.
- VMem: VMem updates its surfel-indexed memory after generation by appending new view indices to observed surfels or creating new surfels from generated-view geometry.This update is repeated autoregressively for long-term consistent generation.
- Inpainting-based view extrapolation: Inaccurate depth or point-map estimates can propagate distortions across generated views, while high-fidelity 3D representations are costly to store and process for large scenes.Once constructed, representation inaccuracies become difficult to correct.
- Multi-view-based view extrapolation: Multi-view methods avoid explicit 3D modeling by conditioning novel-view generation on previously rendered or generated views.Examples use autoregressive likelihoods, predefined trajectories, cross-view attention, or epipolar constraints.
- Memory-based approaches: Distance- and field-of-view-based memory retrieval can recover spatially correlated views but lacks geometric reasoning for occlusions.The paper contrasts these approaches with VMem's explicit surfel-based memory.
3. Method
The method generates short batches of camera-controlled views while maintaining an unbounded history through Surfel-Indexed View Memory. VMem retrieves geometrically relevant views using coarse surfel geometry, then updates the index with newly generated views.
- Interactive generation: The system autoregressively generates M novel RGB views for user-specified cameras while conditioning on previously generated views.After each batch, the number of generated frames increases by M, supporting interactive exploration.
- View retrieval: VMem replaces recent-frame selection with dynamic retrieval of past views that are relevant to the target scene region.The method prioritizes views sharing visible regions with the novel viewpoint rather than treating all history equally.
- Reading memory: The reading procedure renders surfels from the average target pose and selects the top-K most frequent view indices as generator context.Retrieved views are then passed to the novel-view generator.
- Surfel-indexed memory: Each surfel stores indices of past views that observed it, allowing visible surfels from a novel viewpoint to vote for relevant references while accounting for occlusion.An octree accelerates surfel retrieval based on geometry.
- Memory management: Non-maximum suppression reduces redundant views from similar poses and promotes broader scene coverage during retrieval and memory writing.When writing, highly similar poses cause the older view to be discarded.
- Writing memory: After generation, an off-the-shelf point-map estimator jointly aligns new and retrieved views, converts new point maps into coarse surfels, and updates the memory index.The resulting surfels store the new frame indices and are merged with existing surfels.
4. Experiments
VMem is evaluated on standard novel-view benchmarks, revisitation-focused cycle trajectories, qualitative long sequences, and retrieval ablations. Across these settings, it maintains stronger consistency while reducing context and computation requirements.
- Evaluation setup: The evaluation covers RealEstate10K, Tanks-and-Temples, in-the-wild scenes, and a cycle-trajectory protocol designed to test revisitation.The protocol returns the camera along the same path to the starting position, addressing the limited revisitation in standard benchmarks.
- Benchmark results: VMem substantially outperforms all baselines on short- and long-term novel-view generation metrics.The benchmark evaluates the fifth generated image for short-term synthesis and the final image after at least 200 frames for long-term synthesis.
- Generalization: VMem consistently outperforms baselines across in-domain and out-of-domain cycle evaluations on the majority of metrics.The out-of-domain evaluation uses all six advanced Tanks-and-Temples scenes with indoor and outdoor content and more dynamic camera trajectories.
- Ablation study: The ablation shows VMem improves generator performance for both context sizes, with the largest improvement at K = 4.The compared retrieval strategies include temporal, camera-distance, and field-of-view retrieval.
- Computational efficiency: With K = 4 views, VMem achieves a 12× speed improvement over SEVA while recovering most performance obtained with K = 17 views.Frame generation takes 4.2 seconds for VMem versus 50 seconds for SEVA on an RTX 4090 GPU.
5. Conclusion
VMem is presented as a plug-and-play memory module for long-term autoregressive scene generation. It improves scene consistency and coherence while reducing computational costs and context requirements.
- VMem anchors past views to a surfel-based scene representation and retrieves the most relevant views.
- VMem improves long-term scene consistency while reducing computational costs.
- Experiments on long-term scene synthesis benchmarks show that VMem outperforms existing methods in scene coherence.
- The method enables fewer context views and significantly faster generation.
A. Implementation details
The implementation fine-tunes a more efficient SEVA version by reducing reference and target views and training with LoRA on multiple GPUs.
- The efficient SEVA version uses K = 4 reference frames and M = 4 target views.
- LoRA with rank 256 is applied while context views are randomly sampled online during training.
- Training runs for 600,000 iterations on 8 A40 GPUs with batch size 24 per GPU and AdamW optimization.
B. Average pose calculation
Average camera poses for surfel rendering are computed by averaging translations directly and rotations through aligned, normalized quaternions.
- Translations tT +m are averaged using a simple mean.
- Rotations RT +m are converted into quaternions before averaging.
- Quaternion signs are aligned to a common hemisphere and the mean quaternion is normalized.
C. Autoregressive point map prediction
Autoregressive point-map prediction maintains consistency in a shared coordinate space by freezing user-defined camera trajectories and previously predicted depth maps during optimization.
- Point maps are generated for each view autoregressively in a shared coordinate space.
- User-defined camera parameters are frozen to preserve the fixed camera trajectory.
- Previously predicted depth maps are frozen at each generation step during optimization.
D. Limitations and discussion
VMem is evaluated using cyclic trajectories because no established benchmark exists for long-term consistency in scene video generation. The evaluation has limited scope, and the model also faces generalization and inference-speed constraints.
- Cyclic trajectories serve as a proxy because no established benchmark evaluates long-term consistency in scene video generation.
- The trajectories contain limited occlusions, so they do not fully demonstrate VMem's potential for handling occlusions.
- Existing metrics primarily measure low-level texture similarity rather than true multi-view consistency, motivating standardized evaluation protocols.
- Fine-tuning only on RealEstate10K, dominated by indoor scenes, may limit generalization to natural landscapes and images with moving objects.
- 4.16 seconds per frame on an RTX 4090 falls short of the real-time performance required for applications such as virtual reality.
- VMem's performance is expected to improve as its off-the-shelf image-set generator and point map predictor advance.