Source-linked AI summary
MoVerse: Real-Time Video World Modeling with Panoramic Gaussian Scaffold
Yang Zhou, Ziheng Wang, Yuqin Lu, Haofeng Liu, Jun Liang, Shengfeng He, Jing Li
TL;DR
Generating an interactively navigable, spatially persistent world from one narrow-field-of-view image requires completing unseen surroundings, constructing controllable geometry, and rendering coherent video. MoVerse builds a panoramic Gaussian scaffold and uses a causal Gaussian-conditioned renderer, supporting real-time roaming at 8 FPS on one NVIDIA RTX 4090 GPU.
Problem
Generating a navigable world from one narrow-field-of-view image requires completing missing views, constructing a persistent controllable scene, and rendering coherent high-quality video during interaction.
Method
MoVerse generates a gravity-aligned 360° panorama, lifts it into a persistent 3D Gaussian scaffold, and renders interactive video with a Gaussian-conditioned autoregressive model.
Results
8 FPS: MoVerse supports real-time scene roaming on a single NVIDIA RTX 4090 GPU.
Takeaways & Limitations
A persistent panoramic Gaussian scaffold provides durable spatial memory and camera controllability while the causal renderer synthesizes high-fidelity observations.
Takeaways & Limitations
MoVerse’s final world is bounded by panoramic completion quality, while approximate Gaussian geometry can produce artifacts under difficult scene conditions.
Abstract
from arXiv · showhide
We present MoVerse, a real-time video world model that creates an interactively navigable scene from a single narrow-field-of-view image. This setting is challenging because the input observes only a small fraction of the environment, while interactive roaming requires a complete surrounding world, persistent geometry, controllable camera motion, and temporally coherent high-fidelity observations. MoVerse addresses this problem by separating world construction from observation rendering. It first expands the input into a gravity-aligned 360$^\circ$ panorama with topology-aware diffusion, closing the missing field of view before 3D reasoning. It then lifts the panorama into a persistent 3D Gaussian scaffold using panoramic geometry-aware residual prediction, yielding a dense and directly renderable spatial memory. Finally, a Gaussian-conditioned video renderer translates scaffold renderings along user-specified camera trajectories into photorealistic video. To make this renderer practical for interaction, we train a bidirectional diffusion teacher for high-quality conditional rendering and distill it into a causal autoregressive student for bounded-latency streaming. This design combines the controllability and long-range consistency of explicit 3D representations with the perceptual quality of generative video models. MoVerse supports real-time scene roaming at 8~FPS on a single NVIDIA RTX~4090 GPU, demonstrating a practical path toward single-image world creation with interactive video output.
1 Introduction
MoVerse addresses single-image navigable-world construction by separating world construction from observation rendering. It completes a gravity-aligned panorama, builds a persistent panoramic Gaussian scaffold, and generates interactive video along user-specified camera trajectories.
- Problem: Single NFOV images provide insufficient environmental coverage, yet interactive applications require a complete, spatially persistent world supporting camera movement and coherent observations.The introduction frames missing-field-of-view completion, controllable 3D representation, and persistent rendering as coupled requirements.
- Limitations of prior work: Existing explicit 3D methods offer durable spatial memory and camera control but must infer most scene content from weak evidence when lifted from one NFOV image.Explicit representations include point clouds, meshes, and 3D Gaussian scenes.
- Limitations of prior work: Implicit video and world models produce visually strong videos, but long trajectories or revisits can cause geometric drift, identity changes, or boundary artifacts.Their long-range stability depends on attention windows, recurrent states, or key–value caches.
- MoVerse: MoVerse uses three stages: gravity-aligned 360° panorama generation, persistent 3D Gaussian scaffold construction, and Gaussian-conditioned autoregressive video rendering.The system separates offline scaffold construction from interactive observation rendering along user-specified camera trajectories.
- Design rationale: The factorized design supplies omnidirectional evidence, converts it into a reusable renderable asset, and preserves scaffold-encoded camera motion and layout during causal video rendering.Stage I completes the panorama, Stage II creates persistent scene state, and Stage III improves perceptual quality and temporal coherence without replacing scaffold control.
2 Method · 2.1 Overview
MoVerse turns a single NFOV image into a real-time navigable video stream through panoramic completion, persistent Gaussian scene construction, and Gaussian-conditioned video rendering. Offline scaffold construction separates computationally heavy world creation from online trajectory-conditioned rendering for interactive exploration.
- 2.1 Overview: MoVerse processes a single NFOV image through three sequential stages: panoramic completion, persistent Gaussian scene generation, and interactive video rendering.The stages respectively complete omnidirectional context, store the world in a Gaussian scaffold, and translate scaffold renderings into temporally coherent video.
- 2.1 Overview: Stage I expands the input into a gravity-aligned 360° equirectangular panorama before 3D lifting, providing structurally complete observations for every viewing direction.This ordering reduces the need for 3D reasoning to hallucinate large unobserved regions.
- 2.1 Overview: Stage II lifts the panorama into a panoramic 3D Gaussian scene whose elements encode center, covariance, opacity, and appearance.The scaffold is represented as G = {(µk, Σk, αk, ck)}K and provides persistent spatial memory for subsequent rendering.
- 2.1 Overview: 3D Gaussians are directly splatting-renderable at real-time frame rates, producing dense, camera-controllable RGB conditioning video that is more informative than sparse point-cloud projections.The Gaussian-rendered conditioning remains imperfect as final photorealistic observation, motivating the learned video renderer.
- 2.1 Overview: The panoramic-generation module auto-levels the input and uses topology-aware ERP completion with circular latent encoding and shift-equivariant generation to preserve the horizontal S1 boundary.These operations establish a gravity-aligned canonical viewing space while respecting the panorama’s circular topology.
- 2.1 Overview: Stage III translates Gaussian-rendered conditioning video into high-fidelity output video while preserving temporal coherence across successive viewpoints and the scaffold-implied camera motion and scene layout.The renderer generates sequences rather than independent frames, enhancing perceptual quality without altering the requested trajectory or scene structure.
- 2.1 Overview: Stages I and II run once offline to produce a persistent 3DGS asset in seconds, while Stage III renders requested trajectories online and streams enhanced frames in real time.This offline–online split prevents panorama synthesis and 3D lifting from bottlenecking interactive exploration.
2.2 Stage I: Panoramic Generation
Stage I converts a single narrow-field-of-view image into a complete, gravity-aligned equirectangular panorama by canonicalizing the input and completing missing regions with topology-aware latent diffusion. The resulting horizontally periodic panorama is centered on the input view and provides the canonical input for subsequent 3D construction.
- Pipeline: Stage I expands the input NFOV image into a complete equirectangular panorama through canonicalization, masked diffusion completion, and topology-aware decoding.The pipeline stabilizes the input in a canonical panoramic frame, uses it as masked visual context, and produces a horizontally closed ERP panorama.
- Latent diffusion: The generator uses masked latent diffusion to preserve the observed input while synthesizing the unobserved panoramic field of view.Training encodes target panoramas into latent space with an ERP mask, while inference conditions denoising on the aligned input context.
- Canonicalization: Canonicalization aligns the environmental horizon with the ERP equator and keeps vertical structures plumb, yielding stable longitude–latitude coordinates.The canonical viewing space removes pitch and roll ambiguity and normalizes the panoramic observation.
- Topology: Circular latent encoding and shift-equivariant diffusion enforce horizontal periodicity and reduce dependence on a fixed ERP seam.Circular padding wraps latent convolutions around the azimuthal axis, while shifted denoising branches and randomized inference shifts discourage boundary artifacts.
- Output and training: The final output P = Gθ(I) is an equirectangular, gravity-aligned, horizontally periodic panorama centered around the input view for the following 3D construction stage.Horizon360 training uses canonicalized panoramas and perspective views sampled across variable camera poses, fields of view, and aspect ratios.
2.3 Stage II: Gaussian Scene Generation
Stage II converts the gravity-aligned panorama into a persistent, directly renderable 3D Gaussian scaffold. It combines ERP-aware spherical initialization with residual prediction in angular–inverse-depth space, then trains the scaffold for faithful, stable novel-view rendering.
- Gaussian scaffold generation: Stage II runs a feed-forward 3DGS predictor once per panorama to produce a reusable Gaussian scaffold and RGB conditioning frames for downstream video rendering.The output is a standard splatting-renderable 3DGS asset.
- ERP-aware initialization: The method initializes Gaussian primitives on a strided ERP grid by spherical back-projection from panorama depth, with latitude-aware scales that shrink toward the poles.A lower bound prevents degenerate splats near the poles.
- Panoramic residual prediction: The generator adapts depth-guided residual 3DGS prediction to panoramas by modeling spherical geometry, latitude-varying sampling density, and closed horizontal topology.The prediction head is zero-initialized, so training begins from the physically meaningful spherical scaffold and learns local corrections.
- Panoramic residual prediction: Gaussian centers are corrected through angular–inverse-depth offsets for longitude, latitude, and depth, preserving consistency with ERP sampling and horizontal S1 closure.Other attributes use bounded residual updates, including scale, rotation, color, and opacity, while maintaining valid parameter ranges.
- Training objectives: Training combines target-view reconstruction, source-panorama reconstruction, inverse-depth supervision, and residual regularization to make the scaffold faithful, geometrically stable, and directly renderable.The model is trained on HM3D using differentiable rendering.
2.4 Stage III: Gaussian-Conditioned Video Rendering
Stage III transforms Gaussian-scaffold renderings into photorealistic, trajectory-consistent video while repairing rendering artifacts rather than hallucinating a world from scratch. A bidirectional diffusion teacher is distilled into a causal renderer for bounded-latency interactive roaming, reaching 8 FPS on one NVIDIA RTX 4090 GPU.
- Rendering role: Stage III enhances Gaussian-scaffold renderings into final video, preserving requested camera motion, persistent scene layout, and long-range spatial anchoring while repairing floaters, aliasing, disocclusion holes, and temporal artifacts.The scaffold provides controllable spatial conditioning, while the renderer improves appearance instead of reconstructing the world independently.
- Student distillation: The causal autoregressive student generates one latent block at a time from current scaffold frames and recent history, converting the teacher into a bounded-latency streaming renderer.It is trained under autoregressive self-rollout conditions and uses a local key–value cache.
- Training curriculum: The training curriculum progresses from trajectory-aligned geometric proxies to explicit Gaussian renders and panorama-conditioned Gaussian renders, matching increasingly realistic deployment conditions.This progression teaches artifact repair under camera motion, Gaussian-splatting failure modes, and the Stage II-to-Stage III interface.
- Deployment: 8 FPS end-to-end roaming is achieved on a single NVIDIA RTX 4090 GPU with K = 1 latent frame per block, two-step denoising, and MemRoPE-style local caching.Each latent block corresponds to four pixel-space frames under the Wan VAE temporal stride.
3 Results
MoVerse is evaluated qualitatively across its full three-stage pipeline, from a single NFOV image to panoramic completion, explicit Gaussian-scaffold rendering, and autoregressive video output. The results also examine Stage I panorama coherence, Stage II camera-controllable rendering, Stage III trajectory-conditioned enhancement, and the bidirectional teacher used for distillation.
- Full pipeline: End-to-end examples show the input NFOV image, Stage I ERP panorama, Stage II Gaussian renderings, and Stage III outputs along the same camera trajectory.The results illustrate the complementary roles of the explicit scaffold and learned renderer.
- Stage I panoramic generation: Stage I completes perspective NFOV inputs into gravity-aligned ERP panoramas that support multiple perspective views, including seam-crossing views.Seam-crossing views test coherence under the ERP domain’s horizontal S1 topology.
- Stage II Gaussian scaffold rendering: Stage II renders novel views directly from the 3D Gaussian scaffold while preserving the completed panorama’s global layout along camera trajectories.The scaffold provides camera-controllable Gaussian-rendered observations.
- Stage III autoregressive rendering: Stage III converts Gaussian-rendered conditioning frames into visually coherent video observations while streaming along user-specified camera trajectories and preserving the explicit scene layout.The outputs enhance scaffold renderings during trajectory-conditioned generation.
- Bidirectional conditional teacher: The bidirectional Gaussian-conditioned teacher attends to the full temporal window and supplies the visual target for distilling the causal autoregressive student.It serves as a quality model for Gaussian-conditioned video rendering.
4 Discussion
MoVerse separates persistent long-range spatial memory in a panoramic 3D Gaussian scaffold from causal video rendering for local, high-fidelity observations. This factorization improves practical controllability but remains limited by panorama quality and approximate geometry, motivating feedback between stages and extensions to dynamic, editable worlds.
- Hybrid design: MoVerse stores long-range spatial memory in a persistent panoramic 3D Gaussian scaffold while using a causal renderer for local, high-fidelity observation synthesis.The scaffold fixes camera trajectories and scene layout over long horizons, while the video model repairs splatting artifacts and fills small gaps.
- Limitations: The final world is bounded by Stage I panorama quality, because semantic, horizon, or behind-camera errors can be faithfully lifted into the scaffold and rendered more convincingly.The factorization may preserve and amplify incorrect scene content rather than correct it.
- Future work: Future work should connect stages through uncertainty-guided density allocation, error-triggered panorama or geometry refinement, and renderer-to-scaffold consistency signals.The paper also proposes extending the static-scene assumption to dynamic objects and editable worlds.