Source-linked AI summary
Lyra 2.0: Explorable Generative 3D Worlds
Tianchang Shen, Sherwin Bahmani, Kai He, Sangeetha Grama Srinivasan, Tianshi Cao, Jiawei Ren, Ruilong Li, Zian Wang, Nicholas Sharp, Zan Gojcic, Sanja Fidler, Jiahui Huang, Huan Ling, Jun Gao, Xuanchi Ren
TL;DR
Lyra 2.0 addresses the spatial forgetting and temporal drifting that limit 3D-consistent video generation during long-horizon exploration. It routes information with per-frame 3D geometry, trains on self-augmented histories, and supports high-quality 3D reconstruction for interactive exploration and simulation.
Problem
Long-horizon camera-controlled generation must preserve 3D consistency across large viewpoint changes and revisits, but current models suffer from spatial forgetting and temporal drifting.
Method
Lyra 2.0 uses per-frame 3D geometry to retrieve relevant history and establish correspondences, while self-augmented training exposes the model to imperfect outputs to reduce drift.
Results
Lyra 2.0 enables large-scale, explorable 3D environments with long-horizon consistency and generated scenes suitable for interactive exploration, virtual reality, and simulation.
Takeaways & Limitations
The generated scenes can be directly deployed for interactive exploration, virtual reality experiences, and simulation.
Takeaways & Limitations
The framework focuses on static environments, and training-data exposure variations may produce photometric inconsistencies that cause artifacts in feed-forward 3DGS reconstruction.
Abstract
from arXiv · showhide
Recent advances in video generation enable a new paradigm for 3D scene creation: generating camera-controlled videos that simulate scene walkthroughs, then lifting them to 3D via feed-forward reconstruction techniques. This generative reconstruction approach combines the visual fidelity and creative capacity of video models with 3D outputs ready for real-time rendering and simulation. Scaling to large, complex environments requires 3D-consistent video generation over long camera trajectories with large viewpoint changes and location revisits, a setting where current video models degrade quickly. Existing methods for long-horizon generation are fundamentally limited by two forms of degradation: spatial forgetting and temporal drifting. As exploration proceeds, previously observed regions fall outside the model's temporal context, forcing the model to hallucinate structures when revisited. Meanwhile, autoregressive generation accumulates small synthesis errors over time, gradually distorting scene appearance and geometry. We present Lyra 2.0, a framework for generating persistent, explorable 3D worlds at scale. To address spatial forgetting, we maintain per-frame 3D geometry and use it solely for information routing -- retrieving relevant past frames and establishing dense correspondences with the target viewpoints -- while relying on the generative prior for appearance synthesis. To address temporal drifting, we train with self-augmented histories that expose the model to its own degraded outputs, teaching it to correct drift rather than propagate it. Together, these enable substantially longer and 3D-consistent video trajectories, which we leverage to fine-tune feed-forward reconstruction models that reliably recover high-quality 3D scenes.
1. Introduction
Generative reconstruction uses camera-controlled video to create scalable 3D scenes, but long-horizon exploration is hindered by spatial forgetting and temporal drifting. Lyra 2.0 addresses these failures with 3D-guided information routing, self-augmented training, and a reconstruction model adapted to generated sequences.
- Generative reconstruction synthesizes dense novel views along a prescribed camera trajectory and lifts them into explicit 3D geometry and appearance.
- Long-horizon exploration suffers from spatial forgetting when revisited regions leave temporal context and from temporal drifting caused by accumulated autoregressive errors.
- Lyra 2.0 uses per-frame 3D geometry to retrieve relevant history and establish correspondences while leaving appearance synthesis to the diffusion prior.
- Self-augmentation trains the model on its own imperfect predictions, exposing it to degraded conditioning so it can correct temporal drift.
- A feed-forward 3D Gaussian Splatting model is fine-tuned on generated sequences to tolerate minor inconsistencies and produce reliable reconstructions.
- Lyra 2.0 supports arbitrary long-horizon trajectories from a single image and reconstructs explored environments as high-quality 3D Gaussians and meshes.
2. Related Work
Prior work adds camera control and memory to video generation, but persistent 3D consistency remains limited during long explorations and revisits. Lyra 2.0 routes information through per-frame geometry and combines long-video generation with scalable feed-forward reconstruction.
- Camera-Conditioned Video Generation: Camera-conditioned video methods improve viewpoint control, yet generated scenes often lose 3D consistency when revisiting previously generated regions.
- Memory-Aware Long Video Generation: Memory-aware approaches address fixed temporal context through frame retrieval, accumulated 3D representations, or architectural changes for temporal coherence.
- Memory-Aware Long Video Generation: Lyra 2.0 maintains per-frame geometry only for retrieving history and establishing correspondences, while the generative prior synthesizes appearance.
- 3D Scene Generation: Feed-forward 3D scene-generation methods typically have limited view coverage, whereas Lyra 2.0 generates long consistent videos from one image and lifts them into Gaussians and meshes.
3. Preliminaries
The preliminaries describe latent video diffusion, autoregressive segment generation, camera conditioning, and history compression. These components provide the basis for controlling viewpoints and extending temporal context during scene exploration.
- DiT-Based Latent Video Diffusion: A VAE compresses RGB videos into spatiotemporal latents that a decoder reconstructs, supporting DiT-based latent video diffusion.
- DiT-Based Latent Video Diffusion: Long videos are generated as fixed-length segments autoregressively, with each segment conditioned on previously generated frames.
- Camera-Conditioned Video Generation: Depth-based warping and Plücker ray injection provide complementary camera conditioning for ordinary and drastic viewpoint changes.
- Context Compression via FramePack: FramePack compresses history using finer tokenization for recent frames and more aggressive compression for distant frames, extending the temporal horizon.
- Context Compression via FramePack: The initial image remains at full resolution as a fixed anchor that preserves the original scene appearance during generation.
4. Method
Lyra 2.0 generates long, camera-controlled videos through iterative spatial retrieval, guided synthesis, and memory updates, then lifts them into explicit 3D representations. Its anti-forgetting and anti-drifting mechanisms use per-frame geometry for routing and self-augmented histories for robust autoregressive generation.
- Overview: Lyra 2.0 autoregressively retrieves relevant history, generates a video segment from spatial and temporal context, and updates memory for continued navigation.Users provide a 3D camera trajectory and optional text prompt; generated segments are progressively lifted into 3D point clouds.
- Anti-Forgetting: Canonical coordinate warping establishes dense spatial correspondences without transferring RGB appearance, leaving novel-view synthesis to the diffusion prior.Warped correspondence maps include canonical coordinates and depth, are encoded with positional encoding, and injected into DiT self-attention.
- Anti-Forgetting: The 3D cache stores each frame’s depth, camera parameters, and downsampled world-coordinate point cloud independently rather than fusing them globally.Per-frame storage avoids accumulating cross-view misalignment from degraded depth estimates into one corrupted reconstruction.
- Anti-Forgetting: Geometry-aware retrieval selects history frames with maximal target-view visibility and greedily maximizes uncovered-pixel coverage, enabling recalls after hundreds of frames.Retrieved observations are injected as spatial slots alongside temporally compressed history and generation tokens.
- Anti-Drifting: Self-augmentation trains the model on corrupted or self-predicted history so it learns to recover quality from imperfect context during autoregressive inference.The procedure addresses observation bias between clean training histories and imperfect inference outputs, requiring one additional DiT forward pass.
- 3D Reconstruction: The generated videos are lifted into explicit 3D representations for downstream applications including embodied-AI simulation and virtual reality.The system exports reconstructed content as 3D Gaussians or meshes.
5. Experiments
Experiments show that Lyra 2.0 sustains realistic, controllable long-horizon videos and converts them into higher-quality, larger-scale 3D scenes, while ablations validate its core design choices and applications demonstrate practical utility.
- Long Video Generation: Lyra 2.0 achieves the best results on both datasets across nearly all long-video metrics, combining visual fidelity with accurate camera control.The evaluation includes SSIM, LPIPS, FID, subjective quality, style consistency, camera controllability, and reprojection error.
- Long Video Generation: At approximately frame 800, Lyra 2.0 preserves realistic geometric structures and appearances when revisiting regions, unlike baselines that degrade, distort, or drift.VMem shows severe degradation and structural collapse; GEN3C and Yume-1.5 show geometric distortions, while CaM and SPMem drift.
- 3D Scene Generation: Both Ours + DAv3 and Ours Full achieve the best 3D-scene metrics on both datasets, and Ours Full consistently outperforms Ours + DAv3.The results support fine-tuning the reconstruction model to tolerate artifacts in generated videos.
- 3D Scene Generation: Iterative camera-trajectory exploration produces scenes with substantially greater spatial extent and complexity than Lyra and FantasyWorld, whose short videos limit scene scale.The pipeline reconstructs realistic 3D scenes and supports progressively expanding environments.
- Ablation Study: Global point-cloud conditioning reduces Camera Controllability from 63.87 to 49.86 and Style Consistency from 85.07 to 82.42.The ablation attributes this degradation to accumulated depth errors corrupting the conditioning signal over long horizons.
- Ablation Study: Removing self-augmentation raises per-frame Subjective Quality from 43.35 to 47.88 but lowers Style Consistency to 77.98 and Camera Controllability to 53.92.Without imperfect-conditioning exposure during training, errors compound across autoregressive segments.
- Applications: The framework supports interactive scene generation, in-the-wild reconstruction across indoor and outdoor environments, and robot simulation in NVIDIA Isaac Sim.Generated 3D Gaussian Splatting representations and meshes can be exported for physically grounded navigation and interaction.
6. Discussion
Lyra 2.0 enables large-scale, explorable 3D environments and supports deployment for interactive exploration, virtual reality, and simulation. The framework remains limited to static environments and can reproduce photometric inconsistencies from its training data.
- Lyra 2.0 enables creation of large-scale, explorable 3D environments.
- Generated scenes can be deployed for interactive exploration, virtual reality experiences, and simulation.
- The framework focuses on static environments and does not explicitly model dynamic scenes.
- Exposure variations in DL3DV may be reproduced during generation, causing photometric inconsistencies and artifacts in feed-forward 3DGS reconstruction.
A.1. Model Architecture
The model injects camera and spatial information through depth-warped conditioning, Plücker rays, and canonical coordinates. It retrieves spatial memory frames to balance coverage of revisited regions against inference efficiency.
- Base Model: The backbone is Wan 2.1-14B DiT, with video encoding at 8× spatial and 4× temporal downsampling and 832×480 resolution.
- Camera Conditioning Modules: Depth-warped conditioning forwards the latest frame to target viewpoints, while Plücker ray coordinates encode camera geometry for history and generation tokens.
- Canonical Coordinate Injection: Canonical coordinate maps are injected into self-attention queries and keys at every transformer block, excluding values.
- Number of Spatial Slots: N_s=5 provides a good trade-off between previously visited-region coverage and inference efficiency.
A.2. Training
Training uses retrieved spatial memory and self-augmentation alongside AdamW optimization, rectified flow matching, and mixed-precision distributed training.
- Spatial Memory: The system retrieves five spatial memory frames per autoregressive step from a downsampled 3D cache.
- Self-Augmentation: Self-augmentation uses an augmentation probability of p_aug = 0.7.
- Optimization: Training uses AdamW with learning rate 3×10^-5, weight decay 0.1, batch size 64 across 64 NVIDIA GB200 GPUs, and 7,000 iterations.
- Flow Matching: The model uses rectified flow matching with logit-normal timestep sampling and uniform time weighting during training.
A.3. Inference
Inference applies classifier-free guidance and uses a multistep flow scheduler. Runtime is substantially reduced with Ours DMD, while spatial memory retrieval remains under one second per step.
- Classifier-Free Guidance: Classifier-free guidance uses a text-prompt scale of 5.0.
- Runtime: 194 seconds per 80-frame autoregressive step falls to approximately 15 seconds with Ours DMD.
- Runtime: Spatial memory retrieval takes less than 1 second per step in both inference configurations.
A.4. 3D Reconstruction
The reconstruction pipeline fine-tunes a feed-forward model on autoregressively generated videos and extracts scene meshes from Gaussian-based depth and signed-distance estimates.
- 3,000 one-minute videos are generated from DL3DV images and camera trajectories to construct the fine-tuning dataset.DAv3 is fine-tuned on these scenes for 10,000 iterations with learning rate 5×10−5 and batch size 8.
- Mesh extraction uses a hierarchical sparse grid, computes a signed distance field from Gaussian reconstruction depth, and applies marching cubes to produce one merged surface mesh.
A.5. Related Work
Related work spans 3D generation, multi-view coherence, iterative environment expansion, and feed-forward 3D inference, while prior feed-forward methods are generally limited to static scenes.
- The paper provides additional related-work discussion beyond Section 2.
- 3D generation: 3D generation evolved from category-specific GAN-based object synthesis toward CLIP-supervised, text-conditioned, and diffusion-based pipelines.
- 3D generation: Multi-view coherence is addressed through multi-view generation or supervision, iterative inpainting for environment expansion, and lifting 2D observations into NeRF, Gaussian, or mesh representations.
- Feed-forward 3D models: Feed-forward 3D models infer 3D structure directly in a single pass, but are generally restricted to static scene representations.