PaperlayerView exact revision on arXiv

Source-linked AI summary

Lyra: Generative 3D Scene Reconstruction via Video Diffusion Model Self-Distillation

Sherwin Bahmani, Tianchang Shen, Jiawei Ren, Jiahui Huang, Yifeng Jiang, Haithem Turki, Andrea Tagliasacchi, David B. Lindell, Zan Gojcic, Sanja Fidler, Huan Ling, Jun Gao, Xuanchi Ren

arXiv:2509.19296v1cs.CVcs.GR

TL;DR

Existing 3D reconstruction methods depend on captured multi-view data, while video diffusion models lack explicit 3D representations for interactive simulation. Lyra distills video diffusion knowledge into a 3D Gaussian Splatting decoder, generating static scenes from text or images and dynamic scenes from monocular video; it reports state-of-the-art results across single-image 3D and single-video 4D generation.

  • Problem

    3D reconstruction is limited by the need for captured multi-view data, accurate camera poses, and large-scale 3D training data, especially for dynamic scenes.

  • Method

    Lyra self-distills a camera-controlled video diffusion model into a latent-space 3DGS decoder that generates explicit static or dynamic scenes from monocular inputs.

  • Results

    Lyra achieves state-of-the-art results in single-image 3D scene generation and single-video 4D scene generation, outperforming prior methods across all reported benchmarks and metrics.

  • Takeaways & Limitations

    The framework enables scalable, real-time explicit 3D scene generation without captured multi-view data or post-processing optimization.

  • Takeaways & Limitations

    Pixel-space 3DGS decoding causes out-of-memory failures, motivating Lyra’s latent-space design for processing many input frames.

Abstract

from arXiv · show

The ability to generate virtual environments is crucial for applications ranging from gaming to physical AI domains such as robotics, autonomous driving, and industrial AI. Current learning-based 3D reconstruction methods rely on the availability of captured real-world multi-view data, which is not always readily available. Recent advancements in video diffusion models have shown remarkable imagination capabilities, yet their 2D nature limits the applications to simulation where a robot needs to navigate and interact with the environment. In this paper, we propose a self-distillation framework that aims to distill the implicit 3D knowledge in the video diffusion models into an explicit 3D Gaussian Splatting (3DGS) representation, eliminating the need for multi-view training data. Specifically, we augment the typical RGB decoder with a 3DGS decoder, which is supervised by the output of the RGB decoder. In this approach, the 3DGS decoder can be purely trained with synthetic data generated by video diffusion models. At inference time, our model can synthesize 3D scenes from either a text prompt or a single image for real-time rendering. Our framework further extends to dynamic 3D scene generation from a monocular input video. Experimental results show that our framework achieves state-of-the-art performance in static and dynamic 3D scene generation.

1 INTRODUCTION

Lyra introduces a self-distillation framework that converts video diffusion models’ implicit 3D knowledge into explicit 3D Gaussian Splatting representations without captured multi-view data. It supports scalable static scene generation and extends to dynamic 4D reconstruction from monocular video, achieving state-of-the-art results in both settings.

  • Motivation: The method addresses reconstruction limits caused by accurate pose requirements, high-quality imagery, synchronized multi-camera setups, and restriction to observed content.These limitations particularly affect scalability and dynamic-scene reconstruction.
  • Contribution: Lyra trains a 3DGS student decoder under supervision from a pre-trained camera-controlled video diffusion model, eliminating captured real-world multi-view data.The RGB decoder acts as teacher while the 3DGS decoder operates directly in the video model’s latent space.
  • Contribution: The framework generates explicit 3D environments from video diffusion models across diverse scenarios, enabling real-time rendering, physical interaction, and consistent multi-view synthesis.Latent-space processing supports multiple views without the heavy memory overhead of pixel-space feed-forward reconstruction methods.
  • Contribution: Lyra extends self-distillation to dynamic 4D generation from monocular video by learning time-conditioned 3DGS representations for novel-view synthesis.The video model provides space–time supervision as teacher while the student produces dynamic scene representations.
  • Contribution: The framework generalizes across diverse scenes and achieves state-of-the-art results in single-image 3D scene generation and single-video 4D scene generation.These claims summarize the reported overall performance of the proposed model.

2 RELATED WORK

Prior work advances multi-view generation, camera-controlled video synthesis, and feed-forward 3D reconstruction, while recent methods increasingly target scene-scale generation. However, scene-level reconstruction remains constrained by training-distribution scenes, motivating approaches that generate 3D scenes from learned generative models.

  • Multi-view image generation: Multi-view generation progressed from object-centric scenes without backgrounds to scene-scale models, including methods that generate views from real or synthesized images.
  • Camera-conditioned video generation: Camera-conditioned video generation evolved from pose-conditioned video models to pixel-wise camera conditioning with Plücker coordinates and other camera-training strategies.
  • Feed-forward 3D models: Feed-forward 3D models predict NeRFs or 3DGS from text or images, with later work extending from object-centric reconstruction toward scene-level reconstruction.
  • Feed-forward 3D models: Most scene-level reconstruction methods remain limited by training-distribution scenes and generalize poorly to generated scenes, while newer methods generate scenes using pointmaps or camera-controlled video models.

3 SELF-DISTILLATION USING VIDEO DIFFUSION MODELS

Lyra distills a camera-controlled video diffusion model’s implicit 3D knowledge into an explicit 3DGS decoder through teacher–student supervision in latent space. GEN3C provides spatiotemporal caches and rendered guidance, while multi-trajectory synthetic supervision trains coherent Gaussians without captured multi-view data.

  • GEN3C backbone: GEN3C constructs an L × V spatiotemporal 3D cache of colored point clouds by unprojecting estimated depth from images or videos across times and camera views.The cache is designed to improve video consistency and camera-control precision.
  • GEN3C backbone: Rendering cached point clouds from camera poses produces RGB images and disocclusion masks that provide structured guidance for video generation.The masks identify regions that the diffusion model should fill in.
  • Teacher–student self-distillation: The 3DGS decoder is trained entirely with synthetic supervision generated by the camera-controlled video diffusion model.Training uses diverse text prompts, image-diffusion-generated images, and GEN3C-expanded multi-view sequences.
  • Teacher–student self-distillation: The 3DGS decoder renders explicit Gaussians whose views are supervised to match RGB frames decoded from the frozen video diffusion model’s latent output.The RGB decoder acts as teacher, while the 3DGS decoder acts as student in the same latent-space pipeline.
  • Multi-trajectory supervision: Six camera trajectories are sampled per input image, and the 3DGS decoder fuses their diffusion latents into coherent Gaussians while filling disoccluded regions.Operating in latent space enables efficient multi-trajectory aggregation, while explicit Gaussians support geometric consistency, downstream simulation, and real-time rendering.

4 FEED-FORWARD RECONSTRUCTION FROM MULTI-VIEW VIDEO LATENTS

The feed-forward 3DGS decoder converts camera-conditioned multi-view video latents into an explicit, arbitrarily viewable 3D representation while avoiding pixel-space scaling bottlenecks. It predicts Gaussian features from latent visual and Plücker inputs and is trained with image, depth, and opacity-based supervision.

  • Architecture: The 3DGS representation enables rendering from arbitrary viewpoints while providing explicit geometry and fast rendering.The decoder is designed to transform synthesized multi-view latents into an explicit 3D representation, motivating the choice of 3D Gaussian Splatting.
  • Scaling Latent-Based 3D Reconstruction: Operating in compressed video latent space avoids the pixel-space attention costs that grow with the number of input pixels and views.The latent tensor is denoted Z ∈ R^V×L′×C×h×w.
  • Architecture: The decoder maps multi-view latents and encoded Plücker embeddings to per-pixel 3D Gaussian features without requiring an additional visual encoder.Its static formulation is G = Ds(Z, E), with G ∈ R^V×L×H×W×14.
  • Training Objectives: Training combines MSE, LPIPS, scale-invariant depth, and opacity regularization losses to improve appearance, geometry, and Gaussian sparsity.The loss weights are λmse = 1.0, λlpips = 0.5, λdepth = 0.05, and λopacity = 0.1; the lowest 80% opacity Gaussians are removed.

5 EXTENSION TO DYNAMIC 3D SCENES

The framework extends to dynamic 3D scenes by conditioning on a monocular video with camera poses and generating time-dependent 3D Gaussians. Motion-reversed supervision balances temporal coverage and prevents low-opacity artifacts caused by restricted timestamp correspondence.

  • Self-distillation for dynamic scenes: The dynamic setup takes a length-L video with camera poses and generates multi-view video latents representing the same underlying motion.
  • Dynamic 3DGS decoder architecture: The dynamic 3DGS decoder augments its input with encoded source and target time embeddings to decode time-dependent Gaussians across video frames.A target timestep is randomly sampled during training, and the corresponding Gaussians are supervised across trajectories.
  • Dynamic data augmentation: Reversing the input video creates paired near and far supervision views for each timestep, balancing temporal coverage and preventing low-opacity artifacts in early-timestep Gaussians.The augmentation addresses artifacts arising when extreme-pose early timesteps lack supervision coverage.
  • Dynamic data augmentation: Because only frames at the corresponding timestamp validly supervise dynamic 3DGS, the model can ignore information from other timesteps and learn a trivial solution.

6 EXPERIMENTS

Experiments show that Lyra achieves state-of-the-art single-image-to-3D reconstruction across multiple benchmarks and metrics, while producing consistent novel views for 3D/4D scenes. Ablations indicate that self-distillation, depth and LPIPS losses, learned multi-view fusion, opacity pruning, and joint Transformer/Mamba-2 design improve quality, robustness, compactness, or efficiency.

  • Quantitative results: Lyra outperforms previous methods on all evaluated benchmarks and metrics for single-image-to-3D reconstruction.Evaluation uses RealEstate10K, DL3DV, and Tanks-and-Temples with PSNR, SSIM, and LPIPS.
  • Qualitative results: Lyra generates high-quality novel views for unseen 3D/4D regions while maintaining consistency with the input image or video.Novel-view renderings are shown in Figures 1 and 6, with additional qualitative comparisons on the Lyra dataset in Appendix C.
  • Ablation study: Self-distillation alone matches or exceeds joint training with real multi-view data, supporting its diversity and consistency for learning reconstruction.The real-data-only baseline uses RealEstate10K and DL3DV, which have limited out-of-distribution generalizability.
  • Ablation study: Depth and LPIPS losses improve geometry, image-based metrics, robustness to input inconsistencies, and preservation of high-frequency details, while learned fusion outperforms independent trajectory fusion.Depth loss prevents flat geometry, and fusion is learned through reconstruction blocks whose tokens attend to one another.
  • Ablation study: Opacity pruning compacts the Gaussian representation, slightly improves visual quality, and reduces rendering time from 30ms to 18ms, a 1.67× speedup.The comparison uses H = 704 and W = 1280.
  • Ablation study: Joint Transformer/Mamba-2 blocks provide slightly higher quality and reduce one-pass inference time from 20922ms to 3213ms, a 6.5× speedup.The timing comparison uses V = 6, L = 121, H = 704, and W = 1280; pixel-space 3DGS instead causes out-of-memory at up to 726 input frames.

7 CONCLUSION

Lyra is a 3D and 4D generation framework that uses camera-controlled video diffusion teachers to self-distill a student 3DGS decoder from single-image or video inputs without multi-view datasets. The authors note that modeling motion and tracking within the reconstruction network could improve visual motion quality.

  • Conclusion: Lyra generates 3D and 4D reconstructions from a single image or video, replacing multi-view data collection with camera-controlled video diffusion teachers and self-distillation.Its student 3DGS decoder operates in the video model’s latent space and directly reconstructs 3D Gaussians without post-processing or optimization.
  • Conclusion: Modeling motion and tracking within the reconstruction network is identified as a potential way to improve visual motion quality.This direction is noted in relation to concurrent work by Lin et al. (2025).

8 ETHICS STATEMENT · A ADDITIONAL DETAILS OF VIDEO DIFFUSION MODEL · B MODEL DETAILS

The paper discusses responsible use and safeguards while detailing conservative rendering, progressive training, Gaussian subsampling, architecture, camera/time encoding, and dynamic augmentation for scalable 3D/4D generation.

  • 8 ETHICS STATEMENT: The method supports realistic embodied-agent simulation and scalable, customizable scene generation without multi-view capture or expensive per-scene optimization, while posing misuse risks.The authors recommend provenance tracking, dataset documentation, and good evaluation practices to mitigate deceptive or wrongful use.
  • A ADDITIONAL DETAILS OF VIDEO DIFFUSION MODEL: Conservative mask refinement assumes occluded regions are occupied, correcting background leakage and enabling more reliable foreground completion under large viewpoint changes.The refined masks identify potentially visible foreground regions missed by standard point-based rendering.
  • B MODEL DETAILS: The model is trained progressively because direct training on six 121-frame trajectories at 704 × 1280 resolution is expensive, requiring six days on eight NVIDIA A100 80 GB GPUs.The final static pretrained model initializes the dynamic model, and gsplat provides the 3DGS implementation.
  • B MODEL DETAILS: Gaussian subsampling reduces 654,213,120 per-pixel Gaussians to 10,222,080 by using one Gaussian per 8 × 8 neighborhood, then opacity pruning yields 2,044,416.The spatial subsampling reduces the Gaussian count by a factor of 64 and makes outputs more compact.
  • B MODEL DETAILS: The reconstruction network uses two joint blocks totaling 16 layers: each block combines one Transformer layer with seven Mamba-2 layers and 512 hidden dimensions.This network is smaller than Long-LRM and Wonderland, while Mamba-2 offers faster training and inference than pure Transformers under the same budget.
  • B MODEL DETAILS: Plücker embeddings encode camera rays by separately processing directions and ray-direction cross products, while sinusoidal time embeddings independently encode source and target times.Both encodings are projected into the video latent space through the pretrained RGB encoder.
  • B MODEL DETAILS: Dynamic augmentation pairs supervision views from identical motion states but extreme viewpoints, using motion-reversed videos to provide full spatial coverage at each timestep.The procedure helps prevent low-opacity artifacts in early timesteps and includes zoom-in and zoom-out trajectories.
  • B MODEL DETAILS: Compared with BTimer (GEN3C), the method produces generated scenes with significantly fewer artifacts and higher fidelity across five visualized views.The comparison is presented for image-to-3DGS generation.

C ADDITIONAL EVALUATION · C.1 ADDITIONAL BASELINE DETAILS

The additional evaluation compares Lyra with BTimer for joint 3D and 4D reconstruction, adapting BTimer to use Lyra’s GEN3C video-diffusion backbone and camera trajectories.

  • C ADDITIONAL EVALUATION: Lyra is compared against BTimer, a recent method for joint 3D and 4D reconstruction from posed images or videos.
  • C.1 ADDITIONAL BASELINE DETAILS: The BTimer experiments use the same scenes as Lyra after coordination with BTimer’s authors.
  • C.1 ADDITIONAL BASELINE DETAILS: Because BTimer is purely regression-based, the evaluation integrates it with Lyra’s GEN3C video diffusion backbone.
  • C.1 ADDITIONAL BASELINE DETAILS: The adapted BTimer receives RGB-decoded videos generated by GEN3C as input.
  • C.1 ADDITIONAL BASELINE DETAILS: The comparison uses the same camera trajectories as Lyra’s model.
  • C.1 ADDITIONAL BASELINE DETAILS: This setup enables a regression-based BTimer comparison under Lyra’s video-diffusion-generated inputs and scene trajectories.

C.2 ADDITIONAL STATIC 3D EVALUATION

The method produces higher-quality image-to-3DGS generations with fewer artifacts than BTimer (GEN3C).

  • C.2 ADDITIONAL STATIC 3D EVALUATION: Qualitative comparisons show higher quality and fewer artifacts than BTimer (GEN3C) for image-to-3DGS generation.The comparison appears in Fig. 10, with additional qualitative results in Tab. 4a.

C.3 DYNAMIC 3D EVALUATION

On 100 out-of-distribution videos from the dynamic Lyra dataset, the method significantly outperforms BTimer (GEN3C) under a resolution-matched evaluation setup. The quantitative results are reported in Table 4b within the broader static and dynamic Lyra evaluations.

  • C.3 DYNAMIC 3D EVALUATION: The method significantly outperforms BTimer (GEN3C) on dynamic 3D evaluation using 100 out-of-distribution videos from the dynamic Lyra dataset.Results are reported in Tab. 4b.
  • C.3 DYNAMIC 3D EVALUATION: For fair comparison, the videos are cropped to 512 × 512 because BTimer was mainly trained at that resolution.
  • C.3 DYNAMIC 3D EVALUATION: Table 4 presents quantitative results on the static and dynamic Lyra datasets, with the dynamic comparison shown in Tab. 4b.

C.4 DEPTH VISUALIZATION · D ROBOT SIMULATION · E RELATED WORK

The framework improves geometric depth quality, supports robot simulation through an export pipeline, and is positioned among prior work in 3D, 4D, feed-forward, and camera-conditioned generation.

  • C.4 DEPTH VISUALIZATION: Depth supervision prevents flat geometries without sacrificing visual quality in generated 3DGS renderings.The paper visualizes rendered scenes and corresponding depth maps in Figure 11.
  • D ROBOT SIMULATION: The method enables robot simulation by exporting generated 3D Gaussians from .ply to .usdz for import into NVIDIA Isaac.The pipeline first generates text-conditioned 3D scenes, then converts the exported files using 3DGUT.
  • E RELATED WORK: Prior 3D-generation work spans single-object GANs, CLIP-supervised text-driven assets, multi-view consistency, and lifting images into NeRF, Gaussian, or mesh representations.These approaches progressively address flexibility, structural coherence, and 2D-to-3D reconstruction.
  • E RELATED WORK: Feed-forward 3D models directly predict 3D content from image or text inputs, complementing optimization-based reconstruction approaches.The related work identifies fast feed-forward prediction as a distinct research direction.
  • E RELATED WORK: 4D-generation methods synthesize dynamic 3D scenes from text, images, or videos, while earlier 3D-GAN extensions face limited datasets, object-centricity, and lower photorealism.The paper contrasts these limitations with the fidelity of recent video-generation models.
  • E RELATED WORK: Camera-conditioned video models add camera control to video diffusion through extrinsic conditioning, U-Net or transformer architectures, synthetic data, and space-time modeling.Related methods include MotionCtrl, SynCamMaster, ReCamMaster, and 4DiM.
Loading 2509.19296v1…