Source-linked AI summary
LucidDreamer: Domain-free Generation of 3D Gaussian Splatting Scenes
Jaeyoung Chung, Suyoung Lee, Hyeongjin Nam, Jaerin Lee, Kyoung Mu Lee
TL;DR
Existing 3D scene generation models are restricted to specific domains by scan-based training data. LucidDreamer alternates Dreaming and Alignment to generate and integrate multi-view-consistent geometry, producing diverse, high-quality scenes across domains and input conditions.
Problem
3D scene generation models trained on 3D scans produce limited diversity and quality and restrict target scenes to specific domains.
Method
LucidDreamer alternates Dreaming, which generates and lifts geometrically consistent images into 3D, with Alignment, which integrates new points before Gaussian-splat optimization.
Results
LucidDreamer consistently generates high-quality, diverse 3D scenes across realistic, anime, lego, indoor, and outdoor domains.
Takeaways & Limitations
The pipeline supports text, RGB, and RGBD inputs, combines multiple conditions, changes conditions during generation, and preserves input-image style.
Abstract
from arXiv · showhide
With the widespread usage of VR devices and contents, demands for 3D scene generation techniques become more popular. Existing 3D scene generation models, however, limit the target scene to specific domain, primarily due to their training strategies using 3D scan dataset that is far from the real-world. To address such limitation, we propose LucidDreamer, a domain-free scene generation pipeline by fully leveraging the power of existing large-scale diffusion-based generative model. Our LucidDreamer has two alternate steps: Dreaming and Alignment. First, to generate multi-view consistent images from inputs, we set the point cloud as a geometrical guideline for each image generation. Specifically, we project a portion of point cloud to the desired view and provide the projection as a guidance for inpainting using the generative model. The inpainted images are lifted to 3D space with estimated depth maps, composing a new points. Second, to aggregate the new points into the 3D scene, we propose an aligning algorithm which harmoniously integrates the portions of newly generated 3D scenes. The finally obtained 3D scene serves as initial points for optimizing Gaussian splats. LucidDreamer produces Gaussian splats that are highly-detailed compared to the previous 3D scene generation methods, with no constraint on domain of the target scene. Project page: https://luciddreamer-cvlab.github.io/
1. Introduction
LucidDreamer addresses domain restrictions in 3D scene generation by combining Stable Diffusion with 3D Gaussian splatting. Its alternating Dreaming and Alignment processes support diverse inputs, domains, and changing conditions while producing realistic scenes.
- Existing 3D scene generators suffer from limited diversity and quality because their training data relies on 3D scans.
- LucidDreamer initializes a point cloud from an image and depth map, then alternates Dreaming and Alignment to build a unified 3D scene.
- LucidDreamer generates more realistic scenes across diverse domains, including realistic, anime, lego, indoor, and outdoor settings.
- Dreaming uses the point cloud as a geometric guideline for generating multi-view images, while Alignment integrates those images into one 3D scene.
- The pipeline supports text, RGB, and RGBD inputs, including simultaneous conditions and changing inputs during generation.
2. Related Work
Prior 3D scene representations trade detail, controllability, speed, or storage efficiency, while 3D generation methods face quality and scalability limitations. Gaussian splatting is presented as an explicit representation that improves training and rendering efficiency.
- 3D Scene Representation: Explicit representations such as point clouds, meshes, and voxels provide direct control and fast rasterized rendering but require many elements for detailed scenes.
- 3D Scene Representation: Implicit neural representations can optimize photorealistic scenes from multiple images, but storing scenes in networks makes them difficult to handle and slow.
- 3D Scene Representation: Local explicit structures, including sparse voxels, featured point clouds, tensors, triplanes, polygons, and Gaussian splats, improve training and rendering speed.
- 3D Scene Generation: Earlier 3D generation approaches produced multiview images, voxels, or point clouds but were limited by GAN instability and 3D memory requirements.
3. Method
LucidDreamer constructs a large point cloud by repeatedly navigating the camera, generating geometrically guided views, lifting new pixels into 3D, and aligning them with existing points before Gaussian-splat optimization.
- Point cloud construction: LucidDreamer expands an initial point cloud into a large scene using Stable Diffusion inpainting, monocular depth estimation, and explicit 3D representation.The approach supports text, RGB, and RGBD inputs and preserves the input style across generated scenes.
- Initialization: The initial point cloud P0 is formed by lifting pixels from the input RGBD image [I0, D0] into 3D space.For text input, Stable Diffusion first generates the RGB image and a monocular depth model estimates its depth map.
- Navigation: At each navigation step, the current point cloud is transformed and projected into the camera plane using the camera intrinsics and pose.The camera moves from Pi−1 to Pi, exposing regions that cannot be filled by existing points.
- Dreaming: Dreaming inpaints unfilled regions of the projected image with Stable Diffusion and estimates a corresponding monocular depth map for lifting new pixels into 3D.The mask Mi identifies pixels already filled by existing points, while inpainted pixels are lifted to avoid overlaps.
- Dreaming: The relative depth map is rescaled by estimating di from overlapping point pairs so newly lifted points connect to the existing point cloud.Mi = 1 marks overlapping regions used to estimate the depth scale coefficient.
- Alignment: Alignment moves newly generated points toward corresponding points in the previous cloud, while ray constraints and interpolation preserve image content and overall shape.The aligned points are merged with the original cloud, and repeating the process produces PN whose reprojections are high-quality and multi-view consistent.
- Gaussian splats optimization: The completed point cloud PN initializes Gaussian splatting, whose points are optimized using projected images while losses are computed only on valid image regions.Gaussian representations naturally fill missing pixels during rasterization, and PN initialization encourages the model to focus on representation details.
4. Experiments
Experiments evaluate LucidDreamer across input formats, datasets, styles, and design choices. Results emphasize domain generalization, input conditioning, reconstruction quality, and artifact reduction.
- Experiment settings: LucidDreamer supports text, RGB, and RGBD inputs, using pretrained diffusion and captioning models to construct scenes without model-specific training datasets.For RGB(D) inputs, LAVIS generates captions for diffusion conditioning; Stable Diffusion performs masked inpainting.
- Applicability to various inputs: Different text prompts applied to the same initial image produce distinct intermediate generations and final 3D scenes.The experiment isolates text conditioning while keeping the initial image fixed.
- Applicability to various inputs: LucidDreamer generates high-quality scenes across realistic, anime, lego, indoor, and outdoor styles, while preserving conditioned RGB content under different text prompts.The system also permits changing text conditions during scene construction and supports multiple camera paths.
- Comparison with RGBD2: CLIP-Score and CLIP-IQA both favor LucidDreamer over RGBD2 across the reported quantitative comparisons.The table caption states that LucidDreamer performs better on all metrics.
- Ablations on design choices: LucidDreamer’s point-cloud initialization yields better reconstruction metrics than COLMAP initialization and reaches high-quality scenes in fewer Gaussian-splatting iterations.Valid masks during Gaussian-splatting training reduce boundary artifacts and remove black splinters from backgrounds.
5. Conclusion
LucidDreamer is a domain-free pipeline that alternates Dreaming and Alignment to build multiview-consistent scenes before converting them into Gaussian splats. Experiments report consistently high-quality and diverse scenes across varied situations.
- Conclusion: LucidDreamer alternates Dreaming and Alignment to generate multiview-consistent images and integrate them into an existing point cloud.The completed point cloud is converted into 3D Gaussian splats to enhance scene quality.
- Conclusion: Extensive experiments show that LucidDreamer consistently generates high-quality and diverse 3D scenes in various situations.