Source-linked AI summary
GAUDI: A Neural Architect for Immersive 3D Scene Generation
Miguel Angel Bautista, Pengsheng Guo, Samira Abnar, Walter Talbott, Alexander Toshev, Zhuoyuan Chen, Laurent Dinh, Shuangfei Zhai, Hanlin Goh, Daniel Ulbricht, Afshin Dehghan, Josh Susskind
TL;DR
GAUDI addresses the challenge of generatively modeling complex 3D scenes without shared camera-pose distributions or canonical orientations. It disentangles radiance fields and camera paths in optimized latents, learns their distribution with a diffusion prior, and achieves state-of-the-art performance across multiple datasets while supporting image- and text-conditioned generation.
Problem
Generative modeling of unconstrained 3D scenes is difficult because valid camera-pose distributions vary by scene rather than being shared across samples.
Method
GAUDI first optimizes disentangled latent representations of radiance fields and camera poses, then learns their distribution with a denoising diffusion model.
Results
GAUDI obtains state-of-the-art generation performance across multiple datasets and metrics in unconditional generation.
Takeaways & Limitations
The model supports unconditional generation and conditional 3D-scene generation from image observations or text descriptions.
Takeaways & Limitations
Inference is not real-time because DDPM sampling and radiance-field rendering are inefficient.
Abstract
from arXiv · showhide
We introduce GAUDI, a generative model capable of capturing the distribution of complex and realistic 3D scenes that can be rendered immersively from a moving camera. We tackle this challenging problem with a scalable yet powerful approach, where we first optimize a latent representation that disentangles radiance fields and camera poses. This latent representation is then used to learn a generative model that enables both unconditional and conditional generation of 3D scenes. Our model generalizes previous works that focus on single objects by removing the assumption that the camera pose distribution can be shared across samples. We show that GAUDI obtains state-of-the-art performance in the unconditional generative setting across multiple datasets and allows for conditional generation of 3D scenes given conditioning variables like sparse image observations or text that describes the scene.
1 Introduction
GAUDI addresses scalable generative modeling of complex 3D scenes by disentangling radiance fields from camera poses, then supports unconditional and conditional scene generation.
- Motivation: Generative 3D models aim to capture distributions of scenes and render views from sampled scenes, including completions conditioned on images or text.The motivation emphasizes immersive scene generation and conditional inference.
- Problem: Radiance-field GANs face mode collapse and difficulty training without a canonical coordinate system, while scene camera-pose distributions vary by scene.Valid scene poses depend on walls and objects and may span SE(3).
- Approach: GAUDI optimizes a latent representation for each trajectory that separately encodes the radiance field and camera path.The latent representations are learned through a reconstruction-based optimization process scalable to thousands of trajectories.
- Contributions: GAUDI scales 3D scene generation to thousands of indoor scenes and hundreds of thousands of images without mode collapse or canonical-orientation issues.This contribution targets the training challenges posed by large, unconstrained scene collections.
- Contributions: GAUDI introduces a denoising optimization objective for jointly modeling radiance fields and camera poses in disentangled latent representations.The objective is a central methodological contribution.
- Contributions: GAUDI achieves state-of-the-art generation performance across multiple datasets and supports unconditional generation plus image- or text-conditioned generation.The paper presents both the benchmark result and the supported generative settings.
2 Related Work
Prior work combines radiance fields with generative modeling, but extending object-focused methods to unconstrained 3D scenes remains unresolved because scenes lack shared canonical orientations and camera distributions.
- Generative modeling: DDPMs learn to reverse a diffusion Markov chain and have become strong alternatives to adversarial and autoregressive image-generation models.The related-work discussion places diffusion models in the broader generative-modeling landscape.
- Radiance fields: NeRF represents scenes as radiance fields that map 3D points and camera information to density and color, with views produced by volumetric rendering.Radiance fields provide the 3D representation used by many subsequent generative approaches.
- Generative radiance fields: Most radiance-field generative models target single objects with known canonical orientations and shared camera-pose distributions.These assumptions simplify the modeling problem compared with unconstrained scenes.
- Open problem: Extending radiance-field generation from single objects to unconstrained 3D scenes remains unsolved, while GSN retains GAN pathologies such as mode collapse.The lack of canonical orientation exacerbates these training problems as datasets grow.
- View-conditioned prediction: Single- or multi-view prediction methods use deterministic reconstruction objectives even though a source view can admit multiple equally likely scene completions.This motivates probabilistic generative treatment of scene completion.
3 GAUDI
GAUDI learns disentangled scene and camera-pose latents from trajectories, decodes them through radiance-field and pose networks, and models their distribution with a DDPM for unconditional or conditional generation.
- Data representation: A trajectory is a variable-length sequence of corresponding RGB images, depth images, and 6DOF camera poses from a 3D scene.The model learns from an empirical distribution of such trajectories.
- Two-stage model: GAUDI decomposes learning into latent optimization for each trajectory followed by distribution learning over separate scene and pose latent vectors.The latent is z = [zscene, zpose].
- Decoder architecture: The decoder uses separate networks for camera poses, scene geometry, and radiance-field rendering.The architecture is designed to disentangle camera poses from scene appearance and geometry.
- Camera pose decoder: The camera-pose decoder maps normalized temporal positions and zpose to valid SE(3) poses using orientation and translation outputs.It represents orientation with a normalized quaternion and translation with a 3D vector.
- Scene and radiance-field decoders: The scene decoder maps zscene to an axis-aligned tri-plane representation, whose sampled features condition the radiance-field decoder.The radiance-field network predicts density and appearance for 3D points and renders image-level outputs volumetrically.
- Latent optimization: A denoising reconstruction objective jointly optimizes decoder parameters and per-example latents while perturbing latents with scaled additive noise.The objective combines scene reconstruction and camera-pose reconstruction losses.
- Prior learning: A DDPM learns the distribution of optimized latents, enabling scalable unconditional and conditional 3D-scene generation.At inference, Gaussian noise is iteratively denoised before decoding a radiance field and camera path; conditioning augments the denoising model with y.
4 Experiments
GAUDI is evaluated across four datasets spanning synthetic and real indoor scenes, varied complexity, and different trajectory structures. Experiments assess reconstruction, latent-space interpolation, and dataset coverage using qualitative and quantitative analyses.
- 4.1 Data: Experiments use four datasets—Vizdoom, Replica, VLN-CE, and ARKitScenes—that vary in scene count, complexity, and trajectory characteristics.Vizdoom is synthetic and simple; Replica contains 18 realistic scenes; VLN-CE contains 3.6K navigation trajectories with text descriptions; ARKitScenes contains more than 5K scans across about 1.6K indoor spaces.
- 4.2 Reconstruction: The reconstruction evaluation compares random-trajectory reconstructions across Vizdoom, Replica, VLN-CE, and ARKitScenes against ground truth.Table 1 reports reconstruction results, using 10 randomly sampled images per trajectory for VLN-CE and ARKitScenes because of their large image counts.
- 4.3 Interpolation: Latent-space interpolation on VLN-CE produces smooth transitions in scene geometry and texture across different interpolation paths.The visualization interpolates encoded radiance fields, with each row representing a different path between latent scenes.
4.4 Unconditional generative modeling
GAUDI models unconditional distributions of complex 3D scenes and renders diverse samples from their corresponding sampled camera poses. It achieves state-of-the-art performance across four scene datasets and supports conditional generation from text, images, and categorical environment labels.
- Unconditional generative modeling: GAUDI obtains state-of-the-art performance across four scene datasets and both FID and SwAV-FID metrics.The evaluation compares GAUDI with GRAF, π-GAN, and GSN using 5k sampled images per model and dataset.
- Unconditional generative modeling: GAUDI generates diverse and realistic 3D scenes from the learned unconditional distribution, rendered from sampled camera poses.
- Text conditioning: Text conditioning generates multiple plausible scenes and camera paths that match navigation descriptions.The model uses navigation text from VLN-CE and a pretrained RoBERTa-base encoder.
- Image conditioning: Image conditioning samples scenes containing the same or contextually similar view from a stochastic viewpoint.
- Categorical conditioning: Categorical conditioning improves FID over the unconditional model across all evaluated indoor environments.The comparison uses per-environment FID on real images from the corresponding indoor environment.
5 Conclusion
GAUDI captures distributions of complex and realistic 3D scenes through a scalable two-stage approach. It achieves state-of-the-art performance across multiple datasets and supports both unconditional and conditional generation, including text-based scene generation.
- GAUDI first learns latent representations that disentangle radiance fields and camera poses, then models their distribution with a learned prior.
- GAUDI achieves state-of-the-art performance against recent baselines across multiple 3D datasets and metrics.
- GAUDI supports unconditional and conditional 3D scene generation, including generation from text descriptions.
A Limitations, Future Work and Societal Impact
GAUDI’s current limitations include non-real-time inference and unresolved efficiency challenges in diffusion sampling, radiance-field rendering, high-resolution generation, and latent inference. The paper also identifies societal risks involving misuse, privacy, bias, and detachment from reality.
- Limitations and future work: Inference is not real-time because DDPM sampling and radiance-field rendering remain inefficient.The paper suggests faster DDPM methods and more efficient 3D representations as possible directions.
- Limitations and future work: Scaling an encoder-based replacement for latent optimization to trajectories containing thousands of images remains unclear.
- Societal impact: Generative 3D models raise concerns about disingenuous data, training-data leakage, privacy, bias amplification, and detachment from reality.
B Experimental Settings and Details
The experiments use 2048-dimensional scene and pose latents with dataset-specific trajectory, spatial-scale, and rendering configurations. These settings accommodate substantial variation in trajectory lengths and camera motion across the four datasets.
- Model settings: Scene and pose latents each have 2048 dimensions, with reshaping and UNet denoising configurations differing between training stages.
- Dataset-specific settings: Trajectory lengths and rendering scales vary across datasets, so near and far planes are adjusted accordingly.
- Dataset-specific settings: Vizdoom trajectories average 600 steps, with 0.5-game-unit forward moves or 30-degree rotations.
- Dataset-specific settings: Replica trajectories contain 100 steps, with 15-centimeter forward moves or 25-degree rotations.
- Dataset-specific settings: VLN-CE trajectories contain approximately 30–150 steps, with 15-centimeter forward moves or 25-degree rotations.
- Dataset-specific settings: ARKitScenes trajectories average around 1000 steps and allow continuous camera movement in any direction and orientation.
C Decoder Architecture Design and Details
The decoder separates scene content, camera motion, and radiance-field rendering into dedicated modules. Latents are optimized jointly with decoder parameters to reconstruct trajectories.
- The decoder comprises scene, camera pose, and radiance field modules.The scene decoder produces features for the radiance field, while the other modules model camera poses and image reconstruction.
- The scene decoder uses a VQGAN-style convolutional architecture with self-attention and outputs a 64 × 64 × 768 feature map.Its channels are split into three chunks to form the tri-plane representation.
- The camera pose decoder is a residual MLP with four conditional batch-normalization blocks and hidden size 256, conditioned on zpose.It applies positional encoding to normalized temporal inputs.
- Training initializes all latents to zero and jointly optimizes them with model parameters using Adam.Latents use learning rate 0.001, model parameters use 0.0001, and training runs on 8 A100 GPUs for 2–7 days.
D Prior Architecture Design and Details
GAUDI learns a latent prior with DDPMs and supports conditioning from multiple modalities through modality-specific encoders. The prior is trained with separate settings for unconditional and conditional experiments.
- A DDPM with a UNet denoiser learns the latent distribution p(Z).Training uses uniformly sampled diffusion timesteps, while inference follows the DDPM denoising process.
- Conditional inputs are projected into embeddings through modality-specific encoders for text, images, and categorical variables.Text uses pre-trained RoBERTa-base, while images use an ImageNet-pre-trained ResNet-18.
- The DDPM prior uses learning rate 4.0e−06 and 1,000 diffusion steps with a linearly decreasing noise schedule.Training takes 1–3 days for unconditional learning and 3–5 days for conditional experiments on one A100 GPU.
E Ablation Study
The ablations examine latent optimization parameters, correspondence between scene and pose latents, denoising-network design, and conditioning mechanisms. They use reconstruction and prior-learning experiments to assess these choices.
- Optimization parameters: Increasing β makes it harder to find latent codes with high reconstruction accuracy.The study evaluates RGB-image and camera-pose reconstruction on 100 VLN-CE trajectories while varying latent dimensionality and β.
- Prior design choices: The prior ablations test whether corresponding scene and pose latents, denoising-network width, and noise scale affect GAUDI’s design.These factors are evaluated in the design-choice study reported in Table 5.
- Conditioning mechanism: The conditioning ablation compares cross-attention conditioning with FiLM conditioning in the DDPM denoising architecture.FiLM combines pooled conditioning and timestep embeddings to predict affine modulation parameters.
E.1 Additional Visualizations
Additional visualizations examine latent interpolations, unconditional samples, and multimodal conditional samples. They illustrate smooth scene transitions, sampled camera-path renderings, and conditioning by text, images, or categories.
- Latent interpolations: Latent interpolations tend to change smoothly while aligning similar scene content.Each interpolation path connects a random pair of VLN-CE latents and can be rendered while navigating with a moving camera.
- Unconditional samples: Unconditional samples are rendered from their corresponding sampled camera paths across VLN-CE, ARKitScenes, and Replica.The visual results are described as reinforcing the fidelity and variability of GAUDI’s learned distribution.
- Conditional samples: Conditional samples cover text, image, and categorical conditioning modalities.The supplementary videos correspond to the conditional-generation results.