PaperlayerView exact revision on arXiv

Source-linked AI summary

Unconstrained Scene Generation with Locally Conditioned Radiance Fields

Terrance DeVries, Miguel Angel Bautista, Nitish Srivastava, Graham W. Taylor, Joshua M. Susskind

arXiv:2104.00670v1cs.CVcs.LG

TL;DR

Complex indoor scenes require models that can represent large environments and extrapolate beyond sparse observations, while prior generative radiance-field methods were mainly constrained to single objects. GSN decomposes scenes into many locally conditioned radiance fields and learns a scene prior for free-camera generation and completion. Across several scene datasets, it produces higher-quality renderings than existing models, while the authors note that reconstruction metrics do not fully capture stochastic hallucination quality.

  • Problem

    Existing generative radiance-field models are specialized for constrained single-object views and lack sufficient capacity for complex indoor scenes.

  • Method

    GSN decomposes a global scene latent into a grid of locally conditioned radiance fields and renders them from controlled camera poses.

  • Results

    GSN achieves higher-quality scene renderings than GRAF and π-GAN across datasets, improving FID by 10–14 absolute points.

  • Takeaways & Limitations

    The learned prior supports free-camera scene generation, smooth coherent trajectories, and completion from sparse observations.

  • Takeaways & Limitations

    The authors caution that reconstruction metrics do not fully describe hallucination quality because view synthesis is stochastic.

Abstract

from arXiv · show

We tackle the challenge of learning a distribution over complex, realistic, indoor scenes. In this paper, we introduce Generative Scene Networks (GSN), which learns to decompose scenes into a collection of many local radiance fields that can be rendered from a free moving camera. Our model can be used as a prior to generate new scenes, or to complete a scene given only sparse 2D observations. Recent work has shown that generative models of radiance fields can capture properties such as multi-view consistency and view-dependent lighting. However, these models are specialized for constrained viewing of single objects, such as cars or faces. Due to the size and complexity of realistic indoor environments, existing models lack the representational capacity to adequately capture them. Our decomposition scheme scales to larger and more complex scenes while preserving details and diversity, and the learned prior enables high-quality rendering from viewpoints that are significantly different from observed viewpoints. When compared to existing models, GSN produces quantitatively higher-quality scene renderings across several different scene datasets.

1. Introduction

Existing scene representations struggle to extrapolate beyond observed viewpoints because they lack a learned scene prior. GSN addresses this gap with a generative, locally decomposed radiance-field model for unconstrained indoor scenes.

  • Motivation: Models that interpolate observed views generally cannot infer unobserved scene regions because they lack a prior over scenes.This limits performance for disocclusion inpainting and views beyond observed boundaries.
  • Motivation: Conditional auto-encoders tend toward point estimates and produce blurry renderings when extrapolating far from observations.
  • Motivation: A learned scene prior supports unconditional scene generation and conditional completion from sparse observations.The paper identifies free-camera exploration and scene completion as practical uses.
  • Contribution: GSN introduces a generative model for unconstrained scene-level radiance fields and free-moving-camera view synthesis.
  • Contribution: GSN decomposes latent scenes into locally conditioned radiance fields, supports inversion from sparse observations, and learns smooth, coherent trajectories across multiple scenes.

2. Related Work

Prior radiance-field methods either optimize a separate model per scene, target single objects, or rely on stored observations. GSN instead targets freely moving cameras in full scenes using a spatially decomposed representation.

  • Radiance-field models: Scene-specific radiance-field methods can model high-resolution scenes but require optimizing a new model for every scene.That optimization usually takes days on commodity hardware.
  • Generative radiance fields: Generative radiance-field models commonly target single objects viewed from constrained spheres and use one-dimensional latent representations.
  • GSN representation: GSN’s architecture uses a global latent grid to condition local radiance fields for volumetric rendering.
  • Generative radiance fields: A one-dimensional global latent code does not scale efficiently to full, densely populated scenes.
  • Free-camera synthesis: Free-moving-camera synthesis is harder than single-object viewing because models must represent complete scenes containing many objects.
  • Free-camera synthesis: Observation-memory methods can explore scenes but depend on stored observations, limiting their ability to handle unseen viewpoints.GSN adds a learned scene prior for extrapolation beyond the observations.

3. Method

GSN combines a global scene generator with a grid of locally conditioned radiance fields, enabling viewpoint-controlled rendering of large scenes. It uses local coordinates, occupancy-aware camera sampling, and RGB-D discriminator training.

  • Model overview: GSN maps a global latent code and camera parameters to rendered images through a generator composed of global and locally conditioned radiance-field subnetworks.Camera pose and intrinsics explicitly control viewpoint and field of view.
  • Global generator: A global generator converts one scene code into a two-dimensional grid of local latent codes representing a latent floorplan.Each code conditions a radiance field for a local scene region.
  • Radiance field: The locally conditioned field predicts occupancy and appearance for sampled 3D points and renders pixels through volumetric integration.The model evaluates points along rays and upsamples a rendered feature map with a convolutional refinement network.
  • Local coordinates: Local coordinate systems allow latent codes to represent the same scene part independently of its grid position.This design supports spatial sharing across local radiance fields.
  • Camera sampling: GSN samples free camera poses in SE(3), including potentially invalid locations inside solid objects.The model uses occupancy-weighted sampling over empirical candidate poses to reduce invalid samples.
  • Training: The discriminator receives RGB images concatenated with normalized depth maps, while training uses RGB-D samples from recorded camera trajectories.

4. Experiments

GSN is evaluated for scene generation, ablated to test its representation and training choices, and compared on free-moving-camera view synthesis. Across these experiments, locally conditioned radiance fields, long trajectories, sparse depth, and a learned scene prior support performance on complex scenes.

  • Generation Performance: GSN is evaluated on VizDoom, Replica, and AVD using 64 × 64 images, with FID and SwAV-FID comparisons against recent generative radiance-field models.The evaluation uses 5,000 real and 5,000 generated images for each metric.
  • Generation Performance: GSN improves FID by 10-14 absolute points over GRAF and π-GAN across all three datasets.The paper attributes the improvement to the expressiveness of locally conditioned radiance fields rather than the specific layer type.
  • Generation Performance: Latent interpolations align geometry and appearance features, producing smooth transitions between scenes without unrealistic off-manifold samples.Examples include walls, picture frames, and doorways rendered from a fixed camera pose.
  • Ablation: Models trained on short trajectories fail on long displacements, whereas models trained with long trajectories remain stable when evaluated on short trajectories.Figure 7 evaluates FID across trajectory lengths and contrasts 4-step with 40-step training trajectories.
  • Ablation: GSN retains generated image quality when depth is reduced to a single pixel, while removing depth information causes training failure.The authors suggest depth guides early learning before additional depth supervision becomes unnecessary.
  • View Synthesis: GSN outperforms GTM-SM and ISS on nearly all view-synthesis tasks despite not being trained to explicitly map source views to target views.The evaluation separates source-view memorization from target-view hallucination using L1 and SSIM on VizDoom and AVD.

5. Conclusions

GSN decomposes indoor scenes into local radiance fields that support free-camera rendering, scene coherence, and sparse-observation inference. The authors identify future work in improving rendering, scaling training, and downstream applications, while noting evaluation and reproducibility boundaries.

  • Contributions: GSN decomposes scenes into many local radiance fields that can be rendered by a free-moving camera.The decomposition is intended to scale to large scenes while preserving details and distribution coverage.
  • Learned prior: GSN learns a scene prior from multiple scenes, producing smooth, consistent trajectories that maintain scene coherence.
  • Learned prior: The learned prior supports inference from arbitrary cameras given sparse observations.
  • Reproducibility: The authors followed reported data splits and settings for ISS because that method had no public code or data release.
  • Evaluation caveat: View-synthesis metrics proxy reconstruction quality but do not fully capture hallucination quality because the task is stochastic.
  • Future work: Future work includes improving rendering performance, training on large-scale datasets, and exploring downstream tasks such as reinforcement learning, SLAM, and 3D completion.

A. Model Architectures and Training Details

This section summarizes the model architectures, hyperparameters, and training details used for the GSN models.

  • Scope: The appendix documents GSN’s model architectures, hyperparameter settings, and other training details.

A.1. Mapping Network

The mapping network transforms the global latent code into an intermediate nonlinear latent space using normalization and three LeakyReLU-activated linear layers.

  • Mapping network: The mapping network maps the global latent code z to an intermediate nonlinear latent space.
  • Architecture: It consists of a normalization step followed by three linear layers with LeakyReLU activations.

A.2. Global Generator

GSN uses a global generator to produce a spatial grid of local latent codes, which condition radiance-field networks and are rendered into images through volumetric accumulation and refinement.

  • Global generator: The global generator maps one global latent code z to a 2D grid of local latent codes W representing the scene’s spatial layout.
  • Resolution: The global generator output resolution controls the spatial region represented by each local latent code, and is set to 32 × 32 in all experiments.
  • Local generator: The local generator maps coordinates and view direction to appearance and occupancy, volumetrically renders features along rays, and refines them into RGB images.
  • Radiance field: Each locally conditioned radiance-field network uses modulated linear layers conditioned on its local latent code wij.
  • Rendering: Volumetric rendering uses softplus-thresholded occupancy and 64 samples per ray, with feature-map resolution depending on the target image resolution.
  • Refinement: Refinement blocks upsample rendered feature maps to the target resolution, while directly sampling higher-resolution feature maps improves quality at higher computational cost.

A.4. Discriminator

The discriminator follows a StyleGAN2-based design, while the decoder upsamples low-resolution discriminator features to recover the original image resolution. Training also samples realistic camera poses and uses an egocentric coordinate system centered on the latent grid.

  • Discriminator: The discriminator uses StyleGAN2 components, including residual blocks and a minibatch standard deviation layer.Depth inputs are normalized to [0, 1].
  • Decoder: The decoder takes 4 × 4 discriminator feature maps and progressively upsamples them with stride-2 transposed convolutions and bilinear filtering.The process continues until the original resolution is recovered.
  • Camera sampling: Camera poses are sampled from training trajectories because they better represent occupable locations than uniform sampling across the scene.Each generated scene still uses one sampled camera pose during training.

A.6. Training Details

GSN inversion combines an encoder-predicted latent-grid initialization with optimization against posed source views. Training uses StyleGAN-inspired stabilization techniques and substantial compute for 64 × 64 models.

  • Training details: Training uses RMSprop, a reduced mapping-network learning rate, exponential moving-average generator weights, differentiable augmentation, and lazy R1 regularization.The mapping-network learning rate is 100× lower than the rest of the network, and R1 is applied every 16 iterations.
  • Training details: 64 × 64 generation models were trained for 500k iterations with batch size 32, requiring four days on two 40GB NVIDIA A100 GPUs.Mixed precision was applied to the generator but not the discriminator because discriminator stability decreased.
  • GSN inversion: The inversion encoder predicts an initial local latent-code grid from posed source views using image features back-projected into a shared feature volume.The encoder uses a UNet with a ResNet-50 encoder and averages the resulting volume features.
  • GSN inversion: The inversion loss jointly reconstructs the latent grid and encourages the locally conditioned radiance field to reproduce the input views.The first term matches the latent grid, while the second matches rendered views.
  • GSN inversion: At inference, the initialized grid is optimized for 1000 SGD iterations after selecting a preferred global orientation, then rendered for source reconstruction and target-view prediction.The generator remains frozen during reconstruction-based optimization.

C. Qualitative Results on Local vs. Global Coordinate Systems

The qualitative view-synthesis results evaluate GSN on unseen Vizdoom sequences and Replica scenes using sparse source views to reconstruct inputs and predict target views. The figures organize each row by a different source-view set and distinguish inputs from predictions.

  • Coordinate-system evaluation: The coordinate-system experiment evaluates latent-code rotations of 0, 90, 180, and 270 degrees for sampled scenes.The experiment tests robustness to rigid rearrangement of the latent codes.
  • Vizdoom: GSN inversion produces a local latent-code grid that reconstructs source views and predicts target views from their camera poses.Green frames are inputs, while blue frames are predictions.
  • Replica: Replica results include both training scenes and held-out test scenes, with the bottom three rows representing unseen scenes.Each row corresponds to a different set of source views.

D. Scene Editing

GSN’s local latent grid supports direct scene editing and coordinate transformations, enabling mirrored or rotated scene variants. Editing quality depends on coordinate-system choice, latent-grid resolution, and alignment between source scenes.

  • Coordinate systems: Local coordinates produce higher rendering quality than global coordinates as latent-code rotation increases.The global-coordinate representation degrades with larger rotation angles.
  • Latent-grid editing: Directly altering the local latent grid provides manual control over scene synthesis beyond random generator sampling.The paper demonstrates manipulating local codes to create modified scenes.
  • Editing limitations: Current low-resolution latent grids mainly support high-level edits, while larger grids could enable finer changes such as furniture rearrangement.The paper presents larger local grids as a route toward more detailed control.
  • Latent-grid editing: Mirroring local latent codes along the horizontal axis produces unique scene variants.The figures show panoramas alongside their corresponding local latent codes.
  • Editing limitations: Scene compositions are most convincing when the input scenes are well aligned in appearance and geometry.The demonstrations use codes from single scenes to avoid alignment requirements between different inputs.
Loading 2104.00670v1…