PaperlayerView exact revision on arXiv

Source-linked AI summary

NeRF-VAE: A Geometry Aware 3D Scene Generative Model

Adam R. Kosiorek, Heiko Strathmann, Daniel Zoran, Pol Moreno, Rosalia Schneider, Soňa Mokrá, Danilo J. Rezende

arXiv:2104.00587v1stat.MLcs.LG

TL;DR

Existing scene-reconstruction and generative-rendering methods either optimize separately for each scene or lack explicit 3D geometry. NeRF-VAE combines a latent-conditioned NeRF decoder with amortized inference, enabling geometrically consistent reconstruction and rendering from few views, including out-of-distribution cameras.

  • Problem

    Existing methods lack a unified way to combine learned scene priors, efficient novel-scene inference, and explicit geometric rendering.

  • Method

    NeRF-VAE is a VAE that uses a latent-conditioned NeRF decoder and an encoder to infer scene latents from context images and camera poses.

  • Results

    NeRF-VAE infers and renders geometrically consistent novel scenes from few views, generalizes to out-of-distribution cameras, and achieves lower MSE with uniformly lower worst-case errors than NeRF trained on fewer than 100 views.

  • Takeaways & Limitations

    Explicit geometry and learned scene priors provide view consistency and efficient inference across previously unseen environments.

  • Takeaways & Limitations

    NeRF-VAE has lower per-scene expressivity than NeRF because capacity is divided between shared across-scene information and a limited per-scene latent.

Abstract

from arXiv · show

We propose NeRF-VAE, a 3D scene generative model that incorporates geometric structure via NeRF and differentiable volume rendering. In contrast to NeRF, our model takes into account shared structure across scenes, and is able to infer the structure of a novel scene -- without the need to re-train -- using amortized inference. NeRF-VAE's explicit 3D rendering process further contrasts previous generative models with convolution-based rendering which lacks geometric structure. Our model is a VAE that learns a distribution over radiance fields by conditioning them on a latent scene representation. We show that, once trained, NeRF-VAE is able to infer and render geometrically-consistent scenes from previously unseen 3D environments using very few input images. We further demonstrate that NeRF-VAE generalizes well to out-of-distribution cameras, while convolutional models do not. Finally, we introduce and study an attention-based conditioning mechanism of NeRF-VAE's decoder, which improves model performance.

1 Introduction

NeRF-VAE combines NeRF-style geometric rendering with shared structure across scenes and amortized inference. It targets efficient reconstruction and generation from few views while retaining geometric consistency.

  • Motivation and contribution: NeRF-VAE models multiple scenes with shared structure and uses amortized inference to reason about novel scenes without per-scene retraining.Its generative formulation also supports missing data and sampling completely new scenes.
  • Motivation and contribution: Unlike convolutional rendering models, NeRF-VAE uses explicit 3D geometry and rendering to produce geometrically consistent views.The model defines a distribution over scene functions that can render arbitrary views of a scene.
  • Motivation and contribution: The model infers scene parameters including colours, geometry, and depth maps, and can sample novel scenes from its prior.Training uses several views with associated camera positions and orientations from multiple scenes.
  • Motivation and contribution: NeRF-VAE learns scene priors across multiple environments, enabling reconstruction of unobserved scenes from fewer input views.The learned shared information includes common textures, shapes, and environmental properties.

2 Neural Radiance Fields (NeRF)

NeRF represents scenes as continuous radiance fields queried along camera rays and rendered differentiably. Its architecture separates position-dependent density from view-dependent colour to encourage multi-view consistency.

  • Scene representation: NeRF maps ray position and orientation to RGB colour and volume density using a neural scene function.The function takes position x and orientation d and outputs ((r, g, b), σ).
  • Scene representation: NeRF makes volume density depend only on position while emitted colour depends on both position and ray orientation.This architectural separation encourages multi-view consistency.
  • Volumetric rendering: Differentiable volumetric rendering casts one ray per image pixel and computes each ray colour as a density-weighted average of colours along the ray.The renderer approximates the accumulation process differentiably.
  • Volumetric rendering: The rendering process takes a camera and scene function as inputs and outputs a rendered image.Camera parameters determine the image plane and corresponding pixel rays.

3 NeRF-VAE

NeRF-VAE replaces per-scene NeRF optimization with a VAE whose latent variable captures scene-specific information while shared parameters capture structure across scenes. An encoder infers the latent from context views, and a conditional NeRF decoder renders images or novel scenes.

  • Generative model: A latent variable z defines each scene, while conditional scene-function parameters θ capture structure shared across scenes.Sampling z from a prior produces novel scene functions for generation.
  • Amortized inference: Amortized inference avoids estimating NeRF parameters separately for every scene, addressing a time-consuming, compute-intensive, and data-hungry procedure.Iterative amortized inference can trade additional computation for improved inference by refining posterior parameters.
  • Amortized inference: The encoder maps context images and camera poses into an approximate posterior over z, which conditions the scene function used for arbitrary-camera rendering.Context elements are encoded separately with a shared encoder and aggregated before producing posterior parameters.
  • Training objective: NeRF-VAE trains θ and encoder parameters φ with an ELBO combining image reconstruction likelihood and KL divergence to the latent prior.In practice, the ELBO is approximated by uniformly subsampling pixels.
  • Geometric decoder: NeRF-VAE’s decoder preserves explicit geometric structure, allowing iterative inference to use geometry through ELBO gradients rather than a geometry-agnostic feedforward encoder.The conditional scene function is evaluated through the NeRF rendering process.
  • Conditioning the scene function: The attention-based scene function lets input points attend to a spatial latent representation, while density remains position-dependent as in NeRF.Bottleneck attention blocks reduce computation and memory because the scene function is evaluated many times per pixel.

4 Related Work

NeRF-VAE belongs to amortized neural-rendering and generative scene-modeling work, differing in its compact latent scene representation, explicit geometry, and camera-conditioned formulation. Related approaches vary in whether they require known poses, parameter updates, or support dynamic scenes.

  • Amortized neural rendering: Several amortized neural-rendering methods use NeRF decoders but lack a compact scene representation or require projecting rendered points into input images.Other methods meta-learn NeRF initializations but still update parameters before rendering target observations.
  • Amortized neural rendering: NeRF-VAE differs from sequential ray-marching approaches by using volumetric rendering and demonstrating generalization across scenes rather than only objects from one class.The cited comparison identifies these as distinct design choices.
  • Camera assumptions: Some generative models approximate the marginal distribution of camera poses, whereas the related amortized rendering approaches described here require known input camera poses.This separates pose-conditioned reconstruction methods from pose-distribution-based GAN approaches.
  • Dynamic scenes: NeRF is limited to static scenes and lacks support for varying lighting and transient effects, motivating extensions with per-view latents or temporal components.NeRF-VAE’s explicit scene latent suggests adding a latent dynamics model for videos.

5 Experiments

Experiments evaluate NeRF-VAE on novel-view reconstruction, out-of-distribution camera generalization, decoder and inference variants, and unconditional scene sampling. NeRF-VAE reconstructs scenes from fewer views than NeRF, maintains geometric consistency under novel viewpoints, and produces samples with consistent geometry.

  • Comparison with NeRF: NeRF-VAE achieves lower MSE and uniformly lower worst-case errors than NeRF trained with fewer than 100 views.NeRF-VAE was trained with Nctx = 4 context views and generalizes to different numbers of context views at test time.
  • Comparison with NeRF: NeRF-VAE’s errors concentrate near a small positive value, whereas NeRF has wide MSE distributions and a long tail of large errors.The comparison includes NeRF trained with fewer than 100 views and with 100 views.
  • Comparison with a Convolution-Based Generative Model: NeRF-VAE produces plausible, geometrically consistent outputs from out-of-distribution camera views, while CONV-AR-VAE distorts scene colours and ignores downward orientations.Within-distribution, CONV-AR-VAE also shows inconsistent object persistence, whereas NeRF-VAE renders inferred scene geometry properly.
  • Comparison with a Convolution-Based Generative Model: Iterative inference improves MLP-based scene functions and reduces KL while improving reconstruction, but has a smaller effect on attentive scene functions.Attentive models achieve lower reconstruction errors in high-KL regimes, suggesting higher capacity for complicated data.
  • Samples & Uncertainty: NeRF-VAE samples scenes by drawing z from its prior and rendering multiple views, producing samples resembling the training distribution with consistent geometric structure.Depth estimates of sampled scenes reveal consistent geometry.
  • Samples & Uncertainty: From a single context image, NeRF-VAE can predict a barely visible object while retaining uncertainty about its exact shape.Predictive variance in the depth estimate distinguishes uncertainty between a sphere and an icosahedron.

6 Discussion

NeRF-VAE combines geometric structure with a learned prior over scene functions to support view-consistent inference from few views, while retaining uncertainty under limited information. Its benefits are constrained by degenerate data and limited per-scene expressivity.

  • NeRF-VAE is view-consistent and generalizes to out-of-distribution cameras, unlike convolutional models such as GQN.
  • Sampling latent variables produces scenes whose appearance and variability resemble the training distribution, with depth estimates revealing consistent geometry.
  • The model infers a plausible full object from a single context image in which the object is barely visible.Its predictive depth variance preserves uncertainty about the object's exact shape.
  • Degenerate camera distributions and unnecessary view-dependent color modeling can still lead NeRF-VAE to implausible scene explanations.
  • Limited latent capacity, required for amortized inference, reduces per-scene expressivity compared with NeRF.
  • The low-dimensional latent variable supports potential extensions such as scene interpolation, dynamic scenes and videos, and representations that grow with input complexity.

A Dataset Details

The experiments use randomized CLEVR and Jaytracer scenes, with hierarchical NeRF rendering and encoder-based latent inference. The study also compares scene-function conditioning and convolutional baselines.

  • CLEVR: CLEVR randomizes camera pose and scene object count relative to the original dataset.Camera distance, elevation, azimuth, and object count are varied.
  • Jaytracer: Jaytracer contains diverse cameras, randomized textured environments, two objects, lighting, and raytraced signed-distance-function scenes.Its broad camera distribution makes scene inference more challenging than on GQN and CLEVR.
  • NeRF rendering: NeRF-VAE decoders use hierarchical sampling with coarse and fine scene-function evaluations before volume rendering.The fine function is evaluated at points resampled using coarse density estimates.
  • Encoder and inference: Context images and camera parameters are encoded into posterior parameters, with global pooling for vector latents but feature-map latents for attentive models.Iterative inference can refine these posterior parameters using sampled context pixels and an LSTM.
  • Model comparisons: The study compares orientation-aware and orientation-free scene functions, MLP conditioning variants, and a convolutional autoregressive VAE baseline.The convolutional baseline conditions both prior and posterior on the camera, unlike NeRF-VAE.

E.1 Details: Comparison with NeRF

The comparison evaluates amortized NeRF inference against independently optimized NeRF models on Jaytracer and CLEVR. NeRF-VAE reaches strong performance with far fewer context views, while NeRF remains sensitive to sparse or extreme viewpoints.

  • Experimental setup: NeRF-VAE is evaluated against NeRF using subsampled rays, distinct training configurations, and CLEVR ray intervals extending to [0.05, 14].NeRF uses more coarse and fine ray samples than NeRF-VAE.
  • Results: On CLEVR, NeRF-VAE performs well from one observed view, whereas NeRF requires up to 50 views for comparable MSE.CLEVR objects are visible in a single image, allowing the inference model to encode the whole scene.
  • Error analysis: With 20 NeRF views, a heavy tail of prediction errors remains across 100 test views, especially for extreme camera elevations.These viewpoints are less likely to be represented among the input views.

E.2 Details: Comparison with a Convolution-Based Generative Model

This section examines out-of-distribution camera trajectories and orientation inputs when comparing NeRF-VAE with a convolutional generative baseline. Orientation inputs preserve depth estimates but can distort colors at unseen orientations.

  • Model setup: The comparison includes NeRF-VAE, CONV-AR-VAE, and scene functions with MLP or attention-based conditioning.Training uses different optimizer schedules and latent representations across datasets and model variants.
  • Orientation inputs: NeRF-VAE with orientation inputs produces expected within-distribution interpolations but distorted colors out of distribution.The depth estimates remain unaffected because density depends only on position.
  • Camera trajectories: Out-of-distribution cameras are lifted from the ground and tilted downward relative to within-distribution interpolations.The trajectories interpolate between endpoint positions and orientations in different camera regimes.

E.4 Generalization on CLEVR

On CLEVR out-of-distribution tests, NeRF-VAE generalizes better than the convolutional baseline to novel camera viewpoints, while attention-based scene functions handle more objects than MLP scene functions.

  • Novel camera viewpoints: NeRF-VAE produces higher-quality images than CONV-AR-VAE on unseen CLEVR camera viewpoints, with lower MSE and KL.The test includes elevation, azimuth, and camera-distance interpolations outside the training distribution.
  • Number of objects: Attention-based scene functions generalize to four or five objects after training on one to three, whereas MLP scene functions miss objects.The attention-based variant also achieves lower MSE.
Loading 2104.00587v1…