Source-linked AI summary
Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations
Vincent Sitzmann, Michael Zollhöfer, Gordon Wetzstein
TL;DR
Existing neural scene representations often ignore or weakly enforce 3D structure, limiting multi-view consistency and discovery under limited data. SRNs address this with a continuous 3D-structure-aware representation and differentiable ray marching trained from posed 2D images, demonstrating novel view synthesis, interpolation, few-shot reconstruction, and non-rigid face-model discovery.
Problem
Prior neural scene representations do not or only weakly enforce 3D structure, leading to failures under limited training data and lacking guarantees of multi-view consistency.
Method
SRNs represent scenes as continuous differentiable functions mapping 3D world coordinates to feature representations and train them end-to-end with differentiable ray marching from posed 2D images.
Results
SRNs demonstrate novel view synthesis, shape and appearance interpolation, few-shot reconstruction, and unsupervised discovery of a non-rigid face model while significantly outperforming recent baselines.
Takeaways & Limitations
The representation jointly captures scene geometry and appearance while producing generally multi-view-consistent novel views and enabling generalization of shape and appearance priors across scenes.
Takeaways & Limitations
Generalization across complex, cluttered 3D environments remains an open problem, and SRNs currently require camera intrinsic and extrinsic parameters.
Abstract
from arXiv · showhide
Unsupervised learning with generative models has the potential of discovering rich representations of 3D scenes. While geometric deep learning has explored 3D-structure-aware representations of scene geometry, these models typically require explicit 3D supervision. Emerging neural scene representations can be trained only with posed 2D images, but existing methods ignore the three-dimensional structure of scenes. We propose Scene Representation Networks (SRNs), a continuous, 3D-structure-aware scene representation that encodes both geometry and appearance. SRNs represent scenes as continuous functions that map world coordinates to a feature representation of local scene properties. By formulating the image formation as a differentiable ray-marching algorithm, SRNs can be trained end-to-end from only 2D images and their camera poses, without access to depth or shape. This formulation naturally generalizes across scenes, learning powerful geometry and appearance priors in the process. We demonstrate the potential of SRNs by evaluating them for novel view synthesis, few-shot reconstruction, joint shape and appearance interpolation, and unsupervised discovery of a non-rigid face model.
1 Introduction
Prior neural scene representations often neglect or weakly enforce 3D structure, while classic geometric representations are discrete and commonly require explicit 3D supervision. SRNs address this gap with a continuous, 3D-structure-aware representation and differentiable renderer trained from posed 2D images.
- Motivation: Prior neural scene representations weakly enforce 3D structure, limiting multi-view consistency and performance with limited training data.Their projection and multi-view geometry operations are delegated to black-box neural renderers.
- Related work: Classic voxel, point-cloud, and mesh representations are discrete, limit spatial resolution, and often require explicit 3D supervision.
- Scene Representation Networks: SRNs represent scenes as continuous differentiable functions mapping 3D world coordinates to feature-based local scene properties.The representation models both scene geometry and appearance.
- Contributions: SRNs train end-to-end from posed 2D images without explicit 3D supervision and support novel view synthesis, interpolation, few-shot reconstruction, and non-rigid face-model discovery.The paper reports that SRNs significantly outperform recent literature baselines.
- Scope: The formulation does not model view- or lighting-dependent effects or translucency, entangles shape with appearance, and is non-probabilistic.
- Scene Representation Networks: A differentiable renderer combines learned ray marching and pixel generation to render SRN scenes from arbitrary novel viewpoints.
3 Formulation
SRNs represent scenes continuously in 3D and render them differentiably from posed images, enforcing multi-view structure while learning geometry and appearance without explicit 3D supervision. The formulation extends across object instances through latent codes and joint end-to-end optimization.
- 3.1 Representing Scenes as Functions: SRNs represent a scene as a continuous function mapping each 3D world coordinate to a learned feature vector of local scene properties.Features may encode color, reflectance, or signed distance, while densely modeling scene properties throughout space.
- 3.1 Representing Scenes as Functions: Because the function input is a world coordinate, SRNs can use multi-view and perspective geometry to enforce 3D-aware, multi-view-consistent representations.Learning approximates unknown scene properties while established geometric operations handle spatial structure.
- 3.2 Neural Rendering: The neural renderer maps a scene representation and camera intrinsic and extrinsic parameters to an RGB image.The renderer is defined over the space of scene functions and supports arbitrary viewpoints.
- 3.2 Neural Rendering: Rendering implicitly defined geometry requires locating ray-surface intersections and converting the sampled feature vector into pixel color.SRNs address these subproblems with adaptive ray marching followed by a learned feature-to-color pixel generator.
- 3.2.1 Differentiable Ray Marching Algorithm: RM-LSTM predicts each ray-marching step length from the feature vector at the current estimated intersection.The procedure repeatedly computes world coordinates, evaluates Φ, predicts a step, and updates the ray distance for a fixed number of iterations.
- 3.2.2 Pixel Generator Architecture: The per-pixel MLP maps each intersection feature independently to RGB, making rendering trivially multi-view consistent when intersections are correct.Unlike 2D convolutions, independent pixel processing avoids dependence on a changing 2D feature neighborhood under camera transformations.
- 3.3 Generalizing Across Scenes: Across instances of one object class, SRNs use latent vectors to generate scene-function parameters through a hypernetwork while sharing the renderer.This assumes class instances share shape and appearance properties lying in a lower-dimensional parameter subspace.
- 3.4 Joint Optimization: Joint optimization trains the complete pipeline end-to-end from posed images using image, depth-positivity, and latent-prior losses, without geometry or scene-scale supervision.For a single scene, optimization reduces to solving for the parameters of its scene-function MLP.
4 Experiments
SRNs are evaluated across novel-view synthesis, few-shot reconstruction, latent interpolation, non-rigid face modeling, pose extrapolation, and room-scale scenes. The experiments show strong multi-view consistency and geometry reconstruction, while also exposing ray-marching and out-of-distribution failure cases.
- Latent space interpolation: Latent-code interpolation produces smooth transitions between car and chair models while the camera rotates around the objects.The experiment evaluates meaningful interpolation of object instances in the learned latent space.
- Novel-view synthesis and few-shot reconstruction: SRNs outperform three recent baselines for Shapenet novel-view synthesis across 50-, two-, and single-observation settings, producing multi-view-consistent views and reconstructing geometry.In the single-shot setting, SRNs plausibly complete unseen object parts using learned priors.
- Non-rigid face modeling: SRNs reconstruct face geometry and appearance while smoothly varying 64 expression parameters with identity fixed, including unseen identity-expression combinations.The model also produces reconstructed normal maps from depth maps, indicating learned geometric structure.
- Pose extrapolation: Explicit 3D-aware, per-pixel rendering generalizes to unseen camera transformations, including close-ups and camera roll, despite training on upright camera poses.The formulation supports rendering at arbitrary resolutions and poses through a rendering function operating independently on pixels.
- Failure cases: Ray marching can get stuck in surface holes or near occluders, and objects far from the training distribution may yield texture matching without correct geometry.The reconstructed geometry makes these failures analyzable, unlike black-box alternatives.
- Room-scale scenes: In a room-scale Minecraft scene trained from 500 observations, SRNs infer largely multi-view-consistent images and reconstruct occluding columns and objects, but outputs remain blurry.Failures occur mainly in low-texture areas and thin tubes, while ray-marching artifacts appear at object boundaries and thin structures.
5 Discussion
SRNs provide a continuous, differentiable, 3D-structured scene representation trained from posed 2D images without shape supervision. The discussion identifies extensions including probabilistic modeling, richer image formation, camera-pose estimation, and broader scene generalization.
- SRNs implicitly represent scenes as continuous, differentiable functions mapping 3D coordinates to feature-based scene representations.
- A differentiable ray marcher renders these representations into 2D images for end-to-end training from posed images without shape supervision.
- The demonstrated applications include novel view synthesis, shape and appearance interpolation, and few-shot reconstruction.
- Future work could extend SRNs to probabilistic modeling, view- and lighting-dependent effects, translucency, participating media, and other image-formation models.
- Generalization across complex, cluttered 3D environments remains an open problem, despite SRNs representing room-scale scenes.
Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations –Supplementary Material–
The supplementary material visualizes neural ray marching from intermediate progress to final surface normals. Uniform backgrounds leave depth unconstrained near object silhouettes, while normal maps better expose surface detail.
- Ray-marching visualizations show progress from intermediate intersection estimates to the final normal map.
- Uniformly colored backgrounds do not constrain depth around object silhouettes.
- The supplement reports final normal maps because they visualize surface detail better than the corresponding depth visualizations.
1 Additional Results on Neural Ray Marching
The supplementary analysis explains how neural ray marching produces depth maps at each iteration and how surface normals are computed from depth derivatives. Normal maps are preferred for visualizing fine detail.
- Surface normals are computed as the cross product of numerical horizontal and vertical derivatives of the depth map.
- Normal maps visualize fine surface detail significantly better than depth maps.
- The z-coordinates of running and final intersection estimates produce depth maps that visualize every ray-marching step.
2 Comparison to DeepVoxels
Against DeepVoxels and other baselines on four synthetic objects, SRNs achieve stronger novel-view results and substantially lower model size, while undersampled high-frequency textures expose a consistency–detail trade-off.
- Comparison to DeepVoxels: More than 3 dB: SRNs outperform the best baseline, DeepVoxels, on PSNR averaged over all 4 scenes.DeepVoxels uses a voxel grid with hard-coded multi-view and projective geometry.
- Comparison to DeepVoxels: 550k parameters per model: SRNs achieve this result versus DeepVoxels with more than 160M free variables.
- Comparison to DeepVoxels: SRNs are almost perfectly multi-view consistent, whereas DeepVoxels displays significant flickering artifacts.
- High-frequency texture limitations: Lines of cube lettering occupy as little as a single pixel, causing aliasing across viewing angles and preventing SRNs from generating those details.
- High-frequency texture limitations: A U-Net renderer reproduces undersampled details truthfully but trades away the guarantee of multi-view consistency.
3 Reproducibility
The experiments use publicly reproducible implementations, specified architectures, datasets, training schedules, and compute settings. SRNs are trained with coarse-to-fine image resolutions, while hyperparameter selection is acknowledged as informal.
- Architecture: SRNs use an MLP scene representation with four 256-unit layers, ReLU activations, and layer normalization before each nonlinearity.
- Architecture: In generalization experiments, a hypernetwork maps each scene latent vector to the weights of its corresponding scene representation.
- Complexity: Memory and runtime scale quadratically with image resolution because the per-pixel generator and scene representation process a number of queries proportional to pixel count.
- Training: Training uses coarse-to-fine image resolutions, increasing resolution in stages to enable larger early batch sizes and more independent views per object.
- Reproducibility: Training hyperparameters were selected through informal search rather than a systematic grid search because of the high computational cost.
4 Relationship to per-pixel autoregressive methods
SRNs differ from autoregressive per-pixel methods by conditioning deterministic, independently generated pixel values on a shared scene representation. This design makes the image likelihood depend on the scene representation and camera parameters rather than a sequential pixel factorization.
- PixelCNN and PixelRNN model an image as a one-dimensional sequence and estimate its joint pixel distribution autoregressively.
- Conditioned on scene representation Φ, SRNs assign each pixel value independently and deterministically under camera extrinsic E and intrinsic K parameters.
- Conditional pixel independence motivates SRNs’ per-pixel rendering function Θ.
5 Baseline Discussions
The baseline discussion constructs deterministic comparisons to prior systems, preserving their architectures where possible while removing depth supervision when geometry is unavailable. These baselines are trained and evaluated under explicitly described view-selection protocols.
- Deterministic Variant of GQN: The GQN-inspired baseline is made deterministic by feeding its scene representation directly to a generator rather than parameterizing a latent-variable density.
- Deterministic Variant of GQN: The baseline generator updates recurrent states and a canvas using camera extrinsics, scene representation, downsampling, upsampling, and final convolutional operations.
- Training: Cars and chairs are trained for 180,000 iterations, while lower-resolution Shepard-Metzler objects are trained for 160,000 iterations.
- Testing: For novel-view synthesis, the GQN-inspired baseline receives 15 nearest training views, whereas one-shot and two-shot reconstruction use one or the closer of two reference views.
- Worrall et al. baseline: The Worrall et al. baseline follows the referenced architecture and hyperparameters, using nearest-view selection for synthesis and one- or two-view inputs for reconstruction.
- DeepVoxels: The DeepVoxels comparison uses the architecture from Tatarchenko et al., omits depth-map supervision, and stops training early based on held-out validation performance.
6 Differentiable Ray-Marching in the context of classical renderers
The proposed ray-marcher is positioned between classical rasterization and ray tracing as a sphere-tracing-inspired renderer for implicitly defined geometry. It provides multi-view consistency through the pixel generator but incurs resolution-dependent memory costs and leaves several rendering effects unsupported.
- Classical renderers: Rasterization projects explicit discretized geometry onto the image plane in one step, making it computationally cheap and suitable for real-time rendering.
- Classical renderers: Ray tracing instead follows rays through scene paths toward light sources, with complexity driven by the number of rays needed to sample contributing light paths.
- Differentiable ray marching: SRNs’ ray-marcher is a sphere-tracing-inspired ray tracer for implicitly defined geometry, but it does not currently model multi-bounce ray tracing.
- Pixel-generator trade-offs: The pixel generator guarantees multi-view consistency relative to a 2D-CNN renderer, but its full-resolution layers are comparatively memory-hungry.
- Pixel-generator trade-offs: The pipeline’s limitations include unclear high-frequency texture performance and the absence of view- and lighting-dependent effects, translucency, and probabilistic modeling.
8 Future work
Future work extends SRNs beyond current vision settings, toward broader image-formation models, probabilistic scene inference, realistic effects, and more complex environments.
- SRNs could support robotic manipulation and serve as world models for agents that model environments, navigate, and plan interactions.
- A differentiable forward model would allow SRNs to extend to computer tomography, magnetic resonance imaging, or direct 3D observations.The ray-marcher could be adapted to integrate features or sample predefined locations, and omitted for direct 3D measurements.
- A proposed probabilistic extension would infer distributions over feasible scenes consistent with observations, but it is not experimentally verified for SRNs.The formulation requires approximate-posterior optimization of an evidence lower bound.
- Current SRNs omit view- and lighting-dependent effects and translucency, while entangling shape with appearance and remaining non-probabilistic.Future extensions could condition pixel generation on ray direction and integrate features along rays for participating media.
- Generalization across complex, cluttered 3D environments remains an open problem, especially for low-dimensional embeddings of photorealistic general scenes.