Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations

1 Introduction

• Prior neural scene representations either omit or weakly enforce 3D structure, leaving black-box renderers to learn projection and risking poor structure discovery with limited data.The paper also identifies missing guarantees of multi-view consistency, limited interpretability, and difficulty generalizing to unseen camera intrinsics and transformations.Source: §1 · p. 1
• SRNs encode geometry and appearance as a continuous function from world coordinates to local scene features, with a learned renderer for novel viewpoints.The representation and renderer enforce 3D structure and multi-view consistency while interfacing with multi-view and projective geometry.Source: §1 · p. 2, Figure 1 · p. 3
• Trained end-to-end using only posed 2D images, SRNs render at arbitrary resolutions and generalize to camera transformations and intrinsics unseen during training.They operate on individual pixels without 2D convolutions and can render close-ups despite training only on objects viewed from a constant distance.Source: §1 · p. 2
• SRNs are evaluated on novel view synthesis, few-shot reconstruction, shape and appearance interpolation, and unsupervised discovery of a non-rigid face model, outperforming recent-literature baselines.The contribution summary also describes the representation and renderer as efficiently encapsulating scene geometry and appearance.Source: §1 · p. 2
• The current formulation does not model view- or lighting-dependent effects or translucency, entangles shape with appearance, and is non-probabilistic.The paper identifies these as directions for future work.Source: §1 · p. 2

2 Related Work

• Geometric deep-learning methods use discrete grids, point clouds, or meshes, while function-space methods model decision boundaries or signed distance fields; these often need 3D supervision and do not efficiently represent appearance.The introduction notes that discrete representations limit spatial resolution and sparsely sample smooth scene surfaces.Source: §1 · p. 1, §2 · p. 2
• Prior neural scene methods use latent codes for pose-conditioned rendering or latent-space view transformations, while Generative Query Networks remain oblivious to scene 3D structure.Other approaches infer voxel scenes, use 3D-aware voxel generative models, capture structure with graph neural networks, or represent scenes as programs.Source: §2 · p. 2
• Two-dimensional generative models parameterize natural-image manifolds but struggle with multi-view-consistent generation because they cannot exploit underlying 3D scene structure.The related-work discussion includes adversarial and conditional models, coordinate-to-color functions, spatial or perspective transformations, and graphics-engine inversion.Source: §2 · p. 3

3.1 Representing Scenes as Functions

• A continuous function maps each 3D world coordinate to a feature describing local scene properties, such as color, reflectance, or signed distance.An MLP parameterizes the function, so practical spatial resolution is limited by its capacity; unlike sparse point clouds or voxel grids, it models properties continuously over R3.Source: §3.1 · p. 3

3.2 Neural Rendering

• The renderer maps a scene representation and camera intrinsics and extrinsics to an image, requiring implicit surface intersections and feature-to-color conversion.These are handled by ray marching to find intersections and a pixel generator to map intersection features to colors.Source: §3.2 · p. 4

3.2.1 Differentiable Ray Marching Algorithm

• A ray-marching LSTM uses the feature at each current ray estimate to predict the next step length, producing an intersection estimate after a fixed number of steps.Fixed iteration counts avoid relying on a dynamic stopping rule while the scene function and marcher are randomly initialized; the resulting depth maps are differentiable and interpretable, but are not required for training.Source: §3.2.1 · pp. 4–5

3.2.2 Pixel Generator Architecture

• A per-pixel MLP maps each intersection feature to RGB, making rendering multi-view consistent when intersections are correct and allowing arbitrary image resolutions and poses.The trade-off is that ray marching, scene representations, and the generator operate at the same resolution, requiring substantial memory and forgoing CNN architectural benefits.Source: §3.2.2 · p. 5

3.3 Generalizing Across Scenes

• For a class of scenes, a hypernetwork maps low-dimensional latent codes to scene-function parameters, while the rendering-function parameters are shared across instances.Instance codes are optimized jointly with the hypernetwork and renderer under an auto-decoder framework with a zero-mean diagonal-covariance Gaussian prior; the paper demonstrates this generalization for single-object classes.Source: §3.3 · p. 5

3.4 Joint Optimization

• The joint objective combines rendered-image ℓ2 loss, a depth regularizer for positive ray distances, and a Gaussian latent-code prior; stochastic gradient descent trains the pipeline end-to-end.For a single scene, optimization instead solves for the scene MLP parameters; few-shot reconstruction fixes the learned renderer and hypernetwork and estimates a new instance's latent code.Source: §3.4 · p. 6

Shepard-Metzler objects

• SRNs achieve 30.41 dB PSNR on training Shepard-Metzler objects versus 20.85 dB for dGQN, and 24.36 dB versus 18.56 dB in two-shot reconstruction.In the one-shot setting, dGQN achieves 17.51 dB and SRNs achieve 18.11 dB; both reconstruct an object resembling the mean of feasible objects.Source: §4 · p. 7

Shapenet v2

• Across Shapenet v2 chair and car tests, SRNs outperform dGQN, TCO, and WRL in all training- and few-shot settings, with multi-view consistency and plausible completions.Views are generally multi-view consistent, except for objects with fine geometric details such as convertible windscreens; baselines do not produce multi-view consistent views.Source: §4 · pp. 7–8, Figure 6 · p. 7

Supervising parameters for non-rigid deformation

• Conditioned on Basel face identity and expression parameters, SRNs reconstruct geometry and appearance and smoothly animate unseen identity-expression combinations.The face model uses 224 parameters, including 160 identity and 64 expression dimensions; reconstructed normal maps demonstrate learned geometry.Source: §4 · p. 8

Geometry reconstruction

• SRNs reconstruct geometry in a fully unsupervised manner as needed to explain observations in 3D, including 50-shot, single-shot, and single-scene reconstructions.Source: §4 · p. 8

Latent space interpolation

• Interpolating learned latent codes for Shapenet cars and chairs produces smooth transitions between models as the camera rotates.Source: Figure 5 · p. 7, §4 · p. 8

Pose extrapolation

• SRNs generalize to unseen 3D camera transformations, including close-ups and camera roll, despite training only on upright camera poses around objects.Source: §4 · p. 8

Failure cases

• Ray marching can fail in surface holes or near occluders, and objects far from the training distribution may match texture without reconstructing geometry.The resulting continuous or stepped-through surfaces can be analyzed through reconstructed geometry, unlike failures from black-box alternatives.Source: §4 · p. 8

Towards representing room-scale scenes

• On a Minecraft room-scale scene, SRNs infer geometry and appearance and produce largely multi-view-consistent views, with failures on low-texture areas and thin tubes.Generated images are blurry, with ray-marching artifacts mainly at object boundaries and thin structures; occluding columns and objects are reconstructed correctly.Source: §4 · p. 9

Discussion

• SRNs encode scenes as continuous, differentiable functions mapping 3D coordinates to scene features, trained from posed 2D images through differentiable ray marching without shape supervision.The paper demonstrates novel view synthesis, shape and appearance interpolation, and few-shot reconstruction.Source: §5 · p. 9
• Future work could extend SRNs to probabilistic scene sampling, view- and lighting-dependent effects, translucency, participating media, and image formation models such as CT or MRI.Source: §5 · p. 9
• SRNs currently require camera intrinsic and extrinsic parameters, though future work could integrate differentiable camera parameters with learned pose-estimation algorithms [72].The paper notes that these parameters can be obtained robustly via bundle adjustment.Source: §5 · p. 9
• Generalization across complex, cluttered 3D environments remains open despite SRNs' ability to represent room-scale scenes; meta-learning may enable generalization with weaker manifold-dimensionality assumptions [73].The discussion also proposes exploring SRNs for robotic manipulation or as an independent agent's world model.Source: §5 · p. 9

• Ray marching progresses from the top left to the bottom right, with the final normal map visualizing surface detail.Figure 1 shows both the marching progress and the final normal map.Source: Scene Representation Networks:… · p. 14
• Because the uniformly colored background does not constrain depth, depth is unconstrained around object silhouettes; the main document therefore reports only final normal maps.Source: Scene Representation Networks:… · p. 14

Computation of Normal Maps

• Normal maps visualize fine surface detail significantly better than depth maps, and surface normals are the cross product of numerical horizontal and vertical depth-map derivatives.The paper reports normal maps in the main submission rather than depth maps.Source: §1 · p. 14

Ray Marching Progress Visualization

• Ray-march intersection estimates at each iteration yield camera-coordinate depth maps that visualize every step; examples show ray marches alongside their final normal maps.The passage describes two example ray marches.Source: §1 · p. 14

2 Comparison to DeepVoxels

• The comparison evaluates single-scene novel-view synthesis on four synthetic objects against DeepVoxels, Pix2Pix, Tatarchenko et al., Worrall et al., and Cohen and Welling.DeepVoxels uses a feature voxel grid with multi-view and projective geometry hard-coded into its architecture.Source: §2 · p. 14
• SRN renders are almost perfectly multi-view consistent while DeepVoxels flickers, but SRNs blur some very high-frequency textures.The qualitative comparison shows reconstructed geometry normals, SRN outputs, and ground truth for each object.Source: §2 · p. 14, Figure 2 · p. 15
• SRNs beat DeepVoxels, the best baseline, by more than 3 dB across four scenes with 550k parameters versus over 160M free variables.Table 1 compares PSNR and SSIM averaged over all four scenes.Source: §2 · p. 14, Table 1 · p. 16
• Undersampled cube-side lettering aliases across views, violating SRN's view-independent color assumption for each world coordinate; the nonsampled CVPR logo is reproduced accurately.Letter lines can be less than two pixels wide, and some occupy only a single pixel; the DeepVoxels 2D downsampling also introduced blur that is not multi-view consistent.Source: §2 · pp. 15–16
• A U-Net-based 2D CNN renderer reconstructs the undersampled lettering and other detail truthfully, but sacrifices the multi-view consistency guarantee.The paper presents this as a trade-off between robustness to faulty training data and multi-view consistency.Source: Figure 4 · p. 15, §2 · p. 16

3 Reproducibility

• The authors will make all code and datasets publicly available, and evaluated each model on the test sets exactly once.Source: §3 · p. 16

3.1 Architecture Details

• Across experiments, Φ is a four-layer, 256-unit ReLU MLP with layer normalization before each nonlinearity.Generalization experiments generate Φ’s weights with Ψ, which uses separate three-layer MLPs for each Φ layer; its hidden width is 256 for Shepard–Metzler and ShapeNet and 224 for Basel faces. The ray-marching LSTM has 16 hidden units and zero initial state; the five-layer, 256-unit pixel generator uses ReLU and layer normalization. For DeepVoxels, Φ is scene-specific and Kaiming-Normal initialized.Source: §3.1 · pp. 16–17

3.2 Time & Memory Complexity

• Φ and the pixel generator scale linearly with query count and quadratically with image resolution, while Ψ’s final layer dominates model parameters.With four Φ layers of 256 units, Φ has about 2 × 10^5 parameters, while Ψ’s final layer has about 5 × 10^7; Ψ need only be queried once. Ray-marching memory and runtime also scale linearly with steps and quadratically with image resolution.Source: §3.2 · p. 17

3.3 Dataset Details

• Dataset construction used Blender-rendered Shepard–Metzler objects, ShapeNet cars and chairs, sampled Basel faces, and the DeepVoxels dataset.ShapeNet views were sampled on spheres of radius 1.3 for cars and 2.0 for chairs; 40 faulty cars were discarded. Basel identity parameters were sampled from N(0, 0.7), and expression parameters uniformly from (−0.4, 1.6). Shepard–Metzler meshes came from a modified open-source renderer, while DeepVoxels followed.Source: §3.3 · pp. 17–18

3.4.1 General details

• Training used coarse-to-fine image resolutions to enable larger early batches and more independent views per object, alongside ADAM and single-GPU PyTorch runs.The ADAM settings were β1 = 0.9 and β2 = 0.999; training used RTX6000 or RTX8000 GPUs. SRN hyperparameters were selected through informal search rather than a systematic grid search because of computational cost.Source: §3.4.1 · p. 18

3.4.2 Per-experiment details

• Schedules varied by experiment: Shepard–Metzler trained directly at 64 × 64 for 352 epochs; cars, chairs, and faces increased from 64 × 64 to 128 × 128, while DeepVoxels used three stages.At 64 × 64, batch size was 72 and decreased fourfold with each doubling of resolution; λdepth was 1 × 10−3, λlatent was 1, and the default ADAM learning rate was 4 × 10−4 unless otherwise reported. Cars trained at 64 × 64 for 5k iterations then 128 × 128 for 70 epochs; chairs used 20k iterations then 12 epochs; faces used 15k then 5k iterations; DeepVoxels used 12 × 128 for 20k iterations at 4 × 10−4, 256 × 256 for 30k at 1 × 10−4, then 512 × 512 for 30k at 4 × 10−6.Source: §3.4.2 · p. 18

4 Relationship to per-pixel autoregressive methods

• PixelCNN and PixelRNN [10, 11] model an image as a one-dimensional sequence of pixel values and estimate their joint distribution.Source: §4 · p. 19
• Conditioned on scene representation Φ, SRNs deterministically assign each pixel a value, making pixel values conditionally independent and simplifying image probability to scene probability under camera parameters E and K.Source: §4 · p. 19
• This conditional independence motivates the per-pixel design of the rendering function Θ.Source: §4 · p. 19

5.1 Deterministic Variant of GQN

• The GQN-inspired baseline is deterministic to isolate scene-representation properties; its results need not represent unaltered GQN, and probabilistic SRNs and comparison to unaltered GQN are left to future work.Source: §5.1 · p. 19
• The baseline feeds the Tower representation directly to a deterministic autoregressive skip-convolutional LSTM generator, which updates a canvas recurrently and produces the final image.The generator conditions state updates on camera extrinsics and uses convolutional downsampling and upsampling; extra layers accommodate higher-resolution car and chair images.Source: §5.1 · p. 19
• Training used 180,000 iterations for cars and chairs and 160,000 for Shepard-Metzler objects; testing supplied 15 nearest views for training-set synthesis and the closest reference view for reconstruction.Batch sizes were 140 for cars and chairs and 192 for Shepard-Metzler objects.Source: §5.1 · p. 19

5.2 Tatarchenko et al.

• The Tatarchenko et al. baseline uses the architecture and hyperparameters described in, with approximately 70 · 10^6 parameters; it receives no depth supervision and training stops early based on held-out validation performance.Source: §5.2 · p. 20
• For Tatarchenko et al. [3], test-time inputs are the nearest view for training-set synthesis and the closer of two references—or the sole reference—for reconstruction.Source: §5.2 · p. 20

5.3 Worrall et al.

• The Worrall et al. baseline follows its prescribed architecture and hyperparameters, including nearest-neighbor upsampling, leaky ReLU activations, and batch normalization.Source: §5.3 · p. 20
• For Worrall et al. [4], test-time inputs are the nearest view for training-set synthesis and the closer of two references—or the sole reference—for reconstruction.Source: §5.3 · p. 20

6 Differentiable Ray-Marching in the context of classical renderers

• Sphere tracing steps by the signed distance to the nearest surface, but is guaranteed to converge only with infinitely many steps; the Ray-Marching LSTM can learn step lengths.Classical sphere tracing commonly takes many steps and assigns the closest surface's color near the intersection, but this heuristic can fail in constructed cases.Source: §6 · p. 21
• Each ray-marching step evaluates the scene representation for every pixel, and each evaluation requires a full forward pass through a multilayer perceptron.The paper identifies ray marching itself as the main driver of the proposed renderer's computational and memory cost.Source: §6 · p. 21
• Rasterization projects explicit surface primitives onto the image plane in one step, enabling computationally cheap real-time rendering that has approached photorealism.Its image-formation model is not appropriate for physically accurate effects such as light transport, refraction, and translucency.Source: §6 · p. 21
• Physics-based rendering commonly uses ray tracing, tracing rays from each pixel through the scene toward light sources, with complexity driven by sampling the paths that contribute to pixel values.With explicit discrete representations such as meshes, intersection testing is cheap; the number of rays needed to sample contributing paths is the main complexity driver.Source: §6 · p. 21
• The proposed ray-marcher is a sphere-tracing-inspired ray-tracer for implicit geometry, but it does not currently model multi-bounce ray tracing.The paper notes that multi-bounce ray tracing could potentially be added in future work.Source: §6 · p. 21

7 Trade-offs of the Pixel Generator vs. CNN-based renderers

• The pixel generator ensures multi-view consistency but uses full-resolution layers, forgoes memory-saving CNN designs, and has unclear limits on high-frequency detail generation.It is not essential to SRNs and can be replaced by a 2D-CNN renderer; U-Net is cited as an example of a CNN architecture with resolution bottlenecks and skip connections.Source: §7 · p. 21

8 Future work

• SRNs may support robotics and independent agents by serving as environment models for navigation, manipulation, and interaction planning.The paper identifies robotic manipulation and world models for independent agents as natural applications outside vision.Source: §8 · p. 21
• SRNs could extend to CT, MRI, or other image-formation models given a differentiable forward model, adapting or omitting the ray-marcher as appropriate.For direct 3D observations, the ray-marcher may be omitted and φ sampled directly.Source: §8 · p. 22
• An unverified probabilistic extension would infer distributions over feasible scenes from observations, following Kumar et al. and Eslami et al..It sums observation codes into a permutation-invariant scene code, samples latents with autoregressive DRAW, and decodes scene-network parameters through a hypernetwork for differentiable rendering. Training uses 2D images and camera parameters; the intractable optimization requires an approximate-posterior evidence lower bound, and the fixed scene representation yields multi-view consistency.Source: §8 · p. 22
• Future scene modeling could incorporate view- and lighting-dependent effects, multiple ray bounces, translucency, and participating media.The pixel generator could receive camera-ray direction and, assuming no occlusions, light-ray direction; the ray-marcher could sum features along a ray for translucency and participating media.Source: §8 · p. 22
• Generalization across complex, cluttered 3D environments remains open, and the feasibility of low-dimensional embeddings for photo-realistic general environments has not been demonstrated.The paper suggests meta-learning as a possible way to generalize across scenes without restricting them to a highly low-dimensional manifold.Source: §8 · p. 22