Source-linked AI summary
DeepVoxels: Learning Persistent 3D Feature Embeddings
Vincent Sitzmann, Justus Thies, Felix Heide, Matthias Nießner, Gordon Wetzstein, Michael Zollhöfer
TL;DR
Existing generative neural networks lack robust 3D understanding for coherent novel-view synthesis. DeepVoxels addresses this with a persistent Cartesian 3D feature volume trained using 2D supervision and structured image formation, achieving high-quality results across challenging scenes. The approach remains memory inefficient and generalization to substantially different viewpoints is challenging.
Problem
Existing 2D generative architectures struggle to generate coherent views of the same scene because they do not robustly represent 3D layout and large 3D transformations.
Method
DeepVoxels learns a persistent, viewpoint-invariant Cartesian 3D feature grid, fuses lifted image features into it, and renders target views through differentiable perspective projection and a 2D network.
Results
DeepVoxels produces high-quality novel views across challenging synthetic and real scenes, outperforming baselines by an average 7dB on four scanned objects.
Takeaways & Limitations
Persistent 3D scene representations can support high-quality novel-view synthesis while using 2D supervision and avoiding explicit geometry modeling.
Takeaways & Limitations
The dense 3D volume is memory inefficient, forcing a trade-off between local resolution and spatial extent, while substantially different viewpoints remain challenging.
Abstract
from arXiv · showhide
In this work, we address the lack of 3D understanding of generative neural networks by introducing a persistent 3D feature embedding for view synthesis. To this end, we propose DeepVoxels, a learned representation that encodes the view-dependent appearance of a 3D scene without having to explicitly model its geometry. At its core, our approach is based on a Cartesian 3D grid of persistent embedded features that learn to make use of the underlying 3D scene structure. Our approach combines insights from 3D geometric computer vision with recent advances in learning image-to-image mappings based on adversarial loss functions. DeepVoxels is supervised, without requiring a 3D reconstruction of the scene, using a 2D re-rendering loss and enforces perspective and multi-view geometry in a principled manner. We apply our persistent 3D scene representation to the problem of novel view synthesis demonstrating high-quality results for a variety of challenging scenes.
1. Introduction
DeepVoxels addresses the difficulty of generating coherent novel views by introducing a persistent 3D feature representation that uses native 3D operations without explicitly reconstructing geometry. It is trained with 2D supervision while enforcing perspective, occlusion, and multi-view structure.
- Motivation: Existing 2D generative networks struggle to generate coherent views of the same scene because large 3D transformations are difficult to learn through 2D convolutions and skip connections.These architectures must map 3D operations into 2D kernels, while skip connections can bypass higher-level reasoning.
- Technical contributions: Learned soft visibility enables explicit occlusion reasoning, improving novel-view quality and generalization to unseen viewpoints.The occlusion mechanism is a stated technical contribution of the representation.
- Training: Differentiable image formation enforces perspective and multi-view geometry during training without requiring 3D reconstruction or 3D supervision.The method uses a 2D re-rendering objective and learns from posed multi-view images.
- Scope: The investigation assumes Lambertian scenes and does not explicitly model specular or other view-dependent effects.The approach can handle light specularities, but these are not modeled explicitly.
2. Related Work
The related work spans generative image synthesis, 3D geometric learning, learned novel-view synthesis, model-based rendering, and image-based rendering. DeepVoxels is positioned between these areas by combining 3D structure with high-fidelity image generation.
- Neural Image Synthesis: Neural image synthesis has used autoencoders, autoregressive models, and conditional GANs, often with encoder-decoder or U-Net architectures.These methods produce strong 2D image and video synthesis results but are discussed as background to the paper’s 3D-structured approach.
- 3D Deep Learning: 3D deep-learning methods represent geometry with occupancy grids, signed distance fields, point clouds, or meshes, but these representations do not directly provide sufficiently high-resolution color for view synthesis.Related work also includes learned image embeddings and spatial transformation methods for scene properties such as rotation and illumination.
- View Synthesis and Rendering: Novel-view synthesis methods predict reference-view lookups, blend multi-view images, learn layered representations, or decode low-dimensional scene embeddings into new views.The section also covers model-based rendering from reconstructed geometry and image-based rendering through warped or blended input images.
3. Method
DeepVoxels learns a persistent, viewpoint-invariant 3D voxel feature representation from registered multi-view images without explicit geometry supervision. The architecture fuses lifted observations into this volume, reprojects it to target views, reasons about occlusion, and renders images with 2D networks.
- 3. Method: DeepVoxels is a persistent, viewpoint-invariant 3D voxel grid whose spatial structure organizes learned per-voxel feature codes.The representation serves as the latent space of an encoder-decoder architecture.
- 3.1. Training Corpus: Training uses a source image with camera pose and two randomly selected target views from registered multi-view images, with camera intrinsics and extrinsics available.The source is sampled from the top-5 nearest neighbors by view direction to one target, increasing the likelihood that source points are visible there and encouraging multi-view consistency.
- 3.2. Architecture Overview: A 2D feature extractor lifts source-image features into a temporary 3D volume, which a persistent GRU-based integration network fuses across observations.The hidden state remains persistent across training steps, while a 3D inpainting U-Net fills holes in the resulting feature representation.
- 3.2. Architecture Overview: An occlusion network predicts soft visibility along depth rays and collapses the canonical volume into a 2D feature map for rendering.The visibility weights can also be used to compute a depth map, while the rendering network predicts target pixels from the flattened volume.
- 3.2. Architecture Overview: The projection layer maps 3D code vectors into each target camera’s canonical view volume using perspective geometry and trilinear interpolation.The camera model uses intrinsic and extrinsic matrices; the inverse mapping associates canonical-view voxels with positions in the persistent world-space grid.
4. Analysis
DeepVoxels achieves high-quality novel-view synthesis on synthetic and real scenes, outperforming the evaluated baselines while explicitly modeling voxel visibility for more robust rendering.
- Implementation: The implementation uses 170 million parameters and renders a single test frame in 71ms.Models are trained separately for each scene using a 32^3 voxel volume unless otherwise specified.
- Object-specific Novel View Synthesis: DeepVoxels achieves the best PSNR and SSIM results on all evaluated objects, outperforming the baselines by an average of 7dB.The comparison uses four high-quality 3D scans and includes Pix2Pix, autoencoder, and rotation-equivariant latent-space baselines.
- Object-specific Novel View Synthesis: The model preserves challenging fine details, including foreshortened cube letters and high-frequency vase features, despite a voxel grid 16 times smaller than the image resolution.Trilinear interpolation in lifting and projection may support this fine-grained representation.
- Occlusion Reasoning and Interpretability: The occlusion module raises PSNR from 23.26dB to 28.42dB on the cube and from 30.02dB to 32.35dB on the pedestal.It also produces fully unsupervised depth maps that expose learned 3D scene structure.
- Occlusion Reasoning and Interpretability: Explicit occlusion reasoning improves rendering by forcing a softmax-weighted selection of voxels along each ray, preventing hidden parts from shining through.Without the occlusion module, combining features from multiple depths produces severe artifacts at inference time.
- Novel View Synthesis for Real Captures: On real captures, DeepVoxels generally produces high-quality novel views, with artifacts mainly affecting details significantly smaller than one voxel.Highly specular surfaces can introduce inconsistencies, as illustrated by the coffee bag.
5. Limitations
DeepVoxels has memory and viewpoint-generalization limitations. Dense voxel volumes constrain the trade-off between local resolution and spatial extent, while large pose changes can reduce detail.
- Dense 3D volumes are memory inefficient, forcing a trade-off between local resolution and spatial extent.A 64^3 voxel volume with 8 feature channels fills 12GB of GPU memory.
- Viewpoint generalization remains challenging when synthesized poses differ significantly from the training set.Detail is lost for poses far from the training views, although the scene's 3D structure is generally preserved.
- Future sparse neural networks may replace the dense representation at the method's core.The authors also report that compelling results can be achieved with relatively small volume resolutions.
6. Conclusion
The paper proposes DeepVoxels, a 3D-structured scene representation that encodes view-dependent appearance using only 2D supervision. It introduces native 3D operations as a first step toward overcoming limitations of existing 2D generative models.
- DeepVoxels encodes the view-dependent appearance of a 3D scene using only 2D supervision.
- The representation introduces native 3D operations into the network to address limitations of existing 2D generative models.