Source-linked AI summary
Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision
Michael Niemeyer, Lars Mescheder, Michael Oechsle, Andreas Geiger
TL;DR
Existing image-based 3D reconstruction methods are limited by costly 3D supervision or discretized voxel and mesh representations. DVR analytically differentiates depth for implicit shape and texture representations, enabling learning from images alone while rivaling full-3D-supervision models and producing watertight meshes in multi-view reconstruction.
Problem
Existing implicit-representation methods require costly 3D ground truth, while differentiable image-based approaches are limited to discretized voxels or meshes.
Method
DVR derives analytic depth gradients for implicit shape and texture networks, enabling their training from multi-view images and object masks.
Results
DVR learns implicit 3D shape representations without 3D supervision, rivals full-3D-supervision models, and directly produces watertight meshes for multi-view reconstruction.
Takeaways & Limitations
DVR broadens the scope of implicit shape and texture representations to image-supervised single- and multi-view reconstruction on synthetic and real data.
Takeaways & Limitations
The method currently requires object masks and camera information and estimates texture rather than more complex material properties.
Abstract
from arXiv · showhide
Learning-based 3D reconstruction methods have shown impressive results. However, most methods require 3D supervision which is often hard to obtain for real-world datasets. Recently, several works have proposed differentiable rendering techniques to train reconstruction models from RGB images. Unfortunately, these approaches are currently restricted to voxel- and mesh-based representations, suffering from discretization or low resolution. In this work, we propose a differentiable rendering formulation for implicit shape and texture representations. Implicit representations have recently gained popularity as they represent shape and texture continuously. Our key insight is that depth gradients can be derived analytically using the concept of implicit differentiation. This allows us to learn implicit shape and texture representations directly from RGB images. We experimentally show that our single-view reconstructions rival those learned with full 3D supervision. Moreover, we find that our method can be used for multi-view 3D reconstruction, directly resulting in watertight meshes.
1. Introduction
Learning-based 3D reconstruction can infer models from single images but typically depends on accurate 3D supervision, limiting real-world use. DVR addresses this by deriving analytic depth gradients for continuous implicit shape and texture representations, enabling reconstruction from 2D images and masks.
- Motivation: Most learning-based methods require accurate 3D ground truth, restricting them mainly to synthetic data despite strong single-image reconstruction performance [12] [13] [17].Recent alternatives use 2D supervision, including depth maps or multi-view images, by modifying the rendering process.
- Motivation: Existing differentiable-rendering approaches use voxel- or mesh-based representations that suffer discretization artifacts and computational limits on resolution or template deformation [11] [15] [21] [33] [43].Implicit shape and texture representations avoid discretization during training [12].
- Contribution: DVR derives analytic gradients of predicted depth with respect to implicit representation parameters, making volumetric rendering differentiable for shape and texture.The formulation learns implicit 3D representations solely from multi-view images and object masks.
- Contribution: DVR’s memory footprint is independent of depth-sampling accuracy because it need not store volumetric data during the forward pass.The method supports single- and multi-view reconstruction on synthetic and real data.
- Contribution: Unlike, DVR learns shared geometry and texture parameters without conditioning texture on geometry.A single model represents both geometry and texture.
2. Related Work
Related work spans voxel-, point-, mesh-, and implicit 3D representations, as well as classical and learned reconstruction pipelines using differentiable rendering. This work distinguishes itself by making volumetric rendering for implicit shape and texture representations differentiable, enabling learning from 2D supervision.
- 3D Representations: Implicit representations avoid spatial discretization and maintain a fixed memory footprint, addressing resolution, connectivity, and topology limitations of voxel-, point-, and mesh-based methods.Voxel methods remain limited to relatively small resolutions, point methods require post-processing for missing connectivity, and mesh methods often rely on deformable templates.
- Differentiable Rendering: This work shows that volumetric rendering for implicit representations is inherently differentiable, enabling implicit shape and texture learning from 2D supervision.The approach learns representations directly from image-based supervision rather than requiring 3D supervision.
- 3D Reconstruction: Classical multi-view stereo matches features or reconstructs voxel grids, while learned methods refine depth, optimize MVS components, or replace the pipeline end-to-end.Feature-based methods produce depth maps requiring lossy fusion, whereas this method is supervised from 2D images alone and outputs a consistent 3D representation.
- Differentiable Rendering: Mesh-based differentiable-rendering methods, including OpenDR and soft rasterization, often require deformable template meshes, restricting output topology.These approaches approximate or soften the traditional graphics pipeline and have produced compelling reconstruction results.
- Differentiable Rendering: Voxel-grid differentiable-rendering methods provide probabilistic ray-potential formulations but require storing intermediate evaluations, limiting resolution.Other implicit-rendering methods use silhouettes or LSTM-based differentiable rendering; the former cannot recover concave shapes from silhouettes alone, while the latter cannot extract geometry directly.
3. Method
DVR represents shape and texture with implicit neural fields and learns them from 2D images by rendering surface colors and differentiating surface depth analytically. Its implementation uses ray-based surface extraction, implicit differentiation, and automatic differentiation to optimize photometric reconstruction losses efficiently.
- Shape: An occupancy network defines the object surface as the level set fθ = τ, enabling arbitrary-resolution isosurface extraction from an image-conditioned latent code.The occupancy network assigns occupancy probabilities to 3D points, while the encoder conditions predictions on the input image.
- Texture: A jointly implemented texture field evaluates surface color at the reconstructed point, with shape and texture conditioned on the same object embedding.The texture is defined on the surface fθ = τ, and fθ and tθ are implemented as one neural network with two shallow heads.
- Rendering: DVR renders each pixel by casting a camera ray, finding its first intersection with the implicit surface, and evaluating the texture field at that point.The predicted image is compared with the observed image using a photometric reconstruction loss, such as a robust ℓ1-based measure.
- Gradients: Implicit differentiation yields a closed-form gradient for surface depth, allowing gradients to flow through the rendered color without storing voxel-based intermediate volumes.The resulting parameter gradients combine derivatives of the texture field, surface point, and occupancy network, and are computed with reverse-mode automatic differentiation.
- Implementation and supervision: The forward pass samples occupancy along each ray and applies an iterative secant method at the first free-space-to-occupied transition, while batching surface-depth computation.Implicit differentiation avoids unrolling the forward pass or storing intermediate results; training can use one image per object and optionally incorporate depth information.
- Training losses: The training objective supports RGB losses on valid interior rays, mask-based handling of exterior rays, and an optional normal smoothness prior for unconstrained real-world regions.The normal prior encourages more natural shapes when supervision is only 2D or 2.5D.
4. Experiments
Experiments show that the method reconstructs accurate shape and texture from single RGB views without 3D supervision, including with only one training view per object. It also produces watertight meshes for real-world multi-view reconstruction while rivaling tuned MVS methods.
- Experimental Scope: The experiments evaluate single-view reconstruction with multi-view or single-view supervision and real-world multi-view reconstruction on challenging DTU scans with changing appearance.DTU experiments use full-resolution images and exclude views with profound lighting changes; texture results for DRC and SoftRas are unavailable because their official code lacks texture prediction.
- Single-View Reconstruction: Our method performs best among 2D-supervised approaches and rivals fully 3D-supervised methods for single-view shape reconstruction, while also predicting texture.This result uses Chamfer-L1 distances against ground-truth meshes and is supported by both quantitative and qualitative evaluations.
- Single-View Supervision: A model trained with one RGB-D view per object achieves Chamfer distance 0.410, comparable to 0.383 for the 24-view depth-supervised model.The single-view model infers appropriate 3D shape and texture by aggregating information across training instances in canonical object coordinates.
- Multi-View Reconstruction: On real-world DTU scans, our method rivals highly tuned MVS algorithms and directly produces watertight meshes for multi-view reconstruction.Its accurate cavity modeling uses texture information to improve over the visual hull; the evaluation covers RGB images with masks and optional sparse depth.
5. Conclusion and Future Work
DVR derives analytic depth gradients for implicit representations, enabling learning from multi-view imagery without 3D supervision and supporting multi-view 3D reconstruction. Future work targets removing object-mask and camera-information requirements and modeling richer material properties.
- Conclusion: DVR analytically derives depth gradients with respect to network parameters because volumetric rendering is differentiable for implicit representations.This formulation enables learning implicit 3D shape representations from multi-view imagery without 3D supervision.
- Conclusion: DVR achieves results comparable to highly optimized multi-view stereo methods on DTU scans 65, 106, and 118.Baselines use screened Poisson surface reconstruction with trim parameters 0, 5, and 7 to produce final outputs.
- Conclusion: DVR learns implicit 3D shape representations from multi-view imagery without 3D supervision, rivaling models learned with full 3D supervision.The method also supports multi-view 3D reconstruction and broadens applications of implicit shape and texture representations.
- Future Work: Future work will remove the need for object masks and camera information, potentially through soft-mask prediction, and estimate more complex material properties beyond texture.These directions address current input requirements and extend the method's appearance modeling scope.