PaperlayerView exact revision on arXiv

Source-linked AI summary

Volume Rendering of Neural Implicit Surfaces

Lior Yariv, Jiatao Gu, Yoni Kasten, Yaron Lipman

arXiv:2106.12052v2cs.CV

TL;DR

Neural volume rendering often learns noisy, low-fidelity geometry from generic density functions and arbitrary density level sets. VolSDF models density as a Laplace CDF applied to a signed distance function, enabling bounded opacity error and geometry–radiance disentanglement. It produces high-quality reconstructions on multiview datasets and supports switching shape and appearance between scenes.

  • Problem

    Neural volume rendering commonly represents density generically and extracts geometry at arbitrary density level sets, yielding noisy, low-fidelity reconstructions.

  • Method

    VolSDF models volume density as a Laplace CDF applied to a learnable signed distance function.

  • Results

    VolSDF improves geometry approximation over previous neural volume-rendering techniques while maintaining view-synthesis quality and enabling density–radiance disentanglement.

  • Takeaways & Limitations

    The signed-distance density provides an inductive bias for geometry, bounds opacity approximation error for high-fidelity sampling, and supports switching fields between scenes.

  • Takeaways & Limitations

    The method lacks a correctness proof for its sampling algorithm and cannot represent non-watertight manifolds or manifolds with boundaries using an SDF.

Abstract

from arXiv · show

Neural volume rendering became increasingly popular recently due to its success in synthesizing novel views of a scene from a sparse set of input images. So far, the geometry learned by neural volume rendering techniques was modeled using a generic density function. Furthermore, the geometry itself was extracted using an arbitrary level set of the density function leading to a noisy, often low fidelity reconstruction. The goal of this paper is to improve geometry representation and reconstruction in neural volume rendering. We achieve that by modeling the volume density as a function of the geometry. This is in contrast to previous work modeling the geometry as a function of the volume density. In more detail, we define the volume density function as Laplace's cumulative distribution function (CDF) applied to a signed distance function (SDF) representation. This simple density representation has three benefits: (i) it provides a useful inductive bias to the geometry learned in the neural volume rendering process; (ii) it facilitates a bound on the opacity approximation error, leading to an accurate sampling of the viewing ray. Accurate sampling is important to provide a precise coupling of geometry and radiance; and (iii) it allows efficient unsupervised disentanglement of shape and appearance in volume rendering. Applying this new density representation to challenging scene multiview datasets produced high quality geometry reconstructions, outperforming relevant baselines. Furthermore, switching shape and appearance between scenes is possible due to the disentanglement of the two.

1 Introduction

VolSDF addresses noisy, low-fidelity geometry in neural volume rendering by defining density from signed-distance geometry. This representation improves reconstruction, ray sampling, and disentanglement while preserving view-synthesis quality.

  • Neural volume rendering synthesizes novel views from sparse input images but often produces noisy, low-fidelity geometry.
  • VolSDF defines volumetric density as a function of signed distance to the scene surface rather than using a generic density model.
  • The signed-distance density provides an inductive bias that improves geometry approximation and helps disentangle density from radiance.
  • Bounding opacity approximation error enables viewing-ray sampling that more faithfully couples density and radiance in the rendering integral.
  • On DTU and BlendedMVS, VolSDF reconstructs surfaces more accurately than NeRF and NeRF++ while avoiding object masks.
  • VolSDF can switch density and radiance fields between scenes, whereas this disentanglement fails in NeRF-based models.

2 Related work

Related work spans neural implicit surfaces, classical multi-view stereo, and concurrent volume-rendering methods with implicit representations. VolSDF combines volume rendering with neural implicit surfaces while avoiding masks and using signed-distance-specific opacity bounds.

  • Multi-view 3D reconstruction: Classical multi-view stereo methods estimate depth or voxels, then commonly require fusion and an additional meshing step to obtain surfaces.
  • Concurrent volume-rendering methods: Concurrent work incorporates implicit surfaces into volume rendering with an occupancy network and an explicit surface-smoothing loss.
  • Concurrent volume-rendering methods: VolSDF instead uses a signed-distance representation regularized with an Eikonal loss, without explicit smoothing, and derives an opacity approximation bound.

3 Method

VolSDF parameterizes volume density as a transformed signed distance function, enabling principled surface extraction and bounded opacity approximation for sampling the rendering integral.

  • 3.1 Density as transformed SDF: VolSDF defines density as αΨβ(−dΩ(x)), where dΩ is a learnable signed distance function and Ψβ is a Laplace CDF.The parameters α and β are learnable, with β controlling the boundary smoothing amount.
  • 3.1 Density as transformed SDF: The transformed SDF provides a surface prior and makes the learned surface recoverable as the zero level-set of dΩ rather than an arbitrary density level-set.The density models a homogeneous object with density α and smoothly decreases near its boundary.
  • 3.2 Volume rendering of σ: Volume rendering computes expected radiance along a camera ray from opacity, transparency, and radiance, using numerical quadrature over discrete samples.The radiance field may depend on position, surface normal, and viewing direction.
  • 3.2 Volume rendering of σ: Adaptive sampling is important because the ray PDF is concentrated near object boundaries, while generic opacity models do not provide an explicit inverse opacity function.NeRF addresses inverse sampling with a separately trained coarse network whose density may not faithfully represent the final density.
  • 3.3 Bound on the opacity approximation error: VolSDF bounds the opacity approximation error for rectangle-rule integration by bounding density variation along each ray interval.The bound uses endpoint distances and density parameters, and sufficiently dense sampling can reduce it below any chosen ϵ.

4 Experiments

VolSDF is evaluated on DTU and BlendedMVS for multiview surface reconstruction, comparison with neural rendering and implicit-surface baselines, and geometry–appearance disentanglement.

  • Evaluation setup: VolSDF is evaluated on DTU and BlendedMVS, using quantitative and qualitative surface-reconstruction comparisons with relevant baselines.DTU includes 15 selected scans; BlendedMVS evaluation uses 9 scenes with complex backgrounds.
  • Reconstruction results: VolSDF improves NeRF reconstructions considerably while remaining on-par with NeRF in rendering quality measured by PSNR.
  • Baseline comparisons: VolSDF produces more accurate surface reconstructions than NeRF and NeRF++, with reconstruction comparable to IDR while avoiding object masks.
  • Baseline comparisons: With ground-truth masks, IDR introduces extraneous surface parts in the compared scene, whereas mask-free VolSDF provides a more faithful result.
  • Geometry–appearance disentanglement: VolSDF successfully switches materials between two objects, while both tested NeRF disentanglement variants fail to produce correct disentanglement.The experiment switches radiance fields between two trained scenes.

5 Conclusions

The paper concludes that VolSDF represents volume density through a transformed signed distance function, improving geometry approximation and supporting disentanglement and high-fidelity sampling. It also identifies limitations involving sampling guarantees, representable manifolds, density assumptions, and potential misuse.

  • VolSDF represents volume density as a transformed signed distance function to define learned surface geometry within a volume-rendering framework.
  • This representation provides an inductive bias for disentangling geometry and radiance, improves geometry approximation, and bounds opacity approximation error for high-fidelity sampling.
  • Limitations: The sampling algorithm works well in practice, but the authors do not have a proof of correctness for it.
  • Limitations: The SDF formulation cannot represent non-watertight manifolds or manifolds with boundaries, and the current formulation assumes homogeneous density.
  • Limitations: Accurate geometry reconstruction from images can be used for malicious purposes, although the authors report no immediate negative societal impact.

A.1 Sampling ablation study

The sampling ablation compares uniform, hierarchical, and VolSDF sampling, finding that alternative procedures reduce reconstruction and rendering accuracy and introduce artifacts.

  • Sampling strategies: VolSDF sampling is compared against uniform 256-sample, hierarchical coarse-and-fine, and one- or five-iteration alternatives.The study evaluates geometry with Chamfer distances and rendering with PSNRs.
  • Results: Alternative sampling procedures lead to lower accuracy in both geometry reconstruction and rendering.The reported artifacts include geometry defects around the head top and nose, plus salt-and-pepper noise and over-smoothed regions in renderings.

A.2 Positional encoding ablation

The positional-encoding ablation shows a tradeoff between smoother geometry and more detailed rendering, while VolSDF alleviates rendering artifacts and improves geometry fidelity relative to NeRF.

  • VolSDF ablation: Higher positional encoding improves VolSDF specular highlights and details but adds undesired reconstruction noise.VolSDF uses level 6 positional encoding, while NeRF uses level 10 in the comparison.
  • Positional encoding levels: Level 6 positional encoding produces smoother geometry, whereas level 10 yields more detailed rendering with slightly higher noise.The comparison reports both rendered-image PSNR and surface Chamfer distance.
  • Qualitative results: VolSDF rendering alleviates NeRF’s salt-and-pepper artifacts while producing higher-fidelity geometry approximation on DTU and BlendedMVS.Figures 10 and 11 provide additional qualitative results for the two datasets.

A.4 Limitations

The supplied evidence highlights robustness to sampling choices and successful disentanglement, while also documenting failure cases and conditions affecting disentanglement quality.

  • Rendering robustness: VolSDF produces consistent rendering with random and regularly spaced sampling, unlike NeRF, which exhibits different artifacts.The comparison uses the same scene and sampling strategy for both methods.
  • Disentanglement conditions: Geometry and radiance disentanglement succeeds when normal information is incorporated and partially fails without it.The normal dependency has a similar effect in VolSDF to that reported for surface rendering.
  • Scene transfer: Radiance fields can be switched between independently trained scenes, transferring material and lighting across geometries.Additional BlendedMVS results show off-diagonal post-training combinations and graceful transfer between three scenes.
  • Evaluation boundary: The evaluation uses known camera poses, object masks for a DTU comparison, and dataset-specific surface-processing procedures.DTU masking removes non-visual-hull parts, while BlendedMVS evaluation uses ground-truth meshes and largest connected components.

B.3 Additional implementation details

The implementation uses separate geometry and radiance networks, positional encoding, dataset-specific training schedules, zero-level-set mesh extraction, and bounded background modeling.

  • Architecture: The geometry network is an eight-layer MLP with width 256, one skip connection, and outputs an SDF value plus a 256-dimensional feature vector.The network receives a 3D position and uses geometric initialization.
  • Encoding: Positional encoding uses 6 levels for position and 4 levels for view direction.The position encoding choice is consistent with the reported VolSDF configuration.
  • Training: Training uses ADAM with a learning rate decayed from 5e-4 to 5e-5 over 100K DTU or 200K BlendedMVS iterations.A DTU model takes approximately 12 hours on a single Nvidia V-100 GPU.
  • Mesh extraction: Meshes are extracted with Marching Cubes from the zero level set of the signed distance function.The extracted surface is defined by d(x).
  • Background modeling: The background model uses a scene bounding sphere and inverted-sphere parameterization so rays that miss surfaces become eventually occluded.The sphere radius is set to 3, with maximal ray-sample depth 2r and 32 background samples.
  • Baselines: NeRF and NeRF++ baselines reconstruct geometry by extracting a fixed 50 level set of their learned foreground density.NeRF++ uses separate foreground and background networks with uniform and hierarchical samples.

C Proofs and additional lemmas

The section establishes Lipschitz-based bounds for the density and opacity approximation, then explains how dense sampling and numerical quadrature control rendering error.

  • Bounds for density and distance: The density’s Lipschitz constant can be bounded on each ray segment despite the distance function not being everywhere differentiable.The proof uses the distance function’s global Lipschitz property and properties of the Laplace distribution.
  • Opacity approximation bounds: The opacity approximation error is bounded interval-by-interval using the Lipschitz bound and an explicit lower distance bound between geometric sets.The lower bound is defined through the minimum distance between a ray segment and the boundary of the complement of neighboring balls.
  • Sampling guarantees: For any ϵ > 0, sufficiently dense sampling provides an opacity-error bound below ϵ, implying that dense sampling can achieve arbitrarily low error.This result follows from bounding the relevant inner sum by the maximum sample interval.
  • Numerical quadrature: The rendering integral is discretized with the rectangle rule, while transmittance is represented through segment survival probabilities and cumulative products.The final discrete probability includes an implementation-only terminal sample whose probability is approximately zero outside the bounding sphere.
Loading 2106.12052v2…