PaperlayerView exact revision on arXiv

Source-linked AI summary

NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction

Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, Wenping Wang

arXiv:2106.10689v3cs.CVcs.GR

TL;DR

Existing neural surface methods can require masks and struggle with local optimization around occlusions, while volume-rendered fields lack sufficient surface constraints. NeuS trains a neural SDF with a new volume-rendering formulation, achieving high-quality reconstruction on DTU and BlendedMVS, including complex and occluded scenes.

  • Problem

    Existing surface-rendering methods require mask supervision and can struggle with complex structures, while volume-rendered representations lack sufficient constraints for high-fidelity surface extraction.

  • Method

    NeuS represents surfaces as neural signed distance functions and trains them with a novel volume-rendering scheme based on SDF-induced density.

  • Results

    NeuS produces high-quality reconstructions and outperforms state-of-the-art methods qualitatively and quantitatively, including scenes with severe occlusions and complex structures.

  • Takeaways & Limitations

    NeuS combines the accurate surface representation of neural SDFs with volume rendering’s robust training under abrupt depth changes.

  • Takeaways & Limitations

    SDF rendering involves a trade-off: standard density formulations are biased, whereas an unbiased construction is not occlusion-aware.

Abstract

from arXiv · show

We present a novel neural surface reconstruction method, called NeuS, for reconstructing objects and scenes with high fidelity from 2D image inputs. Existing neural surface reconstruction approaches, such as DVR and IDR, require foreground mask as supervision, easily get trapped in local minima, and therefore struggle with the reconstruction of objects with severe self-occlusion or thin structures. Meanwhile, recent neural methods for novel view synthesis, such as NeRF and its variants, use volume rendering to produce a neural scene representation with robustness of optimization, even for highly complex objects. However, extracting high-quality surfaces from this learned implicit representation is difficult because there are not sufficient surface constraints in the representation. In NeuS, we propose to represent a surface as the zero-level set of a signed distance function (SDF) and develop a new volume rendering method to train a neural SDF representation. We observe that the conventional volume rendering method causes inherent geometric errors (i.e. bias) for surface reconstruction, and therefore propose a new formulation that is free of bias in the first order of approximation, thus leading to more accurate surface reconstruction even without the mask supervision. Experiments on the DTU dataset and the BlendedMVS dataset show that NeuS outperforms the state-of-the-arts in high-quality surface reconstruction, especially for objects and scenes with complex structures and self-occlusion.

1 Introduction

Neural implicit surface reconstruction offers high-quality modeling of complex objects, but surface rendering can struggle with abrupt depth changes and occlusions. NeuS combines SDF-based surface representation with volume rendering to improve optimization and reconstruction.

  • IDR’s single-intersection surface rendering provides overly local gradients, causing optimization failures near abrupt depth changes and occlusions.Its reconstruction can fail near object edges and require masks as supervision for convergence.
  • Volume rendering samples multiple points along each ray, allowing near and far surfaces to contribute gradient signals during optimization.
  • A bamboo-planter example illustrates NeuS handling top occlusions with better reconstruction quality than state-of-the-art methods.
  • NeuS represents surfaces with neural SDFs and introduces a novel volume rendering scheme designed to retain accurate geometry and robust training.The method targets complex structures and severe occlusions while avoiding the bias introduced by standard SDF-associated volume density.

2 Related Works

Prior reconstruction methods trade off correspondence dependence, voxel resolution, representation flexibility, and rendering behavior. Neural implicit methods offer continuous high-resolution representations, while volume rendering handles depth changes but weakly constrains surfaces.

  • Classical Multi-view Surface and Volumetric Reconstruction: Classical point- and surface-based methods estimate per-pixel depth from photometric consistency before fusing depth maps into global point clouds.
  • Classical Multi-view Surface and Volumetric Reconstruction: Classical correspondence matching can produce artifacts and missing parts on textureless objects, while voxel-based volumetric methods are limited by achievable resolution.
  • Neural Implicit Representation: Neural implicit representations encode geometry continuously and can achieve high spatial resolution.
  • Rendering-based Methods: Surface-rendering methods assume ray color depends on an intersection with scene geometry, whereas volume-rendering methods aggregate sampled-point colors along rays.
  • Rendering-based Methods: Volume rendering handles sudden depth changes and synthesizes high-quality images, but density-based fields provide insufficient constraints for extracting high-fidelity surfaces.

3 Method

NeuS represents surfaces as zero-level sets of neural signed distance functions and trains them through a volume-rendering scheme designed to be both occlusion-aware and unbiased to first order. Its rendering design addresses the limitations of naïve density conversion while supporting robust learning from 2D images.

  • Scene representation: NeuS represents the object surface as the zero-level set of a neural signed distance function and jointly models geometry and appearance with MLPs.The SDF maps spatial positions to signed distances, while a color field maps positions and viewing directions to colors.
  • Rendering procedure: The method introduces S-density, a logistic density distribution derived from the SDF, to connect implicit geometry with volume rendering.Its inverse standard deviation is trainable and approaches zero as training converges.
  • Rendering procedure: NeuS accumulates colors along camera rays using weights assigned to sampled points, producing rendered pixel colors for training against input images.The rendered color depends on the ray, the point weights, and the viewing-direction-conditioned color field.
  • Rendering procedure: The desired weight function peaks at the zero-level-set intersection and gives greater weight to nearer points when multiple points share the same SDF value.These requirements make the rendering both unbiased with respect to the surface location and occlusion-aware.
  • Rendering procedure: The naïve standard-density formulation is occlusion-aware but biased because its weight maximum occurs before the ray reaches the actual surface intersection.This displacement introduces inherent geometric errors in reconstructed surfaces.
  • Rendering procedure: NeuS defines a new opaque density and weight construction that combines occlusion awareness with first-order unbiasedness, including rays intersecting multiple surfaces.The construction is derived first for a single planar intersection and then generalized to multiple intersections; a theorem establishes first-order unbiasedness.

4 Experiments

NeuS is evaluated against established reconstruction methods on DTU and BlendedMVS, with mask and mask-free settings, qualitative comparisons, and ablations. It achieves strong reconstruction quality on abrupt depth changes and thin structures, while the full weight construction improves over simpler alternatives.

  • Datasets and baselines: Experiments use 15 DTU scenes and additional challenging scenes, evaluated with and without foreground masks against IDR, NeRF, and other baselines.The DTU scenes include varied materials, appearance, and geometry, including non-Lambertian surfaces and thin structures.
  • Quantitative comparisons: NeuS outperforms baseline methods on DTU Chamfer distance both with and without mask supervision.The comparisons use the same Chamfer-distance evaluation protocol as UNISURF and IDR.
  • Qualitative comparisons: NeuS reconstructs surfaces more faithfully than baselines without masks and handles abrupt depth changes better than UNISURF in qualitative comparisons.With masks, IDR struggles with thin metal parts and sudden depth changes, while NeRF produces noisy meshes.
  • Ablation studies: The full weight construction performs better than the naive solution in Chamfer distance, while direct construction produces severe artifacts.The naive solution is worse because it introduces bias into surface reconstruction.
  • Ablation studies: Removing Eikonal regularization or geometric initialization leaves Chamfer distance on par with the full model but prevents correct signed-distance-function outputs.The SDF prediction error is reflected by MAE between predicted and ground-truth SDF values.
  • Thin structures: NeuS accurately reconstructs thin structures, especially edges with abrupt depth changes, and also handles scenes combining thin structures with other content.The evaluation uses two challenging thin objects with 32 input images.

5 Conclusion

The conclusion presents NeuS as a neural-SDF volume-rendering method that delivers high-quality reconstruction, including for severe occlusions and complex structures, while noting textureless-object degradation and computational demands.

  • NeuS represents surfaces as neural SDFs and trains the implicit representation with a new volume-rendering method.
  • NeuS produces high-quality reconstruction and outperforms state-of-the-art methods qualitatively and quantitatively, including on severe occlusions and complex structures.
  • Performance degrades for textureless objects, and the method uses one shared scale parameter across spatial locations.
  • Training requires a large amount of computational resources, consistent with the method's learning-based design.

Acknowlegements

The authors acknowledge contributors, funding support, and the HKU GPU Farm for computational resources.

  • Michael Oechsle provided UNISURF results used in the comparisons.
  • Christian Theobalt and Lingjie Liu received support from ERC Consolidator Grant 770784 and a Lise Meitner Postdoctoral Fellowship, respectively.
  • The HKU GPU Farm mainly provided the computational resources.

A Derivation for Computing Opacity αi

The appendix derives the discrete opacity α_i used by NeuS from the SDF-induced density, treating intervals where rays enter or exit a surface separately.

  • The derivation starts from the opaque density function ρ(t) and computes the discrete opacity α_i for each sampled interval.
  • Entering the surface: For intervals where a ray enters a surface, the signed distance decreases and −(∇f(p(t)) · v) remains positive.
  • Opacity computation: The integral term is evaluated using a constant C to obtain the discrete opacity expression.
  • Exiting the surface: For intervals where a ray exits a surface, the signed distance increases, the density becomes zero, and the derivation concludes.

B.1 Proof of Unbiased Property of Our Solution

The proof shows that NeuS’s weight function reaches its local maximum at the surface intersection under a first-order SDF approximation, unlike the naive formulation.

  • B.1 Proof of Unbiased Property of Our Solution: The S-density uses the logistic density, the derivative of a sigmoid, with positive values around the surface.The proof relies on positivity of the density and sigmoid terms.
  • B.1 Proof of Unbiased Property of Our Solution: Under a locally planar SDF approximation, the gradient is the outward unit normal and the view-direction term is constant.The approximation assumes a sufficiently small planar patch near the surface intersection.
  • B.1 Proof of Unbiased Property of Our Solution: Because the logistic density is unimodal and peaks at zero, NeuS’s weight function attains its local maximum when f(p(t)) = 0.The zero level set is the surface intersection.
  • B.1 Proof of Unbiased Property of Our Solution: The conclusion remains valid when the camera ray intersects multiple surfaces, because the proof assumes no intervening surface between the camera and sample point.This establishes the result locally without requiring a single surface intersection along the entire ray.
  • B.1 Proof of Unbiased Property of Our Solution: The naive volume-rendering weight is biased because its local maximum does not occur at the surface point.Its opacity is set directly to the S-density, producing an inherent geometric error.

C Second-order Bias Analysis

The second-order analysis compares NeuS with the naive solution and finds faster convergence of the local-maximum position error for NeuS, alongside rendering and reconstruction evaluations.

  • C Second-order Bias Analysis: The analysis assumes that the SDF decreases monotonically along the ray and that its second derivative is fixed near the surface intersection.The local maximum is written as t̄ = t* + ∆t.
  • C Second-order Bias Analysis: The local-maximum error is O(s^-2) for NeuS versus O(s^-1) for the naive solution.Here 1/s is the standard deviation of the S-density.
  • C Second-order Bias Analysis: NeuS therefore has quadratic convergence, while the naive solution has linear convergence as the S-density standard deviation approaches zero.The comparison is based on numerical solutions for different fixed values of µ and τ.
  • C Second-order Bias Analysis: NeuS renders 1600x1200 images in about 320 seconds with default volume rendering and about 60 seconds with sphere-tracing-guided sampling on one Nvidia 2080Ti GPU.The faster strategy samples only regions near surfaces after sphere tracing.
  • C Second-order Bias Analysis: On DTU novel-view synthesis without mask supervision, NeuS achieves performance comparable to NeRF in PSNR and SSIM.The evaluation holds out 10% of DTU images for testing.
  • C Second-order Bias Analysis: The full model predicts a more accurate SDF than variants without Eikonal regularization or geometric initialization, while random initialization produces axis-aligned artifacts.The reported comparison concerns signed-distance-field quality and initialization behavior.
  • C Second-order Bias Analysis: During BlendedMVS training, reconstructed surfaces become sharper while the trainable S-density standard deviation decreases.The optimization automatically reduces the standard deviation as training proceeds.
  • C Second-order Bias Analysis: NeuS fails to correctly reconstruct textureless regions in a metal-rabbit case because those regions are ambiguous for neural rendering.This is identified as a failure case rather than a general performance characterization.

E.6 Additional Results

Additional results compare NeuS with baseline methods on DTU and BlendedMVS, including settings with and without mask supervision, and show training-progress visualizations.

  • E.6 Additional Results: A failure reconstruction case contains textureless regions.Figure 16 identifies the failure case visually.
  • E.6 Additional Results: The additional experiments report qualitative comparisons with baseline methods on DTU and BlendedMVS under both mask-supervised and mask-free settings.Figure 17 covers both settings, while Figure 15 focuses on the mask-free setting.
  • E.6 Additional Results: The Durian reconstruction becomes sharper across training iterations, accompanied by a changing trainable standard deviation.The progression is shown in Figure 14.
  • E.6 Additional Results: The additional figures present rendered comparisons using reference images, NeuS results, and NeRF results.The displayed comparison labels are Reference Image, Ours, and NeRF.
Loading 2106.10689v3…