PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Signed Distance Field for Multi-view Surface Reconstruction

Jingyang Zhang, Yao Yao, Long Quan

arXiv:2108.09964v1cs.CV

TL;DR

Conventional reconstruction can accumulate errors across image-to-point-cloud-to-mesh conversions, while implicit rendering methods may depend on object masks. MVSDF combines SDF geometry and a surface light field with MVS depth supervision, feature consistency, and rendered-image optimization; it reports better geometry accuracy and rendering fidelity than previous methods.

  • Problem

    Conventional reconstruction can accumulate errors across image-to-point-cloud-to-mesh conversions, while implicit-rendering approaches can have geometry-appearance ambiguities and depend on object masks.

    Source: §1 · pp. 1–2

  • Method

    MVSDF represents geometry with an SDF and appearance with a surface light field, using MVS depth supervision and multi-view feature consistency alongside render-loss optimization.

    Source: §1 · p. 2

  • Results

    MVSDF outperforms previous methods in geometry accuracy and rendering fidelity, with evaluations on multiple datasets showing recovery of complex geometry without object masks.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    The reported results show that MVSDF can reconstruct complex scene geometry without input object masks.

    Source: §1 · p. 2

    IDR's manually labeled masks are impractical for many images and can be ill-defined for real-world inputs.

    Source: §1 · p. 2

Abstract

from arXiv · show

Recent works on implicit neural representations have shown promising results for multi-view surface reconstruction. However, most approaches are limited to relatively simple geometries and usually require clean object masks for reconstructing complex and concave objects. In this work, we introduce a novel neural surface reconstruction framework that leverages the knowledge of stereo matching and feature consistency to optimize the implicit surface representation. More specifically, we apply a signed distance field (SDF) and a surface light field to represent the scene geometry and appearance respectively. The SDF is directly supervised by geometry from stereo matching, and is refined by optimizing the multi-view feature consistency and the fidelity of rendered images. Our method is able to improve the robustness of geometry estimation and support reconstruction of complex scene topologies. Extensive experiments have been conducted on DTU, EPFL and Tanks and Temples datasets. Compared to previous state-of-the-art methods, our method achieves better mesh reconstruction in wide open scenes without masks as input.

1. Introduction

MVSDF combines implicit surface estimation with MVS depth and feature constraints to reconstruct complex geometry without object masks. The introduction positions this approach against accumulated errors in conventional reconstruction pipelines and mask dependence in implicit rendering methods.

1. Introduction

  • Conventional image-to-point-cloud-to-mesh pipelines can accumulate conversion errors, allowing point-cloud errors to produce incorrect mesh topology that is difficult to recover.Learning-based methods had improved individual MVS and mesh-extraction steps, but end-to-end surface reconstruction from images remained desirable.Source: §1 · p. 1
  • On Family and Horse, MVSDF is reported to recover correct mesh topology with fine details on highly textureless and reflective surfaces.Figure 1 illustrates these Tanks and Temples reconstructions.Source: Figure 1 · p. 1
  • Implicit differentiable rendering offers end-to-end geometry and image synthesis, but geometry-appearance ambiguities motivate IDR's use of object masks, which are difficult to obtain at scale and can be ill-defined on real images.MVSDF aims to improve topology and support complex geometries without input masks.Source: §1 · p. 2
  • MVSDF represents geometry as an SDF and appearance as a surface light field, with MVS depth supervision and multi-view feature consistency refining the surface.The geometry and appearance are jointly optimized through render loss; the paper also reports improved geometry accuracy and preservation of high-fidelity details.Source: §1 · p. 2
  • On DTU, EPFL, and Tanks and Temples, MVSDF recovers complex geometries without object masks, with qualitative and quantitative results.The evaluation also compares mesh reconstruction and view-synthesis quality with classical meshing pipelines and differentiable-rendering networks.Source: §1 · p. 2

2. Related Works

Prior work spans MVS, implicit surface representations, and differentiable rendering, but conversions between depth maps, point clouds, and meshes can lose information. MVSDF combines MVS and feature consistency with an implicit SDF representation to address mask dependence and difficult surface regions.

2. Related Works

  • MVS depth maps are commonly fused into point clouds and converted to meshes, but these conversions can create holes and smooth away surface details.The passage identifies over-filtering of depth maps as a source of holes and point-cloud-to-mesh conversion as a source of smoothing.Source: §2 · p. 2
  • Deep MVS improves depth quality but still struggles to match pixels in textureless or non-Lambertian regions; this work targets those regions with implicit interpolation and view-dependent appearance modeling..Source: §2 · p. 2
  • Implicit neural functions represent continuous surfaces with fixed-sized MLPs and can model complex geometries with arbitrary topologies.Occupancy fields and SDFs are among the representations discussed; meshes are commonly extracted with marching cubes.Source: §2 · p. 2
  • Differentiable-rendering methods estimate geometry by matching rendered and input images, while IDR uses an SDF and surface light field but depends on accurate manually labeled masks.Inaccurate masks can lead to missing or extra mesh surfaces; MVSDF instead introduces MVS and feature-consistency geometry constraints to relax the mask requirement.Source: §2 · p. 2
  • MVSDF samples spatial points for MVS-depth supervision, refines ray-traced surface points with deep feature consistency, then jointly optimizes SDF geometry and a surface light field with render loss.The framework illustration summarizes the method's sequence.Source: Figure 2 · p. 3

3. Method

MVSDF represents geometry with an SDF and appearance with a surface light field, supervising the SDF with MVS depth and refining it through feature and rendered-image consistency. It also learns which surfaces are valid and combines its losses in a staged training schedule.

3.1. Geometry and Appearance Representations

  • The surface is the zero level set of an SDF predicted by an MLP, while a surface light field represents scene appearance.The SDF network maps query locations to distance from the closest surface point.Source: §3.1 · p. 3

3.2. Geometry Supervision

  • MVS depth maps supervise the SDF with approximated signed distances, and mini-batch fusion reduces errors from unreliable single-view observations.The method samples points throughout space and fuses distances from multiple depth maps.Source: §3.2 · pp. 3–4

3.3. Local Geometry Refinement

  • Multi-view deep-feature consistency refines geometry, while rendered-image consistency jointly optimizes geometry and appearance.Feature consistency adds a photo-consistency constraint earlier in the differentiable-rendering pipeline; render loss is more sensitive to local color changes.Source: §3.3 · pp. 4–5

3.4. Valid Surface Indicator

  • A learned indicator marks valid surfaces, and graph-cut trimming removes triangles classified as invalid.The indicator is trained from successfully traced locations and randomly sampled background points.Source: §3.4 · p. 5, Figure 5 · p. 5

3.5. Loss

  • Training first emphasizes depth supervision for topology, then feature consistency for finer structures, and finally render loss for fine-scale detail.The overall objective combines render, feature, depth, Eikonal, and indicator losses.Source: §3.5 · p. 5

4. Experiments

MVSDF combines implicit surface estimation with multi-view stereo, using depth maps, feature consistency, and rendered-image loss to refine geometry. Across the reported benchmarks, it achieves strong reconstruction results, including the best mean Chamfer distance and PSNR on DTU and EPFL, while the ablation study finds benefits from both feature and render losses.

4.1. Implementation

  • Training uses an 8-layer, 512-unit SDF MLP and a 4-layer, 512-unit surface-light-field MLP, with MVS depth maps and image features from a fixed pretrained Vis-MVSNet module.Thin structures receive additional near-surface samples; mesh extraction uses Marching Cubes, followed by graph-cut trimming of extrapolated background surfaces.Source: §4.1 · p. 6

4.2. Benchmark on DTU Dataset

  • 0.88 mean Chamfer distance and 25.92 PSNR are the best results among the compared methods on DTU.The method also produces results without manual masks and has less mesh distortion than IDR.Source: §4.2 · p. 7

4.3. Benchmark on EPFL Dataset

  • 6.61 mean Chamfer distance and 27.01 PSNR are the best results among the compared methods on EPFL.The reconstruction correctly recovers topology where IDR produces inflated surfaces, especially when the wall is included in its input masks.Source: §4.3 · p. 7

4.4. Additional Qualitative Results

  • On Tanks and Temples’ Horse scene, MVSDF produces a complete, accurate mesh and realistic view-dependent rendering where Vis-MVSNet yields an incomplete point cloud and bumpy, noisy results.Source: §4.4 · p. 7

4.5. Ablation Study

  • Both feature loss and render loss improve reconstruction quality, while feature loss is more robust and render loss adds finer detail but can produce erroneous roof geometry.The feature-loss setting refines the surface but remains coarser than the full setting.Source: §4.5 · p. 8, Figure 9 · p. 8

5. Conclusion

MVSDF combines implicit neural surface estimation with MVS networks, supervising geometry with MVS depth maps and refining it through deep feature consistency and rendered-image loss. Evaluations across datasets report improved geometry accuracy and rendering fidelity over previous methods.

  • MVSDF combines implicit neural surface estimation with MVS networks, directly supervises geometry with MVS depth maps, and locally refines it through deep feature consistency and rendered-image loss.Source: §5 · p. 8

1. Implementation Details

The supplementary implementation details specify baseline settings, MVSDF training losses, and evaluation procedures. They also describe how reported reconstruction and rendering metrics are computed.

1.1. Baselines

  • Vis-MVSNet uses two source images and 256 depth samples, while IDR is manually masked on EPFL and trained for 10,000 epochs; Colmap and Vis-MVSNet meshes use screened Poisson reconstruction.Source: §1.1 · p. 11

1.2. MVSDF

  • MVSDF uses Vis-MVSNet-matched MVS settings and stage-dependent distance and feature loss weights, with fixed Eikonal, indicator, and render weights.The distance and feature weights change from 1.0 and 0 in the first training sixth, to 0.1 each in the next third, then 0.01 each in the final half; distance-loss downweighting is limited to points near the surface.Source: §1.2 · p. 11

1.3. Evaluations

  • EPFL Chamfer distance averages both point-cloud directions after cropping ground truth by manual masks and excludes distances above 0.8; PSNR is evaluated only within predefined masks.For DTU, Chamfer distance uses the provided MATLAB code; the PSNR masks are the IDR mask for DTU and the intersection of methods’ render masks for the other datasets.Source: §1.3 · p. 11

2. Ablation Study on MVS module

The ablation compares MVS depth maps at different quality settings and finds results similar to the full setting. The MVS module mainly supplies the initial surface topology and can be replaced by other MVS algorithms if needed.

2. Ablation Study on MVS module

  • Lowres downsamples input depth maps to 256 × 192, while filtered uses precomputed, geometrically filtered maps as in [46].The settings represent lower- and higher-accuracy MVS modules, respectively.Source: §2 · p. 11
  • Lowres and filtered depth-map settings generate results similar to the proposed full setting.The comparison is on DTU; Table 2 reports quantitative results on Tanks and Temples.Source: §2 · pp. 11–12, Table 2 · p. 12
  • The MVS module mainly recovers the correct initial surface topology, and other MVS algorithms can be substituted if necessary.Source: §2 · p. 12

3. Tanks and Temples Dataset

On Francis, M60, and Panther, the method is reported to outperform Colmap and Vis-MVSNet in rendered-image quality.

3. Tanks and Temples Dataset

  • The proposed method consistently outperforms Colmap and Vis-MVSNet in rendered-image quality on Francis, M60, and Panther.The comparison reports PSNR scores in Table 2.Source: §3 · p. 12

4. Additional Qualitative Results

Additional qualitative results are shown across DTU, EPFL, and Tanks and Temples.

4. Additional Qualitative Results

Related papers

Loading 2108.09964v1…