PaperlayerView exact revision on arXiv

Source-linked AI summary

MonoSDF: Exploring Monocular Geometric Cues for Neural Implicit Surface Reconstruction

Zehao Yu, Songyou Peng, Michael Niemeyer, Torsten Sattler, Andreas Geiger

arXiv:2206.00665v2cs.CV

TL;DR

Neural implicit reconstruction struggles with sparse views and textureless regions, where RGB-only supervision leaves geometry underconstrained. MonoSDF adds monocular depth and normal cues to optimization, improving reconstruction quality and efficiency across representations.

  • Problem

    Neural implicit reconstruction struggles with sparse views and textureless regions, where RGB-only supervision leaves scene geometry underconstrained.

    Source: §1 · p. 1

  • Method

    MonoSDF adds monocularly predicted depth and surface-normal cues as supervision alongside RGB reconstruction when optimizing neural implicit surfaces.

    Source: §1 · p. 2

  • Results

    Monocular geometric priors improve reconstruction quality, efficiency, and scalability across neural implicit representations, with faster convergence.

    Source: §1 · p. 3, §5 · p. 10

  • Takeaways & Limitations

    With monocular cues, a simple MLP performs best overall, while converging more slowly than grid-based representations.

    Source: §5 · p. 10

Abstract

from arXiv · show

In recent years, neural implicit surface reconstruction methods have become popular for multi-view 3D reconstruction. In contrast to traditional multi-view stereo methods, these approaches tend to produce smoother and more complete reconstructions due to the inductive smoothness bias of neural networks. State-of-the-art neural implicit methods allow for high-quality reconstructions of simple scenes from many input views. Yet, their performance drops significantly for larger and more complex scenes and scenes captured from sparse viewpoints. This is caused primarily by the inherent ambiguity in the RGB reconstruction loss that does not provide enough constraints, in particular in less-observed and textureless areas. Motivated by recent advances in the area of monocular geometry prediction, we systematically explore the utility these cues provide for improving neural implicit surface reconstruction. We demonstrate that depth and normal cues, predicted by general-purpose monocular estimators, significantly improve reconstruction quality and optimization time. Further, we analyse and investigate multiple design choices for representing neural implicit surfaces, ranging from monolithic MLP models over single-grid to multi-resolution grid representations. We observe that geometric monocular priors improve performance both for small-scale single-object as well as large-scale multi-object scenes, independent of the choice of representation.

1 Introduction

MonoSDF adds monocular depth and normal cues to multi-view neural implicit surface reconstruction, addressing limitations in sparse-view and textureless settings. The paper also studies MLP and grid representations and reports that geometric priors improve reconstruction across them.

1 Introduction

  • RGB-only reconstruction is underconstrained, and current neural implicit methods struggle with limited input views and large textureless regions.The introduction identifies these settings in DTU, ScanNet, and Tanks & Temples scenes.Source: §1 · p. 1
  • MonoSDF uses per-image depth and surface-normal predictions alongside RGB loss; normals help in textureless areas, while photometric cues provide globally accurate geometry in textured regions.The paper attributes the gains to complementary recognition cues from monocular networks and photometric consistency cues from multi-view reconstruction.Source: §1 · pp. 2–3
  • MLPs provide global smoothness but are computationally expensive, whereas grid representations are more efficient through locality but noisier in sparse-view or less-observed areas.The study covers large MLPs, dense SDF grids, single feature grids, and multi-resolution feature grids.Source: §1 · pp. 2–3
  • Monocular geometric priors improve reconstruction across representations and settings, with faster convergence and gains in textureless or less-observed areas.Figure 1 illustrates improved reconstructions and faster optimization when geometric cues are incorporated.Source: §1 · pp. 2–3, Figure 1 · p. 2
  • The paper evaluates object-, room-, and large-scale indoor reconstruction on DTU, Replica [67], ScanNet, and Tanks and Temples.The experiments span multiple challenging datasets and scene scales.Source: §1 · p. 3

2 Related Work

The section situates neural implicit reconstruction among scene representations, multi-view methods, and prior-guided approaches, then distinguishes this work’s use of monocular depth and normal cues across architectures.

Architectures for Neural Implicit Scene Representations

  • Single-MLP fields [9] [42] [50] offer object-level quality but scale poorly; grid-hybrid designs [10] [41] [45] [54] [69] capture detail faster at greater memory cost.The paper systematically compares four architectural design choices for implicit surface reconstruction.Source: §2 · p. 3

3D Reconstruction from Multi-view Images

  • Neural implicit methods [39] [48] avoid discretization but require accurate masks; volume-rendered methods [49] [76] remove that need yet struggle in large, textureless scenes.Classic MVS methods use feature matching for depth or voxel shapes, while learning-based methods replace components such as feature matching, depth fusion, or depth inference.Source: §2 · p. 3

Incorporating Priors into Neural Scene Representations

  • This work uses monocular depth and normal priors [17] for full-scene reconstruction, avoiding Manhattan-world assumptions [11] and extending beyond NeuRIS [75] across MLP-to-grid representations.Manhattan-SDF [21] uses dense COLMAP MVS depth and Manhattan-world priors, and requires semantic segmentation to determine where to apply the assumption; NeuRIS [75] uses normal priors for indoor reconstruction.Source: §2 · p. 3

3 Method

MonoSDF recovers scene geometry from multiple posed images by optimizing neural implicit representations with monocular geometric cues. The method covers SDF representations, differentiable volume rendering, predicted depth and normal cues, and joint reconstruction losses.

3.1 Implicit Scene Representations

  • Scene geometry is an SDF represented as a dense learnable grid, a positional-encoding MLP, or an MLP conditioned on single- or multi-resolution feature grids.A separate network predicts RGB color from a 3D point and viewing direction.Source: §3.1 · pp. 4–5

3.2 Volume Rendering of Implicit Surfaces

  • For differentiable rendering, rays are sampled through each pixel and their predicted SDF values and colors are integrated to render color, expected depth, and surface normals.The SDF-to-volume-rendering transform uses a learnable parameter β.Source: §3.2 · p. 5

3.3 Exploiting Monocular Geometric Cues

  • Pretrained Omnidata predicts relative depth maps and normal maps for each input image; depth supplies semi-local information, while normals capture local geometric detail, making the cues complementary.Because monocular depth has uncertain absolute scale, it is treated as a relative cue.Source: §3.3 · p. 6

3.4 Optimization

  • Optimization combines RGB reconstruction and Eikonal regularization with depth consistency after per-batch scale-and-shift alignment, plus normal consistency using angular and L1 losses.The scale and shift are estimated per batch by least squares because predicted depth maps can differ in scale and shift.Source: §3.4 · p. 6

4 Experiments

The experiments assess scene representations and monocular geometric cues on Replica, compare methods on real-world indoor scenes, and evaluate object reconstruction with sparse and dense views. They examine reconstruction quality and optimization time across these settings.

4.1 Ablation Study

  • On Replica, multi-resolution feature grids are the best-performing grid representation; a single MLP gives decent but over-smoothed results, while a dense SDF grid performs significantly worse.Replacing independent grid-cell SDF values with latent codes and an MLP readout boosts quality; multi-resolution grids further improve it.Source: Table 1 · p. 7, §4.1 · p. 7
  • On Replica, either monocular cue improves both representations, their combination performs best, and adding cues speeds convergence; grids converge faster than the MLP.After 10K iterations, both representations outperform their converged no-cue models, and cue-integration overhead is small. With cues, differences between representations become negligible; the paper also reports similar cue behavior for the other two representations in supplementary material.Source: Table 2 · p. 8, Figure 4 · p. 8, §4.1 · p. 8

4.2 Real-world Large-scale Scene Reconstruction

  • On ScanNet, the MLP variant outperforms all baselines; on Tanks and Temples, monocular cues boost VolSDF to reasonable results on a large-scale indoor scene.The ScanNet MLP is more robust to motion blur and noisy camera poses than grids, whose local geometry updates are harmed by that noise. The paper describes MonoSDF as the first neural implicit model achieving reasonable results on such a large-scale indoor scene.Source: §4.2 · pp. 8–9, Table 3 · p. 9

4.3 Object-level Reconstruction from Sparse Views

  • With only three DTU views, monocular cues significantly improve both representations, but the MLP performs better than grids, which update locally.Without cues, neither representation works well. TSDF fusion of predicted depths struggles with meaningful details due to cue inconsistencies, while classic MVS is quantitatively strong but produces incomplete reconstructions from three images.Source: §4.3 · p. 9, Table 4 · p. 9

4.4 Object-level Reconstruction from Dense Views

  • With dense DTU views, the MLP achieves quality similar to prior methods, while multi-resolution grids outperform prior work by a large margin.The authors attribute the grids’ performance to their expressiveness and the continued effectiveness of monocular cues in suppressing noise, enabling smooth, detailed surfaces.Source: §4.4 · pp. 9–10, Table 5 · p. 10

5 Conclusion

MonoSDF integrates monocular depth and normal cues into neural implicit surface optimization, improving reconstruction quality, efficiency, and scalability across representations. The conclusion highlights representation tradeoffs, dependence on cue quality, and directions for extending the approach.

5 Conclusion

  • Monocular geometric cues improve reconstruction quality, efficiency, and scalability across a variety of neural implicit representations.MonoSDF systematically explores incorporating monocular cues into optimization from multiview images.Source: §5 · p. 10
  • With monocular cues, a simple MLP performs best overall but converges more slowly, while multi-resolution grids converge fast and capture details but are less robust to noise and ambiguity.The conclusion notes that MLPs can represent complex scenes.Source: §5 · p. 10
  • Reconstruction quality depends on monocular cue quality, making filtering strategies for predictor failures a promising direction.The demonstrated reconstruction improvements use depth and normal cues.Source: §5 · p. 10
  • Future directions include exploring occlusion edges, planes, or curvature [17], using higher-resolution cues beyond Omnidata’s 384 × 384-pixel output [17], and jointly optimizing scene representations and camera parameters.Joint optimization is noted as especially relevant to multi-resolution grids for handling noisy camera poses.Source: §5 · p. 10

Checklist … A.3 Additional Details

The checklist reports that the paper’s claims reflect its scope and that limitations are described; supplementary material gives implementation details, including architecture, depth alignment, and optimization settings.

Checklist

  • The authors report that the paper’s claims accurately reflect its contributions and scope, discuss limitations, and cover potential negative societal impacts in supplementary material.They also report conforming to ethics review guidelines.Source: Checklist · p. 15
  • The authors report releasing code and data, specifying training details and computational resources, but not reporting error bars.They state that computational-resource details appear in supplementary material.Source: Checklist · p. 15

A Implementation Details

  • Supplementary Section A covers four scene-representation architectures, Multi-Res. Grid details, depth-loss computation, parameterization, optimization, and evaluation metrics.The implementation discussion is organized across Sections A.1–A.4.Source: A Implementation Details · p. 16, §A.1 · p. 16, §A.2 · p. 16

A.1 Architectures

  • Multi-Res. Grids use geometrically spaced feature levels and a spatial hash at finer levels, with Rmin = 16, Rmax = 2048, L = 16, F = 2, and T = 219.Coarse levels are stored densely; the listed settings are used in all experiments.Source: §A.1 · p. 16

A.2 Depth Consistency Loss

  • The scale-invariant depth loss aligns rendered expected depth to monocular depth by solving for scale and shift with least squares, re-estimated per iteration on rays from one image.The per-iteration estimate accounts for monocular depth maps differing in scale and shift and for changing scene geometry.Source: §A.2 · pp. 16–17

A.3 Additional Details

  • The MLP uses eight 256-unit hidden layers; grid SDF and all color networks use two 256-unit layers, while monocular cues are predicted from 384 × 384 images.Models are optimized for 200k iterations, taking about 6 hours for Multi-Res. Grids and 11 hours for MLP on one NVIDIA RTX3090 GPU.Source: §A.3 · p. 17

A.4 Evaluation Metrics

Evaluation is dataset-specific: DTU uses Accuracy, Completeness, and their mean, Chamfer Distance; Replica and ScanNet add thresholded metrics, while Tanks and Temples uses the evaluation server’s F-score.

A.4 Evaluation Metrics

  • For DTU, Accuracy and Completeness measure mean distances in opposite directions, and Chamfer Distance is their mean.The metrics follow the official evaluation protocol; a Python script computes them for efficiency.Source: §A.4 · p. 18, Table 6 · p. 18
  • For Replica [67] and ScanNet, evaluation reports Accuracy, Completeness, Chamfer Distance, Precision, Recall, and F-score at a 5 cm threshold, following [21] [68].These metrics are defined in Table 6.Source: §A.4 · p. 18, Table 6 · p. 18
  • Replica also reports Normal Consistency because near-perfect ground truth is available, following [21] [42] [53] [54] [68].The metric is defined in Table 6.Source: §A.4 · p. 18, Table 6 · p. 18
  • For Tanks and Temples, reconstructions are submitted to the official evaluation server, which provides the reported F-score.Source: §A.4 · p. 18

B Ablation

The ablations examine monocular geometric cues across scene representations, architecture configurations, input-view counts, and cue predictors, and analyze optimization time. The reported results show improved reconstruction quality across architectures and with more input views.

B.1, B.4, B.5

  • The ablation suite also covers scene representations, cue predictors, and optimization time, alongside architecture and input-view configurations.The section outline assigns these studies to Sections B.1, B.4, and B.5, respectively.Source: B Ablation · p. 18

B.2

  • Monocular geometric cues significantly improve Replica reconstruction quality across all tested architectures.The table caption identifies the scope as an ablation of monocular geometric cues on Replica.Source: Table 7 · p. 19

B.3

  • On Replica, using more input views for training improves reconstruction quality; adding monocular geometric cues also improves quality, and with only 10 input views, cues significantly improve results.Source: Figure 6 · p. 19

B.1 Ablation of Different Cues · B.2 Ablation of Different Architecture Configurations

Monocular geometric cues improve reconstruction across scene representations and architecture configurations. Depth and normal cues work complementarily, while similar gains on Omnidata train and test splits indicate the predictor did not overfit to training data.

B.1 Ablation of Different Cues

  • Geometric cues significantly improve reconstruction quality across all four scene representations, with depth and normal cues together producing the best results.The study evaluates these cues on Replica using Omnidata train/test splits.Source: §B.1 · p. 19
  • Depth and normal cues are complementary: using both yields the best reconstruction results.The cue ablation compares the different geometric cues across scene representations.Source: §B.1 · p. 19
  • Reconstruction quality and gains from geometric cues are similar on the Omnidata train and test splits, indicating no predictor overfitting to training data.Replica is part of Omnidata’s training set and accounts for 0.46% of its training data [17].Source: §B.1 · p. 19

B.2 Ablation of Different Architecture Configurations

  • With different network configurations, monocular geometric cues improve reconstruction quality and convergence speed.The configurations include MLPs with different layer counts and multi-resolution feature grids with different hash-table sizes.Source: §B.2 · p. 20, Figure 7 · p. 20
  • The architecture ablation varies MLP layer counts and multi-resolution feature-grid hash-table sizes to evaluate different model capacities.The learnable parameter counts for these configurations are listed in Table 8.Source: §B.2 · p. 20, Table 8 · p. 20
  • The architecture experiments track performance over optimization processes for the tested configurations.Figure 7 presents the optimization processes, and Table 8 lists the corresponding learnable-parameter counts.Source: §B.2 · p. 20, Figure 7 · p. 20, Table 8 · p. 20

B.3 Ablation of Different Numbers of Input Views … C.2 Tanks and Temples

Monocular geometric cues improve reconstruction across input-view counts and predictor choices, with rapid convergence and stronger qualitative and quantitative results reported for ScanNet and Tanks and Temples. The additional results section also covers DTU.

B.3 Ablation of Different Numbers of Input Views

  • Monocular geometric cues consistently improve reconstruction across different numbers of input views.The experiments vary both input-image count and the use of monocular geometric cues.Source: §B.3 · p. 20

B.4 Ablation of Different Monocular Cues Predictors

  • Omnidata predictors give the best results among the tested depth and normal predictors, while self-supervised depth degrades performance.The authors attribute the self-supervised result to weaker prediction quality and training with an RGB loss, which retains the under-constrained geometry problem.Source: §B.4 · p. 21, Table 9 · p. 21

B.5 Optimization Time

  • With monocular geometric cues, Multi-Res. Grids at 5k iterations outperform converged models without cues at 200k iterations, implying a 40× speedup (5k vs. 200k).Cue prediction runs at 36 FPS on an NVIDIA RTX3090 with batch size one; predicting depth and normal maps for 464 ScanNet images takes less than 26 seconds. For the MLP variant, volume rendering depth and normals adds only 0.0002% of MLP inference time.Source: §B.5 · p. 21

C Additional Results

  • Additional quantitative and qualitative results cover ScanNet, Tanks and Temples, and DTU.The supplied passages give further ScanNet and Tanks and Temples results; they do not specify DTU findings.Source: C Additional Results · p. 21

C.1 ScanNet

  • On ScanNet, the MLP architecture outperforms previous state-of-the-art methods and is more robust to motion blur and partially noisy camera poses, also surpassing concurrent work [75].The paper attributes the MLP result to its robustness to ScanNet’s image and pose noise; the concurrent work used a filtering mechanism.Source: §C.1 · p. 21, Table 10 · p. 22

C.2 Tanks and Temples

  • On Tanks and Temples, monocular geometric cues significantly improve reconstruction quality, with improvements reported for all scenes.The evaluation reports F-score at 10mm using the official server.Source: §C.2 · p. 22, Table 11 · p. 22
  • Compared with a Vis-MVSNet point-cloud reconstruction pipeline, the paper reports more complete reconstructions from its approach.The MVS pipeline fuses predicted depth maps and uses screened Poisson reconstruction; the paper notes that this post-processing can produce floating artifacts and bloated areas in less-observed regions.Source: §C.2 · p. 22

C.3 Preliminary Results of Using High-resolution Monocular Cues

The section presents a proof-of-concept pipeline for generating and integrating higher-resolution monocular depth and normal cues. The authors report that these cues contain finer details and significantly improve reconstruction results.

C.3 Preliminary Results of Using High-resolution Monocular Cues

  • Higher-resolution cues are generated by predicting depth and normals for overlapping image crops, then merging predictions after aligning depth maps and solving normal rotations.The merged cues are used to train the reconstruction model.Source: §C.3 · pp. 22–23
  • The resulting high-resolution monocular cues contain more fine details than low-resolution cues and significantly improve reconstruction results.Other high-resolution depth-generation methods are also possible, such as [43].Source: §C.3 · p. 23

C.4 DTU · D Societal Impact

On DTU, monocular geometric cues improve reconstruction for both MLP and multi-resolution grids and improve three-view novel-view synthesis. The paper also identifies computational, privacy, and misuse risks.

C.4 DTU

  • With three input views, monocular cues improve DTU reconstructions for both MLP and Multi-Res. Grids, enabling high-quality meshes where both fail without cues.Qualitative results also show improved reconstruction quality for both architectures.Source: §C.4 · p. 23, Figure 17 · p. 34
  • Annealing the monocular depth and normal loss weights λ2 and λ3 to 0 over the first 200 optimization epochs improves DTU reconstruction quality.The predictor is imperfect, and the qualitative ablation verifies the importance of weight annealing.Source: §C.4 · p. 23, Figure 20 · p. 37
  • A three-view DTU failure duplicates objects in front of camera frustums because scale-ambiguous monocular depth does not ensure multi-view consistency.The paper suggests explicit multi-view constraints, such as sparse COLMAP point clouds, as additional supervision [14].Source: §C.4 · p. 23, Figure 21 · p. 37
  • 23.64 for MLP w/ cues exceeds 17.65 for MLP on the reported DTU novel-view synthesis results.The comparison uses three input views; the supplied passages do not identify the metric label.Source: §C.4 · p. 23, Table 13

D Societal Impact

  • The method may affect society through predictor training's computational demands and climate impact, scene-reconstruction privacy concerns, and potential malicious uses of accurate geometry.The paper notes that its general-purpose monocular geometric predictor requires large amounts of training data and computational resources.Source: D Societal Impact · p. 24

C.4 DTU

  • With all input views, Multi-Res. feature grids reconstruct more detailed surfaces than prior methods, while the MLP achieves similar results.The paper also reports that its multi-resolution-grid approach produces more accurate reconstructions than state-of-the-art methods.Source: §C.4 · p. 23, Figure 19 · p. 36

Related papers

Loading 2206.00665v2…