PaperlayerView exact revision on arXiv

Source-linked AI summary

SimpleRecon: 3D Reconstruction Without 3D Convolutions

Mohamed Sayed, John Gibson, Jamie Watson, Victor Prisacariu, Michael Firman, Clément Godard

arXiv:2208.14743v1cs.CV

TL;DR

3D reconstruction methods that rely on costly 3D convolutions are challenging to use in resource-constrained settings. SimpleRecon combines a 2D CNN with metadata-informed cost-volume depth prediction, reporting state-of-the-art depth estimation and 3D reconstruction without expensive 3D convolutions.

  • Problem

    Costly 3D convolutions in depth estimation and reconstruction methods limit their suitability for low-power hardware with constrained compute and memory.

    Source: §1 · p. 1

  • Method

    SimpleRecon combines a carefully designed 2D CNN with a cost volume augmented by keyframe and geometric metadata for informed depth-plane scoring.

    Source: §1 · p. 2, §3.1 · p. 5

  • Results

    The authors report state-of-the-art depth estimations and 3D reconstructions without expensive 3D convolutions.

    Source: §5 · p. 14

  • Takeaways & Limitations

    The method’s combination of metadata and architecture avoids 3D-convolution costs and may enable use in embedded, resource-constrained environments.

    Source: §1 · p. 2

    The authors find the ATLAS mesh evaluation inconsistent: self-comparison does not score zero, and metric performance does not match visual quality or correlate well across methods.

    Source: §4.2 · p. 11

Abstract

from arXiv · show

Traditionally, 3D indoor scene reconstruction from posed images happens in two phases: per-image depth estimation, followed by depth merging and surface reconstruction. Recently, a family of methods have emerged that perform reconstruction directly in final 3D volumetric feature space. While these methods have shown impressive reconstruction results, they rely on expensive 3D convolutional layers, limiting their application in resource-constrained environments. In this work, we instead go back to the traditional route, and show how focusing on high quality multi-view depth prediction leads to highly accurate 3D reconstructions using simple off-the-shelf depth fusion. We propose a simple state-of-the-art multi-view depth estimator with two main contributions: 1) a carefully-designed 2D CNN which utilizes strong image priors alongside a plane-sweep feature volume and geometric losses, combined with 2) the integration of keyframe and geometric metadata into the cost volume which allows informed depth plane scoring. Our method achieves a significant lead over the current state-of-the-art for depth estimation and close or better for 3D reconstruction on ScanNet and 7-Scenes, yet still allows for online real-time low-memory reconstruction. Code, models and results are available at https://nianticlabs.github.io/simplerecon

1 Introduction

SimpleRecon returns to per-view depth estimation and fusion, using a 2D CNN and cost-volume metadata to target accurate depth and competitive 3D reconstruction without expensive 3D convolutions. Its qualitative preview reports improvement over monocular MVS depth methods and a match to volumetric reconstruction results.

1 Introduction

  • Traditional reconstruction estimates per-image depth before fusion and surface extraction, while 3D-convolution methods and some depth estimators are costly in time, memory, or compute.These costs can preclude use on low-power hardware such as smartphones.Source: §1 · pp. 1–2
  • SimpleRecon reports state-of-the-art depth accuracy and competitive 3D reconstruction, with its preview improving on monocular MVS methods [12] and matching volumetric results [64].It uses off-the-shelf TSDF fusion for reconstruction.Source: §1 · p. 2, Figure 1 · p. 2
  • The method combines keyframe and geometric metadata in the cost volume with a carefully designed 2D CNN, and may support embedded, resource-constrained use by avoiding 3D convolutions.The CNN uses image priors, a plane-sweep feature volume, and geometric losses.Source: §1 · p. 2

2 Related Work

Prior work spans calibrated stereo, multi-view depth estimation, and scene reconstruction. SimpleRecon uses metadata-informed depth prediction and efficient TSDF fusion as an alternative to computationally expensive volumetric reasoning.

2.1 Depth from Calibrated Stereo Pairs

  • 4D cost volumes processed with 3D convolutions can be computationally expensive, while reducing features to a 3D cost volume permits 2D convolutions but typically sacrifices depth quality.The paper states that simple tricks and reduction techniques can allow a 3D cost-volume method to outperform existing 4D cost-volume methods on depth and reconstruction.Source: §2.1 · p. 3

2.2 Multi-view Stereo Depth

  • MVS estimates reference-view depth from source views at arbitrary locations, and prior learning methods use geometric matching and cost volumes, while SimpleRecon adds readily available metadata.Prior approaches use image features, occlusion or motion cues, temporal fusion, and multiple-reference-view aggregation.Source: §2.2 · pp. 3–4

2.3 3D Scene Reconstruction from Posed Views

  • Classical reconstruction commonly fuses dense per-view depth into surfaces, and TSDF fusion supports real-time mapping from depth maps.Kinect Fusion demonstrated real-time reconstruction using a volumetric TSDF from which a mesh can be extracted.Source: §2.3 · p. 4
  • Deep volumetric methods produce high-quality reconstructions but can be compute- and memory-intensive; SimpleRecon instead predicts high-quality depth for efficient, progressive TSDF fusion without 3D convolutions.Its overview injects metadata into a feature volume, reduces each cell with an MLP, and passes the resulting feature map to a 2D encoder-decoder.Source: §2.3 · pp. 4–5, Figure 2 · p. 5

3 Method

SimpleRecon combines a 2D depth-prediction network with a plane-sweep cost volume enriched by geometric and camera metadata. Its architecture and losses aim for accurate multi-view depth and competitive reconstruction without costly 3D convolutions.

3 Method

  • The model predicts dense reference-view depth from posed images using a plane-sweep cost volume and a 2D convolutional encoder-decoder with image-level features.Training assumes ground-truth depth maps aligned with the RGB images; test-time input includes camera intrinsics and relative poses.Source: §3.1 · p. 5, §3 · p. 5

3.2 Improving the Cost Volume with Metadata

  • The cost volume combines warped image features with geometric metadata, including pose distance, ray information, reprojected depths, and validity masks.An MLP maps each cost-volume feature vector to a scalar interpreted as an initial likelihood for that pixel’s depth plane.Source: §3.2 · pp. 6–7, Metadata motivation — · p. 7
  • Metadata in the cost volume significantly boosts depth accuracy and improves 3D reconstruction quality.Metadata is added explicitly as feature channels and implicitly through feature ordering.Source: Implicit metadata incorporation — · p. 7

3.3 Network Architecture Design

  • A simple 2D encoder-decoder avoids complex temporal fusion; EfficientNetV2 S improves depth accuracy over MnasNet at the cost of more parameters and 10% slower execution.The feature-matching encoder uses the first two ResNet18 blocks; multi-scale image features are connected to the cost-volume encoder.Source: §3.3 · p. 8
  • As additional source frames are added, other methods show diminishing returns, whereas the authors’ method incorporates the added information more effectively and improves performance with up to 8 views.The authors posit that additional per-frame metadata helps the network choose relative feature weights; by contrast, MVDepthNet, MVSNet, ManyDepth and ATLAS give each frame equal weight during an update.Source: §3.3 · p. 8

3.4 Loss

  • Intermediate predictions at lower output scales substantially improve results, and absolute log-depth error produces sharper boundaries and higher fused reconstruction quality than the scale-invariant alternative.The lower-scale depths are nearest-neighbor upsampled before computing the depth loss.Source: §3.4 · p. 9
  • Training combines multi-scale absolute log-depth regression with gradient, normal, and multi-view depth losses.The multi-view loss uses projected reference depth and is applied only at the final output scale; loss weights were selected experimentally on the validation set.Source: §3.4 · p. 9

3.5 Implementation Details

  • The implementation uses EfficientNetV2 S, trains for 100k steps on two A100 GPUs, and predicts depth at half the 512 × 384 input resolution.Training takes 36 hours; keyframes are selected following DeepVideoMVS.Source: §3.5 · p. 10

4 Experiments

Experiments assess depth prediction, fused 3D reconstruction, latency, and ablations, including evaluation on ScanNetv2 and 7-Scenes. Ablations examine architecture, metadata, view count, and the cost volume.

4.1 Depth Estimation

4.2 3D Reconstruction Evaluation

  • The depth-based method beats state-of-the-art depth estimators for fusion by a wide margin and exceeds volumetric methods on some metrics without global refinement.Mesh evaluation follows the TransformerFusion protocol.Source: §4.2 · p. 11, Table 2 · p. 11
  • The ATLAS mesh protocol gives inconsistent scores: self-comparison is nonzero, and metric scores do not align well with visual quality or correlate across methods.The authors provide these scores and further discussion in the supplementary material.Source: §4.2 · p. 11

4.3 3D Reconstruction Latency

  • Although slower than methods such as NeuralRecon per keyframe, SimpleRecon can update the reconstructed volume quickly using online TSDF fusion.Latency is measured per new RGB frame; the comparison omits time waiting for required keyframes.Source: §4.3 · p. 12, Table 3 · p. 12

4.4 Ablations

  • Using 64 feature channels instead of 16 degrades accuracy and is significantly slower; removing the MLP also greatly reduces performance.The baseline uses no MLP and reduces 16 feature channels with a dot product.Source: §4.4 · p. 12
  • Random keyframe ordering harms both models, but access to pose-distance metadata makes the full model less affected.Source: §4.4 · p. 12
  • Ablations show accuracy increasing as the cost-volume MLP receives more information, including features, depth, validity masks, and metadata.All compared models use MLP cost-volume reduction.Source: §4.4 · p. 13
  • Performance improves from 2 to 8 source views, while DeepVideoMVS remains relatively constant beyond three; zeroing the cost volume greatly decreases performance.The ablation indicates that a strong metric-depth estimate from the cost volume is required to resolve scale ambiguity.Source: §4.4 · pp. 13–14

5 Conclusion

SimpleRecon achieves state-of-the-art depth estimations and 3D reconstructions without expensive 3D convolutions. Its metadata-augmented cost volume and architectural choices support high-quality depth-based reconstruction, while leaving room for further refinement.

5 Conclusion

  • SimpleRecon produces state-of-the-art depth estimations and 3D reconstructions without expensive 3D convolutions.Source: §5 · p. 14
  • Metadata boosts scores, and together with architectural design choices leads to state-of-the-art depths.Source: §5 · p. 14
  • The approach remains compatible with 3D convolutions and additional cost-volume or depth refinement when compute is less restricted.Source: §5 · p. 14

Related papers

Loading 2208.14743v1…