PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural RGB-D Surface Reconstruction

Dejan Azinović, Ricardo Martin-Brualla, Dan B Goldman, Matthias Nießner, Justus Thies

arXiv:2104.04532v3cs.CV

TL;DR

NeRF-based volumetric representations reproduce appearance but can fail to produce clean metric surfaces. This paper combines a truncated signed distance surface, volumetric radiance, RGB-D observations, and camera refinement, achieving high-quality reconstructions that outperform traditional and learned RGB-D fusion methods.

  • Problem

    NeRF-based volumetric representations reproduce scene appearance but do not reconstruct clean metric surfaces, motivating room-scale RGB-D surface reconstruction.

  • Method

    The method combines a truncated signed distance surface representation with a volumetric radiance field, RGB-D optimization, and joint pose and camera refinement.

  • Results

    The method achieves high-quality surface reconstructions and outperforms traditional and learned RGB-D fusion methods.

  • Takeaways & Limitations

    Implicit differentiable representations can support high-quality 3D surface reconstruction while retaining volumetric rendering for color.

  • Takeaways & Limitations

    The method runs offline, can miss high-frequency local detail in very large scenes, and is designed only for opaque surfaces.

Abstract

from arXiv · show

Obtaining high-quality 3D reconstructions of room-scale scenes is of paramount importance for upcoming applications in AR or VR. These range from mixed reality applications for teleconferencing, virtual measuring, virtual room planing, to robotic applications. While current volume-based view synthesis methods that use neural radiance fields (NeRFs) show promising results in reproducing the appearance of an object or scene, they do not reconstruct an actual surface. The volumetric representation of the surface based on densities leads to artifacts when a surface is extracted using Marching Cubes, since during optimization, densities are accumulated along the ray and are not used at a single sample point in isolation. Instead of this volumetric representation of the surface, we propose to represent the surface using an implicit function (truncated signed distance function). We show how to incorporate this representation in the NeRF framework, and extend it to use depth measurements from a commodity RGB-D sensor, such as a Kinect. In addition, we propose a pose and camera refinement technique which improves the overall reconstruction quality. In contrast to concurrent work on integrating depth priors in NeRF which concentrates on novel view synthesis, our approach is able to reconstruct high-quality, metrical 3D reconstructions.

1. Introduction

Neural scene representations reproduce novel-view appearance but do not directly yield clean metric meshes. The paper combines dense color and depth with signed-distance geometry and camera refinement to reconstruct room-scale scenes.

  • Color-based volumetric methods synthesize near photo-realistic novel views but do not reconstruct metric, clean meshes.
  • Consumer depth cameras provide broadly accessible range measurements, while RGB observations supply dense color information.
  • The method learns a truncated signed distance field while retaining differentiable volumetric rendering for color reproduction.
  • Joint optimization of the scene representation and camera poses addresses noisy initial poses from depth-based alignment.
  • The method reconstructs color-observed geometry absent from depth maps and reports state-of-the-art quality on synthetic and real ScanNet data.

2. Related Work

Related work spans classical, learned, and coordinate-based reconstruction methods. This paper combines NeRF rendering with an implicit surface representation and explicit depth integration for metric room-scale reconstruction.

  • Classical 3D Reconstruction: Classical reconstruction uses stereo, Structure-from-Motion, or SLAM with patch, volumetric, or mesh representations.
  • Deep Learning for 3D Reconstruction: Deep-learning methods predict depth, learn multiview stereo, complete scans, improve SLAM features, or infer scene structure from images.
  • Neural Radiance Fields: NeRF represents scenes with a coordinate-based model and fixed volumetric rendering to synthesize novel views from color images.
  • This Work: The proposed hybrid representation combines an implicit signed-distance surface with a volumetric radiance field and incorporates depth for robust metric reconstruction.
  • Concurrent Work: Concurrent methods use depth to constrain NeRF sampling or predictions, while NeuS uses signed distance geometry but only RGB inputs.

3. Method

The method combines a TSDF-based implicit surface with volumetric radiance, integrating RGB-D observations while refining camera parameters during optimization. Signed-distance-based rendering concentrates weights near surfaces and supports detailed, artifact-reduced reconstruction.

  • Hybrid Scene Representation: The hybrid representation uses a TSDF for geometry and a volumetric representation for view-dependent radiance.An MLP evaluates arbitrary 3D positions to predict truncated signed distance and radiance values, conditioned on positional and viewing-direction encodings.
  • Signed-Distance Rendering: Rendering weights are computed from signed distances rather than density probabilities, with a peak at the surface’s zero-crossing.The truncation distance controls how quickly weights decay away from the surface, and samples beyond the first truncation region are suppressed.
  • Signed-Distance Rendering: Signed-distance rendering predicts hard occupied-free-space boundaries and reduces the noisy free-space artifacts associated with density-based models.The differentiable weighting scheme also enables depth inference through a photometric loss, although it is not physically based.
  • Network Architecture: The radiance MLP models view-dependent effects, while per-frame latent codes compensate for exposure and white-balance variation.The shape MLP predicts distance to the nearest surface; the radiance MLP uses encoded viewing direction and intermediate shape features.
  • Pose and Camera Refinement: Camera poses and a shared image-plane deformation field are refined jointly with the scene representation to address pose, distortion, and intrinsic-parameter inaccuracies.The poses are initialized with BundleFusion, and the deformation residual shifts rays before 3D unprojection.
  • Optimization: Optimization samples RGB-D pixels and ray points, combining RGB, free-space, and truncated signed-distance objectives over batches.The signed-distance loss compares predicted and depth-sensor-observed distances; experiments use tr = 5 cm, 1024 rays per iteration, and 2 × 10^5 iterations.

4. Results

On synthetic and ScanNet data, the method produces high-quality reconstructions by combining signed-distance geometry, RGB-D observations, and camera refinement. Ablations show that photometric supervision recovers missing geometry, while pose and image-plane refinement reduce alignment artifacts; the method is offline and limited for very large or transparent scenes.

  • Results on real data: On ScanNet, joint optimization of the scene representation and camera poses substantially reduces misalignment artifacts compared with BundleFusion reconstructions.The comparison covers room-scale scenes captured with noisy RGB-D measurements.
  • Quantitative evaluation: The method achieves the lowest Chamfer distance and highest IoU, normal consistency, and F-score among the evaluated reconstruction methods.The evaluation uses 10 synthetic scenes, with point-cloud metrics and voxelized IoU.
  • Effect of the photometric term: Photometric supervision reconstructs geometry missing from depth measurements by leveraging RGB observations.This effect is demonstrated for missing table legs and a meshed basket.
  • Effect of pose refinement: Camera pose optimization improves reconstruction quality by reducing geometry misalignment artifacts both quantitatively and qualitatively.The method jointly optimizes rotation and translation parameters initialized from BundleFusion.
  • Limitations and future work: The method runs offline for around 9 hours per 2 × 10^5 iterations and can miss high-frequency local detail in very large scenes; it handles only opaque surfaces.A global scene-specific MLP creates the large-scene detail limitation.

5. Conclusion

The paper presents a hybrid implicit-surface and volumetric-radiance method for RGB-D surface reconstruction, achieving high-quality reconstructions that outperform traditional and learned RGB-D fusion methods.

  • The method combines an implicit surface function with volumetric radiance representation for RGB-D sequence reconstruction.This incorporates depth observations while retaining differentiable volumetric rendering.
  • The resulting reconstructions are high quality and outperform traditional and learned RGB-D fusion methods.
  • The appendix provides per-scene quantitative evaluations, ScanNet and synthetic-data ablations, and optimization parameters for reproducibility.

A. Implementation Details

Implementation uses Adam optimization over randomly sampled rays and multi-scale positional encodings, with scene normalization and an image-plane deformation field for camera calibration errors.

  • The method uses Adam with learning rate 5 × 10^-4 and exponential decay of 10^-1 over 2.5 × 10^5 iterations.
  • Each iteration computes gradients from 1024 randomly chosen rays, using 16 S′f samples per ray.
  • S′c is selected to average one sample every 1.5 cm of ray length, with scenes internally scaled into a [-1, 1]^3 cube.
  • Depending on scene size, convergence takes 9–13 hours on one NVIDIA RTX 3090, with loss weights λ1 = 0.1, λ2 = 10, and λ3 = 6×10^3.
  • The implementation uses 8 positional-encoding bands for point coordinates and 4 bands for view directions.
  • A 6-layer, width-128 MLP deformation field corrects camera pixel-space distortions and inaccurate intrinsic parameters.

B. Per-scene Quantitative Evaluations

The evaluation measures reconstruction quality on visible geometry using standard surface metrics across synthetic and real-scene experiments, with scene-dependent sampling settings and controlled data sources.

  • Reconstruction quality is evaluated using Chamfer distance, IoU, normal consistency, and F-score on surfaces visible in the input color and depth streams.
  • Ray lengths and sample counts vary by scene size, as documented for ScanNet and synthetic experiments.
  • Meshes are subdivided below 1.5 cm edge length, and invisible or occluded triangles are culled using the ground-truth trajectory.
  • The synthetic benchmark contains 10 licensed scenes rendered with color and noisy depth, covering varied sizes, complexity, and materials.
  • BundleFusion supplies the initial camera trajectory for all methods except COLMAP to support fair comparison.
  • The photometric energy term encourages correct depth prediction where the depth sensor captured no measurements.

C. Ablation Studies

Ablations show that signed-distance geometry, photometric supervision, and camera refinement improve reconstruction, while color data recovers geometry absent from depth measurements.

  • Additional real-data results demonstrate advantages from the signed distance field and camera refinement.
  • The photometric term reconstructs missing geometry from RGB observations, including simulated table legs and a meshed basket.
  • The piano stool legs are reconstructed from color data despite being absent from every depth map.
  • Using fewer input frames evaluates how reconstruction quality depends on the amount of RGB-D data.

C.3. Robustness to Noisy Pose Initialization

The method remains robust to substantial camera-pose noise and missing frames, but refinement fails when cameras begin inside geometry. Batch size and truncation width trade reconstruction quality against computational cost.

  • Reconstruction remains robust to camera errors up to 5 cm in position and 5° in orientation.
  • Pose refinement remains robust up to 3 cm and 3° of initialization noise.
  • At σ = 10 cm, cameras can intersect geometry, preventing pose refinement and degrading reconstruction quality.
  • Good reconstruction results persist with as few as 13 frames.
  • Smaller batches increase noise and can miss areas without depth supervision, whereas oversized batches slow optimization and increase GPU memory without improving reconstruction quality.

D. Comparison to RGB-based methods

RGB-only signed-distance methods struggle with large textureless regions in indoor scenes, whereas the RGB-D approach uses depth to reconstruct those regions more effectively.

  • RGB-only methods lack sufficient multi-view constraints to properly reconstruct large textureless regions such as white walls.
  • The proposed method is designed for indoor reconstruction where depth complements insufficient color-based constraints.
  • The comparison in Fig. 12 evaluates NeuS against the proposed method on the ‘morning apartment’ scene.

E. Color Reproduction of Classic and NeRF-style Methods

Classic and NeRF-style methods differ in how they reproduce unseen views: classic methods average observations into textures, while depth-constrained NeRF improves over unconstrained NeRF.

  • NeRF without depth constraints exhibits severe fogging when rendering novel views.
  • Adding depth constraints resolves the fogging observed in unconstrained NeRF rendering.
  • Classic methods produce sharp textures but can show incorrect view-dependent effects and misalignment artifacts, while the proposed method resembles depth-constrained NeRF.

F. Runtime and Memory Requirements

The proposed method has substantially higher runtime and memory demands than several baselines, with costs increasing for larger scenes or denser sampling. Qualitative comparisons emphasize smoother, more complete geometry than competing reconstructions.

  • With S′c = 256 samples, the method requires 9 hours and 8.5 GB for 2×10^5 iterations; with 512 samples, it requires 13 hours and 10.5 GB.
  • BundleFusion takes 10–40 minutes and uses about 5.8 GB, while NeRF + Depth takes 6 hours and uses 4.7 GB.
  • Runtime ranges from 40 minutes to 6 hours for RoutedFusion, about 4 hours for COLMAP alignment, and 6–12 hours for SIREN.
  • The proposed method fills geometry gaps while maintaining the smoothness of classic fusion approaches; BundleFusion is incomplete, while other baselines fit or retain noise.
Loading 2104.04532v3…