PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Depth Completion of a Single RGB-D Image

Yinda Zhang, Thomas Funkhouser

arXiv:1803.09326v2cs.CV

TL;DR

Commodity RGB-D cameras leave substantial depth holes on challenging surfaces. The paper predicts surface normals and occlusion boundaries from RGB, combines them with observed depths through global optimization, and reports better completion than baseline approaches, while noting that normals and boundaries alone cannot always determine absolute depth.

  • Problem

    Commodity RGB-D cameras produce missing depth for challenging surfaces, while existing training data generally lacks completed ground truth for unobserved pixels.

  • Method

    The method predicts surface normals and occlusion boundaries from color, then solves global surface structure using observed depths as soft constraints.

  • Results

    The proposed approach outperforms previous depth-inpainting and depth-estimation baselines on a new benchmark.

  • Takeaways & Limitations

    The approach completes depth using local geometric predictions from color together with global structure recovered from observed depth measurements.

  • Takeaways & Limitations

    Surface normals and occlusion boundaries alone cannot determine depth relationships in pathological configurations such as a wall visible through a window.

Abstract

from arXiv · show

The goal of our work is to complete the depth channel of an RGB-D image. Commodity-grade depth cameras often fail to sense depth for shiny, bright, transparent, and distant surfaces. To address this problem, we train a deep network that takes an RGB image as input and predicts dense surface normals and occlusion boundaries. Those predictions are then combined with raw depth observations provided by the RGB-D camera to solve for depths for all pixels, including those missing in the original observation. This method was chosen over others (e.g., inpainting depths directly) as the result of extensive experiments with a new depth completion benchmark dataset, where holes are filled in training data through the rendering of surface reconstructions created from multiview RGB-D scans. Experiments with different network inputs, depth representations, loss functions, optimization methods, inpainting methods, and deep depth estimation networks show that our proposed approach provides better depth completions than these alternatives.

1. Introduction

The paper addresses large missing regions in commodity RGB-D depth images with a two-stage approach that predicts local geometric cues from color and reconstructs global depth using observed measurements.

  • Commodity RGB-D cameras can miss more than 50% of depth pixels when surfaces are glossy, bright, thin, close, or distant.
  • The authors introduce 105,432 RGB-D images with completed depths rendered from large-scale surface reconstructions in 72 real-world environments.
  • Direct depth regression performs poorly for large holes, so the network predicts local differential properties that are easier to estimate than absolute monocular depth.
  • The proposed method predicts surface normals and occlusion boundaries from color, then solves for completed depths using global optimization constrained by observed depths.
  • The two-stage decomposition achieves significantly smaller relative error than alternatives and does not require retraining for new depth sensors.

2. Related Work

Prior work estimates depth, fills depth holes, improves depth resolution, or reconstructs depth from sparse samples, whereas this paper focuses on completing large, complex missing regions.

  • Monocular depth-estimation methods use hand-tuned models, shape cues, or deep networks, but are trained to reproduce raw observed camera depth.
  • Depth inpainting methods use smoothness, diffusion, surface extrapolation, edge alignment, low-rank completion, and patch-based techniques.
  • Depth super-resolution assumes complete, regularly sampled low-resolution measurements and targets spatial resolution rather than missing regions.
  • Unlike these settings, depth completion must synthesize large-scale content for holes that can be large and complex.
  • Sparse-sample reconstruction methods augment color images with limited depth measurements, including wavelet and contourlet representations.

3. Method

The method creates completed RGB-D training data from multi-view surface reconstructions, predicts local surface geometry from color, and globally solves for missing depths using observed measurements. It separates color-based prediction from depth-constrained optimization, while recognizing that normals alone cannot determine every depth relationship.

  • Dataset: The dataset renders completed depth images from multi-view RGB-D surface meshes, avoiding the cost of pairing low-cost and high-cost depth sensors.Matterport3D meshes are reconstructed from multi-view scans and rendered from sampled RGB-D viewpoints.
  • Dataset: 64.6% of pixels missing from raw depth images are filled by the multi-view reconstruction process.Combining viewpoints also recovers surfaces distant from one camera or viewed obliquely by another.
  • Dataset: The completed images generally preserve close-surface resolution, improve distant-surface resolution, and reduce noise through multi-view filtering and averaging.The reconstruction provides finer projected detail for distant surfaces and is especially useful beyond 4 meters, where raw measurements are quantized and noisy.
  • Depth Representation: Instead of regressing absolute depth directly, the method predicts local surface properties such as normals and occlusion boundaries, then solves back for absolute depth.These properties depend on local image neighborhoods and are more accessible from color than absolute monocular depth.
  • Depth Representation: Normals and occlusion boundaries cannot theoretically determine depths in regions enclosed by occlusions, so observed depths provide regularization during global optimization.In real-world scenes, connected paths and available observations often make completion practical despite this indeterminacy.
  • Network Architecture and Training: Training on all pixels and rendered normals performs better than training on only observed or unobserved pixels and raw normals; color-only prediction also outperforms depth-based prediction in large holes.The network is therefore trained from color to predict normals and boundaries, with observed depth used later in optimization.

4. Experimental Results

Experiments evaluate the proposed color-only prediction and depth-optimization pipeline against alternative inputs, representations, boundary handling, sparse observations, inpainting methods, and depth-estimation methods.

  • Evaluation setup: Evaluations measure errors primarily on pixels unobserved by the test depth image, using relative error, RMSE, and threshold accuracy metrics.
  • Ablation studies: Color-only input outperforms RGB-D input for normal prediction, with median error 17.28° versus 23.07°, and slightly better depth completion Rel 0.089 versus 0.090.The authors expect observed depth encourages interpolation rather than synthesizing depth in large holes.
  • Ablation studies: Surface normals produce the best depth completion representation, achieving Rel = 0.089 versus 0.167 for depths, 0.100 for derivatives, and 0.092 for normals plus derivatives.Normals encode surface orientation without depth-dependent scale.
  • Ablation studies: Occlusion-boundary weighting improves results by approximately 19%, reducing Rel from 0.110 to 0.089.Boundary predictions help the optimizer down-weight noisy or incorrect normals near depth discontinuities.
  • Ablation studies: The method works with sparse observations: 2,000 input depth pixels represent only 2.5% of all pixels, while unobserved-pixel estimation remains harder than observed-pixel estimation.This supports potential use with depth sensors that provide sparse measurements.
  • Comparison to baseline methods: Against hand-tuned inpainting methods, the proposed method achieves Rel = 0.089 versus 0.103–0.151 for the alternatives.The authors attribute the improvement to learned surface-normal priors rather than simple geometric heuristics.
  • Comparison to baseline methods: Compared with deep color-to-depth methods, the proposed method is 23–40% better on both observed and unobserved evaluation pixels and preserves scene structure and fine details.Qualitative results show this advantage even with only one raw-depth pixel.

5. Conclusion

The paper presents a two-stage deep-learning framework for completing missing depth in commodity RGB-D images and trains it using rendered completed-depth data.

  • The framework predicts surface normals and occlusion boundaries from color, then solves completed depths from those predictions.
  • Supervised training uses data rendered from large-scale surface reconstructions, and experiments show improvements over baseline depth-inpainting and depth-estimation approaches.

Supplementary Material

The supplementary material contains additional implementation details and experimental results beyond the main paper.

  • The document adds ablation studies, cross-dataset experiments, and comparisons with other inpainting methods.

A. Further Implementation Details

The supplementary implementation details describe completed-depth data construction from Matterport3D meshes and show that the reconstructed meshes preserve most relevant surface details.

  • The supplementary material states that data and code will be released upon acceptance to support reproducibility.
  • Matterport3D scenes are partitioned into room-like regions and reconstructed into dense point clouds while excluding depths beyond 4 meters to avoid mesh noise.
  • The reconstruction process produces meshes with 12.2M triangles that reproduce most surface details and cover regions where RGB-D holes typically occur.

A.2. Network architecture

The architecture evaluates color, depth, and combined inputs, while predicting depth-related representations and occlusion boundaries through task-specific outputs and losses.

  • Architecture: The implementation derives its networks from a prior surface-normal model and removes the last two batch-normalization layers for better practical performance.
  • Preprocessing: Color images are resized with bilinear interpolation, whereas depth images use nearest-neighbor resizing to avoid inaccurate smoothing.
  • Inputs: The network accepts color, depth, or concatenated Color+Depth inputs, producing 3, 2, or 5 input channels respectively.Depth includes absolute depth and a binary valid-depth mask.
  • Outputs: The outputs include one-channel absolute depth, three-channel surface normals, eight-channel depth derivatives, and three-class occlusion-boundary probabilities.The boundary classes are no edge, depth crease, and occlusion boundary.
  • Losses: Depth, surface normal, and derivative regression use SmoothL1 or cosine embedding losses, while occlusion-boundary classification uses cross entropy.SmoothL1 trains depth and derivatives; cosine embedding trains surface normals.

A.3. Training schema

The training and testing pipeline uses RMSprop with fixed batch size and scheduled learning-rate reduction, while the section situates additional experiments on alternative design choices.

  • Optimization: RMSprop trains all tasks with momentum 0.9, batch size 1, and an initial learning rate of 0.001 halved every 100K iterations.All models converge within 300K iterations.
  • Experimental scope: The additional experiments cover ablations and cross-dataset comparisons with other depth-completion methods.The ablations examine inputs, training data, losses, depth representations, and optimization methods.

B.1. Ablation Studies

Ablations favor color-only surface prediction with rendered-depth supervision and solving depth from normals over direct depth or disparity prediction.

  • Training inputs and supervision: Training with color alone, all available pixels, and rendered depth supervision produces the best qualitative surface-normal predictions among the tested settings.The selected setting consequently benefits global depth optimization.
  • Training inputs and supervision: Rendered-depth supervision outperforms raw-depth supervision for training surface prediction models.The comparison attributes part of the improvement to supervision on unobserved pixels.
  • Depth representation: Rel = 0.089 versus 0.408 after correcting a table characterization, preserving the conclusion that normals plus depth solving outperform direct depth prediction.The corrected result is reported for color input and direct complete-depth prediction from input depth.
  • Depth representation: Estimating disparity is not better than estimating depth when either color or depth is used as input for depth completion.

B.2. Cross-Dataset Experiments

Cross-dataset tests show strongest performance when training and testing datasets match, while models still transfer reasonably across datasets and depth sensors without fine-tuning.

  • Cross-dataset transfer: Models work best on the dataset matching their training source, while Matterport3D-trained models generalize better than ScanNet-trained models.The authors associate this difference with Matterport3D’s more diverse camera viewpoints.
  • Cross-dataset transfer: Matterport3D- and ScanNet-trained models both work reasonably well on the other dataset without fine-tuning.
  • Sensor transfer: Intel RealSense provides better short-range depth but more missing area than Structure Sensor and Kinect, especially for distant or special-material surfaces.
  • Sensor transfer: The method runs effectively on RGB-D images from depth sensors with significantly different missing-depth patterns.

B.3. Comparisons to Depth Inpainting Methods

The paper compares its depth-completion method with several depth-inpainting baselines using depth, surface-normal, and 3D point-cloud visualizations. Across these comparisons, the proposed method is reported to produce better completions and 3D geometry than the alternatives.

  • Baseline comparisons: The study compares the proposed method with DCT, FCN, context encoder, and other baseline inpainting methods.DCT fills missing values by solving a penalized least-squares system, while FCN and CE learn to generate completed depth images.
  • Baseline comparisons: DCT results mostly interpolate the missing holes and are similar to other inpainting comparisons.
  • Baseline comparisons: Methods designed for color inpainting, including FCN and CE, are reported to be ineffective for depth inpainting.The paper attributes this partly to depth-specific characteristics and differences in network architecture and resolution.
  • Quantitative and qualitative results: None of the compared alternatives, including filling missing regions with a mean depth value, works as well as the proposed method.
  • Quantitative and qualitative results: The proposed method significantly outperforms baseline inpainting methods in Table 8.
  • 3D geometry comparisons: Point-cloud visualizations show that the proposed method produces better 3D geometry with fewer bleeding issues at boundaries.The visualizations project completed depth using Matterport3D camera intrinsics and copy color intensity from the color image.
Loading 1803.09326v2…