PaperlayerView exact revision on arXiv

Source-linked AI summary

Sparse-to-Dense: Depth Prediction from Sparse Depth Samples and a Single Image

Fangchang Ma, Sertac Karaman

arXiv:1709.07492v2cs.ROcs.AIcs.CV

TL;DR

Dense depth prediction from monocular RGB is unreliable, motivating the use of sparse depth measurements as additional input. The paper trains a single deep regression network on RGB-D data, showing large gains with few samples and applications in SLAM mapping and LiDAR super-resolution.

  • Problem

    RGB-based depth prediction remains inaccurate and unreliable, while available depth sensors have cost, range, sparsity, or computational limitations.

  • Method

    A single deep regression network takes an RGB image and sparse depth samples as input and predicts a full-resolution depth image.

  • Results

    100 sparse depth samples reduce NYU-Depth-v2 RMSE by over 50% and improve KITTI reliable prediction from 59% to 92%.

  • Takeaways & Limitations

    The method supports dense-map creation as a plug-in for sparse SLAM and visual-inertial odometry and enables LiDAR super-resolution.

  • Takeaways & Limitations

    Accuracy saturates as sparse-depth samples increase, and predictions retain blurry boundaries even with many samples.

Abstract

from arXiv · show

We consider the problem of dense depth prediction from a sparse set of depth measurements and a single RGB image. Since depth estimation from monocular images alone is inherently ambiguous and unreliable, to attain a higher level of robustness and accuracy, we introduce additional sparse depth samples, which are either acquired with a low-resolution depth sensor or computed via visual Simultaneous Localization and Mapping (SLAM) algorithms. We propose the use of a single deep regression network to learn directly from the RGB-D raw data, and explore the impact of number of depth samples on prediction accuracy. Our experiments show that, compared to using only RGB images, the addition of 100 spatially random depth samples reduces the prediction root-mean-square error by 50% on the NYU-Depth-v2 indoor dataset. It also boosts the percentage of reliable prediction from 59% to 92% on the KITTI dataset. We demonstrate two applications of the proposed algorithm: a plug-in module in SLAM to convert sparse maps to dense maps, and super-resolution for LiDARs. Software and video demonstration are publicly available.

I. INTRODUCTION

The paper addresses unreliable monocular depth prediction by combining RGB images with sparse depth measurements in a deep regression model. With few additional samples, the method substantially improves depth reconstruction and supports dense mapping applications.

  • Motivation: Existing depth sensors have practical limitations, while RGB-based prediction remains inaccurate in indoor and outdoor settings.LiDARs can be expensive and sparse, structured-light sensors are range- and sunlight-limited, and stereo requires calibration and substantial computation.
  • Applications: The model significantly outperforms RGB-based and other fusion-based methods.The paper also applies it as a plug-in for sparse SLAM and visual odometry and for LiDAR measurement super-resolution.
  • Approach: The method combines a single RGB image with sparse depth measurements to reconstruct a full-resolution depth image.Sparse measurements can come from low-resolution depth sensors or visual SLAM and visual-inertial odometry algorithms.
  • Results: 100 sparse depth samples reduce NYU-Depth-v2 RMSE by over 50% and raise KITTI reliable prediction from 59% to 92%.These results show substantial gains from adding sparse depth to RGB input.

II. RELATED WORK

Prior RGB-only depth estimation methods used hand-crafted features, graphical models, or image retrieval to infer depth from photometric similarity. These approaches represent earlier alternatives to the paper’s RGB-plus-sparse-depth formulation.

  • RGB-based depth prediction: Early RGB-based methods combined hand-crafted image features with probabilistic graphical models to estimate absolute depth scales.Saxena et al. inferred depth using a Markov Random Field model after estimating scales for image patches.
  • RGB-based depth prediction: Non-parametric methods retrieved images with similar photometric content and combined their depths to estimate a query image’s depth.

III. METHODOLOGY

The proposed system uses a CNN encoder-decoder to map RGB images and randomly sampled sparse depth images to dense depth predictions. Its design includes dataset-specific encoders, upsampling decoders, and training-time sparse-depth sampling for robustness.

  • CNN architecture: The network is tailored to RGB and sparse-depth inputs and uses separate architectures for KITTI and NYU-Depth-v2.The separate networks account for KITTI images being triple the size of NYU-Depth-v2 images and the resulting GPU-memory requirement.
  • CNN architecture: The encoder consists of a ResNet followed by a 3×3 convolution, while the decoder uses four upsampling layers and bilinear upsampling.ResNet-18 is used for KITTI and ResNet-50 for NYU-Depth-v2; UpProj is the default upsampling module.
  • Depth sampling: During training, sparse depth inputs are sampled randomly from valid ground-truth depth pixels using a Bernoulli probability based on the target sample count.The actual number of nonzero samples varies around the target expectation m.
  • Depth sampling: Random sparse-depth sampling serves as data augmentation and improves robustness to different numbers of input depth samples.The sampling strategy is explicitly distinguished from dropout because its purpose is input robustness and creation of additional training data.

C. Data Augmentation

The training pipeline augments RGB-D data with geometric and color transformations and uses L1 loss as the default regression objective. The experiments favor L1 over L2 and berHu, while the implementation uses pretrained ResNet encoders and regularization.

  • Data Augmentation: Online augmentation applies random scaling, rotation, color jitter, normalization, and horizontal flips to the training data.Nearest-neighbor interpolation preserves sparse depth points during scaling and rotation, followed by center cropping for consistent network input size.
  • Loss Function: L2 loss is sensitive to outliers and produces visually over-smooth depth boundaries.
  • Loss Function: L1 produces slightly better results than berHu and is used as the default loss for simplicity and performance.The comparison uses the same network architecture, and both L1 and berHu outperform L2.
  • Implementation: The models use ImageNet-pretrained ResNet encoders, batch size 16, 20 training epochs, a decaying learning rate, and weight decay of 10^-4.

A. The NYU-Depth-v2 Dataset

NYU-Depth-v2 provides RGB and depth images from 464 indoor scenes, with an official 249-scene training split and 215-scene testing split. The final benchmark uses 654 labeled test images.

  • NYU-Depth-v2 contains RGB and depth images collected from 464 indoor scenes using a Microsoft Kinect.
  • The official split assigns 249 scenes to training and 215 scenes to testing.
  • The final benchmark evaluates performance on 654 labeled test images.
  • Training data are sampled spatially evenly from each raw video sequence, generating roughly 48k synchronized depth-RGB image pairs.
  • Frames are downsampled from 640×480, center-cropped, and processed into final images of size 304×228.

B. The KITTI Odometry Dataset

The KITTI odometry dataset pairs camera and LiDAR measurements across 22 sequences, using separate sequence subsets for training and evaluation. LiDAR projections are sparse and concentrated in the bottom image crop.

  • KITTI odometry contains 22 sequences, with one half used for training and the other half for evaluation.
  • The training set uses all 46k images from training sequences, while final evaluation uses 3200 randomly selected test-sequence images.
  • Both left and right RGB cameras are used as unassociated shots, with Velodyne LiDAR measurements projected onto the RGB images.
  • Only the bottom crop measuring 912×228 is used because LiDAR returns no measurements in the upper image region.
  • KITTI ground truth is sparse, typically containing 18k projected measurements among 208k image pixels.
  • Methods are evaluated using RMSE, REL, and δi metrics.

V. RESULTS

The experiments assess component choices, compare the proposed method with state-of-the-art methods on NYU-Depth-v2 and KITTI, and study sparse-sample counts. They also demonstrate two applications.

  • The experiments evaluate loss functions and network components for depth-prediction accuracy.
  • The proposed method is compared with state-of-the-art methods on the NYU-Depth-v2 and KITTI datasets.
  • The study explores how the number of sparse depth samples affects prediction performance.
  • Two applications are demonstrated in later subsections.The supplied passage identifies the applications but truncates their names.

A. Architecture Evaluation

Architecture evaluation selects L1 loss, UpProj upsampling, and a regular convolution for the first layer. These choices are based on comparisons of losses, receptive fields, and RGB-depth channel handling.

  • Loss Functions: L1 and berHu outperform L2, while L1 performs slightly better than berHu and is selected as the default loss.
  • Upsampling Layers: UpProj outperforms the other upsampling layers with its larger 4×4 receptive field and is selected as the default.
  • Upsampling Layers: DeConv3 outperforms DeConv2 in every metric, while DeConv3 and UpConv have comparable performance because both use a 3×3 receptive field.
  • First Convolution Layer: The first convolution study compares regular convolution, depthwise separable convolution, and channel dropout for RGBd inputs.
  • First Convolution Layer: With an average of 100 sparse input samples, DepthWise and Conv perform similarly and both significantly outperform ChanDrop.
  • First Convolution Layer: The regular convolution layer is used in all experiments for comparison consistency.

1) NYU-Depth-v2 Dataset:

On NYU-Depth-v2, the proposed fusion method is compared with RGB-based and other fusion approaches, showing that sparse depth substantially improves prediction and that RGB adds further accuracy.

  • NYU-Depth-v2 Dataset: The study compares RGB-based approaches and a fusion approach using an additional 2D laser scanner against the proposed method.The quantitative comparisons are reported in Table II.
  • NYU-Depth-v2 Dataset: 20 sparse depth samples without color already produce significantly better predictions than RGB-only input.This comparison uses the same network architecture across the RGB and sparse-depth groups.
  • NYU-Depth-v2 Dataset: Adding RGB to sparse depth improves prediction accuracy when the methods use the same number of samples.The model learns a suitable representation from both RGB images and sparse depth images.
  • NYU-Depth-v2 Dataset: 100 spatially uniform samples outperform a competing method using 225 laser measurements.The authors attribute this comparison to the greater information provided by spatially uniform samples than by a line measurement.

2) KITTI Dataset:

On KITTI, RGB-only depth prediction performs poorly outdoors, while adding sparse LiDAR depth substantially improves accuracy and reliability despite the use of a smaller training dataset than some comparisons.

  • KITTI Dataset: RGB-based depth prediction has a pixel-wise RMSE close to 7 meters in outdoor scenarios.KITTI is challenging because its maximum distance is 100 meters, compared with 10 meters for NYU-Depth-v2.
  • KITTI Dataset: The method uses sparsely labeled depth images projected from LiDAR rather than dense stereo-derived disparity maps.The authors note that this gives them a much smaller training dataset than the cited stereo-based work.
  • KITTI Dataset: 500 additional depth samples reduce RMSE to 3.3 meters, half the RGB approach’s error.This result is reported for the KITTI evaluation.

C. On Number of Depth Samples

The experiments examine how depth-sample count affects RGBd prediction and demonstrate applications in dense mapping and LiDAR super-resolution. Accuracy improves with additional samples, while saturation and blurry boundaries remain limitations.

  • Number of Depth Samples: 100 samples reduce RGBd NYU-Depth-v2 RMSE from 51cm to around 25cm, while REL falls from 0.15 to 0.05.RGBd outperforms RGB beyond 10 samples, and the gap increases as sample count grows.
  • Number of Depth Samples: At 1000 samples, RGBd and sparse-depth-only approaches perform equally well, with sparse samples dominating prediction and color becoming almost irrelevant.The 1000 samples represent less than 1.5% of image pixels.
  • Number of Depth Samples: 100 samples reduce KITTI RGBd RMSE from 7 meters to 3.5 meters and REL from 0.21 to 0.07.The reported percentage improvement is almost identical to that on NYU-Depth-v2.
  • Limitations: Accuracy saturates as sample count increases, and predictions retain blurry boundaries even with many samples.The paper attributes both effects to fine-detail loss in bottleneck encoder-decoder architectures and leaves skip connections for future study.
  • Dense Mapping from Visual Odometry: The method produces a dense prediction map that closely resembles ground truth and is much denser than the sparse visual-odometry point cloud.Predictions outside the trusted convex-hull region are removed, leaving gaps such as on untracked white walls.
  • LiDAR Super-Resolution: For LiDAR super-resolution, RGB plus sparse depth achieves 4.9% average REL versus 20.8% using RGB alone, with cars more recognizable in predictions.The method addresses vertically sparse point clouds caused by LiDAR’s low vertical angular resolution.
Loading 1709.07492v2…