Estimating Depth from RGB and Sparse Sensing

1 Introduction

• Depth estimation is needed for applications including virtual/mixed reality, autonomous vehicles, and robotics, but consumer-grade Kinect v2 consumes about 15W and works indoors only to about 4.5m.The introduction contrasts this with a future VR/MR head-mounted camera target of 1/100th the power and a 1–80m range.Source: §1 · p. 1
• State-of-the-art monocular models still exceed 12% mean absolute relative error on NYUv2, compared with roughly 1% relative depth error for Kinect indoors.The paper links the monocular error level to challenges for 3D reconstruction and tracking.Source: §1 · p. 1
• D3 takes an RGB image and sparse measured depth as input to estimate a full depth map, using the measurements to resolve monocular depth’s scale ambiguity.The sparse measurements could come from lower-power sensors, confident stereo matches, LiDAR-like sensors, or custom sparse sensors.Source: §1 · pp. 1–2
• The model is designed to work across indoor and outdoor scenes and accommodate arbitrary sparse depth patterns.The main body focuses on regular grids; ORB-pattern experiments are described in the Supplementary Materials.Source: §1 · p. 2
• The paper identifies three contributions: sensor-comparable accuracy, indoor/outdoor operation with robustness to common measurement errors, and flexible sparse-input parametrization.The parametrization is intended to accommodate arbitrary sparse patterns during training and testing.Source: §1 · p. 2

2 Related Work

• Deep depth-estimation work has combined CRFs with networks, used geometry and stereo consistency, or explored new architectures.Encoder-decoder networks producing full-resolution pixel-wise maps helped make deep learning suited to depth estimation.Source: §2 · pp. 2–3
• Depth maps serve as outputs or inputs for tasks including tracking, SLAM, and 3D reconstruction or detection.Prior multitask networks also estimate depth alongside surface normals and segmentation, or use depth as a complementary output [3] [12] [34].Source: §2 · p. 3
• Depth superresolution typically starts from a bilinear or bicubic downsampled map that implicitly retains information from all pixels, unlike true sparse sensing.The paper characterizes sparse-to-dense methods and [23] as the more difficult densification setting.Source: §2 · p. 3

3.1 Input Parametrization for Sparse Depth Inputs

• S1 is a nearest-neighbor fill of the sparse depth, while S2 is the Euclidean distance transform of the sampling mask.Together they form an H×W×2 representation; S2 also conveys the sampling pattern and acts as a prior on residual magnitudes.Source: §3.1 · pp. 3–4
• The paired S1 and S2 representation is fast to construct, supports residual prediction relative to S1, and can improve performance and training stability when sparse patterns vary.The procedure involves at most two Euclidean transforms.Source: §3.1 · p. 4

3.2 Sparse Pattern Selection

• Regular grid masks use equally spaced samples in both image directions, with sample count tied to the square of the downsampling factor.For 24×24 downsampling on a 480×640 image, the example uses 0.18% of pixels.Source: §3.2 · p. 5
• Varying the sparse pattern during training reduced training L2 loss and mean relative error by about 3% relative to a static pattern.The decay schedule begins at six times the desired density and smoothly approaches the final density; varying sampling densities can also train one model to perform across densities.Source: §3.2 · p. 5

4.1 Architecture

• The multi-scale architecture feeds RGB, S1, and S2 into DenseNet modules and adds the predicted residual to S1 for the final depth map.The first and last blocks are 3×3 stride-2 convolutions; sparse inputs are also supplied to each module at the appropriate resolution.Source: §4.1 · p. 5, Figure 3 · p. 6

4.2 Datasets

• NYUv2 experiments use 480×640 Kinect V1 depth with a range up to 10m, 26,331 sampled training images, and the standard 654-image validation set.The official split contains 249 training and 215 validation scenes.Source: §4.2 · p. 6
• KITTI experiments use road scenes with depth up to about 85m and evaluate only pixels with ground-truth depth.Training uses random 480×640 crops, with the 480 bottom rows retained vertically because LiDAR ground truth is concentrated near ground level.Source: §4.2 · p. 6

4.3 General Training Characteristics and Performance Metrics

• Training uses Adam, pixel-wise L2 loss, batches of 8 across four Titan X GPUs, and 80,000 batches.The learning rate starts at 1e-3 and decays by 0.2 every 25,000 steps; evaluation uses standard depth metrics including RMSE and delta thresholds.Source: §4.3 · pp. 6–7

5.1 Indoor scenes from NYUv2; 5.4 Generalizing D3 to Outdoor Scenes

• D3 outperforms other sparse-depth methods at every NYUv2 sparsity and beats competing KITTI methods by a large margin.Source: §5.1 · p. 7, Table 1 · p. 8, Table 3 · p. 12

5.1 Indoor scenes from NYUv2

• At 16×16 downsampling, MRE is 0.99%, comparable to consumer-grade sensors; relative error rises at farther, rarer depths.Source: §5.1 · p. 8, Figure 5 · p. 9

5.2 Computational Analysis

• At 0.18% sampling, standard D3 achieves the lowest error and takes 0.11 seconds per VGA pass; a slim version reaches 16fps with mild accuracy loss.Source: §5.2 · p. 9, Table 2 · p. 9

5.3 Generalizing D3 to Multiple Patterns and the Effect of S2

• Training across 0.065%–0.98% sparsities yields strong performance across that range, slightly trailing specialized models at their tuned sparsities but outperforming them away from those settings.Source: §5.3 · pp. 10–11, Figure 7 · p. 11
• Adding distance map S2 improves relative validation L2 loss by 34.4% with random pattern schedules and stabilizes training.Source: §5.3 · p. 11, Figure 7 · p. 11

5.4 Generalizing D3 to Outdoor Scenes

• The mixed model produces high-quality maps on both NYUv2 and KITTI, while KITTI results outperform competing methods by a large margin.Source: §5.4 · pp. 11–12, Table 3 · p. 12

5.5 Robustness Tests

• D3 degrades gracefully under common sparse-input errors and handles constant RGB-depth misregistration robustly, whereas randomly varying misregistration raises error.Source: §5.5 · pp. 12–13, Figure 10 · p. 14

5.6 Discussion

• The discussion attributes performance to multi-scale residual prediction, flexible sparse-input parameterization, and training across varying sparsities.Source: §5.6 · p. 14

• D3 combines sparse depth and registered RGB to produce high-quality dense maps across indoor and outdoor scenes, depths of 1m to 80m, and diverse sparse patterns.Source: §6 · p. 14
• At aggressive indoor sparsities, D3 achieves mean absolute relative error under 1%, comparable to consumer-grade depth sensors.Source: §6 · p. 14
• The authors conclude sparse depth can suffice for applications requiring RGBD input and identify 3D reconstruction, tracking, and surface-normal prediction as future evaluation targets.Source: §6 · p. 14

7.1 Averaging versus Point Sampling

• An 11×11 patch average improves RMSE by ~15% but requires 120 times more data than point sampling, which the setup uses to preserve sparse-sensor power savings.Pointwise ground-truth sampling is chosen so sparse sensing does not access depth beyond the sampled points.Source: Table 4 · p. 17, §7.1 · p. 17

7.2 Additional Results on KITTI

• D3 substantially outperforms all methods and baselines on KITTI across additional sparsity levels, especially at lower pattern densities.The plots compare RMSE and MRE, where lower is better, and δ metrics, where higher is better.Source: §7.2 · p. 17, Figure 11 · p. 18

7.3 Additional Results for Tighter δ Thresholds

• At the tighter δ0.1 threshold, D3 significantly outperforms sparse NN fill, while mixed and exclusive models perform similarly well.The authors report standard δ bounds are too close to 100% for meaningful curves on D3 results.Source: Figure 12 · p. 18, §7.3 · p. 18

7.4 Additional Ablation Studies on D3 Design Choices

• Removing sparse depth from every DenseNet layer chiefly degrades randomized-sparsity training, while removing nearest-neighbor fill substantially degrades performance.Table 5 reports these ablations at 0.18% sparsity and distinguishes standard from randomized sparsity schedules.Source: Table 5 · p. 19, §7.4 · p. 19

7.5 Tabulated Accuracies for Robustness Tests

• Table 6 tabulates D3 performance under different errors in sparse inputs as part of the robustness tests.The supplied discussion introduces the test metrics but does not state their outcomes.Source: §7.5 · p. 19, Table 6 · p. 20

7.6 Preliminary Studies on Interest Point Sampling Maps, and Discussion of Some D3 Weaknesses

• ORB sampling performs worse than regular grids and produces some edge artifacts, while highly irregular patterns can hinder smoothing or omit objects that a forward pass struggles to recover.The authors are exploring smoothness losses and semantic inputs to mitigate these weaknesses; interest-point consistency may reduce temporal jitter.Source: §7.6 · pp. 20–22, Table 7 · p. 21, Figure 13 · p. 21