PaperlayerView exact revision on arXiv

Source-linked AI summary

DFuseNet: Deep Fusion of RGB and Sparse Depth Information for Image Guided Dense Depth Completion

Shreyas S. Shivakumar, Ty Nguyen, Ian D. Miller, Steven W. Chen, Vijay Kumar, Camillo J. Taylor

arXiv:1902.00761v2cs.CV

TL;DR

Sparse depth sensors provide much less spatial detail than high-resolution imagery, motivating depth completion from image context. DFuseNet processes image and depth features in separate branches before combining them; its reported KITTI benchmark RMSE is higher than the cited state-of-the-art result, while the authors report boundary-preserving predictions and cross-dataset evaluation.

  • Problem

    Sparse range sensors typically return an order of magnitude fewer points than camera images, while higher-resolution sensors are expensive and dense range data can benefit planning and obstacle avoidance.

    Source: §I · p. 1

  • Method

    DFuseNet uses separate color and depth branches to extract contextual features, then fuses them for dense depth prediction.

    Source: B. Feature Extraction · p. 3, D. Depth Prediction · p. 3, A. Architecture · p. 7

  • Results

    1206.66 mm mean RMSE, compared with the current published state-of-the-art 772.87 mm; the authors report that their predictions preserve semantic boundaries as well as or better than methods with higher metric accuracy.

    Source: 1) Quantitative Comparison · p. 4

  • Takeaways & Limitations

    The approach produces upsampled depth that respects relevant image boundaries and correlates with available ground truth, and the authors propose a mini-dataset for additional validation.

    Source: C. Conclusion · pp. 7–8

Abstract

from arXiv · show

In this paper we propose a convolutional neural network that is designed to upsample a series of sparse range measurements based on the contextual cues gleaned from a high resolution intensity image. Our approach draws inspiration from related work on super-resolution and in-painting. We propose a novel architecture that seeks to pull contextual cues separately from the intensity image and the depth features and then fuse them later in the network. We argue that this approach effectively exploits the relationship between the two modalities and produces accurate results while respecting salient image structures. We present experimental results to demonstrate that our approach is comparable with state of the art methods and generalizes well across multiple datasets.

I. INTRODUCTION

Sparse depth sensors provide far fewer points than high-resolution cameras, while dense range data is useful for applications such as planning and obstacle avoidance. The paper proposes a CNN that combines image context with sparse depth and evaluates its generalization across four datasets.

I. INTRODUCTION

  • Depth sensors typically return an order of magnitude fewer points than camera images, while higher-resolution sensors are expensive and dense range data can aid planning and obstacle avoidance.The paper frames upsampling sparse measurements to image resolution as the motivating problem.Source: §I · p. 1
  • The proposed CNN is intended to learn global and contextual cues from color images and combine them with sparse depth while preserving edges and smoothness.The authors identify possible inputs from LiDAR, stereo, or structure-from-motion depth sources.Source: §I · p. 1
  • The contributions include a dual-branch CNN with spatial pyramid pooling and multi-scale deconvolutions, a training regime using sources such as stereo imagery, and evaluation on four datasets.The authors also make their mini-dataset publicly available for validation.Source: §I · p. 1

II. RELATED WORK

Related work spans monocular, stereo, and sparse-depth estimation, as well as methods for completing incomplete inputs. The paper positions its approach among learning-based and classical depth-completion methods.

Depth Estimation

  • Monocular depth estimation uses supervised, unsupervised, and self-supervised strategies; Fu et al. report an inverse RMSE score of 12.98 on KITTI depth prediction.CNNs are described as the state of the art in monocular depth estimation at the time of writing.Source: §II · p. 1
  • Stereo methods include learned patch-similarity costs and end-to-end disparity prediction, while Chang et al. use pyramid pooling for global image-dependent features,.End-to-end stereo methods are described as preserving edges.Source: §II · pp. 1–2

Incomplete Input Data

  • Super-resolution and in-painting address completion from incomplete inputs, while multi-scale architectures with skip connections have been used for image and depth upsampling,,.The paper places generative networks for context-aware in-painting outside its scope,.Source: §II · p. 2

Depth Completion

  • Depth-completion approaches range from sparsity-invariant or masked convolutions, [27] to morphological processing and learned filters [14],, confidence masks [15], [30], and depth propagation.The paper says it fills sparse depth before network input, following a strategy similar to.Source: §II · p. 2
  • Other depth-completion work uses multi-scale feature fusion or supervised and self-supervised ResNet-based methods, [33].Source: §II · p. 2

III. APPROACH

The approach uses separate color and depth branches to extract modality-specific context before combining features for dense depth prediction. The architecture uses spatial pyramid pooling and decoder features at multiple scales.

III. APPROACH

  • Independent branches extract features from color and sparse depth before feature concatenation and multi-scale deconvolution produce a dense depth map.The branches are not weight-coupled because the inputs have different modalities.Source: A. Design Overview · pp. 2–3
  • In the illustrated branch ablations, the predicted depth uses both the RGB image and filled depth, and the caption reports that both branches contribute significantly.The caption also states that filled depth is not simply propagated without learning.Source: Figure 2 · p. 3
  • Spatial pyramid pooling blocks learn coarse-to-fine features, which are concatenated and propagated through deconvolution layers for multi-scale depth prediction.Source: A. Design Overview · p. 3

B. Feature Extraction

Feature extraction applies modality-specific processing and pooling before combining features. The subsequent prediction and training stages restore image resolution and combine multiple loss signals.

B. Feature Extraction

  • The color and depth inputs use different feature extractors and pooling: residual blocks and average pooling for color, and fewer, wider-kernel layers with max pooling for depth.Both branches use SPP windows of 64, 32, 16, and 8; the depth input is first filled and normalized to [0, 1].Source: B. Feature Extraction · p. 3

C. Combining Modalities

  • Color and depth features are concatenated, then sequential convolutions reduce channels while increasing spatial resolution to learn a joint feature representation.The concatenated volume also includes an intermediate color-branch residual-block output.Source: C. Combining Modalities · p. 3

D. Depth Prediction

  • Incremental deconvolutions restore resolution, and upsampled outputs from multiple layers are combined for a final convolution and depth prediction.The final output uses a 1x1 convolution on mainly interpolated low-resolution features to minimize checkerboard effects.Source: D. Depth Prediction · p. 3

E. Training

  • Training uses a weighted combination of loss terms, some evaluated across the image and others only where accurate ground-truth depth exists.The weights vary during training according to confidence in each signal.Source: E. Training · p. 3

1) Primary Loss:

The primary loss is evaluated only where ground-truth depth exists, with L2 chosen for better benchmark RMSE; optional stereo supervision adds dense stereo estimates to guide predictions where LiDAR ground truth is absent.

1) Primary Loss:

  • L2 was selected over L1 for the primary loss because it produced better RMSE values on evaluation benchmarks.The loss is computed only at pixels with ground-truth depth and averaged over those points.Source: 1) Primary Loss · p. 3

2) Optional Stereo Supervision:

  • Stereo supervision uses a self-supervised L2 loss between predicted depth and stereo-estimated depth to guide learning where LiDAR ground truth is missing.Stereo estimates come from the paired input images using known intrinsics and extrinsics, with left-right consistency checks discarding noisy or occluded estimates.Source: 2) Optional Stereo Supervision · pp. 3–4
  • Adding the stereo term improves extrapolation where ground truth or LiDAR is unavailable, though accuracy gains are slight and training is faster.Figure 3 compares the input image, prediction without the term, and prediction with it.Source: Figure 3 · p. 4, 2) Optional Stereo Supervision · p. 4

3) Smoothness:

The training objective includes a second-derivative smoothness penalty, while KITTI depth data is sparse in important image regions and leaves distant areas such as the sky unmeasured.

3) Smoothness:

  • The smoothness term is an L1 penalty on the second-order derivative of the predicted dense depth image.It follows a strategy used in unsupervised monocular depth estimation and structure-from-motion networks.Source: 3) Smoothness · p. 4

IV. EXPERIMENTS

  • Training used networks initialized from scratch, ADAM, batches of 20–25, roughly 40 epochs, and an initial learning rate of 10^-4.The learning rate was dropped by 10% every five epochs; weight decay was 10^-4, with α=1, β=0.01, and γ=0.001.Source: §IV · p. 4

A. KITTI Depth Completion

  • KITTI range measurements cluster near the image bottom and often miss object boundaries, while evaluation does not discourage solutions with blurry edges.The KITTI ground truth merges 11 LiDAR scans and is evaluated only at refined ground-truth LiDAR points.Source: A. KITTI Depth Completion · p. 4
  • Because KITTI lacks depth information in distant regions such as the sky, the method aims to predict across more of the image using available context.Some prior approaches crop regions without LiDAR data.Source: A. KITTI Depth Completion · p. 4

1) Quantitative Comparison:

On KITTI, DFuseNet trails several benchmark methods in metric accuracy but is reported to preserve semantic boundaries as well as or better than stronger methods. Additional evaluations report extrapolation with stereo supervision, Virtual KITTI results, and NYUDepthV2 comparisons with a different sample split.

2) Learning to extrapolate with limited ground truth data:

  • Adding stereo supervision yields minimal quantitative improvement, slightly improves accuracy and training speed, and enables extrapolation where LiDAR and ground truth are absent.Figure 3 contrasts predictions without and with the stereo term.Source: 2) Learning to extrapolate with… · p. 4

B. Virtual KITTI

  • 3.458 m RMSE and 1.355 m MAE were reported on the Virtual KITTI validation set, using 10% randomly sampled ground-truth depth as sparse input and a 100 m perception limit.The paper notes Virtual KITTI is not an accurate representation of real-life data; Figure 5 shows color, sparse input, filled depth, prediction, and ground truth.Source: B. Virtual KITTI · p. 4, Figure 5 · p. 5

1) Quantitative Comparison:

  • 1206.66 mm mean RMSE for DFuseNet versus 772.87 mm for the published state of the art; several methods outperform it quantitatively, while Figure 4 shows semantic boundary preservation comparable to or better than stronger methods.The compared methods include Van Gansbeke et al. [30], MorphNet, SparseConv [12], Sparse-to-Dense [33], NConv-CNN [15], HMS-Net [27], and CSPN.Source: 1) Quantitative Comparison · p. 4, Figure 4 · p. 5

C. NYU Depth V2

  • On NYUDepthV2, the approach is reported as comparable to Ma et al. and better than Liao et al. at 200 samples, but the comparison uses different training and validation samples.The authors also report accurate results with noisy input filling; Table I explicitly cautions that errors were not computed on the same data.Source: C. NYU Depth V2 · p. 5, Table I · p. 5

D. Number of depth samples

The NYUDepthV2 experiment evaluates RMSE across different sparse-depth sample counts and finds that gains from adding samples saturate around 5,000 points. Even with very sparse input, the RGB branch can guide depth prediction using image context.

D. Number of depth samples

  • Figure 7 plots NYUDepthV2 RMSE against input depth samples as a percentage of image resolution, with a log-scaled x-axis.The number of samples is varied as a percentage of total image resolution.Source: Figure 7 · p. 6
  • RMSE gains saturate at around 5,000 depth samples, roughly 1.7% of image resolution.The experiment trains a separate model for each sample size and evaluates RMSE on the validation set.Source: D. Number of depth samples · pp. 5–6
  • With extremely sparse input, the RGB branch can guide depth prediction using mostly image-based contextual cues.Figure 6 shows examples with 1,000 input samples in the first row and 500 in the second, alongside input, filled depth, prediction, and ground truth.Source: D. Number of depth samples · p. 6, Figure 6 · p. 6

E. Penn Driving LiDAR RGB Dataset

The authors calibrated an Ouster LiDAR to a Zed Mini camera using checkerboard corners detected in the LiDAR intensity image. They then collected a short Philadelphia driving sequence and evaluated predictions using withheld LiDAR points.

E. Penn Driving LiDAR RGB Dataset

  • Checkerboard corners detected in the Ouster intensity image were used to calibrate the LiDAR and Zed Mini camera extrinsics.The projected points were used with standard stereo calibration tools; projected LiDAR points on the RGB image were used to verify calibration.Source: E. Penn Driving LiDAR RGB Dataset · p. 6, Figure 8 · p. 6
  • The calibrated camera–LiDAR assembly collected 15 minutes of driving in downtown Philadelphia.The setup used an Ouster OS-13 64-beam LiDAR and a StereoLabs Zed Mini camera.Source: E. Penn Driving LiDAR RGB Dataset · p. 6
  • For evaluation, the authors sampled 5,000 points from roughly 40,000 LiDAR points and used the withheld points to assess inference accuracy.Points were artificially removed to simulate a lower-resolution sensor, and the pre-trained model was run on the resulting input.Source: E. Penn Driving LiDAR RGB Dataset · p. 6

V. DISCUSSION AND CONCLUSION

The discussion favors independent RGB and depth processing before fusion, because the modalities have different representations and early fusion before pyramid pooling reduced performance. It also notes that hole filling and depth residual connections can interfere with the network's use of RGB cues.

A. Architecture

  • Independent convolution and pyramid-pooling branches process RGB and depth separately before fusion; fusing before the SPP blocks reduced performance.The authors interpret the drop as a sign that independently learned information helps the final fusion and prediction.Source: A. Architecture · pp. 6–7
  • The authors favor regular convolutions with simple hole filling, after sparse convolutions in the depth branch caused a significant performance drop.They describe morphological filters, fill maps, and nearest-neighbor interpolation as simple ways to initialize filled depth input.Source: A. Architecture · p. 7
  • With hole-filled depth input, residual connections from depth layers to penultimate deconvolution layers produced similar accuracy to IPBasic but prevented learning from the RGB branch.The authors suggest such a network may require more careful training and hyperparameter selection.Source: A. Architecture · p. 7

B. Generalization

A KITTI-trained model produces qualitative predictions on Virtual KITTI and NYUDepthV2 without retraining, though cross-dataset depth-scale differences cause quantitative errors. The paper concludes that its separate-branch approach produces depth estimates that respect image boundaries and correlate with available ground truth.

B. Generalization

  • On NYUDepthV2, the KITTI-trained model achieved RMSE 0.607 m and MAE 0.216 m using 10% of ground truth as input samples; on Penn driving, it achieved RMSE 6.690 m and MAE 3.157 m.The Penn error was higher mainly in regions far away.Source: B. Generalization · p. 7
  • The KITTI-trained model was tested on Virtual KITTI, Penn driving, and NYUDepthV2, and Figure 9 shows predictions on Virtual KITTI and NYUDepthV2 without retraining or fine-tuning.The model generated semantically valid predictions, but depth-scale differences across datasets caused quantitative errors that minimal training with KITTI initialization quickly corrected.Source: B. Generalization · p. 7, Figure 9 · p. 7, Figure 10 · p. 7

C. Conclusion

  • The approach produces upsampled depth results that respect relevant image boundaries and correlate well with available ground truth.The authors also propose their own dataset for additional validation.Source: C. Conclusion · pp. 7–8

Related papers

Loading 1902.00761v2…