PaperlayerView exact revision on arXiv

Source-linked AI summary

PENet: Towards Precise and Efficient Image Guided Depth Completion

Mu Hu, Shuling Wang, Bin Li, Shiyu Ning, Li Fan, Xiaojin Gong

arXiv:2103.00783v3cs.CV

TL;DR

Image guided depth completion must infer dense depth from sparse, noisy measurements using complementary color and depth information. PENet uses specialized color-dominant and depth-dominant branches, geometric convolution, adaptive fusion, and DA-CSPN++ refinement. The full model ranked 1st on the KITTI depth completion online leaderboard at submission and was more efficient than most top-ranked methods.

  • Problem

    Image guided depth completion must generate dense depth from sparse depth and a high-quality image, although sparse measurements remain noisy near object boundaries.

  • Method

    PENet uses complementary color-dominant and depth-dominant branches, geometric 3D encoding, confidence-weighted fusion, and efficient DA-CSPN++ refinement.

  • Results

    1st on the KITTI depth completion online leaderboard at submission, the full model also inferred more efficiently than most top-ranked methods.

  • Takeaways & Limitations

    PENet thoroughly exploits and fuses color and depth modalities while remaining trainable from scratch without extra datasets.

Abstract

from arXiv · show

Image guided depth completion is the task of generating a dense depth map from a sparse depth map and a high quality image. In this task, how to fuse the color and depth modalities plays an important role in achieving good performance. This paper proposes a two-branch backbone that consists of a color-dominant branch and a depth-dominant branch to exploit and fuse two modalities thoroughly. More specifically, one branch inputs a color image and a sparse depth map to predict a dense depth map. The other branch takes as inputs the sparse depth map and the previously predicted depth map, and outputs a dense depth map as well. The depth maps predicted from two branches are complimentary to each other and therefore they are adaptively fused. In addition, we also propose a simple geometric convolutional layer to encode 3D geometric cues. The geometric encoded backbone conducts the fusion of different modalities at multiple stages, leading to good depth completion results. We further implement a dilated and accelerated CSPN++ to refine the fused depth map efficiently. The proposed full model ranks 1st in the KITTI depth completion online leaderboard at the time of submission. It also infers much faster than most of the top ranked methods. The code of this work is available at https://github.com/JUGGHM/PENet_ICRA2021.

I. INTRODUCTION

PENet addresses sparse, noisy depth completion by thoroughly fusing color and depth through specialized branches, geometric encoding, adaptive fusion, and efficient refinement.

  • Motivation: Image guided depth completion predicts dense depth from sparse depth and a high-resolution color image, but sparse measurements remain noisy near object boundaries.These challenges matter for applications including autonomous driving, 3D reconstruction, and augmented reality.
  • Motivation: Earlier two-branch methods fuse modalities at multiple stages but require extra datasets such as Cityscapes or synthetic data for pretraining.Other approaches use separate modality encoders and primarily perform late fusion.
  • Proposed framework: PENet uses color-dominant and depth-dominant branches whose complementary dense predictions are adaptively fused with learned confidence weights.The color-dominant branch is more reliable near boundaries, while the depth-dominant branch is overall reliable but affected by boundary noise.
  • Proposed framework: A geometric convolutional layer encodes 3D cues by concatenating a 3D position map to convolutional inputs, while DA-CSPN++ refines the fused depth map efficiently.The refinement module is designed to recover accurate valid input depths that prediction may not preserve.
  • Results: 1st in the KITTI depth completion online leaderboard at submission, the full model also infers more efficiently than most top-ranked methods.The reported ranking is time-specific to submission.

II. RELATED WORK

Related depth-completion methods address sparse, noisy inputs and multimodal fusion with sparse-invariant convolutions and two-branch architectures.

  • Depth completion: Depth completion produces dense depth from sparse measurements, optionally using a reference image to exploit structural information and improve performance.The task involves irregular sparsity, noise, and distinct color and depth modalities.
  • Multimodal fusion: PENet adopts a two-branch architecture similar to prior FusionNet and DeepLiDAR designs but constructs its branches for different purposes.The paper states that its network is more effective than those related designs.

B. Geometric Encoding

The paper motivates geometric encoding for depth completion and pairs a simple geometric convolution with a dilated, accelerated CSPN++ refinement strategy.

  • Geometric encoding: Existing geometric-cue methods use continuous convolution, graph propagation, or surface normals, but can be computationally complicated or require extra learning data.These approaches establish the relevance of 3D geometric information while motivating a simpler alternative.
  • Depth refinement: CSPN++ and related spatial propagation methods refine depth predictions but remain inefficient, motivating dilation and more efficient propagation.The proposed implementation enlarges propagation neighborhoods and accelerates the operation.

III. METHODOLOGY

The methodology combines color-dominant and depth-dominant encoder-decoder branches, confidence-weighted depth fusion, and geometric convolutional inputs.

  • Framework: The end-to-end framework contains a two-branch backbone and a refinement module, with two predictions adaptively fused using learned confidence weights.The fused map is then refined to enhance depth quality.
  • Two-branch backbone: Both branches use similar encoder-decoder networks, but one emphasizes color information while the other emphasizes depth information.The backbone is designed to exploit both modalities through their respective branches.
  • Two-branch backbone: The depth-dominant branch receives the sparse depth map and the color-dominant prediction while concatenating corresponding decoder and encoder features.This design performs multimodal fusion at multiple stages.
  • Geometric encoding: The geometric convolutional layer augments conventional convolution by concatenating X, Y, and Z position channels to its input.The added channels encode a 3D position map derived from sparse depth.

B. The Geometric Convolutional Layer

The geometric convolutional layer augments convolution with a 3D position map derived from sparse depth, allowing geometric information to enter both branches at multiple scales.

  • The layer concatenates a 3D position map with a conventional convolutional layer’s input to encode geometric information.The position map contains X, Y, and Z coordinates.
  • The position map is derived from sparse depth using pixel coordinates and camera intrinsic parameters.The coordinates are X, Y, and Z; u, v, u0, v0, fx, and fy define the projection.
  • Each ResBlock convolution is replaced with geometric convolution, while min-pooled sparse depth supplies Z at smaller scales.This encodes 3D geometric information into features in both color- and depth-dominant branches.

C. The Dilated and Accelerated CSPN++

The refinement module adapts CSPN++ with dilated neighborhoods and tensor-level parallel propagation to refine depth maps more effectively and efficiently.

  • The method refines the backbone’s depth map with CSPN++ because predicted maps may fail to preserve valid input depth values.The refinement adds a dilation strategy and a parallel implementation.
  • At each iteration, every pixel aggregates propagated information from pixels in its neighborhood using learned affinities.The coarse depth map is denoted D0, and the refined map after t iterations is Dt.
  • For a k×k neighborhood, the network learns k×k affinity maps, each representing one neighbor’s affinity to all pixels.Each affinity map is translated in the opposite neighbor direction for alignment.
  • One-hot convolutional kernels translate affinity maps in parallel, converting pixel-wise propagation into a more efficient tensor-level operation.The accelerated implementation conducts propagation fully in parallel.

D. The Training Loss

The training loss compares predicted and ground-truth depth only at pixels with valid ground-truth values, with intermediate predictions also supervised early in training.

  • The loss measures the difference between the predicted depth map and ground truth after masking invalid ground-truth pixels.The mask is 1(Dgt > 0), and ⊙ denotes element-wise multiplication.
  • Intermediate depth predictions receive additional supervision during the early training epochs.
  • The intermediate-loss weights λcd and λdd are empirically set hyper-parameters.

IV. EXPERIMENTS

Experiments evaluate the model and variants on KITTI depth completion using standard depth metrics, inference runtime, and a staged training implementation.

  • The KITTI dataset provides aligned color images and sparse depth maps from projected 3D LiDAR points.It contains 86K training frames, 7K validation frames, and 1K test frames at 1216 × 352 resolution.
  • Sparse depth maps contain about 5% valid pixels, while dense ground truth contains around 16% valid pixels.
  • Evaluation reports RMSE, MAE, iRMSE, iMAE, and inference runtime.Depth errors use millimeters, inverse-depth errors use 1/km, and runtime is also reported.
  • Training uses ADAM and progressively trains the backbone, DA-CSPN++, and full model with separate stages.The backbone and DA-CSPN++ are trained before joint full-model training.

B. Ablation Studies

Ablations show that the two-branch design, geometric encoding, and accelerated refinement each contribute to PENet’s depth-completion performance and efficiency.

  • Two-branch backbone: The two-branch backbone improves performance when the CD branch receives sparse depth and the DD branch receives CD-depth predictions.
  • Two-branch backbone: An additional guidance map from the first branch is unnecessary and slightly hurts performance.
  • Qualitative analysis: DD-depth contributes more to fused maps in most regions, whereas CD-depth is more reliable near object boundaries but sensitive to color and texture.
  • Geometric convolutional layer: Geometric convolution improves B4’s RMSE by a great margin, while depth-only encoding helps and pixel-coordinate encoding may slightly hurt.
  • Geometric convolutional layer: GCL infers better depth information than standard convolution and CoordConv, especially when foreground and background colors are similar.
  • Depth refinement: All refinement variants greatly improve the backbone, with B4+C2 slightly outperforming the other counterparts.
  • Depth refinement: The accelerated refinement implementation greatly reduces spatial-propagation runtime on a single 2080Ti GPU.

C. Comparison with State-of-the-arts

PENet achieves the top KITTI depth-completion leaderboard rank at submission and combines strong RMSE performance with faster inference than most compared methods.

  • PENet ranks 1st on the KITTI online leaderboard at submission and significantly improves RMSE, the most important evaluation metric.
  • ENet, the geometric encoded backbone without refinement, outperforms 9 top-ranked methods, including methods using spatial propagation.
  • The full model infers faster than 8 compared methods and is much faster than spatial-propagation methods in the tested comparison.

V. CONCLUSION

The paper presents a two-branch architecture that separately exploits color-dominant and depth-dominant information for image guided depth completion. Together with geometric encoding and DA-CSPN++ refinement, the model is precise and efficient on the KITTI online leaderboard.

  • The proposed two-branch architecture exploits color- and depth-dominant information, respectively, to fuse multimodal cues thoroughly.
  • A geometric convolutional layer augments the backbone with 3D geometric cues for multimodal depth completion.
  • The model integrates a speedup DA-CSPN++ module for further depth refinement.
  • The entire model is reported as precise and efficient on the KITTI online leaderboard.
Loading 2103.00783v3…