PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Guided Convolutional Network for Depth Completion

Jie Tang, Fei-Peng Tian, Wei Feng, Jian Li, Ping Tan

arXiv:1908.01238v1cs.CV

TL;DR

Sparse LiDAR measurements make dense depth completion necessary, and existing methods often fuse RGB and depth features with simple operations. The paper learns content-dependent spatially variant kernels from RGB guidance, factorizes their convolution for efficient multi-stage fusion, and reports strong benchmark and generalization results.

  • Problem

    LiDAR provides sparse depth measurements, while many existing RGB-guided methods fuse heterogeneous image and depth information through simple addition or concatenation.

  • Method

    The method predicts content-dependent, spatially variant kernels from RGB guidance and applies them to depth features, using convolution factorization to reduce GPU memory consumption.

  • Results

    The method reports strong results on indoor and outdoor real-world and synthetic datasets, including state-of-the-art KITTI and NYUv2 performance and strong cross-condition generalization.

  • Takeaways & Limitations

    Learned guided convolution provides a multi-stage RGB-depth fusion strategy for completing sparse and irregular LiDAR depth.

Abstract

from arXiv · show

Dense depth perception is critical for autonomous driving and other robotics applications. However, modern LiDAR sensors only provide sparse depth measurement. It is thus necessary to complete the sparse LiDAR data, where a synchronized guidance RGB image is often used to facilitate this completion. Many neural networks have been designed for this task. However, they often naïvely fuse the LiDAR data and RGB image information by performing feature concatenation or element-wise addition. Inspired by the guided image filtering, we design a novel guided network to predict kernel weights from the guidance image. These predicted kernels are then applied to extract the depth image features. In this way, our network generates content-dependent and spatially-variant kernels for multi-modal feature fusion. Dynamically generated spatially-variant kernels could lead to prohibitive GPU memory consumption and computation overhead. We further design a convolution factorization to reduce computation and memory consumption. The GPU memory reduction makes it possible for feature fusion to work in multi-stage scheme. We conduct comprehensive experiments to verify our method on real-world outdoor, indoor and synthetic datasets. Our method produces strong results. It outperforms state-of-the-art methods on the NYUv2 dataset and ranks 1st on the KITTI depth completion benchmark at the time of submission. It also presents strong generalization capability under different 3D point densities, various lighting and weather conditions as well as cross-dataset evaluations. The code will be released for reproduction.

I. INTRODUCTION

Dense depth is important for robotics, but LiDAR provides sparse measurements that are insufficient for many applications. The paper introduces guided, content-dependent feature fusion and factorized convolution to enable efficient multi-stage depth completion.

  • Motivation: LiDAR provides reliable depth sensing but only sparse measurements, making dense depth estimation valuable for robotic navigation and related applications.The paper gives 64 vertical scan lines as an example of LiDAR sparsity.
  • Motivation: Existing depth-completion methods commonly use synchronized RGB images to improve sparse-depth prediction.Prior approaches use RGB cues such as surface normals, photo-consistency, and semantic supervision.
  • Proposed approach: The proposed guided network predicts spatially variant convolution kernels from RGB features and applies them to sparse-depth features for multi-modal fusion.This design is inspired by guided image filtering but learns the kernel-generation process end to end.
  • Efficiency: Convolution factorization separates spatially variant channel-wise convolution from spatially invariant cross-channel convolution to reduce GPU memory consumption.The reduction enables guided convolution to operate within multi-stage fusion in an encoder-decoder network.
  • Results: Experiments on outdoor and indoor real-world and synthetic datasets report state-of-the-art performance on KITTI and NYUv2, with strong generalization across conditions.The reported generalization settings include different point densities, lighting, weather, and cross-dataset evaluations.

II. RELATED WORK

Prior depth-completion work includes depth-only and image-guided approaches, while RGB-depth fusion is often implemented with simple addition or concatenation. This paper instead learns guided kernels and extends dynamic filtering to multi-stage multimodal fusion.

  • Method categories: Depth-completion methods are broadly categorized as depth-only methods and image-guided methods.The distinction depends on whether an RGB image guides completion.
  • Depth-only methods: Depth-only methods range from handcrafted operators and classical reconstruction to deep networks that infer dense depth from sparse or low-resolution inputs.Related depth super-resolution methods also use databases, self-similarity, dictionary learning, or variational refinement.
  • Image-guided methods: Image-guided methods exploit RGB-derived semantic, edge, surface, or temporal cues and generally achieve better results than depth-only approaches.Examples include bilateral filtering, surface-normal prediction, photo-consistency, and sparsity-invariant operations.
  • Fusion gap: Most RGB-guided depth-completion methods fuse image and depth features through simple concatenation or element-wise addition.The paper identifies this shared fusion strategy as the main motivation for its guided convolution module.
  • Guided filtering: Unlike differentiable guided filtering based on a fixed closed-form kernel equation, this method learns more general kernels from the guidance image for sparse LiDAR completion.The related guided-filtering operator is described as unsuitable for sparse LiDAR inputs.
  • Dynamic filtering: The approach extends dynamic filtering toward multi-stage multimodal fusion, addressing the GPU-memory limits that previously restricted spatially variant kernels.The paper combines this extension with convolution factorization and studies fusion strategies experimentally.

III. THE PROPOSED METHOD

The method completes sparse depth using RGB guidance by learning spatially variant filters that transform depth features. It also motivates learned filtering from the limitations of handcrafted guided filtering on sparse inputs.

  • Problem formulation: Depth completion maps a sparse depth image S and guidance RGB image I to a dense depth map D.The RGB image supplies object-boundary and scene-content information.
  • Guided filtering: Guided image filtering computes each output pixel from a weighted local average of sparse-depth values using guidance-dependent kernel weights.The weights are associated with pixels i and j within a local neighborhood N(i).
  • Learned guidance: Because handcrafted guided filtering does not work well on sparse and irregular LiDAR inputs, the method learns more general filter kernels from the guidance image.The learned kernels are applied to fuse multimodal features rather than directly filtering the sparse input.

B. Guided Convolution Module

The guided convolution module generates content-dependent, spatially variant kernels from RGB guidance features and applies them to sparse-depth features. A two-stage factorization reduces the memory and computation costs that otherwise make multi-stage fusion impractical.

  • The module generates convolution kernels from guidance-image features and applies them to sparse-depth features for depth-feature extraction.The kernel-generating layer uses differentiable operations, preferably convolution layers for grid images.
  • Content-dependent, spatially variant kernels allow different image regions and objects to receive different filters according to their depth distributions.
  • The factorization splits guided convolution into channel-wise spatially variant convolution followed by cross-channel 1 × 1 convolution.The first stage remains spatially variant; the second stage is content-dependent but spatially invariant and shared across pixels.
  • Naïve spatially variant convolution requires a kernel of size M × N × K^2 × H × B, making memory consumption unaffordable at large feature resolutions and channel counts.
  • The factorization reduces per-output computation from O(M × K^2) to O(M + K^2).The two component operations have complexities O(K^2) and O(M), respectively.
  • 10.7GB to 0.08GB: with M = N = 128, H = 64, B = 304, and K = 3, the proposed two-stage convolution reduces single-layer GPU memory nearly 128 times.This reduction enables the guided convolution module to operate at multiple network scales.

D. Network Architecture

The network uses separate encoder-decoder branches for RGB guidance and sparse LiDAR depth, with hierarchical RGB features generating kernels for depth representations. Decoder-to-encoder fusion is performed repeatedly across stages.

  • Two encoder-decoder networks process the RGB image and sparse LiDAR depth image as GuideNet and DepthNet, respectively.GuideNet learns hierarchical RGB representations containing low-level and high-level information.
  • GuideNet image features generate spatially variant, content-dependent kernels that guide depth feature representations during fusion.
  • The architecture fuses GuideNet decoder features with DepthNet encoder features in multiple stages rather than using only early or late fusion.Decoder features are preferred because they contain more high-level context information; multi-stage fusion shows stronger and more reliable results.

1) Loss Function:

Training uses mean squared error between predicted and ground-truth depth, while experiments cover outdoor, indoor, real-world, synthetic, and varied-condition datasets. The supplied passages also describe KITTI qualitative comparisons and training details.

  • 1) Loss Function:: The loss is mean squared error computed between predicted and ground-truth depth values at valid pixels.For real-world data, invalid or missing ground-truth pixels are excluded from the loss.
  • The model is trained end-to-end from scratch with ADAM, a starting learning rate of 10^-3, weight decay of 10^-6, and learning-rate halving every 50k iterations.Training uses two GTX 1080Ti GPUs with batch size 8 and synchronized cross-GPU batch normalization.
  • Experiments span outdoor and indoor datasets from real-world and synthetic scenes, including KITTI, NYUv2, and varied-density, lighting, weather, and cross-dataset settings.
  • In the KITTI test-set qualitative comparison, the method is reported to recover better 3D details than several top-ranking methods in zoomed regions.

B. Evaluation Metrics

The evaluation uses standard depth-completion metrics for outdoor and indoor scenes, with RMSE serving as the principal ranking metric. Experiments include KITTI test comparisons and visual analysis of predicted depth and guided kernels.

  • Evaluation Metrics: Outdoor evaluation uses RMSE, MAE, iRMSE, and iMAE, with RMSE as the dominant KITTI leaderboard metric.RMSE and MAE measure depth accuracy directly; inverse-depth metrics evaluate errors in inverse depth.
  • Evaluation Metrics: Indoor evaluation uses RMSE, REL, and δi at thresholds 1.25, 1.252, and 1.253.Higher δi indicates better prediction, while larger i represents a softer relative-error constraint.
  • KITTI Evaluation: The method ranks first on the KITTI test server under the primary RMSE metric and performs comparably on other evaluation metrics.The comparison uses state-of-the-art published methods evaluated on the KITTI testing server.
  • KITTI Evaluation: KITTI visual comparisons show more accurate object-boundary details, including recovered background depth between a person’s arms.The method’s results are compared with several state-of-the-art methods on the test set.
  • Guided-Kernel Analysis: Guided-kernel visualization assigns each K×K kernel a Prewitt-derived 2D vector, with similar colors appearing along similar gradients or surface normals.The visualization supports content-dependent, spatially variant guidance in the early fusion stage.

D. Ablation Studies

Ablation studies compare guided convolution against conventional fusion and alternative encoder-decoder fusion placements. The results favor guided, content-dependent fusion and decoder GuideNet features guiding encoder DepthNet features.

  • Comparison with Feature Addition/Concatenation: Replacing guided convolution with addition or concatenation increases RMSE by 31.59 mm and 24.35 mm, respectively.All other network components and settings are kept unchanged in this comparison.
  • Comparison with Feature Addition/Concatenation: Addition performs slightly worse than concatenation because image and depth features are heterogeneous data from different sources.The paper notes that concatenation is commonly used for heterogeneous image-depth fusion.
  • Fusion Scheme of GuideNet and DepthNet: Decoder GuideNet features guiding encoder DepthNet features outperform encoder-to-encoder and decoder-to-decoder fusion alternatives.The paper attributes the advantage to higher-level context in decoder image features and earlier influence on sparse-depth feature extraction.
  • Fusion Scheme of GuideNet and DepthNet: Encoder-to-encoder fusion still outperforms conventional addition or concatenation, supporting guided convolution as an effective fusion module.This variant uses a weaker fusion strategy than the final placement while retaining guided convolution.

3) Fusion Scheme of Multi-stage Guidance:

Multi-stage guidance is more effective than retaining guided fusion only at the first or last stage. On NYUv2, the method also leads across both sparse-sample settings and all five reported metrics.

  • Fusion Scheme of Multi-stage Guidance: Both First Guide and Last Guide perform worse than the full multi-stage guidance scheme.First Guide retains guided fusion only in the first stage, whereas Last Guide retains it only in the last stage.
  • Fusion Scheme of Multi-stage Guidance: First Guide performs slightly better than Last Guide, consistent with more effective image guidance at earlier stages.The paper links early-stage features to subsequent feature extraction, especially for sparse depth images.
  • Fusion Scheme of Multi-stage Guidance: Both partial-guidance variants outperform concatenation, reinforcing the advantage of guided convolution for feature fusion.The comparison uses concatenation in the remaining fusion stages.
  • NYUv2 Evaluation: On NYUv2, the method outperforms all compared methods with both 200 and 500 sparse LiDAR samples across all five evaluation metrics.The network is evaluated without dataset-specific modification.
  • NYUv2 Evaluation: NYUv2 qualitative comparisons show sharper local transitions than Ma et al. and fewer detail losses around thin object boundaries than NConv-CNN.The comparison covers both 200-sample and 500-sample settings.

F. Generalization Capability

The guided network generalizes across sparse LiDAR inputs and challenging lighting and weather conditions. It remains consistently strong relative to the evaluated fusion variants and competing methods.

  • 1) Different Point Densities:: The model degrades gradually as LiDAR point density decreases and remains consistently better than NConv-CNN.Evaluation uses the same model trained only on KITTI, without fine-tuning.
  • 2) Various Lighting and Weather Conditions:: The guided network achieves the best RMSE under clone, fog, morning, overcast, rain, and sunset conditions.Its RMSE remains stable across all evaluated lighting and weather situations.

3) Cross-dataset Evaluation:

Cross-dataset evaluation tests models trained on NYUv2 directly on SUN RGBD using Kinect V1 and Xtion data. The method retains the best RMSE and shows strong transfer, although performance is slightly worse on Xtion data collected with a different device.

  • 3) Cross-dataset Evaluation:: Models trained on NYUv2 are directly evaluated on SUN RGBD to assess cross-dataset generalization.The evaluation includes 500-sample and 200-sample settings.
  • 3) Cross-dataset Evaluation:: The method achieves the best RMSE on SUN RGBD and reports results close to those obtained on NYUv2.This holds for the reported cross-dataset evaluation results.
  • 3) Cross-dataset Evaluation:: All three methods perform slightly worse on Xtion-collected data than on the other evaluated dataset.The passage attributes this difference to camera intrinsic and extrinsic parameters.
Loading 1908.01238v1…