PaperlayerView exact revision on arXiv

Source-linked AI summary

Adaptive Context-Aware Multi-Modal Network for Depth Completion

Shanshan Zhao, Mingming Gong, Huan Fu, Dacheng Tao

arXiv:2008.10833v1cs.CV

TL;DR

Sparse and irregular depth observations make standard convolution ineffective at modeling the contexts needed for dense depth completion. ACMNet uses attention-guided graph propagation and symmetric gated fusion to extract and combine multimodal context, achieving effective performance on KITTI and NYU-v2 while using fewer parameters than recent models.

  • Problem

    Standard convolution is ineffective at modeling observed spatial contexts because sparse depth values are irregularly distributed, limiting depth completion from sparse depth and RGB.

  • Method

    ACMNet applies attention-guided graph propagation across multiscale graphs of observed pixels in both modalities, then uses symmetric gated fusion to combine their features.

  • Results

    ACMNet achieves state-of-the-art performance on KITTI and NYU-v2, with fewer parameters than recent models.

  • Takeaways & Limitations

    The network generates high-quality dense depth maps by adaptively extracting contextual information and exploiting complementary depth and RGB representations.

Abstract

from arXiv · show

Depth completion aims to recover a dense depth map from the sparse depth data and the corresponding single RGB image. The observed pixels provide the significant guidance for the recovery of the unobserved pixels' depth. However, due to the sparsity of the depth data, the standard convolution operation, exploited by most of existing methods, is not effective to model the observed contexts with depth values. To address this issue, we propose to adopt the graph propagation to capture the observed spatial contexts. Specifically, we first construct multiple graphs at different scales from observed pixels. Since the graph structure varies from sample to sample, we then apply the attention mechanism on the propagation, which encourages the network to model the contextual information adaptively. Furthermore, considering the mutli-modality of input data, we exploit the graph propagation on the two modalities respectively to extract multi-modal representations. Finally, we introduce the symmetric gated fusion strategy to exploit the extracted multi-modal features effectively. The proposed strategy preserves the original information for one modality and also absorbs complementary information from the other through learning the adaptive gating weights. Our model, named Adaptive Context-Aware Multi-Modal Network (ACMNet), achieves the state-of-the-art performance on two benchmarks, {\it i.e.}, KITTI and NYU-v2, and at the same time has fewer parameters than latest models. Our code is available at: \url{https://github.com/sshan-zhao/ACMNet}.

1. Introduction

Depth completion must infer dense depth from highly sparse, irregular LiDAR observations and an RGB image. ACMNet addresses convolution’s limited use of observed contexts with adaptive graph propagation and symmetric gated multimodal fusion.

  • Motivation: KITTI LiDAR provides depth for only 5.9% of image pixels, making dense depth recovery from sparse observations challenging.The sparse map is formed by projecting LiDAR data onto the image plane, with missing locations assigned zero.
  • Motivation: Regular convolution kernels ignore irregular observed-depth distributions and associate limited observed contexts with unobserved pixels.This lack of adaptation can produce sub-optimal depth predictions at unobserved locations.
  • Method: ACMNet uses attention-guided graph propagation over multiple graphs built from observed pixels to enhance unobserved-pixel features with spatial context.Compared with convolution alone, graph propagation followed by convolution lets unobserved pixels capture more related observed information.
  • Method: Co-attention-guided propagation extracts multi-scale multimodal features, while symmetric gated fusion combines depth and RGB representations adaptively.Each fusion branch preserves one modality’s information while incorporating complementary information from the other.
  • Evaluation: ACMNet’s effectiveness is demonstrated on the KITTI Depth Completion and NYU-v2 datasets.The introduction identifies both as evaluation benchmarks.

2. Related Work

Prior depth-completion methods progressed from handcrafted optimization to CNN-based learning, while graph models and multimodal fusion provide relevant foundations for ACMNet.

  • Depth Completion: Traditional depth-completion approaches optimize energy functions with handcrafted features, but their performance is limited.CNN-based methods subsequently outperformed these traditional approaches by a wide margin.
  • Depth Completion: CNN depth-completion methods address sparse LiDAR through sparsity-invariant convolutions, geometric cues, self-supervision, and learned depth priors.Examples include surface-normal prediction and stereo-based self-supervised learning.
  • Monocular Depth Estimation: Monocular depth estimation evolved from graphical models with handcrafted features toward deep learning approaches, including supervised methods.The cited development reflects a broad transition in modeling strategies.
  • Graph-based Models: Graph models represent target nodes by propagating neighborhood information in a data-driven way across computer-vision tasks.The related work cites applications including action recognition, point-cloud analysis, few-shot classification, and person re-identification.
  • Multi-modal Information Fusion: Multimodal vision systems commonly fuse modalities by concatenation or summation, although different modalities can provide different information.This motivates more adaptive fusion strategies for heterogeneous inputs.

3.1. Problem Formulation

The task is to learn a mapping from paired sparse depth and RGB inputs to a ground-truth dense depth map. ACMNet implements this objective with graph propagation and multimodal fusion modules.

  • Problem Formulation: ACMNet receives a sparse depth map and a single RGB image and predicts a dense depth map.The sparse depth, RGB image, and target depth are denoted XS, XI, and Y, respectively.
  • Problem Formulation: The learning objective is Y = f(XS, XI), where XS ∈ R^H×W, XI ∈ R^3×H×W, and Y ∈ R^H×W.The mapping is learned from a set of paired samples.
  • Problem Formulation: The network is built around co-attention-guided graph propagation and symmetric gated fusion modules.These modules respectively support contextual representation learning and multimodal complementarity modeling.

3.2. Network Architecture

ACMNet uses a two-stream encoder-decoder that extracts multiscale features with CGPM and fuses depth and RGB representations with SGFM during decoding.

  • Network Architecture: The overall architecture follows a two-stream encoder-decoder for sparse depth and RGB inputs, enhanced with CGPM and SGFM modules.The encoder and decoder process the two modalities before producing the recovered dense depth map.
  • Encoder: The encoder learns discriminative multiscale features from sparse depth and RGB, using CGPM to capture contextual information from observed spatial contexts.This targets the difficulty of extracting rich information from extremely sparse observations.
  • Decoder: The decoder predicts unobserved depth values from multiscale multimodal encoder features.It addresses the challenge of exploiting complementary representations rather than directly concatenating or summing them.
  • Decoder: SGFM combines depth and RGB features progressively at decoder levels to model their complementarity.The extracted features are fused through SGFM before the branches are combined into the final dense depth map.
  • Decoder: The final recovered dense depth map is obtained using either end-integration or feature-integration to combine the two branches.The two alternatives are described as methods for combining the branches after decoder processing.

3.3. Co-Attention Guided Graph Propagation (CGPM)

CGPM constructs scale-specific graphs over observed depth pixels and uses attention-guided propagation to adaptively aggregate contextual information for feature learning and enhancement of unobserved pixels.

  • Graph construction: At each scale, CGPM constructs an individual graph whose nodes are pixels with observed depth values, connecting each node to its k nearest spatial neighbours.The graph is built separately at each resolution, with lower-resolution sparse depth obtained by max-pooling down-sampling.
  • Propagation pipeline: CGPM first propagates features among observed pixels, then enhances unobserved-pixel features using convolution and a residual connection.The two stages respectively aggregate neighbour information and extend the resulting context to unobserved locations.
  • Adaptive weighting: Self-attention learns adaptive weights between neighbouring nodes by modelling their relationships.These weights determine how neighbouring observed features are aggregated during graph propagation.
  • Multi-modal encoding: The same graph-propagation procedure is applied to the depth stream, while the image stream uses the analogous process to learn discriminative features.The encoder therefore performs adaptive contextual feature extraction separately for both input modalities.
  • Co-attention: In the encoder, co-attention learns depth-stream and image-stream propagation weights from both modalities' information.The resulting CGPMs extract multi-scale, multi-modal features while using RGB and sparse depth jointly to guide contextual propagation.

3.4. Symmetric Gated Fusion (SGFM)

SGFM adaptively fuses depth and RGB representations by preserving one branch while absorbing complementary information from the other through learned gating, symmetrically in both directions.

  • Fusion motivation: SGFM uses depth geometry and RGB semantics as complementary sources when fusing the two modality-specific feature representations.Depth describes scene geometry and distances, whereas RGB supplies semantic and appearance information for unobserved pixels.
  • Depth-branch fusion: Adaptive gating weights filter unrelated RGB information before injecting useful content into the depth features.The gated fusion produces an initial combined feature that is then processed by a residual block at applicable decoder levels.
  • Symmetric fusion: The enhanced depth features are upsampled through deconvolution, and the symmetric procedure improves the image branch with complementary depth information.At lower levels, gating weights are learned for the corresponding features; the same fusion design is applied in reverse.
  • Strategy comparison: The implementation compares symmetric gated fusion with direct fusion and direct attention fusion across encoder and decoder features.The paper presents visual and quantitative comparisons of these alternatives.

3.5. Branch Integration

After symmetric fusion, ACMNet integrates the two branch outputs either at the prediction stage using confidence maps or progressively through intermediate feature integration.

  • Integration options: The two SGFM branches produce image- and depth-branch features that can be combined using end-integration or feature-integration.Both alternatives are used to obtain the final prediction.
  • Illustrations: Figure 5 presents alternative fusion strategies, while Figure 6 illustrates feature-integration with some SGFM inputs omitted for simplicity.The implementation considers features from both encoder and decoder stages.
  • End-integration: In end-integration, each branch predicts a dense depth map and a confidence map indicating the reliability of that prediction.The branch predictions are therefore weighted adaptively across the image plane before producing the final dense depth map.
  • Feature-integration: Feature-integration progressively fuses intermediate branch features through convolutional operations before one final convolution predicts the depth map.The intermediate features QSI and QIS are combined into QF before generating the final prediction.

3.6. Loss Function

The network is trained primarily with a masked MSE loss on labeled depth pixels, supplemented by an edge-aware smoothness loss and combined through a weighted full objective.

  • The primary training loss is a masked MSE between the semi-dense ground-truth depth map and the prediction.
  • An edge-aware smoothness loss encourages predicted depths to preserve spatial continuity.

4. Experiments

Experiments evaluate ACMNet on KITTI and NYU-v2, compare it with prior methods, and ablate graph propagation, fusion, and graph-construction choices. ACMNet performs competitively while using fewer parameters, and its components improve qualitative and quantitative depth completion results.

  • KITTI Dataset: ACMNet yields close KITTI performance to CSPN++ and NLSPN with a much smaller model size.NLSPN and CSPN++ have RMSE errors 3mm and 1mm lower than ACMNet, respectively, but approximately four times more parameters.
  • KITTI Dataset: ACMNet preserves more boundary details and performs better on tiny or thin objects than four compared state-of-the-art methods in KITTI qualitative examples.Figure 7 compares DeepLiDAR, Certainty, PwP, Sparse2dense, and ACMNet using predictions, zoomed details, and error maps.
  • NYU-v2 Dataset: ACMNet performs better than almost all methods on NYU-v2 except NLSPN, while using far fewer parameters than NLSPN.The NYU-v2 comparison uses 500 sparse depth samples.
  • Ablation Study: Graph propagation and symmetric gated fusion outperform corresponding direct-fusion or no-propagation configurations in the ablation studies.The comparisons include Baseline, +GP, +SG, +GP+SG, DF, DAF, and SG configurations.
  • Ablation Study: Using 3D coordinates with six nearest neighbours slightly outperforms the corresponding 2D-coordinate graph on RMSE.The authors attribute this difference to learning the scene’s geometric structure in camera coordinates.
  • Ablation Study: Coordinate system, neighbour count, and sampled-point count affect RMSE, but the model performs well in most settings.Using too many neighbours may introduce unrelated contexts, while more or fewer sampled points may degrade performance.
  • Analysis of branch integration: Feature-Integration achieves RMSE 786 versus 789 for End-Integration, indicating stronger learning of branch reliability at the feature level.The comparison evaluates the two integration methods in Table 6.
  • Analysis of branch integration: Confidence-guided fusion substantially improves both branches because the depth branch supplies dense predictions while the image branch captures boundary information.The two branches obtain close individual scores, but learning confidence maps to combine them improves all metrics.

5. Conclusion

The conclusion presents ACMNet as a depth-completion network that combines adaptive graph propagation with symmetric gated fusion. The reported experiments support effective dense-depth prediction and component effectiveness, while sparsity-level comparisons show consistent gains over several methods.

  • 5. Conclusion: ACMNet recovers dense depth maps from sparse LiDAR data and dense RGB data.
  • 5. Conclusion: ACMNet performs better than Certainty, Sparse2dense, and NConv-CNN across all tested input densities.The comparison reports results under different sparsity levels, including lower and larger densities.
  • 5. Conclusion: Co-attention guided graph propagation uses observed-pixel graphs to adaptively extract multi-scale and multi-modal features for enhancing unobserved pixels.
  • 5. Conclusion: Symmetric gated fusion learns the heterogeneity of the two modalities to combine their extracted features effectively.
  • 5. Conclusion: The authors report high-quality dense depth maps and experimental evidence for the effectiveness of the network and its components.
Loading 2008.10833v1…