PaperlayerView exact revision on arXiv

Source-linked AI summary

Channel-Wise Attention-Based Network for Self-Supervised Monocular Depth Estimation

Jiaxing Yan, Hong Zhao, Penghui Bu, YuSheng Jin

arXiv:2112.13047v1cs.CVcs.AIcs.LG

TL;DR

Self-supervised monocular depth estimation still lacks explicit scene-structure modeling and effective local-detail handling. CADepth-Net addresses these gaps with structure perception and detail emphasis modules based on channel-wise attention, achieving state-of-the-art results on KITTI and Make3D.

  • Problem

    Recent self-supervised monocular depth methods lack explicit modeling of scene structure and proper handling of local details, limiting scene understanding and producing blurry artefacts.

  • Method

    CADepth-Net uses self-attention for long-range channel dependencies and channel attention for feature re-calibration, detail emphasis, and multilevel feature fusion.

  • Results

    CADepth-Net achieves state-of-the-art results on the KITTI benchmark and Make3D datasets, with sharper depth estimation reported in experiments.

  • Takeaways & Limitations

    The proposed attention modules improve scene-structure representation and emphasize object-boundary details for sharper depth predictions.

  • Takeaways & Limitations

    The self-supervised training assumes a moving camera and static scene, requiring auto-masking to filter stationary pixels when these assumptions are violated.

Abstract

from arXiv · show

Self-supervised learning has shown very promising results for monocular depth estimation. Scene structure and local details both are significant clues for high-quality depth estimation. Recent works suffer from the lack of explicit modeling of scene structure and proper handling of details information, which leads to a performance bottleneck and blurry artefacts in predicted results. In this paper, we propose the Channel-wise Attention-based Depth Estimation Network (CADepth-Net) with two effective contributions: 1) The structure perception module employs the self-attention mechanism to capture long-range dependencies and aggregates discriminative features in channel dimensions, explicitly enhances the perception of scene structure, obtains the better scene understanding and rich feature representation. 2) The detail emphasis module re-calibrates channel-wise feature maps and selectively emphasizes the informative features, aiming to highlight crucial local details information and fuse different level features more efficiently, resulting in more precise and sharper depth prediction. Furthermore, the extensive experiments validate the effectiveness of our method and show that our model achieves the state-of-the-art results on the KITTI benchmark and Make3D datasets.

1. Introduction

The paper targets incomplete scene-structure perception and blurry depth boundaries in self-supervised monocular estimation. CADepth-Net addresses both issues with channel-wise attention modules and reports state-of-the-art KITTI performance.

  • Motivation: Self-supervised monocular depth estimation avoids the large, costly ground-truth collections required by fully supervised methods.It uses geometrical constraints from monocular video or synchronized stereo image pairs as supervision.
  • Motivation: Prior methods implicitly learn scene structure with convolutions, limiting their perception of overall 3D layouts in complex scenes.The paper identifies insufficient explicit exploration of robust 3D scene geometry as the underlying gap.
  • Motivation: Naive decoder fusion of high- and low-level features fails to preserve details, producing inefficient integration and blurry artefacts near depth discontinuities.This motivates explicit handling of local details and spatial information.
  • Method: CADepth-Net combines a structure perception module with a detail emphasis module to handle scene structure and local details.The network uses channel-wise attention for information aggregation and feature re-calibration.
  • Method: The structure perception module uses self-attention to capture long-range dependencies, aggregate channel features, and enrich scene-structure representations.It fuses local depth responses from non-contiguous regions through weighted aggregation across channel maps.
  • Method: The detail emphasis module uses channel attention to fuse different-scale features and emphasize important details for sharper depth estimation.Its purpose is to improve object-boundary recovery and reduce blurry predictions.
  • Results: Extensive experiments on KITTI and Make3D show that CADepth-Net outperforms existing methods and achieves state-of-the-art results on the KITTI benchmark.The introduction reports the result across both datasets, with KITTI identified as the state-of-the-art benchmark.

2. Related Work

Related work spans supervised depth estimation, self-supervised depth-and-motion learning, and self-attention for long-range dependencies. The framework overview situates CADepth-Net within a U-Net pipeline combining semantic extraction, attention modules, multiscale decoding, and pose estimation.

  • 2.1. Supervised Depth Estimation: Supervised monocular depth methods estimate dense depth from color images but require costly high-quality ground-truth depth.Early work used multiscale networks for coarse global and locally refined predictions.
  • 2.2. Self-supervised Depth Estimation: Self-supervised methods jointly learn depth and egomotion, using view synthesis from monocular video as the supervision signal.Subsequent work modeled non-rigid motion and incorporated direct visual odometry for relative camera pose.
  • 2.3. Self-attention Mechanism: Self-attention has been used to aggregate global context and model long-range dependencies across images, videos, and generative models.The cited applications include spatial-temporal dependency modeling and improved image generation.
  • Framework Overview: CADepth-Net uses a ResNet encoder, structure perception module, detail emphasis module, multiscale UpConv decoding, and a pose network.The pose network takes temporally adjacent images and outputs relative pose for the self-supervised framework.

3. Method

CADepth-Net combines self-supervised view-synthesis training with channel-wise modules that explicitly model scene structure and emphasize local details during multiscale decoding.

  • Self-Supervised Training: The model predicts depth and relative pose from adjacent monocular frames, using synthesized views and photometric reprojection error for joint self-supervision.An auto-masking method filters stationary pixels, while edge-aware smoothness regularizes disparities in textureless regions.
  • Channel-wise Attention-based Network: CADepth-Net uses a fully convolutional U-Net with a pretrained residual backbone, skip connections, multiscale decoding, and progressive inverse-depth upsampling.The decoder sequentially applies detail emphasis modules before producing predictions at the original input resolution.
  • Structure Perception Module: The structure perception module models relationships among channel maps to aggregate complementary regional responses and encode richer scene-structure context.Self-attention computes channel-map similarities and discriminative relationships, then forms each output channel from weighted features across channels plus the original feature.
  • Structure Perception Module: The final aggregated structure features encode long-range dependencies between feature maps, while detail recalibration produces more precise depth estimation.Together, the modules target incomplete scene-layout perception and blurry depth discontinuities caused by simpler feature processing.
  • Detail Emphasis Module: The detail emphasis module concatenates low- and high-level features, then uses channel attention to reweight informative responses and fuse details across scales.Global average pooling and convolutional gating produce channel weights; residual summation improves stability and supports sharper edges.

4. Experiments

Experiments evaluate CADepth-Net on KITTI and Make3D, including module visualizations, ablations, and cross-dataset generalization. The results report sharper depth predictions, improved scene-structure and detail representations, state-of-the-art comparisons, and real-time inference characteristics.

  • CADepth-Net significantly outperforms existing self-supervised approaches across KITTI evaluation metrics.
  • The model produces sharper boundaries and finer details on thin structures, including road signs, poles, trees, and pedestrians.It also estimates depth on a reflective car roof, identified as challenging for previous advanced methods.
  • Make3D Results: When trained on KITTI and evaluated on unseen Make3D data, CADepth-Net produces superior, more accurate, and sharper depth estimates than other self-supervised methods.The Make3D evaluation uses center cropping and median scaling.
  • Structure Perception Module: Structure perception aggregates discriminative channel features to add contextual information from distant regions, including foreground, midground, and vanishing-point areas.The resulting representations support richer scene-geometry perception and more complete region responses.
  • Detail Emphasis Module: Detail emphasis re-calibrates channel maps and prioritizes informative low-level features that describe object boundaries precisely.The module uses feature weights between 0 and 1 and reports the top n = 10 feature maps.
  • Ablation Study: Ablations show steady improvements across almost all evaluation measures and backbones, while the combined ResNet50 model achieves 59M parameters and 28ms inference on an RTX3090 GPU.The structure perception module improves δ < 1.25 with little time cost and no additional parameters.

5. Conclusion

The paper presents CADepth-Net with structure perception and detail emphasis modules for self-supervised monocular depth estimation. Experiments report sharper predictions and state-of-the-art KITTI results.

  • CADepth-Net uses channel-wise attention to aggregate long-range scene-structure features and re-calibrate features emphasizing object boundaries.

Supplementary Material

The supplementary material adds network details, quantitative evaluations, ablation experiments, and qualitative comparisons for CADepth-Net.

  • The supplementary material covers depth-network details, KITTI quantitative results, additional ablations, and qualitative comparisons.

A. Network Architecture

The architecture uses pretrained ResNet50 depth and pose networks, with structure perception in the encoder and detail emphasis applied across decoder scales. It also evaluates KITTI using improved ground truth that better handles moving objects and occlusions.

  • Network Architecture: The structure perception module receives encoder features, while detail emphasis is applied successively at decoder scales, including the highest-resolution stage without a skip connection.
  • Evaluation Ground Truth: The improved KITTI ground truth uses five consecutive frames and stereo pairs to handle moving objects better, covering 652 of 697 Eigen test frames.

C. KITTI Evaluation Server Benchmark

The KITTI evaluation-server benchmark evaluates CADepth-Net under a new split and compares it with existing monocular and stereo self-supervised approaches. The reported results show that CADepth-Net outperforms existing self-supervised methods.

  • The benchmark uses a new split with 72,084 training examples, 6,060 validation examples, and 500 test examples.The evaluation follows the training protocols described in the main paper.
  • The evaluation-server results are computed without median scaling, using a scale factor calculated from 2,000 KITTI training samples with available ground-truth depths.
  • CADepth-Net outperforms existing self-supervised approaches on the KITTI depth-prediction benchmark.
  • The reported comparisons include methods using monocular, stereo, and ground-truth depth supervision.
  • The benchmark also includes comparisons among methods sharing the same backbone and training settings on the KITTI Eigen split.

D. Additional Ablation Experiments

Additional ablations assess the structure perception and detail emphasis modules separately. The structure module improves distant-object accuracy, while the detail module produces finer details and sharper object boundaries.

  • The structure perception module reduces errors for distant objects in absolute relative error evaluations.Qualitative ablations directly compare error maps without and with the module.
  • Accuracy improves across all evaluated depth intervals, with the performance gap increasing at larger distances.
  • The detail emphasis module generates finer details and more accurate boundaries for pedestrians and thin road signs.
  • Comparisons using the same backbone attribute the observed performance gain mainly to the proposed modules.

E. Additional Qualitative Comparisons

Additional qualitative comparisons show sharper and higher-quality depth predictions from CADepth-Net on KITTI and Make3D. Module-specific visualizations associate structure perception with distant-object error reduction and detail emphasis with sharper estimates.

  • On KITTI, CADepth-Net produces higher-quality outputs and clearer borders than existing baselines.
  • On Make3D, the method preserves sharp discontinuities in depth-prediction results.
  • The structure perception ablation shows reduced errors for distant objects when the module is included.
  • The detail emphasis ablation yields more precise and sharper depth estimation.

F. Additional Visualization Results

Intermediate-feature visualizations illustrate how CADepth-Net aggregates distant-region responses for scene understanding and assigns higher weights to critical local details at multiple scales.

  • Structure-perception features aggregate relative depth relationships across distant regions and improve scene understanding.
  • The detail visualization displays the top 8 feature maps with the highest weights from the module.The weights range from 0 to 1, with examples produced by dem2 and dem3.
  • The detail emphasis module highlights critical local details at multiple scales by assigning them higher scores.
  • KITTI visualizations show higher-quality outputs with clear borders in the predicted depth maps.
Loading 2112.13047v1…