PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-supervised Monocular Trained Depth Estimation using Self-attention and Discrete Disparity Volume

Adrian Johnston, Gustavo Carneiro

arXiv:2003.13951v1cs.CVcs.LG

TL;DR

Limited and difficult-to-collect ground-truth RGB-D data motivates self-supervised monocular depth estimation from widely available video. The paper extends this setting with self-attention and discrete disparity volumes, achieving state-of-the-art monocular results on KITTI 2015 and superior results to previous self-supervised methods on Make3D. The approach also produces sharper, more robust depth estimates and pixel-wise uncertainty.

  • Problem

    Ground-truth RGB-D data is limited in variety and abundance, while accurate large-scale collection is difficult because of sensor noise and operating conditions.

  • Method

    The method extends monocular self-supervised depth estimation with self-attention for non-contiguous contextual reasoning and discrete disparity volumes projected into multi-scale disparity maps.

  • Results

    The method achieves state-of-the-art monocular self-supervised depth estimation on KITTI 2015 and superior results to previous self-supervised methods on Make3D.

  • Takeaways & Limitations

    Discrete disparity volumes produce more robust and sharper depth estimates while enabling pixel-wise depth uncertainty computation.

Abstract

from arXiv · show

Monocular depth estimation has become one of the most studied applications in computer vision, where the most accurate approaches are based on fully supervised learning models. However, the acquisition of accurate and large ground truth data sets to model these fully supervised methods is a major challenge for the further development of the area. Self-supervised methods trained with monocular videos constitute one the most promising approaches to mitigate the challenge mentioned above due to the wide-spread availability of training data. Consequently, they have been intensively studied, where the main ideas explored consist of different types of model architectures, loss functions, and occlusion masks to address non-rigid motion. In this paper, we propose two new ideas to improve self-supervised monocular trained depth estimation: 1) self-attention, and 2) discrete disparity prediction. Compared with the usual localised convolution operation, self-attention can explore a more general contextual information that allows the inference of similar disparity values at non-contiguous regions of the image. Discrete disparity prediction has been shown by fully supervised methods to provide a more robust and sharper depth estimation than the more common continuous disparity prediction, besides enabling the estimation of depth uncertainty. We show that the extension of the state-of-the-art self-supervised monocular trained depth estimator Monodepth2 with these two ideas allows us to design a model that produces the best results in the field in KITTI 2015 and Make3D, closing the gap with respect self-supervised stereo training and fully supervised approaches.

1. Introduction

Monocular depth estimation is attractive because RGB video is widely available, but supervised approaches are constrained by limited and difficult-to-collect ground-truth RGB-D data. The paper addresses this challenge with self-attention for global contextual reasoning and discrete disparity volumes for sharper, more robust depth and uncertainty estimation.

  • Motivation: Accurate monocular depth supports perception, localization, mapping, and planning, but conventional range sensors can be expensive or complex to deploy.These systems motivate learning-based alternatives.
  • Motivation: Ground-truth RGB-D data is limited in variety and abundance, and accurate large-scale collection is difficult because of sensor noise and operating conditions.Weather and lighting can restrict sensor capabilities.
  • Self-supervised learning: Monocular video offers widely available training sequences but requires jointly estimating depth and camera ego-motion to minimize photometric reprojection loss.This makes monocular self-supervision more challenging than stereo-based learning with known camera pose.
  • Proposed approach: Self-attention uses non-contiguous image regions as context, unlike local 2D and 3D convolutions, to estimate similar depth across globally related regions.The architecture passes encoded features through a Self-Attention Context Module.
  • Proposed approach: Discrete disparity volumes target more robust and sharper depth estimates while enabling pixel-wise depth uncertainty estimation.The figure depicts estimated disparity, attention maps, and pixel-wise uncertainty as model outputs.

2. Related Work

Prior work established supervised and self-supervised routes for monocular depth estimation, while highlighting limitations from costly data, ill-posed geometry, rigid-scene assumptions, and local context. Related approaches also explored self-attention and probabilistic disparity volumes for broader contextual modeling and uncertainty-aware refinement.

  • Monocular depth estimation: Monocular depth is inherently ill-posed because a pixel can correspond to multiple plausible depths.Supervised learning mitigates this challenge but depends on costly training data.
  • Self-supervised depth estimation: Self-supervised stereo methods replace ground-truth depth with a stereo photometric reprojection warping loss that implicitly learns depth.This approach reduces dependence on direct depth labels.
  • Self-supervised depth estimation: Monocular-video estimators jointly learn depth and ego-motion, but rely on a static-scene assumption in which only the camera moves.The passage identifies this assumption as a source of degenerate behavior.
  • Self-attention: Self-attention models long-range dependencies and can represent feature groups with similar semantic similarity beyond the local reach of standard convolutions.This motivates using non-contiguous context for monocular depth prediction.
  • Disparity volumes: Prior disparity-volume methods use cost or probability volumes, with Bayesian filtering applied in one approach to refine outliers using computed uncertainty.These methods connect discrete matching representations with uncertainty-aware refinement.

3. Methods

The model extends Monodepth2 with self-attention, a discrete disparity volume, and a pose estimator trained jointly through photometric reprojection and smoothness losses.

  • Model overview: Monodepth2 provides the UNet-based baseline, while the proposed model focuses on self-attention and discrete disparity volume contributions.
  • Self-attention: The ResNet-101 encoder produces low-resolution features that feed a self-attention module forming query, key, and value representations.The encoder output is defined on a lattice at one-eighth of the input resolution.
  • Discrete disparity volume: A low-resolution discrete disparity volume produces K disparity channels, and multi-scale decoding predicts disparity at one-eighth, one-quarter, one-half, and full resolution.Each decoder stage uses upconvolution and skip connections from the corresponding encoder resolution.
  • Pose estimation: The pose estimator takes images from different time steps and returns their relative transformation for reprojection-based training.The pose model is parameterized separately from the depth model.
  • Training and inference: Training minimizes a per-pixel photometric reprojection error between source and target frames, with edge-aware smoothness regularization and stationary-point auto-masking.The depth and pose models are trained jointly, and inference outputs the full-resolution disparity map.

4. Experiments

Experiments on KITTI and Make3D evaluate the proposed model against self-supervised monocular baselines, including Monodepth2, and analyze its components through ablation and qualitative results. The method improves thin-structure and boundary estimation, while self-attention and DDV provide additional uncertainty and interpretability signals.

  • Experimental setup: The model is trained and evaluated on KITTI 2015, then tested on Make3D using KITTI-trained weights and median scaling.Evaluation follows the Eigen split, removes static frames before training, and limits predicted depths to 80m.
  • KITTI results: On KITTI 2015, the method outperforms existing self-supervised monocular methods and is competitive with stronger-supervision approaches, including Monodepth2.The authors report comparable results to Monodepth2, the current state-of-the-art method in the cited comparison.
  • Make3D results: On Make3D, the model produces superior results compared with previous methods that also rely on self-supervision.The evaluation uses the Monodepth2 testing protocol and criteria from prior work.
  • Ablation study: The ablation study shows steady improvement in almost all evaluation measures when adding DDV and self-attention to the Monodepth2 baseline.The study also compares ResNet-18 and ResNet-101 backbones with dilated convolutions.
  • Qualitative results: Qualitative results show sharper estimates for thin structures and difficult boundaries, while the method succeeds on reflective car roofs and difficult street signs in Monodepth2 failure cases.The KITTI examples include trees, signs, bollards, poles, and boundaries overlapping foliage.
  • Ablation study: Self-attention drastically improves close-range accuracy (δ < 1.25) for both backbone models, with a larger gain for ResNet-101.The authors associate the larger ResNet-101 improvement with the larger receptive field from dilated convolutions.
  • Self-attention and uncertainty: Self-attention maps expose non-contiguous image relationships, while DDV supports uncertainty estimation from disparity-probability variance.Reported uncertainty generally increases with distance, whereas background regions estimated at near-infinite depth have low uncertainty; foliage and shadows show high uncertainty.

5. Conclusion

The method combines self-attention and a discrete disparity volume to improve monocular-video depth estimation, achieving state-of-the-art KITTI 2015 results while estimating pixel-wise uncertainty.

  • Self-attention improves contextual information for monocular depth estimation by incorporating relationships across non-contiguous image regions.
  • Attention maps expose learned relationships between objects and disparity, highlighting foreground, midground, or background regions.
  • The discrete disparity volume regularises training and produces more robust, sharper depth estimates while enabling pixel-wise depth uncertainty.

A. KITTI Improved Ground Truth

The improved KITTI ground truth uses multiple LiDAR frames and stereo images to better handle occlusions and other motion-related evaluation issues.

  • Improved KITTI ground truth reprojects five consecutive LiDAR frames and uses stereo images for better handling of occlusions.
  • The Eigen evaluation ground truth reprojects LiDAR points but does not handle occlusions, non-rigid motion, or camera motion.

B. Network Architecture

The network uses a dilated ResNet-101 encoder, a decoder with skip connections and ELU activations, and a Monodepth2-style pose model.

  • Experiments generally use a ResNet-101 encoder with pretrained ImageNet weights and atrous convolutions in the final two residual blocks.
  • The encoder uses ReLU activations, while the decoder uses ELU activations and skip connections between intermediate encoder and decoder outputs.
  • The pose model retains the ResNet-18 encoder and pose decoder defined by Monodepth2.

C. Additional Qualitative Results

Additional results compare the method qualitatively with prior approaches, visualize uncertainty and attention, and document the network architecture and KITTI evaluation setup.

  • Additional Qualitative Results: The method produces sharper predictions for thin structures and complex shapes such as people than multiple previous works.
  • Additional Qualitative Results: Low-contrast and unknown-texture regions correspond with high uncertainty and can exhibit texture-copy artefacts in predicted depth.
  • Additional Qualitative Results: Additional attention maps are randomly selected from the 512 output channels of the context module.
  • Network Architecture: The network architecture table specifies convolution kernels, strides, channels, dilation, resolution scales, inputs, activations, and upsampling.
  • KITTI Improved Ground Truth: The KITTI improved-ground-truth table compares methods by supervision level and reports best and second-best results for each category.
Loading 2003.13951v1…