PaperlayerView exact revision on arXiv

Source-linked AI summary

SuperDepth: Self-Supervised, Super-Resolved Monocular Depth Estimation

Sudeep Pillai, Rares Ambrus, Adrien Gaidon

arXiv:1810.01849v1cs.CVcs.AIcs.LGcs.RO

TL;DR

Self-supervised monocular depth methods have generally operated at low resolution, although high-resolution prediction is important for high-fidelity perception. SuperDepth adds sub-pixel disparity super-resolution and differentiable flip augmentation, achieving state-of-the-art self-supervised disparity performance on KITTI and improved depth and pose results.

  • Problem

    Self-supervised depth methods are constrained to low-resolution regimes by memory requirements, despite high resolution being important for high-fidelity prediction and autonomous driving.

  • Method

    The method uses sub-pixel convolution to super-resolve disparities and differentiable flip augmentation to fuse original and flipped predictions during self-supervised stereo training.

  • Results

    0.112 abs. rel. is achieved by Ours-SP versus 0.116 abs. rel. for Ours at the same input resolution, while Ours-FA achieves state-of-the-art KITTI Eigen-split performance.

  • Takeaways & Limitations

    High-resolution disparity estimation with the proposed layers improves self-supervised monocular depth and pose estimation on the public KITTI benchmark.

Abstract

from arXiv · show

Recent techniques in self-supervised monocular depth estimation are approaching the performance of supervised methods, but operate in low resolution only. We show that high resolution is key towards high-fidelity self-supervised monocular depth prediction. Inspired by recent deep learning methods for Single-Image Super-Resolution, we propose a sub-pixel convolutional layer extension for depth super-resolution that accurately synthesizes high-resolution disparities from their corresponding low-resolution convolutional features. In addition, we introduce a differentiable flip-augmentation layer that accurately fuses predictions from the image and its horizontally flipped version, reducing the effect of left and right shadow regions generated in the disparity map due to occlusions. Both contributions provide significant performance gains over the state-of-the-art in self-supervised depth and pose estimation on the public KITTI benchmark. A video of our approach can be found at https://youtu.be/jKNgBeBMx0I.

I. INTRODUCTION

Self-supervised depth and pose estimation avoids costly ground-truth measurements, but existing methods operate at low resolution despite high-resolution depth being important for autonomous navigation. The paper addresses this gap with sub-pixel disparity super-resolution and differentiable flip augmentation.

  • Motivation: Self-supervised depth and pose methods learn latent scene variables from geometric and temporal constraints without requiring ground-truth labels.Supervised and semi-supervised approaches require large volumes of depth and pose measurements that can be difficult to obtain.
  • Motivation: Existing self-supervised depth methods are limited to low-resolution training because of model and loss memory requirements.The paper motivates high-resolution prediction for safer autonomous driving and improved long-term perception, prediction, and planning.
  • Contributions: The method uses subpixel-convolutional layers to super-resolve disparities from lower-resolution outputs.These layers replace deconvolution or resize-convolution up-sampling in disparity decoders.
  • Contributions: A differentiable flip-augmentation layer fuses original and horizontally flipped predictions, reducing boundary artifacts and occluded-region effects without separate post-processing.The model is trained self-supervised with synchronized stereo imagery and evaluated on KITTI.
  • Background: Monocular depth estimation is an ill-posed inverse problem because multiple 3D scenes can correspond to the same 2D image.Methods therefore rely on geometric, ordinal, temporal, or learned statistical priors.

III. SELF-SUPERVISED, SUPER-RESOLVED MONOCULAR DEPTH ESTIMATION

The model learns a disparity function from synchronized stereo imagery and converts disparity to metric depth using known camera parameters. It builds on an encoder-decoder architecture with skip connections and a single disparity output channel.

  • Model formulation: The model learns a disparity function fd from synchronized stereo images in a self-supervised manner.For each input pixel, fd predicts disparity, which is then used to estimate depth.
  • Model formulation: Metric depth is estimated from disparity using the camera focal length and stereo baseline, both assumed known during training.The supplied formulation identifies focal length f and baseline B as the relevant camera parameters.
  • Architecture: The disparity network extends a DispNet-style encoder-decoder with skip connections and outputs a single disparity channel.Two additional components are incorporated into this base architecture.

1) Sub-pixel Convolution for Depth Super-Resolution:

The paper replaces conventional high-resolution interpolation with sub-pixel convolution for disparity super-resolution and introduces differentiable flip augmentation to improve boundary predictions. Qualitative comparisons show sharper object boundaries and better recovery of thin and distant objects.

  • Sub-pixel Convolution: Conventional deconvolution, resize-convolution, and interpolation up-sample disparities in high-resolution space with limited representational capacity.The paper instead introduces an ESPCN-inspired sub-pixel convolutional layer for depth super-resolution.
  • Differentiable Flip Augmentation: The flip-based design addresses poor boundary-pixel priors caused by unobservable left-image boundary scene points in the right image.Earlier methods used alpha-blending as a separate post-processing step to reduce these artifacts.
  • Qualitative Results: Qualitative comparisons report crisper depth boundaries and more reliable reconstruction of thin and far-off objects than previous methods.Examples include poles and traffic signs, where super-resolved depths recover underlying geometry with higher fidelity.
  • Differentiable Flip Augmentation: Differentiable flip augmentation enables end-to-end fine-tuning by fusing predictions from original and horizontally flipped images with a pixel-wise mean.The model reverts the flipped disparity before fusion and handles borders similarly to prior post-processing.

B. Self-supervising Depth with Stereopsis

The method formulates self-supervised disparity learning as photometric reconstruction across camera views, combining appearance, smoothness, and occlusion regularization losses. These terms are averaged across pixels, pyramid scales, and image batches during training.

  • Photometric Supervision: Disparity estimation is recast as synthesizing a target image by re-projecting a source image using predicted disparity and camera geometry.The photometric objective compares the target image with the synthesized target across available views.
  • Photometric Supervision: For stereo cameras, the relative camera transformation is known a priori and incorporated as a constant in the minimization objective.The available source views are indexed by s in S.
  • Training Loss: The overall loss combines photometric, disparity smoothness, and occlusion regularization terms.The combined loss is given as LD(It, ˆIt) = Lp(It, ˆIt) + λ1 Ls(It) + λ2 Lo(It).
  • Training Loss: Appearance matching uses SSIM together with an L1 photometric term to compare target and synthesized images.This defines the photometric component of the training objective.
  • Training Loss: The edge-aware smoothness term regularizes disparities in textureless regions and is weighted across pyramid levels with downsampling.The 0th pyramid level starts with weight 1, with weights decaying by a factor of 2.
  • Training Loss: Occlusion regularization minimizes shadow areas across high-gradient disparity regions by applying an L1 penalty to disparity estimates.The term encourages background depths, corresponding to lower disparities.
  • Training Loss: The final loss is averaged per pixel, pyramid scale, and image batch during training.

A. Dataset

Experiments use the KITTI dataset and standard Eigen-split protocols, evaluating a modified DispNet baseline and proposed variants under self-supervised training.

  • Experiments use 22,600 training, 888 validation, and 697 test stereo image pairs from KITTI Eigen splits.
  • The study evaluates disparities with metrics described in Eigen et al. on the standard KITTI Disparity Estimation benchmark.
  • The modified DispNet baseline is compared with proposed sub-pixel and flip-augmentation variants.

1) Effect of High-Resolution in Disparity Estimation:

Increasing resolution improves self-supervised disparity estimation, motivating sub-pixel convolution to super-resolve disparities efficiently and accurately.

  • At 1024 x 384, 1536 x 576, and 2048 x 768, most disparity metrics continue improving as input resolution increases.Improvements include Abs. Rel, Sq. Rel, RMSE, and RMSE log.
  • The baseline saturates at 1536 x 576, while the fraction of disparities within δ < z improves at higher resolutions.The saturation is attributed to the original KITTI stereo image resolution of 1392 x 512 pixels.
  • 0.112 Abs. Rel is achieved by Ours-SP versus 0.116 Abs. Rel for the Ours baseline at the same input resolution.
  • Naively increasing input resolution improves δ < z and decreases Abs Rel, Sq Rel, and RMSE log without changing the loss function.
  • Table I reports state-of-the-art self-supervised disparity estimation performance on the KITTI Eigen test set using the proposed components.

3) Improving Disparity Estimation with Differentiable Flip-Augmentation Fine-Tuning:

Differentiable flip-augmentation replaces hand-engineered fusion with end-to-end training, improving predictions and contributing to state-of-the-art KITTI performance.

  • Flip-augmentation improves Ours and Ours-SP on all metrics except Abs. Rel, which remains unchanged.
  • Training with Ours-SP followed by flip-augmentation fine-tuning achieves state-of-the-art performance on the KITTI Eigen split benchmark.
  • The experiments consider self-supervised training from scratch rather than ImageNet-pretrained weights or alternate-dataset adaptation.
  • Table I compares self-supervised stereo and monocular methods on KITTI Eigen depths below 80m using error and threshold-accuracy metrics.Lower is better for Abs Rel, Sq Rel, RMSE, and RMSE log; higher is better for the δ metrics.
  • Qualitative comparisons report sharper object boundaries, especially for small objects such as poles and traffic signs.

C. Pose Estimation

The pose network estimates 6 DoF transformations between temporally adjacent monocular frames and is jointly trained with disparity using photometric objectives. Bootstrapping pose with stereo-trained disparity and combining monocular and stereo losses supports metrically accurate, low-drift trajectories on KITTI.

  • The pose network takes a target image and temporally adjacent context images, then predicts 6 DoF transformations between them.Rotations are parameterized using the logarithm of a unit quaternion in R3.
  • Pose training uses a photometric loss between the target image and its reconstruction inferred by mapping a context image into the target view.
  • α2 = 0.05 weights the multi-view photometric loss toward its L1 component because sequential images often violate fronto-parallel assumptions.The loss combines L1 and SSIM terms, but the stated setting favors L1 during pose optimization.
  • Jointly optimizing disparity and pose prevents disparity from diverging and supports metrically accurate trajectory recovery.
  • Evaluation reports average translational and rotational RMS drift over 100-800m trajectories on KITTI odometry sequences 00-10.The networks are trained on the KITTI Eigen split, which includes images from sequences 01, 02, 06, 08, 09, and 10.
  • The proposed method outperforms UnDeepVO quantitatively and recovers metrically accurate trajectories with little extended-period drift.The authors attribute the comparison to bootstrapping pose with more accurate depth estimates.

D. Implementation

Implementation uses a PyTorch disparity network with sub-pixel convolution and differentiable flip augmentation, trained with parallelized computation across eight Titan V100 GPUs. The section also visualizes pose predictions against KITTI ground truth for sequences 00 and 08.

  • The disparity network is implemented in PyTorch, using native PixelShuffle and index select operations for the two proposed components.
  • Figure 5 visualizes pose estimates on KITTI odometry sequences 00 and 08, rendering predictions in blue and ground truth in red.
  • Training uses eight Titan V100 GPUs, 200 Adam epochs, and typically batch sizes of 4 or 8 with an initial learning rate of 5e-4.The learning rate and batch size were selected through hyper-parameter search.

V. CONCLUSION

The paper extends self-supervised monocular disparity estimation with super-resolution and differentiable flip augmentation. Experiments report state-of-the-art KITTI depth performance and strong quantitative pose-estimation results.

  • Sub-pixel convolutions super-resolve disparities from lower-resolution features, while differentiable flip augmentation reduces boundary artifacts and ambiguities.
  • The super-resolved representation reduces ambiguities in self-supervised photometric-loss estimation compared with lower-resolution predictions.
  • Experiments show significant gains and state-of-the-art depth-estimation performance on the public KITTI benchmark, alongside strong quantitative pose results.
Loading 1810.01849v1…