PaperlayerView exact revision on arXiv

Source-linked AI summary

Geometry meets semantics for semi-supervised monocular depth estimation

Pierluigi Zama Ramirez, Matteo Poggi, Fabio Tosi, Stefano Mattoccia, Luigi Di Stefano

arXiv:1810.04093v2cs.CV

TL;DR

Monocular depth estimation from one image is ill-posed, and supervised training depends on costly depth labels. This paper jointly trains depth estimation and semantic segmentation using geometric reprojection signals and semantic labels. On KITTI, the authors report improved monocular depth estimation over the state of the art, while semantic segmentation remains below that task’s state of the art.

  • Problem

    Single-image depth estimation is ill-posed, and supervised methods require large datasets of costly per-pixel depth labels.

    Source: §1 · p. 2

  • Method

    A semi-supervised single-encoder, dual-decoder network uses unsupervised image re-projection for depth and semantic groundtruth labels for segmentation, with a cross-domain discontinuity loss linking the tasks.

    Source: §1 · p. 3

  • Results

    The authors report that their jointly trained architecture outperforms the state-of-the-art approach for unsupervised monocular depth estimation.

    Source: §5 · p. 14

  • Takeaways & Limitations

    On KITTI, joint depth and semantic prediction improves depth estimation, while obtaining semantic labels is more feasible than depth labeling.

    Source: §5 · p. 14

    Semantic segmentation results remain far from state of the art, and the observed improvements in segmentation from joint training are minor.

    Source: §4.3 · pp. 13–14

Abstract

from arXiv · show

Depth estimation from a single image represents a very exciting challenge in computer vision. While other image-based depth sensing techniques leverage on the geometry between different viewpoints (e.g., stereo or structure from motion), the lack of these cues within a single image renders ill-posed the monocular depth estimation task. For inference, state-of-the-art encoder-decoder architectures for monocular depth estimation rely on effective feature representations learned at training time. For unsupervised training of these models, geometry has been effectively exploited by suitable images warping losses computed from views acquired by a stereo rig or a moving camera. In this paper, we make a further step forward showing that learning semantic information from images enables to improve effectively monocular depth estimation as well. In particular, by leveraging on semantically labeled images together with unsupervised signals gained by geometry through an image warping loss, we propose a deep learning approach aimed at joint semantic segmentation and depth estimation. Our overall learning framework is semi-supervised, as we deploy groundtruth data only in the semantic domain. At training time, our network learns a common feature representation for both tasks and a novel cross-task loss function is proposed. The experimental findings show how, jointly tackling depth prediction and semantic segmentation, allows to improve depth estimation accuracy. In particular, on the KITTI dataset our network outperforms state-of-the-art methods for monocular depth estimation.

1 Introduction

Monocular depth is ill-posed from a single image, while depth labels are costly to obtain. The paper combines unsupervised geometric depth signals with supervised semantic labels to improve depth estimation and produce both outputs.

1 Introduction

  • Single-image depth estimation is ill-posed, and supervised approaches require large labeled datasets with costly, often sparse and noisy depth measurements.Stereo pairs or moving-camera views can instead provide unsupervised training signals through image re-projection error.Source: §1 · pp. 1–2
  • The paper proposes joint semantic segmentation and single-image depth estimation, motivated by the link between learned depth features and scene semantics.The authors conjecture that explicitly guiding the network with scene semantics may improve depth prediction.Source: §1 · p. 2
  • The proposed semi-supervised framework uses unsupervised image re-projection for depth and pixel-level groundtruth semantic maps for segmentation.Semantic annotation is required, but the authors describe it as less prohibitive than obtaining groundtruth depth.Source: §1 · p. 3
  • The authors position the work as the first integration of unsupervised monocular depth estimation with supervised semantic segmentation.Their stated contribution is to improve a state-of-the-art encoder-decoder depth architecture.Source: §1 · p. 3
  • A semantic decoder uses the depth-estimation features so the architecture learns both tasks jointly.The paper introduces a cross-domain discontinuity loss to encourage spatial proximity between depth discontinuities and semantic contours.Source: §1 · p. 3
  • On KITTI, jointly tackling semantic segmentation and depth estimation improves monocular depth estimation over a depth-from-mono approach without scene-semantic awareness.The network also produces a semantic segmentation as well as a depth map.Source: §1 · p. 3

2 Related work

Prior work addresses monocular depth through image reconstruction and semantic segmentation through learned visual features. The paper distinguishes its approach from joint depth-and-semantics methods that require groundtruth labels for both tasks.

Unsupervised Monocular Depth

  • The related-work review frames image reconstruction as a way to avoid costly groundtruth depth labels, with prior monocular methods using stereo pairs or video sequences for training.Video-based methods predict relative poses as well as depth, while stereo-based methods use view reconstruction losses.Source: §2 · pp. 3–4
  • Stereo-supervised image reconstruction and bilinear warping established strong unsupervised monocular depth results, while video-based supervision avoids stereo pairs but was less accurate than.The authors follow, which they describe as the undisputed state of the art for unsupervised monocular depth estimation.Source: §2 · p. 4

Semantic Segmentation

  • Modern semantic-segmentation methods mainly use fully convolutional networks, including encoder-decoder architectures that recover fine details from low-resolution features.The paper’s semantic network follows the encoder-decoder group to fit its monocular depth model.Source: §2 · p. 4

Related joint estimation methods

  • Prior joint depth-and-semantics methods require groundtruth labels for both tasks and use multiple training stages, unlike this paper’s semantic-only supervision for depth improvement.The cited approaches include models combining depth regression or estimation with semantic classification or segmentation.Source: §2 · p. 5

3 Proposed method

The method jointly estimates depth and semantics from one image using a shared encoder and task-specific decoders. Semantic supervision and a cross-domain loss connect scene labels to depth prediction.

3 Proposed method

  • The example shows the framework preserving guard-rail structure better than the depth prediction from.The figure pairs the framework's depth and semantic maps with the input image and the depth map.Source: Figure 3 · p. 7, §3 · p. 7
  • The shared encoder learns features for depth and semantic decoders, whose combined training is intended to improve depth prediction.The decoders use different losses, while both gradient flows update the shared encoder.Source: §3 · pp. 5–7

3.1 Loss functions

  • The depth loss combines image reprojection, disparity smoothness, and left-right consistency, and is computed at four scales.Reprojection uses SSIM and L1 differences between original and warped images; smoothness is relaxed at strong image-intensity gradients.Source: Depth term · p. 8
  • The semantic decoder minimizes cross-entropy between predicted and ground-truth pixel-wise labels, computed at full resolution.The text describes this semantic term as a combination of entropy and KL-divergence.Source: Semantic term · p. 8
  • The cross-domain discontinuity loss links ground-truth semantic labels to depth prediction by aligning semantic boundaries with depth discontinuities.It uses predicted disparity and ground-truth semantic maps; the method expects gradient peaks between adjacent pixels assigned to different classes.Source: Cross-domain discontinuity term · pp. 8–9

4 Experimental results

The experiments evaluate joint depth estimation and semantic segmentation using a shared encoder and task-specific decoders, with semantic labels limiting the training samples available. On KITTI 2015, semantic supervision and the cross-domain discontinuity loss improve depth results, while the reported segmentation accuracy remains far from state of the art.

4.2 Monocular depth estimation: evaluation on KITTI 2015

  • The evaluation uses 200 KITTI 2015 stereo pairs, split into 160 training and 40 evaluation samples, after training on Cityscapes and fine-tuning on KITTI.The authors note that their numerical results are not directly comparable with because the fair comparison uses only the semantically annotated samples.Source: §4 · p. 9, §4.2 · p. 10
  • The full loss with ResNet50 gives the best configuration across all metrics; adding the cross-domain discontinuity loss usually improves results further when semantic and depth losses are combined.Using only the depth and cross-domain discontinuity losses performs worse than the depth-only baseline.Source: §4.2 · p. 11
  • With post-processing, the full loss gives the best results on most scores, and the VGG full-loss model beats the depth-loss-only ResNet50 baseline except on δ3.Post-processing averages depth predictions from the input image and its horizontally flipped counterpart after flipping that prediction back.Source: §4.2 · p. 11, Table 1 · p. 11
  • On the same KITTI evaluation split, the tested self-supervised methods,, and perform worse than both and the proposed method.Their available weights were trained on more data, and the comparison uses a different ground-truth quality than the Eigen split.Source: §4.2 · pp. 11–12, Table 2 · p. 12
  • The qualitative examples show improved car shapes, including around reflective or transparent windows, and better pole structures compared with.The figure displays input images, depth maps, and the proposed method's semantic and depth maps for three evaluation images.Source: §4.2 · p. 12, Figure 4 · p. 13

4.3 Semantic segmentation: evaluation on KITTI 2015

  • Semantic segmentation reaches 88.51% per-pixel accuracy with VGG and 88.19% with ResNet50, but the authors report that results remain far from state of the art.They attribute the limitation to an architecture inspired by and optimized for unsupervised depth prediction; they describe the observed segmentation improvements as minor.Source: §4.3 · pp. 13–14

5 Conclusion and future work

The paper concludes that joint depth and semantic training improves unsupervised monocular depth estimation using ground-truth labels only for segmentation. It presents the approach as a way to improve self-supervised training without adding image samples and identifies single-camera supervision and stronger segmentation as future work.

5 Conclusion and future work

  • Jointly predicting depth and semantics with a single encoder and two decoders outperforms the state-of-the-art unsupervised monocular depth approach.Ground-truth labels are used only for semantic segmentation.Source: §5 · p. 14
  • Semantic annotations are tedious, but the authors consider them more feasible than depth labels, which require expensive active sensors and are nearly infeasible to obtain offline.They describe the method as improving self-supervised training without adding image samples.Source: §5 · p. 14
  • Future work will investigate single-camera sequences for supervision and ways to improve semantic segmentation within the joint framework.Source: §5 · p. 14

Related papers

Loading 1810.04093v2…