PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Supervised Monocular Depth Estimation: Solving the Dynamic Object Problem by Semantic Guidance

Marvin Klingner, Jan-Aike Termöhlen, Jonas Mikolajczyk, Tim Fingscheidt

arXiv:2007.06936v2cs.CV

TL;DR

Self-supervised monocular depth estimation must handle moving dynamic-class objects that violate the static-world assumption used by geometric training. SGDepth combines cross-domain semantic segmentation and depth learning with semantic masking and frame detection, and it exceeds all baselines on the KITTI Eigen split without test-time refinement.

  • Problem

    Moving dynamic-class objects violate the static-world assumption used in self-supervised monocular depth estimation from video.

  • Method

    SGDepth jointly trains supervised semantic segmentation and self-supervised depth with task-specific heads, masks moving-object pixels, and detects non-moving-object frames for depth learning.

  • Results

    SGDepth exceeds all baselines on the KITTI Eigen split without test-time refinement and outperforms previous mono approaches in 6 out of 7 measures.

  • Takeaways & Limitations

    Semantic guidance supports self-supervised monocular depth estimation while preserving learning from non-moving dynamic-class objects.

  • Takeaways & Limitations

    The approach requires training data containing some frames with non-moving dynamic-class objects from predefined semantic classes.

Abstract

from arXiv · show

Self-supervised monocular depth estimation presents a powerful method to obtain 3D scene information from single camera images, which is trainable on arbitrary image sequences without requiring depth labels, e.g., from a LiDAR sensor. In this work we present a new self-supervised semantically-guided depth estimation (SGDepth) method to deal with moving dynamic-class (DC) objects, such as moving cars and pedestrians, which violate the static-world assumptions typically made during training of such models. Specifically, we propose (i) mutually beneficial cross-domain training of (supervised) semantic segmentation and self-supervised depth estimation with task-specific network heads, (ii) a semantic masking scheme providing guidance to prevent moving DC objects from contaminating the photometric loss, and (iii) a detection method for frames with non-moving DC objects, from which the depth of DC objects can be learned. We demonstrate the performance of our method on several benchmarks, in particular on the Eigen split, where we exceed all baselines without test-time refinement.

1 Introduction

The paper introduces SGDepth, which combines supervised semantic segmentation with self-supervised monocular depth estimation to address moving dynamic-class objects that violate static-world assumptions. It masks moving-object pixels during photometric training while retaining non-moving dynamic-class objects for learning, and reports superior benchmark performance without test-time refinement.

  • Motivation: Self-supervised monocular depth estimation learns from geometric image projections and photometric errors without depth labels, but requires stereo or sequential images during training.These methods infer depth from single images at inference time.
  • Motivation: Moving dynamic-class objects such as cars, trucks, and pedestrians violate the static-world assumption underlying video-based geometric projection.Occlusions and insufficient camera motion create additional photometric-error problems.
  • Method: SGDepth jointly optimizes depth and semantic segmentation across source and target domains using task-specific network heads.Supervised segmentation guides self-supervised depth estimation in the target domain.
  • Method: The method excludes pixels belonging to moving dynamic-class objects from the photometric loss instead of extending the image projection model.This avoids contaminating training while keeping the geometric model simpler.
  • Method: A detection method identifies frames with non-moving dynamic-class objects so their depth can be learned normally, while moving objects remain excluded from the loss.The approach therefore distinguishes when dynamic-class pixels should contribute to training.
  • Results: SGDepth exceeds all baselines on the KITTI Eigen split without test-time refinement and is evaluated on two further KITTI benchmarks.The paper presents the method as a three-part contribution spanning cross-domain training, semantic masking, and moving-object detection.

2 Related Work

Prior self-supervised depth work uses stereo or video geometry and addresses reconstruction challenges such as occlusion, while dynamic-object methods often add optical-flow, segmentation, or object-specific transformations. SGDepth instead presents a joint depth–segmentation framework with semantic cross-task guidance.

  • Depth Estimation: Earlier depth-estimation methods inferred depth from stereo pairs, image sequences, or single images supervised by LiDAR scans.Neural-network benchmarks are now dominated by depth-estimation algorithms.
  • Self-Supervised Depth Estimation: Self-supervised monocular depth estimation optimizes photometric error from geometric projections between stereo images or video frames.Joint depth and relative-pose prediction extends this formulation to sequential images.
  • Self-Supervised Depth Estimation: Related self-supervised approaches address occlusions with reprojection losses and use alternatives including calibration learning, synthetic depth labels, teacher-student learning, GANs, stereo proxies, or recurrent networks.These methods still rely on geometric projection assumptions whose static-world limitations motivate dynamic-object handling.
  • Multi-Task Learning: Multi-task learning has been applied to domain adaptation, depth estimation, and semantic segmentation, including semantic cues for self-supervised depth with stereo images.Prior work computes cross-entropy between predicted and warped segmentation outputs.
  • Handling Dynamic-Class Objects: Dynamic-object methods commonly predict optical flow alongside depth and impose cross-task consistency, geometric constraints, or modified reconstruction losses.Other approaches use segmentation to apply separate rigid transformations to dynamic objects, increasing projection-model complexity.
  • Proposed Framework: SGDepth jointly predicts depth and semantic segmentation, with red framework components representing semantic cross-task guidance between the two tasks.In the framework diagram, grey blocks are neural networks, blue blocks plain self-supervised depth, orange blocks plain supervised segmentation, and block numbers reference equations.

3 Method

The method combines self-supervised monocular depth estimation with supervised semantic segmentation and uses semantic guidance to handle dynamic-class objects during training.

  • 3.1 Self-Supervised Monocular Depth Estimation: During training, preceding and succeeding frames are warped into the current frame using camera intrinsics, predicted relative poses, and estimated depth.The projection maps pixels to 3D, applies the relative pose, and reprojects them into another image before bilinear sampling.
  • 3.1 Self-Supervised Monocular Depth Estimation: The photometric loss combines absolute image differences with structural similarity, while the smoothness loss encourages nearby pixels to have similar depths.Smoothness is applied to mean-normalized inverse depth and permits larger depth changes near image-intensity boundaries.
  • 3.2 Supervised Semantic Segmentation: Semantic segmentation assigns each pixel a class using per-pixel posterior scores trained with a weighted cross-entropy loss.The segmentation mask is obtained by selecting the highest-scoring class for each pixel.
  • 3.3 Semantic Guidance: A shared encoder with task-specific depth and segmentation decoder heads trains segmentation in a supervised source domain and depth in a self-supervised target domain.The framework propagates gradients from the separate decoders through the shared encoder.
  • 3.3 Semantic Guidance: The semantic masking strategy excludes pixels belonging to dynamic-class objects in the target or projected frames from the photometric loss.Projected semantic masks use nearest-neighbor sampling, and the resulting mask contains 0 for dynamic-class pixels and 1 otherwise.
  • 3.3 Semantic Guidance: The method detects static images from segmentation-mask alignment, then combines masked and unmasked photometric losses so depth can be learned from non-moving dynamic-class objects.The alignment indicator equals 1 for perfect alignment without moving dynamic-class objects and 0 when the share of moving objects is high; a threshold determines which images are treated as static.

4 Experimental Setup

SGDepth uses separate depth and segmentation heads with shared encoder training across Cityscapes and KITTI, evaluated under multiple image resolutions and depth metrics.

  • Network Topology: SGDepth uses an encoder–decoder architecture with skip connections and an ImageNet-pretrained ResNet18 encoder.The depth and segmentation decoders share the encoder but use task-specific outputs.
  • Network Topology: The depth head predicts dense depth maps, while the segmentation head produces class probabilities through a softmax output.Depth values are constrained to the range [0.1, 100].
  • Training Aspects: Training uses gradient scaling between encoder and decoder connections and gradually increases the fraction of images trained without masked photometric loss.The gradient scale factor is λ = 0.1; the masking schedule changes after 30 epochs.
  • Training Aspects: Models are trained for 40 epochs with Adam, batch sizes of 12 for single-task and 6 for multi-task models, and a learning rate reduced after 30 epochs.The learning rate changes from 10^-4 to 10^-5.
  • Databases: Semantic segmentation is trained on Cityscapes while self-supervised depth estimation uses KITTI subsets, enabling cross-dataset training.Evaluation includes the KITTI Eigen and Benchmark splits, plus a joint depth-segmentation KITTI split.
  • Evaluation Metrics: Depth evaluation uses four error metrics and three accuracy metrics comparing predicted and ground-truth depth.The error metrics include Abs Rel, Sq Rel, RMSE, and RMSE log.

5 Evaluation and Discussion

SGDepth is evaluated against monocular and stereo baselines on KITTI depth benchmarks and through component studies, with qualitative and semantic-segmentation results showing sharper object structure and improved task performance.

  • 5.1 Depth Evaluation w.r.t. the Baselines: On the Eigen split, full SGDepth outperforms all comparable baselines at each of three reported resolutions without test-time refinement.The comparison excludes methods using test-time refinement or significantly larger architectures.
  • 5.1 Depth Evaluation w.r.t. the Baselines: On the Benchmark split, SGDepth outperforms the only other listed self-supervised approach and reduces the gap to supervised methods.Results were computed using the KITTI online evaluation server with a global validation-derived scale factor.
  • 5.1 Depth Evaluation w.r.t. the Baselines: SGDepth sharpens depth boundaries around dynamic-class objects and detects small objects such as traffic signs that other methods miss.Figure 5 highlights improved distinction of small objects and sharper object boundaries.
  • 5.2 Ablation Studies: The full training approach sharpens depth boundaries and reduces artifacts inside segmentation masks compared with depth-only and segmentation-only baselines.These qualitative differences are shown in Figure 6.
  • 5.2 Ablation Studies: The final SGDepth model outperforms the Liu et al. mono approach in 6 out of 7 measures and Chen et al.’s stereo approach in 5 out of 7 measures.This comparison summarizes the ablation-study performance of the final model.
  • 5.3 Semantics Evaluation: Multi-task training improves depth estimation and semantic segmentation across Cityscapes and KITTI.The reported KITTI mIoU improves from 43.1% to 51.6%, while Cityscapes mIoU improves from 63.3% to 68.2%.

6 Conclusion

SGDepth combines cross-domain semantic segmentation and self-supervised depth estimation with semantic masking to address moving dynamic-class objects. It exceeds baselines on KITTI evaluations, while requiring training frames containing non-moving dynamic-class objects.

  • 6 Conclusion: SGDepth uses multi-task cross-domain training and semantic masking to improve monocular depth estimation around moving objects.The method trains depth estimation and semantic segmentation jointly, using semantic guidance for dynamic-object handling.
  • 6 Conclusion: On the KITTI Eigen split, SGDepth exceeds all baselines without test-time refinement.
  • 6 Conclusion: On the KITTI split, SGDepth outperforms previous monocular approaches in 6 of 7 measures and a stereo approach in 5 of 7 measures.
  • 6 Conclusion: The method is advantageous only when training data includes frames with non-moving dynamic-class objects, such as parked vehicles.These frames provide examples from which dynamic-class-object depth can be learned.

Supplementary

This supplementary passage only introduces additional experimental-setup details for deeper understanding.

  • Supplementary: The supplementary section provides additional details about the experimental setup.

A.1 Detailed Dataset Overview

The experiments combine Cityscapes semantic labels with KITTI depth data across several evaluation splits, while defining dynamic classes from human and vehicle categories.

  • A.1 Detailed Dataset Overview: Cityscapes provides 2,975 labeled training images for semantic segmentation and an official validation set of 500 labeled images.The validation set is not used for hyperparameter optimization.
  • A.1 Detailed Dataset Overview: The depth network is trained on KITTI splits, including the Eigen split with 697 test images and the Benchmark split with 500 labeled test images.Benchmark labels are available only through an evaluation server.
  • A.1 Detailed Dataset Overview: The KITTI split uses 200 official KITTI 2015 Stereo training images with labels for both depth and semantic segmentation.This makes the split suitable for observing multi-task training benefits.
  • A.1 Detailed Dataset Overview: Dynamic-class objects include person, rider, car, truck, bus, train, motorcycle, and bicycle.These classes belong to the human and vehicle categories in Cityscapes.
  • A.1 Detailed Dataset Overview: Table 4 summarizes the datasets used and the labels available for each.
  • A.1 Detailed Dataset Overview: Road, sidewalk, building, wall, fence, pole, traffic light, traffic sign, vegetation, terrain, and sky are treated as static classes.

A.3 Evaluation Metrics

Evaluation covers depth and semantic segmentation using standard metrics, dataset-specific scaling and cropping, and qualitative comparisons of SGDepth predictions.

  • A.3 Evaluation Metrics: Depth maps are evaluated against sparse LiDAR ground truth, with median scaling applied before evaluation.
  • A.3 Evaluation Metrics: The Eigen split applies a standard crop, whereas the Benchmark split uses a global validation-derived scale factor because test labels are unavailable.
  • A.3 Evaluation Metrics: Semantic segmentation is evaluated with mean intersection over union, computed from true positives, false positives, and false negatives.Counts are aggregated across the test set before mIoU is calculated.
  • A.3 Evaluation Metrics: Qualitative comparisons examine SGDepth against baselines and a depth-only variant, focusing on sharper object boundaries and dynamic-object depth.The full method produces sharper boundaries and better-learned depth for pedestrians and cyclists in the reported examples.

B.2 Benefits of Multi-Task Training

On KITTI, joint SGDepth training improves depth estimation and semantic segmentation relative to single-task baselines, with semantic guidance benefiting individual-image predictions.

  • Joint SGDepth training visibly improves depth predictions over the depth-only baseline on KITTI examples.The comparison uses models trained for depth estimation alone versus the full SGDepth model.
  • The qualitative comparisons indicate that semantic guidance benefits depth estimation in individual images.
  • SGDepth also improves semantic segmentation compared with a segmentation-only model trained on Cityscapes.The authors attribute this to KITTI-suitable features learned through self-supervised depth estimation, which may bridge the Cityscapes–KITTI domain shift.

B.3 KITTI Eigen Split Ablation

The KITTI Eigen split ablation confirms the multi-task advantage: all multi-task models outperform the depth-only baseline, while full SGDepth performs best on key depth metrics.

  • All multi-task models outperform the single-task SGDepth only depth baseline on the KITTI Eigen split.The ablation evaluates the obtained models on the Eigen split after training and optimization on the KITTI split.
  • SGDepth full achieves the best results in Abs. Rel. and δ < 1.25 on the Eigen split.The authors identify these as important metrics and report the same pattern on the KITTI split.
  • Table 5 reports the Eigen-split ablation across model variants and resolutions, with CS and K denoting Cityscapes and KITTI training settings.

B.4 Pose Evaluation

Pose estimation is evaluated on KITTI odometry sequences 9 and 10, where joint depth–segmentation training has a negative effect but remains competitive with baselines.

  • Joint depth and semantic-segmentation training appears to negatively affect pose estimation.The authors identify optimization of pose within the multi-task setting as a possible direction for future work.
  • Despite this effect, SGDepth achieves competitive pose-estimation results against the cited baselines.The comparison uses training on sequences 0–8 and evaluation on sequences 9 and 10.
Loading 2007.06936v2…