PaperlayerView exact revision on arXiv

Source-linked AI summary

Semantically-Guided Representation Learning for Self-Supervised Monocular Depth

Vitor Guizilini, Rui Hou, Jie Li, Rares Ambrus, Adrien Gaidon

arXiv:2002.12319v1cs.CVcs.LG

TL;DR

Monocular depth networks can exploit category-level appearance–geometry relationships, but existing methods do not explicitly use them while remaining fully self-supervised. The paper introduces fixed semantic guidance through pixel-adaptive convolutions and two-stage resampling, and reports consistent state-of-the-art improvements, including for dynamic categories.

  • Problem

    Existing depth methods either do not explicitly leverage category-conditioned geometric structure or rely on strong semantic supervision that departs from self-supervision.

  • Method

    A fixed pretrained semantic segmentation network guides self-supervised depth-feature learning through pixel-adaptive convolutions, followed by two-stage resampling to reduce dynamic-object bias.

  • Results

    The method consistently improves self-supervised monocular depth performance across pixels, semantic classes, and dynamic categories, establishing a new state of the art.

  • Takeaways & Limitations

    Semantic-aware geometric representations can improve self-supervised monocular depth estimation across different depth architectures and fine-grained evaluation axes.

  • Takeaways & Limitations

    The self-supervised photometric loss assumes a static world and therefore cannot properly model some dynamic objects, including objects moving at the camera’s speed.

Abstract

from arXiv · show

Self-supervised learning is showing great promise for monocular depth estimation, using geometry as the only source of supervision. Depth networks are indeed capable of learning representations that relate visual appearance to 3D properties by implicitly leveraging category-level patterns. In this work we investigate how to leverage more directly this semantic structure to guide geometric representation learning, while remaining in the self-supervised regime. Instead of using semantic labels and proxy losses in a multi-task approach, we propose a new architecture leveraging fixed pretrained semantic segmentation networks to guide self-supervised representation learning via pixel-adaptive convolutions. Furthermore, we propose a two-stage training process to overcome a common semantic bias on dynamic objects via resampling. Our method improves upon the state of the art for self-supervised monocular depth prediction over all pixels, fine-grained details, and per semantic categories.

1 INTRODUCTION

The paper targets monocular depth estimation by making semantic structure more explicit while preserving self-supervision. It introduces semantic guidance and resampling to improve depth prediction, including for dynamic objects.

  • Motivation: Accurate monocular depth supports perception, navigation, and planning while enabling supervision from large-scale raw-video datasets.Self-supervised methods use geometry rather than direct depth measurements as supervision.
  • Motivation: Monocular depth networks learn appearance–geometry representations conditioned on object category, but existing methods either underuse this structure or require semantic supervision.The latter departs from the self-supervised paradigm through a semantic proxy task.
  • Contributions: The proposed architecture uses a fixed pretrained semantic segmentation network and pixel-adaptive convolutions to guide self-supervised geometric representation learning.The convolutions learn semantic-dependent representations intended to capture category-conditioned geometric relationships.
  • Contributions: A two-stage training process automatically detects a dynamic-object bias and resamples training data to address predictions at infinite depth.The bias is associated with dynamic objects and projections at infinity.
  • Results: The method improves state-of-the-art self-supervised monocular depth estimation on KITTI across pixels, classes, and dynamic categories.The supplied implementation note indicates that source code and pretrained models are available.

2 RELATED WORK

Prior work uses semantics mainly to handle dynamic regions or reduce geometric uncertainty. This paper instead uses pretrained semantic features to guide depth representation learning across multiple architectures.

  • Prior depth estimation: Supervised monocular depth methods improved substantially, but large-scale depth labels became a bottleneck, especially for outdoor applications.This motivated self-supervised alternatives based on raw imagery and geometry.
  • Semantic information: Earlier semantic approaches use semantic or instance information to mask or model dynamic image regions and reduce uncertainty in geometric estimation.Examples include object-aware MRFs and instance segmentation combined with epipolar constraints.
  • This work: The proposed approach uses a pretrained semantic segmentation network to guide depth features through pixel-adaptive convolutions rather than treating semantics as a separate proxy task.This representation-learning approach can modify ResNet and PackNet depth networks.
  • This work: Semantic feature guidance yields consistent performance gains across depth architectures ranging from ResNets to PackNet.The paper reports applicability to widely used and state-of-the-art architectures.

3 SELF-SUPERVISED STRUCTURE-FROM-MOTION

The self-supervised SfM setup jointly predicts depth and camera motion, reconstructing target images from temporal context. Training combines photometric reconstruction with edge-aware depth smoothness and established self-supervised refinements.

  • Structure-from-motion: The SfM setting uses a monocular depth model to predict per-pixel depth and an ego-motion estimator to predict 6 DoF transformations between target and source images.The depth model maps an image to a depth prediction, while the pose model operates on target and source image pairs.
  • Structure-from-motion: Experiments use the preceding and following frames as temporal source images for the target frame.The reported source images are I_t−1 and I_t+1.
  • Training objective: The networks are trained simultaneously with an appearance-matching loss on synthesized and target images plus a depth regularization term for edge-aware smoothing.The final objective is averaged per pixel, pyramid scale, and image batch.
  • Training objective: The total loss combines photometric loss Lp with weighted depth smoothness Ls: L(It, ˆIt) = Lp(It, ˆIt) + λ1 Ls(ˆDt).λ1 weights the photometric and depth-smoothness terms.
  • Training objective: Edge-aware smoothness regularizes depth gradients using image gradients, with pyramid-level weights decaying by a factor of 2 after each downsampling.The finest pyramid level starts with weight 1.
  • Training objective: The implementation incorporates auto-masking, minimum reprojection error, and inverse-depth upsampling to improve self-supervised monocular depth estimation.These refinements follow prior self-supervised depth methods.
  • Architecture: The baseline uses PackNet packing and unpacking blocks, but the proposed semantic guidance is not restricted to that architecture.PackNet was selected because of its state-of-the-art self-supervised monocular depth performance.

4 SEMANTICALLY-GUIDED GEOMETRIC REPRESENTATION LEARNING

The method injects fixed pretrained semantic features into a self-supervised depth network through pixel-adaptive decoder convolutions, then uses two-stage resampling to address dynamic-object bias and infinite-depth failures.

  • 4 SEMANTICALLY-GUIDED GEOMETRIC REPRESENTATION LEARNING: The architecture combines a depth network trained end-to-end with a fixed pretrained semantic network that supplies guidance.Only the depth network is optimized during self-supervised learning.
  • 4.1 SEMANTICALLY-GUIDED DEPTH FEATURES: Pixel-adaptive convolutions use processed multi-level semantic feature maps to generate semantic-dependent geometric features in the depth decoder.Semantic maps are processed with 3 × 3 and 1 × 1 convolutions, Group Normalization, and ELU nonlinearities before guiding corresponding layers.
  • 4.1 SEMANTICALLY-GUIDED DEPTH FEATURES: Guidance kernels act as content-dependent secondary weights on standard convolutional weights, changing their influence on resulting depth features.This limits the use of features across semantically different regions, such as sky and pedestrian areas.
  • 4.1 SEMANTICALLY-GUIDED DEPTH FEATURES: The semantic guidance network uses a Feature Pyramid Network with a ResNet backbone, although the proposed architecture is not restricted to this choice.The selected FPN has been used for semantic and instance-level prediction in panoptic segmentation.
  • 4.3 TWO-STAGE TRAINING: Dynamic objects create infinite-depth failures because photometric self-supervision assumes a static world and models only camera ego-motion.The proposed semantic features may reinforce this bias because the failure occurs mainly on dynamic categories such as cars and motorcycles.
  • 4.3 TWO-STAGE TRAINING: Two-stage training filters the dataset using first-stage depth predictions that violate a geometric prior, with ground-plane estimates obtained through robust nearby-ground pixels and RANSAC.The filtered data are then used to train the second-stage depth network.

5 EXPERIMENTAL RESULTS

On KITTI, the proposed semantically guided architecture improves self-supervised monocular depth estimation across standard benchmarks, depth networks, semantic classes, and fine-grained structures. Two-stage training particularly improves class-average performance for dynamic-object classes affected by infinite-depth predictions.

  • Overall depth estimation: The proposed architecture further improves the state of the art in self-supervised monocular depth estimation on KITTI and outperforms methods using semantic information, including ground-truth labels.Table 1 evaluates depths up to 80m and distinguishes monocular, stereo, depth-supervised, semantic, and instance-aware methods.
  • Qualitative results: The semantically guided architecture produces sharper boundaries and better object delineation, especially for distant or visually ambiguous structures.Qualitative comparisons use input images, baseline depth maps, and proposed depth maps.
  • Ablation across depth networks: The method consistently improves different depth networks across all considered metrics when using either ResNet-18 or ResNet-50 encoders.The same pretrained semantic network is used in both encoder variations.
  • Class-specific performance: Depth performance improves across all predicted classes, with especially clear gains for fine-grained structures and sharp boundaries such as poles and traffic signs.Class-specific evaluation bins pixels using predicted semantic categories rather than ground-truth segmentation.
  • Two-stage training: 0.121 to 0.117 Abs-Rel is the class-average improvement from two-stage training, while cars improve from 0.200 to 0.177 and motorcycles from 0.091 to 0.069.Pixel-average performance does not change significantly because pixels affected by infinite depth are a small fraction of all pixels.

6 CONCLUSION

The paper introduces semantic guidance for self-supervised monocular depth estimation through a fixed pretrained network and pixel-adaptive convolutions, alongside two-stage resampling for dynamic-object bias. Experiments show consistent improvements across different monodepth architectures and establish a new state of the art.

  • Contribution: A fixed pretrained semantic network guides multi-level depth features through pixel-adaptive convolutions within self-supervised monocular depth estimation.The architecture learns semantic-aware geometric representations in a structure-from-motion setting.
  • Training strategy: Two-stage training resamples data to overcome the bias toward predicting dynamic objects at infinite depth.The training strategy addresses a failure mode associated with dynamic objects.
  • Conclusion: Experiments show consistent performance improvements across different monodepth architectures, establishing a new state of the art in self-supervised monocular depth estimation.The paper identifies instance masks, optical flow, surface normals, and self-supervised semantic-network fine-tuning as future directions.

A PRE-TRAINING THE SEMANTIC SEGMENTATION NETWORK

Pre-training the semantic network is essential for useful guidance: semantic information improves depth estimation, while jointly fine-tuning the semantic network can erase that benefit.

  • Adding semantic information improves depth estimation, but also increases model complexity and trainable parameters.The analysis tests whether the gain comes from semantic features rather than secondary-network complexity.
  • Untrained semantic features produce significantly worse results because the depth network must filter spurious guidance.Jointly fine-tuning both networks improves these results, but still does not surpass the baseline.
  • Half-CityScapes pre-training transfers useful information despite lower validation mIoU of around 70% versus 75% for full pre-training.Fine-tuning both networks in this setting hurts performance, attributed to forgetting information in the secondary network.
  • Full CityScapes pre-training with depth-only fine-tuning yields the best reported performance, whereas fine-tuning both networks returns results similar to the baseline.Table 3 distinguishes ImageNet and CityScapes pre-training, including half-dataset pre-training, from depth or semantic-network fine-tuning.

B UNCERTAINTY AND GENERALIZATION TO DIFFERENT OBJECTS

The framework can use semantic features when they are reliable and disregard them when they are erroneous, supporting depth reconstruction across varied object-level failures.

  • Decoded semantic features remain useful despite domain gaps because the framework does not use semantic predictions directly.The paper evaluates robustness to semantic-network degradation and erroneous predictions.
  • Erroneous semantic predictions can coexist with accurate depth predictions in the proposed semantically-guided framework.Figure 6 presents qualitative examples of this behavior.
  • Imprecise boundaries and wrong classifications do not prevent recovery of object shape and depth with sharp delineation.Traffic signs and trucks are reconstructed accurately even when their semantic contours or classes are incorrect.
  • The framework reconstructs objects absent from the ontology and removes hallucinated object contours.Examples include a trash can labeled as fence and an erroneously introduced person contour.
  • These examples indicate that the framework leverages semantic information when accurate and discards it when photometric reconstruction favors another result.The conclusion is framed as reasoning over uncertainty in semantic classification.

C GENERALIZATION TO DIFFERENT DATASETS

Evaluation across datasets indicates improved generalization: the semantically-guided architecture outperforms both state-of-the-art methods and the baseline on substantially different environments.

  • The proposed architecture outperforms state-of-the-art methods and the Guizilini et al. (2019) baseline on the cross-dataset evaluation.The result provides additional evidence of generalization improvements on significantly different data.
  • Table 4 evaluates networks trained on KITTI and CityScapes using NuScenes data from different platforms and environments.The training domains are Karlsruhe, Germany, while evaluation includes Boston, USA and Singapore.
Loading 2002.12319v1…