PaperlayerView exact revision on arXiv

Source-linked AI summary

Three Ways to Improve Semantic Segmentation with Self-Supervised Depth Estimation

Lukas Hoyer, Dengxin Dai, Yuhua Chen, Adrian Köring, Suman Saha, Luc Van Gool

arXiv:2012.10782v2cs.CV

TL;DR

Dense segmentation labels are costly, and this paper investigates using self-supervised depth estimation from unlabeled image sequences to improve semi-supervised segmentation. Its approach combines depth-feature transfer, geometry-aware DepthMix augmentation, and SDE-based sample selection; on Cityscapes, it reports gains for all three components and state-of-the-art semi-supervised results.

  • Problem

    Semantic segmentation requires costly dense annotations, motivating methods that improve performance with fewer labeled examples.

    Source: §1 · p. 1

  • Method

    The method uses self-supervised monocular depth estimation to transfer features, mix labeled images and labels according to scene geometry, and select annotation samples using SDE.

    Source: §1 · pp. 1–2

  • Results

    With 1/30 of available labels, the method achieves 92% of full-annotation baseline performance and slightly outperforms it with 1/8 labels, while improving the previous state of the art.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The paper presents SDE as a shared resource for representation learning, augmentation, and annotation selection in semi-supervised segmentation.

    Source: §1 · p. 1, §5 · p. 8

Abstract

from arXiv · show

Training deep networks for semantic segmentation requires large amounts of labeled training data, which presents a major challenge in practice, as labeling segmentation masks is a highly labor-intensive process. To address this issue, we present a framework for semi-supervised semantic segmentation, which is enhanced by self-supervised monocular depth estimation from unlabeled image sequences. In particular, we propose three key contributions: (1) We transfer knowledge from features learned during self-supervised depth estimation to semantic segmentation, (2) we implement a strong data augmentation by blending images and labels using the geometry of the scene, and (3) we utilize the depth feature diversity as well as the level of difficulty of learning depth in a student-teacher framework to select the most useful samples to be annotated for semantic segmentation. We validate the proposed model on the Cityscapes dataset, where all three modules demonstrate significant performance gains, and we achieve state-of-the-art results for semi-supervised semantic segmentation. The implementation is available at https://github.com/lhoyer/improving_segmentation_with_selfsupervised_depth.

1. Introduction

The paper uses self-supervised depth estimation (SDE) to improve semi-supervised semantic segmentation through representation learning, geometry-aware augmentation, and automatic sample selection. On Cityscapes, it reports strong results with substantially fewer annotations.

1. Introduction

  • The three strategies apply SDE across representation learning, data augmentation, and annotation selection to improve segmentation while reducing annotation needs.The paper frames their shared resource as knowledge learned from unlabeled image sequences.Source: §1 · p. 1
  • DepthMix blends images and labels according to scene geometry, producing fewer artifacts than prior mixing methods [69] [47].The paper describes it as a geometry-aware augmentation strategy.Source: §1 · pp. 1–2
  • Automatic Data Selection for Annotation uses SDE as a proxy-task oracle and selects samples iteratively by diversity and uncertainty, without human-in-the-loop segmentation labeling.The approach is presented as improving flexibility, scalability, and efficiency, particularly for crowdsourced annotation.Source: §1 · p. 2
  • With 1/30 of the available labels, the method reaches 92% of full-annotation baseline performance and slightly outperforms that baseline with 1/8 of the labels.All three components demonstrate significant performance gains, and the combined method improves the previous state of the art for semi-supervised segmentation.Source: §1 · p. 2
  • SDE serves as an auxiliary task through transfer learning and joint segmentation-depth learning, improving segmentation especially when supervision is limited.The paper presents SDE as a way to exploit unlabeled image sequences for semi-supervised segmentation.Source: §1 · p. 2

2. Related Work

The related work surveys semi-supervised segmentation, active learning, and SDE-based approaches, positioning geometry-aware augmentation and SDE-driven annotation selection as differences from prior methods.

2.1. (Semi-Supervised) Semantic Segmentation

  • Prior semi-supervised segmentation uses adversarial training or consistency methods, including CutMix and ClassMix; DepthMix differs by respecting scene structure during mixing.The section also notes that earlier self-supervised representation methods were outperformed by ImageNet pretraining and were not considered practical for semi-supervised segmentation.Source: §2.1 · p. 2

2.2. Active Learning

  • Active learning commonly selects informative samples for human labeling using uncertainty or diversity, whereas this work replaces the human oracle with SDE.Prior unsupervised selection methods were limited in model scope or did not iteratively adapt selection to the current model's uncertainty.Source: §2.2 · p. 2

2.3. Improving Segmentation with SDE

  • Earlier SDE-segmentation studies generally targeted depth estimation or fully supervised segmentation; this work instead uses SDE to improve semi-supervised segmentation.The related work distinguishes prior depth-focused joint learning and semantic-segmentation methods restricted to full supervision.Source: §2.3 · p. 3

3. Methods

The methods use self-supervised depth estimation (SDE) across sample selection, data augmentation, and joint learning to improve semantic segmentation. Depth features guide which images to annotate, geometry-preserving mixing, and learning segmentation with depth as an auxiliary task.

3.3. Semi-Supervised Semantic Segmentation

  • SDE pretraining and joint learning with depth as an auxiliary task support semantic segmentation, with unlabeled image sequences also used to generate pseudo-labels for training.The method combines SDE and segmentation through a shared encoder and uses attention-guided distillation to exchange intermediate decoder features.Source: §3 · p. 3, §3.3 · p. 5

3. Methods

  • The framework covers annotation selection, DepthMix augmentation, and multi-task learning, with SDE learned from unlabeled image sequences.Multi-task pretraining combines SDE with image classification.Source: §3 · p. 3

3.1. Automatic Data Selection for Annotation

  • Automatic annotation selection iteratively combines SDE-feature diversity with uncertainty estimated from a single-image depth student trained on the selected images.The student is retrained from scratch at each selection step; its log-scale disparity error against the SDE teacher signals uncertainty.Source: §3.1 · pp. 3–4

3.2. DepthMix Data Augmentation

  • DepthMix creates virtual samples by copying image regions with a binary mask and applying the same mask to their labels.Unlike geometry-agnostic mixing, its depth-based mask favors closer objects, mitigating scene-structure violations such as distant objects pasted over nearer ones.Source: §3.2 · pp. 4–5

4. Experiments

The Cityscapes experiments evaluate SDE-based semi-supervised segmentation, its component contributions, and automatic data selection. The proposed method outperforms competing approaches across labeled subset sizes, with each component adding performance gains.

4.1. Implementation Details

  • The implementation uses Cityscapes labeled images and nearby unlabeled frames, with a shared ResNet101 encoder and separate segmentation and SDE decoders.The multi-task network also uses an attention-guided distillation module; training details include an equal weighting of segmentation and SDE losses.Source: §4.1 · p. 6

4.2. Semi-Supervised Semantic Segmentation

  • The method outperforms other approaches across labeled subset sizes, and adding data selection lets it slightly exceed the fully supervised baseline with 1/8 of the labels.The comparison is on Cityscapes; the reported exception for original ClassMix at 100 samples does not hold when ClassMix uses the authors’ setting.Source: §4.2 · pp. 6–7, Table 1 · p. 7
  • Adding Cityscapes unlabeled sequences to ClassMix produces no significant difference in the reported comparison.The authors suggest this may reflect high correlation between Cityscapes images and video frames.Source: §4.2 · p. 7

4.3. Ablation Study

  • With 372 (2975) labeled samples, SDE transfer and multi-task learning add +2.10 (+1.99), DepthMix with pseudo-labels +5.00 (+2.06), and data selection +5.11 (–) mIoU points.Combining SDE multi-tasking and DepthMix adds +7.52 (+3.40); all three components add +8.87 (–) mIoU points.Source: §4.3 · pp. 7–8, Table 2 · p. 7
  • DepthMix exceeds ClassMix by 0.98 (0.23) percentage points for 372 (2975) annotated samples.The authors attribute the comparison to geometry-aware augmentation; depth estimates model occlusions and produce synthetic samples with realistic appearance.Source: §4.3 · p. 8, Figure 5 · p. 8
  • Depth multi-task learning improves fence, traffic light, traffic sign, rider, truck, and motorcycle, while DepthMix further improves wall, truck, bus, and train.The authors suggest characteristic depth profiles may explain the multi-task gains, and presenting difficult objects in another context may help DepthMix generalize.Source: Figure 4 · p. 7, §4.3 · p. 8
  • The authors report that SDE can be trained on a different dataset from segmentation when both datasets have similar visual domains.This applicability is reported in supplementary material.Source: §4.3 · p. 8

4.4. Automatic Data Selection for Annotation

  • The data-selection combination DS+US outperforms random selection and active learning based on segmentation entropy, without requiring human-in-the-loop annotation.The authors report that selected samples improve truck, bus, and train performance and support the relationship between depth-estimation and segmentation sample difficulty.Source: §4.4 · p. 8, Table 3 · p. 8

5. Conclusion

The conclusion presents three ways to use self-supervised depth estimation to improve semantic segmentation. Cityscapes experiments show gains over baselines and competing methods, with state-of-the-art semi-supervised performance.

  • The paper combines SDE feature transfer and joint depth-segmentation learning, geometry-aware DepthMix, and SDE-based automatic data selection without human-in-the-loop annotations.The authors report that DepthMix avoids inconsistent geometry in generated images.Source: §5 · p. 8

A. Further Implementation Details

The supplementary implementation details specify the network architecture, data-selection model, and timing measurement setup.

Network Architecture

  • The network uses DeepLabv3 with U-Net decoders for depth and segmentation, a ResNet101 encoder, ASPP features, and skip-connected upsampling blocks.An attention-guided multi-modal distillation module with a segmentation side output follows the third decoder block.Source: A. Further Implementation Details · p. 12

Runtime

Data Selection

  • The data-selection experiment uses a slimmed ResNet50 fSIDE architecture, while the depth student is trained with berHu loss.The selected annotated subset is evaluated for segmentation with the default architecture and training hyperparameters.Source: A. Further Implementation Details · p. 12

B. Cross-Dataset Transfer Learning

The paper evaluates whether Cityscapes SDE sequences can support semi-supervised segmentation on CamVid, a street-scene dataset. The reported results show gains over ClassMix and improvement with the fully labeled dataset.

  • For 50 labeled CamVid samples, SDE pretraining adds 3.6 mIoU points, pseudo-labels and DepthMix add another 4.07, and data selection adds another 1.41.The method outperforms ClassMix by 2.34 points with 50 labeled samples and 2.14 with 100; on the fully labeled dataset, it improves performance by 3.29 points.Source: B. Cross-Dataset Transfer Learning · p. 12

C. Further Example Predictions

Examples with 100 annotated training samples show clearer segmentation contours at pronounced depth boundaries and better differentiation among similar vehicle classes, while SDE can fail on cars moving directly in front of the camera.

C. Further Example Predictions

  • SDE improves contours near pronounced depth discontinuities and differentiation among similar classes such as truck, bus, and train.The figure provides further predictions for 100 annotated training samples.Source: C. Further Example Predictions · p. 12, Figure S6 · p. 13
  • SDE can fail to reconstruct cars driving directly in front of the camera, but this does not hinder transfer of depth-learned features to segmentation.Such cars remain at the same image location across the sequence, violating reconstruction assumptions.Source: C. Further Example Predictions · p. 12
  • Table S5 reports CamVid test-set mIoU with standard deviation over three random seeds, using SDE trained on Cityscapes sequences.The caption identifies DT as SDE Transfer Learning, XD as DepthMix, and S as Data Selection.Source: Table S5 · p. 13

D. DepthMix Real-World Examples

DepthMix generally produces realistic synthetic samples by using self-supervised depth estimates to model occlusions, but inaccurate depth can cause artifacts, especially around dynamic objects and mismatched depth and class boundaries.

D. DepthMix Real-World Examples

  • DepthMix generally models occlusions correctly and produces realistic synthetic samples.The examples show source images and depth estimates, mixed images and depth, and mixed segmentations; source segmentation labels are ground truth for simplicity.Source: D. DepthMix Real-World Examples · p. 13, Figure S7 · p. 14
  • Inaccurate SDE for dynamic objects can create inaccurate structure in the mixed image.The figure presents typical DepthMix failure cases.Source: D. DepthMix Real-World Examples · p. 13, Figure S8 · p. 15
  • When depth discontinuities do not align with class borders, DepthMix can produce artifacts in mixed images and segmentations, including sky within a building.The same kind of issue can occur in ClassMix when pseudo-labels define the mix mask.Source: D. DepthMix Real-World Examples · p. 13

Related papers

Loading 2012.10782v2…