Source-linked AI summary
Domain Adaptive Semantic Segmentation with Self-Supervised Depth Estimation
Qin Wang, Dengxin Dai, Lukas Hoyer, Luc Van Gool, Olga Fink
TL;DR
Semantic segmentation under source–target domain shift lacks reliable auxiliary supervision in many settings. The paper uses self-supervised depth available on both domains, learning semantic–depth correlations and depth-based adaptation difficulty to refine pseudo-labels; it reports state-of-the-art results on SYNTHIA-to-Cityscapes and GTA-to-Cityscapes.
Problem
Semantic segmentation must handle distribution shifts between labeled source and unlabeled target domains, while auxiliary-task supervision is not always available.
Method
CorDA learns domain-shared semantic–depth feature correlations and uses discrepancies between source and target depth decoders to estimate adaptation difficulty for refining target pseudo-labels.
Results
55.0% mIoU (stereo depth) is achieved on SYNTHIA-to-Cityscapes, a 6.7% absolute improvement over DACS.
Takeaways & Limitations
Self-supervised depth estimation can be integrated into existing UDA segmentation frameworks to improve target-domain segmentation.
Takeaways & Limitations
The method considers depth estimation as its auxiliary task.
Abstract
from arXiv · showhide
Domain adaptation for semantic segmentation aims to improve the model performance in the presence of a distribution shift between source and target domain. Leveraging the supervision from auxiliary tasks~(such as depth estimation) has the potential to heal this shift because many visual tasks are closely related to each other. However, such a supervision is not always available. In this work, we leverage the guidance from self-supervised depth estimation, which is available on both domains, to bridge the domain gap. On the one hand, we propose to explicitly learn the task feature correlation to strengthen the target semantic predictions with the help of target depth estimation. On the other hand, we use the depth prediction discrepancy from source and target depth decoders to approximate the pixel-wise adaptation difficulty. The adaptation difficulty, inferred from depth, is then used to refine the target semantic segmentation pseudo-labels. The proposed method can be easily implemented into existing segmentation frameworks. We demonstrate the effectiveness of our approach on the benchmark tasks SYNTHIA-to-Cityscapes and GTA-to-Cityscapes, on which we achieve the new state-of-the-art performance of $55.0\%$ and $56.6\%$, respectively. Our code is available at \url{https://qin.ee/corda}.
1. Introduction
The paper addresses semantic segmentation under source–target domain shift by using self-supervised depth estimation available across domains. CorDA learns domain-shared semantic–depth correlations and uses depth-based adaptation difficulty to refine target pseudo-labels, achieving state-of-the-art results on two benchmarks.
- Semantic segmentation models face domain shift from differences in illumination, appearance, viewpoints, and backgrounds between training and test data.
- Self-supervised depth estimation provides auxiliary supervision for learning domain-robust correlations between semantic and depth features.The paper motivates this through stable relationships such as skies being far away and roads and sidewalks being flat.
- CorDA learns task feature correlations with domain-shared multi-modal distillation and transfers them from the source domain to improve target segmentation.The correlation module models interactions and complementarity between semantic and depth features.
- The method estimates pixel-wise adaptation difficulty from discrepancies between domain-specific depth decoders and uses it to refine target semantic pseudo-labels.
- CorDA achieves new state-of-the-art segmentation performance on SYNTHIA-to-Cityscapes and GTA-to-Cityscapes.
2. Related Work
Prior work addresses domain adaptation through distribution alignment, geometric auxiliary information, multi-task distillation, and self-supervised learning. This paper is positioned within these directions while focusing on task correlation in UDA without target ground truth.
- UDA methods commonly reduce domain shift by aligning source and target distributions in output, feature, or input spaces.
- Geometric information, including auxiliary depth regression, has been used to support semantic learning and reduce domain gaps.
- Multi-task distillation combines task-specific representations through shared backbones and multi-modal distillation units.Most prior multi-task learning work assumes fully supervised tasks without domain shift.
- Self-supervised UDA methods improve target generalization with auxiliary predictions, but prior auxiliary tasks may not exploit correlations with the main task.
3. Methodology
CorDA uses self-supervised depth information from both domains to learn domain-robust semantic–depth correlations and refine target pseudo-labels according to estimated adaptation difficulty. Its shared correlation module and domain-specific depth decoders integrate these signals into an existing UDA segmentation architecture.
- Depth supervision: Target depth pseudo-ground truth is generated from stereo pairs or video sequences and used as fixed supervision during training.Off-the-shelf methods such as Semi-Global Matching and MonoDepth2 can generate the pseudo-depth information.
- Correlation-aware architecture: The framework uses a shared backbone, domain-specific depth heads, a shared semantic head, and a domain-shared task feature correlation module.The architecture produces intermediate semantic and depth predictions before learning their cross-task interaction.
- Training objectives: Intermediate semantic predictions use cross-entropy losses, while depth predictions use reverse Huber losses with separate source and target depth predictions.The target semantic loss uses pixel-wise pseudo-label weights, and inverse depth is adopted for depth learning losses.
- Correlation-aware architecture: The correlation module uses spatial attentions to capture mutual relationships between semantic and depth features and transfer more robust correlations across domains.The attentions capture complementary information while suppressing irrelevant information, and are shared from source to target.
- Final prediction: The final architecture retains independent source and target depth decoders while using distilled semantic and depth features for the final predictions.A shared semantic decoder generates target semantic predictions from the distilled semantic features.
- Pseudo-label refinement: CorDA re-weights target semantic pseudo-labels using a pixel-wise domain-gap indicator derived from source–target depth prediction discrepancies.The complete framework combines shared correlation learning with depth-guided difficulty refinement during training.
4. Experiments
Experiments evaluate CorDA on SYNTHIA-to-Cityscapes and GTA5-to-Cityscapes, including component ablations, depth-source robustness, and comparisons with prior methods. CorDA achieves 55.0% mIoU on SYNTHIA-to-Cityscapes and 56.6% mIoU on GTA5-to-Cityscapes.
- Ablation study: individual modules: 49.6% mIoU is obtained by directly adding source and target depth as auxiliary tasks to DACS, providing only a small improvement.The ablation uses stereo depth estimation for target data.
- Ablation study: individual modules: 54.2% mIoU follows explicit depth–semantic correlation learning, while removing its correlation modules reduces performance to 51.7% mIoU.The correlation-aware architecture yields a 4.6% absolute improvement over the corresponding baseline.
- Ablation study: individual modules: The adaptation-difficulty pseudo-label refinement adds a further 0.8% absolute improvement, producing the final CorDA framework.The refinement compares prediction discrepancies between source and target depth decoders.
- Ablation study: choice of pseudo depth ground truth: 54.5% mIoU with monocular depth and 55.0% mIoU with stereoscopic depth show robustness to the pseudo-depth source.Both depth estimations use only stereo pairs or image sequences from the same dataset.
- Comparison to the state-of-the-art approaches: 55.0% mIoU is achieved on SYNTHIA-to-Cityscapes with stereo depth, a 6.7% absolute improvement over DACS.The method outperforms other approaches using either monocular or stereo depth estimations.
- Results on GTA5-to-Cityscapes: 56.6% mIoU on GTA5-to-Cityscapes improves over DACS by 4.5% absolute and improves predictions for confusable sidewalk and road classes.The GTA5 setup uses monocular depth for GTA5 and stereo depth for Cityscapes.
5. Conclusions
The framework leverages self-supervised auxiliary-task guidance to bridge domain gaps in semantic segmentation. It transfers domain-shared semantic–depth correlations and uses depth-based adaptation difficulty to refine predictions, achieving state-of-the-art results on two benchmarks.
- The framework explicitly learns domain-shared correlations between semantic and auxiliary-task features to improve knowledge transfer to the target domain.A domain-shared task feature correlation module models these relationships.
- Depth-decoder prediction discrepancy approximates adaptation difficulty and is used to refine segmentation predictions.
- The integrated approach achieves state-of-the-art performance on SYNTHIA-to-Cityscapes and GTA-to-Cityscapes.