Source-linked AI summary
DADA: Depth-aware Domain Adaptation in Semantic Segmentation
Tuan-Hung Vu, Himalaya Jain, Maxime Bucher, Matthieu Cord, Patrick Pérez
TL;DR
Semantic-segmentation UDA must bridge source-target domain gaps, while existing methods often ignore available source depth. DADA uses depth as training-only privileged information through auxiliary prediction, feature fusion, and depth-aware adaptation, and reports consistent gains across synthetic-to-real benchmarks.
Problem
Domain gaps can sharply reduce target-domain segmentation performance, while most UDA methods use semantic labels but ignore available source depth.
Method
DADA uses source depth as privileged information through an auxiliary depth-prediction task, feature fusion, and depth-aware adversarial adaptation.
Results
DADA consistently outperforms other UDA methods and achieves state-of-the-art performance on challenging synthetic-to-real benchmarks.
Takeaways & Limitations
Depth can support target-domain semantic segmentation when used as complementary training supervision and in domain alignment.
Takeaways & Limitations
The framework has failure cases involving indistinguishable objects from blurry depth outputs, and excessive attention to depth regression can hurt the main task.
Abstract
from arXiv · showhide
Unsupervised domain adaptation (UDA) is important for applications where large scale annotation of representative data is challenging. For semantic segmentation in particular, it helps deploy on real "target domain" data models that are trained on annotated images from a different "source domain", notably a virtual environment. To this end, most previous works consider semantic segmentation as the only mode of supervision for source domain data, while ignoring other, possibly available, information like depth. In this work, we aim at exploiting at best such a privileged information while training the UDA model. We propose a unified depth-aware UDA framework that leverages in several complementary ways the knowledge of dense depth in the source domain. As a result, the performance of the trained semantic segmentation model on the target domain is boosted. Our novel approach indeed achieves state-of-the-art performance on different challenging synthetic-2-real benchmarks.
1. Introduction
UDA addresses the domain gap that can sharply reduce semantic-segmentation performance when models trained on source imagery are evaluated on target imagery. DADA exploits source-domain depth as privileged information through auxiliary depth learning and depth-aware adaptation to improve target-domain segmentation.
- The domain gap between source and target image distributions can cause drastic drops in target semantic-segmentation performance.
- Synthetic-to-real UDA is attractive because synthetic source labels are free, while virtual engines can provide dense depth alongside images.SYNTHIA supplies virtual depth maps for synthesized urban scenes, but most UDA methods ignore such extra information.
- DADA uses source depth as privileged information through an auxiliary depth-regression task that enriches features and modifies domain adaptation.
- The proposed depth-aware adversarial training uses late fusion because visual information at different depth levels should be treated differently.
- DADA learning jointly aligns segmentation-based and depth-based source and target information while accounting for scene geometry.
- DADA architecture fuses depth-specific and standard CNN appearance features before forwarding them through segmentation classifiers.
- Evaluations report state-of-the-art performance on challenging synthetic-to-real benchmarks, with ablation studies examining the framework.
2. Related works
Related UDA work for semantic segmentation includes discrepancy minimization, adversarial alignment, generative source translation, and privileged-information approaches. DADA differs from SPIGAN by using depth beyond generator regularization.
- The review focuses on UDA approaches targeting visual semantic segmentation.
- UDA methods for segmentation commonly use adversarial training to make intermediate or final representations domain-agnostic.
- Generative approaches translate source samples into target-like images while preserving source semantic content and labels for supervised segmentation training.
- Other UDA methods combine adversarial and generative approaches, self-training, or curriculum-style learning.
- Privileged information is additional source-data information available only during training, and SPIGAN uses depth regression to support pixel-level adaptation.
- DADA also uses depth regression, but unlike SPIGAN, it does not use depth only as regularization for a generator.
3. Depth-aware domain adaptation
DADA integrates monocular depth prediction into a segmentation network and uses depth-aware adversarial alignment to adapt source and target outputs. Its fusion operations inject geometry into representations and discriminator inputs.
- DADA Network Architecture.: DADA modifies a semantic-segmentation network with a monocular depth-regression branch and trains it using source segmentation and depth supervision.
- DADA Network Architecture.: The auxiliary branch encodes backbone features into depth predictions, then decodes and fuses encoded features back into the main branch.
- DADA Network Architecture.: Element-wise product fusion combines depth-specific and backbone features before the remaining classification modules produce segmentation predictions.
- DADA Network Architecture.: The network jointly predicts a soft segmentation map and depth, with parameters learned by minimizing source segmentation and depth losses.
- DADA Network Architecture.: The reverse Huber loss penalizes larger depth residuals with an ℓ2 term while emphasizing gradients from small-residual samples with ℓ1.
- DADA Adversarial Learning Scheme: UDA aligns source and target soft segmentation outputs so a discriminator cannot distinguish their domains.
- DADA Adversarial Learning Scheme: DADA fusion multiplies weighted self-information by inverse depth, emphasizing scene elements closer to the ego-camera before adversarial adaptation.
- DADA Adversarial Learning Scheme: The resulting depth-aware map carries 3D structural and geometrical information intended to remain consistent across domains.
4. Experiments
Experiments evaluate DADA on synthetic-to-real semantic-segmentation benchmarks, using SYNTHIA as the annotated source and Cityscapes or Vistas as targets. Across benchmarks and ablations, depth-aware adaptation improves performance, especially for vehicle and human categories, while reducing annotation requirements and exposing depth-quality limitations.
- Benchmarks: DADA achieves state-of-the-art performance across three synthetic-to-real benchmarks using SYNTHIA as source and Cityscapes or Vistas as target domains.The experiments include 16-class and 7-class Cityscapes protocols and a 7-class Vistas protocol.
- Quantitative results: 1.8% D-Gain on SYNTHIA→Cityscapes nearly doubles SPIGAN’s gain, with improvements concentrated in car (+7%), bus (+8.1%), and bike (+5.1%).DADA also reports moderate gains on object classes such as light and pole and outperforms other baselines on the 13-class subset.
- Quantitative results: +3.2% mIoU over AdvEnt on the 7-class Cityscapes benchmark, including +3.7% on vehicle and +3% on human.At 320 × 640 resolution, using depth adds +14.0% to human-category IoU.
- Quantitative results: +2.4% mIoU over AdvEnt on SYNTHIA→Vistas, while DADA achieves 30% negative transfer versus SPIGAN’s 42%.The reported Vistas results include the best per-class IoUs on that benchmark.
- Ablation studies: Ablations show that separate depth-space adaptation helps, while combining feature fusion with DADA fusion performs best by emphasizing closer objects during adversarial training.The seven setups compare source-only training, AdvEnt components, depth adaptation, feature fusion, and DADA fusion.
- Annotation effort: Using 50% of SYNTHIA with segmentation and depth annotations matches AdvEnt trained on all images: 40.9% versus 40.8%.The authors identify reduced manual semantic-annotation requirements as practically important when depth annotation is automatic.
5. Conclusion
DADA uses source-domain depth as privileged information to improve target-domain semantic segmentation in UDA. It combines auxiliary depth prediction, feature fusion, and depth-aware adaptation, consistently outperforming other UDA methods on synthetic-to-real benchmarks.
- DADA leverages source-domain depth as privileged information to assist target-domain semantic segmentation.
- An auxiliary depth-prediction task enriches visual features through fusion and modifies the original adaptation loss.
- DADA consistently outperforms other UDA methods across different synthetic-to-real semantic segmentation benchmarks.
- Future work considers real-world settings where source-domain depth is only sparsely available, such as from automotive LiDAR scanners.