Source-linked AI summary
Classes Matter: A Fine-grained Adversarial Approach to Cross-domain Semantic Segmentation
Haoran Wang, Tong Shen, Wei Zhang, Lingyu Duan, Tao Mei
TL;DR
Domain-adaptive segmentation must address domain shift while avoiding the class mismatches that global feature alignment can introduce. The paper proposes fine-grained adversarial alignment using a class-aware discriminator and domain encodings, achieving new state-of-the-art results across three benchmarks.
Problem
Domain shift causes performance drops, while global feature alignment overlooks class structure and can mix samples with different semantic labels.
Method
FADA uses a fine-grained discriminator modeling domain and class information, with prediction-derived domain encodings supervising class-level feature alignment.
Results
FADA achieves new state-of-the-art results and large-margin improvements on GTA5 →Cityscapes, SYNTHIA →Cityscapes, and Cityscapes →Cross-City.
Takeaways & Limitations
Class-level adversarial alignment preserves semantic category structure while adapting features across domains.
Takeaways & Limitations
Global alignment methods can mix samples with different semantic labels, producing mismatched classes across domains.
Abstract
from arXiv · showhide
Despite great progress in supervised semantic segmentation,a large performance drop is usually observed when deploying the model in the wild. Domain adaptation methods tackle the issue by aligning the source domain and the target domain. However, most existing methods attempt to perform the alignment from a holistic view, ignoring the underlying class-level data structure in the target domain. To fully exploit the supervision in the source domain, we propose a fine-grained adversarial learning strategy for class-level feature alignment while preserving the internal structure of semantics across domains. We adopt a fine-grained domain discriminator that not only plays as a domain distinguisher, but also differentiates domains at class level. The traditional binary domain labels are also generalized to domain encodings as the supervision signal to guide the fine-grained feature alignment. An analysis with Class Center Distance (CCD) validates that our fine-grained adversarial strategy achieves better class-level alignment compared to other state-of-the-art methods. Our method is easy to implement and its effectiveness is evaluated on three classical domain adaptation tasks, i.e., GTA5 to Cityscapes, SYNTHIA to Cityscapes and Cityscapes to Cross-City. Large performance gains show that our method outperforms other global feature alignment based and class-wise alignment based counterparts. The code is publicly available at https://github.com/JDAI-CV/FADA.
1 Introduction
Domain shift causes segmentation performance to drop across synthetic and realistic domains, while global alignment can ignore class structure. FADA addresses this with class-level adversarial alignment supervised by domain encodings.
- Synthetic-to-real deployment suffers performance drops because source and target domains have different data distributions.
- Most adversarial adaptation methods align global feature distributions without explicitly preserving relationships among semantic classes.
- FADA models joint domain-and-class information with a fine-grained discriminator to align features according to semantic categories.
- Domain encodings derived from predictions on both domains replace binary domain labels as supervision for fine-grained adversarial alignment.
- Class Center Distance analysis verifies class-level alignment, while experiments report significant improvements on GTA5 →Cityscapes, SYNTHIA →Cityscapes, and Cityscapes →Cross-City.
2 Related Work
Domain adaptation methods address distribution differences in semantic segmentation, but global alignment can mix differently labeled samples. The proposed framework instead uses fine-grained class-level discrimination.
- 2.1 Semantic Segmentation: Semantic segmentation predicts a unique semantic label for every pixel in an input image.
- 2.2 Domain Adaptation: Domain adaptation addresses performance drops caused by different training and testing distributions through feature alignment, including adversarial training.
- 2.3 Domain Adaptive Semantic Segmentation: Global feature alignment can mix samples with different semantic labels and produce class mismatches across domains.
- 2.3 Domain Adaptive Semantic Segmentation: The proposed framework feeds source and target semantic features to a convolutional fine-grained discriminator supervised by prediction-derived domain encodings.
3 Method
FADA extends adversarial feature alignment from binary domain discrimination to class-aware alignment using domain encodings derived from segmentation predictions. Its fine-grained discriminator models domain and class structure while adversarial training encourages domain-invariant features without hurting feature–class relationships.
- Traditional feature alignment: Traditional adaptation aligns source and target feature distributions with a binary discriminator, but global alignment can overlook class-level structure.The discriminator distinguishes domains while the feature network learns domain-invariant representations.
- Fine-grained adversarial learning: The discriminator splits each binary domain channel into K channels, representing domain confidence as a class-wise distribution.This design lets the discriminator model underlying relationships between classes and encourages class-level alignment.
- Domain encodings: Domain encodings replace binary labels with [a; 0] for the source and [0; a] for the target, where a is K-dimensional classifier-derived knowledge.The paper compares binary labels, one-hot hard labels, and multi-channel soft labels as ways to construct these encodings.
- Training and inference: The adversarial loss makes target features resemble source features while preserving their relationship with classes, and the adaptation component is removed during inference.The final model uses the original segmentation network with adapted weights.
- Extracting class knowledge: One-hot encodings select the most confident predicted class, optionally ignoring samples below a confidence threshold to reduce noise.Soft encodings instead use temperature-scaled logits to produce class probability distributions.
4 Experiments
FADA is evaluated on three unsupervised domain-adaptive segmentation benchmarks and achieves improved semantic segmentation, class-level feature alignment, and component-wise gains.
- Benchmarks: FADA is evaluated on Cityscapes →Cross-City, SYNTHIA →Cityscapes, and GTA5 →Cityscapes benchmarks.
- Cross-city adaptation: 8.5% improvement over source-only baselines and 2.25% over the previous best method are achieved on average across four Cross-City cities.
- Synthetic-to-real adaptation: 16.4% and 13.9% gains over source-only models are reported for VGG16 and ResNet101, respectively, on SYNTHIA →Cityscapes.
- Synthetic-to-real adaptation: 15.5% and 12.4% improvements are reported on different baselines for GTA5 →Cityscapes, with over 4% general improvement over feature-level methods.
- Feature distribution: FADA achieves the lowest mean CCD value, 1.1, indicating better class-level alignment and preservation of consistent class structures between domains.
- Ablation studies: Fine-grained adversarial training improves performance by 10.1%, while self distillation and multi-scale testing add 2.3% and 0.7%, respectively.
5 Conclusion
The paper proposes fine-grained adversarial training for domain-adaptive semantic segmentation, using category-level information to guide feature alignment. Experiments validate its effectiveness across three popular tasks, with large performance gains over other methods.
- A fine-grained discriminator distinguishes domains and captures category-level information to guide fine-grained feature alignment.
- Binary domain labels are generalized to domain encodings to incorporate class information during adversarial training.
- The method achieves new state-of-the-art results on three popular domain-adaptation tasks, outperforming other methods by a large margin.