Source-linked AI summary
Taking A Closer Look at Domain Shift: Category-level Adversaries for Semantics Consistent Domain Adaptation
Yawei Luo, Liang Zheng, Tao Guan, Junqing Yu, Yi Yang
TL;DR
Unsupervised domain adaptation in semantic segmentation must reduce source–target domain shift without disrupting category-level alignments. CLAN uses co-training to adapt adversarial loss by local semantic alignment, and matches state-of-the-art segmentation accuracy on two transfer tasks.
Problem
Global adversarial alignment matches marginal feature distributions but can ignore category-level joint shifts, incorrectly mapping features and causing negative transfer.
Method
CLAN uses co-training to estimate each feature’s semantic alignment, reducing adversarial weight for well-aligned features and increasing it for poorly aligned features.
Results
CLAN matches state-of-the-art segmentation accuracy on GTA5 → Cityscapes and SYNTHIA → Cityscapes.
Takeaways & Limitations
Category-level adaptive adversarial weighting protects semantic consistency while pursuing domain alignment in unsupervised segmentation.
Takeaways & Limitations
Traditional adversarial alignment can still produce negative transfer by deconstructing existing local alignment despite perfect marginal alignment.
Abstract
from arXiv · showhide
We consider the problem of unsupervised domain adaptation in semantic segmentation. The key in this campaign consists in reducing the domain shift, i.e., enforcing the data distributions of the two domains to be similar. A popular strategy is to align the marginal distribution in the feature space through adversarial learning. However, this global alignment strategy does not consider the local category-level feature distribution. A possible consequence of the global movement is that some categories which are originally well aligned between the source and target may be incorrectly mapped. To address this problem, this paper introduces a category-level adversarial network, aiming to enforce local semantic consistency during the trend of global alignment. Our idea is to take a close look at the category-level data distribution and align each class with an adaptive adversarial loss. Specifically, we reduce the weight of the adversarial loss for category-level aligned features while increasing the adversarial force for those poorly aligned. In this process, we decide how well a feature is category-level aligned between source and target by a co-training approach. In two domain adaptation tasks, i.e., GTA5 -> Cityscapes and SYNTHIA -> Cityscapes, we validate that the proposed method matches the state of the art in segmentation accuracy.
1. Introduction
Unsupervised adaptation for semantic segmentation must bridge source–target domain shift without sacrificing category-level semantic consistency. CLAN adaptively weights adversarial alignment using local alignment estimates and achieves state-of-the-art-level results on two transfer tasks.
- Motivation: Semantic segmentation requires costly dense pixel-level annotations, motivating simulated data whose labels are plentiful but whose models generalize poorly to realistic domains.The generalization problem is attributed to different source and target data distributions, known as domain shift.
- Limitation of global alignment: Global adversarial alignment can match marginal feature distributions while ignoring category-level joint distributions, causing negative transfer for already aligned features.The problem worsens when the adversarial loss receives a larger weight.
- Proposed approach: CLAN prioritizes category-level alignment by protecting well-aligned classes and increasing adversarial pressure on classes whose feature distributions differ across domains.Co-training estimates semantic alignment: consistent predictions reduce adversarial influence, whereas disagreement increases it.
- Contributions: Adaptive adversarial weighting emphasizes category-level feature alignment as a means of reducing domain shift.The method applies different adversarial-loss weights to different features.
- Results: CLAN matches state-of-the-art segmentation accuracy on GTA5 → Cityscapes and SYNTHIA → Cityscapes.The reported evaluation covers two unsupervised domain adaptation transfer tasks.
2. Related Works
The related work centers on adversarial learning and co-training for unsupervised domain adaptation. CLAN differs by using co-training to enforce category-level feature alignment and adaptively weight adversarial loss according to local alignment.
- Adversarial learning: Adversarial adaptation methods reduce domain divergence, one component of the adaptation-loss bound, by aligning feature distributions across domains.This line of work focuses primarily on the domain-divergence term after treating source error and ideal-joint-hypothesis error as comparatively addressed.
- Co-training: Co-training alternates learners across distinct views and can assign pseudo-labels to unlabeled target samples for direct target-domain classification-loss optimization.Classifier diversity can be induced through dropout, consensus regularization, or parameter diversity.
- Position of this work: Unlike global feature alignment or classifier-consensus maximization, CLAN uses co-training to adaptively weight adversarial loss according to each feature’s local alignment.The paper presents this as an early attempt in segmentation to combine local alignment assessment with adaptive adversarial weighting.
3. Method
CLAN extends adversarial domain adaptation with category-level weighting so aligned features are protected while poorly aligned features receive stronger adaptation. It combines co-training, discrepancy-based weighting, segmentation, and adversarial objectives to preserve semantic consistency during global alignment.
- Motivation: Traditional adversarial adaptation aligns marginal feature distributions but can disrupt local category alignment and cause negative transfer.This can map same-class samples farther apart or redirect already aligned infrequent classes toward prevalent classes.
- Architecture: CLAN divides the generator into a feature extractor E and two classifiers C1 and C2 whose predictions provide diverse views of each pixel.The source pathway uses ensemble predictions for supervised segmentation and adversarial losses, while the target pathway additionally computes prediction discrepancy.
- Adaptive adversarial loss: CLAN uses classifier prediction discrepancy as a pixel-level alignment signal and multiplies it with the discriminator loss to adapt adversarial weighting.Large discrepancy increases the adversarial force on poorly aligned features, whereas small discrepancy reduces punishment for features whose category-level distributions already overlap.
- Training objective: The method combines segmentation, weight discrepancy, and self-adaptive adversarial losses, with hyperparameters controlling their relative importance.Training alternates optimization of G and D until the overall CLAN loss converges.
- Analysis: CLAN reduces adversarial loss for already aligned classes, preserving well-segmented infrequent categories such as poles and traffic signs during adaptation.The contrastive analysis attributes their preservation to category-level weighting rather than unrestricted global alignment.
- Analysis: The co-training discrepancy loss encourages E to capture domain-invariant pixel semantics rather than domain-specific visual properties.C1 and C2 are encouraged to differ in captured characteristics while agreeing on predictions, strengthening semantic consistency across domains.
4. Experiment
CLAN is evaluated on two synthetic-to-real segmentation transfers using standard benchmark comparisons, implementation settings, quantitative alignment analyses, and qualitative results. Across both tasks, CLAN improves segmentation and produces more semantically aligned features, while parameter studies examine convergence and adaptation performance.
- Datasets: CLAN is evaluated on GTA5 → Cityscapes and SYNTHIA → Cityscapes against existing domain adaptation methods.Cityscapes is the target domain; GTA5 and SYNTHIA are synthetic source domains.
- Implementation: The experiments use DeepLab-v2 with ImageNet-pretrained ResNet-101, parallel classification modules for co-training, and SGD or Adam optimization.A VGG-16 variant is also used for fair comparison with methods using the VGG backbone.
- Comparative Studies: +18.7% mIoU on VGG-16 and +6.6% on ResNet-101 are reported for CLAN over source-only segmentation on GTA5 → Cityscapes.CLAN also improves over TAN by over +1.6% mIoU in both architectures and is reported as on par with CBST.
- Comparative Studies: 39.3% mIoU is reported for CLAN on VGG-16 in SYNTHIA → Cityscapes, a 19.1% improvement over non-adaptive segmentation.On ResNet-101, CLAN improves over source-only segmentation by 9.2% and over TAN by 1.9%; it also outperforms CBST by 3.2% mIoU.
- Feature Distribution: CLAN’s cluster center distance decreases monotonically during training and converges to a smaller value than TAN, indicating better category-level feature alignment.The final cluster center distance is smaller for CLAN in most classes, especially infrequent classes prone to negative transfer.
- Parameter Studies: Varying λlocal and ϵ shows that CLAN can steadily outperform TAN near the best parameter values while maintaining convergence and improving adaptation performance.Figure 4 compares mean IoU and discriminator convergence across parameter settings.
5. Conclusion
CLAN addresses semantic inconsistency from global feature alignment in unsupervised domain adaptation by adaptively weighting adversarial loss according to category-level alignment. The method prevents well-aligned features from being incorrectly mapped and achieves competitive segmentation accuracy against state-of-the-art UDA approaches.
- CLAN addresses semantic inconsistency incurred by global feature alignment during unsupervised domain adaptation.
- The method examines category-level data distributions and adaptively weights adversarial loss for each feature according to its category-level alignment.
- CLAN prevents well-aligned features from being incorrectly mapped as a side effect of pure global distribution alignment.
- Experimental results show very competitive segmentation accuracy compared with state-of-the-art unsupervised domain adaptation approaches.