Source-linked AI summary
Category Anchor-Guided Unsupervised Domain Adaptation for Semantic Segmentation
Qiming Zhang, Jing Zhang, Wei Liu, Dacheng Tao
TL;DR
Semantic segmentation UDA is challenged by domain shift and category-agnostic alignment, while target labels are unavailable. CAG-UDA uses source category centroids as anchors for target pseudo-labeling and category-aware feature alignment, with progressive training. It achieves state-of-the-art results on GTA5→Cityscapes and SYNTHIA→Cityscapes, including 50.2 mIoU in GTA5→Cityscapes.
Problem
Domain shift and category-agnostic feature alignment hinder semantic segmentation UDA when target-domain labels are unavailable.
Method
CAG-UDA uses source category centroids as anchors to identify active target features, assign pseudo-labels, align categories, and progressively train the model.
Results
CAG-UDA achieves new state-of-the-art performance in GTA5→Cityscapes and SYNTHIA→Cityscapes, reaching 50.2 mIoU in GTA5→Cityscapes.
Takeaways & Limitations
Explicit category anchors support category-wise feature alignment and shared discriminative representations for source and target domains.
Takeaways & Limitations
CAG-UDA depends on reliable pseudo-labels; removing its warm-up stage causes a 6.3 mIoU drop, and a stage-free end-to-end model remains future work.
Abstract
from arXiv · showhide
Unsupervised domain adaptation (UDA) aims to enhance the generalization capability of a certain model from a source domain to a target domain. UDA is of particular significance since no extra effort is devoted to annotating target domain samples. However, the different data distributions in the two domains, or \emph{domain shift/discrepancy}, inevitably compromise the UDA performance. Although there has been a progress in matching the marginal distributions between two domains, the classifier favors the source domain features and makes incorrect predictions on the target domain due to category-agnostic feature alignment. In this paper, we propose a novel category anchor-guided (CAG) UDA model for semantic segmentation, which explicitly enforces category-aware feature alignment to learn shared discriminative features and classifiers simultaneously. First, the category-wise centroids of the source domain features are used as guided anchors to identify the active features in the target domain and also assign them pseudo-labels. Then, we leverage an anchor-based pixel-level distance loss and a discriminative loss to drive the intra-category features closer and the inter-category features further apart, respectively. Finally, we devise a stagewise training mechanism to reduce the error accumulation and adapt the proposed model progressively. Experiments on both the GTA5$\rightarrow $Cityscapes and SYNTHIA$\rightarrow $Cityscapes scenarios demonstrate the superiority of our CAG-UDA model over the state-of-the-art methods. The code is available at \url{https://github.com/RogerZhangzz/CAG_UDA}.
1 Introduction
The paper addresses unsupervised domain adaptation for semantic segmentation, where synthetic-to-real domain discrepancies degrade performance and category-agnostic alignment can confuse target categories. CAG-UDA uses source category anchors, category-aware alignment, and progressive training to improve adaptation.
- Motivation: Synthetic-to-real differences in textures, layouts, colors, and illumination create domain discrepancies that sharply degrade segmentation performance.Pixel-level annotation is also expensive, motivating synthetic datasets such as GTA5 and SYNTHIA.
- Motivation: Global marginal-distribution matching does not ensure that target samples from different categories remain properly separated.This limitation can compromise predictions under category-agnostic feature alignment.
- CAG-UDA: CAG-UDA uses source category-wise feature centroids as explicit anchors to identify active target features and assign pseudo-labels.The approach is motivated by the tendency of same-category features to cluster together.
- CAG-UDA: Anchor-based distance and discriminative losses respectively reduce intra-category variance and increase inter-category feature variance.The model also uses stagewise training to reduce error accumulation from incorrect pseudo-labels.
- Results: CAG-UDA achieves new state-of-the-art performance in both GTA5→Cityscapes and SYNTHIA→Cityscapes scenarios.This is the paper's reported overall contribution across the two benchmark adaptation settings.
2 Related Work
Prior segmentation adaptation methods align global or category-level distributions and use pseudo-labeling, but CAG-UDA directly aligns category-wise features with explicit anchors. Its design targets local category mismatches and category imbalance more directly than implicit alignment approaches.
- Feature distribution alignment: Earlier methods align domain distributions at the appearance, feature, or output level, including global marginal-distribution matching.Global matching can leave local category-wise feature shifts unresolved.
- Feature distribution alignment: Implicit category-level methods use weighted adversarial learning or task-specific classifiers to align category-level feature distributions.These approaches seek discriminative domain-invariant features without directly using category anchors.
- CAG-UDA: CAG-UDA differs by directly aligning category-wise features in both domains through a category anchor-guided method.This design explicitly addresses category-level mismatches rather than relying only on implicit alignment.
- Pseudo-label assignment: Pseudo-labeling methods iteratively generate target labels and retrain models, while category-center methods minimize distances or contrastive domain discrepancies.The reviewed methods address adaptation through self-training or center-based objectives, including category-imbalance control in iterative self-training.
3 A category anchor-guided UDA model for semantic segmentation
CAG-UDA addresses category-agnostic alignment in semantic segmentation by using source-domain category anchors to guide target-feature selection, pseudo-labeling, and category-wise adaptation. Its objective combines source supervision with anchor-based target alignment, while stagewise training limits error accumulation.
- Network architecture: CAG-UDA uses an encoder, feature transformer fD, and classifier, with a category anchor-guided alignment module operating on fD features.The module includes category anchor construction, active target sample identification, and pseudo-label assignment.
- Category anchor construction: Category anchors are source-domain category feature centroids computed from fD features and fixed throughout each training stage.Each centroid represents the feature distribution of one category.
- Active target sample identification: Active target samples are near one category anchor and far from others, enabling reliable pseudo-label assignment without target annotations.The method compares the closest and second-closest anchor distances using a predefined margin Δd.
- Pseudo-label assignment: Pseudo-labels assign each active target feature the category of its closest anchor, providing supervision that is less dependent on the source-biased classifier.The paper motivates anchor-based pseudo-labels as more reliable than labels derived only from predicted probabilities.
- Objective functions: The objective combines source cross-entropy, source and target category-wise distance losses, and target cross-entropy losses based on anchor-derived pseudo-labels.These losses reduce intra-category variation and adapt the encoder, decoder, and classifier to the target domain.
- Stagewise training: Stagewise training pretrains on the source domain and progressively updates anchors, active states, and pseudo-label supervision to avoid iterative error accumulation.Single-stage updating was unstable and could trap the network at performance below 30 mIoU.
4 Experiments
Experiments evaluate CAG-UDA on GTA5→Cityscapes and SYNTHIA→Cityscapes using quantitative, qualitative, and ablation studies. The model achieves strong adaptation performance, particularly for small objects, while warm-up and stagewise training support reliable pseudo-labeling.
- Quantitative Results: 50.2 mIoU on GTA5→Cityscapes surpasses the source-only model by 16.1 and exceeds CLAN and DISE by 4.5.CAG-UDA performs especially well on fence, traffic sign, rider, train, and bike, while BLF performs better on several stuff categories.
- Quantitative Results: 51.7 mIoU on the Cityscapes testing set indicates strong reported generalization for GTA5→Cityscapes.The testing-set result is presented separately from the validation-set comparison.
- Quantitative Results: CAG-UDA outperforms previous state-of-the-art methods in SYNTHIA→Cityscapes despite the larger domain shift and remains particularly strong on small objects.The reported advantage includes pole, sign, person, and bike categories.
- Qualitative Results: Qualitative results show improved predictions for road, person, pole, sign, and other small objects, with CAs-based pseudo-label assignment activating small objects.Source-only training misclassifies road and person as sidewalk and building in one example.
- Ablation Studies: Ablations show category anchor-guided PLA, cross-loss complementarity, and stagewise training improve performance, while removing warm-up decreases mIoU by 6.3.An extra Stage 4 produces no improvement beyond 50.2 mIoU, indicating saturation in that experiment.
5 Conclusion
CAG-UDA adapts semantic segmentation to the target domain through category-wise feature alignment guided by category anchors. Its modules, complementary losses, and stagewise training support progressive adaptation with reduced error accumulation.
- CAG-UDA adapts segmentation models through category-wise feature alignment guided by category anchors.The model includes category anchor construction, active target sample identification, and pseudo-label assignment modules.
- Distance and cross-entropy losses based on identified active target samples complementarily enhance adaptation performance.
- Stagewise training progressively adapts CAG-UDA while reducing error accumulation.
- Experiments on GTA5 and SYNTHIA demonstrate superior generalization to Cityscapes compared with representative methods.