Source-linked AI summary
Pixel-Level Cycle Association: A New Perspective for Domain Adaptive Semantic Segmentation
Guoliang Kang, Yunchao Wei, Yi Yang, Yueting Zhuang, Alexander G. Hauptmann
TL;DR
Domain-adaptive semantic segmentation seeks accurate target-domain pixel predictions from source annotations, but prior discrepancy-minimization approaches often overlook pixel-wise relationships. The paper cyclically associates source and target pixels and contrastively strengthens valid pairs, reporting improved results on two benchmarks. The authors present the approach as a new perspective for domain adaptation.
Problem
Domain-adaptive segmentation seeks target-domain pixel predictions using source annotations, while prior discrepancy-minimization methods often overlook pixel-wise relationships.
Source: §1 · p. 1Method
The method cyclically associates source and target pixels by feature similarity and contrastively strengthens pairs whose source endpoints share a category.
Source: §1 · p. 2, §3.2 · p. 4Results
More than 10% mIoU improvement over source-only training is reported on GTAV → Cityscapes and SYNTHIA → Cityscapes, with favorable comparisons to prior state-of-the-art methods.
Source: §1 · p. 2Takeaways & Limitations
The authors present pixel-wise cross-domain similarity as a new perspective and their method as a baseline intended to inspire future research.
Source: §5 · p. 8
Abstract
from arXiv · showhide
Domain adaptive semantic segmentation aims to train a model performing satisfactory pixel-level predictions on the target with only out-of-domain (source) annotations. The conventional solution to this task is to minimize the discrepancy between source and target to enable effective knowledge transfer. Previous domain discrepancy minimization methods are mainly based on the adversarial training. They tend to consider the domain discrepancy globally, which ignore the pixel-wise relationships and are less discriminative. In this paper, we propose to build the pixel-level cycle association between source and target pixel pairs and contrastively strengthen their connections to diminish the domain gap and make the features more discriminative. To the best of our knowledge, this is a new perspective for tackling such a challenging task. Experiment results on two representative domain adaptation benchmarks, i.e. GTAV $\rightarrow$ Cityscapes and SYNTHIA $\rightarrow$ Cityscapes, verify the effectiveness of our proposed method and demonstrate that our method performs favorably against previous state-of-the-arts. Our method can be trained end-to-end in one stage and introduces no additional parameters, which is expected to serve as a general framework and help ease future research in domain adaptive semantic segmentation. Code is available at https://github.com/kgl-prml/Pixel- Level-Cycle-Association.
1 Introduction
Domain-adaptive segmentation seeks target-domain pixel predictions using only source annotations, but existing transfer approaches often miss cross-domain pixel relationships. This paper introduces cyclic pixel association with contrastive strengthening and reports gains on two benchmarks.
1 Introduction
- Prior adversarial methods often address domain discrepancy globally or treat pixels independently, overlooking pixel-wise relationships and offering less discriminative features.Self-training can improve segmentation, but may rely on good initialization, be difficult to tune, and yield high variance.Source: §1 · pp. 1–2
- The authors frame explicit domain-discrepancy minimization as orthogonal to self-training and report its effectiveness on two representative benchmarks.They summarize the contribution as a new pixel-similarity perspective with cross-domain cyclic association.Source: §1 · p. 2
- The method cyclically associates source and target pixels by feature similarity, then contrastively strengthens pairs whose source endpoints share a category.Spatial feature aggregation uses internal target-pixel similarities to propagate gradients from associated pixels to unassociated ones.Source: §1 · p. 2
- On GTAV → Cityscapes and SYNTHIA → Cityscapes, the method improves mIoU by more than 10% over source-only training and performs favorably against prior state-of-the-art methods.The reported evaluation uses mean Intersection-over-Union (mIoU).Source: §1 · p. 2
2 Related Work
Prior domain-adaptation work addresses domain gaps through image-level translation or representation-level adaptation, while adaptive segmentation also uses adversarial training and self-training. This paper focuses on cross-domain pixel-wise relationships and describes its one-stage training as orthogonal to self-training and other approaches.
2 Related Work
- Domain-adaptation methods mitigate gaps through image-level translation or representation-level approaches, including adversarial training and Maximum Mean Discrepancy.The review also notes methods combining image-level and representation-level adversarial training.Source: §2 · p. 2
- Adversarial adaptive-segmentation methods use style transfer, make features or predictions indistinguishable across domains, or combine both strategies.Examples include DISE's separation of domain-invariant structure and domain-specific texture, and adversarial training on multi-level predictions.Source: §2 · p. 2
- Self-training methods iteratively improve segmentation, but the review identifies dependence on good initialization and sensitivity to noise as concerns.The passage also notes uncertainty about how much pretraining is needed and a risk of collapse.Source: §2 · p. 2
- The paper focuses on cross-domain pixel-wise relationships overlooked by most prior work and presents its one-stage method as orthogonal to self-training and style-transfer augmentation.Figure 1 illustrates the cycle association: a source pixel selects a target pixel by similarity, which then selects a source pixel.Source: §2 · pp. 2–3, Figure 1 · p. 3
3 Method
PLCA addresses the limits of global or image-level domain alignment by associating source and target pixels cyclically and contrastively. Spatial aggregation diffuses gradients across target pixels, while associations at feature and prediction levels address multi-granular pixel relationships.
3.1 Background
- The setting uses labeled source data and unlabeled target data with the same underlying categories, aiming to classify each target pixel into one of those classes.The stated challenge is pixel-wise target prediction using only out-of-domain source annotations.Source: §3.1 · p. 3, §3.2 · p. 3
3.2 Pixel-Level Cycle Association
- PLCA selects a most-similar target pixel for each source pixel, then a most-similar source pixel in return; cycle consistency requires the source pixels to share a semantic class.Associated pairs are strengthened contrastively rather than by directly maximizing similarity, which could introduce bias because their context or structure may differ.Source: §3.2 · pp. 3–4
3.3 Gradient Diffusion via Spatial Aggregation
- Spatial aggregation represents each target feature using features from other pixels, allowing associated seed pixels to diffuse gradients unevenly across the image.The gradient contribution depends on receiver-to-seed feature similarity; aggregation also strengthens target features and can reduce invalid associations early in adaptation.Source: §3.3 · pp. 4–5, Figure 2 · p. 5
3.4 Multi-Level Association
- Associating both backbone features and final predictions may reduce source overfitting and exploit complementary multi-granular pixel relationships to alleviate domain shift.Prediction-level association uses negative KL divergence as its similarity measure and also applies contrast normalization and spatial aggregation.Source: §3.4 · p. 5
3.5 Objective
- The adaptation objective combines source pixel-wise cross-entropy and Lovász-softmax losses with multi-level association and label-smooth regularization losses.Lovász-softmax is imposed on source data to address class imbalance; adaptive smoothing is intended to balance cross-domain association and data fitting.Source: §3.5 · pp. 5–6
4 Experiments
Across GTAV→Cityscapes and SYNTHIA→Cityscapes, PLCA outperforms prior adversarial methods and performs comparably to or better than self-training methods. Ablations and qualitative comparisons report benefits from the proposed modules and smoother predictions than source-only training.
4.1 Dataset
- The evaluation transfers from synthetic GTAV or SYNTHIA images to real Cityscapes images, using mIoU on the specified common classes.GTAV uses 19 classes; SYNTHIA results follow protocols for 13 and 16 common classes.Source: §4.1 · p. 6
4.2 Implementation Details
- Training uses DeepLab-V2 with a ResNet-101 backbone, and associations are built at 92 × 92 resolution.The implementation uses ImageNet-pretrained weights and reports task-specific training iterations of about 11K for GTAV and 3K for SYNTHIA.Source: §4.2 · p. 6
4.4 Comparisons with the state-of-the-arts
- PLCA outperforms previous state-of-the-art adversarial methods and performs comparably to or better than self-training methods across the reported benchmarks.For SYNTHIA→Cityscapes, PLCA exceeds CRST by more than 3%; for GTAV→Cityscapes, it reports 47.7% mIoU versus CRST’s 47.1%.Source: Table 2 · p. 7, Table 3 · p. 8, §4.4 · p. 8
4.3 Ablation Studies
- 47.7% and 46.8% mIoU are reported for GTAV and SYNTHIA, respectively, exceeding source-only results by +13.4% and +10.4%.The ablation study progressively adds the proposed modules to the source-only baseline.Source: §4.3 · p. 7, Table 1 · p. 7
- Adding association on predictions improves results, PLCA beats Sim-PLCA, and removing target spatial aggregation noticeably lowers mIoU.The reported findings respectively support multi-level associations, contrastively strengthening associated pairs, and spatial aggregation on target pixels.Source: §4.3 · p. 7
4.5 Visualization
- PLCA predicts masks that are smoother and contain fewer spurious areas than the source-only model’s masks.The paper reports that these visualizations show improved pixel-level prediction accuracy.Source: §4.5 · p. 8
5 Conclusion
PLCA cyclically associates cross-domain pixels to mitigate domain shift and achieves superior performance to previous state-of-the-arts. Figure 3 compares its predicted masks with source-only predictions on Cityscapes.
5 Conclusion
- PLCA achieves superior performance to previous state-of-the-arts while training end-to-end in one stage.The conclusion also proposes evaluating associations across multiple feature layers and on more recent architectures.Source: §5 · p. 8
- PLCA mitigates domain shift by cyclically associating cross-domain pixels using pixel-wise similarities.The conclusion presents this as a new perspective for domain adaptive semantic segmentation.Source: §5 · p. 8
- Future work will apply associations to multiple feature layers and evaluate the method on more recent architectures.The authors hope the exploration will provide a baseline and inspire further research.Source: §5 · p. 8
- Figure 3 compares Cityscapes masks from a source-only network and PLCA, both trained with GTAV as the source.The figure also shows the original image and ground-truth mask.Source: Figure 3 · p. 9
Broader Impact
The authors describe potential benefits of semantic segmentation without target-domain annotations, including lower annotation costs and more robust deployment in distinct scenarios. They also caution that biased training data may affect the adapted system.
Broader Impact
- The method may reduce the cost of annotating out-of-domain data for semantic segmentation applications without annotations.The authors also describe this as economically and environmentally friendly.Source: Broader Impact · p. 9
- The authors state that the adapted segmentation system may be more robust when deployed in a distinct scenario, potentially reducing accident risk in some applications.Source: Broader Impact · p. 9
- Biased collected training data may affect the adapted system despite the method not leveraging data bias.Source: Broader Impact · p. 9