Source-linked AI summary
Category Contrast for Unsupervised Domain Adaptation in Visual Tasks
Jiaxing Huang, Dayan Guan, Aoran Xiao, Shijian Lu, Ling Shao
TL;DR
UDA must learn useful target-domain representations from unlabelled data while overcoming the semantic limitations of ordinary instance contrast. CaCo builds a category-aware dictionary mixing source and target samples with pseudo labels, and experiments show superior performance across visual UDA tasks while complementing existing methods.
Problem
Existing instance contrast methods provide limited category semantics and can create category collisions, which is suboptimal for semantic visual UDA tasks.
Method
CaCo uses a category-aware, domain-mixed dictionary with source labels and target pseudo labels, applying category contrast to pull same-category samples together and push different-category samples apart.
Results
CaCo achieves superior performance across segmentation, classification, and detection UDA tasks and consistently improves existing UDA methods when incorporated.
Takeaways & Limitations
CaCo provides a generic category contrastive loss that works across visual UDA tasks and is complementary to existing UDA methods.
Takeaways & Limitations
Ordinary instance contrast methods are limited by little category prior information and category collisions in large category-agnostic dictionaries.
Abstract
from arXiv · showhide
Instance contrast for unsupervised representation learning has achieved great success in recent years. In this work, we explore the idea of instance contrastive learning in unsupervised domain adaptation (UDA) and propose a novel Category Contrast technique (CaCo) that introduces semantic priors on top of instance discrimination for visual UDA tasks. By considering instance contrastive learning as a dictionary look-up operation, we construct a semantics-aware dictionary with samples from both source and target domains where each target sample is assigned a (pseudo) category label based on the category priors of source samples. This allows category contrastive learning (between target queries and the category-level dictionary) for category-discriminative yet domain-invariant feature representations: samples of the same category (from either source or target domain) are pulled closer while those of different categories are pushed apart simultaneously. Extensive UDA experiments in multiple visual tasks (e.g., segmentation, classification and detection) show that CaCo achieves superior performance as compared with state-of-the-art methods. The experiments also demonstrate that CaCo is complementary to existing UDA methods and generalizable to other learning setups such as unsupervised model adaptation, open-/partial-set adaptation etc.
1. Introduction
The paper reframes UDA through instance contrast, proposing a category-aware, domain-mixed dictionary and category contrastive loss for discriminative, domain-invariant representations. CaCo balances categories, aligns source and target domains, and consistently improves UDA performance across visual tasks.
- UDA addresses poor generalization to new domains by exploiting unlabelled target-domain data.
- Instance contrastive learning can be viewed as matching an encoded query with positive and negative dictionary keys.
- CaCo builds a category-aware, domain-mixed dictionary with evenly sampled source and target keys, assigning target keys predicted category labels.
- Category contrast pulls same-category samples together and pushes different-category samples apart, producing category-discriminative yet domain-invariant representations.
- Category-balanced keys mitigate imbalance by distributing contrastive comparisons evenly across categories.
- CaCo achieves superior UDA performance consistently compared with state-of-the-art methods across visual tasks and complements existing UDA methods.
2. Related Works
The related-work discussion connects UDA's existing unsupervised losses with instance contrastive learning, then identifies semantic limitations of category-agnostic contrastive dictionaries. CaCo addresses these limitations by introducing category priors through a categorical domain-mixed dictionary.
- UDA methods commonly use adversarial, image-translation, or self-training losses to learn from unlabelled target data.
- CaCo introduces a generic category contrastive loss and categorical domain-mixed dictionary to address these limitations in UDA.
- Instance contrastive learning trains embeddings by pulling positive samples toward an anchor and pushing negative samples away, often through dictionary-based designs.
- Existing instance contrast methods lack category priors, limiting high-level semantic information needed by segmentation, detection, and classification.
- Category-agnostic dictionaries can create category collisions by pushing semantically similar negative pairs apart, impairing semantic-level discrimination.
- Related contrastive approaches include semantic-distribution contrast with category centroids and pixel-level contrast using memory banks for segmentation.
3. Method
CaCo adapts instance contrastive learning to UDA by constructing a category-aware, domain-mixed dictionary and applying category contrastive learning to target queries. This design learns category-discriminative, domain-invariant representations while supporting theoretical interpretations through maximum likelihood and EM.
- Category Contrast: CaCo builds a category-aware, domain-mixed dictionary from source and target keys for category contrastive learning on unlabeled target data.Source and target samples are encoded by separate query and momentum key encoders; target queries contrast against the resulting dictionary.
- Representation Properties: The category-aware and domain-mixed dictionary learns category-discriminative representations while aligning representations within and across source and target domains.Category awareness supports predefined-category recognition tasks, while domain mixing contrasts target queries with keys from both domains.
- Categorical Dictionary: The classifier assigns predicted category labels to encoded keys, which are grouped into categorical dictionary entries and maintained with a dynamic M-size queue.Keys are sampled from either domain, classified into categories, and queued by category as older entries are replaced.
- Category Contrastive Loss: The category contrastive loss pulls target queries toward same-category keys and separates them from different-category keys through a C-way softmax-based objective.The loss uses category matching, temperature parameters, and dot-product similarities; each categorical group contributes one positive key for the query.
- Relation to Instance Contrast: CaCo generalizes instance contrast by treating InfoNCE as a special case of category contrast, with each instance serving as a category and a fixed temperature.CaCo instead permits different temperatures for keys whose predicted labels have different uncertainties, scaled by prediction entropy.
- Theoretical Insights: Category contrastive learning is modeled as maximum likelihood optimized by EM and is convergent under certain conditions.The paper states both propositions and places their proofs in the appendix.
4. Experiments
Experiments evaluate CaCo across semantic segmentation, object detection, and image classification UDA tasks, along with ablations, complementarity, and generalization studies. Across these evaluations, CaCo is consistently competitive, improves existing UDA methods, and benefits from combining source- and target-domain dictionary keys.
- Experimental setup: Experiments cover UDA for semantic segmentation, object detection, and image classification across multiple domain-shift benchmarks.The evaluated tasks include GTA5→Cityscapes, SYNTHIA→Cityscapes, Cityscapes→Foggy Cityscapes, Cityscapes→BDD100k, VisDA17, and Office-31.
- Semantic segmentation: CaCo achieves comparable segmentation performance with state-of-the-art methods on GTA5→Cityscapes and SYNTHIA→Cityscapes.The GTA5→Cityscapes evaluation also reports clear and consistent gains when CaCo is added to existing adversarial, image-translation, and self-training methods.
- Ablation studies: Ablations show that source-only and target-only category dictionaries each outperform the baseline, while combining both performs clearly best.The source- and target-domain variants provide orthogonal inter-domain and intra-domain category-contrastive signals, making their keys complementary.
- Object detection: CaCo clearly outperforms state-of-the-art methods in object detection on Cityscapes→Foggy Cityscapes and Cityscapes→BDD100k.Adding CaCo to state-of-the-art methods consistently boosts detection performance across both tasks.
- Image classification: CaCo clearly outperforms state-of-the-art methods in image classification on VisDA17 and Office-31.Adding CaCo to existing methods consistently improves classification performance on both benchmarks.
- Generalization and complementarity: CaCo performs competitively across segmentation, detection, and classification, and improves existing UDA methods consistently across these visual tasks.The experiments also report that CaCo is effective beyond standard UDA, including unsupervised model adaptation, partial-set UDA, and open-set UDA.
- Comparison with representation learning: Existing unsupervised representation-learning methods perform poorly on GTA→Cityscapes because they lack semantic priors and domain-gap modeling.CaCo is reported to work effectively for UDA by learning category-discriminative yet domain-invariant representations.
5. Conclusion
The paper presents CaCo, a category contrast technique for visual UDA that uses a semantics-aware dictionary spanning source and target domains. Experiments show superior performance across visual tasks, complementarity with existing UDA methods, and applicability to broader adaptation settings.
- CaCo introduces a generic category contrastive loss designed to work across visual UDA tasks.
- Its semantics-aware dictionary combines source and target samples, assigning each target sample a pseudo category from source category priors.
- Category contrast pulls same-category samples together and pushes different-category samples apart to learn category-discriminative, domain-invariant representations.
- Extensive experiments across segmentation, classification, and detection show CaCo achieves superior performance over highly optimized state-of-the-art methods.
- CaCo is complementary to existing UDA methods and generalizes to unsupervised model adaptation, open-set adaptation, and partial-set adaptation.