Source-linked AI summary
CDTrans: Cross-domain Transformer for Unsupervised Domain Adaptation
Tongkun Xu, Weihua Chen, Pichao Wang, Fan Wang, Hao Li, Rong Jin
TL;DR
UDA must transfer knowledge across domains despite noisy target pseudo labels that hinder category-level alignment. CDTrans combines center-aware pseudo-label generation with a weight-sharing triple-branch Transformer using self- and cross-attention. It achieves state-of-the-art performance across four UDA datasets, with nearly 21% improvement over the baseline on VisDA-2017.
Problem
Category-level UDA requires target pseudo labels, but these labels are often too noisy for accurate domain alignment.
Method
CDTrans generates target pseudo labels with two-way center-aware labeling and uses a weight-sharing triple-branch Transformer for source, target, and cross-domain alignment.
Results
CDTrans achieves state-of-the-art results on four popular UDA datasets and outperforms previous methods by a large margin.
Takeaways & Limitations
Cross-attention provides a robust feature-alignment mechanism for UDA despite noise in pseudo-labeled input pairs.
Takeaways & Limitations
On Office-Home and Office-31, TVT performs better, using a ViT backbone pretrained on ImageNet21K rather than CDTrans's ImageNet1K pretraining.
Abstract
from arXiv · showhide
Unsupervised domain adaptation (UDA) aims to transfer knowledge learned from a labeled source domain to a different unlabeled target domain. Most existing UDA methods focus on learning domain-invariant feature representation, either from the domain level or category level, using convolution neural networks (CNNs)-based frameworks. One fundamental problem for the category level based UDA is the production of pseudo labels for samples in target domain, which are usually too noisy for accurate domain alignment, inevitably compromising the UDA performance. With the success of Transformer in various tasks, we find that the cross-attention in Transformer is robust to the noisy input pairs for better feature alignment, thus in this paper Transformer is adopted for the challenging UDA task. Specifically, to generate accurate input pairs, we design a two-way center-aware labeling algorithm to produce pseudo labels for target samples. Along with the pseudo labels, a weight-sharing triple-branch transformer framework is proposed to apply self-attention and cross-attention for source/target feature learning and source-target domain alignment, respectively. Such design explicitly enforces the framework to learn discriminative domain-specific and domain-invariant representations simultaneously. The proposed method is dubbed CDTrans (cross-domain transformer), and it provides one of the first attempts to solve UDA tasks with a pure transformer solution. Experiments show that our proposed method achieves the best performance on public UDA datasets, e.g. VisDA-2017 and DomainNet. Code and models are available at https://github.com/CDTrans/CDTrans.
1 INTRODUCTION
The paper addresses UDA's noisy pseudo-label problem with Transformer cross-attention and introduces CDTrans, a triple-branch framework with center-aware labeling. It reports strong performance on public benchmarks.
- UDA transfers knowledge from labeled source data to an unlabeled target domain, but category-level alignment depends on noisy target pseudo labels.
- Cross-attention is used because it can align different distributions and remain robust to noise in pseudo-labeled input pairs.
- CDTrans uses three weight-sharing Transformer branches for source learning, target learning, and source-target feature alignment.
- The two-way center-aware labeling method produces higher-quality target pseudo labels by weighting cross-domain similarities to weaken noise.
- CDTrans achieves the best performance with a large margin on the VisDA-2017 and DomainNet datasets.
2 RELATED WORK
Related UDA work includes domain-level and category-level alignment, while pseudo labeling supports conditional alignment and regularization. Transformer research provides the broader architectural context.
- Pure Transformer models, including ViT and DeiT, have become increasingly popular for computer-vision tasks.
- UDA methods primarily align domains globally or align category distributions between source and target data.
- Domain-level methods reduce source-target distribution divergence using measures such as Maximum Mean Discrepancy and adversarial learning.
- Category-level alignment performs finer-grained matching within the same label space by pushing target samples toward source category distributions.
- Pseudo labeling assigns labels to unlabeled data from predicted probabilities and has been used for conditional distribution alignment and regularization in domain adaptation.
3 THE PROPOSED METHOD
CDTrans combines cross-attention, two-way center-aware labeling, and a weight-sharing triple-branch Transformer to align source and target domains despite noisy pairs.
- 3.1.1 PRELIMINARY: Cross-attention aggregates target patches according to their similarity with source queries, producing aligned features from paired images.More similar target patches receive larger weights, while dissimilar patches contribute less.
- 3.1.2 ROBUSTNESS TO NOISE: Cross-attention reduces the effect of dissimilar patches in false positive pairs, supporting feature alignment under pair-label noise.The mechanism assigns low weights to dissimilar patches, weakening their negative effects to some extent.
- 3.2.1 TWO-WAY LABELING: The two-way labeling method selects nearest cross-domain pairs in both directions, so the final pair set includes all source and target data.Source-to-target pairs alone involve only part of the target data; the reverse direction removes this training bias.
- 3.2.2 CENTER-AWARE FILTERING: Center-aware filtering uses source-model probability distributions and target-domain centers to refine pseudo labels, retaining only label-consistent pairs.The method performs one refinement round and discards pairs whose target pseudo label disagrees with the source label.
- 3.3 CDTRANS: CROSS-DOMAIN TRANSFORMER: CDTrans contains weight-sharing source, target, and source-target Transformer branches using self-attention for domain-specific features and cross-attention for aligned features.The source and target branches use cross-entropy, while the source-target branch guides the target branch through distillation.
4 EXPERIMENTS
CDTrans is evaluated on four UDA benchmarks against state-of-the-art methods and backbone-controlled baselines. It achieves strong gains across datasets, while ablations support the value of center-aware labeling, cross-attention, and distillation.
- Datasets and implementation: CDTrans is evaluated on VisDA-2017, Office-Home, Office-31, and DomainNet using DeiT-small and DeiT-base backbones.The experiments use 224×224 input images and compare against multiple UDA baselines and state-of-the-art methods.
- Comparison to SoTA: CDTrans outperforms the baseline by a large margin on all four datasets, including nearly 21% on VisDA-2017.The method also achieves the best reported performance on VisDA-2017 compared with the listed state-of-the-art methods.
- Comparison to SoTA: On VisDA-2017, performance for the hard “person” category rises from 10.3% to 88.6% despite noisier pseudo labels.The authors interpret this result as evidence that CDTrans is robust to labeling noise to some extent.
- Comparison to SoTA: TVT performs better than CDTrans on Office-Home and Office-31, using ViT pretrained on ImageNet21K rather than ImageNet1K.This comparison involves different pretraining data, which the authors identify as a fairness-related distinction.
- Different pseudo labeling: The two-way center-aware labeling method improves precision and final performance over two-way labeling without center awareness, while slightly reducing recall.It also performs better than other pseudo-label generation methods and approaches the ground-truth-pair upper bound.
- Different losses: Ablations show that target-branch classification, source-target cross-attention, and distillation loss each contribute to improved VisDA-2017 performance.Using distillation rather than classification on the source-target branch further improves the results.
5 CONCLUSION
The paper introduces CDTrans, a pure transformer-based UDA network with three branches and a two-way center-aware labeling method for high-quality pseudo labels. Training with these labels yields a robust solution and state-of-the-art results on four popular UDA datasets.
- CDTrans is a pure transformer-based three-branch network for unsupervised domain adaptation.
- The method generates high-quality target pseudo labels using a two-way center-aware labeling method.
- Training CDTrans with the generated pseudo labels yields a robust UDA solution.
- CDTrans achieves state-of-the-art results on four popular UDA datasets, outperforming previous methods by a large margin.