Source-linked AI summary
Asymmetric Tri-training for Unsupervised Domain Adaptation
Kuniaki Saito, Yoshitaka Ushiku, Tatsuya Harada
TL;DR
Unsupervised domain adaptation must transfer a classifier from labeled source data to an unlabeled target domain, but distribution matching alone may not produce target-discriminative representations. The paper introduces asymmetric tri-training, where two classifiers pseudo-label target samples and a third learns from them; it reports state-of-the-art results, including a more-than-10% improvement on MNIST→SVHN.
Problem
Unsupervised domain adaptation seeks accurate target-domain classification from labeled source samples and unlabeled target samples, while distribution matching alone may not ensure target accuracy.
Method
Asymmetric tri-training uses three networks: two assign pseudo-labels to target samples, and one learns target-discriminative representations from those pseudo-labeled samples.
Results
The method achieves state-of-the-art performance across evaluated visual-recognition and sentiment-analysis tasks, outperforming other methods by more than 10% on MNIST→SVHN.
Takeaways & Limitations
Pseudo-labeling target samples enables the method to train target-specific representations within a simply implemented asymmetric tri-training framework.
Takeaways & Limitations
The evaluation excludes the Office dataset and focuses on settings with one available source domain and one target domain.
Abstract
from arXiv · showhide
Deep-layered models trained on a large number of labeled samples boost the accuracy of many tasks. It is important to apply such models to different domains because collecting many labeled samples in various domains is expensive. In unsupervised domain adaptation, one needs to train a classifier that works well on a target domain when provided with labeled source samples and unlabeled target samples. Although many methods aim to match the distributions of source and target samples, simply matching the distribution cannot ensure accuracy on the target domain. To learn discriminative representations for the target domain, we assume that artificially labeling target samples can result in a good representation. Tri-training leverages three classifiers equally to give pseudo-labels to unlabeled samples, but the method does not assume labeling samples generated from a different domain.In this paper, we propose an asymmetric tri-training method for unsupervised domain adaptation, where we assign pseudo-labels to unlabeled samples and train neural networks as if they are true labels. In our work, we use three networks asymmetrically. By asymmetric, we mean that two networks are used to label unlabeled target samples and one network is trained by the samples to obtain target-discriminative representations. We evaluate our method on digit recognition and sentiment analysis datasets. Our proposed method achieves state-of-the-art performance on the benchmark digit recognition datasets of domain adaptation.
1. Introduction
Unsupervised domain adaptation seeks target-domain classifiers from labeled source and unlabeled target samples, but distribution matching alone may not yield discriminative target performance. The paper proposes asymmetric tri-training, using pseudo-labels and three classifiers with different roles, and reports strong benchmark results.
- Different source and test distributions can make neural networks difficult to apply across domains, while collecting extensive labels remains difficult.
- Unsupervised domain adaptation trains a target-domain classifier using labeled source samples and unlabeled target samples.
- Matching source and target feature distributions alone may fail when the resulting representations are nondiscriminative for target classification.
- The method assigns pseudo-labels to unlabeled target samples and trains target-specific networks as if those labels were true.
- Three classifiers are asymmetric: two label target samples, while the remaining network learns from pseudo-labeled target samples.
- Across digit, traffic-sign, and sentiment tasks, the method demonstrates state-of-the-art performance in nearly all experiments, exceeding other methods by more than 10% on MNIST→SVHN.
2. Related Work
Prior adaptation methods commonly minimize source loss and domain divergence, while self-training and tri-training use pseudo-labels without specifically addressing cross-domain labeling. The paper positions asymmetric tri-training as a domain-adaptation method that gives three classifiers different roles.
- Many domain-adaptation methods minimize source classification loss together with divergence between source and target domains.
- Similarity-based transductive methods provide labels for unlabeled samples, whereas this method backpropagates target category loss from pseudo-labels.
- Self-training retrains a classifier using its own high-confidence predictions, while co-training and tri-training use multiple classifiers to label unlabeled samples.
- The proposed network uses a shared feature extractor, two source-and-target classifiers, and one target-specific classifier trained only on pseudo-labeled target samples.
- Standard tri-training does not assume that unlabeled samples come from a different distribution than labeled samples.
3. Method
The method constructs a target-specific network from pseudo-labeled target samples while using two labeling networks and shared representations. Pseudo-labels require classifier agreement and high confidence, and training alternates between source and pseudo-labeled data.
- Network structure: Two labeling networks assign pseudo-labels to target samples, while a third target-specific network trains only on those pseudo-labeled samples.The shared network F receives gradients from all three networks.
- Network structure: The labeling networks learn from source and pseudo-labeled target samples, whereas the target-specific network learns only from pseudo-labeled target samples.
- Multiview features: A weight constraint makes the two labeling networks use different features, encouraging predictions from different viewpoints.The constraint is applied to the fully connected layers’ weights W1 and W2.
- Learning procedure: Pseudo-labels are retained only when the two classifiers agree and at least one classifier’s maximum predicted probability exceeds 0.9 or 0.95.Resampling candidates at each step is used to prevent overfitting to pseudo-labels.
- Learning procedure: Training alternates between updating the shared and labeling networks with source plus pseudo-labeled data and updating the shared and target-specific networks with pseudo-labeled data.The pseudo-labeled candidate pool begins at 5,000 and is gradually increased to a maximum of 40,000.
- Learning procedure: Using source samples alongside pseudo-labeled samples is intended to preserve accuracy while target-specific training supplies target-discriminative information.The paper describes this as a cycle that gradually enhances target-domain accuracy.
4. Analysis
The analysis emphasizes that target error depends not only on source error and domain divergence but also on the shared error of an ideal joint hypothesis. The method uses pseudo-labels to approximately evaluate and minimize this otherwise unmeasurable term while also addressing domain discrepancy.
- Existing theory: The domain divergence term measures disagreement between hypotheses across source and target domains.The paper identifies H∆H-distance as a measure of adaptability between domains.
- Existing theory: Target-domain expected error is upper bounded by source error, domain divergence, and the error of an ideal joint hypothesis.
- Existing theory: The ideal-joint-hypothesis error is important when representations are learned by deep models because it is determined by the output of the final fully connected layer.
- Connection to the method: The analysis models pseudo-labeled target samples as carrying false labels at ratio ρ and defines their shared error with source samples as C′.
- Connection to the method: Because labeled target samples are unavailable, the method uses pseudo-labels to approximately evaluate and minimize the ideal-joint-hypothesis error.The analysis states that the approach considers both the domain-divergence and shared-error terms.
5. Experiment and Evaluation
The experiments evaluate asymmetric tri-training across visual and sentiment domain-adaptation settings, including ablations and representation analyses. The method generally outperforms competing approaches, while gradient-flow and domain-discrepancy analyses clarify when its target-specific network helps.
- Experimental setup: The evaluation covers digit, traffic-sign, and Amazon Reviews domain-adaptation tasks, comparing the method with five established approaches.The visual experiments use digits and traffic signs; Amazon Reviews contains 12 adaptation scenarios.
- Visual-domain results: In every visual-domain setting, the proposed method outperforms other methods by a large margin.Table 1 reports results for digits and traffic-sign datasets, while source-only results are included for comparison.
- Representation analysis: About 7%: the method outperforms the other existing method in another reported visual-domain comparison, with adapted target representations becoming more dispersed.The visualization compares source-only and adapted embeddings using target and source samples.
- Visual-domain results: More than 10%: the method exceeds other approaches in MNIST→SVHN and achieves state-of-the-art performance in both evaluated adaptation scenarios.The authors describe MNIST→SVHN as especially challenging because MNIST and SVHN have distinct appearances.
- Ablation and analysis: Backpropagating all gradients gives clear performance improvements, whereas using only Ft can worsen performance when SYN SIGNS and GTSRB are similar and pseudo-labels are noisy.For MNIST→MNIST-M, gradients from Ft help construct target-discriminative representations; for MNIST→SVHN, all gradients yield the best performance.
- Sentiment results: Nine of twelve: on Amazon Reviews, the method performs better than VFAE and DANN in nine settings.The reported metric is accuracy across 12 domain-adaptation scenarios.
6. Conclusion
The paper proposes a simply implemented asymmetric tri-training method that uses pseudo-labeled target samples to learn discriminative representations. It evaluates the method on visual recognition and sentiment analysis, outperforming other methods, including by more than 10% on MNIST→SVHN.
- Asymmetric tri-training uses two networks to assign pseudo-labels to unlabeled target samples and a third network to learn from them.The method aims to learn discriminative target representations from pseudo-labeled samples.
- The method is simply implemented for unsupervised domain adaptation.
- The method outperformed other methods on both visual recognition and sentiment analysis domain-adaptation tasks.
- More than 10% improvement was achieved over other methods in the MNIST→SVHN adaptation task.
Proof of Theorem
The proof defines the ideal joint hypothesis and bounds its combined source-target error using pseudo-labeled target data. It then relates this bound to pseudo-label noise through the triangle inequality.
- The ideal joint hypothesis h* minimizes the combined source and target error, whose corresponding error is C = R_S(h*) + R_T(h*).
- The proof models pseudo-labeled target data as containing false labels at ratio ρ and denotes the minimum shared source error as C′.
- Using the 0-1 loss, the proof compares expected errors computed from true-labeled and pseudo-labeled target sets.
- The triangle inequality bounds combined source-target error by source error, pseudo-labeled-target error, and the difference between pseudo-labeled and true target errors.
- The resulting bound establishes the theorem’s main inequality.
CNN Architectures and training detail
The experiments use four architecture types based on prior work, with topology-specific batch normalization and dropout choices. Training uses validation-selected hyperparameters, and extending training to 100 steps reaches about 97% accuracy.
- Four architecture types are used, based on Ganin and Lempitsky, with network topologies shown in Figures 6–8.
- The learning rate is 0.05 for SVHN↔MNIST and 0.01 in the other scenarios.
- Training batch sizes are 128 for F_t and F, and 64 for F_1, F_2, and F.
- Dropout and batch normalization are assigned differently across adaptation scenarios, including no dropout for MNIST→SVHN.
- About 97% accuracy is achieved when the number of steps increases to 100.
Supplementary experiments on MNIST→MNIST-M
The supplementary experiment increases training steps to 100 and observes about 97% accuracy, while the architectures use scenario-specific batch normalization and dropout configurations.
- About 97% accuracy is reached when training is extended to 100 steps.The main experiments use 30 steps, but further improvement is expected from additional steps.
- The MNIST→MNIST-M architecture adds batch normalization in the last convolutional layer and fully connected layers F1 and F2.
- The SVHN-related architectures place batch normalization in different convolutional or fully connected layers depending on the adaptation direction.
- The Synthetic Signs→GTSRB architecture adds batch normalization after the last convolutional layer and uses dropout.