Source-linked AI summary
Unsupervised Domain Adaptation through Self-Supervision
Yu Sun, Eric Tzeng, Trevor Darrell, Alexei A. Efros
TL;DR
Unsupervised domain adaptation must transfer predictions from labeled source data to an unlabeled target domain. This paper jointly trains source classification with self-supervised tasks on both domains, avoiding adversarial minimax optimization; it reaches state-of-the-art accuracy on four of seven benchmarks. The method’s success depends on choosing self-supervised tasks that fit the application, and the paper identifies failures on SVHN and a lack of statistical guarantees for its tuning heuristic.
Problem
Unsupervised domain adaptation seeks good target-domain predictions from labeled source data when target data are unlabeled, while existing discrepancy-based approaches use difficult minimax optimization.
Method
The paper jointly trains the source classifier and auxiliary self-supervised tasks on source and target samples, selecting tasks that avoid domain-specific variation.
Results
The method achieves state-of-the-art accuracy on four out of seven benchmarks and excels at natural-scene object recognition, especially with all three tasks together.
Takeaways & Limitations
Self-supervision provides a simple, stable alternative to adversarial learning for domain alignment when auxiliary tasks fit the application.
Takeaways & Limitations
The method fails on SVHN when rotation prediction admits trivial solutions, and its hyper-parameter and early-stopping heuristic has no statistical guarantee.
Abstract
from arXiv · showhide
This paper addresses unsupervised domain adaptation, the setting where labeled training data is available on a source domain, but the goal is to have good performance on a target domain with only unlabeled data. Like much of previous work, we seek to align the learned representations of the source and target domains while preserving discriminability. The way we accomplish alignment is by learning to perform auxiliary self-supervised task(s) on both domains simultaneously. Each self-supervised task brings the two domains closer together along the direction relevant to that task. Training this jointly with the main task classifier on the source domain is shown to successfully generalize to the unlabeled target domain. The presented objective is straightforward to implement and easy to optimize. We achieve state-of-the-art results on four out of seven standard benchmarks, and competitive results on segmentation adaptation. We also demonstrate that our method composes well with another popular pixel-level adaptation method.
1 INTRODUCTION
The paper targets unsupervised domain adaptation by aligning source and target representations without target labels. It uses auxiliary self-supervised tasks trained jointly across both domains and the source classification task to obtain stable, discriminative alignment.
- Unsupervised domain adaptation uses labeled source data to predict on an unlabeled target domain sharing the source label space.
- Existing alignment methods often minimize feature-space discrepancies with MMD or domain discriminators, producing difficult minimax optimization.These objectives can cause unstable discrepancy losses and sudden divergence when the opposing forces are poorly balanced.
- The method replaces unavailable target labels with auxiliary self-supervised tasks that create labels directly from data and train on both domains.Multiple tasks align domains along multiple task-relevant directions while the original classifier is trained on labeled source data.
- The approach is empirically evaluated against the risk that separate domain-specific decision boundaries could overfit the training data.The paper relates this risk to high-complexity decision boundaries and implicit regularization from stochastic gradient descent.
- The paper proposes a simple, stable training algorithm that connects unsupervised domain adaptation with self-supervised learning without adversarial learning.The algorithm uses batches containing samples from both domains and reports state-of-the-art results on several standard benchmarks.
2 RELATED WORK
Prior unsupervised domain adaptation methods align domains through feature discrepancies, image translation, or pseudo-labeling, while self-supervision is usually used for pre-training. This paper instead jointly trains self-supervised tasks with the source task to align domains and preserve task performance.
- Unsupervised domain adaptation: Feature-space adaptation methods measure source-target discrepancy with MMD or adversarial discriminators, but both use difficult minimax optimization.Practical modifications improve benchmarks while making state-of-the-art algorithms more complicated.
- Unsupervised domain adaptation: Pixel-level methods transform source images toward target appearance and can be followed by additional representation-space adaptation.The paper demonstrates composition between its method and a popular pixel-level adaptation approach.
- Unsupervised domain adaptation: Pseudo-labeling methods estimate target labels from a source-trained model and retrain on selected confident predictions, unlike this paper’s joint training.The paper distinguishes its approach from co-training and self-ensembling methods.
- Self-supervised feature learning: Self-supervised learning uses data-derived auxiliary tasks to learn representations, commonly through pre-training followed by fine-tuning on labeled data.
- Self-supervised feature learning: This paper trains self-supervised tasks simultaneously with the main supervised task, encouraging a representation that aligns domains and supports the main task.Standard self-supervised pre-training followed by source fine-tuning gives almost no benefit over the no-adaptation baseline in this setting.
3 DESIGNING SELF-SUPERVISED TASKS FOR ADAPTATION
Self-supervised tasks should encode structural information relevant to the main task rather than domain-specific appearance factors. The paper therefore favors classification-based transformations whose labels can align domains without preserving irrelevant visual differences.
- Suitable self-supervised labels should avoid requiring information about factors of variation that adaptation aims to eliminate.
- Pixel-reconstruction tasks such as inpainting, colorization, and denoising can preserve brightness and appearance differences that separate domains.Experiments found colorization and denoising performance little better than source-only training, and sometimes worse.
- The method favors classification tasks predicting structural labels over reconstruction tasks predicting pixels.
- Rotation prediction classifies whether an image was rotated by 0°, 90°, 180°, or 270°.
- Flip prediction classifies whether an input image was randomly flipped vertically.
- Patch location prediction classifies where randomly cropped patches originated within the input image.
- When domains differ only by brightness scaling, rotation, flip, and location tasks remain suitable because brightness is orthogonal to their predictions.
4 METHOD
The method jointly trains a source-supervised prediction task with self-supervised tasks on both domains, using shared features to induce alignment. A mean-distance heuristic supports hyper-parameter tuning and early stopping without minimax optimization.
- 4 METHOD: The optimization combines one supervised source-task loss with K self-supervised losses trained on source and target data.Each task has a separate head, while all heads share the feature extractor.
- 4 METHOD: Unlike adversarial discrepancy methods, the objective does not pose a minimax problem; all task losses instead work toward inducing alignment.The method uses self-supervision on both domains rather than target-only auxiliary training.
- 4 METHOD: The shared encoder supplies high-level features to task-specific heads, which produce predictions in their respective label spaces.The heads are low-capacity linear layers, optionally followed by softmax or sigmoid outputs.
- 4 METHOD: Self-supervised tasks transform inputs and generate labels, allowing the same task to be trained on labeled-source-derived and unlabeled-target-derived samples.The original labels are discarded when generating self-supervised samples.
- 4 METHOD: At test time, the self-supervised heads are discarded, and predictions use the main head applied to the shared representation.The paper reports smooth convergence of task errors and centroid distance in two example adaptations.
- 4.1 A HEURISTIC FOR HYPER-PARAMETER TUNING AND EARLY STOPPING: The mean distance between source and target representations is used with source validation error for hyper-parameter tuning and early stopping.The combined normalized measurement selects the epoch minimizing the sum of the two quantities.
5 EXPERIMENTS
Experiments show that self-supervised domain alignment achieves strong object-recognition results and improves semantic-segmentation adaptation, but task suitability determines success.
- Object recognition: The method achieves state-of-the-art accuracy on four of seven standard benchmarks.
- Object recognition: The method fails on SVHN benchmarks because rotation prediction can exploit adjacent digits rather than semantic content.
- Object recognition: Adding location and flip prediction improves STL-10→CIFAR-10 by 9% beyond the source-only improvement.
- Semantic segmentation: Despite classification-oriented self-supervision, the method significantly improves GTA5→Cityscapes segmentation over the source-only baseline.
- Semantic segmentation: Combining the method with CyCADA further improves segmentation performance over CyCADA alone.
6 DISCUSSION
The discussion positions self-supervision as an alternative to adversarial adaptation and distinguishes the paper from related robustness, domain-generalization, and few-shot settings.
- The paper proposes studying self-supervised tasks as an alternative to adversarial learning for unsupervised domain adaptation.
- Its possible advantage for very small target datasets is left for future work because it was not evaluated in this paper.
- Unlike prior denoising-autoencoder adaptation, the method uses self-supervision on both source and target domains.
- Robustness and few-shot-learning studies lack the unlabeled-target component central to this setting.
- Domain generalization uses multiple labeled distributions and therefore differs from unsupervised domain adaptation despite superficial similarity.
B ADDITIONAL ALGORITHMIC DETAILS
The implementation balances source and target contributions through mixed batches and optimizes self-supervised objectives sequentially for memory efficiency.
- Balanced batches prevent the larger dataset from dominating each self-supervised loss.
- Each self-supervised batch contains half source samples and half target samples.
- The implementation adds each self-supervised task as a module with structural modifications and a loss function.
- For memory efficiency, stochastic-gradient optimization loops over self-supervised tasks instead of storing gradients for all losses simultaneously.
C ADDITIONAL DISCUSSION ON THE MEAN DISTANCE
The paper distinguishes using mean distance for model selection from using discrepancy-based minimax training, arguing that the former remains easy to optimize.
- Mean distance is used for hyper-parameter tuning and early stopping, while task losses are optimized during training.
- The method avoids minimax optimization by minimizing self-supervised task losses that are intended to reduce mean distance.
- Model parameters have many degrees of freedom and can overfit mean distance, whereas model selection uses only a few hyper-parameters and an early-stopping parameter.
D DETAILS OF THE SIX DATASETS USED FOR OBJECT RECOGNITION
The paper uses six standard object-recognition datasets, covering handwritten digits, street-view digits, and natural objects. CIFAR-10 and STL-10 are reduced to nine shared classes for adaptation.
- MNIST contains 60,000 training and 10,000 test grayscale handwritten-digit images.
- MNIST-M blends MNIST digits with random BSDS500 color patches and retains MNIST’s training and test set sizes.
- SVHN contains cropped Google Street View house numbers, with 73,257 training, 26,032 test, and 531,131 additional easier training samples.
- USPS provides 7,291 training and 2,007 test grayscale handwritten-digit images that are only slightly different from MNIST.
- CIFAR-10 and STL-10 contain centered natural-object images and are made nine-class datasets by deleting their mismatched category.
E IMPLEMENTATION DETAILS ON THE OBJECT RECOGNITION BENCHMARKS
Object-recognition experiments use a 26-layer pre-activation ResNet and standard SGD training settings. The reported source-only baseline is weaker than a close competitor, indicating that gains come through adaptation.
- The test-time model is a 26-layer pre-activation ResNet, with h0 as the prediction layer and φ as the preceding representation network.
- The source-only results are worse than those of Shu et al. (2018), supporting the claim that improvements arise through adaptation rather than the base architecture.
- Self-supervised heads are linear layers attached to the end of φ, without other modifications to the standard ResNet.
- Training uses SGD with weight decay 5e-4, momentum 0.9, batch size 128, initial learning rate 0.1, and two tenfold learning-rate drops.
F IMPLEMENTATION DETAILS ON GTA5 →CITYSCAPES
For GTA5 → Cityscapes segmentation adaptation, experiments initialize from ImageNet-pretrained DeepLab-v3 and use regression-based location prediction as the self-supervised task.
- The segmentation model is initialized from an ImageNet-pretrained DeepLab-v3 architecture.
- Each self-supervised head applies global average pooling to the pre-logit layer followed by a single linear layer.
- Location prediction uses a continuous regression formulation trained with square loss to regress image coordinates.