Unsupervised Domain Adaptation through Self-Supervision

• Unsupervised domain adaptation uses labeled source data to predict on an unlabeled target domain sharing the source label space.Source: §1 · p. 1
• Existing alignment methods often minimize feature-space discrepancies with MMD or domain discriminators, producing difficult minimax optimization.These objectives can cause unstable discrepancy losses and sudden divergence when the opposing forces are poorly balanced.Source: §1 · p. 1
• The method replaces unavailable target labels with auxiliary self-supervised tasks that create labels directly from data and train on both domains.Multiple tasks align domains along multiple task-relevant directions while the original classifier is trained on labeled source data.Source: §1 · pp. 1–2, Figure 1 · p. 2
• The approach is empirically evaluated against the risk that separate domain-specific decision boundaries could overfit the training data.The paper relates this risk to high-complexity decision boundaries and implicit regularization from stochastic gradient descent.Source: §1 · p. 2
• The paper proposes a simple, stable training algorithm that connects unsupervised domain adaptation with self-supervised learning without adversarial learning.The algorithm uses batches containing samples from both domains and reports state-of-the-art results on several standard benchmarks.Source: §1 · p. 2

Unsupervised domain adaptation

• Feature-space adaptation methods measure source-target discrepancy with MMD or adversarial discriminators, but both use difficult minimax optimization.Practical modifications improve benchmarks while making state-of-the-art algorithms more complicated.Source: §2.1 · pp. 2–3
• Pixel-level methods transform source images toward target appearance and can be followed by additional representation-space adaptation.The paper demonstrates composition between its method and a popular pixel-level adaptation approach.Source: §2.1 · p. 3
• Pseudo-labeling methods estimate target labels from a source-trained model and retrain on selected confident predictions, unlike this paper’s joint training.The paper distinguishes its approach from co-training and self-ensembling methods.Source: §2.1 · p. 3

Self-supervised feature learning

• Self-supervised learning uses data-derived auxiliary tasks to learn representations, commonly through pre-training followed by fine-tuning on labeled data.Source: §2.2 · p. 3
• This paper trains self-supervised tasks simultaneously with the main supervised task, encouraging a representation that aligns domains and supports the main task.Standard self-supervised pre-training followed by source fine-tuning gives almost no benefit over the no-adaptation baseline in this setting.Source: §2.2 · p. 3

• Suitable self-supervised labels should avoid requiring information about factors of variation that adaptation aims to eliminate.Source: §3 · p. 4
• Pixel-reconstruction tasks such as inpainting, colorization, and denoising can preserve brightness and appearance differences that separate domains.Experiments found colorization and denoising performance little better than source-only training, and sometimes worse.Source: §3 · p. 4
• The method favors classification tasks predicting structural labels over reconstruction tasks predicting pixels.Source: §3 · p. 4
• Rotation prediction classifies whether an image was rotated by 0°, 90°, 180°, or 270°.Source: §3 · p. 4
• Flip prediction classifies whether an input image was randomly flipped vertically.Source: §3 · p. 4
• Patch location prediction classifies where randomly cropped patches originated within the input image.Source: §3 · p. 4
• When domains differ only by brightness scaling, rotation, flip, and location tasks remain suitable because brightness is orthogonal to their predictions.Source: §3 · p. 4

4 METHOD

• The optimization combines one supervised source-task loss with K self-supervised losses trained on source and target data.Each task has a separate head, while all heads share the feature extractor.Source: §4 · pp. 4–5
• Unlike adversarial discrepancy methods, the objective does not pose a minimax problem; all task losses instead work toward inducing alignment.The method uses self-supervision on both domains rather than target-only auxiliary training.Source: §4 · p. 4, §4.1 · p. 6
• The shared encoder supplies high-level features to task-specific heads, which produce predictions in their respective label spaces.The heads are low-capacity linear layers, optionally followed by softmax or sigmoid outputs.Source: Figure 2 · p. 5, §4 · p. 5
• Self-supervised tasks transform inputs and generate labels, allowing the same task to be trained on labeled-source-derived and unlabeled-target-derived samples.The original labels are discarded when generating self-supervised samples.Source: §4 · p. 5
• At test time, the self-supervised heads are discarded, and predictions use the main head applied to the shared representation.The paper reports smooth convergence of task errors and centroid distance in two example adaptations.Source: §4 · p. 6, Figure 3 · p. 6

4.1 A HEURISTIC FOR HYPER-PARAMETER TUNING AND EARLY STOPPING

• The mean distance between source and target representations is used with source validation error for hyper-parameter tuning and early stopping.The combined normalized measurement selects the epoch minimizing the sum of the two quantities.Source: §4.1 · p. 6

Object recognition

• The method achieves state-of-the-art accuracy on four of seven standard benchmarks.Source: §5.1 · p. 7
• The method fails on SVHN benchmarks because rotation prediction can exploit adjacent digits rather than semantic content.Source: §5.1 · p. 7
• Adding location and flip prediction improves STL-10→CIFAR-10 by 9% beyond the source-only improvement.Source: §5.1 · pp. 7–8

Semantic segmentation

• Despite classification-oriented self-supervision, the method significantly improves GTA5→Cityscapes segmentation over the source-only baseline.Source: §5.2 · p. 8
• Combining the method with CyCADA further improves segmentation performance over CyCADA alone.Source: §5.2 · p. 8, Table 3 · p. 8

• The paper proposes studying self-supervised tasks as an alternative to adversarial learning for unsupervised domain adaptation.Source: §6 · p. 8
• Its possible advantage for very small target datasets is left for future work because it was not evaluated in this paper.Source: §6 · p. 8
• Unlike prior denoising-autoencoder adaptation, the method uses self-supervision on both source and target domains.Source: A ADDITIONAL DISCUSSION ON SPECIFIC… · p. 13
• Robustness and few-shot-learning studies lack the unlabeled-target component central to this setting.Source: A ADDITIONAL DISCUSSION ON SPECIFIC… · p. 13
• Domain generalization uses multiple labeled distributions and therefore differs from unsupervised domain adaptation despite superficial similarity.Source: A ADDITIONAL DISCUSSION ON SPECIFIC… · p. 13

• Balanced batches prevent the larger dataset from dominating each self-supervised loss.Source: B ADDITIONAL ALGORITHMIC DETAILS · pp. 13–14
• Each self-supervised batch contains half source samples and half target samples.Source: B ADDITIONAL ALGORITHMIC DETAILS · p. 14
• The implementation adds each self-supervised task as a module with structural modifications and a loss function.Source: B ADDITIONAL ALGORITHMIC DETAILS · p. 14
• For memory efficiency, stochastic-gradient optimization loops over self-supervised tasks instead of storing gradients for all losses simultaneously.Source: B ADDITIONAL ALGORITHMIC DETAILS · p. 14

• Mean distance is used for hyper-parameter tuning and early stopping, while task losses are optimized during training.Source: C ADDITIONAL DISCUSSION ON THE MEAN… · p. 14
• The method avoids minimax optimization by minimizing self-supervised task losses that are intended to reduce mean distance.Source: C ADDITIONAL DISCUSSION ON THE MEAN… · p. 14
• Model parameters have many degrees of freedom and can overfit mean distance, whereas model selection uses only a few hyper-parameters and an early-stopping parameter.Source: C ADDITIONAL DISCUSSION ON THE MEAN… · p. 14

• MNIST contains 60,000 training and 10,000 test grayscale handwritten-digit images.Source: D DETAILS OF THE SIX DATASETS USED FOR… · p. 14
• MNIST-M blends MNIST digits with random BSDS500 color patches and retains MNIST’s training and test set sizes.Source: D DETAILS OF THE SIX DATASETS USED FOR… · p. 14
• SVHN contains cropped Google Street View house numbers, with 73,257 training, 26,032 test, and 531,131 additional easier training samples.Source: D DETAILS OF THE SIX DATASETS USED FOR… · p. 15
• USPS provides 7,291 training and 2,007 test grayscale handwritten-digit images that are only slightly different from MNIST.Source: D DETAILS OF THE SIX DATASETS USED FOR… · p. 15
• CIFAR-10 and STL-10 contain centered natural-object images and are made nine-class datasets by deleting their mismatched category.Source: D DETAILS OF THE SIX DATASETS USED FOR… · p. 15

• The test-time model is a 26-layer pre-activation ResNet, with h0 as the prediction layer and φ as the preceding representation network.Source: E IMPLEMENTATION DETAILS ON THE OBJECT… · p. 15
• The source-only results are worse than those of Shu et al. (2018), supporting the claim that improvements arise through adaptation rather than the base architecture.Source: E IMPLEMENTATION DETAILS ON THE OBJECT… · p. 15
• Self-supervised heads are linear layers attached to the end of φ, without other modifications to the standard ResNet.Source: E IMPLEMENTATION DETAILS ON THE OBJECT… · p. 15
• Training uses SGD with weight decay 5e-4, momentum 0.9, batch size 128, initial learning rate 0.1, and two tenfold learning-rate drops.Source: E IMPLEMENTATION DETAILS ON THE OBJECT… · p. 15

• The segmentation model is initialized from an ImageNet-pretrained DeepLab-v3 architecture.Source: F IMPLEMENTATION DETAILS ON GTA5… · p. 15
• Each self-supervised head applies global average pooling to the pre-logit layer followed by a single linear layer.Source: F IMPLEMENTATION DETAILS ON GTA5… · p. 15
• Location prediction uses a continuous regression formulation trained with square loss to regress image coordinates.Source: F IMPLEMENTATION DETAILS ON GTA5… · p. 15