Source-linked AI summary
Return of Frustratingly Easy Domain Adaptation
Baochen Sun, Jiashi Feng, Kate Saenko
TL;DR
Domain shift degrades classifiers when training and test distributions differ, and unlabeled target domains make supervised adaptation unsuitable. CORAL performs unsupervised adaptation by aligning source and target second-order statistics, and it performs well across standard object-recognition and sentiment-prediction tasks while remaining simple and efficient.
Problem
Domain shift harms conventional classifiers, while practical target domains often lack the labels required by supervised adaptation methods.
Method
CORAL aligns source and target feature distributions by matching their second-order statistics, using covariance-based whitening and re-coloring of source features.
Results
CORAL outperforms many existing methods across object-recognition and sentiment-prediction tasks, including methods using deep CNN adaptation.
Takeaways & Limitations
A covariance-based transformation provides a simple and efficient unsupervised adaptation method that works with both shallow and deep feature representations.
Abstract
from arXiv · showhide
Unlike human learning, machine learning often fails to handle changes between training (source) and test (target) input distributions. Such domain shifts, common in practical scenarios, severely damage the performance of conventional machine learning methods. Supervised domain adaptation methods have been proposed for the case when the target data have labels, including some that perform very well despite being "frustratingly easy" to implement. However, in practice, the target domain is often unlabeled, requiring unsupervised adaptation. We propose a simple, effective, and efficient method for unsupervised domain adaptation called CORrelation ALignment (CORAL). CORAL minimizes domain shift by aligning the second-order statistics of source and target distributions, without requiring any target labels. Even though it is extraordinarily simple--it can be implemented in four lines of Matlab code--CORAL performs remarkably well in extensive evaluations on standard benchmark datasets.
1 Introduction
Domain shift causes classifiers trained on one distribution to perform poorly on another, while common supervised adaptation methods require labeled target examples. CORAL addresses unlabeled-target adaptation by aligning source and target covariances with a simple, efficient transformation and performs well across object-recognition and sentiment tasks.
- Domain shift can substantially reduce performance when classifiers trained on one domain are applied directly to another.
- Supervised adaptation methods often require labeled target examples, limiting their use when the target domain is unlabeled.
- CORAL aligns source and target feature distributions by matching second-order statistics through covariance-based whitening and re-coloring.It computes covariance statistics in both domains, transforms source features, and then trains a classifier on the transformed source data.
- CORAL is evaluated on object recognition and sentiment prediction, where it outperforms many existing methods.
- CORAL works with both bag-of-words and deep CNN features while requiring only source and target covariance computation rather than deep-network retraining.
2 Related Work
Prior domain adaptation work includes supervised feature replication, reweighting, dictionary learning, subspace methods, and adaptive deep networks. These approaches vary in their assumptions and computational requirements, motivating a simpler unsupervised alternative.
- Supervised adaptation methods regularize target learning, reduce source-target distance, or transform features, but require labeled target examples.
- Unsupervised approaches include loss reweighting, dictionary learning, and subspace-based representations that reduce differences between source and target domains.
- Adaptive deep methods optimize domain-invariant representations with additional architectures or loss layers, increasing training cost and complexity.
3 Correlation Alignment for Unsupervised Domain Adaptation
CORAL performs unsupervised domain adaptation by transforming source features so their covariance matches the target covariance. The method uses whitening and re-coloring, remains efficient and classifier-agnostic, and avoids subspace projection while accounting for covariance eigenvectors and eigenvalues.
- Core idea: CORAL aligns source and target feature distributions by matching their second-order statistics, specifically covariance.It is unsupervised and requires no target labels.
- Formulation: CORAL applies a linear transformation that minimizes the Frobenius-norm distance between transformed-source and target covariance matrices.The transformed covariance is written as A^T C_S A.
- Formulation: The analytical solution handles low-rank covariance matrices by using r = min(r_C_S, r_C_T) and the target covariance's leading singular components.The rank constraint arises because linear transformation cannot increase source covariance rank.
- Implementation: In practice, classical whitening and coloring are faster and more stable than the analytical solution, with similar performance across λ; the paper sets λ to 1.A reported transformation takes less than one minute for source data in R^795×4096 and target data in R^2817×4096 on a regular laptop.
- Implementation: After transformation, a classifier trained on adjusted source features can be applied directly to target features, and CORAL works with any base classifier.For linear classifiers, the equivalent parameter transformation can improve efficiency when target data are numerous and high-dimensional.
- Relationship to existing methods: Unlike subspace-mapping methods, CORAL aligns complete covariance matrices, accounting for both eigenvectors and eigenvalues without selecting a subspace dimensionality.The paper presents this as a more general and simpler alternative to subspace projection.
4 Experiments
The experiments evaluate CORAL for unsupervised domain adaptation in object recognition and sentiment analysis, using shallow and deep features across standard benchmark domain shifts. CORAL consistently outperforms the compared adaptation methods and no-adaptation baselines, including under larger-scale training.
- Experimental Setup: CORAL is evaluated on object recognition and sentiment analysis with both shallow and deep features, assuming unlabeled target domains.The experiments use standard benchmarks and protocols for these two tasks.
- Object Recognition with Shallow Features: The Office-Caltech10 evaluation covers 12 source-to-target domain shifts with SURF bag-of-words features and compares CORAL with five published methods and no adaptation.The compared methods are SVMA, DAM, GFK, SA, and TCA.
- Object Recognition with Deep Features: On the standard Office dataset with deep features, CORAL consistently outperforms no adaptation and is the only method with better average performance across all 6 shifts.CORAL also beats DAN and ReverseGrad in 2 of the 3 shifts they report.
- A Larger Scale Evaluation: With full source training, CORAL outperforms all baseline methods on both Office-Caltech10 and the Cross-Dataset Testbed, with larger margins on deep features.The performance gap between no adaptation and other methods becomes smaller as more source data is used.
- 4.2 Sentiment Analysis: CORAL’s margin over other published methods is larger on deep than bag-of-words features, while image improvements exceed those on text.The paper relates this pattern to stronger feature correlations in deep image representations and extreme sparsity in text features.
5 Discussion
CORAL’s advantage is especially pronounced for deep features, where stronger feature correlations may make covariance alignment more effective. The authors therefore suggest strong value for CORAL in deep learning while identifying deep text features as future work.
- 64.0 for CORAL-fc6 versus 49.1 for SA-fc6 shows a larger deep-feature margin than on bag-of-words features.The authors suggest stronger correlations in deep features may explain this difference.
- Image improvements are much larger than text improvements, possibly because bag-of-words text features are extremely sparse and less correlated.
- Deep features represent higher-level parts or objects, which should be more strongly correlated than edges.The paper gives jointly appearing arms and heads as an intuition for stronger correlations.
- The findings suggest that CORAL is extremely valuable in the era of deep learning.Applying CORAL to deep text features is identified as future work.
6 Conclusion
The paper concludes that CORAL is simple, efficient, and effective, requiring only covariance-based re-coloring of whitened source features. Extensive standard-benchmark experiments show superiority over many state-of-the-art methods across feature types and tasks.
- CORAL is simple, efficient, and effective for domain adaptation.
- Re-coloring whitened source features with the target covariance is the only computation required.
- Extensive standard-benchmark experiments demonstrate superiority over many existing state-of-the-art methods.
- CORAL applies across highly performing deep features and tasks including computer vision and natural language processing.