Source-linked AI summary
Towards Accurate Model Selection in Deep Unsupervised Domain Adaptation
Kaichao You, Ximei Wang, Mingsheng Long, Michael I. Jordan
TL;DR
Deep UDA lacks an accurate, standardized way to select models without labeled target data. DEV embeds adapted feature representations into validation and uses control variates to reduce variance, with experiments showing performance nearly on par with Target Risk and theoretical support for its guarantees.
Problem
Deep UDA model selection lacks an accurate standardized procedure without labeled target data, complicating fair comparison and evaluation.
Method
DEV embeds adapted feature representations into validation to estimate target risk unbiasedly with bounded variance, then applies control variates for further variance reduction.
Results
DEV performs nearly on par with Target Risk and significantly surpasses previous model-selection methods in theoretical and empirical evaluations.
Takeaways & Limitations
DEV provides an accurate, non-intrusive model-selection method for Deep UDA when labeled target data are unavailable.
Takeaways & Limitations
IWCV's variance is unbounded, while DEV's variance guarantee depends on Rényi divergence and benefits more for deep models with smaller divergence.
Abstract
from arXiv · showhide
Deep unsupervised domain adaptation (Deep UDA) methods successfully leverage rich labeled data in a source domain to boost the performance on related but unlabeled data in a target domain. However, algorithm comparison is cumbersome in Deep UDA due to the absence of accurate and standardized model selection method, posing an obstacle to further advances in the field. Existing model selection methods for Deep UDA are either highly biased, restricted, unstable, or even controversial (requiring labeled target data). To this end, we propose \textit{Deep Embedded Validation} (\textbf{DEV}), which embeds adapted feature representation into the validation procedure to obtain unbiased estimation of the target risk with bounded variance. The variance is further reduced by the technique of control variate. The efficacy of the method has been justified both theoretically and empirically.
1. Introduction
Deep UDA transfers knowledge from labeled source data to unlabeled target data, but model selection is difficult because target labels are inaccessible and existing validation strategies have important shortcomings.
- Deep UDA reduces labeling work by transferring knowledge from labeled source data to related unlabeled target data.
- Target labels are inaccessible during training and selection, while source risk can be inconsistent with target risk.
- Fixed hyperparameters may depend on another task with labeled target data and therefore fail in practical single-task adaptation settings.
- Source-risk selection lacks theoretical guarantees and becomes highly biased when the domain gap is large.
- IWCV is unbiased in principle but requires density-ratio estimation and has unbounded variance, making it unstable.
- DEV embeds adapted feature representations into validation, estimating target risk without target labels while providing bounded variance and further variance reduction through control variates.
2. Related Work
Related work covers standard validation procedures and domain-adaptation-specific methods, highlighting the challenge of selecting models under distribution shift without labeled target data.
- Validation in supervised learning: Hold-out validation separates training, validation, and test data, whereas k-fold cross-validation reuses data across k training and validation runs.
- Domain adaptation: Domain adaptation addresses violations of the standard assumption that training and test distributions are identical.
- Model selection in domain adaptation: IWCV weights validation risk using a known density ratio to obtain an unbiased target-risk estimator under covariate shift.
- Model selection in domain adaptation: Importance-weighted methods can have variance bounded by Rényi divergence, but this does not itself lower the variance or its bound.
- Model selection in domain adaptation: TrCV handles marginal and conditional shifts but requires labeled target data, creating controversy similar to selecting hyperparameters by target risk.
- Model selection in domain adaptation: DEV extends validation to adapted feature representations, preserving unbiasedness, tightening variance bounds, and reducing variance further with control variates.
3. Preliminaries
The preliminaries introduce Rényi divergence and a control-variate estimator for reducing variance while preserving unbiasedness.
- 3.1. Rényi Divergence: Rényi divergence measures the difference between distributions p and q and is non-negative, equaling zero exactly when p = q.As α approaches 1, it becomes the Kullback–Leibler divergence.
- 3.2. Control Variates: The control-variate method combines an unbiased estimator z with a related unbiased estimator t and a tunable coefficient η.The resulting estimator is constructed by adjusting z with the centered control variate t − τ.
- 3.2. Control Variates: The adjusted estimator remains unbiased because the control variate has zero expected deviation from its target parameter.This follows from linearity of expectation.
- 3.2. Control Variates: The optimal coefficient minimizes a quadratic variance expression, yielding Var[z⋆] ≤ Var[z].The reduction depends on the correlation between z and t.
4. Method
DEV performs model selection for Deep UDA by embedding adapted feature representations into importance-weighted validation, producing unbiased target-risk estimates with theoretically bounded and control-variate-reduced variance.
- 4.3. Algorithm: DEV applies to a finite set of candidate models and can also extend beyond feature-adaptation methods, including settings with generated auxiliary domains.The complete procedure computes DEV risks for candidate models and selects the model with the best estimated risk.
- 4.1. Importance-Weighted Cross-Validation: Deep UDA model selection is difficult because target labels are unavailable, while source risk can be highly biased when source and target distributions differ.The candidate model is decomposed into a feature extractor F and task function T, enabling validation in the learned feature space.
- 4.1. Importance-Weighted Cross-Validation: Under covariate shift, importance-weighted cross-validation is unbiased when the density ratio is known, but its variance is bounded by input-space Rényi divergence and is not reduced merely by continued adaptation.The support condition supp(p) ⊃ supp(q) is required; otherwise importance weights may diverge.
- 4.2. Deep Embedded Validation: DEV estimates the feature-space density ratio by discriminating source from target features, using domain labels and a constant factor derived from domain sample sizes.The discriminative component estimates the ratio through the feature-space joint distribution, and a two-layer logistic regression model is sufficient for this estimation.
- 4.4. Variance Reduction by Control Variate: The control variate uses the feature-space density ratio because its expectation is model-independent, unlike the loss expectation, which is unknown and task-specific.This choice explicitly reduces the variance of the target-risk estimate within DEV.
- 4.2. Deep Embedded Validation: DEV embeds adapted deep features into validation, yielding an unbiased target-risk estimate whose variance is bounded by feature-space Rényi divergence and further reduced with a control variate.The method is tailored to Deep UDA because adaptation generally makes feature distributions closer than input distributions, tightening the variance bound.
5. Experiments
Experiments evaluate DEV across toy regression, Deep UDA benchmarks, and partial domain adaptation, showing accurate target-risk estimation and improved model selection when adapted features and variance control are used.
- 5.1. Toy Dataset: DEV shows significantly smaller variance than IWCV while both correspond well with Target Risk across 1,000 toy experiments.Source Risk deviates from Target Risk and is not a reasonable estimator; the lower DEV variance supports the control-variate technique.
- 5.2. VisDA Dataset: DEV selects the best MCD model on VisDA, whereas IWCV degrades because pre-trained features retain a large domain gap and produce unstable importance weights.DEV instead estimates density ratios on adapted features, which have a smaller domain gap and imply bounded variance under importance-weighting theory.
- 5.3. Office-31 Dataset: DEV performs nearly as well as Target Risk on Office-31 CDAN, surpassing IWCV, Source Risk, and results reported in the original papers.Source Risk works well mainly for small-gap transfers D →W and W →D, whereas DEV remains effective for the larger-gap transfer D →A.
- 5.4. Digits Dataset: DEV improves GTA (Original) by 1.4% on Digits without altering the model, while Source Risk performs poorly on the large-gap SVHN →MNIST task.The source-risk failure is attributed to deviation from target risk under the large visual domain gap.
- 5.5. Beyond Standard Domain Adaptation: DEV continues to correspond well with Target Risk for PADA in partial domain adaptation and exceeds the original reported results without modification.The method relies on the adapted-feature support condition in which the source feature distribution contains the target feature support.
- 5.5. Beyond Standard Domain Adaptation: In partial domain adaptation, DEV generally estimates lower density ratios for source-specific classes than for shared classes, enabling unbiased target-risk estimation with controlled variance.The source-specific class 8 exception is associated with confusion between desktop computer, laptop computer, and monitor classes.
6. Conclusion
The paper concludes that Deep Embedded Validation provides accurate model selection for Deep UDA without labeled target data. It combines adapted representations with control variates and performs nearly on par with Target Risk while surpassing previous methods.
- 6. Conclusion: DEV embeds adapted deep representations into validation to estimate target risk more reliably and uses control variates to reduce variance.The method is presented as non-intrusive model selection when target labels are unavailable.
- 6. Conclusion: Theoretical analysis and extensive experiments show DEV performs nearly on par with Target Risk and significantly surpasses previous model-selection methods.The conclusion characterizes DEV as an accurate method for Deep UDA without labeled target data.