Source-linked AI summary
Unsupervised Domain Adaptation with Residual Transfer Networks
Mingsheng Long, Han Zhu, Jianmin Wang, Michael I. Jordan
TL;DR
Unsupervised domain adaptation must transfer models despite feature and classifier mismatches between labeled source and unlabeled target domains. The paper introduces Residual Transfer Networks to jointly adapt classifiers and features, and reports stronger benchmark performance than comparison methods. Its classifier adaptation assumes the source and target classifiers differ by a small perturbation function.
Problem
Unsupervised domain adaptation faces mismatches in both features and classifiers because the target domain lacks labeled data and differs distributionally from the source.
Method
Residual Transfer Networks jointly adapt classifiers and features by learning a residual between source and target classifiers, matching multi-layer feature distributions, and training with back-propagation.
Results
RTN outperforms comparison methods on most Office-31 transfer tasks, substantially improving hard transfers while matching comparable accuracy on easier transfers.
Takeaways & Limitations
The approach enables end-to-end adaptive-classifier and transferable-feature learning and can be trained with standard back-propagation in most deep-learning packages.
Takeaways & Limitations
Classifier adaptation assumes the source and target classifiers differ only by a small perturbation function.
Abstract
from arXiv · showhide
The recent success of deep neural networks relies on massive amounts of labeled data. For a target task where labeled data is unavailable, domain adaptation can transfer a learner from a different source domain. In this paper, we propose a new approach to domain adaptation in deep networks that can jointly learn adaptive classifiers and transferable features from labeled data in the source domain and unlabeled data in the target domain. We relax a shared-classifier assumption made by previous methods and assume that the source classifier and target classifier differ by a residual function. We enable classifier adaptation by plugging several layers into deep network to explicitly learn the residual function with reference to the target classifier. We fuse features of multiple layers with tensor product and embed them into reproducing kernel Hilbert spaces to match distributions for feature adaptation. The adaptation can be achieved in most feed-forward models by extending them with new residual layers and loss functions, which can be trained efficiently via back-propagation. Empirical evidence shows that the new approach outperforms state of the art methods on standard domain adaptation benchmarks.
1 Introduction
Deep networks need extensive labeled data, motivating domain adaptation from related source domains despite distribution shifts. This paper proposes jointly adapting classifiers and features rather than assuming a shared classifier.
- Massive labeled datasets drive deep-network gains, but collecting sufficient labels for diverse domains is often prohibitive.
- Domain adaptation leverages labeled source data and unlabeled target data, but distribution shifts obstruct direct transfer of predictive models.
- Prior deep adaptation methods commonly learn domain-invariant features so a source-trained classifier can be applied to the target domain.
- The paper challenges the strong assumption that source and target classifiers can be shared after feature adaptation.
- Residual Transfer Networks jointly learn adaptive classifiers and transferable features from labeled source and unlabeled target data within one deep architecture.
2 Related Work
Related work frames domain adaptation as reducing distribution discrepancies while developing transferable representations. Residual learning motivates explicitly modeling classifier mismatch, extending prior perturbation approaches to the unlabeled-target setting.
- Domain adaptation addresses shifts between training and test distributions by learning representations that bridge source and target domains without target labels.
- Deep networks can reduce, but not remove, cross-domain discrepancies, leaving dataset shift as a bottleneck for feature transferability.
- Residual networks reformulate layers as residual functions relative to their inputs, inspiring the paper’s residual treatment of classifier mismatch.
- Unlike prior classifier-perturbation methods requiring target labels and using input features, this approach models the residual with reference to the target classifier for unsupervised adaptation.
3 Residual Transfer Networks
Residual Transfer Networks jointly adapt features and classifiers for unsupervised domain adaptation, addressing mismatches between source and target distributions without target labels. The architecture combines tensor-based feature adaptation, residual classifier adaptation, and entropy refinement in an end-to-end framework.
- Problem: Unsupervised domain adaptation must address mismatches in both feature distributions and classifiers when source and target domains differ.The target domain is unlabeled, so the source classifier cannot necessarily be applied directly.
- Architecture: RTN extends deep CNNs to jointly learn transferable features and adaptive classifiers from labeled source and unlabeled target data.The model integrates deep feature learning, feature adaptation, and classifier adaptation end to end.
- Feature Adaptation: Tensor MMD fuses multiple feature layers before matching source and target distributions, capturing cross-layer interactions with one penalty.This contrasts with adapting each layer separately using independent MMD penalties.
- Classifier Adaptation: RTN models the source classifier as the target classifier plus a learned residual function, using residual layers to represent classifier mismatch.The residual formulation is implemented by mapping fT(x) to fS(x) through the residual block.
- Classifier Adaptation: Residual layers are trained with source supervision while their perturbation depends functionally on the target classifier, avoiding the need for target labels.The source classifier is the residual-block output so standard back-propagation can train it from labeled source data.
- Classifier Adaptation: Entropy minimization further refines the target classifier by encouraging predictions to separate through low-density regions of the target distribution.The entropy penalty makes the target classifier directly responsive to unlabeled target data.
4 Experiments
Experiments evaluate RTN on Office-31 and Office-Caltech under unsupervised adaptation protocols, compare it with conventional and deep-transfer baselines, and examine its modules and behavior. RTN performs best on most transfer tasks while classifier adaptation, entropy minimization, and residual responses support its design.
- Experimental setup: RTN is evaluated on six Office-31 and twelve Office-Caltech transfer tasks using standard unsupervised adaptation protocols.All labeled source data and unlabeled target data are used, with average classification accuracy reported across three random experiments.
- Experimental setup: The comparison includes conventional transfer methods, AlexNet, and deep domain-adaptation methods DDC, DAN, and RevGrad.The baselines span MMD-based, manifold-learning, supervised deep-learning, and adversarial domain-adaptation approaches.
- Results: RTN outperforms comparison methods on most transfer tasks, substantially improving hard tasks while remaining comparable on easier tasks.The reported hard tasks include A →W and C →W; easier tasks include D →W and W →D.
- Results: RTN sets a new state-of-the-art result on the benchmark datasets by adapting classifier layers in addition to feature layers.The classifier adaptation bridges source and target classifiers in an end-to-end residual learning framework.
- Ablation study: Ablations show that tensor MMD, entropy minimization, and residual transfer each contribute, with RTN (mmd+ent+res) performing best across variants.The entropy penalty improves target-classifier adaptation, while residual transfer supports more adaptive classifiers; using both prevents the residual function from tending toward zero mapping.
- Discussion: t-SNE visualizations indicate that DAN predictions poorly discriminate target categories, whereas RTN produces larger class-to-class distances through the target classifier.These observations support residual transfer of classifiers as an extension to feature adaptation.
- Discussion: Layer responses show that the residual function generally has much smaller responses than the target-classifier shortcut, supporting the small-gap modeling assumption.The analysis compares outputs from the target-classifier, residual, and summed layers.
- Discussion: Classifier weights trained separately on source and target data are substantially different, while RTN accuracy follows a bell-shaped curve as entropy parameter γ varies.Sensitivity is tested on A →W with 31 classes and C →W with 10 classes across seven γ values.
5 Conclusion
The proposed approach jointly learns adaptive classifiers and transferable features for unsupervised domain adaptation. It supports end-to-end training with standard back-propagation and complements existing feature-adaptation techniques.
- A residual transfer module bridges source and target classifiers, unlike methods that adapt features without classifier adaptation.
- Feature adaptation matches feature distributions across domains, complementing the proposed classifier adaptation.
- Standard back-propagation makes the approach scalable and implementable in most deep learning packages.
- The paper identifies semi-supervised domain adaptation extensions as future work.