Source-linked AI summary
Learning Transferable Features with Deep Adaptation Networks
Mingsheng Long, Yue Cao, Jianmin Wang, Michael I. Jordan
TL;DR
As deep features become task-specific, their transferability declines across domains, especially when domain discrepancy increases. DAN addresses this by matching multi-layer feature distributions with multi-kernel mean embeddings and significantly outperforms comparison methods on most transfer tasks.
Problem
Deep networks’ higher-layer features become increasingly task-specific and less transferable as domain discrepancy grows, limiting cross-domain adaptation with scarce target labels.
Method
DAN matches source and target mean embeddings for task-specific layers in a reproducing kernel Hilbert space using multi-layer, multi-kernel adaptation.
Results
DAN significantly outperforms comparison methods on most transfer tasks and performs comparably on easy tasks where source and target domains are similar.
Takeaways & Limitations
Multi-layer mean-embedding matching with optimal multi-kernel selection substantially enhances feature transferability across domains and supports linear-time estimation.
Takeaways & Limitations
The paper leaves principled selection of the general-to-specific boundary and adaptation of convolutional layers for future study.
Abstract
from arXiv · showhide
Recent studies reveal that a deep neural network can learn transferable features which generalize well to novel tasks for domain adaptation. However, as deep features eventually transition from general to specific along the network, the feature transferability drops significantly in higher layers with increasing domain discrepancy. Hence, it is important to formally reduce the dataset bias and enhance the transferability in task-specific layers. In this paper, we propose a new Deep Adaptation Network (DAN) architecture, which generalizes deep convolutional neural network to the domain adaptation scenario. In DAN, hidden representations of all task-specific layers are embedded in a reproducing kernel Hilbert space where the mean embeddings of different domain distributions can be explicitly matched. The domain discrepancy is further reduced using an optimal multi-kernel selection method for mean embedding matching. DAN can learn transferable features with statistical guarantees, and can scale linearly by unbiased estimate of kernel embedding. Extensive empirical evidence shows that the proposed architecture yields state-of-the-art image classification error rates on standard domain adaptation benchmarks.
1. Introduction
Domain adaptation reduces labeling costs by transferring labeled data across related but differently distributed domains, yet deep features become less transferable and can enlarge domain discrepancy in higher layers. DAN addresses this challenge by layerwise adapting task-specific representations and explicitly reducing domain discrepancy with multiple kernels.
- Motivation: Limited training data can cause large generalization error, while manually labeling sufficient data across diverse domains may be prohibitive.This motivates algorithms that reduce labeling costs by leveraging labeled data from relevant source domains.
- Domain adaptation challenge: Domain adaptation transfers knowledge between related domains with different distributions, but domain discrepancy obstructs predictive-model adaptation.Models trained on annotated images may fail under substantial changes in pose, occlusion, or illumination.
- Deep feature transferability: Deep features transition from general to specific across the network, causing transferability to drop significantly in higher layers.Higher-layer features depend strongly on the specific dataset and task and are not safely transferable to novel tasks.
- Deep feature transferability: Higher-layer representations can enlarge domain discrepancy, making domains more mutually distinguishable and potentially causing statistically unbounded target-task risk.Although deep features remain salient for discrimination, enlarged dataset bias can deteriorate domain-adaptation performance.
- DAN contribution: DAN generalizes deep convolutional neural networks to domain adaptation by enhancing task-specific feature transferability through explicit domain-discrepancy reduction.Its architecture adapts all task-specific layers layerwise and explores multiple kernels, improving adaptation effectiveness compared with single-kernel methods while providing statistical guarantees.
2. Related Work
Related work frames domain adaptation as transfer learning that reduces domain discrepancy and labeling effort. Existing approaches often match distributions through shallow representations, while deep networks offer transferable nonlinear features but prior transferability methods may require substantial target labels.
- Transfer learning: Transfer learning bridges domains or tasks while explicitly considering domain discrepancy and reducing manual labeling effort.It has been applied across machine learning and computer vision.
- Distribution matching: Matching different domain distributions effectively is a major bottleneck, and most existing methods learn shallow representations to reduce domain discrepancy explicitly.These methods do so without learning deep features.
- Deep representations: Deep neural networks learn nonlinear representations that disentangle explanatory factors and capture invariant factors transferable from original tasks to similar novel tasks.This motivated their use in domain adaptation, multimodal learning, and multi-source learning.
- Prior deep transfer methods: Yosinski et al. (2014) studied deep convolutional feature transferability when learning tasks differ across domains.Their method requires sufficient target labeled examples so a source network can be fine-tuned to the target task, limiting direct applicability when novel target labels are scarce.
3. Deep Adaptation Networks
DAN adapts task-specific deep representations by matching source and target distributions with multi-kernel maximum mean discrepancy (MK-MMD). It freezes general early layers, fine-tunes intermediate layers, regularizes higher layers, and alternates scalable parameter and kernel optimization with theoretical target-risk guarantees.
- MK-MMD adaptation: DAN embeds hidden representations in an RKHS and minimizes MK-MMD between source and target distributions across multiple task-specific layers.MK-MMD is the RKHS distance between distribution mean embeddings, with a characteristic multi-kernel selected to improve test power and reduce domain discrepancy.
- DAN architecture: DAN extends AlexNet by freezing conv1–conv3, fine-tuning conv4–conv5, and adapting fc6–fc8 because transferability declines from general to specific layers.The implementation applies distribution-adaptation regularization to layers 6–8 while preserving generic early features and fragile co-adaptation in intermediate layers.
- DAN architecture: Multi-layer adaptation addresses dataset bias more effectively than adapting a single layer because transferability worsens in conv4–conv5 and drops significantly in fc6–fc8.The paper identifies multi-layer adaptation as a distinguishing advantage of DAN over approaches that adapt only one layer.
- Optimization: Alternating mini-batch SGD updates for Θ and quadratic-programming updates for β each cost O(n), making DAN scalable to large datasets.The kernel coefficients β are optimized to consolidate feature transferability while the network parameters Θ are updated iteratively.
- Theoretical guarantees: Explicitly minimizing MK-MMD across layers decreases the upper bound on target risk, while jointly optimizing β reduces Type II two-sample test error.The source classifier and two-sample classifier also provide a way to assess adaptation performance and facilitate model selection.
4. Experiments
Experiments evaluate DAN against shallow and deep transfer-learning baselines under unsupervised and semi-supervised adaptation protocols. DAN generally outperforms comparison methods, with analyses attributing its gains to multi-kernel, multi-layer distribution matching.
- Experimental setup: DAN is evaluated against state-of-the-art transfer-learning and deep-learning methods on unsupervised and semi-supervised adaptation tasks.The comparisons focus on multi-layer adaptation with multi-kernel MMD.
- Experimental setup: The experiments use Office-31’s three domains across transfer tasks including A → W, D → W, and W → D, with standard and classical evaluation protocols.Unsupervised evaluation uses labeled source and unlabeled target examples; semi-supervised evaluation requires 3 labeled target examples per category.
- Main results: DAN significantly outperforms comparison methods on most transfer tasks and achieves comparable performance on the easier D → W and W → D tasks.The study reports average classification accuracies and standard errors for each task.
- Main results: Deep-learning methods outperform shallow transfer-learning methods, while DDC generally improves over CNN through single-layer, single-kernel MMD adaptation.LapCNN provides no improvement over CNN in the reported semi-supervised comparison.
- Ablation analysis: DAN7 and DAN8 outperform DDC, indicating that multi-kernel MMD better bridges domain discrepancy by matching low-order and high-order moments.DANSK also exceeds DDC, supporting multi-layer adaptation across task-specific feature abstractions.
- Feature analysis: DAN features discriminate target points and align source–target categories better than DDC features, while DAN’s domain discrepancy is intended to be smaller than CNN’s.Transfer accuracy follows a bell-shaped relationship with λ, increasing and then decreasing across the tested values.
5. Conclusion
The paper proposes Deep Adaptation Network (DAN) to enhance task-specific feature transferability through multi-layer mean-embedding matching across domains. It also identifies principled boundary selection and convolutional-layer adaptation as directions for further improving transferability.
- Conclusion: DAN enhances the transferability of features from task-specific neural-network layers.The architecture is proposed specifically to address reduced transferability in task-specific representations.
- Conclusion: General features can generalize to novel tasks, whereas task-specific features cannot effectively bridge domain discrepancy.The conclusion contrasts the transfer behavior of general and original-task-specific representations.
- Conclusion: Mean-embedding matching across multi-layer representations substantially enhances feature transferability.The approach matches representations across domains to improve transferability.
- Conclusion: Future work should develop principled criteria for the boundary between general and specific features.The paper highlights boundary selection as an open question arising from feature transition through the network.
- Conclusion: Applying distribution adaptation to CNN convolutional layers may further enhance feature transferability.The passage identifies convolutional-layer distribution adaptation as another direction for improvement.