Source-linked AI summary
Deep Transfer Learning with Joint Adaptation Networks
Mingsheng Long, Han Zhu, Jianmin Wang, Michael I. Jordan
TL;DR
Limited labeled data and source–target distribution shifts make adapting deep models to target tasks difficult. JAN aligns joint distributions across multiple task-specific layers using JMMD and adversarial training, with efficient linear-time back-propagation. The approach reports state-of-the-art results on standard datasets and reduces measured joint-distribution shifts.
Problem
Adapting models is difficult when target labels are scarce and source and target domains differ in their joint distributions of inputs and labels.
Method
JAN aligns joint distributions of activations from multiple task-specific layers using JMMD, CNN fine-tuning, and adversarial training.
Results
JAN models outperform comparison methods on most Office-31 transfer tasks and report state-of-the-art results on standard datasets.
Takeaways & Limitations
Reducing shifts in joint distributions of multiple task-specific activations enables end-to-end learning of transferable representations.
Takeaways & Limitations
Kernel-based MMD may suffer vanishing gradients with low-bandwidth kernels or fail to capture complex distances in high-dimensional spaces.
Abstract
from arXiv · showhide
Deep networks have been successfully applied to learn transferable features for adapting models from a source domain to a different target domain. In this paper, we present joint adaptation networks (JAN), which learn a transfer network by aligning the joint distributions of multiple domain-specific layers across domains based on a joint maximum mean discrepancy (JMMD) criterion. Adversarial training strategy is adopted to maximize JMMD such that the distributions of the source and target domains are made more distinguishable. Learning can be performed by stochastic gradient descent with the gradients computed by back-propagation in linear-time. Experiments testify that our model yields state of the art results on standard datasets.
1. Introduction
Deep transfer learning addresses the lack of labeled target data by leveraging labeled source data, but distribution shifts hinder adaptation. JAN aligns joint distributions across multiple domain-specific layers and uses JMMD with adversarial training for efficient learning.
- Motivation: Limited labeled data in target domains motivates leveraging richly labeled source domains for transfer learning.Manual labeling sufficient data for diverse applications is often prohibitive.
- Related work: Recent deep transfer methods embed domain adaptation into deep-learning pipelines to learn transferable representations and match marginal distributions.These methods use deep networks to disentangle explanatory variation factors while adapting representations.
- Motivation: Distribution shifts across source and target domains are a major obstacle to adapting predictive models.The challenge becomes especially significant when joint distributions of input features and output labels change.
- JAN: JAN aligns joint distributions of activations from multiple domain-specific layers across source and target domains.The method uses joint maximum mean discrepancy (JMMD) to measure the distance between kernel mean embeddings of empirical joint distributions.
- JAN: Adversarial training maximizes JMMD to make source and target distributions more distinguishable, while linear-time estimation enables mini-batch back-propagation.Experiments report state-of-the-art results on standard datasets.
2. Related Work
Transfer learning seeks models that generalize across domains with different distributions, but joint-distribution shifts make adaptation difficult. JAN directly manipulates joint distributions without assuming how marginal or conditional components change.
- Transfer learning: Transfer learning aims to build learning machines that generalize across domains following different probability distributions.The paradigm has applications including computer vision and natural language processing.
- Existing methods: Most existing transfer methods minimize domain discrepancy using shallow representations, while deep networks reduce but do not remove cross-domain discrepancy.Deep networks learn abstract representations that disentangle explanatory variation factors.
- Joint-distribution shift: Joint-distribution shifts may arise from changes in marginal input distributions, conditional distributions, or both.Conditional shifts commonly require additional assumptions to make the problem tractable.
- JAN: JAN directly manipulates joint distributions without assumptions on marginal or conditional distributions, addressing an unresolved issue in deep architectures.The paper states that accounting for joint-distribution shifts within deep architectures remained unclear.
3. Preliminary
Kernel mean embeddings represent distributions in reproducing kernel Hilbert spaces, enabling nonparametric comparison from samples. Tensor-product feature spaces extend this representation to joint distributions, while MMD measures discrepancies between distributions through embedding distances.
- Hilbert space embedding: A reproducing kernel Hilbert space represents a distribution through its expected feature map, called a kernel mean embedding.The embedding supports evaluating expectations of RKHS functions as inner products.
- Hilbert space embedding: Finite samples provide empirical kernel embeddings that converge to their population counterparts at rate O(n^-1/2).The estimate is formed from i.i.d. samples drawn from the distribution.
- Joint embeddings: Joint distributions of multiple variables are embedded in tensor-product feature spaces using the tensor product of their feature maps.The construction uses one kernel and feature map for each variable.
- Joint embeddings: The empirical joint embedding estimates the population joint embedding from finite samples and converges at a similar rate to marginal embedding.The sample is drawn i.i.d. from the joint distribution.
- Maximum Mean Discrepancy: Maximum Mean Discrepancy is a kernel two-sample test that compares source and target distributions using the distance between their mean embeddings.With a universal RKHS, MMD equals zero if and only if the two distributions are identical, and unbiased empirical estimates can be computed from samples.
4. Joint Adaptation Networks
JAN adapts unsupervised source-to-target transfer by matching joint distributions across multiple task-specific layers, while adversarial training strengthens JMMD-based discrimination. A linear-time JMMD estimate enables mini-batch SGD and back-propagation.
- Joint Adaptation Networks: JAN aligns joint distributions of activations across multiple domain-specific layers within a CNN transfer pipeline.The method targets activations whose transferability decreases as deep features become task-specific.
- Joint Maximum Mean Discrepancy: JMMD measures discrepancy between source and target joint activation distributions and captures interactions among variables across layers.Unlike MMD’s uniform per-layer weighting, JMMD uses non-uniform weights reflecting other layers’ influence.
- Joint Adaptation Networks: JAN jointly minimizes CNN classification error and JMMD over selected task-specific layers, while leaving lower transferable layers unmatched.AlexNet uses fc6, fc7, and fc8; ResNet uses pool5 and fc.
- Efficient Optimization: A linear-time unbiased JMMD estimate fits mini-batch SGD, supports back-propagation, and scales linearly to large samples.Mini-batches use equal numbers of source and target points to eliminate bias caused by domain size.
- Adversarial Training: JAN-A adds neural layers that maximize JMMD, making source and target distributions more distinguishable during adversarial training.This addresses vanishing gradients and limited function classes associated with kernel-based MMD.
5. Experiments
Experiments on Office-31 and ImageCLEF-DA show that JAN generally outperforms comparison methods, especially on difficult domain transfers. Analyses attribute this performance to joint-distribution alignment, while examining representation visualization, discrepancy, parameter sensitivity, and convergence.
- Office-31 results: JAN outperforms comparison methods on most Office-31 transfer tasks, with substantial gains on difficult transfers and comparable accuracy on easier ones.The difficult tasks involve substantially different domains, whereas D →W and W →D involve similar domains.
- Method comparison: JAN sets new state-of-the-art results by adapting joint distributions across multiple task-specific layers rather than independently adapting layer-wise marginal distributions.JAN uses a JMMD penalty, whereas DAN uses separate MMD penalties and assumes feature and classifier layers are independent.
- ImageCLEF-DA results: JAN models outperform comparison methods on most ImageCLEF-DA tasks, but with smaller improvements than on Office-31.The more balanced ImageCLEF-DA domains provide a controlled complement to Office-31, whose domain sizes differ.
- Feature visualization: JAN produces clearer target-category discrimination than DAN in t-SNE visualizations of ResNet activations for task A →W.The visualization indicates that multilayer joint-distribution adaptation improves the source classifier's separation of target categories.
- Distribution discrepancy: JAN features yield smaller A-distance and JMMD than CNN and DAN features, indicating reduced marginal and joint distribution discrepancies.The A-distance analysis covers tasks A →W and W →D, while JMMD is computed using fc7 features and ground-truth fc8 labels.
- Sensitivity and convergence: Transfer accuracy follows a bell-shaped curve as JMMD weight λ varies, and JAN converges fastest while JAN-A matches RevGrad's convergence speed with higher accuracy.The λ analysis varies λ from 0.01 to 1 on A →W; convergence is evaluated using test errors on the same task.
6. Conclusion
The paper presents an end-to-end approach that reduces shifts in joint distributions across multiple task-specific layers, approximating shifts between input features and output labels. It computes these discrepancies in a tensor-product Hilbert space with linear scalability and reports empirical efficacy.
- JAN enables end-to-end learning of transferable representations.
- Unlike methods matching marginal feature distributions, JAN reduces shifts in joint distributions across multiple task-specific activation layers.
- The joint activation-distribution shift approximates the shift in joint distributions of input features and output labels.
- Joint-distribution discrepancy is computed in a tensor-product Hilbert space, scaled linearly to large samples, and implemented in most deep networks.
- Experiments testify to the efficacy of the proposed approach.