Source-linked AI summary
A Survey on Deep Transfer Learning
Chuanqi Tan, Fuchun Sun, Tao Kong, Wenchang Zhang, Chao Yang, Chunfang Liu
TL;DR
Deep learning applications are constrained by costly acquisition and annotation of sufficiently large training datasets. This survey defines and categorizes deep transfer learning, reviews its four technique-based categories, and concludes that combined technologies are often used in practice while unsupervised and semi-supervised transfer, negative transfer, and transferability remain open issues.
Problem
Deep learning depends on massive training data, while some domains face expensive data collection and annotation that make large, high-quality datasets difficult to construct.
Method
The survey defines deep transfer learning, categorizes it into four technique-based categories, and reviews research works within those categories.
Results
The survey presents instances-based, mapping-based, network-based, and adversarial-based deep transfer learning and reviews representative research in these categories.
Takeaways & Limitations
Multiple deep transfer learning technologies are often combined in practical applications, while unsupervised or semi-supervised transfer remains a future research focus.
Takeaways & Limitations
Most current research focuses on supervised learning, and negative transfer, transferability measures, and stronger physical support for transfer knowledge require further research.
Abstract
from arXiv · showhide
As a new classification platform, deep learning has recently received increasing attention from researchers and has been successfully applied to many domains. In some domains, like bioinformatics and robotics, it is very difficult to construct a large-scale well-annotated dataset due to the expense of data acquisition and costly annotation, which limits its development. Transfer learning relaxes the hypothesis that the training data must be independent and identically distributed (i.i.d.) with the test data, which motivates us to use transfer learning to solve the problem of insufficient training data. This survey focuses on reviewing the current researches of transfer learning by using deep neural network and its applications. We defined deep transfer learning, category and review the recent research works based on the techniques used in deep transfer learning.
1 Introduction
Deep learning achieves strong application potential through hierarchical feature learning but depends heavily on large, annotated datasets. The survey motivates deep transfer learning as a way to address data scarcity and organizes prior work into four categories.
- Deep learning automatically learns high-level features from mass data through unsupervised or semi-supervised hierarchical representation learning.
- Data dependence is a major challenge because deep models require massive training data to discover latent patterns.
- Expensive data collection and annotation make large, high-quality datasets difficult to build in domains such as bioinformatics.
- Transfer learning relaxes the i.i.d. requirement and avoids training target-domain models from scratch, reducing target-domain data and training-time demands.
- The survey defines deep transfer learning, categorizes it into four categories, and reviews current research using standardized descriptions and sketches.
2 Deep Transfer Learning
The paper formalizes transfer learning and defines deep transfer learning as transfer learning whose target predictive function is a nonlinear deep-neural-network function. It presents transfer learning as knowledge transfer across related but non-identical domains or tasks.
- The learning process of transfer learning is illustrated in Fig. 1.
- A domain consists of a feature space and probability distribution, while a task consists of a label space and target prediction function.
- Transfer learning improves a target task by discovering and transferring latent knowledge from a different source domain or task.
- The survey distinguishes transfer-learning methods according to relationships between source and target domains and tasks.
- Deep transfer learning uses a nonlinear target predictive function that reflects a deep neural network.
3 Categories
The survey classifies deep transfer learning into four technique-based categories: instances-based, mapping-based, network-based, and adversarial-based methods. These categories differ in whether they reweight source instances, map domains into a shared space, reuse pretrained network components, or learn domain-indiscriminate representations.
- Deep transfer learning is categorized into instances-based, mapping-based, network-based, and adversarial-based approaches.The survey organizes the categories according to the techniques used in deep transfer learning.
- Instances-based deep transfer learning: Instances-based methods select source-domain instances and assign them weights before combining them with target-domain training data.Instances dissimilar to the target domain are excluded, while similar instances are included with appropriate weights.
- Mapping-based deep transfer learning: Mapping-based methods transform source- and target-domain instances into a new data space where they are more similar, then use the mapped instances for neural-network training.Deep distribution-matching methods extend techniques such as MMD, MK-MMD, and JMMD to learn domain-adapted representations.
- Network-based deep transfer learning: Network-based methods reuse a source-pretrained network's structure and parameters as part of a target-domain network, optionally updating the transferred subnetwork through fine-tuning.The survey also relates network structure to transferability, identifying transferable features and suitable architectures.
- Adversarial-based deep transfer learning: Adversarial-based methods learn representations that remain discriminative for the task while making source- and target-domain features difficult to distinguish.An adversarial layer evaluates feature-domain discriminability, and its performance guides the network toward more transferable general features.
4 Conclusion
The survey classifies deep transfer learning into four categories and reviews current research in the field. It identifies combined techniques, unsupervised and semi-supervised transfer, transferability, negative transfer, and stronger physical support as future research directions.
- Deep transfer learning is classified into four categories: instances-based, mapping-based, network-based, and adversarial-based.
- Multiple deep transfer learning technologies are often combined in practical applications to achieve better results.
- Future research should address unsupervised and semi-supervised learning, negative transfer, transferability measures, and physical support for knowledge transfer.