PaperlayerView exact revision on arXiv

Source-linked AI summary

A Comprehensive Survey on Transfer Learning

Fuzhen Zhuang, Zhiyuan Qi, Keyu Duan, Dongbo Xi, Yongchun Zhu, Hengshu Zhu, Hui Xiong, Qing He

arXiv:1911.02685v3cs.LGstat.ML

TL;DR

Transfer learning research needs a systematic account of methods and mechanisms because labeled target-domain data can be difficult to obtain. This survey organizes over forty approaches by data and model perspectives and finds that model selection matters across applications and datasets.

  • Problem

    Sufficient labeled training data are often expensive, time-consuming, or unrealistic to collect in real-world machine-learning scenarios.

  • Method

    The survey systematizes over forty transfer-learning approaches by data and model perspectives and evaluates representative models across multiple datasets.

  • Results

    Model performance varies across datasets and tasks, with HIDC, SFA, and MTrick showing relatively consistent performance in the reported experiments.

  • Takeaways & Limitations

    Selecting an appropriate transfer-learning model is an important and complex issue in practical applications.

  • Takeaways & Limitations

    FAM may not work well for heterogeneous transfer-learning tasks when source and target domains use different feature representations.

Abstract

from arXiv · show

Transfer learning aims at improving the performance of target learners on target domains by transferring the knowledge contained in different but related source domains. In this way, the dependence on a large number of target domain data can be reduced for constructing target learners. Due to the wide application prospects, transfer learning has become a popular and promising area in machine learning. Although there are already some valuable and impressive surveys on transfer learning, these surveys introduce approaches in a relatively isolated way and lack the recent advances in transfer learning. Due to the rapid expansion of the transfer learning area, it is both necessary and challenging to comprehensively review the relevant studies. This survey attempts to connect and systematize the existing transfer learning researches, as well as to summarize and interpret the mechanisms and the strategies of transfer learning in a comprehensive way, which may help readers have a better understanding of the current research status and ideas. Unlike previous surveys, this survey paper reviews more than forty representative transfer learning approaches, especially homogeneous transfer learning approaches, from the perspectives of data and model. The applications of transfer learning are also briefly introduced. In order to show the performance of different transfer learning models, over twenty representative transfer learning models are used for experiments. The models are performed on three different datasets, i.e., Amazon Reviews, Reuters-21578, and Office-31. And the experimental results demonstrate the importance of selecting appropriate transfer learning models for different applications in practice.

1 INTRODUCTION

Transfer learning addresses the difficulty of obtaining abundant labeled data by transferring knowledge across related domains, while recognizing that prior knowledge can also cause negative transfer. This survey systematizes transfer learning from data and model perspectives, emphasizing homogeneous approaches and evaluating model selection across datasets.

  • Motivation: Traditional machine learning often requires abundant same-distribution labeled data, but collecting sufficient data can be expensive, time-consuming, or unrealistic.
  • Motivation: Transfer learning transfers knowledge across domains to address limited training data, provided the learning activities share a meaningful connection.
  • Challenges: Previous experience can produce negative transfer when it interferes with learning a new task, as related languages may still differ in vocabulary, pronunciation, or conjugation.
  • Taxonomy: Transfer learning is divided into homogeneous and heterogeneous categories according to domain discrepancy; homogeneous methods address domains with the same feature space.

2 RELATED WORK

This section situates transfer learning among related machine-learning paradigms and clarifies their connections and differences. It emphasizes that multi-task learning transfers knowledge through simultaneous learning of related tasks, whereas transfer learning transfers knowledge across related domains with greater focus on the target task.

  • 2 RELATED WORK: The section introduces related areas and clarifies their connections and differences with transfer learning.It frames this discussion as related work on transfer learning.
  • 2 RELATED WORK: Semi-supervised learning lies between supervised learning with completely labeled instances and unsupervised learning.The passage identifies semi-supervised learning as both a machine-learning task and method.
  • 2 RELATED WORK: Multi-task learning enhances task generalization by exploiting inter-task interconnections while simultaneously learning related tasks.It considers both inter-task relevance and inter-task difference.
  • 2 RELATED WORK: Transfer learning differs from multi-task learning by transferring knowledge across related domains and prioritizing the target task over the source task.Multi-task learning pays equal attention to each task, whereas transfer learning focuses more on the target task.

3 OVERVIEW

The overview defines domains and tasks as the basic elements of transfer learning, then characterizes transfer learning by source-target settings and knowledge-transfer strategies. It also explains that the survey organizes approaches primarily from data and model perspectives rather than strictly following prior categorizations.

  • Definitions: A domain D consists of a feature space X and marginal distribution P(X), written D = {X, P(X)}.Instances are represented as X = {x|xi ∈X, i = 1, · · · , n}.
  • Definitions: A task T consists of a label space Y and an implicitly learned decision function f, written T = {Y, f}.For probabilistic models, f(xj) can output conditional distributions over labels.
  • Definitions: Transfer learning uses knowledge from observed source domains and tasks to improve learned decision functions on one or more target domains.Source observations commonly contain instance-label pairs, whereas target observations may contain unlabeled or limited labeled instances.
  • Categorization: The framework distinguishes single-source from multi-source transfer learning according to whether mS equals 1 or exceeds 1, while most studies focus on mT = 1.Domain adaptation is described as adapting source domains to transfer knowledge and reduce differences between domains.
  • Categorization: Transfer learning approaches are categorized as instance-based, feature-based, parameter-based, or relational-based, with feature-based methods divided into asymmetric and symmetric variants.Asymmetric methods align source features with target features, whereas symmetric methods map both into a common latent space.
  • Survey organization: The survey interprets approaches from data and model perspectives, broadly covering instance- and feature-based methods through data-based analysis and parameter-based methods through model-based analysis.Its organization does not strictly follow the previously mentioned categorizations.

4 DATA-BASED INTERPRETATION

Data-based transfer learning approaches transfer knowledge by adjusting and transforming data. Their strategies and objectives are organized from a data perspective, including space adaptation, while the survey emphasizes homogeneous transfer learning.

  • Data-based approaches transfer knowledge through data adjustment and transformation.
  • The survey presents data-based strategies and objectives from the data perspective in Fig. 3.
  • Space adaptation is an objective required mostly in heterogeneous transfer learning scenarios.
  • The survey focuses more on homogeneous transfer learning than on heterogeneous scenarios.

4.1 Instance Weighting Strategy

Instance weighting reduces source–target marginal-distribution differences by assigning source instances weights in the loss, with ideal weights given by the target-to-source density ratio. KMM and related methods estimate these weights, while iterative and heuristic frameworks downweight harmful or mismatched source instances before training the target classifier.

  • Instance Weighting Strategy: Instance weighting assigns source-domain instances weights in the loss to reduce marginal distribution differences when P S(X) ≠ P T(X) but P S(Y |X) = P T(Y |X).The strategy addresses settings with many labeled source instances and limited target-domain instances.
  • Instance Weighting Strategy: The theoretical weighting parameter is βi = P T(xi)/P S(xi), but this ratio is generally unknown and difficult to obtain using traditional methods.The parameter applies for i = 1, 2, · · · , nS.
  • Instance Weighting Strategy: Kernel Mean Matching estimates unknown density ratios by matching source and target instance means in an RKHS, after which weighted source instances can train a learner.Its optimization can be converted into quadratic programming using the kernel trick, allowing incorporation into existing algorithms.
  • Instance Weighting Strategy: KLIEP estimates instance weights by minimizing KL divergence and includes a built-in model selection procedure.Other instance-based frameworks combine instance weights with domain weights before training the target classifier on reweighted source and labeled target instances.
  • Instance Weighting Strategy: TrAdaBoost iteratively decreases the weights of source instances that negatively affect the target learner, extending AdaBoost with successive weak-classifier updates.MsTrAdaBoost selects the source–target candidate classifier with minimal target-domain classification error to update instance weights.

4.2 Feature Transformation Strategy

Feature transformation transfers knowledge by mapping source and target data into representations that reduce distribution differences while preserving useful structure. The survey covers distribution-based metrics, feature augmentation for homogeneous and heterogeneous settings, and mapping or encoding methods such as TCA, JDA, and SDA.

  • Feature Transformation Strategy: Feature-based approaches transform original features into new representations, often using common latent features as a bridge between related source and target domains.The objectives include reducing marginal and conditional distribution differences while preserving useful information.
  • Distribution Difference Measurement: MMD measures distribution difference through the distance between instance means in a reproducing kernel Hilbert space and is widely used in transfer learning.Kernel tricks enable efficient computation, while related criteria include Wasserstein distance and central moment discrepancy.
  • Feature Augmentation: FAM augments a single-source feature space to three times its original size using general, source-specific, and target-specific features.Although its elegant expansion has useful properties, the augmentation is redundant and may be ineffective for heterogeneous feature representations.
  • Feature Augmentation: HFA addresses heterogeneous transfer by mapping source and target features into a common space, then stacking mapped, original, and zero elements in a structured representation.It is designed to overcome the limitations of directly replicating features and padding zero vectors across domains with different feature representations.
  • Mapping-Based Feature Extraction: TCA minimizes marginal distribution difference with MMD and a scatter constraint, whereas JDA minimizes both marginal and conditional differences using MMD, pseudo-labels, and eigen-decomposition.TCA learns a linear mapping from an empirical kernel feature space, while JDA learns a transformation matrix through trace optimization.

5 MODEL-BASED INTERPRETATION

This section interprets transfer learning approaches from the model perspective, organizing them by strategies and objectives. Models target accurate predictions in the target domain and may combine sub-modules with different roles.

  • 5 MODEL-BASED INTERPRETATION: Model-based interpretation organizes transfer learning approaches according to their strategies and objectives.The corresponding strategies and objectives are presented in Fig. 4.
  • 5 MODEL-BASED INTERPRETATION: The main objective of a transfer learning model is accurate prediction on the target domain, including classification or clustering results.
  • 5 MODEL-BASED INTERPRETATION: A transfer learning model may contain classifiers, extractors, or encoders that serve roles such as feature adaptation or pseudo-label generation.
  • 5 MODEL-BASED INTERPRETATION: The approaches are introduced in an order corresponding to the strategies shown in Fig. 4.

5.1 Model Control Strategy

Model control strategies transfer source-model knowledge by adding model-level regularizers to the target learner’s objective. The survey presents DAM and related extensions that regulate target loss, decision-function complexity, source-model agreement, domain dependence, and Universum information.

  • Model Control Strategy: DAM transfers knowledge from pre-obtained source models by adding model-level regularizers to the target learner’s objective during training.DAM is a general framework designed for multi-source transfer learning.
  • Model Control Strategy: The DAM objective combines labeled target-domain classification loss, regularizers, and a term controlling the complexity of the final decision function.The classification loss may use square error or cross-entropy, and some transfer-learning approaches are special cases of this framework.
  • Consensus Regularizer: CRF constructs one classifier per source domain and uses a cross-entropy consensus regularizer to enforce agreement on an unlabeled target domain.The regularizer also reduces uncertainty in target-domain predictions.
  • Domain-dependent Regularizer: Fast-DAM uses a domain-dependent regularizer motivated by manifold and graph-based assumptions, weighting source decision functions by relevance measured with MMD.An ε-insensitive loss term can improve the computational efficiency of the resulting model.
  • Domain-dependent Regularizer + Universum Regularizer: Univer-DAM extends Fast-DAM with a Universum regularizer that treats source-domain instances as examples outside the target domain’s positive and negative classes.Like Fast-DAM, it can also use an ε-insensitive loss function.

5.2 Parameter Control Strategy

Parameter control strategies transfer knowledge by sharing or constraining model parameters between source and target learners. Sharing may freeze most source-network layers or identify stable cross-domain structure through matrix tri-factorization, whereas restriction requires source and target parameters to remain similar.

  • Parameter sharing: Parameter sharing directly transfers source-learner parameters, commonly freezing most neural-network layers and fine-tuning only the final layers for the target task.This strategy is especially common in network-based approaches.
  • Matrix-factorization-based sharing: MTrick uses matrix tri-factorization to identify stable knowledge linking document classes and word-cluster concepts across source and target domains.It decomposes document-to-word matrices into document-to-cluster, connection, and cluster-to-word matrices, with the connection matrix serving as the shared parameter.
  • Matrix-factorization-based sharing: TriTL extends MTrick by separating concepts into domain-independent, transferable domain-specific, and nontransferable domain-specific types.Its optimization is solved iteratively, with domain-independent and transferable-specific components initialized from PLSA clustering and nontransferable-specific components randomly initialized.
  • Parameter restriction: Parameter restriction requires source and target model parameters to be similar rather than enforcing that the models share specific parameters.Category-learning methods such as SMKL and MMKL use weighting parameters to control transfer from one or multiple pre-obtained decision functions.

5.3 Model Ensemble Strategy

Model ensemble addresses multi-source transfer when directly combining domains is ineffective because their distributions differ. The section describes ensemble strategies that construct and combine weak classifiers using global or locally adaptive weighting.

  • Model Ensemble Strategy: Model ensemble combines multiple weak classifiers to produce final predictions when source-domain distributions differ.Directly combining data or models from multiple product domains into one domain may not succeed because the domain distributions differ.
  • Model Ensemble Strategy: TaskTrAdaBoost constructs candidate classifiers per source domain, then selects and ensembles suitable classifiers using a revised AdaBoost procedure.A threshold is used to select suitable classifiers and reduce overfitting during candidate construction.
  • Model Ensemble Strategy: LWE assigns each learner an adaptive local weight for each target instance rather than learning one global weight per learner.Learners may come from different source domains or different learning algorithms applied to one source domain.
  • Model Ensemble Strategy: LWE weights learners by similarity between source-learner and target-instance local structures, measured by the percentage of common neighbors in their graphs.The graphs connect instances receiving the same predicted class and represent target instances through clustering.
  • Model Ensemble Strategy: ENCHOR focuses on constructing weak learners by selecting anchors and generating anchor-based representations.Each instance’s feature vector is multiplied by a coefficient measuring its distance from the anchor, and the operations for different anchors can be parallelized.

5.4 Deep Learning Technique

Deep transfer-learning methods are organized into reconstruction-based and discrepancy-based approaches, with models adapting representations, distributions, or predictions across domains. Representative techniques include TLDA, DAN, residual and joint adaptation networks, Deep CORAL, and the multi-source MFSAN framework.

  • Nonadversarial deep transfer learning: TLDA uses parameter-sharing autoencoders for source and target domains to minimize reconstruction error, distribution difference, model complexity, and regression error.It can predict directly from the encoder or use the autoencoder as a feature extractor before training a target classifier.
  • Nonadversarial deep transfer learning: Deep discrepancy-based adaptation emerged because shallow architectures often struggle to learn high-performing domain-independent representations.Deep models instead adapt distributions within neural-network representations using discrepancy measures.
  • Nonadversarial deep transfer learning: DAN combines domain-specialized fully connected networks with layer-wise distribution adaptation, classification-error minimization, and MK-MMD to measure source-target differences.Its architecture extracts features through five convolutional layers before feeding them into source- and target-specific three-layer fully connected networks.
  • Nonadversarial deep transfer learning: Later deep adaptation work adds entropy-based target regularization, generalizes architectures and discrepancy metrics, and includes residual transfer networks, JAN, and Deep CORAL.JAN adapts joint distributions across multiple layers, while Deep CORAL adds a CORAL loss for feature adaptation.
  • Multi-source transfer learning: MFSAN addresses multi-source transfer learning with a common extractor, domain-specific extractors and classifiers, and objectives for classification error, distribution adaptation, and consensus regularization.Target features are processed by all domain-specific extractors, producing multiple classification predictions whose discrepancy is regularized.

6 APPLICATION

Transfer learning is applied across medical imaging, bioinformatics, transportation, and recommender systems to address limited labels, distribution differences, sparse associations, changing conditions, and insufficient personalized data. The surveyed approaches transfer pretrained models, latent features, instances, or cross-domain knowledge, with reported benefits depending on the target-data regime.

  • Medical imaging: Medical imaging uses transfer learning because specialized images are expensive and difficult to label sufficiently, including diagnosis, detection, and osteoarthritis assessment.Reported approaches fine-tune pretrained deep networks such as AlexNet for Alzheimer’s disease detection and other computer-aided detection tasks.
  • Bioinformatics: Bioinformatics applications address distribution differences in biological sequences, sparse gene–phenotype associations, and protein–protein interaction prediction by transferring organism, association, or network knowledge.Collective matrix factorization can transfer linkage knowledge by sharing a factor matrix across source and target PPI networks.
  • Transportation: Transportation applications transfer information across weather and lighting conditions for traffic scenes and historical driver data for modeling newly involved drivers.Driver-model adaptation can help the target driver when few target-domain data are available, but its superiority is not obvious with very small or very large target-instance counts.
  • Recommender systems: Recommender systems use instance-based and feature-based transfer learning to reduce dependence on large training datasets by importing source-domain data or latent feature information.Coordinate System Transfer constructs user- and item-side coordinate systems from source data and constrains the target domain with them; it significantly outperforms average filling and latent factorization baselines at all data sparsity levels.

7 EXPERIMENT

The experiments compare representative transfer learning models across object recognition, text classification, and sentiment classification using Office-31, Reuters-21578, and Amazon Reviews. Results show that model effectiveness depends on the dataset, task, domain relationship, and suitability of parameter settings.

  • Datasets and tasks: The experiments cover classification tasks in object recognition, text classification, and sentiment classification using Office-31, Reuters-21578, and Amazon Reviews.Office-31 contains three visual domains, Reuters-21578 supplies text-classification tasks, and Amazon Reviews contains four product-review domains.
  • Experimental setup: Over twenty algorithms are evaluated across three datasets, with eight on Office-31, fourteen on Reuters-21578, and eleven on Amazon Reviews.Accuracy is the evaluation metric; source instances are labeled, target instances are generally unlabeled except for TrAdaBoost, and each algorithm is run three times with average results reported.
  • Amazon Reviews: About seventy percent average accuracy is achieved by TCA, mSLDA, and CD-PLSA across the Amazon Reviews tasks.HIDC, SCL, SFA, and MTrick are also identified among the better-performing algorithms, representing feature reduction, encoding, alignment, and selection strategies.
  • Reuters-21578: Most algorithms perform relatively well on Orgs vs People but poorly on People vs Places, indicating a larger discrepancy between People and Places.TrAdaBoost performs relatively well because it uses target-domain labels, while HIDC, SFA, and MTrick show relatively consistent performance across the three tasks.
  • Limitations: Default or recommended parameter settings may be unsuitable for selected datasets, contributing to poor performance; GFK averages about sixty-two percent accuracy when transferred to text classification.GFK was originally designed for object recognition but is directly applied to text classification in the experiment.

8 CONCLUSION AND FUTURE DIRECTION

The survey unifies data- and model-based perspectives on transfer learning, introduces applications, and evaluates representative models in object recognition and text categorization. Future work should extend transfer learning to broader applications and more complex scenarios, including cross-company source data.

  • Conclusion: The survey systematizes transfer-learning mechanisms and strategies through unified definitions and symbols from both data and model perspectives.It introduces objectives and strategies from the data perspective and mechanisms and strategies from the model perspective.
  • Conclusion: Experiments evaluate representative transfer-learning models in object recognition and text categorization, with model comparisons reported.The experiments cover two mainstream application areas.
  • Future Direction: Future research should apply transfer-learning techniques to a wider range of applications and address knowledge transfer in more complex scenarios.The survey identifies broader application and complex-scenario transfer as available directions.
  • Future Direction: Cross-company source-domain data creates a real-world challenge for transferring knowledge while protecting user interests.The passage gives this as an example of a more complex transfer-learning scenario.
Loading 1911.02685v3…