PaperlayerView exact revision on arXiv

Source-linked AI summary

A Review of Deep Transfer Learning and Recent Advancements

Mohammadreza Iman, Khaled Rasheed, Hamid R. Arabnia

arXiv:2201.09679v2cs.LGcs.AIcs.CV

TL;DR

Deep learning is constrained by extensive labeled-data requirements and costly training. This review synthesizes DTL definitions, taxonomies, methods, applications, experiments, and limitations; its discussed approaches include model expansion that preserves prior knowledge and supports continual learning.

  • Problem

    Deep learning requires extensive labeled data and expensive training, motivating methods that reduce both constraints.

  • Method

    The paper reviews DTL definitions, taxonomy, recent applied studies, common model-based methods, experimental analyses, limitations, and proposed solutions.

  • Results

    Vertical expansion increases a pretrained model’s learning capacity while keeping previously obtained knowledge intact, leaving the final model valid on source data.

  • Takeaways & Limitations

    DTL can support continual learning through transfer processes while preserving validity on previous training sources.

  • Takeaways & Limitations

    Model-based DTL approaches face catastrophic forgetting and overly biased pretrained models.

Abstract

from arXiv · show

Deep learning has been the answer to many machine learning problems during the past two decades. However, it comes with two major constraints: dependency on extensive labeled data and training costs. Transfer learning in deep learning, known as Deep Transfer Learning (DTL), attempts to reduce such dependency and costs by reusing an obtained knowledge from a source data/task in training on a target data/task. Most applied DTL techniques are network/model-based approaches. These methods reduce the dependency of deep learning models on extensive training data and drastically decrease training costs. As a result, researchers detected Covid-19 infection on chest X-Rays with high accuracy at the beginning of the pandemic with minimal data using DTL techniques. Also, the training cost reduction makes DTL viable on edge devices with limited resources. Like any new advancement, DTL methods have their own limitations, and a successful transfer depends on some adjustments for different scenarios. In this paper, we review the definition and taxonomy of deep transfer learning and well-known methods. Then we investigate the DTL approaches by reviewing recent applied DTL techniques in the past five years. Further, we review some experimental analyses of DTLs to learn the best practice for applying DTL in different scenarios. Moreover, the limitations of DTLs (catastrophic forgetting dilemma and overly biased pre-trained models) are discussed, along with possible solutions and research trends.

1 Introduction

Deep learning addresses challenging nonlinear applications but requires extensive labeled data and costly training. Deep transfer learning is reviewed as an approach to these constraints, alongside its taxonomy, applications, evaluations, limitations, and solutions.

  • Deep learning supports applications across image processing, natural language processing, numerical analysis, prediction, and voice recognition but requires extensive labeled data and costly training.
  • Transfer learning in deep learning can reduce training time, cost, and dependence on large datasets, including for medical images and edge devices.
  • The paper reviews DTL definitions and taxonomy, recent practical studies, experimental evaluations, limitations, and possible remedies.

2 Deep Learning

Deep learning uses layered neural networks to model nonlinear datasets and has rapidly expanded across practical applications. Training commonly adjusts network weights through backpropagation and optimization, with CNNs prominent in visual tasks.

  • Deep learning is a machine-learning subcategory that handles nonlinear datasets through layered nodes with activation functions and associated weights.
  • Deep neural networks are typically trained by adjusting weights through backpropagation and optimization algorithms.
  • CNN layers have improved visual deep-learning models since 2011 and are used in most current deep-learning systems.

3 Deep Transfer Learning (DTL)

Deep transfer learning reuses knowledge from a source task and dataset to reduce target-task learning costs, including when labeled target data are scarce. Its source and target data may differ in distribution, distinguishing it from related learning paradigms.

  • Deep transfer learning reuses knowledge from another task and dataset to reduce learning costs when large labeled datasets are difficult to obtain.
  • Initializing a network with weights from a trained model can improve training performance over random initialization, even with a distant source dataset.
  • Unlike semi-supervised learning, DTL permits source and target datasets with different distributions, whereas semi-supervised learning uses one dataset with unlabeled target examples.

4 From Transfer Learning to Deep Transfer Learning, Taxonomy

DTL can be categorized by data homogeneity, labeling, and applied approach, with network-based methods most commonly used. These methods generally adapt pretrained models through freezing, fine-tuning, or added layers, while progressive learning seeks to limit forgetting.

  • Label setting: DTL can be categorized as transductive, inductive, or unsupervised according to which source and target labels are available.
  • Applied approaches: DTL approaches are categorized as instance-based, feature-based, parameter-based or network-based, and relational-based or adversarial-based.
  • Applied approaches: Network-based approaches are the most applied DTL techniques because they adapt models between source and target data, including distant domains.
  • Network-based methods: Network-based DTL commonly combines pretraining, freezing, fine-tuning, and adding new layers to adapt a pretrained model to target data.
  • Progressive learning: Progressive learning freezes the pretrained model and adds layers for new tasks, helping prevent catastrophic forgetting compared with fine-tuning.
  • Network-based methods: A commonly effective DTL strategy freezes earlier and middle layers while fine-tuning later layers, with progressive learning adding layers near the model’s end.

5 Review of Recent Advancements in DTL

The review surveys recent practical DTL studies and identifies finetuning as the most common approach, while noting its susceptibility to catastrophic forgetting. It also covers freezing CNN layers and selected publications.

  • 21 of 38 selected studies used finetuning from a related trained model, making it the most common DTL method.The selection used a systematic literature review of reproducible practical studies published during the past five years.
  • Finetuning can reduce training costs and extensive target-data requirements but remains prone to catastrophic forgetting.
  • Freezing CNN layers and finetuning lateral fully connected layers is identified as the second popular DTL approach.CNN layers extract features, while fully connected layers are adapted to the target classification task.
  • Table 1 lists selected recent DTL publications, while Fig. 2 presents common DTL approaches.

6 Experimental Analyzations of Deep Transfer Learning

The reviewed experiments show that transfer learning generally outperforms random initialization, with gains shaped by source–target domain similarity, dataset size, and target-data availability. Additional analyses link transfer success to feature reuse, faster convergence, initialization, and layer criticality.

  • Feature reuse plays a key role: IMAGENET pre-training produced the largest improvement on the visually similar real domain, while distant targets still gained.Pre-trained models also converged faster than randomly initialized models.
  • Pre-trained models retain a stronger prior, making adaptation harder for ambiguous samples and causing initialization to strongly affect feature similarity.Pre-trained models were correct on more straightforward samples, whereas random initialization sometimes handled samples that pre-training marked incorrect.
  • Finetuning a pre-trained network kept models in the same loss-landscape basin, unlike repeated random initialization, which reached different basins.
  • Module criticality increased toward the output, with fully connected layers becoming critical in pre-trained models.The pattern is consistent with earlier layers learning more general features and later layers learning target-specialized features.
  • The experiments varied source–target image domains, task types, and dataset sizes across more than 1200 transfer-learning experiments.The evaluation covered 20 datasets, seven image domains, and four task types.
  • Transfer learning outperformed training from scratch in all experiments, and 85% of target tasks had a source surpassing ILSVRC’12 pre-training.The strongest gains occurred within the same image domain, with transfer effects stronger when the target training set was small.

7 Discussion

DTL model-based approaches face catastrophic forgetting and overly biased pre-trained models, motivating methods that expand or selectively adapt models while preserving prior knowledge.

  • 7 Discussion: Model-based DTL is constrained by catastrophic forgetting and overly biased pre-trained models.These constraints affect whether transferred knowledge remains useful and whether models can learn target-specific features.
  • 7 Discussion: Finetuning can drastically change weights, wiping out source knowledge and limiting successful transfer to tightly related source and target data.Adding a limited number of source samples to target training data is described as a way to reduce forgetting.
  • 7 Discussion: Freezing earlier CNN layers reduces forgetting but can bias the model toward source features, while lateral layers may still forget knowledge.The technique is most successful when source and target data and tasks are related.
  • 7 Discussion: Progressive learning adds layers to a frozen model, balancing forgetting and bias for related task transfers but failing on distant source and target data.Frozen earlier layers cannot learn new features, whereas new lateral layers help adjust to the target task.
  • 7 Discussion: Vertical expansion adds nodes to frozen layers, increasing learning capacity while preserving prior knowledge and keeping the final model valid on source data.The approach is presented as enabling successful transfer and opening the door to deep continual learning.

8 Conclusion

The paper reviews DTL definitions, taxonomies, applied studies, common model-based methods, experiments, limitations, and possible solutions.

  • 8 Conclusion: The review covers DTL definitions, taxonomies, more than thirty applied studies, three common model-based methods, and two experimental studies.It also discusses catastrophic forgetting, overly biased pre-trained models, and possible solutions.
  • 8 Conclusion: Finetuning is the most common transfer technique because publicly available pre-trained models can be adapted simply.The reviewed methods are reported to work across various machine learning problems.
  • 8 Conclusion: The summarized experiments clarify details of successful DTL approaches across different scenarios.The paper presents these findings alongside method limitations and possible solutions.
Loading 2201.09679v2…