PaperlayerView exact revision on arXiv

Source-linked AI summary

Transferability in Deep Learning: A Survey

Junguang Jiang, Yang Shu, Jianmin Wang, Mingsheng Long

arXiv:2201.05867v1cs.LG

TL;DR

Deep learning’s reliance on large-scale data motivates transferability: reusing knowledge from prior tasks to make learning more data-efficient. This survey unifies transferability across pre-training, adaptation, and evaluation, and provides a benchmark library for fair assessment. It also highlights challenges including catastrophic forgetting, negative transfer, and limitations in some evaluation methods.

  • Problem

    Deep learning often requires large amounts of labeled data, motivating methods that transfer knowledge to unseen tasks and domains for greater data efficiency.

  • Method

    The survey synthesizes principles and methods across pre-training, task adaptation, domain adaptation, and evaluation throughout the deep learning lifecycle.

  • Results

    The survey provides a unified perspective on transferability and implements TLlib, an open-source library for fair evaluation and reproduction of transfer-learning methods.

  • Takeaways & Limitations

    Transferability should be investigated across the complete lifecycle of acquiring, preserving, applying, and evaluating reusable knowledge.

  • Takeaways & Limitations

    The survey does not cover all transferability methods and notes that some evaluation bounds remain loose or computationally costly.

Abstract

from arXiv · show

The success of deep learning algorithms generally depends on large-scale data, while humans appear to have inherent ability of knowledge transfer, by recognizing and applying relevant knowledge from previous learning experiences when encountering and solving unseen tasks. Such an ability to acquire and reuse knowledge is known as transferability in deep learning. It has formed the long-term quest towards making deep learning as data-efficient as human learning, and has been motivating fruitful design of more powerful deep learning algorithms. We present this survey to connect different isolated areas in deep learning with their relation to transferability, and to provide a unified and complete view to investigating transferability through the whole lifecycle of deep learning. The survey elaborates the fundamental goals and challenges in parallel with the core principles and methods, covering recent cornerstones in deep architectures, pre-training, task adaptation and domain adaptation. This highlights unanswered questions on the appropriate objectives for learning transferable knowledge and for adapting the knowledge to new tasks and domains, avoiding catastrophic forgetting and negative transfer. Finally, we implement a benchmark and an open-source library, enabling a fair evaluation of deep learning methods in terms of transferability.

1. Introduction

Transferability enables deep models to reuse knowledge from upstream tasks and domains for data-efficient learning on new tasks. The survey unifies pre-training, adaptation, and evaluation across this lifecycle.

  • Transferability is the ability to extract reusable representations from source tasks and adapt them to improve related target-task learning.
  • Deep learning applications commonly pre-train models on large-scale labeled or unlabeled upstream data, then adapt them to downstream tasks or domains.
  • Pre-training seeks generic transferability across many downstream tasks, whereas adaptation targets specific tasks or related domains.
  • The survey connects isolated transferability-related areas across the full lifecycle while emphasizing core principles and methods rather than exhaustive coverage.
  • Its evaluation component reviews methods on large-scale datasets and provides TLlib as an open-source library for fair evaluation and reproduction.

2. Pre-Training

Deep learning’s dependence on abundant labeled data limits applications in data-scarce settings. Pre-training addresses this challenge by learning transferable representations or models from large-scale upstream tasks.

  • Large amounts of labeled data are needed for deep learning methods to generalize well, limiting use in domains and tasks with scarce data or annotations.
  • Pre-training obtains transferable representations or models from large-scale upstream tasks to improve downstream performance.

2.1 Pre-Training Model

Pre-training model architecture shapes transferability through capacity and inductive bias. Learned inductive bias from large-scale pre-training can support strong downstream generalization with limited labeled data.

  • Deeper models can capture more knowledge from large-scale pre-training, but increasing depth also makes optimization more difficult.
  • Downstream performance is highly correlated with pre-training accuracy, making model capacity and architecture critical factors in transferability.
  • Classic architectures impose designed inductive biases that improve data efficiency but can limit model expressiveness and transferability.
  • Pre-training supplies learned inductive bias, and larger pre-trained models may constrain the effective hypothesis space during fine-tuning.
  • As pre-training data increases, learned inductive bias can outperform manually designed inductive bias in transferability.

2.2 Supervised Pre-Training

Supervised pre-training learns transferable representations from large-scale labeled data for downstream reuse, while related methods target faster adaptation, domain transfer, and robustness to distribution changes.

  • 2.2 Supervised Pre-Training: Large-scale labeled pre-training learns reusable feature representations that are adapted to downstream tasks, usually discarding the task-specific head.ImageNet classification is a common computer-vision pre-training task.
  • 2.2 Supervised Pre-Training: Web-scale weakly supervised and semi-supervised pre-training improve transfer performance over training with only clean labeled data.These approaches address the labeling cost that limits the size of annotated datasets.
  • 2.2 Supervised Pre-Training: More pre-training data does not necessarily improve transferability when datasets become extremely large, motivating importance weighting toward target-relevant examples.The passage distinguishes data quantity from data quality as influences on transferability.
  • 2.2 Supervised Pre-Training: Standard supervised pre-training can be limited by its dependence on sufficient labeled data and vulnerability to adversarial examples.The survey presents meta-learning and causal learning as alternative approaches targeting efficient adaptation and distributional robustness.
  • 2.2.1 Meta-Learning: Meta-learning uses meta-training to learn knowledge that helps a model adapt quickly during meta-testing on new tasks.Its bi-level objective updates the model on task training data and evaluates the updated model on a task test set.
  • 2.2.2 Causal Learning: Causal and invariant-learning methods seek representations whose mechanisms remain stable across environments, supporting out-of-distribution generalization when distributions change.The transferability of invariant learning depends on whether invariance across training environments extends to all environments.

2.3 Unsupervised Pre-Training

Unsupervised pre-training learns transferable representations from unlabeled data through generative, autoregressive, autoencoding, and contrastive objectives. These objectives support transfer across tasks, domains, modalities, and supervision regimes.

  • Unsupervised pre-training uses very large unlabeled datasets to acquire generally transferable knowledge without expensive manual annotation.
  • Generative Learning: Generative pre-training reconstructs original inputs from perturbed inputs, encouraging representations to capture intrinsic and transferable explanatory factors.
  • Generative Learning: Masked language modeling predicts randomly masked tokens from unmasked context, overcoming the unidirectional limitation of autoregressive language modeling.
  • Generative Learning: 175B-parameter GPT-3, pretrained on over 500GB of text, achieves strong few-shot transfer, while multilingual BERT enables cross-lingual transfer across 104 languages.
  • Contrastive Learning: Contrastive pre-training learns by bringing representations of two views of the same instance together and separating representations of different instances.
  • Contrastive Learning: CLIP learns a shared image-text embedding from large-scale Internet data and enables zero-shot transfer competitive with task-specific supervised models.
  • Contrastive Learning: Contrastive pre-training matches supervised pre-training on classification and can outperform it on object detection and semantic segmentation by modeling objects more holistically.

2.4 Remarks

The survey emphasizes that pre-training transferability depends on task relationships, data choices, and the design of unsupervised objectives. It identifies unresolved trade-offs and limitations spanning scalability, labeling, adaptation cost, augmentation, and task alignment.

  • As supervised pre-training accuracy increases, downstream performance can saturate or even conflict with pre-training performance when task gaps are large.
  • Table 2 compares pre-training methods by modality scalability, task scalability, data efficiency, and labeling cost.
  • Unsupervised pre-training avoids manual labeling but can require many fine-tuning steps, while contrastive learning depends on augmentations that are difficult to design across modalities.
  • Meta-learning supports fast adaptation but is mainly evaluated on related tasks and small datasets, leaving its large-scale transferability unclear.
  • The field lacks solid analysis of how unsupervised pre-training bridges task shifts and what mechanisms enable its transferability.

3. Adaptation

Most applications still require adaptation because target tasks differ from pre-training tasks, and domain adaptation can help when target labels are scarce.

  • Task adaptation remains necessary for most applications because target tasks usually differ from the pre-training task.
  • Domain adaptation can use labeled data from a related source domain to improve performance on a target domain when target-task labels are insufficient.

3.1 Task Adaptation

Task adaptation reuses pre-trained knowledge for downstream tasks while addressing catastrophic forgetting, negative transfer, parameter inefficiency, and limited labeled data. Methods span fine-tuning, domain-adaptive and regularized tuning, residual and prompt-based adaptation, with trade-offs across performance, efficiency, and data requirements.

  • Catastrophic Forgetting: Catastrophic forgetting occurs when adaptation erodes knowledge from pre-training, with dataset shift identified as an important source.Domain-adaptive tuning first adapts on related or target-domain data before task fine-tuning, while regularization constrains deviation from the pre-trained model.
  • Negative Transfer: Negative transfer arises when upstream and downstream tasks are weakly related, domains differ substantially, target data are abundant, or adaptation algorithms trade positive transfer against harmful transfer.The survey reports that stronger positive transfer can coincide with more severe negative transfer.
  • Parameter Efficiency: 3.6% tunable parameters allow Adapter Tuning to match fully fine-tuned BERT on GLUE while extending pre-trained models with much lower storage cost.Adapters insert residual modules into frozen layers, preserving a near-identity path at initialization.
  • Data Efficiency: Hundreds or thousands of labeled samples may still be needed for strong fine-tuning, motivating cross-task transfer, labeled source domains, and task reformulation.Few-shot and zero-shot adaptation address settings where labels are expensive or unavailable.
  • Data Efficiency: Cosine-distance classifiers make feature transfer a strong few-shot baseline by classifying examples through distances to class prototypes without updating representations.The method uses each class’s mean labeled feature as its prototype.
  • Remarks: Fine-tuning performs best with enough downstream labels, whereas prompt learning needs fewer labels but remains limited beyond NLP; residual, difference, and prompt methods improve parameter efficiency.Negative transfer remains unresolved across these adaptation strategies.

3.2 Domain Adaptation

Domain adaptation addresses performance degradation caused by distribution shifts when transferring a model from labeled source data to an unlabeled target domain. Its methods use theoretical bounds, statistics matching, adversarial objectives, classifier discrepancy, semantic consistency, and consistency regularization to reduce source–target mismatch.

  • Unlabeled target data can still cause performance degradation under distribution shift, motivating domain adaptation to reduce the source–target discrepancy.
  • Theory: Domain-adaptation theory bounds target risk using source risk and distribution distances, often assuming the ideal joint error is sufficiently small.
  • Theory: H∆H-Divergence estimates source–target discrepancy from unlabeled samples through domain discrimination, while Disparity Discrepancy provides a tighter hypothesis-induced bound and extends to multiclass margin loss.
  • Statistics Matching: Statistics-matching methods align distributions using measures such as MMD, covariance, higher-order moments, or optimal transport, but MMD can ignore geometry and alter feature scale.
  • Deep Domain Adaptation: Adversarial and classifier-based methods address domain mismatch through gradient-reversal training or classifier discrepancy, while translation methods must preserve semantics and consistency regularization remains modality-limited.

4. Evaluation

The survey evaluates transferability across architectures, pre-training, task adaptation, and domain adaptation using large-scale datasets, unified tooling, and benchmarks. Its results show that transferability depends strongly on architectures, pre-training strategies, task relatedness, and dataset scale.

  • 4. Evaluation: The evaluation framework combines large-scale, diverse datasets with TLlib to support unified, quantitative, fair, and reproducible comparisons.The listed resources cover cross-task and cross-domain transferability, while TLlib implements representative adaptation algorithms in a common codebase.
  • 4.3 Benchmarks: The benchmark evaluates pre-training methods with adaptation procedures and data augmentations held constant for fair comparison.Hyperparameters are selected using target validation performance.
  • 4.3 Benchmarks: Architectures and pre-training strategies greatly influence both cross-task and cross-domain transferability.The survey reports these effects through benchmarks on language, image recognition, ImageNet-Sketch, and ImageNet-R.
  • 4.3.2 Task Adaptation: Task-adaptation gains remain limited on average across many datasets and largely depend on relatedness between target and pre-training tasks.Regularization can outperform vanilla fine-tuning on individual datasets, but those gains do not generalize broadly across the benchmark.
  • 4.3.3 Domain Adaptation: Many domain-adaptation methods that perform well on small datasets do not perform well on large-scale DomainNet and ImageNet evaluations.The survey therefore calls for greater attention to cross-domain transferability on large-scale datasets.

5. Conclusion

The survey investigates how to acquire and apply transferability across the whole deep-learning lifecycle. It connects pre-training, task adaptation, and domain adaptation in a unified perspective.

  • 5. Conclusion: The survey unifies how deep-learning systems acquire and apply transferability across pre-training, task adaptation, and domain adaptation.It examines architecture, pre-training task, training strategy, preservation and use of transferable knowledge, and bridging domain gaps.
  • 5. Conclusion: The survey aims to focus community attention on the fundamental role of transferability in deep learning.This expectation follows its unified treatment of previously isolated areas.
Loading 2201.05867v1…