PaperlayerView exact revision on arXiv

Source-linked AI summary

Transfer Learning in Deep Reinforcement Learning: A Survey

Zhuangdi Zhu, Kaixiang Lin, Anil K. Jain, Jiayu Zhou

arXiv:2009.07888v7cs.LGcs.AIstat.ML

TL;DR

Reinforcement learning faces costly and potentially unsafe exploration, motivating transfer of external expertise to target tasks. This survey systematically categorizes transfer learning in deep reinforcement learning, analyzes methods and evaluation, and identifies future challenges. It covers recent DRL transfer developments while noting unresolved limitations such as imperfect demonstrations, scalability, and robustness across task settings.

  • Problem

    Reinforcement learning can require prohibitive or unsafe interaction samples, while transfer learning in RL remains less mature and more complex than in supervised learning.

  • Method

    The survey categorizes DRL transfer approaches by transferred knowledge, RL backbones, goals, methodologies, applications, and evaluation perspectives.

  • Results

    The survey provides a comprehensive account of recent DRL transfer methods and their evaluation, including policy transfer, representation transfer, and emerging knowledge-transfer directions.

  • Takeaways & Limitations

    The framework highlights future work on reasoning over diverse knowledge forms and efficient, principled knowledge use in DRL.

  • Takeaways & Limitations

    The survey notes that many policy-transfer evaluations emphasize multi-teacher settings, leaving generalization and robustness dependent on measures such as performance sensitivity and fixed-epoch performance.

Abstract

from arXiv · show

Reinforcement learning is a learning paradigm for solving sequential decision-making problems. Recent years have witnessed remarkable progress in reinforcement learning upon the fast development of deep neural networks. Along with the promising prospects of reinforcement learning in numerous domains such as robotics and game-playing, transfer learning has arisen to tackle various challenges faced by reinforcement learning, by transferring knowledge from external expertise to facilitate the efficiency and effectiveness of the learning process. In this survey, we systematically investigate the recent progress of transfer learning approaches in the context of deep reinforcement learning. Specifically, we provide a framework for categorizing the state-of-the-art transfer learning approaches, under which we analyze their goals, methodologies, compatible reinforcement learning backbones, and practical applications. We also draw connections between transfer learning and other relevant topics from the reinforcement learning perspective and explore their potential challenges that await future research progress.

1 INTRODUCTION

Deep reinforcement learning extends reinforcement learning with deep neural networks for complex domains, but RL still faces costly exploration and transfer challenges. This survey reviews recent transfer-learning approaches for DRL, organizes them systematically, and identifies future directions.

  • Motivation: Deep reinforcement learning combines reinforcement learning with deep neural networks to address complex domains beyond traditional tabular settings.Applications include robotics control, game playing, health informatics, electricity networks, and intelligent transportation systems.
  • Motivation: Partial observability, sparse feedback, and large state-action spaces can make collecting sufficient interaction samples prohibitively expensive or unsafe.The paper specifically highlights automatic driving and health informatics as safety-sensitive domains.
  • Motivation: Transfer learning uses external expertise to benefit learning in a target domain and has become a crucial topic for reinforcement learning.The survey frames transfer learning as a response to practical RL difficulties.
  • Survey scope: The survey investigates recent transfer-learning research in RL, whose complexity arises because knowledge must transfer within a Markov Decision Process.Expert knowledge can take different forms and therefore require different transfer mechanisms.
  • Contributions: The survey categorizes DRL transfer methods, analyzes their differences and evaluation metrics, and discusses future directions for reasoning over diverse knowledge forms.It also covers newer schemes such as representation disentanglement and policy distillation.
  • Organization: The paper organizes the survey around RL preliminaries, transfer-learning definitions, categorization frameworks, evaluation metrics, and knowledge formats such as reward shaping.The organization also includes practical applications and related topics.

2 DEEP REINFORCEMENT LEARNING AND TRANSFER LEARNING

This section introduces RL as sequential decision-making over Markov Decision Processes, reviews model-free and model-based DRL foundations, and defines transfer learning as using source and target information to learn a target policy. It also distinguishes transfer learning from related paradigms such as continual learning.

  • RL preliminaries: A Markov Decision Process specifies initial states, state and action spaces, transition probabilities, rewards, discounting, and policy behavior.The agent interacts with the environment, receives rewards, and accumulates time-discounted returns.
  • RL preliminaries: Standard reinforcement learning aims to learn an optimal policy by maximizing expected return from environment interactions.Goal-conditional RL additionally conditions the policy and reward on a goal distribution.
  • RL methods: RL methods include model-free approaches that omit explicit environment models and model-based approaches that use learned or provided transition or cost models.Model-free foundations include Monte-Carlo, temporal-difference, and policy-gradient methods; model-based methods support planning and data generation.
  • Transfer learning: Transfer learning learns a target-domain policy using both exterior information from source domains and interior information from the target domain.Ordinary RL is recovered as the special case in which source information is empty.
  • Related topics: Continual learning sequentially transfers knowledge across unforeseen changing tasks while avoiding catastrophic forgetting, making it a more challenging specialized transfer setting.Most transfer-learning methods in the survey instead address fixed groups of tasks.

3 ANALYZING TRANSFER LEARNING

The survey analyzes transfer learning by the transferred knowledge, compatible RL framework, and transfer objective, using HalfCheetah to illustrate domain differences and knowledge forms. It also proposes metrics that assess target-agent learning and transferred-knowledge requirements.

  • Categorization: Transfer-learning approaches are organized by what knowledge is transferred, which RL frameworks support it, and whether transfer is zero-shot, few-shot, or sample-efficient.Transferred knowledge may include demonstrations, action distributions, or potential functions.
  • Domain variation: Source and target domains may differ in state space, action space, rewards, transition dynamics, initial states, or trajectory length.HalfCheetah illustrates these differences through changes in movement positions, torques, physical rules, starting conditions, and episode duration.
  • Transferred knowledge: Transferable knowledge can include demonstrated trajectories, model dynamics, teacher policies, and teacher value functions.These forms respectively provide behavior, physical rules, action probabilities, or state-action quality estimates.
  • Evaluation metrics: Transfer-learning evaluation considers jumpstart, asymptotic, accumulated, transfer-ratio, time-to-threshold, fixed-epoch, and performance-sensitivity metrics.These metrics capture initial or ultimate returns, learning-curve area, relative asymptotic performance, adaptation speed, and robustness to hyperparameters.
  • Evaluation metrics: The survey adds necessary knowledge amount and necessary knowledge quality to evaluate how much and how reliable transferred knowledge must be.Examples include the number of source tasks, expert policies, or demonstrations, and whether knowledge is near-oracle or suboptimal.
  • Evaluation metrics: Together, the metrics evaluate transfer mastery and generalization: ultimate target performance and rapid adaptation to the target domain.The framework therefore evaluates both learning outcomes and transfer requirements.

4 RELATED WORK

Earlier surveys categorized RL transfer mainly by problem settings and objectives, but did not cover the subsequent decade of research. Other surveys focus on narrower areas including sim-to-real transfer, benchmarks, and multi-task RL.

  • Prior surveys: Early RL transfer surveys categorized scenarios by the number of domains and differences in state-action spaces.A later survey refined these dimensions by also analyzing transfer objectives.
  • Prior surveys: The pioneering surveys did not cover research from the last decade, including newer developments enabled by deep learning.One emphasized task-mapping methods suited to tabular or mildly dimensional state-action spaces.
  • Focused surveys: Other surveys target specific intersections of RL and transfer learning, including sim-to-real robotics, transfer benchmarks, and multi-task RL.These works respectively emphasize robotics applications, evaluation resources, and multi-task learning progress.

5 TRANSFER LEARNING APPROACHES DEEP DIVE

This section organizes transfer-learning approaches by the type of knowledge transferred and analyzes each type using additional criteria and evaluation metrics.

  • 5 TRANSFER LEARNING APPROACHES DEEP DIVE: Transfer-learning approaches are grouped mainly by answering what knowledge is transferred.The survey then applies other criteria from Section 3 and summarizes applicable evaluation metrics for each approach.

5.1 Reward Shaping

Reward shaping transfers external knowledge by augmenting the target environment’s rewards, with methods evolving from static state potentials to learned dynamic state-action potentials.

  • 5.1 Reward Shaping: Reward shaping reconstructs the target reward distribution with auxiliary rewards that guide policy learning using external knowledge.The shaping function maps states, actions, and transitions to additional reward signals alongside environment rewards.
  • 5.1 Reward Shaping: Potential-based reward shaping defines shaping rewards as differences between potential functions and can preserve policy invariance without further restrictions.The potential function evaluates state quality based on expertise, and the original and transformed optimal Q-functions are related through that potential.
  • 5.1 Reward Shaping: PBA extends potential-based shaping by making the potential function evaluate action benefits from a given state.Because the next action follows the learning policy, PBA requires on-policy learning and can be sample-costly.
  • 5.1 Reward Shaping: Dynamic potential methods extend shaping to time-dependent potentials, while DPBA learns a state-action value potential in parallel with the environment value function.DPBA incorporates prior knowledge through an extra reward and provides a framework for shaping arbitrary knowledge as auxiliary rewards.
  • 5.1 Reward Shaping: The survey describes a progression from static state potentials through state-action and dynamic potentials to dynamic state-action potentials.This progression is summarized as PBRS, PBA, DPB, and DPBA in the reward-shaping comparison.

5.2 Learning from Demonstrations

Learning from demonstrations uses externally supplied behavior to improve exploration, initialization, value learning, or policy updates across multiple reinforcement-learning frameworks. Its main challenges are imperfect demonstrations and limited coverage of visited states.

  • 5.2 Learning from Demonstrations: Learning from Demonstrations uses external demonstrations for more efficient exploration, usually when source and target MDPs are identical.Some work instead learns from demonstrations generated in a different domain.
  • 5.2 Learning from Demonstrations: Demonstrations support offline pretraining or offline RL, and can initialize value functions, policies, or transition-dynamics models.Online methods use demonstrations during reinforcement learning rather than only for initialization.
  • 5.2 Learning from Demonstrations: Demonstration-based methods span policy iteration, Q-learning, and policy-gradient frameworks.Examples include DQfD and LfDS for Q-learning, and GAIL, POfD, and related methods for policy gradients.
  • 5.2 Learning from Demonstrations: GAIL learns discriminator-based rewards that encourage the current policy’s state-action occupancy distribution to match the expert’s distribution.The expert distribution is estimated from demonstrations, and the discriminator output becomes a new reward.
  • 5.2 Learning from Demonstrations: Imperfect demonstrations and covariate drift remain central challenges because limited or biased demonstrations can misguide learning and leave unseen states unsupported.Proposed responses include robust losses, selective use of demonstrations, entropy regularization, decaying demonstration guidance, and disagreement regularization.
  • 5.2 Learning from Demonstrations: Demonstrations can encourage expert-like behavior, shape value functions, or guide policy updates through auxiliary objectives.DQfD and DDPGfD use demonstration replay, LfDS shapes value-related rewards, and GAIL or POfD regularize policy learning.

5.3 Policy Transfer

Policy transfer uses pretrained policies from source domains to support learning in a target domain. The survey distinguishes policy distillation, which trains a student from teacher policies, from policy reuse, which directly incorporates source policies.

  • Policy transfer addresses many-to-one settings by leveraging one or multiple pretrained teacher policies from source domains for a target policy.
  • Policy distillation: Policy distillation trains a student by minimizing divergence between teacher and student action distributions, using either teacher-generated or student-generated trajectories.
  • Policy distillation: Student distillation empirically provides better exploration than teacher distillation, especially when teacher policies are deterministic.
  • Policy distillation: Distillation can also reshape rewards: teacher-likelihood terms encourage expert-like actions, while entropy terms encourage exploration.
  • Policy reuse: Policy reuse directly combines source policies, with reuse probabilities linked to expected performance gains on the target task.
  • Policy reuse: Generalized Policy Improvement selects actions using the highest estimated value across policies, while a shared representation supports cross-task Q-function estimation.
  • Evaluation and open questions: Policy transfer is commonly evaluated through sensitivity to teacher-policy or source-task variation and performance after fixed training epochs.
  • Evaluation and open questions: Existing approaches generally assume that one or more expert policies are always available, leaving imperfect or budget-limited teachers as open questions.

5.4 Inter-Task Mapping

Inter-task mapping transfers knowledge by learning or using correspondences between source and target domains. Mapped states, values, or transitions can directly support target learning or guide it through rewards and objectives, but methods often target moderate-dimensional tasks.

  • Inter-task mapping methods assume one-to-one correspondences between source and target domains and differ in mapping scope and downstream use.
  • Earlier approaches require provided state and action mappings, from which mappings over source and target Q-values can be derived.
  • Later approaches learn mappings over agent-specific or task-specific state representations, using encoder-decoder structures, trajectory alignment, or related methods.
  • Transition-space mapping embeds source and target transitions into a shared latent space to identify correspondences despite different transition-space dimensionalities.
  • Mapped knowledge may comprise state spaces, Q-functions, or state-action-state transition representations.
  • Mapped representations can be applied directly in the target domain or used as shaping rewards and loss objectives to guide learning.
  • Evaluation and scope: Most inter-task mapping methods address moderate-dimensional maze or tabular tasks, with evaluation including transfer time and varying numbers of target-domain initial states.

5.5 Representation Transfer

Representation transfer supports deep reinforcement learning by reusing or disentangling task-relevant representations under assumptions about task-invariant structure. The surveyed methods include progressive and modular architectures, successor representations, and universal value function approaches, but remain constrained by task and environment differences.

  • Overview: Representation-transfer methods assume that state, action, or reward spaces contain task-invariant subspaces across domains.Knowledge is transferred through universal subspaces shared by source and target tasks.
  • Reusing Representations: Progressive networks freeze prior task columns and transfer their representations through collateral connections to newly trained task columns.The architecture grows as new tasks arrive, while preserving previously learned weights.
  • Reusing Representations: Modular networks separate task-specific and agent-specific policy modules so knowledge can transfer across agents or tasks.The task module can be reused by different agents, while the agent module can be reused across tasks.
  • Disentangling Representations: Successor representations separate state features from reward distributions, enabling transfer when domains differ only in rewards.Deep extensions represent successor features with neural networks and use generalized policy improvement for multi-task transfer.
  • Disentangling Representations: Extended successor representations avoid a linear reward-function assumption by learning matrix-valued latent features whose basis functions can represent unseen tasks.This broadens task representation but comes with a looser performance lower-bound under generalized policy improvement.
  • Disentangling Representations: UVFA transfers state embeddings across goal-conditioned tasks, but matrix factorization can be time-consuming and end-to-end fine-tuning remains necessary.These costs are practical concerns in complex environments with large state spaces.
  • Summary and Discussion: Representation transfer relies on assumptions about invariant rewards, dynamics, policies, or state-based task definitions, leaving generalization across changing dynamics and state distributions as future work.The survey also notes uncertainty about how many and what kinds of subtasks are needed for a sufficiently general latent space.

6 APPLICATIONS

The survey connects transfer learning in deep reinforcement learning to applications including robotics, games, natural language processing, large-model training, health informatics, transportation, and finance. These applications use demonstrations, reward shaping, policy transfer, or human preferences to improve learning in domain-specific tasks.

  • Robotics Learning: Robotics transfer learning uses human or robot demonstrations and shares policies or episodic demonstrations across collaborative robots.A central application goal is fast and robust adaptation to unseen tasks.
  • Game Playing: Game-playing is a common test-bed, spanning grid-worlds, online strategy games, video games, and platforms such as MineCraft, Atari, and Starcraft.AlphaGo used expert demonstrations for offline pre-training before policy optimization with Monte-Carlo Tree Search.
  • Natural Language Processing: Natural-language applications use expert demonstrations, reward shaping, and policy transfer in dialogue, visual question answering, sequence generation, question answering, image captioning, and structured prediction.These applications include both language-only and vision-language tasks.
  • Large Model Training: Large-model training uses human preferences as reward signals for reinforcement-learning fine-tuning.Preference rankings are treated as shaped rewards in systems including GPT4, Sparrow, PaLM, and LaMDA.
  • Health Informatics: Health-informatics applications include medical diagnosis, health-resource scheduling, drug discovery, and drug-delivery individualization, where prior knowledge helps address limited clinical data.The survey identifies Q-learning for individualized drug delivery as one example.
  • Others: Deep reinforcement learning has also been applied to traffic signal scheduling, transportation resource allocation, portfolio management, asset allocation, and trading optimization.These examples extend transfer-related reinforcement-learning applications beyond robotics, games, and language.

7 FUTURE PERSPECTIVES

The survey identifies future directions for transfer learning in deep reinforcement learning, including learning from black-box resources, disentangling and fusing knowledge, framework-agnostic transfer, standardized evaluation, and transfer to and from large pretrained models.

  • Transfer Learning from Black-Box: Black-box transfer must address noisy or biased teacher demonstrations and pretrained function approximators by estimating their optimality.Suggested directions include inferring reasoning mechanisms, designing feedback schemes, and improving interpretability.
  • Knowledge Disentanglement and Fusion: Knowledge disentanglement can support fusion across multiple source domains with different qualities and modalities.The survey links this direction to action, state, and reward representations, as well as hierarchical reinforcement learning and prototype learning.
  • Framework-Agnostic Knowledge Transfer: Framework-agnostic transfer remains difficult because methods are often restricted to discrete-action or continuous-action reinforcement-learning settings.The survey proposes unified frameworks as a route toward standardization.
  • Evaluation and Benchmarking: Evaluation needs generalized metrics because no single metric captures transfer efficacy, alongside standardized measures of ethics, groundedness, stereotypes, and model bias.The survey emphasizes quantifying the appropriateness of transferred knowledge.
  • Knowledge Transfer to and from Pre-Trained Large Models: Transfer to and from pretrained large models raises challenges beyond black-box learning because their training processes may be inaccessible.The survey highlights efficient fine-tuning with knowledge distillation and principled prompt engineering for knowledge extraction.
Loading 2009.07888v7…