PaperlayerView exact revision on arXiv

Source-linked AI summary

A Tutorial on Meta-Reinforcement Learning

Jacob Beck, Risto Vuorio, Evan Zheran Liu, Zheng Xiong, Luisa Zintgraf, Chelsea Finn, Shimon Whiteson

arXiv:2301.08028v4cs.LG

TL;DR

Deep RL is limited by poor data efficiency and policy generality, motivating meta-RL as a way to learn better RL algorithms. This survey organizes meta-RL settings, methods, applications, and open problems, highlighting sample-efficient adaptation alongside its training-cost and generalization trade-offs.

  • Problem

    Deep RL often has poor data efficiency and limited policy generality, restricting its broader adoption and motivating better RL algorithms.

  • Method

    The survey defines and categorizes meta-RL by task distributions and per-task learning budgets, then reviews algorithms, applications, and open problems.

  • Results

    The survey finds that most research focuses on few-shot multi-task meta-RL, while also identifying many-shot settings and promising robotics applications.

  • Takeaways & Limitations

    Meta-RL is a promising route to sample-efficient adaptation, particularly when frequent deployment adaptation justifies substantial upfront meta-training.

  • Takeaways & Limitations

    Meta-RL increases sample complexity during meta-training and may reduce transferability and interpretability at test time.

Abstract

from arXiv · show

While deep reinforcement learning (RL) has fueled multiple high-profile successes in machine learning, it is held back from more widespread adoption by its often poor data efficiency and the limited generality of the policies it produces. A promising approach for alleviating these limitations is to cast the development of better RL algorithms as a machine learning problem itself in a process called meta-RL. Meta-RL is most commonly studied in a problem setting where, given a distribution of tasks, the goal is to learn a policy that is capable of adapting to any new task from the task distribution with as little data as possible. In this survey, we describe the meta-RL problem setting in detail as well as its major variations. We discuss how, at a high level, meta-RL research can be clustered based on the presence of a task distribution and the learning budget available for each individual task. Using these clusters, we then survey meta-RL algorithms and applications. We conclude by presenting the open problems on the path to making meta-RL part of the standard toolbox for a deep RL practitioner.

Introduction

Meta-RL frames the design of RL algorithms as a meta-learning problem to improve sample efficiency, while trading higher training cost and potentially weaker generalization for faster adaptation at test time.

  • Motivation: Meta-RL learns RL algorithms, or components of them, through machine learning to produce more sample-efficient reinforcement learning.It is a special case of meta-learning in which the learned algorithm is an RL algorithm.
  • Motivation: Deep RL remains highly sample inefficient, limiting its real-world applications despite successes in games, navigation, and robotics.
  • Motivation: Meta-RL may improve sample efficiency or provide solutions to previously intractable RL problems.
  • Trade-offs: Meta-learning trades improved test-time sample efficiency for greater training data requirements and reduced generality on unseen data.The learned algorithm is fitted to meta-training data, which may limit its ability to generalize.
  • Applications: Meta-RL is especially useful when efficient adaptation is frequent enough that the upfront cost of meta-training is relatively small.Safety-critical domains are one motivating setting because data collection can be necessary while novel-behavior exploration is costly or dangerous.

228 Introduction

The survey provides an entry point to meta-RL by organizing its problem settings, algorithms, applications, and open research questions around adaptation speed, task distributions, and learning budgets.

  • Survey scope: The survey covers machine-learning research on meta-RL from major conferences and workshops, primarily from 2017 to 2022, while aiming for a holistic rather than exhaustive overview.It excludes meta-RL research in fields such as neuroscience and also discusses selected work outside the main search scope.
  • Survey contributions: The survey introduces meta-RL, reviews its algorithms and applications, and discusses open problems including broader generalization, optimization, and lower meta-training costs.Its structure includes problem settings, example algorithms, applications such as robotics, and open research areas.
  • Few-shot meta-RL: Few-shot meta-RL learns an RL algorithm that adapts to a task within a handful of episodes, typically using a task distribution.The survey illustrates this setting with navigation to initially unknown goal positions.
  • Many-shot meta-RL: Many-shot meta-RL targets general-purpose RL algorithms and may train across tasks or alongside standard RL on a single task.
  • Survey contributions: Representative papers are collected in section-specific summary tables to provide high-level summaries of the cited meta-RL research.

Background

Meta-RL learns an RL algorithm that maps task-interaction data to policy parameters, using an inner loop for adaptation and an outer loop for meta-training. Its formal setting evaluates adapted policies across task distributions, with task inference represented through history or beliefs.

  • Definition: Meta-RL learns an RL algorithm f that maps collected data to policy parameters rather than directly designing the policy.The learned algorithm is the inner loop; machine learning of that algorithm is the outer loop.
  • Inner and outer loops: The inner loop adapts policy parameters from meta-trajectory data, while the outer loop updates meta-parameters using trajectories from multiple MDPs.The inner loop may use all data collected so far within an MDP.
  • Meta-RL interaction: A trial or lifetime concatenates data from multiple episodes on one task into a meta-trajectory used for adaptation.The task horizon H specifies the maximum number of episodes used for learning.
  • Parameterization: The inner loop f_θ outputs base-policy parameters ϕ=f_θ(D), often adapting only task-specific parameters and sometimes updating them at every MDP step.These adapted parameters are also called task parameters.
  • Objective: Meta-RL performance measures returns from policies produced during trials on tasks sampled from p(M), optionally excluding an initial exploration period defined by shot K.The objective can discount rewards across episodes in the meta-trajectory.
  • POMDP formalization: Meta-RL can be formulated as a POMDP, yielding history-dependent black-box methods or belief-dependent task-inference methods.A task posterior b=p(M|τ) is a sufficient statistic when the hidden state is the task identity.
  • Example algorithms: MAML differentiates through the inner-loop update, whereas RL2 uses an RNN hidden state to carry task information across episodes.MAML can generalize beyond the task distribution under specific conditions, while RL2 directly approximates the Bayes-optimal policy but may generalize poorly outside its training distribution.

Many-Shot Meta-RL

Many-shot meta-RL concerns settings where adaptation may require many episodes, with categories organized by task-distribution structure and learning horizon. The survey distinguishes multi-task transfer from single-task learning and highlights fast exploration as a central meta-RL challenge.

  • Problem categories: Meta-RL settings are organized by whether the task distribution contains multiple tasks and whether the task horizon is short or long.These dimensions produce four clusters, three of which yield practical algorithms.
  • Many-shot multi-task: Many-shot multi-task meta-RL allows an inner loop to use thousands of episodes or more while exploiting similarities among related tasks.The goal remains to make adaptation as data-efficient as possible.
  • Single-task settings: Single-task meta-learning investigates whether efficiency can improve without access to a distribution of related tasks.This setting resembles standard RL, where agents often train on one complex task over many optimization steps.
  • Single-task settings: The few-shot single-task setting lacks known research because a short lifetime may not provide enough time to learn a data-efficient adaptation procedure.The limitation concerns learning the adaptation procedure and producing a policy within only a few episodes.
  • Related fields: Meta-RL differs from meta-supervised learning because adapted policies collect additional task data, creating a data-collection problem during adaptation.Meta-RL tasks are defined by MDPs rather than fixed datasets.
  • Exploration: Fast exploration is required when agents must adapt quickly, making exploration a distinctive challenge in meta-RL.The survey treats fast exploration and its challenges as a central theme of few-shot meta-RL.
  • Related fields: Meta-RL differs from multi-task RL because the agent must infer the encountered MDP rather than receive a ground-truth task representation.This makes multi-task RL easier in settings where the task representation is known.

Few-shot Meta-RL

Few-shot meta-RL learns inner-loop RL algorithms that adapt to related tasks within a few timesteps or episodes. The survey organizes methods by inner-loop parameterization and examines exploration, supervision, model-based choices, and adaptation trade-offs.

  • Few-shot meta-RL requires agents to adapt quickly to new but related tasks using only a few timesteps or episodes.
  • Method categories: The survey categorizes most methods into parameterized policy gradient, black box, and task inference approaches according to their inner-loop parameterization.
  • Adapted policy parameters: Meta-RL methods additionally differ in which policy parameters the inner-loop adapts and which remain outer-loop meta-parameters.
  • Exploration: Few-shot adaptation depends on efficient exploration, because the agent must collect informative data during its limited adaptation shots.
  • Supervision and modeling: The survey compares methods by supervision assumptions and by whether they explicitly model environment dynamics and rewards.
  • Parameterized policy gradients: Parameterized policy gradients can generalize adaptation structure, but their high-variance gradients may require many episodes and perform poorly in sparse-reward few-shot settings.
  • Parameterized policy gradients: Ignoring sampling-correction terms produces biased meta-gradient estimates because earlier policies influence later collected data.
  • Parameterized policy gradients: Parameterized policy gradients may eventually converge on out-of-distribution tasks, while practical convergence can fail in environments with sparse rewards.

3.2 Black Box Methods

Black box meta-RL represents the inner-loop as a sequence model that learns through activations, often producing a context vector or task-specific policy parameters. Its architectures include recurrent networks, attention, and biologically inspired updates, with trade-offs in task diversity, computation, and generalization.

  • Black box methods represent the inner-loop as a sequence model whose activations perform learning without directly updating its parameters.
  • Adapted policy parameters: For task distributions requiring little adaptation, black box methods commonly produce a context vector from history and condition the base policy on it.
  • Adapted policy parameters: When tasks require substantially different behaviors, sharing base-policy weights across tasks may impede adaptation, motivating direct generation or modulation of policy parameters.
  • Architectures: A recurrent network can generate all policy weights and biases as a hypernetwork, or output a context vector on which the policy is conditioned.
  • Architectures: Black box inner-loops also use biologically inspired Hebbian updates, convolution combined with attention, attention over recurrent states, and self-attention.
  • Attention: Attention combines value vectors using weights computed from query-key similarity, with single-query and multi-query forms expressed by softmax-weighted matrix products.
  • Attention: Attention over recurrent states uses the current hidden state as a query and projected prior hidden states as keys and values.
  • Trade-offs: Attention may aid generalization to novel tasks and complex planning, but its computation and memory generally scale as O(t^2), unlike recurrent networks’ O(1) per timestep.

3.3 Task Inference Methods

Task inference methods adapt by identifying the hidden task or task-relevant representation, often using supervised or multi-task objectives. They trade broader generalization for potentially faster adaptation when the task distribution is inferable.

  • Task inference: Task inference methods identify the MDP to which the agent must adapt, treating inner-loop learning as task identification.The inferred task can be represented as a distribution that narrows as the agent collects data.
  • Task inference: When the task is known, multi-task RL maps that task directly to a policy; meta-RL instead infers the task before using such a mapping.Task inference therefore moves part of the meta-learning problem toward the easier multi-task setting.
  • Privileged information: Privileged-information methods train a predictor of the task representation from collected data and use the prediction as the policy’s adapted task parameters.The supervised objective requires the task representation to be available during meta-training.
  • Multi-task training: Multi-task training learns an encoding gθ(cM) that retains information relevant for solving tasks, rather than passing potentially uninformative or irrelevant task representations directly.For circle navigation, the encoding can reduce (xgoal, ygoal, xobject, yobject) to (xgoal, ygoal).
  • Additional advantages: Task inference methods can model rewards and dynamics to sample imagined tasks and can also support exploration.These capabilities are identified as additional advantages beyond comparisons with policy-gradient and black-box methods.

3.4 Exploration and Meta-Exploration

Meta-RL exploration gathers information about an unknown task before exploitation, using task-distribution knowledge to make adaptation efficient. Methods range from implicit end-to-end exploration to posterior sampling and task-inference objectives, each with distinct trade-offs.

  • Exploration: Inner-loop exploration collects information about the current task, while meta-exploration occurs in the outer loop during standard RL training.Meta-RL additionally uses knowledge of the task distribution to target informative data collection.
  • Exploration: In few-shot adaptation, the agent uses its first K episodes to gather information before evaluation over the remaining H −K episodes.The exploration problem is to choose early actions that improve later task performance.
  • End-to-end methods: End-to-end methods learn exploration implicitly by maximizing the meta-RL objective, but sophisticated exploration can be extremely sample inefficient.They face a circular dependency: exploitation needs prior exploration, while exploration relies on exploitation to generate reward.
  • Posterior sampling: Posterior sampling maintains a task-identity distribution, samples an identity each episode, and acts as if that sampled identity were true.PEARL iteratively refines uncertainty through interaction, but uses the same task-conditioned policy for exploration and exploitation.
  • Task inference: Task-inference exploration adds intrinsic rewards for gathering information that reduces uncertainty about task dynamics, rewards, or identity.These methods often use separate policies for exploration and exploitation when exploration episodes are freely available.

3.5 Bayes-adaptive Optimality

The Bayes-adaptive MDP formalizes meta-RL exploration by augmenting environment states with beliefs over unknown dynamics and rewards. Its Bayes-optimal policy conditions exploration and exploitation on current uncertainty, although exact planning is generally intractable.

  • BAMDP formulation: A BAMDP represents uncertainty about an unknown MDP and evaluates expected rewards and transitions under the current belief.The belief is updated from observed states and rewards generated by the underlying MDP.
  • Bayes-optimality: The Bayes-optimal policy explicitly conditions on current uncertainty to decide when and what to explore or exploit.This policy maximizes returns in the BAMDP and therefore in the corresponding unknown MDP setting.
  • BAMDP formulation: BAMDP hyperstates s+_t = (s_t, b_t) combine the environment state with the belief, making the history-dependent problem Markov.The hyperstate is a sufficient statistic for the agent’s history in this formulation.
  • Bayes-optimality: As H increases, the agent is incentivized to explore more because it has more time to benefit from discovering higher-reward solutions.The BAMDP objective corresponds exactly to the standard meta-RL objective when K = 0.
  • Approximate solutions: Exact Bayes-optimal planning is generally intractable because hyperstates contain beliefs over transition dynamics and reward functions.Practical methods therefore learn approximate Bayes-optimal policies, including variational belief approximations.

3.5. Bayes-adaptive Optimality

Meta-RL spans supervision settings ranging from reward-free meta-training to reward-free meta-testing, alongside methods for task inference and exploration. The section also contrasts few-shot exploration, which separates information gathering from exploitation, with zero-shot settings that may interleave them.

  • Exploration: Black-box meta-RL can struggle with temporally extended exploration because end-to-end optimization must learn information gathering and exploitation together.Learning one behavior can be difficult without already having learned the other, and successful end-to-end training may require curricula, distillation, and many meta-training samples.
  • Exploration: Few-shot methods use free episodes to reduce task uncertainty before exploitation, whereas zero-shot methods may interleave exploration and exploitation.The BAMDP objective can represent free exploration by setting initial rewards to zero.
  • Supervision settings: Meta-RL supervision varies from no rewards during meta-training to no rewards during meta-testing, with the standard setting providing rewards in both phases.Meta-imitation learning is treated as a related but separate problem.
  • Unsupervised meta-RL: DIAYN can create a distribution over reward functions by learning latent variables and maximizing mutual information between states and those variables.A separate MAML agent is then trained over the resulting latent reward distribution.
  • Supervision settings: Reward-free meta-testing methods remove rewards from the inner loop and instead rely on reward correlates, manually designed features, or unsupervised learning rules.These settings assume that task identity can be inferred without reward information; one example varies only task dynamics.

3.7 Model-based Meta-RL

Model-based meta-RL explicitly learns MDP dynamics and reward models, then uses them for planning or policy training on model-generated data. This provides an alternative to the predominantly model-free algorithms discussed earlier.

  • Model-based meta-RL: Model-based meta-RL learns an explicit model of MDP dynamics and reward.
  • Model-based meta-RL: The learned model can define a policy through planning or generate data for policy training.
  • Model-based meta-RL: Most previously discussed meta-RL algorithms are model-free because they do not learn an explicit dynamics-and-reward model.

3.7. Model-based Meta-RL

Model-based meta-RL learns environment models that can support planning, imagined data, or policy adaptation. Its benefits depend on model accuracy and planning feasibility, while theory connects meta-RL performance to task-distribution structure and learned representations.

  • Model-based methods: Model-based meta-RL includes gradient-based, recurrent, transition-history, and variational approaches for adapting environment models.These approaches parallel representation choices in model-free meta-RL.
  • Model uses: Learned models can drive model-predictive control, generate additional training data, or provide adapted parameters to a policy.Some methods use models as sub-procedures within standard RL algorithms.
  • Trade-offs: Model-based meta-RL can be highly sample-efficient with accurate models and can often learn off-policy, but may require extra components and achieve lower asymptotic performance.The trade-off is especially relevant for longer-horizon tasks where an off-the-shelf planner is insufficient.
  • Trade-offs: Model-based meta-RL can supplement scarce task distributions with imagined tasks and may simplify exploration when planning handles complicated exploration strategies.This can make planning preferable to learning exploration directly in some inner loops.
  • Theory: Meta-RL theory uses Bayesian analysis to study learning algorithms, task distributions, Bayes-optimal policies, regret, and generalization.The survey also incorporates results from supervised meta-learning and multitask representation learning.
  • Theory: Representation-learning results show that related tasks can improve downstream sample efficiency, while PAC bounds explain weaker guarantees for broader task distributions.Some bounds depend exponentially on the task distribution’s degrees of freedom.

Many-shot Meta-RL

Many-shot meta-RL learns general-purpose RL algorithms for long-horizon adaptation, in either multi-task or single-task settings. Its methods use surrogate objectives and learned components to address long-horizon optimization, while remaining constrained by high computational costs and meta-training trade-offs.

  • Many-shot settings: Many-shot meta-RL learns general-purpose RL algorithms applied for thousands of updates rather than a handful, targeting practice-like use beyond narrow task distributions.The setting includes multi-task training across broader task distributions and single-task meta-learning alongside standard RL training.
  • Multi-task many-shot meta-RL: Multi-task many-shot meta-RL retains the few-shot objective but uses much longer trials and broader task distributions, potentially with differing action and observation spaces.Few-shot methods often peak after tens of episodes, whereas many-shot methods may require tens of thousands or more episodes to converge.
  • Optimization challenges: Many-shot optimization is computationally demanding: inner-loops can require tens of thousands to tens of millions of interactions, and outer-loop convergence can require up to ten billion environment steps.High inner-loop costs restrict which learning architectures can be efficiently trained; optimizing an RNN over millions of timesteps is beyond the current state of the art.
  • Many-shot methods: Learned auxiliary tasks, intrinsic rewards, and online tuning of inner-loop hyperparameters can improve sample efficiency and performance over base or handcrafted alternatives.Some auxiliary-task approaches achieved state-of-the-art model-free RL performance on Atari at the time of publication.
  • Many-shot methods: Learned objective functions show promising generalization across settings, including transfer from simple gridworld tasks to more complex Atari tasks and eventual outperformance of IMPALA.Other approaches can generalize to completely unseen environments, indicating benefits beyond adaptation within a fixed narrow task distribution.
  • Outer-loop algorithms: Because direct optimization over long horizons causes vanishing or exploding gradients and infeasible memory requirements, many-shot methods commonly maximize a surrogate objective after only a few inner-loop updates.This surrogate is biased relative to performance at the end of the trial, although it can still produce practical gains.

Applications

Meta-RL is applied where agents must adapt quickly across changing tasks, including robotics, control, education, and multi-agent settings. Robotics commonly uses simulation and task-inference or model-based methods to reduce data demands, while practical deployment remains constrained by sim-to-real transfer and meta-training cost.

  • Application domains: Meta-RL supports rapid adaptation in traffic signal control, building energy control, automatic code grading, robotics, and multi-agent RL.The surveyed applications span control, education, manipulation, locomotion, and interactions with other agents.
  • Robotics: Robotics uses meta-RL for manipulation and locomotion across varying robot hardware, objects, loads, terrains, and environmental conditions.Training separate policies on physical robots can require millions of steps and may be time-consuming or dangerous.
  • Robotics: Most robotic methods meta-train in simulation and then deploy in the real world through sim-to-real transfer, reducing meta-training cost.Simulation makes it easier to create tasks by changing simulator parameters.
  • Robotics: Task-inference methods condition policies on context vectors inferred from historical data, whereas model-based methods adapt learned dynamics models or use them for control.Task-inference methods mainly differ in the loss used to train task inference; model-based methods can be more sample efficient for some robotic tasks.
  • Robotics: Robotic meta-RL faces a significant sim-to-real gap, while exploration is often less challenging because platforms or terrains present themselves immediately.The survey highlights privileged simulator information and related task-inference methods as ways to mitigate transfer difficulties.
  • Multi-agent RL: In multi-agent RL, meta-learning addresses generalization to unseen opponents or teammates and non-stationarity caused by other agents.Other agents may be learned policies or humans, and their changing behavior creates a generalization problem.

Open Problems

Open problems center on evaluating and improving generalization beyond narrow task distributions, making many-shot optimization reliable, and developing broader data regimes and benchmarks. The survey also emphasizes trade-offs involving upfront data, interpretability, adaptation, and the need for specialized methods.

  • Generalization: Narrow task distributions make task inference trivial and produce inductive biases that transfer poorly to entirely new behaviors.A study cited by the survey found meta-RL no better than multi-task pre-training on complex visual Atari generalization.
  • Benchmarks: Broader benchmarks should combine clear task structures with parametric and non-parametric diversity across simulation, games, robotics, and real-world coding tasks.Examples include manipulation tasks with different goals and object interactions, procedurally generated games, and systematic program-error discovery.
  • Generalization: OOD generalization may fail when learned inductive biases are misspecified; agents must adaptively choose whether to use or adjust them.The survey contrasts PPG methods, which can in principle recover with enough data, with black-box methods that cannot guarantee this behavior.
  • Many-shot optimization: Many-shot meta-RL still lacks standard benchmarks, while single-task truncated optimization introduces biased meta-gradients and unresolved bias-variance trade-offs.Proposed approaches include bootstrapped objectives and weighted meta-gradients, but each introduces or retains sources of bias.
  • Data regimes: Offline meta-RL settings remain underexplored and often require extra assumptions or limited forms of online interaction that constrain their applicability.The survey identifies offline RL via online RL as an open direction for learning more effective offline algorithms.
  • Practical trade-offs: Meta-RL is most applicable when upfront data collection is relatively cheap or deployment adaptation is expensive and frequent.Simulators can make meta-training affordable, while safety-critical settings can make efficient deployment adaptation valuable.
  • Practical trade-offs: Meta-RL trades interpretability and transferability against sample efficiency and engineering burden, replacing some domain expertise with computation.The survey notes that learned systems may adapt efficiently while yielding less interpretable insights.
  • Method design: Specialized meta-RL methods may be unnecessary in some settings because recurrent networks, hypernetworks, POMDP methods, and general exploration methods can perform similarly.The survey presents this as a question about whether the problem setting requires specialized algorithms at all.

Conclusion

The survey organizes meta-RL around few-shot and many-shot settings, reviews their algorithms and applications, and identifies generalization and data efficiency as central challenges. It concludes that broader task distributions and diverse benchmarks are needed to advance real-world use.

  • Survey scope: Most existing research studies few-shot multi-task meta-RL, learning algorithms that rapidly adapt to new tasks from a known task distribution.The survey categorizes common methods as parameterized policy gradient, black-box, and task-inference approaches.
  • Survey scope: Many-shot meta-RL addresses both generalization to broader task distributions and faster learning on a single task, often by augmenting standard RL algorithms with learned components.The survey finds these seemingly opposite problems are addressed with surprisingly similar methods.
  • Future direction: Broader and more diverse task distributions are needed for training and testing meta-RL algorithms and for enabling new applications.The conclusion links this need to the sample-efficiency barrier in real-world controller learning and sim-to-real transfer.

List of Venues Surveyed

The survey primarily covers meta-RL research presented at major machine learning, robotics, and multi-agent conferences and workshops from 2017 to 2022.

  • Machine learning venues: The surveyed venues include NeurIPS and ICLR workshops on meta-learning, plus ICLR, NeurIPS, and ICML.These venues cover core machine learning and meta-learning research.
  • Multi-agent and robotics venues: The survey also covers AAMAS, AAAI, CoRL, RSS, and IROS.These venues represent multi-agent systems, artificial intelligence, and robotics research.
Loading 2301.08028v4…