Towards Continual Reinforcement Learning: A Review and Perspectives

Khimya Khetarpal; Matthew Riemer; Irina Rish; Doina Precup

Source-linked AI summary

Towards Continual Reinforcement Learning: A Review and Perspectives

Khimya Khetarpal, Matthew Riemer, Irina Rish, Doina Precup

arXiv:2012.13490v2Dec 25, 2020cs.LGcs.AI

TL;DR

Continual RL addresses how agents can learn across changing tasks while retaining and reusing prior abilities. The paper reviews the field through unified taxonomies of non-stationarity and approaches, discusses evaluation, and identifies open challenges. It concludes that continual RL remains early-stage but has promise for increasingly realistic applications and connections with neuroscience.

Problem
Current deep RL agents often master narrow tasks, while continual RL seeks agents that adapt across changing tasks and retain previously learned abilities.
Method
The paper reviews continual RL formulations, characterizes non-stationarity by its scope and driver, surveys approach families, and discusses evaluation, neuroscience connections, and open problems.
Results
The surveyed approaches comprise explicit knowledge retention (18.5%), leveraging shared structure (40.8%), and learning to learn (40.7%).
Takeaways & Limitations
Continual RL has promise for developing incremental learners suited to increasingly realistic applications where non-stationarity matters, including healthcare, education, logistics, and robotics.
Takeaways & Limitations
Changepoint detection is generally reactive and may not exploit cross-task dependencies effectively to improve sample efficiency.

Abstract

from arXiv · show

In this article, we aim to provide a literature review of different formulations and approaches to continual reinforcement learning (RL), also known as lifelong or non-stationary RL. We begin by discussing our perspective on why RL is a natural fit for studying continual learning. We then provide a taxonomy of different continual RL formulations by mathematically characterizing two key properties of non-stationarity, namely, the scope and driver non-stationarity. This offers a unified view of various formulations. Next, we review and present a taxonomy of continual RL approaches. We go on to discuss evaluation of continual RL agents, providing an overview of benchmarks used in the literature and important metrics for understanding agent performance. Finally, we highlight open problems and challenges in bridging the gap between the current state of continual RL and findings in neuroscience. While still in its early days, the study of continual RL has the promise to develop better incremental reinforcement learners that can function in increasingly realistic applications where non-stationarity plays a vital role. These include applications such as those in the fields of healthcare, education, logistics, and robotics.

1. Introduction

Continual RL studies agents that learn and adapt across an ongoing stream of tasks, extending abilities while retaining and reusing prior knowledge. This survey clarifies the field’s scope through taxonomies of formulations, approaches, evaluation, and open problems.

Continual learning develops increasingly complex behaviors incrementally, building new capabilities on previously developed ones while adapting and generalizing to new situations.
A continual learner should learn online and incrementally, retain prior abilities, minimize catastrophic forgetting, and adapt efficiently to changes over time.
Continual RL treats sequential decision making over a stream of tasks, with each task modeled as a stationary Markov Decision Process.
The survey provides taxonomies of continual RL problem formulations and approach families, alongside evaluation metrics and benchmarks for generating and assessing non-stationarity.
The survey covers multi-task, metalearning, never-ending, non-stationary, and lifelong learning in RL, while limiting detailed coverage of transfer, representation learning, domain adaptation, and domain randomization.

2. Background

The background formalizes reinforcement learning with Markov decision processes and distinguishes two complementary views of non-stationarity. It situates continual RL among broader learning settings, including transfer, domain adaptation, and meta-learning.

Reinforcement-learning foundations: Reinforcement learning is typically formalized as a finite, discrete-time MDP with states, actions, transition probabilities, rewards, and a discount factor.The agent selects actions through a policy, transitions between states, and receives rewards from the environment.
Reinforcement-learning foundations: The agent aims to maximize expected discounted return, evaluated through state-value and action-value functions under a policy.Model-based methods use an environment model for planning, whereas many practical algorithms are model-free because the model is unknown.
Views of non-stationarity: The survey distinguishes a non-stationary function view, where environment components such as transitions, rewards, observations, or actions may depend on time.Tasks are represented as stationary MDPs, potentially forming a discrete set that can be infinite in the extreme case.
Views of non-stationarity: The partially observable view treats non-stationarity as arising from the agent’s perspective, and the survey presents it as complementary to the function view.The function view aligns more readily with supervised and bandit formulations, while the partially observable view facilitates convergence analysis.
Related learning settings: Transfer and domain adaptation involve multiple tasks, domains, or skills, but commonly assume separate policies or known task boundaries rather than continual adaptation during training.Domain adaptation adapts a skill-specific policy to a new domain, while transfer learning addresses learning multiple potentially diverse tasks with limited data.

3. RL: A Natural Fit For Studying Continual Learning

RL naturally supports continual-learning research because agents interact sequentially with environments whose data are time-correlated and non-stationary. Continuing RL objectives also expose the tension between retaining past abilities and adapting to new experiences.

Neural networks face a stability-plasticity dilemma between adapting to recent experiences and retaining past capabilities.
Continuing RL maximizes expected long-term discounted returns over the future distribution of states, rather than optimizing only the current state.
Non-stationary dynamics can make standard episodic policy-gradient and supervised SGD updates myopic and biased toward current experiences.
Episodic objectives optimize only until episode termination, whereas continuing objectives extend over the agent’s lifetime.
RL data are time-correlated and do not come from a fixed distribution, making RL useful for analyzing long-term memory in continual learning.

4. A Taxonomy of Continual RL Problem Formalisms

The survey organizes continual RL formulations from a general problem allowing every environment component to vary over time to structured assumptions that make non-stationarity learnable. It emphasizes that additional functional assumptions are needed to avoid vacuous problem statements.

4. A Taxonomy of Continual RL Problem Formalisms: The taxonomy treats the setting where all components of the RL problem may be non-stationary as the most general continual RL formulation.
4.1 General CRL Problem: The General CRL Problem includes state, action, observation, reward, transition, and observation-function components.
4.1 General CRL Problem: Each component can vary as a function f(i, t), where i is the component-specific input and t indexes time.
4.2 Common Non-stationary Functional Forms: Without structural assumptions, arbitrary non-stationarity provides no consistent learning signal and can make the problem vacuous.
4.2 Common Non-stationary Functional Forms: Lipschitz non-stationarity bounds temporal change by C|t − t′| for every input i.
4.2 Common Non-stationary Functional Forms: The Lipschitz condition also bounds the function’s first time derivative.
4.2 Common Non-stationary Functional Forms: Piecewise non-stationarity models intervals of stationary functions separated by time points t0, t1, t2, ...
4.2 Common Non-stationary Functional Forms: A fixed variation budget is another possible assumption about the form of non-stationarity.

4.3 Key Properties of Non-stationarity: Scope and Drivers

The survey characterizes continual RL non-stationarity along two dimensions: scope, which identifies changing interaction components, and driver, which captures assumptions about how those changes evolve. It then relates non-stationary MDPs, POMDPs, and active multi-agent settings.

The taxonomy categorizes non-stationarity by its scope and driver.
Scope specifies which elements of the agent-environment interaction process can change over time.
Driver specifies causal assumptions about the evolution of non-stationary environment dynamics.
A non-stationary MDP is a continual RL problem when non-stationarity affects a subset of state, action, reward, or transition components.
This POMDP perspective can support theoretical analysis because the augmented system is stationary and has defined long-term behavior.
A non-stationary MDP with changing rewards or transitions can equivalently be represented as a POMDP by incorporating time into an unobserved state variable.
Active Markov games appear non-stationary or partially observable because other agents’ actions, parameters, and update functions are unobserved.
Stationary policies of all agents yield a Nash equilibrium, whereas individually optimal non-stationary policies yield an active equilibrium.

4.5 A Unified View

The survey presents continual RL as a unified generalization of existing problem settings, organized by the scope and driver of non-stationarity.

The general CRL problem broadly captures existing formulations in the literature.
Scope and driver of non-stationarity provide two primary dimensions for characterizing CRL problem formalisms.
CRL is presented as a strict generalization of existing settings, offering a unified formulation.
HiP-MDPs are special CRL problems with non-stationarity scope α ⊆{p, r} and stationary driver β.
HM-MDPs and DP-MDPs are special CRL problems with α ⊆{p, r} and β ∈{stationary, passive}.
The taxonomy defines categories of settings rather than necessarily describing concrete settings.

4.6 Assumptions About Shared Structure

The survey treats shared structure as important for making continual RL learnable and for enabling generalization across changing environments, while organizing non-stationarity by its affected components.

Generic MDPs without assumed structure have lower-bound regret and sample complexity that preclude provable generalization to novel state-action pairs.
Real-world applications often possess shared structure in observations, states, transition dynamics, or reward dynamics.
Shared-structure assumptions include linear POMDPs, low-rank models, Block MDPs, and Factored MDPs.
Continual RL can involve non-stationarity in rewards, transitions, observations, or action spaces, though fully general settings remain rarely explored.
Reward-only, transition-only, and joint reward-transition changes are represented in existing approaches, with joint changes described as especially challenging.
Changing action spaces and observation functions remain underexplored, and allowing every environment component to vary is an open problem likely requiring additional assumptions.

4.8 Literature Review: The Drivers of Non-stationarity

The survey distinguishes passive, active, and hybrid drivers of non-stationarity according to whether the agent can influence how tasks evolve.

Non-stationarity may be drawn from a stationary distribution beyond the agent’s influence or may be influenced by the agent’s behavior.
Stationary Task Distributions: Multi-Task Learning: Multi-task learning commonly assumes tasks are sampled from an unknown but fixed distribution p(z).
Stationary Task Distributions: Multi-Task Learning: Simultaneous learning across diverse tasks can produce interfering gradients, especially when gradient magnitudes differ or loss landscapes have high curvature.
Stationary Task Distributions: Meta-Learning: Meta-learning uses meta-training and meta-testing under a stationary task distribution, with generalization depending on shared commonalities between distributions.
Stationary Task Distributions: Meta-Learning: Out-of-distribution generalization and distributional shifts are often not explicitly addressed in these settings.
Passive Non-stationarity: Passive non-stationarity models task evolution with p(z′|z), without accounting for effects of the agent’s changing policy on that distribution.
Active Non-stationarity: Active non-stationarity includes settings where agent behavior influences environmental change, such as intrinsic curiosity and learned curricula.
Active and Passive Non-stationarity: Hybrid non-stationarity combines agent influence with causal mechanisms beyond the agent’s control and is presented as representative of many real-world applications.

5. A Taxonomy of Continual RL Approaches

The survey organizes continual RL approaches into three broad, non-mutually-exclusive families and reports how the reviewed literature is distributed among them.

The three approach families are explicit knowledge retention, leveraging shared structure, and learning to learn.
The taxonomy’s categories are not mutually exclusive, although the survey examines each family separately.
Explicit knowledge retention accounts for 18.5% of the reviewed approaches.
Leveraging shared structure accounts for 40.8% of the reviewed approaches.
Learning to learn accounts for 40.7% of the reviewed approaches.
Learning to adapt is predominant across the approach subcategories reviewed.
Latent parameter storage, distillation, state abstraction, and auxiliary-task techniques constitute only a small fraction of the approaches covered.

5.1 Explicit Knowledge Retention

Explicit knowledge retention methods stabilize continual RL by preserving information from previous tasks, but they trade off storage, plasticity, transfer, or adaptation efficiency.

5.1 Explicit Knowledge Retention: Explicit knowledge retention methods address catastrophic forgetting by preserving information from prior tasks during continual learning.The reviewed strategies include task-specific storage, parameter protection, distillation, and replay.
5.1.1 Parameter Storage Based: Storing an independent model for each task requires task detection and substantial storage while limiting knowledge sharing across tasks.
5.1.1 Parameter Storage Based: Shared latent components and representations can reuse knowledge across tasks, but accumulating past representations increases input dimensionality and storage requirements.A single low-dimensional shared representation is presented as one way to address these costs.
5.1.1 Parameter Storage Based: Parameter-usage priors, superposition, and distillation protect old knowledge by reducing plasticity or representational overlap, but may limit backward transfer.
5.1.1 Parameter Storage Based: Experience replay reinforces past-distribution experiences and can improve storage efficiency through compressed recollections, yet struggles when current and past policies differ substantially.The latter difficulty arises because off-policy learning becomes difficult in that setting.

5.2 Leveraging Shared Structure

Shared-structure approaches seek retention and transfer by representing reusable abstractions, modules, skills, and temporal hierarchies across tasks, while compositional continual learning remains difficult.

5.2 Leveraging Shared Structure: Continual agents can retain and transfer knowledge by reusing structured solutions through function composition and abstract concepts across multiple timescales.
5.2 Leveraging Shared Structure: Compositional task formulations aim to enable generalization by solving new task variations through combinations of previously learned components.
5.2 Leveraging Shared Structure: Neural modules can be composed for related tasks, increasing compositional power and helping divide information to avoid negative transfer.
5.2 Leveraging Shared Structure: A central challenge is the chicken-and-egg problem: learning modules and learning how to combine them become jointly non-stationary as modules change.This challenge can arise even when the environment itself is stationary.
5.2 Leveraging Shared Structure: State abstractions map ground-MDP states to abstract states to capture common structure and potentially facilitate positive forward transfer.Related abstractions preserve reward and transition properties, value equivalence, or context-specific independencies.
5.2 Leveraging Shared Structure: Temporal abstractions such as macros, skills, and options let agents make decisions across multiple timescales rather than at every individual timestep.Options combine an initiation set, internal policy, and termination condition; SMDPs model variable durations between decisions.
5.2 Leveraging Shared Structure: Multi-task option discovery has shown theoretical and empirical potential, including sample-complexity bounds and skill composition for efficient multi-task learning.

5.3 Learning to Learn

Learning-to-learn approaches address hidden task contexts, adaptation, and exploration by using interaction histories, meta-learning, or self-modification, but key deployment assumptions remain challenging.

5.3 Learning to Learn: Meta-learning in continual RL includes learning unknown state-space components, improving adaptation, and learning exploration or curiosity-driven behavior.
5.3.1 Context Detection: Context-detection formulations represent the environment with an observable physical state and an unobserved task state whose evolution may be stationary, passive Markovian, or active Markovian.HiP-MDPs, HM-MDPs, and MOMDPs represent increasingly dynamic assumptions about task-state evolution.
5.3.1 Context Detection: Changepoint methods detect boundaries between evolving task states without an oracle, but are generally reactive and may not exploit cross-task dependencies for sample efficiency.
5.3.1 Context Detection: Bayesian RL infers beliefs about unobserved task states from interaction histories and seeks policies that act optimally relative to uncertainty about the current task.
5.3 Learning to Learn: Meta-learning aims to improve sample efficiency by modifying an agent’s optimization process based on historical learning successes and failures.
5.3.2 Learning to Adapt: Self-modifying policies are an ambitious meta-learning approach that has not yet been scaled to applications with modern deep RL networks.
5.3.2 Learning to Adapt: Meta-training and meta-testing separate fast inner-loop learning from slower outer-loop learning about how to improve that process.In MAML, the inner loop uses gradient-based learning while the outer loop computes gradients to improve inner-loop performance.

6. On Evaluation of Continual RL Agents: Benchmarks & Metrics

Evaluating continual RL remains difficult because existing benchmarks often lack the characteristics needed to study non-stationarity, forgetting, transfer, and adaptation systematically. The paper advocates configurable standardized benchmarks and broader probe-based metrics beyond accumulated reward.

Benchmark limitations: Evaluating continual RL agents remains an open question because widely used RL benchmarks may not provide sufficient continual-learning characteristics.The paper identifies the lack of well-suited environments as a roadblock to rapid progress.
Benchmark limitations: Existing environments support focused measurements, such as abstractions, continuous control, image-based deep RL, or 3D navigation, rather than comprehensive continual-learning evaluation.Different domains expose agents to specific dimensions of capability.
Benchmark limitations: Hand-engineered environment customizations can bias experiments toward the questions designers intended to study.Researchers often design tasks for particular evaluation questions, potentially producing unintended consequences.
Benchmark design: Parameterized environments can generate unseen task variants by modifying internal parameters governing transition and reward dynamics.The paper cites this as a principled way to introduce non-stationarity.
Benchmark design: Continual RL needs standardized benchmarks that support progressive training, skill discovery and composition, and a range of non-stationary settings.CausalWorld is presented as a promising robotic-manipulation benchmark meeting many desirable criteria.
Metrics: Accumulated reward alone may not reveal the full abilities of continually learning agents, motivating auxiliary probe metrics for capabilities such as transfer and forgetting.The proposed evaluation framework combines domains, metrics, controllable non-stationarity, and carefully designed experiments.

7. Looking Forward

Continual RL faces unresolved questions about its scope, objectives, task specification, agent-environment boundaries, scaling, and interaction with other learning agents. The paper looks to neuroscience and increasingly realistic settings for guidance while emphasizing that many foundational assumptions remain open.

Connections to neuroscience: Neuroscience offers potential guidance on memory, credit assignment, reward, modularity, and the balance between model-based and model-free learning.The review presents these connections as possible sources of insight for designing continual RL agents.
Connections to neuroscience: Human learning evidence supports investigating uncertainty-aware learning, hierarchical structure, modular control, and memory mechanisms in continual RL.The paper describes these as trends corroborated by neuroscience findings.
Open problems: The extremely general nature of continual RL and the lack of a concrete definition leave substantial unexplored potential and open problems.The review frames defining appropriate assumptions as a fundamental challenge.
Open problems: Continual RL still requires clearer choices about inductive biases, lifetime utility, and how tasks are specified and related.Broad MDP definitions and competing assumptions motivate specialized formulations such as HM-MDPs, MOMDPs, Block MDPs, HiP-MDPs, and Factored MDPs.
Open problems: The correct agent-environment boundary remains an important open question when other agents participate in a real-world decision maker’s learning.This issue is distinct from ordinary stationary-environment formulations.
Open problems: Scaling laws for continual RL may clarify whether increased data, compute, and hardware help address forgetting, without constituting a complete solution.The paper treats learning at scale as a potentially fruitful research direction rather than a guaranteed remedy.
Open problems: Learning alongside other adaptive agents is a key continual RL scenario, but multi-agent RL remains far from industrial application and largely in its infancy.The resulting non-stationarity is identified as central to multi-agent RL.

8. Conclusion

The paper argues that continual RL should move toward realistic settings while drawing on neuroscience, psychology, cognition, and animal learning. It presents continual RL as a still largely theoretical field with promising relevance to challenging real-world applications.

Conclusion: The review urges continual RL researchers to address open problems while moving toward increasingly realistic settings.It connects this goal to findings from the human brain, psychology, cognitive behavior, and animal learning.
Conclusion: Bridging AI and computational neuroscience is presented as having promising potential to accelerate progress in continual RL.The conclusion frames neuroscience as a foundation for future advances.
Conclusion: Continual RL is described as largely theoretical to date, with potentially promising use cases in challenging real-world applications.The cited application areas include settings where continual adaptation matters.

Appendix A. The Relationship Between Continual Reinforcement Learning and Continual Supervised Learning

The appendix casts supervised learning as a special case of reinforcement learning under simplifying assumptions about policies, states, rewards, transitions, and data. It then explains why non-stationary data make the usual SGD objective biased toward current experience and potentially prone to forgetting.

RL formulation: Supervised learning can be represented as continuing RL by using deterministic policies, decomposed states, differentiable rewards, and action-invariant transitions.The state includes inputs and optimal outputs, while reward is defined as negative loss.
Stationary learning: Under stationary incoming data, linearity of expectations connects the continuing-environment objective to the standard stochastic-gradient supervised-learning objective.The equivalence assumes an i.i.d. input-output distribution.
Non-stationary learning: With non-stationary data p(x, y∗, t), the SGD objective is biased toward current experience and omits long-term effects of parameter changes.The appendix contrasts this with the stationary case.
Non-stationary learning: When past experiences recur in future but not current data, the biased SGD objective may naturally produce catastrophic forgetting.The appendix uses this analysis to motivate RL as a framework broad enough to capture important features of continual supervised learning.