PaperlayerView exact revision on arXiv

Source-linked AI summary

Experience Replay for Continual Learning

David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy P. Lillicrap, Greg Wayne

arXiv:1811.11682v2cs.LGcs.AIstat.ML

TL;DR

Continual reinforcement learning must acquire sequentially presented tasks without losing prior skills, often without explicit task boundaries. The paper proposes CLEAR, which combines fresh on-policy learning with replay-based off-policy learning and behavioral cloning. CLEAR learns new tasks quickly while substantially reducing catastrophic forgetting across Atari and DMLab, with constrained replay memory remaining nearly as effective as unbounded storage.

  • Problem

    Continual reinforcement learning must learn sequential tasks while preserving prior knowledge, even when task boundaries are unknown and new experience is costly to gather.

  • Method

    CLEAR combines on-policy learning from novel experiences with off-policy learning and behavioral cloning from replayed past experience, without relying on task identities.

  • Results

    CLEAR performs better than state-of-the-art Elastic Weight Consolidation and Progress & Compress, almost eliminating catastrophic forgetting while retaining rapid adaptation to new tasks.

  • Takeaways & Limitations

    Replay can be a simple and powerful approach for continual reinforcement learning, and CLEAR remains effective when replay memory is severely constrained.

  • Takeaways & Limitations

    When storing a memory buffer is truly prohibitive, parameter-protection methods such as Progress & Compress may be more suitable.

Abstract

from arXiv · show

Continual learning is the problem of learning new tasks or knowledge while protecting old knowledge and ideally generalizing from old experience to learn new tasks faster. Neural networks trained by stochastic gradient descent often degrade on old tasks when trained successively on new tasks with different data distributions. This phenomenon, referred to as catastrophic forgetting, is considered a major hurdle to learning with non-stationary data or sequences of new tasks, and prevents networks from continually accumulating knowledge and skills. We examine this issue in the context of reinforcement learning, in a setting where an agent is exposed to tasks in a sequence. Unlike most other work, we do not provide an explicit indication to the model of task boundaries, which is the most general circumstance for a learning agent exposed to continuous experience. While various methods to counteract catastrophic forgetting have recently been proposed, we explore a straightforward, general, and seemingly overlooked solution - that of using experience replay buffers for all past events - with a mixture of on- and off-policy learning, leveraging behavioral cloning. We show that this strategy can still learn new tasks quickly yet can substantially reduce catastrophic forgetting in both Atari and DMLab domains, even matching the performance of methods that require task identities. When buffer storage is constrained, we confirm that a simple mechanism for randomly discarding data allows a limited size buffer to perform almost as well as an unbounded one.

1 Introduction

Continual learning requires balancing rapid adaptation to new experience with preservation of old skills, especially when tasks arrive sequentially without reliable boundaries. The paper proposes CLEAR, which combines replay with on- and off-policy learning to reduce forgetting while retaining plasticity.

  • Continual learning must preserve old performance while rapidly acquiring new skills, creating a stability-plasticity dilemma.
  • Catastrophic forgetting arises when sequential training on new tasks causes previously acquired knowledge to deteriorate.
  • Unknown or continuously deforming task boundaries make simultaneous multi-task training infeasible in many reinforcement-learning settings.
  • CLEAR mixes on-policy learning from novel experience with off-policy replay learning, adding behavioral cloning for stability.The approach uses replay buffers for past events without requiring explicit task information.
  • CLEAR performs better than Elastic Weight Consolidation and Progress & Compress, almost eliminating catastrophic forgetting.
  • CLEAR is much simpler than prior forgetting-reduction methods and can be combined with other approaches.

2 Related work

Prior continual-learning work emphasized parameter protection, while replay in reinforcement learning was mainly developed for single-task data efficiency. CLEAR adapts replay to continual RL by combining off-policy learning with behavioral cloning to address policy mismatch.

  • Supervised learning can reorder and replay fixed datasets, but streaming reinforcement learning cannot simply reorder incoming data.
  • Existing forgetting-mitigation methods primarily protect parameters inferred for earlier tasks, as in Elastic Weight Consolidation and Progressive Networks.
  • Experience replay in reinforcement learning has largely targeted data-efficient learning on single tasks rather than continual learning.
  • CLEAR uses behavioral cloning because historical actions may come from an outdated policy, making off-policy replay especially important in reinforcement learning.

3 The CLEAR Method

CLEAR trains an actor-critic network on novel and replayed experiences using V-Trace, applying shared losses to both and behavioral-cloning losses only to replay. The cloning terms constrain policy and value outputs toward the network’s past self to reduce drift on replayed tasks.

  • CLEAR feeds a learner a mixture of novel and replayed experiences through distributed actor-critic training with V-Trace.V-Trace uses truncated importance weights to correct off-policy distribution shifts.
  • V-Trace defines a target for the current value estimate using experience generated by a possibly different policy and truncated importance-sampling corrections.The notation distinguishes the current policy πθ, behavior policy µ, hidden state hs, and target vs.
  • The policy-gradient, value, and entropy losses are applied to both new and replay experiences, typically using a 50-50 mixture.Performance does not appear very sensitive to the novel-to-replay ratio.
  • Replay experiences additionally receive behavioral-cloning losses that match the current policy and value function to historical outputs.The policy term penalizes KL divergence, while the value term penalizes the L2 difference between historical and present value functions.
  • The behavioral-cloning objective aims to prevent network outputs on replayed tasks from drifting while learning new tasks.The policy-divergence direction KL[µ||πθ] ensures the present policy remains nonzero wherever the historical policy is nonzero.

4 Results

Across DMLab and Atari experiments, sequential training causes catastrophic forgetting, while CLEAR preserves performance on prior tasks without sacrificing rapid learning of new ones. CLEAR approaches separate and simultaneous-training performance, remains effective with constrained replay buffers, and compares favorably with prior methods.

  • 4.1 Catastrophic forgetting vs. interference: Sequential training causes task performance to decay immediately when training switches to another task, whereas separate and simultaneous training show little difference in DMLab.The results distinguish catastrophic forgetting from task interference, which is minimal in these DMLab tasks.
  • 4.1 Catastrophic forgetting vs. interference: CLEAR effectively eliminates forgetting on three cyclically trained DMLab tasks while preserving overall training performance and producing little dropoff at task switches.Without behavioral cloning, replay still reduces forgetting, but the effect is weaker.
  • 4.1 Catastrophic forgetting vs. interference: CLEAR attains cumulative performance similar to networks trained on tasks separately and simultaneously, effectively eliminating catastrophic forgetting.Separate and simultaneous training serve as no-forgetting upper-bound scenarios for sequential continual learning.
  • 4.3 Plasticity: CLEAR's probe-task performance remains largely independent of buffer contents and preceding-task identities, but 100% replay degrades performance as the buffer fills.Mixing new and replay experience supports rapid learning of new tasks while retaining prior-task performance.
  • 4.3 Plasticity: 100% replay can increase performance on each task, but both off-policy replay and behavioral cloning contribute to CLEAR's success.Behavioral cloning stabilizes replayed-task outputs, while off-policy learning can improve performance even without on-policy data.
  • 4.5 Limited-size buffers: All tested replay buffers perform well across 900 million environmental frames, although reducing capacity to 5 million slightly decreases robustness to catastrophic forgetting.The compared capacities were 450 million, 50 million, and 5 million experiences.
  • 4.6 Comparison to P&C and EWC: CLEAR obtains slightly better performance than P&C and significantly better performance than EWC despite being simpler and agnostic to task boundaries.The on-policy baseline still rapidly reacquires prior performance after re-exposure, affecting cumulative comparisons on some tasks.

5 Discussion

CLEAR combines behavioral cloning with replay-based off-policy learning to stabilize historical behavior while retaining adaptation to new tasks. The authors frame its applicability as regime-dependent and identify avenues for combining it with other methods.

  • Behavioral cloning keeps the policy distribution close to historical data, lowering the variance of importance-weighted gradient estimates.The loss regularizes the network without preventing eventual convergence.
  • CLEAR’s applicability depends on the regime: parameter-protection methods may suit settings where storing a memory buffer is prohibitive.Task identities or clear boundaries may also reduce memory or computation demands.
  • Changing the action space during training can make behavioral cloning and other forgetting-prevention strategies deleterious to performance.Such scenarios may require algorithms that selectively forget skills as well as protect them.
  • The authors suggest combining CLEAR with weight-consolidation methods and investigating alternative off-policy algorithms such as Retrace.They note that V-Trace is effective for small shifts between present and past policy distributions.
  • CLEAR uses on-policy learning for fresh experience and off-policy learning with behavioral cloning for replay experience.This combination supports rapid adaptation while maintaining and modestly improving performance on past tasks.

A.1 Distributed setup

The distributed setup uses actors that generate and replay unrolls for a learner, with replay storage distributed across actors. Reservoir sampling maintains a uniform sample when buffers reach capacity.

  • Distributed setup: The setup uses multiple CPU actors and a single learner whose weight updates propagate asynchronously to the actors.
  • Actor: Actors generate unrolls, insert them into buffers, and sample replay unrolls for a learner-facing queue.Reservoir sampling is used after the buffer reaches its maximum capacity.
  • Learner: Each learner batch contains pairs of new and replay unrolls, then selects either source according to buffer utilization.No actor contributes more than one example to the batch, reducing batch variance.
  • Network: DMLab uses a shallower network without the additional textual-input LSTM, while Atari copies the Progress & Compress network and hyperparameters.
  • Replay buffer: The replay buffer stores environment inputs, network logits, value outputs, actions, and rewards required by V-Trace.Storage is split equally across actors; a one-million-item buffer across 100 actors gives each actor capacity for 10,000 items.
  • Replay buffer: The default buffer cap is half the number of training environment frames, and reservoir sampling preserves a uniformly random sample of prior unrolls.

A.4 Training

Training uses V-Trace with task-specific DMLab and Atari hyperparameters, while selected experiments add behavioral-cloning losses. CLEAR’s hyperparameters were not fully optimized.

  • V-Trace training uses behavioral-cloning losses with weights 0.01 for policy cloning and 0.005 for value cloning in some experiments.
  • The policy-gradient, value, and entropy losses use weights 1, 0.5, and ≈0.005, respectively, following prior settings.
  • CLEAR’s hyperparameters were not subjected to significant full optimization.

A.5 Evaluation

Networks are evaluated throughout training on every task, rather than only the task currently used for training. Separate testing-actor pools asynchronously track the learner for each designated task.

  • Evaluation covers all tasks during training, not merely the task currently being trained.
  • Each task has a separate pool of testing actors that asynchronously updates its weights to match the learner.
  • Testing actors do not use replay buffers or provide training examples, and they remain on their designated task regardless of the training task.

A.6 Experiments

Experiments switch all actors between tasks after designated learning episodes, while the learner tracks total episodes and receives new-task and replay experiences. Results average three independent runs and report standard-deviation error bars.

  • Actors switch tasks simultaneously after a specified number of learning episodes.The learner monitors total episodes, and subsequent examples include new-task and replay experiences.
  • Each experiment was run independently three times, with figures plotting mean performance and standard-deviation error bars.

B Figures replotted according to cumulative sum

The replotted figures use mean cumulative reward to compare models, while smoothing obscures individual forgetting periods. They show sequential training’s forgetting without CLEAR and compare CLEAR across replay mixtures, buffer sizes, and task-aware baselines.

  • Mean cumulative reward averages rewards through time t, making model performance easier to compare but smoothing away individual forgetting periods.The final cumulative rewards are tabulated at the end of training in Figures 3 and 7.
  • Sequential training without CLEAR shows lower cumulative performance than separate or simultaneous training because of catastrophic forgetting.Thicker line segments mark periods when the network gains new experience on a task.
  • Applying CLEAR to sequential tasks achieves almost the same results as simultaneous training, while CLEAR without behavioral cloning also performs decently.
  • An equal 50-50 mixture of new and replay experience appears to trade off stability and plasticity, whereas 100% replay reduces forgetting but lowers overall performance.
  • Reduced-size replay buffers still allow CLEAR to achieve essentially the same performance.
  • CLEAR achieves comparable or better performance than P&C and EWC, despite those methods requiring task-boundary information.
Loading 1811.11682v2…