PaperlayerView exact revision on arXiv

Source-linked AI summary

Pseudo-Rehearsal: Achieving Deep Reinforcement Learning without Catastrophic Forgetting

Craig Atkinson, Brendan McCane, Lech Szymanski, Anthony Robins

arXiv:1812.02464v6cs.LGcs.AIcs.CV

TL;DR

Sequential learning can make neural networks forget earlier tasks, limiting continual learning. RePR combines a dual-memory system with generative pseudo-rehearsal to retain prior-task knowledge while learning new reinforcement tasks. The paper reports successful sequential learning of complex tasks without revisiting them or storing prior raw data, and experimental evidence of outperforming state-of-the-art EWC methods on a sequential learning task.

  • Problem

    Neural networks can forget previously learned tasks while learning new ones, and pseudo-rehearsal had been virtually unexplored in reinforcement learning despite its use in image classification.

    Source: §1 · p. 2

  • Method

    RePR uses a short-term DQN for the current task and a long-term DQN with a generative network to transfer new knowledge and rehearse representative states from previous tasks.

    Source: §3 · pp. 6–7

  • Results

    RePR sequentially learned complex reinforcement tasks without revisiting them or storing past raw data, and experimentally outperformed state-of-the-art EWC methods on a sequential learning task.

    Source: §7 · p. 23, §8 · p. 28

  • Takeaways & Limitations

    The experiments support pseudo-rehearsal as a less restrictive alternative to weight-constraint methods, allowing network layers to change as new knowledge is incorporated.

    Source: §7 · p. 23

    RePR assumes the agent knows when a task switch occurs; otherwise, a task-detection method would need to be combined with it.

    Source: §7.3 · p. 27

Abstract

from arXiv · show

Neural networks can achieve excellent results in a wide variety of applications. However, when they attempt to sequentially learn, they tend to learn the new task while catastrophically forgetting previous ones. We propose a model that overcomes catastrophic forgetting in sequential reinforcement learning by combining ideas from continual learning in both the image classification domain and the reinforcement learning domain. This model features a dual memory system which separates continual learning from reinforcement learning and a pseudo-rehearsal system that "recalls" items representative of previous tasks via a deep generative network. Our model sequentially learns Atari 2600 games without demonstrating catastrophic forgetting and continues to perform above human level on all three games. This result is achieved without: demanding additional storage requirements as the number of tasks increases, storing raw data or revisiting past tasks. In comparison, previous state-of-the-art solutions are substantially more vulnerable to forgetting on these complex deep reinforcement learning tasks.

1. Introduction

The introduction frames catastrophic forgetting as a barrier to sequential reinforcement learning and presents RePR, which combines dual memory with pseudo-rehearsal to learn new tasks while retaining prior ones. It reports successful application to complex deep reinforcement learning without storing previous raw data, and identifies separating reinforcement learning from continual learning as important.

1. Introduction

  • Pseudo-rehearsal had been widely used in image classification but was virtually unexplored in reinforcement learning, where catastrophic forgetting constrains continuous learning.Source: §1 · p. 2
  • RePR combines a freshly initialized DQN for new tasks, a long-term DQN, and generative pseudo-rehearsal to retain knowledge without storing large amounts of previous-task data.The generative network produces states representative of earlier tasks; the long-term system learns from these while receiving knowledge from the short-term system.Source: §1 · pp. 2–3
  • Deep reinforcement learning complicates pseudo-rehearsal because changing targets increase interference, while low-quality generated states can impair initial learning and later retention.Source: §1 · p. 3
  • RePR isolates reinforcement learning in the short-term system and uses supervised learning with fixed targets in the long-term system; it also transfers new knowledge using real samples.The authors contrast this with prior pseudo-rehearsal applications that use the same cross-entropy loss in both systems.Source: §1 · p. 3
  • The stated contributions include the first successful application of pseudo-rehearsal to complex deep reinforcement learning and evidence that separating reinforcement learning from continual learning facilitates new learning.Source: §1 · pp. 3–4
  • The paper reports above-state-of-the-art performance on sequential complex reinforcement tasks without storing raw data from previously learned tasks.Source: §1 · p. 3

2. Background

The background introduces deep Q-learning and explains why rehearsal-based approaches to catastrophic forgetting motivate pseudo-rehearsal. Generative replay extends the approach to large input spaces and chains of tasks.

2.1. Deep Q-Learning

  • Deep Q-learning trains a neural network to predict discounted rewards for possible actions given the current state.Its predictor and target networks are updated on different schedules, and training examples are sampled from an experience replay of prior experiences.Source: §2.1 · p. 4

2.2. Pseudo-Rehearsal

  • Rehearsal learns new items alongside previously learned ones, but can require storing and retraining on large prior datasets.Storing real data may also be infeasible in settings such as medicine because of privacy regulations.Source: §2.2 · p. 4
  • Pseudo-rehearsal avoids storing large amounts of past data by training on generated pseudo-items alongside new-task data.The original approach generates random inputs, records the original network's outputs, then rehearses those input-output pairs while learning the new dataset.Source: §2.2 · p. 5
  • For large input spaces such as images, GAN-based replay generates pseudo-items representative of prior tasks, and a single GAN can be trained to retain a chain of tasks.The generator learns to produce representative items while the discriminator distinguishes generated from real images.Source: §2.2 · pp. 5–6

3. The RePR Model

RePR separates current-task reinforcement learning from long-term retention: a short-term DQN learns the new task, while a long-term DQN consolidates it and rehearses generated states from prior tasks. Its training sequence also updates a generative network so the system can learn tasks without revisiting them or growing its memory store with the task count.

3. The RePR Model

  • RePR pairs a short-term system that learns the current task with a long-term DQN and GAN that retain knowledge of learned tasks through pseudo-rehearsal.Knowledge transfer uses the long-term DQN as student and the short-term DQN as teacher; real items teach new knowledge, while generated items rehearse prior knowledge.Source: §3 · pp. 6–7

3.1. Training Procedure

  • The training procedure has three stages—short-term DQN training, long-term DQN training, and long-term GAN training—and can repeat until the DQN or GAN lacks sufficient capacity.Source: §3.1 · p. 7

3.1.1. Training the Short-Term DQN

  • For each new task, the short-term DQN is reinitialized and trained solely on that task using the standard DQN loss.The model plays the game while simultaneously training the short-term memory system.Source: §3.1.1 · p. 7, Figure 1 · p. 8

3.1.2. Training the Long-Term DQN

  • The long-term DQN learns from current-task experience-replay examples while pseudo-rehearsing generated states to preserve outputs from the previous long-term DQN.The trade-off between learning new tasks and retaining previous ones is controlled by a scaling factor, 0 < α < 1.Source: §3.1.2 · pp. 7–9, Figure 2 · p. 9

3.1.3. Training the Long-Term GAN

  • The long-term GAN is reinitialized and trained on current-task replay states and states generated by the previous GAN, so it can represent current and earlier tasks.The previous GAN's generated state is intended to resemble a state from one of the prior tasks.Source: §3.1.3 · pp. 9–10

3.2. Requirements of a Continual Learning Agent

  • The stated requirements are sequential learning without revisiting prior tasks, substantial forgetting, memory growth with task count, or storing previous-task raw data.The paper says a single GAN can learn to generate data representative of all previous tasks without growing in size as the task count increases.Source: §3.2 · p. 10

4. Related Work

Prior approaches to catastrophic forgetting either constrain network optimization or add data representative of earlier tasks. The paper positions RePR against limits in these approaches and earlier pseudo-rehearsal work on simple reinforcement tasks.

Optimization constraints

  • Progressive networks can require substantial model growth, while EWC requires storing task-specific weights and Fisher information or past examples.Other weight-constraint variants cited in the paper had been applied only to relatively simple reinforcement tasks.Source: §4 · pp. 10–11

Rehearsal

  • Rehearsal methods require continued access to previous environments or substantial stored training data, whereas pseudo-rehearsal adds generated samples to training.Source: §4 · p. 11

Prior pseudo-rehearsal

  • Prior pseudo-rehearsal work sequentially learned two simple 2D reinforcement tasks, while RePR addresses more distinct, complex reinforcement-learning tasks with large input spaces, requiring deep convolutional networks and a dual memory system.Earlier tasks used a 64 × 3 grid and differed only in the color of objects to collect.Source: §4 · p. 12

DGDMN and RePR

  • DGDMN teaches the long-term system generated, classifier-labeled examples, while RePR uses short-term experience-replay data and separates deep Q-learning from supervised long-term learning.DGDMN uses cross-entropy for both systems; RePR uses deep Q-learning in short-term memory and mean squared error in long-term memory.Source: §4 · pp. 12–13

Progress and Compress

  • Progress and Compress transfers Atari-game learning from short-term to long-term memory using distillation and online-EWC, but the authors report no conclusive evidence that it assists sequential reinforcement learning.Its online-EWC memory requirement does not scale with task count because it stores recent weights and a discounted sum of Fisher information matrices.Source: §4 · pp. 11, 13

Pseudo-rehearsal rationale

  • Pseudo-rehearsal alters training examples rather than constraining weights, allowing internal reconfiguration; the authors suggest it may outperform weight constraints over time, but frame this as a possibility.The paper summarizes RePR as the first pseudo-rehearsal variation successfully applied to continual learning with complex reinforcement tasks.Source: §4 · pp. 13–14

5. Method

The method evaluates RePR on sequential Atari tasks, comparing conditions on an initial three-game sequence and testing an extended sequence. The reported alpha variants perform at least equivalently to the original DQN across tasks, while the extended experiment demonstrates forgetting beyond model capacity.

Experimental sequence

  • Initial experiments compare methods on Road Runner, Boxing, and James Bond, then an extended experiment adds Pong, Atlantis, and Qbert.The extended experiment tested only RePR and rehearsal methods with a single seed because of its computational cost.Source: §5 · p. 14

Extended experiment

  • Larger networks did not prevent forgetting when RePR was pushed beyond its capacity in the extended experiment.That experiment extended the sequence with Pong, Atlantis, and Qbert and used a single seed for RePR and rehearsal methods.Source: §5 · p. 14

Training procedure

  • Each task trained the short-term system for 20 million frames and the long-term system for 20m frames, while the GAN alternated generator and discriminator optimization for 200,000 steps.The first long-term DQN copied the short-term DQN weights, so conditions differed only after Boxing was introduced.Source: §5 · p. 16

Results

  • α=0.35 and α=0.75 produced very similar results, with the final agent performing at least equivalently to the original DQN on all tasks.Source: §5 · pp. 16–17

Evaluation

  • Performance evaluation used 30 episodes per task with an ε-greedy policy at ε = 0.05; conditions were generally run three times with shared seeds.Scores and standard deviations were calculated over the 30 episodes; unless otherwise stated, reported results were averaged across seeds.Source: §5 · p. 17

6. Results

Across sequential Atari tasks, RePR retained prior-task performance close to real rehearsal, and its final scores remained above human-expert levels. The experiments also show that pseudo-items can support retention despite lower quality than real items, while outcomes vary across comparison methods and training settings.

6.1. RePR Performance on CF

  • RePR performed very close to reh, with slight Road Runner degradation likely related to GAN rehearsal; similar tasks shared more important weights.The task evaluations occur after every 1 million observable training frames, with switches from Road Runner to Boxing to James Bond.Source: §6.1 · pp. 17, 19, Figure 3 · p. 18

6.2. Quality of Generated Items

  • GAN-generated images visibly differed from real game frames, and generated items scored below real items yet still effectively prevented catastrophic forgetting.The student-DQN assessment reports scores for training with real or generated items, including single games and all three games together.Source: §6.2 · pp. 19–20, Figure 4 · p. 19, Table 2 · p. 20

6.3. RePR Versus EWC

  • RePR outperformed EWC and online-EWC; EWC retained past experiences better, while online-EWC learned the new task more effectively.Source: §6.3 · p. 20
  • The EWC comparison is setting-dependent: unlike prior evaluations, tasks were not revisited, capacity did not grow, and the LTM retained Q-values rather than only policy.Under a different policy-only, shorter-retention scheme, EWC variants performed comparably to RePR, so the training scheme less comprehensively explores model capabilities.Source: §6.3 · p. 21

6.4. Further Evaluating RePR

  • PR performed worse and converged more slowly than RePR; DGDMN reached about half RePR’s performance on Boxing and James Bond.DGDMN did not struggle on Road Runner because its LTM copied the STM after the first task and learned that task with real data.Source: §6.4 · pp. 21–22
  • With rehearsal constrained to RePR-sized memory, reh-limit forgot Road Runner almost entirely and retained roughly half its Boxing performance; reh-limit-comp also forgot Road Runner.Source: §6.4 · p. 22
  • RePR achieved 80% of individually trained scores, exceeded human-expert scores (7845, 4, 407), and used orders-of-magnitude less LTM storage than reh.Reh achieved 90% of individually trained scores; the table also compares storage requirements and individual-task scores.Source: Table 3 · p. 22, §6.4 · p. 23
  • In the extended task sequence, RePR-extend underperformed the other conditions and gradually lost prior-task performance, while three other conditions retained considerable knowledge.Reh-extend-policy retained noticeably more Road Runner knowledge.Source: §6.4 · p. 23

7. Discussion

RePR’s discussion reports advantages over weight-constraint methods and rehearsal, and identifies both the dual-memory system’s role and practical limits in longer task sequences. Its scope also includes assumptions about task data, task switches, and the retained value information.

7. Discussion

  • RePR outperformed state-of-the-art EWC methods on sequential learning while allowing network weights to change as new knowledge is incorporated.The authors contrast this flexibility with weight constraints that require similar weights across tasks.Source: §7 · p. 23

7.1. Dual Memory

  • Without the dual memory system, learning new tasks converged more slowly and reached lower final performance than RePR, with greater interference between new and previous knowledge.The authors report that matching RePR’s Boxing performance without dual memory required weighting the reinforcement-learning loss much more heavily, with substantial forgetting.Source: §7.1 · p. 25
  • DGDMN performed substantially worse than RePR because generated items were not accurate enough to teach the long-term system effectively on these reinforcement tasks.Generated items that were inadequate for short-term-to-long-term transfer could still serve effectively in pseudo-rehearsal.Source: §7.1 · p. 25
  • RePR assumes real data from the most recent task remains in experience replay to teach the long-term system, then is cleared for data from the next task.The replay buffer is described as temporary and fixed in size.Source: §7.1 · pp. 25–26

7.2. Scalability

  • RePR’s memory requirements do not grow with task count and it stores no raw data from past tasks; it also outperformed rehearsal when storage limited rehearsal items.When rehearsal items were compressed, rehearsal’s long-term allocation was still approximately 0.250 GB, an order of magnitude greater than RePR’s.Source: §7 · p. 23, §7.2 · p. 26
  • In the extended sequence, policy retention showed modest forgetting, while Q-value retention showed substantial forgetting versus rehearsal, generally declining only after millions of training frames.The authors attribute the Q-value condition’s forgetting to limits in the generative network’s capacity and stability.Source: Figure 5 · p. 24, §7.2 · p. 26

7.3. Limitations and Future Research

  • Rehearsal methods become infeasible as task counts approach infinity, and using the same task order across conditions could bias the experiments.RePR also assumes the agent knows when a task switch occurs; some applications may require a task-detection method.Source: §7.3 · p. 27
  • RePR still stores current-task replay for within-task forgetting; experiments use DQNs, and whether retained policy alone suffices to resume learning without retained values remains open.The paper proposes actor-critic networks as a possible extension but leaves the policy-versus-value retention question for future work.Source: §7.3 · p. 27

8. Conclusion

The paper concludes that pseudo-rehearsal with RePR supports sequential learning across complex reinforcement tasks without growing memory with task count or revisiting or storing raw data from past tasks. The experiments concern Atari 2600 games, including Road Runner, Boxing, James Bond, Pong, Atlantis, and Qbert.

8. Conclusion

  • RePR sequentially learned complex reinforcement tasks without memory complexity growing with task count, revisiting past tasks, or storing raw data from them.The authors also report experimental support for benefits over weight-constraint methods and for the dual-memory system’s role in effective continual reinforcement learning.Source: §8 · p. 28
  • The paper reports that pseudo-rehearsal is less restrictive on network weights than weight-constraint methods, while the dual-memory system is important for effective continual reinforcement learning.Source: §8 · p. 28

Appendix A. Details on the Atari 2600 Games

  • The Atari tasks span racing and item collection, boxing, vehicle control, paddle-and-ball play, cannon defense, and navigation across a color-changing pyramid.The games described are Road Runner, Boxing, James Bond, Pong, Atlantis, and Qbert.Source: Appendix A · p. 28

Appendix B. Further Implementation Details

The implementation appendix specifies the DQN changes and architecture used in the experiments, then describes the GAN’s training losses, optimization, and network design. It also identifies the relevant architecture tables and image preprocessing choices.

Appendix B.1. DQN

  • The DQN kept the original network architecture but used uncentered TensorFlow RMSProp with global-norm gradient clipping, biases of 0.01, and weights initialized from N(0, 0.01) with outliers redrawn.The appendix points to Table B.4 for hyperparameters and Table B.5 for the architecture used in most experiments, with an exception for the extended sequence.Source: Appendix B.1 · p. 29, Table B.5 · p. 30

Appendix B.2. GAN

  • The GAN used WGAN-GP with a drift term that keeps discriminator outputs for real and generated inputs from drifting too far from zero.The discriminator and generator are updated on alternating steps using their corresponding loss functions; the appendix gives the loss definitions and parameters.Source: Appendix B.2 · p. 30
  • The GAN used Adam with a mini-batch size of 100; its discriminator uses Leaky ReLU, while its generator uses batch normalization and ReLU except for a final Tanh layer.The appendix identifies Table B.6 as the GAN architecture table and describes rescaling images to −1 to 1 and adding random noise before discriminator input.Source: Appendix B.2 · pp. 30–31, Table B.6 · p. 31

Appendix B.3. EWC

The appendix describes EWC and online-EWC retention constraints, then explains policy transfer and retention in the long-term system, including the cross-entropy adaptation used by rehearsal methods.

Appendix B.3. EWC

  • EWC adds a penalty that keeps current parameters near the previous-task parameters, weighted by Fisher-based parameter importance and scaled by λ.The Fisher information is approximated from a Gaussian posterior and replay states; its expectation uses states drawn uniformly from experience replay.Source: Appendix B.3 · pp. 31–32
  • Online-EWC stores the previous network parameters and one Fisher matrix, updated with a discount γ and min-max normalised across tasks.The normalisation is intended to prevent different reward scales from changing the relative importance of parameters.Source: Appendix B.3 · p. 32
  • Under the alternative training scheme, EWC, online-EWC and RePR all learned new tasks while retaining previously learned tasks.The appendix reports similar performance across these conditions; the long-term agent retained a policy while new tasks were learned in the long-term system for 5 million frames each.Source: Appendix B.3 · pp. 32–33

Appendix B.4. Transferring and Retaining the Policy in the LTM System

  • Policy transfer extracts the short-term DQN’s highest-Q action for each sampled state, one-hot encodes it, and distils that policy into the long-term agent with cross-entropy.The short-term DQN has a linear output layer, while the long-term agent uses a softmax output layer.Source: Appendix B.4 · p. 33
  • For rehearsal-based policy retention, RePR uses cross-entropy on GAN-generated pseudo-items representing states from previously learned games.EWC variants retain policies with weight constraints, whereas rehearsal-based methods adapt their retention loss to cross-entropy.Source: Appendix B.4 · p. 33

Appendix B.5. Extended Task Sequence Experiment

The extended-sequence experiment changes training duration and network size, and compares rehearsal and RePR variants under Q-value or policy retention.

Appendix B.5. Extended Task Sequence Experiment

  • The extended experiment initially sets α to 0.05 and reduces it to 0.01 after the fourth task, slowing learning.The experiment is described as nearly identical to the first experiment except for these changes and the enlarged networks and longer training.Source: Appendix B.5 · pp. 33, 35
  • Figure B.6 evaluates the long-term agent every 1 million observable training frames across Road Runner, Boxing and James Bond, with task switches marked by dashed lines.The figure compares EWC, online-EWC and RePR under a different training scheme; results use a single seed.Source: Figure B.6 · p. 34
  • Long-term DQN training was increased to 40 million frames, and the GAN training to 400,000 steps, with both networks enlarged.The DQN used 64, 128 and 128 convolutional filters and 1,024 fully connected units; the GAN generator’s first fully connected layer used 512 × 7 × 7 units.Source: Appendix B.5 · p. 35
  • The extended-sequence conditions standardise short-term DQN Q-values to reduce interference from games’ substantially different reward functions.Policy-retention variants instead use cross-entropy to learn and retain each game’s policy.Source: Appendix B.5 · p. 35

Appendix C. How well does RePR Share Weights?

The appendix uses Fisher overlap to assess whether RePR relies on shared important weights across tasks, and reports greater overlap between Road Runner and Boxing than other task pairs.

Appendix C. How well does RePR Share Weights?

  • Fisher overlap is bounded between 0 and 1, with higher values indicating more shared important weights for calculating the desired action across two tasks.The calculation uses task Fisher matrices normalised to unit trace and approximated from 100 batches of experience-replay samples.Source: Appendix C · p. 35
  • The comparison examines RePR’s task-pair overlaps after learning Road Runner, Boxing and James Bond, and then after learning the reverse order.The reverse-order analysis tests whether the Road Runner–Boxing overlap persists independent of learning order.Source: Appendix C · pp. 35–36
  • Road Runner and Boxing retain higher Fisher overlap than other task pairs in both task orders, suggesting RePR shares important weights more when tasks are more similar.The appendix reports the precise overlap values in Table C.7.Source: Table C.7 · p. 36, Appendix C · p. 36

Appendix D. Evaluating the Importance of Dual Memory

The appendix compares PR without dual memory against RePR with dual memory on sequential Road Runner and Boxing learning, and includes simplified training procedures and a competing-interest declaration.

Appendix D. Evaluating the Importance of Dual Memory

  • The experiment varies α for PR and RePR on Road Runner followed by Boxing, balancing new-task learning against pseudo-rehearsal retention.For PR, α weights deep Q-learning against pseudo-rehearsal; for RePR, it weights distillation against pseudo-rehearsal.Source: Appendix D · p. 36
  • Without dual memory, PR required α = 0.95 to learn Boxing similarly to RePR, yet no α achieved both successful new-task learning and acceptable Road Runner retention.Both models showed considerable Road Runner forgetting at high α; RePR’s dual-memory system allowed comparatively lower α values to support learning and retention.Source: Appendix D · pp. 36–37, Table D.8 · p. 37

Declaration of Competing Interest

  • The authors declare no known competing financial interests or personal relationships that could have influenced the reported work.Source: Declaration of Competing Interest · p. 37
  • The simplified procedures describe short-term DQN training, long-term RePR updates combining current-task learning with retention, and GAN training from current and generated samples.The long-term procedure weights new-task learning and retention with α; the GAN procedure trains a new generator using current-environment samples and pseudo-items from the previous GAN.Source: Declaration of Competing Interest · pp. 38–40
Loading 1812.02464v6…