PaperlayerView exact revision on arXiv

Source-linked AI summary

Model-Free Generative Replay for Lifelong Reinforcement Learning: Application to Starcraft-2

Zachary Daniels, Aswin Raghavan, Jesse Hostetler, Abrar Rahman, Indranil Sur, Michael Piacentino, Ajay Divakaran

arXiv:2208.05056v2cs.LGcs.AI

TL;DR

Deep lifelong reinforcement learning must acquire new tasks while retaining performance on earlier ones, but storing past experiences may be impractical over long lifetimes. This paper develops model-free generative replay with wake-sleep consolidation and evaluates it on StarCraft 2 and MiniGrid tasks. Hidden replay outperformed other replay mechanisms, and most StarCraft 2 scenarios reached 80–90% of expert performance with 6% of training samples.

  • Problem

    Deep lifelong RL must learn across changing tasks while retaining performance on previous tasks, although storing sufficient examples from all past tasks may be impractical.

    Source: §1 · p. 1

  • Method

    The paper develops model-free generative replay that models policy latent representations and uses wake-sleep cycles to train generative memory and consolidate policies into skills.

    Source: §1 · p. 2

  • Results

    Across StarCraft 2 and MiniGrid scenarios, hidden replay outperformed other replay mechanisms; versus task experts, 6% of training samples achieved 80–90% of expert performance in most StarCraft 2 scenarios.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    Replay of independently sampled observation-action pairs mitigated forgetting and was simpler to generate than trajectories.

    Source: §1 · p. 2

    Storing full observations in the replay buffer limits the number of samples saved under memory constraints.

    Source: §5 · p. 10

Abstract

from arXiv · show

One approach to meet the challenges of deep lifelong reinforcement learning (LRL) is careful management of the agent's learning experiences, to learn (without forgetting) and build internal meta-models (of the tasks, environments, agents, and world). Generative replay (GR) is a biologically inspired replay mechanism that augments learning experiences with self-labelled examples drawn from an internal generative model that is updated over time. We present a version of GR for LRL that satisfies two desiderata: (a) Introspective density modelling of the latent representations of policies learned using deep RL, and (b) Model-free end-to-end learning. In this paper, we study three deep learning architectures for model-free GR, starting from a na\"ive GR and adding ingredients to achieve (a) and (b). We evaluate our proposed algorithms on three different scenarios comprising tasks from the Starcraft-2 and Minigrid domains. We report several key findings showing the impact of the design choices on quantitative metrics that include transfer learning, generalization to unseen tasks, fast adaptation after task change, performance wrt task expert, and catastrophic forgetting. We observe that our GR prevents drift in the features-to-action mapping from the latent vector space of a deep RL agent. We also show improvements in established lifelong learning metrics. We find that a small random replay buffer significantly increases the stability of training. Overall, we find that "hidden replay" (a well-known architecture for class-incremental classification) is the most promising approach that pushes the state-of-the-art in GR for LRL and observe that the architecture of the sleep model might be more important for improving performance than the types of replay used. Our experiments required only 6% of training samples to achieve 80-90% of expert performance in most Starcraft-2 scenarios.

1 INTRODUCTION

Deep lifelong reinforcement learning must balance learning new tasks with retaining performance on previous tasks, while storing examples from all past tasks may be impractical. This paper develops model-free generative replay with wake-sleep consolidation and evaluates its architectures across StarCraft 2 and MiniGrid scenarios.

1 INTRODUCTION

  • Generative replay augments current learning with self-labeled samples from an internal generative model, avoiding an ever-growing store of past experiences.Deep networks can forget previous tasks when trained sequentially, and storing sufficient past examples may be unreasonable in long-lived or resource-limited systems.Source: §1 · p. 1
  • The proposed architectures combine introspective density modeling of policy latent representations with model-free end-to-end learning, avoiding explicit dynamics learning.This avoids density modeling of complex perceptual inputs and learning temporal dynamics and rewards from sparse experiences.Source: §1 · p. 2
  • Replay of independently sampled observation-action pairs was sufficient to mitigate forgetting and is simpler to generate than trajectories.The paper connects this replay form to input-label replay in supervised learning.Source: §1 · p. 2
  • Wake-sleep cycles train current-task policies during wake phases, then train generative memory and distill policy knowledge into reusable skills during sleep.The next wake phase seeds exploration with previously learned skills.Source: §1 · p. 2
  • Hidden replay outperformed other replay mechanisms and prevented drift in the features-to-action mapping of a deep RL agent.The evaluation covers transfer, generalization, adaptation, expert-relative performance, and forgetting.Source: §1 · p. 2

2 BACKGROUND

Lifelong RL studies agents learning from changing tasks while transferring knowledge forward and retaining performance on previous tasks. Generative replay uses generated, self-labeled samples to rehearse past tasks, and this paper examines model-free replay in policy feature space.

2.1 LIFELONG REINFORCEMENT LEARNING (LRL) SETUP

  • A lifelong RL syllabus presents tasks sequentially for fixed interactions, with shared observation dimensionality and action space but differing transition and reward dynamics.The agent maximizes average cumulative reward across tasks despite unknown task identities and change points, limited interactions, and fixed model capacity.Source: §2.1 · pp. 2–3
  • Lifelong RL aims to learn new tasks faster through positive forward transfer while maintaining earlier-task performance and minimizing catastrophic forgetting.The paper describes LRL as less explored than class-incremental classification.Source: §2.1 · p. 2

2.2 GENERATIVE REPLAY (GR)

  • Generative replay continually trains a generator on real experiences, then uses its samples to create self-labeled training examples that may reflect past tasks.Prior work focused mainly on class-incremental classification or generative modeling, with less exploration in LRL.Source: §2.2 · p. 3
  • The paper uses a VAE to encode observations into latent vectors and decode them for reconstruction, while identifying feature-space model-free replay as novel for generic deep RL agents.Earlier trajectory replay modeled dynamics, which the authors describe as challenging in complex settings such as StarCraft 2; naively moving generative modeling to policy feature space caused instability and severe forgetting.Source: §2.2 · p. 3

3 HIDDEN REPLAY FOR LIFELONG RL

The section combines wake-sleep training with model-free replay, then compares three architectures that replay observations or latent features. Hidden replay generates feature vectors directly, while a small random replay buffer helps limit feature drift.

Wake-sleep training

  • Wake-sleep training improves the current-task wake policy, then trains a stable sleep policy and generative memory on wake experiences augmented with generated, pseudo-labelled examples.Wake actions mix the wake and sleep policies, with the probability of choosing the sleep policy decaying over time.Source: §3 · p. 3

Sequential Replay

  • Sequential Replay decodes sampled latent vectors into observations before the policy labels them, but produced blurry reconstructions for image-like observations.The authors hypothesize that observations associated with similar policy actions tend to map to generic reconstructions.Source: §3 · p. 5

Two-Headed Replay

  • Two-Headed Replay separates the VAE from the policy after feature extraction, improving visual reconstructions but requiring two forward passes when sampling.Generated and real observations both pass through the feature extractor before the features reach the separate VAE and policy networks.Source: §3 · p. 5

Hidden Replay

  • Hidden Replay generates feature vectors directly, reducing dimensionality and avoiding the feature extractor during sampling.Unlike veridical replay, it uses features learned by the wake policy rather than relearning representations from raw observations.Source: §3 · p. 5
  • Hidden Replay samples and reconstructions fall within the same action partition as ground-truth features, while possible knowledge transfer remains unvalidated.The feature-only replay setup can let the feature extractor drift or collapse; similar-action transfer is proposed as a possibility requiring further study.Source: §3 · p. 5

Random replay

  • 96 random samples per sleep in Starcraft 2 and 256 in Minigrid were sufficient to alleviate feature drift in those domains.The random replay buffer stores raw observations paired with actions gathered across the agent’s lifetime.Source: §3 · p. 5

4 EXPERIMENTS

The experiments compare lifelong learners across task sequences, lifelong-learning metrics, and two domains. Hidden replay generally outperforms tested architectures, while replay results show benefits from adding a small random buffer and from sleep-phase distillation when adapting to new tasks.

4.1 EVALUATION SCENARIOS

  • The evaluation uses pairwise, alternating, and condensed scenarios to test transfer, retention over repeated tasks, and performance across task orders, respectively.Each scenario is evaluated over multiple syllabi.Source: §4.1 · p. 6

4.2 METRICS FOR LIFELONG RL

  • Evaluation combines reward trends, relative reward against single-task experts, and lifelong-learning metrics for transfer, forgetting, and learning speed.Relative-reward variants measure seen-task performance, overall syllabus performance, and generalization to unseen tasks.Source: §4.2 · p. 6

4.3 DOMAINS

  • Experiments use Starcraft 2 minigames and Minigrid tasks, with non-lifelong baselines and ablations over replay buffers and architectures.Starcraft 2 includes three minigames plus an additional task per game; Minigrid uses five tasks with two variants each.Source: §4.3 · p. 7

4.4 EVIDENCE OF OVERCOMING CATASTROPHIC FORGETTING

  • In alternating-task examples, lifelong and baseline agents perform similarly on similar tasks, while hidden replay shows less forgetting than the baseline on dissimilar tasks.Extreme forgetting was somewhat rare in Starcraft 2, possibly because of the task-set design.Source: Figure 3 · p. 7, §4.4 · p. 8

4.5 COMPARISON OF DIFFERENT TYPES OF REPLAY FOR SLEEP

  • Hidden replay with ER+GR, ER+RaR, or ER+RaR+GR shows higher forward transfer and unseen-task relative reward; the latter effect is more pronounced in condensed Starcraft 2.Small random replay also stabilizes hidden replay, and adding it to ER+GR reduces forgetting; sleep-phase distillation provides a jumpstart on new tasks. Backward-transfer scores vary with how well the first task was learned.Source: §4.5 · pp. 8–9

4.6 COMPARISON OF DIFFERENT ARCHITECTURES FOR SLEEP

  • Across both domains and many scenarios, hidden replay seemed to outperform tested architectures on most metrics; reported strengths included transfer, avoiding forgetting, and adaptation to new tasks.In most Starcraft 2 scenarios, hidden replay required only 6% of training samples to achieve 80–90% of expert performance.Source: §4.6 · p. 10

5 DISCUSSION

The discussion reports strong StarCraft-2 performance relative to task experts and identifies extensions to the replay approach. It also points readers to further experiments in Appendix A.14.

5 DISCUSSION

  • Only 6% of task-expert training samples were needed to reach 80–90% of expert performance in the StarCraft-2 experiments.The experiments also report positive forward and backward transfer, reduced catastrophic forgetting, and fast adaptation to new tasks.Source: §5 · p. 10
  • Full StarCraft-2 gameplay is a proposed next step, alongside reducing the high sample complexity of current deep RL agents in that setting.The reported results concern StarCraft-2 minigames.Source: §5 · p. 10
  • Potential extensions include self-triggered sleep, compressed replay experiences, learned representations from RGB observations, and models that grow or change over an agent’s lifetime.The current approach uses scheduled sleep, full observations in its replay buffer, hand-crafted observation representations, and fixed neural architectures and loss functions.Source: §5 · p. 10
  • Further experiments are reported in Appendix A.14.Source: §5 · p. 10

A.1 POTENTIAL NEGATIVE AND POSITIVE SOCIETAL IMPACTS

The authors identify potential military misuse of the multi-agent real-time strategy tool, while also noting possible societal applications in robotics and healthcare.

A.1 POTENTIAL NEGATIVE AND POSITIVE SOCIETAL IMPACTS

  • The tool could be used to train models for strategic military actions, potentially harming the national security of benign nations.Source: §A.1 · p. 13
  • One proposed societal benefit is training robot swarms to perform useful tasks over long lifetimes, including in agriculture and manufacturing.Source: §A.1 · p. 13
  • The authors also identify healthcare treatment planning as a possible use in domains requiring feedback-based learning under significant distribution shift.Source: §A.1 · p. 13

A.2 LIMITATIONS

The authors report limits in long-term retention, training demands, dependence on wake experts, generated-sample quality, and hidden replay’s memory use.

A.2 LIMITATIONS

  • Over long task syllabi, the agent can struggle to remember tasks and does not perfectly learn or retain them.Source: §A.2 · p. 14
  • Training requires careful definition of reasonable tasks and hyperparameter tuning.Source: §A.2 · p. 14
  • The sleep model depends on a good single-task wake expert; when the wake model fails to learn a task, sleep-model learning can be disrupted.Source: §A.2 · p. 14
  • Hidden replay and the sequential and two-headed models can be fragile when learned hidden spaces or generated samples drift from the true data distribution.In these cases, the sleep model can become unstable and replay may hurt learning.Source: §A.2 · p. 14
  • Hidden replay requires a small random buffer of samples across tasks, making memory use suboptimal; the authors observed that this replay had an outsized effect on lifelong learning compared with generative replay.Source: §A.2 · p. 14

A.3 STARCRAFT 2 TASK VARIANTS

The StarCraft-2 task variants cover shard collection, combat against Zerglings and Banelings, and combat against Roaches, with variants in visibility or enemy-group layout.

A.3 STARCRAFT 2 TASK VARIANTS

  • Collect Mineral Shards – No Fog of War has two Marines collecting endlessly respawned sets of 20 shards, with fog of war disabled.Source: §A.3 · p. 14
  • Collect Mineral Shards – Fog of War uses the same two-Marine, endlessly respawning shard task, but enables fog of war.The agent must learn without full knowledge of the current environment state.Source: §A.3 · p. 14
  • DefeatZerglingsAndBanelings – One Group pits nine Marines against one group of six Zerglings and four Banelings.Source: §A.3 · p. 14
  • DefeatZerglingsAndBanelings – Two Groups places nine Marines between nine Zerglings on one side and six Banelings on the other.Source: §A.3 · p. 14
  • DefeatRoaches – One Group pits nine Marines against four Roaches, with optimal combat requiring focus fire.Source: §A.3 · p. 14
  • DefeatRoaches – Two Groups places nine Marines between two groups of Roaches, three per side, with optimal combat requiring focus fire.Source: §A.3 · p. 15

A.4.1 STARCRAFT 2

The Starcraft 2 experiments use a VTrace wake agent and compare replay architectures across six task variants and three lifelong scenarios. The implementation combines policy distillation with replay during periodic sleep phases.

A.4.1 STARCRAFT 2

  • The wake agent uses a VTrace policy, while the hidden replay model adds feature normalization to improve reconstructions and limit feature drift.Its feature representation has dimension 8192; sequential and two-headed VAEs use convolutional encoders and decoders, while hidden replay uses MLPs.Source: §A.4.1 · p. 15
  • Sleep training uses experience and generative replay for all architectures, with hidden replay additionally using random replay; sleep schedules and loss weights vary by architecture.Each sleep uses 4,500 iterations; the hidden replay variant sleeps twice per task and adds 96 wake-buffer observations to its random replay buffer per sleep.Source: §A.4.1 · p. 15
  • The six SC-2 variants cover combat and resource-collection tasks, with pairwise, alternating, and condensed scenarios using 2M-step learning blocks and 30-episode evaluation blocks.Single-task experts were trained for about 30 million environment steps.Source: §A.4.1 · pp. 15–16
  • Figure 8 measures task similarity through forward transfer using single-task experts.The supplied passage identifies the measure but does not describe the figure’s axes or specific task comparisons.Source: Figure 8 · p. 16

A.4.2 MINIGRID

The Minigrid experiments use PPO agents with a VAE-based sleep model and evaluate lifelong learning across five tasks and ten POMDP variants. The setup applies experience, generative, and random replay during sleep.

A.4.2 MINIGRID

  • The wake agent uses PPO networks that map observations to a 256-dimensional feature space, while the sleep model uses a VAE with a 128-dimensional latent space.The VAE is a four-layer bottleneck MLP; its encoder bounds predicted log variance to [−5, 5].Source: §A.4.2 · p. 16
  • Sleep training combines experience replay, feature-vector generative replay, and random replay, beginning generative and random replay after the first sleep.Each sleep runs 20,000 iterations; 256 wake-buffer observations are added to a random replay buffer capped at 4096.Source: §A.4.2 · p. 16
  • The evaluation covers five Minigrid tasks with two variants each, using pairwise, alternating, and condensed scenarios and sparse rewards for reaching the goal.The observations are 7 × 7 views with object type, color, and state channels; evaluation blocks contain 100 episodes per task.Source: §A.4.2 · p. 16

A.5 MINIGRID TASK LENGTHS

Across the supplied appendix results, hidden replay generally performs best among the compared replay architectures, while random replay helps stabilize feature representations. The experiments also examine distillation, scenario setup, replay frequency, and generative quality.

A.5 MINIGRID TASK LENGTHS; A.6 PARAMETERS OF THE PPO AGENT; A.7 EVALUATION SCENARIOS VISUALIZATION

  • The listed Minigrid task lengths span 200,000 to 700,000 steps: CustomUnlockS5 uses 200,000, SimpleCrossingS9N1 400,000, SimpleCrossingS9N2 and CustomFetchS5T1N2 500,000, CustomUnlockS7 300,000, and CustomFetchS8T1N2 700,000.Source: §A.5 · p. 17

A.8 EXAMPLE LEARNING CURVES FOR LIFELONG CURRICULA ON Starcraft 2; A.9 COMPARISON OF LEARNING CURVES FOR LIFELONG CURRICULA ON MINIGRID

  • The SC-2 six-minigame curve approaches the approximately 100 performance of a converged single-task expert, while the Minigrid LL curve shows learning more tasks without interference and jump starts on similar tasks.The Minigrid figure states that maximum reward is 100.Source: Figure 10 · p. 18, Figure 11 · p. 18

A.10 A SLEEP POLICY TRAINED WITH POLICY DISTILLATION CAN LEARN TO IMITATE A CONVERGED SINGLE TASK EXPERT

  • The two-headed and hidden replay sleep policies typically match or exceed converged SC-2 expert rewards, while the sequential model generally performs worse.For Minigrid, the average reward difference between the expert and sleep policy is 0.14 with standard deviation 0.11.Source: §A.10 · pp. 18, 20, Figure 12 · p. 19, Figure 13 · p. 19

A.11 A SLEEP POLICY TRAINED WITH POLICY DISTILLATION CAN LEARN TO IMITATE THE WAKE POLICY ON A SINGLE TASK

  • Sleep-policy rewards usually track the improving wake policy, although some runs underperform; hidden replay has the smallest reported reward difference among the three architectures.Mean absolute differences are 0.589 ± 0.320 for sequential, 0.422 ± 0.263 for two-headed, and 0.209 ± 0.227 for hidden replay; the hidden-model figure compares wake and sleep reward curves.Source: §A.11 · pp. 20–21

A.12 EFFECT OF VARYING THE NUMBER OF WAKE-SLEEP CYCLES

  • Multiple sleep phases per task improve or maintain most lifelong-learning metrics in the tested alternating SC-2 scenarios, but may negatively affect forward transfer.The comparison varies one, two, and three sleeps per task across four scenarios.Source: §A.12 · p. 21, Table 1 · p. 21

A.14 UNDERSTANDING THE EFFECT OF DIFFERENT MODEL ARCHITECTURES

  • Across Minigrid and Starcraft 2, hidden replay generally outperforms the other replay architectures and simple baselines, with continual learners improving maintenance and generalization metrics.The Minigrid comparison reports significant gains in performance maintenance and RRΩ, while the SC-2 comparison reports better generalization to unseen tasks and learning without forgetting.Source: Table 4 · p. 22, §A.14 · p. 22, Table 5 · p. 23

A.15 QUALITATIVE ANALYSIS OF THE GENERATIVE MODEL

  • The sequential model produces blurry shard reconstructions, the two-headed model sharpens them but still generates density maps, and hidden replay reconstructs DefeatRoaches features closely.The two-headed model’s generated DefeatRoaches samples better resemble ground truth where observations have more structural regularity; hidden replay also generates features overlapping a similar previously learned task.Source: §A.15 · pp. 23–24, Figure 15 · p. 23, Figure 16 · p. 24

A.13 UNDERSTANDING THE EFFECT OF DIFFERENT REPLAY TYPES; A.16 ADDITIONAL RESULTS: UNDERSTANDING THE INTERPLAY BETWEEN DIFFERENT REPLAY MECHANISMS

  • Experience and generative replay alone are insufficient for hidden replay; adding random replay better aligns prior-task features and generally improves PM and FTR.The figures illustrate feature drift without random replay and better alignment with it; the accompanying text attributes the difference to maintaining the feature space.Source: Table 2 · p. 21, Table 3 · p. 22, §A.16 · p. 25
Loading 2208.05056v2…