PaperlayerView exact revision on arXiv

Source-linked AI summary

Continual World: A Robotic Benchmark For Continual Reinforcement Learning

Maciej Wołczyk, Michał Zając, Razvan Pascanu, Łukasz Kuciński, Piotr Miłoś

arXiv:2105.10919v3cs.LGcs.AIcs.RO

TL;DR

Continual reinforcement-learning agents must learn new tasks while balancing resource constraints, forgetting, and transfer. This paper introduces Continual World, a robotic continual-learning benchmark, and evaluates existing methods. The evaluation finds that methods may mitigate forgetting at the expense of forward transfer, while often struggling to outperform simple baselines on transfer.

  • Problem

    Continual reinforcement-learning agents must learn over time while balancing resource constraints, avoiding catastrophic forgetting, and transferring knowledge to new tasks.

    Source: §1 · pp. 1–2

  • Method

    The authors introduce Continual World, a benchmark built on Meta-World robotic tasks, and release code with implementations of seven representative continual-learning algorithms.

    Source: §1 · p. 2, §7 · p. 10

  • Results

    Existing methods can handle catastrophic forgetting relatively well at the expense of forward transfer and struggle to outperform simple baselines on that capability.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors argue for greater attention to forward transfer and to the interaction between forgetting and transfer when evaluating continual-learning systems.

    Source: §7 · p. 10

    Continual World uses a separate output head for each new task; a single-head architecture is an alternative the authors identify for future work.

    Source: §4.4 · p. 6

Abstract

from arXiv · show

Continual learning (CL) -- the ability to continuously learn, building on previously acquired knowledge -- is a natural requirement for long-lived autonomous reinforcement learning (RL) agents. While building such agents, one needs to balance opposing desiderata, such as constraints on capacity and compute, the ability to not catastrophically forget, and to exhibit positive transfer on new tasks. Understanding the right trade-off is conceptually and computationally challenging, which we argue has led the community to overly focus on catastrophic forgetting. In response to these issues, we advocate for the need to prioritize forward transfer and propose Continual World, a benchmark consisting of realistic and meaningfully diverse robotic tasks built on top of Meta-World as a testbed. Following an in-depth empirical evaluation of existing CL methods, we pinpoint their limitations and highlight unique algorithmic challenges in the RL setting. Our benchmark aims to provide a meaningful and computationally inexpensive challenge for the community and thus help better understand the performance of existing and future solutions. Information about the benchmark, including the open-source code, is available at https://sites.google.com/view/continualworld.

1 Introduction

Continual learning addresses the challenge of adapting to non-stationary settings, while evaluation is constrained by computational cost and conflicting objectives. Continual World targets forward transfer with a robotic benchmark and an evaluation of existing methods.

1 Introduction

  • Deep-learning models excel mostly in static domains that satisfy the i.i.d. assumption, whereas deployed systems may need to adapt to changing environments and hardware wear.The authors present continual learning as relevant to such open-ended settings.Source: §1 · p. 1
  • Continual World uses realistic robotic manipulation tasks from Meta-World to provide a computationally cheap benchmark with shorter auxiliary sequences for faster research cycles.Its task relationships are intended to make forward transfer more meaningful to study.Source: §1 · p. 2
  • Many evaluated methods handle catastrophic forgetting relatively well at the expense of other desiderata, particularly forward transfer.The authors argue that this supports greater focus on forward transfer and benchmarks with common task structure.Source: §1 · p. 2
  • The benchmark makes optimizing forward transfer central and finds that existing methods struggle to outperform simple baselines on that capability.The authors also release code for the benchmark and seven continual-learning methods.Source: §1 · p. 2

2 Related work

Prior continual-learning benchmarks span supervised learning and reinforcement-learning settings, but the authors identify limits in their suitability for forward transfer and RL-specific evaluation. They position Continual World as a benchmark aimed at those gaps.

2 Related work

  • Continual-learning surveys and taxonomies describe the field and its methods, with the authors adopting a taxonomy of continual-learning solutions [13] [38].They also point to a high-level overview connecting non-stationary learning with AI development.Source: §2 · p. 2
  • The authors note that no reinforcement-learning continual-learning benchmark has been widely accepted, despite prior work applying continual-learning methods in RL.They cite a recent review of the area.Source: §2 · p. 2
  • Supervised benchmarks such as MNIST, image-classification datasets, CORe50, and language modeling can measure forgetting but are not geared toward forward transfer or key RL-specific characteristics.The cited examples include permuted and split MNIST, CIFAR variants, Omniglot, and mini-ImageNet.Source: §2 · p. 2
  • Atari sequences can require 100M or more steps for ten games and lack meaningful task overlap, while continuous-control evaluations use short sequences and limited experiments.A prior Meta-World evaluation used MT10 but did not provide in-depth task analysis or comparisons across other continual-learning methods.Source: §2 · pp. 2–3
  • Other RL benchmarks vary in focus and scope: maze suites change maps or visuals, StarCraft is computationally demanding, and Lifelong Hanabi studies multi-agent coordination.Causal World does not directly consider continual learning, while the real-world relevance of Jelly Bean World is unclear to the authors.Source: §2 · p. 3

3 Continual learning background

Continual learning concerns sequentially acquiring skills and solving new tasks without forgetting earlier ones. Its evaluation balances transfer and forgetting against resource constraints, which can conflict.

3 Continual learning background

  • Continual learning aims to handle non-stationarity by sequentially acquiring skills and solving novel tasks without forgetting prior ones.It is related to multi-task, curriculum, and meta-learning, which differ in assumptions about task access, ordering, and adaptation.Source: §3 · p. 3
  • Evaluation commonly measures catastrophic forgetting, forward transfer, and backward transfer, alongside constraints on computation, memory, network size, and data.The objectives can conflict; unlimited memory and rehearsal could mitigate forgetting but require high computational cost.Source: §3 · p. 3
  • Figure 1 shows PEG-UNPLUG-SIDE-V1 and an example forward-transfer value of 0.10 from SHELF-PLACE-V1 to PEG-UNPLUG-SIDE-V1.The figure caption identifies the task shown on the left and the transfer example on the right.Source: Figure 1 · p. 3

4 Continual World benchmark

Continual World adapts Meta-World manipulation tasks into a computationally accessible RL benchmark for studying forward transfer. Its design defines transfer-focused metrics and protocols, while acknowledging limits in task inputs, boundaries, and sequence scope.

4 Continual World benchmark

  • Continual World sequences realistic robotic manipulation tasks to study forward transfer, using Meta-World and providing implementations of seven existing algorithms.The benchmark is intended for both CL and RL research and is designed to be computationally accessible.Source: §4 · p. 4

4.1 Metrics

  • Forward transfer compares a method’s training curve with a single-task reference; backward transfer is measured but reported as negligible in these experiments.The benchmark also reports average performance and forgetting; the authors note that the learning-speed metric excludes trivial tasks where it is ill-defined.Source: §4.1 · p. 4

4.2 Continual World tasks

  • CW20 repeats a deliberately ordered set of 10 tasks, selected for varied forward transfer; the task-pair matrix estimates transfer, with a CW20 reference value of 0.46.CW10 is described as informative for many cases, while CW20 is recommended for final evaluation; the benchmark also provides three-task triplets.Source: §4.2 · pp. 5–6, Figure 2 · p. 5

4.3 Training and evaluation details

  • Training uses SAC, separate policy heads for each task, randomized object positions, and a 1M-step budget per task.The main sequence has 20 tasks and a total sample budget of 20M; the authors typically used 20 seeds and bootstrap confidence intervals.Source: §4.1 · p. 4, §4.3 · p. 6

4.4 Limitations of Continual World

  • The benchmark’s scope is limited by its small observation space, reliance on SAC and known task boundaries, separate task heads, and only 10 distinct tasks in CW20.It studies low-level transfer through neural-network features and weights, not compositional behavior or semantic-level transfer.Source: §4.4 · pp. 6–7

5 Methods

The paper evaluates seven continual-learning methods from three method families, alongside multi-task learning. The families differ in how they protect prior-task knowledge: regularization, parameter isolation, or replay.

Regularization-based Methods

  • Regularization methods protect parameters important to prior tasks, from equal L2 penalties to importance-weighted penalties and variational inference.EWC estimates importance with the Fisher information matrix, MAS estimates parameter impact on network output, and VCL minimizes KL divergence between current posterior and prior.Source: §5 · p. 8

Parameter Isolation Methods

  • PackNet isolates tasks by iteratively pruning, freezing, and retraining network parts at task changes.Parameter isolation prevents changes to parameters important for previous tasks and is described as a hard equivalent of regularization.Source: §5 · p. 8

Replay Methods

  • Replay methods retain past-task samples for training or constraints; the study includes Perfect Memory and A-GEM, which projects new-sample gradients to avoid interference.Perfect Memory retains all past samples, while the authors report that A-GEM performs poorly on this benchmark.Source: §5 · p. 8

Multi-task learning

  • Multi-task learning trains tasks simultaneously and avoids forgetting by design, but tasks can compete for a single learning system's attention.Reward normalization with PopArt is reported as essential for good performance.Source: §5 · p. 8

6 Experiments

On CW20, the tested methods generally mitigate forgetting, but forward transfer declines over the sequence and remains modest. The experiments also identify RL-specific challenges involving critic regularization and exploration.

6 Experiments

  • The study evaluates seven representative methods on CW20 under fixed compute, memory, sample-count, and network-architecture constraints.The authors also report that tested methods have relatively small resource overheads; PackNet takes 15% more time than fine-tuning and uses 50% more parameters.Source: §6 · pp. 8–9
  • PackNet approaches 0.8 success rate averaged over tasks, while other methods perform considerably worse and A-GEM and Perfect Memory struggle.The reported comparison uses average success rate; the table caption identifies Table 1 as CW20 results for continual-learning methods and multi-task training.Source: §6 · p. 8, Table 1 · p. 8
  • Most methods mitigate forgetting, but revisiting tasks yields no observed boost; A-GEM is difficult to tune and, at its best settings, resembles fine-tuning.The authors note that relearning internal representations might have affected performance if those representations changed substantially on revisiting a task.Source: §6 · p. 8
  • Only fine-tuning and PackNet achieve positive forward transfer across CW20: 0.20 and 0.19 overall, respectively, below the RT reference of 0.46.Their first-half values are 0.32 and 0.21, and their second-half values are 0.08 and 0.17, respectively; all methods decline on the second ten tasks.Source: §6 · p. 9
  • In task triplets, an intervening task B reduces forward transfer from A to C, and tested continual-learning methods often fail to outperform fine-tuning.Even PackNet struggles in this setting; the authors suggest that task B's activation mask may contribute to the deterioration.Source: §6 · p. 9, Figure 4 · p. 9

6.1 RL-Related Challenges

  • RL adds exploration, varying reward scales, and environmental stochasticity, and some continual-learning methods need careful tuning or conceptual adaptation for this setting.The authors argue for a reliable benchmark to assess methods against these RL-related challenges.Source: §6.1 · p. 10
  • For actor-critic methods, the authors favor reducing forgetting in the actor while letting the critic adapt to the current task, though this choice is controversial and not trivial for replay methods.The critic is used only in training the current task, and the authors examine critic regularization empirically for EWC.Source: §6.1 · p. 10
  • At task changes, clearing SAC's replay buffer and collecting fresh samples, usually with a uniform policy, can reduce current-task performance and forward transfer.The authors report that retaining the buffer or exploring with the current policy improves transfer on the diagonal.Source: §6.1 · p. 10

7 Conclusions and Future Work

Continual World is presented as a benchmark for evaluating continual reinforcement learning and for examining the balance between forgetting and forward transfer. Its simulated robotic tasks share observation, action, and reward structure, and include two long sequences and eight task triples.

7 Conclusions and Future Work

  • The authors argue that continual-learning methods should balance transfer with forgetting, since many appear to sacrifice transfer to alleviate forgetting.They identify parameter isolation as a promising direction and flag critic regularization and efficient use of multi-task replays as pressing RL–CL issues.Source: §7 · p. 10

A.1 Tasks details

  • Continual World uses simulated Sawyer robot tasks from Meta-World, with shared observation and action spaces and similar reward structure.Each task lasts 200 steps; observations are 12-dimensional and actions are 4-dimensional.Source: §A.1 · p. 15
  • The benchmark's task-specific Meta-World rewards are used for training, while evaluation uses a binary success metric.The success criterion depends on object and goal positions and a task-specific threshold.Source: §A.1 · p. 15

A.2 Sequences

  • The benchmark includes long CW10 and CW20 sequences plus eight task triples; CW20 repeats CW10 twice.CW10 contains ten named tasks, including HAMMER-V1, PUSH-WALL-V1, and PEG-UNPLUG-SIDE-V1.Source: §A.2 · p. 15

A.3 Training and evaluation details

The experiments use SAC across task sequences, with continual-learning methods generally applied to policy networks. The appendix describes network design, hyperparameter selection, uncertainty estimation, computational infrastructure, and the method families evaluated.

A.3 Training and evaluation details

  • The study uses a TensorFlow 2 implementation of SAC, retaining actor and critic parameters across tasks; most continual-learning methods are applied only to policy networks.A-GEM and Perfect Memory are exceptions to the policy-only setup.Source: §A.3 · p. 15

A.4 Network architecture

  • Actor and critic use four 256-unit linear layers with Layer Normalization after the first layer, and task-specific heads sized to the sequence.The actor maps observations to action-distribution mean and log standard deviation; the critic maps observation-action inputs to scalar values.Source: §A.4 · p. 16

A.5 Hyperparameters

  • Hyperparameters are tuned in two stages: common SAC settings using fine-tuning on CW20, then method-specific settings using final average performance.The common search space and selected parameters are presented in Table 2; method-specific settings appear in Appendix B.Source: §A.5 · p. 16, Table 2 · p. 16

A.6 Bootstrap confidence intervals

  • The authors use non-parametric bootstrap confidence intervals and typically use 20 seeds to obtain informative intervals.Unless noted otherwise, numerical results use at least 20 seeds and report 90% confidence intervals.Source: §A.6 · p. 16

A.7 Infrastructure used

  • A CW20 experiment takes about 100 hours on a CPU node, and the authors report that GPUs offered only slight wall-time improvement for their small network.Their typical node used an Intel Xeon E5-2697 processor and 128 GB memory; three or four experiments ran concurrently.Source: §A.7 · p. 16

B CL methods

  • The benchmark evaluates seven continual-learning methods spanning regularization, parameter isolation, and replay; some required non-trivial adaptation from supervised learning to RL.The listed methods are L2, EWC, MAS, VCL, PackNet, Perfect Memory, and Reservoir Sampling.Source: B CL methods · p. 17

B.1 Regularization-based methods

This block reviews parameter protection, isolation, and replay methods, then examines resource use and RL-specific critic and Fisher-matrix issues.

B.1 Regularization-based methods

  • Regularization protects parameters important to previous tasks; L2 treats all parameters equally and can reduce forgetting while substantially limiting learning.The paper also describes a memory-efficient multi-task variant that stores summed importance vectors and the weights after the last task.Source: §B.1 · p. 17
  • VCL maintains a factorized Gaussian distribution over parameters and uses variational inference, with several changes proposed to make it usable in RL.The authors tune λ, omit the prior on the first task, use one weight sample, and omit coresets because of computational cost.Source: §B.1 · p. 18

B.2 Parameter isolation methods

  • PackNet prunes and freezes task-associated parameters, keeping model size fixed while leaving fewer parameters available for later tasks.For CW20, the selected subset is 5% of parameters and is fine-tuned using already collected data.Source: §B.2 · p. 18

B.3 Replay methods

  • Reservoir sampling approximates the empirical distribution of observed samples; the paper’s unlimited-capacity replay variant is called Perfect Memory, but replay performs poorly even at batch size 512.That batch size significantly extends training time.Source: §B.3 · pp. 18–19

B.4 Resources comparison

  • Most methods use modest resources, while reservoir sampling takes twice the baseline fine-tuning computation time and VCL uses more parameters.The table reports computational and memory overhead normalized to Fine-tuning.Source: Table 3 · p. 19, §B.4 · p. 19

C.1 Critic regularization

  • Regularized critics struggle to achieve decent performance compared with non-regularized references, despite EWC tests across multiple regularization settings.The authors note that the critic usually represents only the current task’s value function, so forgetting is not a primary concern; they caution that omitting regularization can leave a repeated task’s critic to be trained from scratch.Source: Figure 5 · p. 20, §C.1 · p. 20
  • The authors argue critic regularization is harmful because the critic is generally used only during current-task training, although omitting it can leave a repeated task’s critic to be rebuilt.They identify critic regularization as an important research direction.Source: §C.1 · p. 20

C.2 EWC Fisher matrix derivation

  • The Fisher-matrix appendix derives diagonal coefficients analytically for the Gaussian distribution used by the SAC policy.For the critic experiments, the derivation assumes a normal distribution with constant σ^2 = 1; the proof uses the curvature of the KL divergence.Source: §C.2 · pp. 20–21

C.3 VCL

These sections report that removing VCL’s first-task prior improves its results, examine replay and A-GEM performance, and analyze transfer patterns and same-task returns.

C.3 VCL

  • Abandoning VCL’s first-task prior yields the most successful tested setup, with λ = 1.0 optimal; the original setup underperforms and higher λ values fail under strong prior regularization.Increasing λ to 10^-5 improves results slightly over the original setup.Source: §C.3 · pp. 21–22, Figure 6 · p. 22

C.4 Reservoir Sampling

  • Reservoir Sampling runs with batch size 128 fail to exceed Fine-tuning’s 0.05 final average success, while two batch-size-512 runs reach 0.1–0.15 but remain below regularization and parameter isolation methods.The authors attribute the weaker results to critic regularization; larger batches also trade off training speed.Source: Figure 7 · p. 22, §C.4 · p. 22

C.5 A-GEM not working

  • None of the tested A-GEM batch-size settings performs significantly better than the others, and A-GEM performs poorly on the benchmark.The authors suggest critic regularization and potentially brittle gradient approximation under high-variance RL gradients as possible explanations.Source: Figure 8 · p. 23, §C.5 · p. 23

C.6 PackNet extensive clipping

  • PackNet initially becomes unstable near task boundaries; global gradient-norm clipping at 2 × 10^-5 addresses the observed instability while mostly preserving performance.The authors conjecture that frequent changes to optimized parameters may amplify temporal-difference learning instability.Source: §C.6 · p. 23

D.1 Diagonal transfers

  • Same-task diagonal transfers are smaller than expected; keeping the buffer or skipping random exploration improves performance, while changing to a single head alone makes no significant difference.The authors conjecture that buffer resets and random exploration at task changes create distributional shift that SAC struggles to handle.Source: §D.1 · p. 23

D Transfer matrix

  • Most entries in the transfer matrix are positive, but PEG-UNPLUG-V1 followed by PUSH-V1 interferes; forward transfer varies more across second tasks than first tasks.The reported standard deviations are 0.21 for second-task forward transfer and 0.07 for first-task forward transfer.Source: D Transfer matrix · p. 23, Table 4 · p. 24

E CW20 results

On CW20, methods generally mitigate forgetting but deliver weak forward transfer, with even PackNet and Fine-tuning below the transfer-matrix reference. The section also reports similar CW10 patterns and difficulty matching direct transfer in triplet experiments.

E CW20 results

  • PackNet has the strongest forward transfer among the methods, close to Fine-tuning, but both fall below the transfer-matrix reference value of 0.46.Most methods mitigate forgetting but have unsatisfying forward transfer; Table 6 summarizes CW20 results.Source: E CW20 results · p. 24, Table 6 · p. 25

E.1 Forgetting and backward transfer

  • Fine-tuning and A-GEM show rapid catastrophic forgetting after task switches, while most other methods—including L2—mitigate forgetting, with L2 doing so at the cost of poor transfer.The experiment reports no backward transfer.Source: §E.1 · p. 26

E.2 Forward transfer

  • Fine-tuning is the strongest forward-transfer baseline, with PackNet and A-GEM close behind, although A-GEM has low overall performance; most other methods show negative transfer, often substantially.Transfer on the second half of CW20 is worse despite those tasks having been learned earlier without being forgotten.Source: §E.2 · p. 26, Table 8 · p. 28, Table 9 · p. 29

E.3 Results for CW10

  • CW10 results are mostly consistent with CW20, making the shorter sequence suitable for faster experimentation.Table 10 reports the CW10 results for continual-learning methods and multi-task training.Source: §E.3 · p. 26, Table 10 · p. 29

F Triplet experiments

  • On most triplets, tested methods fail to reach direct A→C transfer on the third task, and Fine-tuning often outperforms the other methods.In triplets 7 and 8, Fine-tuning reaches transfer comparable to A→C, while EWC and PackNet are closer to B→C; Perfect Memory performs poorly across the triplets.Source: F Triplet experiments · p. 30, Table 11 · p. 30, Figure 11 · p. 30

G.1 Other orders of tasks

The experiments examine task-order sensitivity, an alternative single-head setup, and scaling to 30 tasks. Method rankings are preserved across tested orderings, while EWC’s forward transfer varies with order and PackNet leads in the 30-task sequence.

G.1 Other orders of tasks

  • Across three CW10 orderings, method rankings are preserved, but EWC’s forward transfer varies substantially with sequence order.The authors identify reducing this variance as an important research question and speculate that it is related to improving transfer generally.Source: §G.1 · p. 31, Table 12 · p. 31, Figure 12 · p. 31

G.2 Random order of tasks

  • Results averaged over 20 random CW10 orderings are qualitatively similar to those for the fixed ordering.Table 13 summarizes results across the random orderings.Source: §G.2 · p. 31, Table 13 · p. 32

G.3 One-hot inputs

  • With a single-head architecture and one-hot task input, EWC performance drops somewhat while PackNet remains mostly unaffected.The comparison includes both single-head and multi-head setups.Source: §G.3 · p. 31, Table 14 · p. 32, Figure 13 · p. 32

G.4 Experiments with a sequence of 30 tasks

  • On 30 tasks, PackNet significantly outperforms the other tested methods, followed by EWC; reservoir and Fine-tuning show catastrophic forgetting and mostly solve only one task.The study uses 40 seeds and random task orderings; tasks were selected after removing the 10 easiest and 10 hardest Meta-World tasks.Source: §G.4 · p. 32, Table 15 · p. 33

H Multi-task learning

In simultaneous training of 10 CW10 tasks, PopArt reward normalization improves reported performance, and EWC and PackNet achieve results similar to or better than multi-task learning. The experiments also required substantial CPU compute.

H Multi-task learning

  • PopArt reward normalization raises multi-task performance to 0.66 from 0.50 for the standard version, while EWC and PackNet achieve results similar to or better than multi-task learning.The experiment trains 10 CW10 tasks simultaneously without duplicating tasks.Source: H Multi-task learning · p. 33, Figure 14 · p. 34

I Infrastructure used

  • A typical CW20 experiment took 100 hours and used 8 or 12 CPU cores on servers with 24 or 28 cores and 64GB of memory, without GPUs.The project ran more than 45K experiments.Source: I Infrastructure used · p. 33

Related papers

Loading 2105.10919v3…