PaperlayerView exact revision on arXiv

Source-linked AI summary

Continual Reinforcement Learning with Neuroevolution

Eleni Nisioti, Andrea Cossu, Kathrin Korte, Sebastian Risi

arXiv:2610.01583v1cs.NEcs.LG

TL;DR

Continual RL has not consistently balanced retaining prior skills with learning new tasks. This paper compares neuroevolution with RL across environments and task changes, and finds that ES most consistently balances stability and plasticity while GA is most plastic. It relates these differences to return-landscape neighborhoods and their overlap across tasks.

  • Problem

    Continual RL seeks to retain prior skills while acquiring new ones, but no method has consistently achieved a good stability-plasticity balance across task changes.

    Source: §1 · p. 1

  • Method

    The study compares ES and GA with PPO, continual RL variants, and population-based PPO across environments and task changes, and examines return landscapes around solutions.

    Source: §1 · p. 2, §4 · p. 5

  • Results

    ES most consistently achieves a good stability-plasticity trade-off, while GA is most plastic; neighborhood overlap predicts the trade-off, and neuroevolution lacks several reported RL plasticity-loss symptoms.

    Source: §6 · p. 8

  • Takeaways & Limitations

    The findings suggest that parameter-space exploration, which favors wide neighborhoods whose overlap predicts the trade-off, may be a mechanism for balancing stability and plasticity.

    Source: §6 · p. 8

    Neuroevolution is less sample-efficient, often needs large populations for complex tasks, and can underperform RL on finely tuned tasks such as HalfCheetah.

    Source: §6 · p. 8

Abstract

from arXiv · show

Despite many studies about causes and remedies of plasticity loss in Reinforcement Learning (RL) under continual task changes, no RL method has yet consistently achieved a good balance between adaptation and forgetting. Here we turn to an alternative optimization paradigm, neuroevolution (NE): algorithms that search directly in weight space through mutation and selection over a population of neural networks. Across a wide array of environments and environmental changes, with policies ranging from a few hundred parameters to million-parameter networks, we compare evolution strategies (ES) and genetic algorithms (GAs) against state-of-the-art continual RL variants and population-based RL. ES most consistently achieves a good stability-plasticity trade-off, while the GA is the most plastic method but forgets more than ES. To explain this, we study the return landscape around each method's solutions. ES finds the widest neighborhoods, i.e.\ regions of weight space in which perturbed policies still solve the task, and the size of the overlap between the neighborhoods of consecutive tasks correlates with a method's stability-plasticity trade-off. Rewarding behavioral diversity in a GA through novelty search makes the population even more plastic, at the cost of forgetting. Finally, symptoms of plasticity loss commonly reported in RL do not transfer to NE. Overall, these results establish NE as a competitive alternative to RL under continual task changes, and suggest that training under perturbations in weight space may be a useful mechanism for continual learning more broadly.

1 INTRODUCTION

Continual RL seeks agents that retain prior skills while acquiring new ones, but existing methods do not consistently balance stability and plasticity. The paper compares neuroevolution with RL and links their trade-offs to return-landscape neighborhoods around solutions.

1 INTRODUCTION

  • Continual RL methods still lack a consistently good balance between retaining prior skills and acquiring new ones, despite proposed remedies for plasticity loss.The paper notes that task and variation types affect method performance, and symptoms of plasticity loss can be hard to distinguish from its causes.Source: §1 · p. 1
  • Across varied environments and task changes, ES most consistently achieves a good stability-plasticity trade-off, while GA is most plastic and continual RL variants restore plasticity at a stability cost.PPO loses plasticity; policies range from a few hundred to more than a million parameters.Source: §1 · p. 2
  • ES has the widest neighborhoods and greatest overlap between consecutive tasks; that overlap predicts the stability-plasticity trade-off.The figure shows ES solving both tasks at the switch, while GA escapes the smaller overlap and continual RL variants leave the previous-task neighborhood.Source: §1 · p. 2, Figure 1 · p. 2
  • The GA’s broader population spread can help it escape local optima and find a member that already solves the incoming task, but also makes escape from a shared neighborhood more likely.Rewarding behavioral diversity through novelty search further increases the chance that some population member solves the new task.Source: §1 · p. 2
  • Neuroevolution does not accumulate dormant neurons, and weight drift does not cost it plasticity, unlike symptoms reported in continual RL.The authors suggest these symptoms may stem from gradient-based optimization rather than non-stationarity.Source: §1 · p. 2

2 BACKGROUND

The paper frames continual RL as learning a sequence of tasks without being told when tasks change. It contrasts this setting with neuroevolution, which updates neural-network weights through population-based evolutionary search.

2.1 CONTINUAL REINFORCEMENT LEARNING

  • Continual RL is modeled as a sequence of tasks, each defined by an MDP, with the agent maximizing episodic return on each task.The setting is boundary-free: the agent receives no information about task changes.Source: §2.1 · p. 3

2.2 NEUROEVOLUTION

  • Neuroevolution optimizes policy-network weights through mutation and selection over candidate networks evaluated by episodic return.A GA retains an archive and selects fit offspring; ES updates a single network using fitness-weighted perturbations to estimate a gradient of expected return under Gaussian noise.Source: §2.2 · p. 3

3 RELATED WORK

Prior work identifies multiple possible correlates and remedies of plasticity loss in RL, while studies of neuroevolution and diversity have often emphasized adaptation. This paper positions its comparison as examining both plasticity and forgetting in continual settings.

3 RELATED WORK

  • Proposed correlates of RL plasticity loss include overwriting, dormant neurons, weight drift, and churn, but their status as causes or symptoms remains contested.The paper benchmarks ReDo, TRAC, and C-CHAIN, which target dormant neurons, initialization drift, and representation collapse respectively.Source: §3 · p. 3
  • Population-based training runs gradient-based learners in parallel and periodically replaces weaker members with stronger copies; the cited work had not studied it in continual settings.The paper includes PBT-PPO to isolate the contribution of a population.Source: §3 · p. 3
  • Earlier evolutionary studies describe environmental change as a possible driver of adaptation, and quality-diversity methods seek diverse high-performing solutions.Continual adaptation of a diverse archive remains understudied; cited lifelong applications do not explicitly examine diverse populations over long task streams.Source: §3 · p. 4
  • Prior NE transfer and quality-diversity studies report plasticity or recoverability benefits, but do not jointly address forgetting and comparisons with continual RL variants.The paper also notes that evidence on whether ES-based LLM fine-tuning forgets less than RL remains inconclusive.Source: §3 · p. 4
  • Neighborhood linking can connect optima across switching tasks when one task’s solution lies within another’s neighborhood; this paper examines the role of neighborhood width and selection in RL.Earlier supervised-learning work linked wider neighborhoods to less forgetting, whereas this study focuses on how selection under noise affects neighborhood size and the stability-plasticity trade-off.Source: §3 · p. 4

4 EXPERIMENTAL SET-UP

The experiments compare neuroevolution and RL across varied environments, task changes, and policy sizes, using measures of adaptation, forgetting, and population diversity. They evaluate both long task sequences and alternating tasks, with no task-switch signal.

4 EXPERIMENTAL SET-UP

  • Population methods are evaluated through their centroid to compare each method as a single agent and avoid the upward bias of selecting the best of many noisy evaluations.Results generally average ten trials with 95% percentile-bootstrap confidence intervals, with within-setting differences tested using a corrected one-sided Mann-Whitney U test.Source: §4 · pp. 4–5
  • Learning accuracy measures end-of-task return, forgetting measures subsequent return loss, and the trade-off is LA − F; cumulative return and zero-shot transfer capture recovery and performance before training on the next task.Zero-shot transfer is measured at a switch before training on the next task.Source: §4 · p. 4
  • Neighborhood width measures the radius at which half of perturbed policies cease to solve a task, while shared-neighborhood size measures the fraction solving both current and next tasks at radius 0.1.A task is solved at rescaled return of at least 0.5; perturbations scale with each layer’s weight norm.Source: §4 · p. 5
  • The study measures behavioral and genomic diversity, and benchmarks ES, GA, GA + Novelty, PPO, three continual PPO variants, and PBT-PPO.GA + Novelty uses Dominated Novelty Search with AURORA descriptors; the GA also uses a mutation-width heuristic on tasks where its centroid lags.Source: §4 · p. 5
  • The benchmark spans control, navigation, locomotion, pixel-based puzzles, and hard exploration, with observation offsets, transition changes, or action reversal as task variations.Runs use either a 20-task sequence or two tasks alternating ten times; the latter allows a single policy to avoid forgetting and permits basin linking.Source: §4 · p. 5
  • Figure 2 plots learning accuracy against forgetting for the 20-task and alternating-task settings, with equal LA − F lines; GA is most plastic and ES has the best trade-off.The setup also varies hyperparameters and task-switch frequency to test sensitivity to these choices.Source: §4 · pp. 5–6, Figure 2 · p. 6

5 RESULTS

ES most often achieves the strongest stability–plasticity trade-off, while the GA is more plastic but forgets more than ES. Neighborhood overlap tracks the trade-off, and adding novelty search increases GA plasticity.

5 RESULTS

  • ES is circled in eight of eighteen settings, while the GA has the highest learning accuracy in ten and forgets more than ES in every setting.ES falls short under action reversal on Acrobot and MountainCar, while RL methods reach higher returns on HalfCheetah even without task switches.Source: §5 · p. 6
  • ES has the highest zero-shot transfer in ten of fourteen settings, while RL methods transfer less and some settings require relearning after every switch.No method transfers to an unseen Kinetix level.Source: §5 · pp. 6–7, Figure 3 · p. 7
  • Continual RL variants improve plasticity at the expense of stability; PBT-PPO leads RL methods on the trade-off in eight settings.Its population diversity collapses early, so it does not show the qualitative behavior that makes GAs plastic.Source: §5 · p. 7
  • ES has the widest neighborhoods, and shared neighborhood size correlates with the stability–plasticity trade-off across methods and within each family.PPO’s neighborhood is similarly narrow along its update path and in random directions; ReDo-PPO’s is about twice as wide along its path.Source: §5 · p. 7, Figure 4 · p. 8
  • Novelty search makes GA populations more behaviorally diverse and improves the best individual’s cumulative return in both tested settings.On DeepSea, GA + Novelty re-finds the new action map after 57% of switches, versus 11% for the GA; the centroid does not track this gain.Source: §5 · pp. 7–8
  • NE’s dormant fraction does not increase at task switches, and weight drift does not correlate with plasticity loss in NE.For the GA, the most plastic method, weights can drift by orders of magnitude.Source: §5 · p. 8

6 DISCUSSION

Across continual task changes, neuroevolution offers a competitive stability-plasticity alternative: ES most consistently balances the trade-off, while overlap between consecutive-task neighborhoods predicts it.

6 DISCUSSION

  • ES most consistently achieves the best stability-plasticity trade-off, while the GA is usually most plastic; both beat RL at the same budget on million-parameter Kinetix levels.The GA’s greater plasticity comes with more forgetting than ES.Source: §6 · pp. 8–9
  • Rewarding behavioral diversity through novelty search makes the GA population more plastic, as reflected in measures of return and behavioral and genomic diversity.Figure 5 tracks elite and centroid return alongside both diversity measures across task switches.Source: §6 · p. 8, Figure 5 · p. 9
  • Across methods, the size of the shared neighborhood between consecutive tasks predicts the stability-plasticity trade-off.The proposed mechanism is that parameter-space exploration favors wide neighborhoods more likely to overlap with the next task’s neighborhood.Source: §6 · p. 8
  • NE is less sample-efficient and may require large populations on complex tasks; on finely tuned HalfCheetah, it falls short of RL’s stationary performance.The authors attribute the HalfCheetah gap to gradient steps climbing narrow return peaks that ES smooths over.Source: §6 · p. 8
  • The authors identify parameter-space exploration for stability and population diversity for plasticity as possible directions for RL, and online mutation-range adaptation for NE.Source: §6 · p. 9

A.1 ENVIRONMENT SUITES AND THEIR CONTINUAL VARIANTS

A.2 METHODS

A.2.3 HYPERPARAMETERS OF EVERY METHOD AND ENVIRONMENT

The appendix specifies method hyperparameters and shared evaluation measures, then defines neighborhood measures for robustness to weight perturbations and overlap across tasks.

A.2.3 HYPERPARAMETERS OF EVERY METHOD AND ENVIRONMENT

  • NE and RL settings are tabulated by method family; continual PPO variants use PPO settings, and no variant is told when tasks change.GA + Novelty is run only on classic control and DeepSea.Source: Table 3 · p. 22, Table 4 · p. 22, §A.2.3 · p. 22

A.3 METRICS

  • All methods are evaluated on their reported agent, while population methods additionally use behavioral and genomic diversity measures.For ES, genomic diversity is its search width.Source: §A.3 · pp. 23–24
  • Return averages ten greedy-action episodes on held-out evaluation seeds; learning accuracy and forgetting use saved centroids, while cumulative return integrates current-task return over environment steps.For alternating tasks, forgetting measures return lost at the next switch in both directions.Source: §A.3 · p. 23
  • Task returns are rescaled within each setting, and a policy counts as solving a task at rescaled return 0.5.The same affine map applies to every method within a setting.Source: §A.3 · p. 24
  • Dormancy is based on normalized hidden-unit activation, while the empirical NTK uses output-gradient similarities to characterize cross-state churn and available output directions.NTK measurements use 128 probe states from the current task; dormant-unit scores use fixed random-policy states pooled across tasks.Source: §A.3 · p. 24

A.4 NEIGHBORHOOD ANALYSIS

  • Neighborhood width is the perturbation radius at which half of 64 perturbed policies no longer solve their task; action change is a cheaper width proxy, and shared neighborhood counts perturbations solving consecutive tasks.Widths are measured only at checkpoints that solve their own task; shared-neighborhood correlation with learning accuracy minus forgetting is reported as radius-independent.Source: §A.4 · p. 25

B PERFORMANCE WITHOUT TASK SWITCHES

Without task switches, methods are generally well tuned, though their performance varies by environment. In continual settings, NE avoids several plasticity-loss symptoms observed in RL.

B PERFORMANCE WITHOUT TASK SWITCHES

  • PPO-family methods generally collect the most reward on stationary classic control, MiniGrid, and HalfCheetah, while NE methods have environment-specific exceptions.ES is slowest with TRAC-PPO on MountainCar; the GA finishes lowest on Acrobot but is the only method to collect reward on Kinetix level h16.Source: B PERFORMANCE WITHOUT TASK SWITCHES · p. 25, Figure 7 · p. 26

C.1 RETURN OVER TRAINING

  • Figure 8 tracks centroid returns through sequences and alternations of tasks; these runs supply the metrics reported for continual settings.Dashed lines mark task switches.Source: Figure 8 · p. 27, §C.1 · p. 27

C.2 SYMPTOMS OF PLASTICITY LOSS

  • PPO’s dormant-unit fraction rises across ReLU continual settings, whereas GA and ES remain flat at environment-set levels; ReDo-PPO keeps dormancy near zero.MountainCar starts highly dormant for every method because random-policy observations vary little; HalfCheetah and Kinetix have almost no dormant units.Source: §C.2 · p. 27, Figure 9 · p. 28, Table 6 · p. 29
  • No NE method loses NTK rank, while rank loss appears for PPO and ReDo-PPO on MiniGrid and for ReDo-PPO on two action-reversal settings.MiniGrid rank also alternates with room size because larger rooms yield fewer distinct views during a random walk.Source: §C.2 · p. 27
  • GA weights grow by one to two orders of magnitude in classic control and Kinetix, yet mutations preserve adaptation after switches despite vanishing policy gradients at those scales.ES and RL weights stay close to their initial scale; GA does not show this growth on HalfCheetah.Source: §C.2 · p. 28

C.3 GA + NOVELTY IN ALL SEVEN CLASSIC CONTROL AND DEEPSEA SETTINGS

Novelty search improves the GA elite’s returns in several settings, but its centroid does not retain most of that lead. PPO minibatch size shifts its stability-plasticity position without materially changing retention.

C.3 GA + NOVELTY IN ALL SEVEN CLASSIC CONTROL AND DEEPSEA SETTINGS

  • GA + Novelty has the most behavioral diversity on Acrobot and MountainCar under both changes, but not on DeepSea, where PBT-PPO is more diverse.Its genomic diversity is 20 to 170 times that of PBT-PPO in every setting.Source: §C.3 · p. 29
  • The GA + Novelty centroid loses cumulative return to the GA on five settings, ties on CartPole under noise, and gains only on DeepSea.The elite-centroid gap is larger than the GA’s in every setting, largest on Acrobot and MountainCar.Source: §C.3 · p. 29
  • The GA + Novelty elite beats ES in all seven settings and PBT-PPO in six, while improving over the GA in four and tying it in three.Its PBT-PPO exception is CartPole under action reversal; the reported pair is 484 versus 463.Source: §C.3 · p. 29, Figure 10 · p. 30

D EFFECT OF HYPERPARAMETERS

  • The hyperparameter study varies one value at a time under the same task schedule, budget, task draw, and seeds.For PPO it varies minibatch size; for GA and ES it varies search-distribution width.Source: D EFFECT OF HYPERPARAMETERS · pp. 30–31

D.1 PPO: THE MINIBATCH SIZE

  • PPO minibatch size moves learning accuracy and forgetting together on seven of nine settings, leaving retained performance nearly unchanged; no value is best on cumulative return everywhere.A low-forgetting result at a setting that barely learns is not evidence of stability.Source: §D.1 · p. 31, Figure 11 · p. 31

D.2 NEUROEVOLUTION: THE SEARCH WIDTH OF THE GA AND ES

The GA learns across the tested search widths, whereas ES succeeds only near its reported width. Where ES learns, it forgets less than the GA, and no width reverses the paper’s main conclusions.

D.2 NEUROEVOLUTION: THE SEARCH WIDTH OF THE GA AND ES

  • ES learns only at ×1 and ×3, staying near the untrained score at narrower widths and collapsing at ×10; the GA learns at every width.On MiniGrid, the GA also loses some seeds at its two narrowest widths.Source: §D.2 · p. 32, Figure 12 · p. 32
  • Wherever ES learns, it forgets less than the GA; at CartPole and MiniGrid, ES’s good-width trade-off exceeds the GA’s at every width.On Acrobot and MountainCar under action reversal, ES remains frozen on one action ordering at every learning width while the GA relearns each ordering and forgets the other.Source: §D.2 · p. 32

E EFFECT OF THE FREQUENCY OF VARIATION

Across task lengths and environments, neuroevolution reaches stability-plasticity trade-offs no RL method matches on both axes: GA leads in learning accuracy, while ES has lower forgetting in most settings. Longer tasks bring more forgetting, with little change in learning accuracy except on MountainCar.

E EFFECT OF THE FREQUENCY OF VARIATION

  • The comparison holds task, seeds, hyperparameters, and a 4000-generation budget fixed while varying task duration from 50 to 400 generations around the 200-generation main setting.Figure 13 reports the stability-plasticity trade-off at each duration.Source: E EFFECT OF THE FREQUENCY OF VARIATION · p. 32, Figure 13 · p. 33
  • Across task lengths and environments, no RL method matches neuroevolution on both trade-off axes: GA has the highest learning accuracy, while ES has lower forgetting in most settings.Cumulative return also favors GA on Acrobot and MountainCar at every length; an RL method leads on CartPole, within the GA’s interval.Source: Figure 13 · p. 33, E EFFECT OF THE FREQUENCY OF VARIATION · p. 33
  • Forgetting grows with task length in every environment, while learning accuracy hardly changes except on sparse-reward MountainCar.MountainCar’s sparse reward takes many updates to find again after a switch; RL methods gain most there but remain well below GA.Source: E EFFECT OF THE FREQUENCY OF VARIATION · p. 33

F POPULATION DIVERSITY

Across settings, greater genomic diversity tracks both better learning and more forgetting, while ES’s stability does not follow the diversity ranking. GA + Novelty is especially diverse, but that pattern does not hold on larger policies.

F POPULATION DIVERSITY

  • GA + Novelty members disagree on 0.25–0.44 of classic-control probe states, versus 0.08–0.23 for GA, and have the most genomic diversity there at 4–5.5 per weight.On HalfCheetah and MiniGrid, GA genomic diversity is no greater than ES; DeepSea 12 instead has the most behavioral diversity under PBT-PPO (0.43 of probe states).Source: F POPULATION DIVERSITY · p. 34
  • Fitness variance rises at every task switch for every population, while behavioral diversity usually falls after the first task and stays low except under GA + Novelty.GA + Novelty’s novelty term keeps members apart; ES and PBT-PPO sit near 0.1 genomic diversity per weight.Source: F POPULATION DIVERSITY · p. 34
  • Across ten settings, genomic diversity correlates positively with cumulative reward, learning accuracy, and forgetting (Spearman ρ = 0.63, 0.41, and 0.25); ES’s stability is not explained by its diversity ranking.The ordering follows GA, but does not order ES against PBT-PPO.Source: F POPULATION DIVERSITY · p. 34
  • The diversity record covers six classic-control settings and DeepSea 12, but omits HalfCheetah, MiniGrid, and Kinetix for GA + Novelty.Figure 14 displays fitness spread, behavioral diversity, and genomic diversity per weight.Source: F POPULATION DIVERSITY · pp. 33–34, Figure 14 · p. 34

G NE IN TOY LANDSCAPES

Toy landscapes show that ES and GA retain shared solutions under different landscape and search-width conditions: ES holds a smooth shared plateau when its search is narrow, while GA reaches rugged shared peaks across ripple depths. Increasing intrinsic dimensionality can undermine GA search, but the toy result does not establish that this explains its performance on large policies.

G NE IN TOY LANDSCAPES

  • On the smooth landscape, ES converges to and holds the shared plateau at every width, while GA leaves it at h = 0.1 in 23 of 24 seeds.ES also leaves when its search reaches the slope at σ ≥ 0.1; GA holds the plateau in 21 of 24 seeds at σ = 0.01.Source: G NE IN TOY LANDSCAPES · p. 35, Figure 15 · p. 36
  • On the rugged landscape, GA reaches the shared peak at every ripple depth, while ES at σ = 0.2 stops at local optima once a ≥ 0.8.At a = 1.6, the valley score is zero, below the highest peak’s 0.70, so reaching the shared peak requires jumping between islands.Source: G NE IN TOY LANDSCAPES · p. 35, Figure 15 · p. 36
  • GA holds the rugged shared peak at every σ ≥ 0.2, while ES holds it only at the intermediate width σ = 0.4.ES cannot see past a valley when narrower; at σ = 0.8 it reaches but ends at the shared peak in only 4 of 24 seeds.Source: G NE IN TOY LANDSCAPES · p. 35, Figure 15 · p. 36
  • With 128 parameters but intrinsic dimensionality k = 2, both methods hold the shared peak; as k grows, GA fails sooner than ES, which holds through k = 128 unless the ripple is deep.At a = 0.4, GA holds through k = 16 but never from k = 32; the table varies k, the number of rippled coordinates.Source: G NE IN TOY LANDSCAPES · pp. 35–36, Table 7 · p. 36
  • The GA’s failure at high intrinsic dimensionality concerns finding the shared peak, not keeping it; whether HalfCheetah has such a landscape remains unshown.Only on HalfCheetah do children mutated from saved GA agents lose clearly more than copies of the parent.Source: G NE IN TOY LANDSCAPES · p. 36

H NEIGHBORHOOD WIDTH AND RETURN LANDSCAPES

H.1 ROBUSTNESS OF THE WIDTH

Across robustness checks, ES remains the widest method on classic-control and MiniGrid panels, and its width lead grows during training. On tanh panels the comparison is weaker; shared-neighborhood overlap tracks the stability-plasticity trade-off, while switching outcomes vary by setting.

H.1 ROBUSTNESS OF THE WIDTH

  • 2.2–5.7 times PPO: ES remains widest across alternative neighborhood-width definitions, while ReDo-PPO is no wider than PPO.The per-panel ratios correlate with the reported definition at Spearman ρ ≥ 0.92.Source: Table 8 · p. 37, §H.1 · p. 37

H.2 SIGNIFICANCE OF NEIGHBORHOOD WIDTH

  • ES is wider than PPO in all four environments (34 on CartPole, 3.8 on Acrobot, 2.7 on MountainCar, and 5.6 on MiniGrid; each p < 0.001).The GA is wider in three environments but not MountainCar; there its mutation scale reaches 10^-5, leaving a cloud of near-copies that does not test neighborhood width.Source: §H.2 · p. 37

H.3 NEIGHBORHOOD WIDTH OVER TRAINING

  • 6.9 to 7.4 times PPO: ES’s width lead grows over training, while GA’s falls from 4.9 to 2.6 times PPO.Every method narrows as training progresses.Source: §H.3 · p. 38, Figure 18 · p. 40

H.4 NEIGHBORHOOD WIDTH ON TANH NETWORKS

  • No continual PPO variant is significantly wider than PPO on tanh panels; on Kinetix return widths, ES is as wide as PPO and GA is narrower.The hidden layers are tanh, so the authors use equal absolute noise across weights rather than relative noise.Source: §H.4 · p. 38, Table 9 · p. 39

H.5 THE SHARED NEIGHBORHOOD

  • ρ = 0.73–0.77: shared-neighborhood size correlates with LA − F across tested radii, and ES has the largest shared neighborhood in the same five settings.With moves as large as each method’s own step, MiniGrid shared-neighborhood share falls from 0.70 to 0.27 for GA and from 0.94 to 0.76 for ES; along its own path, only ReDo-PPO keeps the task further than in random directions (ratio 1.97).Source: §H.5 · pp. 40–41, Figure 19 · p. 41, Figure 20 · p. 41

H.6 RETURN LANDSCAPES OF THE TWO-TASK PANELS

  • 13%: PPO is stuck on switches across ten settings, more than TRAC-PPO (4%) and ReDo-PPO (9%), but not C-CHAIN (11%); under action reversal, ES is most often stuck.Figure 21 slices the return landscape through the centroid, scoring each point on both tasks.Source: Figure 21 · p. 42, §H.6 · pp. 42–43

I ANALYSIS OF POPULATION-BASED PPO

PBT-PPO’s advantage over other RL methods comes from giving each member fewer updates: this delays plasticity loss where PPO struggles, but does not reduce forgetting and can cost learning when tasks need many updates.

I ANALYSIS OF POPULATION-BASED PPO

  • PBT-PPO uses eight PPO learners that alternate updates; every ten updates, the worst fifth copy weights from the best fifth, and the reported agent is their weight centroid.Members retain PPO’s hyperparameters throughout training.Source: I ANALYSIS OF POPULATION-BASED PPO · p. 43
  • PBT-PPO’s wider neighborhood, behavioral diversity, member averaging, higher entropy, and trainer do not explain its advantage over PPO.Its remaining advantage is the update budget: each member makes one-eighth as many updates under matched compute.Source: I ANALYSIS OF POPULATION-BASED PPO · p. 43
  • With an eighth of PPO’s updates, a single PPO learns and keeps about as much as PBT-PPO on Acrobot and MountainCar under ten-task noise, so selection is not needed.PPO with a learning rate ten times smaller shows the same pattern.Source: Table 11 · p. 43, I ANALYSIS OF POPULATION-BASED PPO · p. 43
  • Under ten-task noise, PBT-PPO forgets more than PPO on all three classic-control tasks, while matched one-eighth-update PPO keeps about as much as PBT-PPO on CartPole but far less than ES.For action reversal, noise, and the longer pole, matched PPO keeps 0, 0.40, and 0.67 versus ES’s 0.46, 0.60, and 1.00, respectively.Source: I ANALYSIS OF POPULATION-BASED PPO · p. 43
  • On HalfCheetah, one-eighth-update PPO learns less than PBT-PPO because the task needs many updates, and copying the best member makes up the learning difference.Neither keeps much; ES keeps more in both reported noise and action-reversal settings.Source: I ANALYSIS OF POPULATION-BASED PPO · pp. 43–44
  • PBT-PPO leads the RL methods because each member makes fewer updates, which reduces plasticity loss where PPO loses it but neither reduces forgetting nor preserves learning when tasks need many updates.Fewer updates delay PPO’s learning-accuracy decline on Acrobot and MountainCar; they reduce learning on HalfCheetah.Source: I ANALYSIS OF POPULATION-BASED PPO · pp. 43–44

Related papers

Loading 2610.01583v1…