Source-linked AI summary
Continual Evolution Strategies in Control Tasks
Nicola Pitzalis, Eleni Nisioti, Antonio Carta, Davide Bacciu, Andrea Cossu
TL;DR
Continual control requires agents to adapt to changing tasks without forgetting prior behaviors, but ES remains underexplored in strictly sequential task streams. This paper evaluates sequential ES and replay on MuJoCo locomotion tasks, finding that replay improves retention and can produce selective positive transfer, while larger replay budgets reduce plasticity.
Problem
Evolution Strategies remain comparatively underexplored in continual control, where agents must adapt to changing tasks while retaining previously acquired behaviors.
Method
The study trains ES sequentially across MuJoCo locomotion tasks without parameter resets and evaluates adaptation, retention, transfer, and replay.
Results
Replay substantially improves retention and can induce selective positive transfer, but stronger replay preserves earlier tasks at the cost of reduced adaptation to later tasks.
Takeaways & Limitations
ES can support continual adaptation in control, with replay providing an effective mechanism for mitigating forgetting without gradients or architectural changes.
Takeaways & Limitations
The task-agnostic setting assumes strict parameter sharing across tasks, with only the input layer task-specific and hidden and output layers shared.
Abstract
from arXiv · showhide
We study Evolution Strategies (ES) for continual control, where agents must adapt to changing tasks without forgetting previous ones. On sequential MuJoCo locomotion tasks, naive ES suffers from severe catastrophic forgetting. Replay substantially improves retention and can induce positive transfer, while larger replay budgets reduce plasticity. Overall, these results show that ES can support continual adaptation in control and that replay is an effective mechanism for mitigating forgetting.
1 Introduction
Learning-based controllers often assume stationary environments and morphologies, but realistic changing conditions demand continual adaptation. This work characterizes Evolution Strategies in sequential MuJoCo control, showing catastrophic forgetting under naive sequential exposure and improved retention through replay.
- Motivation: Controllers optimized for stationary dynamics and morphology can degrade sharply when operating conditions change, producing brittle agents with limited adaptability.The introduction motivates continual control because realistic operating conditions cannot all be specified in advance.
- Research gap: Continual learning is well studied for gradient-based methods, whereas Evolution Strategies under strictly sequential task exposure remain largely unexplored.ES optimize distributions of candidate solutions and have demonstrated effectiveness in continuous control.
- Contributions: Naive sequential ES adapts to new sequential MuJoCo locomotion tasks but suffers from catastrophic forgetting, while replay substantially improves retention and can enable positive transfer.The study presents an empirical characterization of ES in continual control and incorporates replay naturally into the ES setting.
2 Related Work
Continual learning addresses adaptation across sequential tasks without overwriting prior knowledge, with replay as a common strategy for mitigating catastrophic forgetting. Evolutionary approaches remain relatively unexplored, particularly for ES-based control, where forgetting, transfer, and adaptation are central dimensions.
- Continual learning: Continual learning studies sequential task adaptation while preserving previously acquired knowledge, but sequential neural-network training typically causes catastrophic forgetting.Replay interleaves new and previous experiences to mitigate forgetting.
- Continual learning: Replay is among the most common strategies for mitigating catastrophic forgetting in continual learning.It combines training on new experiences with training on previous ones.
- Evolutionary approaches: Evolutionary continual adaptation has been rarely explored, with prior work combining evolutionary search and replay-based actor-critic updates or studying supervised classification.The present work focuses specifically on ES-based control and quantifies forgetting, transfer, and adaptation.
3 Continual Evolution Strategies
This section defines continual Evolution Strategies for sequential MuJoCo control, training one policy across ordered tasks without parameter resets. It evaluates adaptation, retention, and transfer using normalized episodic returns and interference-related metrics, with replay introduced to mitigate forgetting.
- Continual training: ES trains a single policy sequentially across an ordered task stream without resetting parameters, enabling evaluation of adaptation, retention, and transfer.Each task receives a fixed number of ES generations before training proceeds to the next task.
- Continual training: Experiments use Hopper-v5, Walker2d-v5, and Swimmer-v5 with shallow feedforward policies combining task-specific input projections with shared hidden and output layers.The environments differ in morphology and control structure, creating varying similarity and conflict during sequential training.
- Evolution Strategies: Parallel ES samples Gaussian perturbations, evaluates candidate policies, rank-normalizes fitness, and updates parameters using population-size and learning-rate settings.Training uses a fixed generation budget for each task.
- Replay: Replay adds fitness evaluations on previously encountered tasks, jointly rank-normalizing all returns so updates favor solutions that retain performance across tasks.Replay samples receive the same weight as standard ES samples and alter only the returns used to estimate the update.
- Evaluation: Performance is measured with normalized episodic returns alongside backward transfer, forward transfer, and acquisition interference to characterize forgetting, facilitation, and inhibition.Negative BWT indicates forgetting, positive BWT indicates beneficial backward transfer, positive FWT indicates facilitated future learning, and interference below zero indicates facilitation.
4 Experiments
Sequential ES adapts to newly introduced control tasks but suffers catastrophic forgetting, while replay substantially improves retention and preserves acquisition ability. Replay also reveals a stability–plasticity trade-off and can produce selective positive transfer across task orderings.
- Task-agnostic continual learning: Sequential ES adapts to each new task but sharply forgets earlier behaviors, including Hopper after later tasks and Swimmer after transition to Walker2d.The setting uses shared hidden and output layers without parameter resets across tasks.
- Replay and forgetting: Replay substantially reduces post-switch performance collapse, stabilizes previously learned tasks, and still permits acquisition of newly encountered tasks.Replay acts as a regularizer that biases search toward parameter regions preserving performance across tasks.
- Replay and forgetting: Intermediate replay provides the best stability–plasticity compromise, improving retention without strongly impairing learning on later tasks.At the largest budget, earlier tasks are preserved more effectively, but later tasks are fitted less efficiently.
- Replay and forgetting: Intermediate replay reduces acquisition interference, whereas overly strong stabilization at the largest budget increases interference and hinders learning new tasks.Little or no replay causes stronger interference during later-task learning.
- Knowledge transfer: Transfer is limited and asymmetric: backward transfer is generally near zero, with positive Hopper←Swimmer BWT, while positive FWT occurs mainly toward Swimmer from Hopper and Walker2d.The Hopper←Swimmer exception has high variability across curricula, whereas transfer from Swimmer to legged agents is weak or negative.
5 Discussion and Future Work
In sequential locomotion tasks, naive ES suffers catastrophic forgetting, whereas replay improves retention and can induce positive transfer. However, stronger replay creates a stability–plasticity trade-off, and future work should test additional continual-learning strategies for black-box evolutionary optimization.
- Discussion: Naive sequential ES suffers catastrophic forgetting, while replay substantially improves retention and can induce positive transfer.Increasing replay can better preserve earlier tasks but may reduce adaptation to later ones.
- Discussion: Transfer is selective and asymmetric: forward transfer is generally limited, backward transfer stays near zero under replay, and benefits depend on task-pair compatibility.Beneficial transfer emerges only for specific task pairs, likely reflecting compatibility between their parameter optima.
- Future Work: Future work should investigate adapting regularization-based methods, including distillation and importance weighting, to black-box evolutionary optimization.These methods are proposed as additional continual-learning strategies to combine with ES.
A Algorithm
The continual ES procedure trains sequentially across ordered tasks without resetting policy parameters, while replay interleaves evaluations on earlier tasks during selected generations.
- A Algorithm: Training proceeds sequentially over an ordered stream of tasks.
- A Algorithm: Policy parameters are not reset between tasks.
- A Algorithm: Replay interleaves additional evaluations on previously encountered tasks during selected ES generations.
B The task-aware variant
The task-aware variant preserves task-specific action mappings by assigning each task a distinct output head, while retaining task-specific input projections and a shared hidden representation. This contrasts with the task-agnostic variant’s single shared output head, which can create interference across heterogeneous tasks.
- Comparison: Figure 6 places the task-agnostic architecture in the top panel and the task-aware multi-head architecture in the bottom panel.The task-agnostic version is used in the main continual setting, whereas the task-aware variant is discussed in supplementary results.
- Architecture: Policies use shallow feedforward neural networks with task-specific input projection layers for heterogeneous observation spaces.The hidden layer serves as the main shared representational bottleneck across tasks.
- Output heads: The task-agnostic variant uses a single shared output head for all tasks.Its output parameters must express action policies for morphologically different agents and environments.
- Output heads: The task-aware variant assigns each task a distinct output head, making the final action mapping task-specific.Previously learned action mappings are isolated in separate heads, so updates on a new task do not overwrite those output parameters.
- Continual-control motivation: Shared output parameters can acquire task-dependent semantics, making the output layer a direct source of interference and forgetting.The task-aware multi-head design instead isolates previously learned action mappings across tasks.
C Baselines
The baseline experiments establish single-task learning curves and unnormalized rewards for three MuJoCo tasks, which provide denominators for continual-performance normalization. They also include a non-continual multi-task comparison trained by alternating among all tasks.
- Single-task baselines: Single-task baselines report learning curves and unnormalized rewards for Hopper-v5, Swimmer-v5, and Walker2d-v5.These results serve as the denominators for normalizing continual learners’ performance.
- Multi-task baseline: The multi-task baseline trains one agent on a combined dataset containing all tasks in a non-continual setting.The agent alternates between environments step by step, with results reported in Table 2.
- Single-task baselines: Figure 7 shows normalized average reward over training iterations for policies trained from scratch on each individual MuJoCo task.Normalization uses the maximum reward achieved in the corresponding single-task run.
D Task-aware continual control
Task-aware continual control uses task-specific input and output layers with a shared hidden representation, substantially reducing catastrophic forgetting. The remaining interference comparison indicates that conflicts primarily arise from the shared action-mapping output layer, while shared representations can support positive transfer.
- Task-aware architecture: Task-aware policies share only the hidden representation, while input and output layers remain task-specific and task identity is available at inference.This setup largely removes output-level forgetting by construction, isolating the role of the shared hidden representation.
- Learning dynamics: Previously learned Hopper, Swimmer, and Walker2d tasks remain largely stable after transitions under the distinct-output architecture.Some learning curves slightly improve after subsequent tasks are introduced, suggesting positive knowledge transfer through the shared hidden representation.
- Interference: The results identify the shared output layer, rather than the shared representation, as a major source of forgetting in these control environments.Substantially reduced interference with task-specific heads indicates that conflict primarily lies in the action mapping.
- Interference: Interference is lower in nearly all task permutations with task-specific outputs than with the replay-based shared-output model.The comparison evaluates acquisition of the second and third tasks across task sequences, with the shared-output baseline using 192 replay steps.
E Replay budget study
Replay budget creates a stability–plasticity trade-off in the fixed Hopper → Swimmer → Walker2d sequence. Intermediate replay best balances retention and adaptation, while excessive replay constrains learning and increases interference.
- Replay budget study: 12, 192, and 288 replay perturbations reveal a stability–plasticity trade-off: smaller budgets preserve plasticity but weaken retention, whereas intermediate replay best balances retention and adaptation.These configurations were compared in the shared-output setting for Hopper → Swimmer → Walker2d.
- Replay budget study: Moderate replay substantially reduces interference during acquisition on later tasks, but excessive replay increases interference again.The effect was measured under increasing replay budgets for Hopper → Swimmer → Walker2d.
- Replay budget study: Replay stabilizes learning only up to a point; additional replay eventually constrains learning and hinders adaptation.This conclusion follows from the combined retention, adaptation, and interference patterns across replay budgets.
F All permutations
Across all six task permutations, shared-output agents show strong order dependence and substantial forgetting after task transitions. Distinct output heads improve stability, while replay is evaluated across budgets and task orders, including its effects on backward and forward transfer.
- Shared output: Shared-output learning curves across all six task orders show strong dependence on task order and pronounced drops on previously learned tasks after transitions.These drops indicate substantial forgetting in sequential continual learning.
- Distinct output: Distinct output heads improve stability across task permutations, with step-wise best and worst returns summarized for each order.The distinct-output results are reported across all six task orders.
- Replay budgets: For Hopper →Swimmer →Walker2d, shared-output replay is tested with +12 steps (∼1.5%), +192 steps (∼25%), and +288 steps (∼37.5%).Figure 10 reports normalized reward curves under these three replay budgets.
- Replay: Replay is evaluated for all six task orders using maximum and minimum rewards, enabling comparison with the shared-output setting.The replay condition is summarized in Table 4 and illustrated with sequential learning curves.
- Transfer: Backward transfer is reported for shared-output policies with and without replay and for distinct-output policies, while forward transfer is reported for shared- and distinct-output settings.The transfer analyses cover task-wise BWT and FWT over training iterations across task orderings.