Source-linked AI summary
Teacher algorithms for curriculum learning of Deep RL in continuously parameterized environments
Rémy Portelas, Cédric Colas, Katja Hofmann, Pierre-Yves Oudeyer
TL;DR
The paper asks how a teacher can train an unknown DRL student across diverse, continuously parameterized environments. It formulates curriculum generation as a continuous bandit problem driven by absolute learning progress and introduces ALP-GMM. Across BipedalWalker benchmarks, ALP-GMM consistently outperformed random heuristics and sometimes expert-designed curricula, including settings with unfeasible tasks and irrelevant dimensions.
Problem
The central challenge is ordering procedurally generated environments for an unknown DRL student when some regions may be difficult or unlearnable.
Method
The paper formulates teacher sampling as a continuous bandit problem and introduces ALP-GMM, which models absolute learning progress with Gaussian mixtures.
Results
ALP-GMM consistently outperformed random heuristics and occasionally expert-designed curricula across difficult parameter spaces with irrelevant dimensions and many unfeasible tasks.
Takeaways & Limitations
Learning-progress-based teachers can guide generalist DRL agents across continuously parameterized environments without prior knowledge of student abilities or precise task-space structure.
Takeaways & Limitations
The teacher assumes difficulty varies as a non-linear, piece-wise smooth function, while the parameter space may contain unfeasible or irrelevant dimensions.
Abstract
from arXiv · showhide
We consider the problem of how a teacher algorithm can enable an unknown Deep Reinforcement Learning (DRL) student to become good at a skill over a wide range of diverse environments. To do so, we study how a teacher algorithm can learn to generate a learning curriculum, whereby it sequentially samples parameters controlling a stochastic procedural generation of environments. Because it does not initially know the capacities of its student, a key challenge for the teacher is to discover which environments are easy, difficult or unlearnable, and in what order to propose them to maximize the efficiency of learning over the learnable ones. To achieve this, this problem is transformed into a surrogate continuous bandit problem where the teacher samples environments in order to maximize absolute learning progress of its student. We present a new algorithm modeling absolute learning progress with Gaussian mixture models (ALP-GMM). We also adapt existing algorithms and provide a complete study in the context of DRL. Using parameterized variants of the BipedalWalker environment, we study their efficiency to personalize a learning curriculum for different learners (embodiments), their robustness to the ratio of learnable/unlearnable environments, and their scalability to non-linear and high-dimensional parameter spaces. Videos and code are available at https://github.com/flowersteam/teachDeepRL.
1 Introduction
The paper studies learning-progress-based teachers for generalist DRL agents in continuously parameterized environments. It introduces ALP-GMM, the CTS setup, and benchmark environments for testing curriculum personalization and robustness.
- The study evaluates whether learning-progress-based curriculum methods can scaffold generalist DRL agents in continuously parameterized environments.
- ALP-GMM models absolute learning progress with Gaussian mixtures, while RIAC is adapted to the same setting.
- The benchmark compares teachers across different learner morphologies and contrasts the approach with POET’s population of agents specialized to deterministic environments.
- The CTS framework handles parameter spaces that encode distributions of environments without assuming a direct parameter-to-environment mapping.
- Two parameterized BipedalWalker environments benchmark curriculum learning over continuous spaces with procedural generation.
2 Related work
Related work applies curriculum learning and learning progress across supervised learning, reinforcement learning, robotics, and education. This paper extends teacher-based curriculum learning to continuous environment spaces containing potentially unlearnable regions.
- Curriculum learning orders tasks to improve an agent’s performance on one or more target tasks.
- Learning progress has supported automated curricula in developmental robotics, neural sequence models, and personalized educational exercise sequences.
- Earlier Teacher-Student work studied discrete environment sets, whereas this paper addresses continuous parameter spaces that map to task distributions.
- Unlike work varying only reward-defined goals, this study varies environment state spaces and avoids requiring an externally specified intermediate-difficulty range.
3 The Continuous Teacher-Student Framework (CTS)
CTS formalizes teacher-student interaction when parameters select distributions of episodic tasks. The teacher uses observed rewards and interaction history to choose parameters while treating the student as a black box.
- In CTS, teachers sample parameters from a continuous space, with each parameter directly selecting a task distribution.
- A student interacts with episodic POMDP tasks by receiving observations, choosing actions, and collecting rewards.
- The teacher proposes tasks from a selected distribution, records average episodic reward, and stores parameter-reward history for future sampling.
- The teacher maximizes final competence across the parameter space while relying only on interaction history, without access to the student’s internal state or capabilities.
- CTS permits non-linear, piece-wise smooth difficulty and parameter spaces containing unfeasible regions or irrelevant dimensions.
- The reported study restricts interactions to one sampled task per parameter and weights all parameter subspaces equally.
4 Methods
The methods use absolute learning progress to guide sampling in continuously parameterized procedural environments. ALP-GMM fits Gaussian mixtures to recent parameter-progress data and is evaluated against RIAC, random sampling, and an expert curriculum across BipedalWalker variants.
- Absolute learning progress: Teacher algorithms select parameter subspaces as bandit arms and use absolute learning progress to prioritize regions with changing competence.
- Absolute learning progress: RIAC partitions the parameter space into hyperboxes and samples within regions according to their absolute learning progress.
- ALP-GMM: ALP-GMM fits a Gaussian mixture to previously sampled parameters and their ALP values, then samples from components with high ALP utility.
- ALP-GMM: For each new parameter, ALP is computed as the absolute reward difference from the closest previously sampled parameter.
- ALP-GMM: ALP-GMM fits mixtures on the most recent 250 parameter-ALP pairs, selects the number of components using Akaike’s Information Criterion, and retains all history for nearest-neighbor ALP.
- Baselines: The evaluation compares ALP-based teachers with random parameter sampling and an expert curriculum that moves from easier to more complex regions using performance thresholds.
- Environments: The benchmark uses procedurally generated BipedalWalker tracks, including stump tracks and a challenging 12-dimensional Hexagon Tracks environment with two distractor dimensions.
- Students: Experiments use Soft Actor-Critic students with three walker morphologies, including the default biped, a short-legged biped, and a larger quadruped.
5 Experimental Results
The experiments evaluate teacher algorithms across student embodiments, increasing unfeasible-task proportions, and complex high-dimensional parameter spaces. ALP-GMM generally provides the strongest or most robust performance, while learning-progress teachers outperform random sampling.
- Evaluation setup: The evaluation measures mastered track distributions using a binary competence threshold of episodic reward 230.Results compare teacher conditions across seeded runs and training horizons.
- Curriculum emergence: ALP-GMM focuses sampling on regions where the student shows high absolute learning progress, producing an emergent curriculum in representative Stump Tracks runs.Sampling shifts toward track distributions with stump heights of 1 to 1.8 and favorable spacing values.
- Stump Tracks comparisons: 14.9% final mean performance for ALP-GMM exceeded Covar-GMM at 10.6% and RIAC at 8.6% for short walkers, while all learning-progress teachers surpassed Random.ALP-GMM was significantly better than RIAC but not significantly different from Covar-GMM in this condition.
- Stump Tracks comparisons: ALP-GMM significantly outperformed Oracle, RIAC, and Covar-GMM for default bipedal walkers, whereas quadrupedal walkers mastered nearly 100% of the test set under most conditions.Oracle performed significantly worse than the other methods for quadrupedal walkers.
- Unfeasible tasks: Learning-progress teachers partially maintain performance as unfeasible tasks increase, with ALP-GMM showing the strongest robustness among the tested conditions.The experiments vary stump-height ranges to increase the proportion of unfeasible tracks.
- High-dimensional task spaces: 80% final average performance for ALP-GMM exceeded Oracle at 68% in 12-dimensional Hexagon Tracks, and ALP-GMM also significantly surpassed Covar-GMM, RIAC, and Random.The setting includes irrelevant dimensions, non-linear difficulty gradients, and initially unfeasible tasks.
- High-dimensional task spaces: Additional abstract experiments without DRL students found that GMM-based approaches performed best as dimensionality and the proportion of initially unfeasible subspaces increased.These experiments complement the DRL evaluations across relevant and irrelevant dimensions.
6 Discussion and Conclusion
LP-based teacher algorithms can guide DRL agents through difficult continuous environments containing irrelevant dimensions and many unfeasible tasks. Without prior knowledge of student abilities or precise task-space structure, ALP-GMM consistently outperformed random heuristics and sometimes expert-designed curricula.
- LP-based teachers successfully guided DRL agents in environments with irrelevant dimensions and large proportions of unfeasible tasks.
- ALP-GMM uses few crucial hyperparameters and is described as conceptually simple for curriculum-learning applications inside and outside DRL.
A Experiments on an n-dimensional toy parameter space
Toy-space experiments evaluated curriculum teachers as meaningful dimensions, irrelevant dimensions, and the number of hypercubes increased. GMM-based methods generally scaled better than RIAC, although Covar-GMM outperformed ALP-GMM in these toy spaces.
- Toy-space setup: The toy space maps parameters to history-dependent rewards and divides the parameter space into hypercubes for controlled student-learning experiments.
- Toy-space setup: Unlocked hypercubes yield increasing positive rewards, while locked hypercubes yield none; sampling 75 parameters in an unlocked hypercube unlocks its neighbors.
- Reference experiment: All learning-progress methods significantly outperformed Random in the 2D reference space, with Covar-GMM performing best.
- Meaningful dimensions: Increasing meaningful dimensions harmed RIAC more than GMM-based approaches; in 6D, ALP-GMM and Covar-GMM reached 100% median performance after 1M episodes.
- Irrelevant dimensions: With 20 or 50 irrelevant dimensions, ALP-GMM performed best throughout training, while with 10 irrelevant dimensions it reached 100% median performance after 52k episodes.
- Overall scaling: GMM-based approaches scaled better than RIAC as relevant or irrelevant dimensions and initially unfeasible hypercubes increased, while Covar-GMM exceeded ALP-GMM in toy spaces.
B Implementation details
The implementation combines SAC students with modified RIAC, Covar-GMM, Oracle, and ALP-GMM teacher procedures. Oracle advances a sampling window using reward thresholds, whereas ALP-GMM models recent learning progress and Oracle can become trapped in unmasterable regions.
- Student implementation: SAC was used for all DRL experiments with identical two-layer (400, 300) Q, V, and policy networks using ReLU activations.
- RIAC: Modified RIAC attempts to prevent oversplitting through capacity limits, candidate split constraints, minimum region sizes, and mixed sampling strategies.
- Covar-GMM: Covar-GMM selects Gaussian components using positive covariance between relative time and episodic reward after concatenating parameters with returns and time.
- Oracle limitation: Oracle’s sampling window reached an unmasterable region after 10,500 episodes and remained there through 15,000 episodes, causing forgetting issues.
- Oracle: Oracle moves its sampling window when recent mean competence exceeds a reward threshold, using a fixed window step and bounded parameter space.
C Additional visualizations for Stump Tracks experiments
Additional Stump Tracks visualizations show how teacher sampling evolves across task parameters and how final performance changes with agent type and unfeasible tracks. Oracle can focus on unmasterable regions, while ALP-GMM remains robust as unfeasibility increases.
- Oracle sampling: Oracle initially sampled easy Stump Tracks, shifted toward intermediate stump heights and spacings, then reached a challenging subspace the student could not master.
- RIAC sampling: RIAC initially sampled randomly, then focused on low stump heights where the SAC student made progress before spreading toward higher stump heights.
- Final performance: For short agents, Random mastered 0% of the test track distributions in nearly all runs, with one run mastering three distributions.
- Robustness to unfeasibility: ALP-GMM maintained a statistically significant performance advantage over other conditions across all three settings as unfeasible tracks increased.
D Additional visualization for Hexagon Tracks experiments
After 80M environment steps on Hexagon Tracks, ALP-GMM achieved the strongest and most consistent final performance, while some other conditions ended with complete loss of test performance.
- ALP-GMM had the highest median performance and narrowest distribution after 80M environment steps.Figure 11 summarizes each condition with medians, quartiles, whiskers, and outliers.
- Only Oracle and ALP-GMM achieved positive final performance in all 32 repeats.Covar-GMM, RIAC, and Random ended with 0% performance in 8/32, 5/32, and 16/25 runs, respectively.
- Every condition temporarily mastered part of the test set, but some runs later ended at 0% because of catastrophic forgetting.The reported distributions therefore distinguish final performance from mastery achieved earlier during training.
E Parameterized BipedalWalker Environments
The study uses parameterized BipedalWalker environments with procedurally generated tracks and obstacle geometries, spanning multiple observation and parameter dimensions.
- The two bipedal walkers use 24-dimensional observation vectors, while the quadrupedal walker uses 34 dimensions.Observations include lidar distances, body kinematics, joint states, and leg-ground contact indicators.
- Stump Tracks vary mean stump height µ_h over [0, 3] and stump spacing Δ_s over [0, 6].These bounds define the continuous parameter ranges for the procedurally generated tracks.
- Hexagon Tracks use a 12-dimensional parameter space with every dimension bounded in [0, 4].The parameters control procedurally generated obstacle configurations.
- Figures 12 and 14 show examples of randomly generated Stump Tracks and Hexagon Tracks, respectively.Figure 13 additionally illustrates how the parameterized hexagonal obstacles are generated.
- In Hexagon Tracks, the first 10 parameter values positively offset obstacle vertices along x and y.The first and last vertices’ y positions remain constrained so each obstacle retains at least one ground-contacting edge.