PaperlayerView exact revision on arXiv

Source-linked AI summary

Teacher algorithms for curriculum learning of Deep RL in continuously parameterized environments

Rémy Portelas, Cédric Colas, Katja Hofmann, Pierre-Yves Oudeyer

arXiv:1910.07224v1cs.LGcs.ROstat.ML

TL;DR

The paper asks how a teacher can train an unknown DRL student across diverse, continuously parameterized environments. It formulates curriculum generation as a continuous bandit problem driven by absolute learning progress and introduces ALP-GMM. Across BipedalWalker benchmarks, ALP-GMM consistently outperformed random heuristics and sometimes expert-designed curricula, including settings with unfeasible tasks and irrelevant dimensions.

  • Problem

    The central challenge is ordering procedurally generated environments for an unknown DRL student when some regions may be difficult or unlearnable.

    Source: §1 · p. 1, §3 · p. 3

  • Method

    The paper formulates teacher sampling as a continuous bandit problem and introduces ALP-GMM, which models absolute learning progress with Gaussian mixtures.

    Source: §1 · pp. 1–2, §4.1 · p. 3

  • Results

    ALP-GMM consistently outperformed random heuristics and occasionally expert-designed curricula across difficult parameter spaces with irrelevant dimensions and many unfeasible tasks.

    Source: §6 · p. 8

  • Takeaways & Limitations

    Learning-progress-based teachers can guide generalist DRL agents across continuously parameterized environments without prior knowledge of student abilities or precise task-space structure.

    Source: §1 · p. 1, §6 · p. 8

    The teacher assumes difficulty varies as a non-linear, piece-wise smooth function, while the parameter space may contain unfeasible or irrelevant dimensions.

    Source: §3 · p. 3

Abstract

from arXiv · show

We consider the problem of how a teacher algorithm can enable an unknown Deep Reinforcement Learning (DRL) student to become good at a skill over a wide range of diverse environments. To do so, we study how a teacher algorithm can learn to generate a learning curriculum, whereby it sequentially samples parameters controlling a stochastic procedural generation of environments. Because it does not initially know the capacities of its student, a key challenge for the teacher is to discover which environments are easy, difficult or unlearnable, and in what order to propose them to maximize the efficiency of learning over the learnable ones. To achieve this, this problem is transformed into a surrogate continuous bandit problem where the teacher samples environments in order to maximize absolute learning progress of its student. We present a new algorithm modeling absolute learning progress with Gaussian mixture models (ALP-GMM). We also adapt existing algorithms and provide a complete study in the context of DRL. Using parameterized variants of the BipedalWalker environment, we study their efficiency to personalize a learning curriculum for different learners (embodiments), their robustness to the ratio of learnable/unlearnable environments, and their scalability to non-linear and high-dimensional parameter spaces. Videos and code are available at https://github.com/flowersteam/teachDeepRL.

1 Introduction

The paper studies learning-progress-based teachers for generalist DRL agents in continuously parameterized environments. It introduces ALP-GMM, the CTS setup, and benchmark environments for testing curriculum personalization and robustness.

  • The study evaluates whether learning-progress-based curriculum methods can scaffold generalist DRL agents in continuously parameterized environments.Source: §1 · pp. 1–2
  • ALP-GMM models absolute learning progress with Gaussian mixtures, while RIAC is adapted to the same setting.Source: §1 · pp. 1–2
  • The benchmark compares teachers across different learner morphologies and contrasts the approach with POET’s population of agents specialized to deterministic environments.Source: §1 · pp. 1–2
  • The CTS framework handles parameter spaces that encode distributions of environments without assuming a direct parameter-to-environment mapping.Source: §1 · pp. 1–2
  • Two parameterized BipedalWalker environments benchmark curriculum learning over continuous spaces with procedural generation.Source: §1 · p. 2

2 Related work

Related work applies curriculum learning and learning progress across supervised learning, reinforcement learning, robotics, and education. This paper extends teacher-based curriculum learning to continuous environment spaces containing potentially unlearnable regions.

  • Curriculum learning orders tasks to improve an agent’s performance on one or more target tasks.Source: §2 · p. 2
  • Learning progress has supported automated curricula in developmental robotics, neural sequence models, and personalized educational exercise sequences.Source: §2 · p. 2
  • Earlier Teacher-Student work studied discrete environment sets, whereas this paper addresses continuous parameter spaces that map to task distributions.Source: §2 · p. 2
  • Unlike work varying only reward-defined goals, this study varies environment state spaces and avoids requiring an externally specified intermediate-difficulty range.Source: §2 · p. 2

3 The Continuous Teacher-Student Framework (CTS)

CTS formalizes teacher-student interaction when parameters select distributions of episodic tasks. The teacher uses observed rewards and interaction history to choose parameters while treating the student as a black box.

  • In CTS, teachers sample parameters from a continuous space, with each parameter directly selecting a task distribution.Source: §3 · pp. 2–3
  • A student interacts with episodic POMDP tasks by receiving observations, choosing actions, and collecting rewards.Source: §3 · p. 3
  • The teacher proposes tasks from a selected distribution, records average episodic reward, and stores parameter-reward history for future sampling.Source: §3 · p. 3
  • The teacher maximizes final competence across the parameter space while relying only on interaction history, without access to the student’s internal state or capabilities.Source: §3 · p. 3
  • CTS permits non-linear, piece-wise smooth difficulty and parameter spaces containing unfeasible regions or irrelevant dimensions.Source: §3 · p. 3
  • The reported study restricts interactions to one sampled task per parameter and weights all parameter subspaces equally.Source: §3 · p. 3

4 Methods

The methods use absolute learning progress to guide sampling in continuously parameterized procedural environments. ALP-GMM fits Gaussian mixtures to recent parameter-progress data and is evaluated against RIAC, random sampling, and an expert curriculum across BipedalWalker variants.

Absolute learning progress

  • Teacher algorithms select parameter subspaces as bandit arms and use absolute learning progress to prioritize regions with changing competence.Source: §4.1 · p. 3
  • RIAC partitions the parameter space into hyperboxes and samples within regions according to their absolute learning progress.Source: §4.1 · p. 3

ALP-GMM

  • ALP-GMM fits a Gaussian mixture to previously sampled parameters and their ALP values, then samples from components with high ALP utility.Source: §4.1 · p. 4
  • For each new parameter, ALP is computed as the absolute reward difference from the closest previously sampled parameter.Source: §4.1 · p. 4
  • ALP-GMM fits mixtures on the most recent 250 parameter-ALP pairs, selects the number of components using Akaike’s Information Criterion, and retains all history for nearest-neighbor ALP.Source: §4.1 · p. 4

Baselines

  • The evaluation compares ALP-based teachers with random parameter sampling and an expert curriculum that moves from easier to more complex regions using performance thresholds.Source: §4.2 · pp. 4–5

Environments

  • The benchmark uses procedurally generated BipedalWalker tracks, including stump tracks and a challenging 12-dimensional Hexagon Tracks environment with two distractor dimensions.Source: §4.3 · p. 5

Students

  • Experiments use Soft Actor-Critic students with three walker morphologies, including the default biped, a short-legged biped, and a larger quadruped.Source: §4.3 · p. 5, Figure 1 · p. 5

5 Experimental Results

The experiments evaluate teacher algorithms across student embodiments, increasing unfeasible-task proportions, and complex high-dimensional parameter spaces. ALP-GMM generally provides the strongest or most robust performance, while learning-progress teachers outperform random sampling.

Evaluation setup

  • The evaluation measures mastered track distributions using a binary competence threshold of episodic reward 230.Results compare teacher conditions across seeded runs and training horizons.Source: §5 · p. 5

Curriculum emergence

  • ALP-GMM focuses sampling on regions where the student shows high absolute learning progress, producing an emergent curriculum in representative Stump Tracks runs.Sampling shifts toward track distributions with stump heights of 1 to 1.8 and favorable spacing values.Source: §5.1 · p. 6, Figure 2 · p. 6

Stump Tracks comparisons

  • 14.9% final mean performance for ALP-GMM exceeded Covar-GMM at 10.6% and RIAC at 8.6% for short walkers, while all learning-progress teachers surpassed Random.ALP-GMM was significantly better than RIAC but not significantly different from Covar-GMM in this condition.Source: §5.1 · p. 6
  • ALP-GMM significantly outperformed Oracle, RIAC, and Covar-GMM for default bipedal walkers, whereas quadrupedal walkers mastered nearly 100% of the test set under most conditions.Oracle performed significantly worse than the other methods for quadrupedal walkers.Source: §5.1 · pp. 6–7

Unfeasible tasks

  • Learning-progress teachers partially maintain performance as unfeasible tasks increase, with ALP-GMM showing the strongest robustness among the tested conditions.The experiments vary stump-height ranges to increase the proportion of unfeasible tracks.Source: §5.2 · pp. 7–8

High-dimensional task spaces

  • 80% final average performance for ALP-GMM exceeded Oracle at 68% in 12-dimensional Hexagon Tracks, and ALP-GMM also significantly surpassed Covar-GMM, RIAC, and Random.The setting includes irrelevant dimensions, non-linear difficulty gradients, and initially unfeasible tasks.Source: §5.3 · p. 8
  • Additional abstract experiments without DRL students found that GMM-based approaches performed best as dimensionality and the proportion of initially unfeasible subspaces increased.These experiments complement the DRL evaluations across relevant and irrelevant dimensions.Source: §5.3 · p. 8

6 Discussion and Conclusion

LP-based teacher algorithms can guide DRL agents through difficult continuous environments containing irrelevant dimensions and many unfeasible tasks. Without prior knowledge of student abilities or precise task-space structure, ALP-GMM consistently outperformed random heuristics and sometimes expert-designed curricula.

  • LP-based teachers successfully guided DRL agents in environments with irrelevant dimensions and large proportions of unfeasible tasks.Source: §6 · p. 8
  • ALP-GMM uses few crucial hyperparameters and is described as conceptually simple for curriculum-learning applications inside and outside DRL.Source: §6 · p. 8

A Experiments on an n-dimensional toy parameter space

Toy-space experiments evaluated curriculum teachers as meaningful dimensions, irrelevant dimensions, and the number of hypercubes increased. GMM-based methods generally scaled better than RIAC, although Covar-GMM outperformed ALP-GMM in these toy spaces.

Toy-space setup

Reference experiment

Meaningful dimensions

Irrelevant dimensions

Overall scaling

B Implementation details

The implementation combines SAC students with modified RIAC, Covar-GMM, Oracle, and ALP-GMM teacher procedures. Oracle advances a sampling window using reward thresholds, whereas ALP-GMM models recent learning progress and Oracle can become trapped in unmasterable regions.

Student implementation

RIAC

  • Modified RIAC attempts to prevent oversplitting through capacity limits, candidate split constraints, minimum region sizes, and mixed sampling strategies.Source: B Implementation details · p. 13

Covar-GMM

  • Covar-GMM selects Gaussian components using positive covariance between relative time and episodic reward after concatenating parameters with returns and time.Source: B Implementation details · p. 13

Oracle limitation

  • Oracle’s sampling window reached an unmasterable region after 10,500 episodes and remained there through 15,000 episodes, causing forgetting issues.Source: B Implementation details · p. 13

Oracle

C Additional visualizations for Stump Tracks experiments

Additional Stump Tracks visualizations show how teacher sampling evolves across task parameters and how final performance changes with agent type and unfeasible tracks. Oracle can focus on unmasterable regions, while ALP-GMM remains robust as unfeasibility increases.

Oracle sampling

  • Oracle initially sampled easy Stump Tracks, shifted toward intermediate stump heights and spacings, then reached a challenging subspace the student could not master.Source: Figure 7 · p. 15

RIAC sampling

  • RIAC initially sampled randomly, then focused on low stump heights where the SAC student made progress before spreading toward higher stump heights.Source: Figure 8 · p. 15

Final performance

  • For short agents, Random mastered 0% of the test track distributions in nearly all runs, with one run mastering three distributions.Source: Figure 9 · p. 16

Robustness to unfeasibility

  • ALP-GMM maintained a statistically significant performance advantage over other conditions across all three settings as unfeasible tracks increased.Source: Figure 10 · p. 16

D Additional visualization for Hexagon Tracks experiments

After 80M environment steps on Hexagon Tracks, ALP-GMM achieved the strongest and most consistent final performance, while some other conditions ended with complete loss of test performance.

E Parameterized BipedalWalker Environments

The study uses parameterized BipedalWalker environments with procedurally generated tracks and obstacle geometries, spanning multiple observation and parameter dimensions.

  • The two bipedal walkers use 24-dimensional observation vectors, while the quadrupedal walker uses 34 dimensions.Observations include lidar distances, body kinematics, joint states, and leg-ground contact indicators.Source: E Parameterized BipedalWalker… · p. 18
  • Stump Tracks vary mean stump height µ_h over [0, 3] and stump spacing Δ_s over [0, 6].These bounds define the continuous parameter ranges for the procedurally generated tracks.Source: E Parameterized BipedalWalker… · p. 18
  • Hexagon Tracks use a 12-dimensional parameter space with every dimension bounded in [0, 4].The parameters control procedurally generated obstacle configurations.Source: E Parameterized BipedalWalker… · p. 18, Figure 13 · p. 19
  • Figures 12 and 14 show examples of randomly generated Stump Tracks and Hexagon Tracks, respectively.Figure 13 additionally illustrates how the parameterized hexagonal obstacles are generated.Source: Figure 12 · p. 18, Figure 14 · p. 19, Figure 13 · p. 19
  • In Hexagon Tracks, the first 10 parameter values positively offset obstacle vertices along x and y.The first and last vertices’ y positions remain constrained so each obstacle retains at least one ground-contacting edge.Source: Figure 13 · p. 19

Related papers

Loading 1910.07224v1…