Source-linked AI summary
Teacher-Student Curriculum Learning
Tambet Matiisen, Avital Oliver, Taco Cohen, John Schulman
TL;DR
Sparse-reward tasks make direct training and curriculum design difficult, motivating automatic selection of subtasks. TSCL has a Teacher choose tasks based on Student learning progress and forgetting, and it matches or exceeds hand-crafted curricula across decimal addition and Minecraft navigation. The authors also identify discrete task parameterization and omitted exploration-bonus comparisons as scope boundaries.
Problem
Sparse-reward tasks are challenging for direct reinforcement learning, while manual curricula require difficult task ordering, mastery thresholds, and anti-forgetting mixtures.
Method
TSCL uses a Teacher to select Student subtasks and proposes algorithms that prioritize high learning progress while revisiting tasks whose performance is worsening.
Results
TSCL matches or surpasses carefully designed curricula on decimal addition and Minecraft navigation, including a Window algorithm matching a manual curriculum on the challenging 5-task Minecraft problem.
Takeaways & Limitations
Automatic learning-progress curricula can support both supervised and reinforcement learning and can outperform uniform sampling in the reported decimal-addition and Minecraft experiments.
Takeaways & Limitations
The work considers only discrete task parameterizations and leaves automatic curricula with undefined subtasks and comparisons with exploration bonuses for future work.
Abstract
from arXiv · showhide
We propose Teacher-Student Curriculum Learning (TSCL), a framework for automatic curriculum learning, where the Student tries to learn a complex task and the Teacher automatically chooses subtasks from a given set for the Student to train on. We describe a family of Teacher algorithms that rely on the intuition that the Student should practice more those tasks on which it makes the fastest progress, i.e. where the slope of the learning curve is highest. In addition, the Teacher algorithms address the problem of forgetting by also choosing tasks where the Student's performance is getting worse. We demonstrate that TSCL matches or surpasses the results of carefully hand-crafted curricula in two tasks: addition of decimal numbers with LSTM and navigation in Minecraft. Using our automatically generated curriculum enabled to solve a Minecraft maze that could not be solved at all when training directly on solving the maze, and the learning was an order of magnitude faster than uniform sampling of subtasks.
1 Introduction
TSCL addresses the difficulty of designing curricula for sparse-reward tasks by letting a Teacher automatically select subtasks for a Student. Its algorithms use learning progress to emphasize rapidly improving or forgotten tasks, and the framework is evaluated on supervised and reinforcement-learning problems.
- Sparse-reward tasks remain difficult because random exploration may require exponentially more samples as the number of steps to reward increases.
- Hand-designed curricula require ordering subtasks, choosing mastery thresholds, and mixing easier tasks to prevent forgetting.
- TSCL has a Teacher monitor the Student and select training tasks at each step to maximize progression through the curriculum.
- The Teacher algorithms prioritize tasks with the steepest learning-curve slopes and revisit tasks whose performance is declining.
- The paper formalizes TSCL as a POMDP, proposes learning-progress algorithms that address forgetting, and evaluates them on decimal addition and Minecraft navigation.
2 Teacher-Student Setup
The Teacher-Student setup treats curriculum selection as an interaction in which a Teacher chooses tasks and receives Student scores. The paper formalizes this interaction with simple and batch POMDPs and defines rewards from changes in task performance.
- At each timestep, the Teacher chooses subtasks, the Student trains on them, and the Student returns a score.
- The Teacher’s goal is to help the Student succeed on a final task using as few training steps as possible.
- TSCL models the Teacher’s curriculum objective as a POMDP with simple and batch formulations for reinforcement and supervised learning.
- In the simple formulation, actions select task parameters, observations expose task scores, and rewards measure score changes after training.
- In the batch formulation, the Teacher selects a probability distribution over tasks, observes all task scores, and receives the sum of score changes.
- The optimization objective sums each task’s performance at its last training step because the final task may initially provide no meaningful feedback.
3 Algorithms
TSCL uses a Teacher to select subtasks according to the Student’s learning progress, while also addressing forgetting. Its algorithms estimate progress from task scores and balance exploration with exploitation when choosing tasks.
- Idealized curriculum: In the idealized curriculum, sampling shifts toward a task while its score improves, then moves to the next task as the curve flattens.Initially tasks are sampled uniformly; after all task curves flatten, sampling returns to uniform.
- Learning-progress curriculum: The Teacher should emphasize tasks with the fastest learning progress and revisit tasks whose performance is worsening to counter forgetting.Learning progress is represented by the slope of a task’s learning curve, using its absolute value to include negative progress.
- Teacher algorithms: Because task scores are noisy, TSCL adapts non-stationary multi-armed bandit ideas to estimate learning progress and balance exploration with exploitation.The paper notes that the progress heuristic alone does not specify how to estimate slopes or choose among tasks.
- Online algorithm: The Online algorithm tracks each task’s expected return with an exponentially weighted moving average and selects tasks using ϵ-greedy or Boltzmann exploration.ϵ-greedy samples randomly with probability ϵ; Boltzmann selection uses a temperature parameter τ.
- Progress estimation: The Naive algorithm estimates each task’s learning-curve slope by linear regression after training it K times, whereas the Window algorithm regresses over a FIFO buffer of the last K scores.The regression coefficient becomes the reward used by the non-stationary bandit algorithm; the Window method uses recording timesteps as input variables.
- Sampling algorithm: The Sampling algorithm removes exploration hyperparameters by sampling one recent reward from each task’s K-reward buffer and choosing the task with the highest sampled reward.Tasks with recently high rewards are therefore sampled more often, making exploration part of the selection rule.
4 Experiments
Experiments apply TSCL to decimal addition and Minecraft navigation, comparing automatic curricula with manual and uniform baselines. TSCL surpasses manual and uniform curricula in 9-digit 1D addition and performs comparably to manual curriculum in Minecraft, while 2D addition favors simpler baselines.
- 4.1.2 Addition with 2-dimensional Curriculum: In 9-digit 2D addition, the task is easier, manual curriculum is hard to beat, and uniform sampling is competitive.The two-dimensional task assigns lengths to the two addends separately, and independent task distributions performed worse.
- 4.1.3 Observations: TSCL trains on shorter numbers first, taking a distinctively different approach from the best manual curriculum in 4-digit 2D addition.Figure 6 compares accuracy progress for TSCL and the best manual curriculum.
- 4.2.1 5-step Curriculum: The Minecraft curriculum contains five tasks, progressing from a single room with a target to four rooms separated by randomized lava and walls.Manual training used 200,000 steps for the first task, 400,000 for tasks two through four, and 600,000 for the fifth task.
- 4.2.1 5-step Curriculum: In Minecraft evaluation, direct training on the last task made no progress, uniform mixing was slow, and TSCL performed comparably to the manual curriculum.The learning curves report mean episode reward per 10,000 timesteps, with means and standard deviations based on three random seeds.
5 Related Work
Earlier curriculum-learning methods generally relied on manually designed curricula, while related automatic approaches differed from TSCL in task generation, progress criteria, or learning setting.
- Prior curriculum-learning work manually designed curricula rather than generating them automatically.
- Learning progress has been used as a reward in developmental robotics, classroom teaching, and intrinsic-motivation frameworks.
- Adversarial-bandit methods lacked assumptions about reward dynamics, while curriculum learning permits the assumption that rewards change smoothly over time.
- Automatic goal-generation methods can create new subtasks during training, but their reported benefit mainly concerns exploration and is not guaranteed to improve final-task learning.
- A concurrent approach applied automatic curricula only to supervised sequence learning and used EXP3.S, whereas TSCL also addressed reinforcement learning with non-stationary-bandit-inspired algorithms.
- For decimal addition, the authors evaluated curriculum approaches rather than improving task performance, so they made no direct comparison with prior addition results.
- Compared with related Minecraft work, TSCL used continuous movement without explicit memory requirements, and unlike exploration bonuses it changed Teacher task selection rather than the Student algorithm.
6 Conclusion
The conclusion presents TSCL as an automatic curriculum-learning framework for supervised and reinforcement learning, using learning progress to select tasks and address forgetting.
- TSCL provides a framework for automatic curriculum learning applicable to supervised and reinforcement learning tasks.
- The proposed Teacher algorithms select tasks using learning-curve progress and retrain tasks whose absolute slope indicates the Student may be forgetting them.
- In decimal addition, the Sampling algorithm outperformed the best manually designed curriculum and uniform sampling.
- On the 5-task Minecraft navigation problem, the Window algorithm matched a carefully designed manual curriculum and significantly outperformed uniform sampling.
- TSCL can avoid manually ordering subtask difficulty and hand-designing curricula when curriculum learning is necessary.
7 Future Work
Future work extends TSCL beyond the discrete, predefined task parameterizations considered in this work.
- The study considered only discrete task parameterizations.
- Future research could apply TSCL to continuous task parameterizations.
- Another proposed direction is automatically generating subtasks when they have not been predefined.
- Candidate sources for automatically generated subtasks include a generative model or different initial states within the same environment.
A Simple versions of the algorithms
The simple TSCL algorithms choose tasks from estimated learning-progress signals, update task values from observed scores, and differ in whether progress is estimated online, over windows, or by sampling.
- Online algorithm: The online algorithm maintains FIFO score and timestep buffers for each task, fits a linear-regression slope, and updates the task’s expected return from that progress estimate.
- Naive algorithm: The naive algorithm resets a score list for the selected task, collects K observations, estimates a regression coefficient, and updates its expected return.
- Window algorithm: The window algorithm initializes task-specific FIFO buffers and expected returns, selects according to absolute expected progress, and trains the Student on the chosen task.
- Sampling algorithm: The sampling algorithm samples a reward from each task’s buffer, selects the task with the largest absolute sampled reward, and stores the observed reward.
- Batch versions: The batch naive and window variants reset task buffers or maintain FIFO buffers, regress task scores, and update scalar or vector expected returns.
- Batch versions: Batch variants compute score changes as rt = ot − ot−1, store per-task rewards, and update expected returns from the resulting progress signals.
C Decimal Number Addition Training Details
The decimal-addition experiments used an LSTM encoder-decoder implementation and measured training efficiency by the steps required to reach 99% validation accuracy.
- Model and data: The encoder and decoder were LSTMs with 128 units, and the encoder’s last output was supplied to every decoder input.The hidden state was not passed directly from encoder to decoder.
- Model and data: Each curriculum training step used 40,960 samples, while validation used 4,096 samples with a batch size of 4,096.
- Evaluation: Training comparisons used the number of steps needed to reach 99% validation-set accuracy.The experiments fixed ϵ=0.1, τ=0.0004, α=0.1, and K=10.
D Minecraft Training Details
The Minecraft experiments trained an image-based PPO agent to navigate mazes with curriculum learning, using parallel environments and several task-specific training choices. The curriculum setup included a simplified first task, corrective penalties, and a mixed-action extension for more complex rooms.
- Task and reward: The agent navigated randomly generated mazes where touching the target yielded 1,000 points, movement cost -0.1, and death or timeout yielded -1,000 points.Timeouts were 30 seconds in the first task and 45 seconds in subsequent tasks.
- Policy architecture: PPO used four convolutional layers, one LSTM layer, and a 40 × 30 color-image input to produce Gaussian movement and turning actions.A state-value output provided the baseline.
- Parallel training: Training used 10 parallel Minecraft instances, with runners collecting experience and a GPU trainer performing batch updates.Runners periodically refreshed their policy snapshots and did not train locally.
- Training optimizations: Frame skipping processed every fifth frame, accelerating learning while preserving policy performance without frame skipping.Auxiliary depth-prediction loss produced only minor improvements.
- Curriculum setup: Minecraft automatic curriculum learning used the Window algorithm with a 10,000-timestep window and exploration rate 0.1.Other algorithms were not implemented because score changes were difficult to calculate under parallel training.
- Curriculum adjustment: The first curriculum task was too simple, so a backward-movement penalty was added; unsuccessful runs were sometimes discarded, and results report only successful runs.The original task could be solved with backward circles that earned nearly the same reward.
- Action extensions: Combining Gaussian movement actions with Bernoulli jump and use actions showed preliminary success in rooms containing doors, switches, or jumping obstacles.