Source-linked AI summary
Automatic Curriculum Learning For Deep RL: A Short Survey
Rémy Portelas, Cédric Colas, Lilian Weng, Katja Hofmann, Pierre-Yves Oudeyer
TL;DR
Deep Reinforcement Learning lacks a unified account of how to organize training situations around agents’ changing capabilities. This survey defines and classifies Automatic Curriculum Learning mechanisms, then unifies them by their goals, controlled task-MDP elements, and surrogate objectives; it concludes that systematic benchmarks and theory remain needed while identifying directions toward open-ended learning agents.
Problem
The survey addresses the need to understand mechanisms that automatically organize DRL training situations according to agent capabilities and the lack of comparative studies, standard benchmarks, and theoretical understanding.
Method
The paper provides an accessible survey and typology of ACL mechanisms, organizing them by purpose, controlled task-MDP element, and optimized surrogate objective.
Results
The survey unifies ACL mechanisms used with DRL under a common framework connecting goals, controlled task-MDP elements, and surrogate objectives.
Takeaways & Limitations
The framework highlights potential new ACL directions and suggests that ACL mechanisms could support the development of open-ended learning agents.
Takeaways & Limitations
Comparative studies, standard benchmark environments, and theoretical understanding of ACL mechanisms remain limited.
Abstract
from arXiv · showhide
Automatic Curriculum Learning (ACL) has become a cornerstone of recent successes in Deep Reinforcement Learning (DRL).These methods shape the learning trajectories of agents by challenging them with tasks adapted to their capacities. In recent years, they have been used to improve sample efficiency and asymptotic performance, to organize exploration, to encourage generalization or to solve sparse reward problems, among others. The ambition of this work is dual: 1) to present a compact and accessible introduction to the Automatic Curriculum Learning literature and 2) to draw a bigger picture of the current state of the art in ACL to encourage the cross-breeding of existing concepts and the emergence of new ideas.
1 Introduction
Automatic Curriculum Learning emerged from curriculum-learning and curiosity-driven exploration ideas as mechanisms that adapt training situations to DRL agents’ capabilities.
- Origins: Curriculum learning organizes training situations by complexity, progressing from simpler situations toward more difficult ones.Examples include training cart-pole controllers with long, light poles before shorter, heavier poles.
- Transition to DRL: ACL connects earlier curriculum-learning ideas with Deep Reinforcement Learning by automatically organizing learning situations during training.
- Definition: Automatic Curriculum Learning is defined as mechanisms that automatically adapt training-data distributions by selecting situations according to agent capabilities.
2 Automatic Curriculum Learning for DRL
ACL for DRL formalizes adaptive task selection and classifies mechanisms by their purpose, controlled task component, and optimized surrogate objective.
- Formalization: DRL agents learn action sequences in environments to maximize cumulative reward, with tasks commonly formalized as Markov Decision Processes.
- Formalization: ACL learns a task-selection function D:H→T from interaction history to maximize performance on target tasks after N training steps.
- Formalization: ACL can be viewed as metalearning because the task-selection function is learned during training to improve subsequent learning.
- ACL Typology: The ACL typology classifies methods by why ACL is used, what it controls, and what surrogate objective it optimizes.Examples include environments or goals as controlled elements and learning progress or diversity as surrogate objectives.
3 Why use ACL?
ACL is used to improve learning on restricted task sets, solve difficult or sparse-reward tasks, support generalization, train multi-goal agents, and organize open-ended exploration.
- Restricted task sets: ACL improves sample efficiency or asymptotic performance on restricted task sets.
- Hard tasks: ACL addresses hard or sparse-reward tasks by guiding agents through auxiliary tasks, progressively increasing difficulty, or organizing exploration.
- Generalization: ACL shapes learning trajectories to improve generalization, including generalization across task distributions and from simulation to the real world.
- Multi-goal agents: In multi-goal reinforcement learning, ACL helps agents improve performance on predefined testing goals through goal-conditioned behavioral repertoires.
- Open-ended exploration: ACL organizes open-ended exploration by supporting discovery of achievable goals and acquisition of robust, diverse behavior repertoires.
4 What does ACL control?
ACL controls task presentation and stored experience at different stages of DRL, modifying task-MDP elements or replayed transitions to shape learning data.
- Data stages: Off-policy DRL separates data collection from data exploitation, motivating distinct ACL approaches for each stage.
- Data collection: During data collection, ACL generates discrete or continuous tasks by acting on task-MDP elements such as rewards, transitions, and initial-state distributions.
- Data collection: Controlling initial-state distributions can move agents from states near a target toward states farther away, creating progressively harder auxiliary tasks.
- Data collection: Procedural Content Generation can create rich task spaces in which ACL detects relevant niches of learning progress.
- Data collection: Self-play forms ACL by organizing sequences of opponents, which correspond to tasks with different transition functions and possibly state spaces.
- Data exploitation: During data exploitation, ACL acts on replay-memory data through transition selection or transition modification.
- Data exploitation: Prioritized experience replay selects transitions judged more informative for policy updates, while Hindsight Experience Replay substitutes achieved goals into stored transitions.
5 What Does ACL Optimize?
ACL methods optimize surrogate objectives that make difficult training goals more tractable, including rewarded experience, intermediate difficulty, learning progress, diversity, and opponent diversity.
- ACL methods use surrogate objectives because average performance on target tasks can be difficult to optimize directly.
- Reward: Reward-focused selection increases high-reward transitions, but data collection avoids solved tasks because they hinder further learning.
- Intermediate difficulty: Intermediate difficulty targets tasks that are neither too easy nor too difficult, supporting maximal learning progress.
- Intermediate difficulty: GoalGAN proposes goals producing intermediate agent performance, while judge networks predict goal feasibility for the current learner.
- Learning progress: Learning progress compares final and initial scores and guides practice toward tasks that are neither too easy nor too difficult without requiring an intermediate threshold.
- Diversity: Diversity objectives favor low-density goals, trajectories, or state regions, while opponent pools maintain diversity to improve robustness and fight catastrophic forgetting.
6 Discussion
The survey unifies ACL mechanisms and identifies open directions, including broader control targets and systematic evaluation. It also connects ACL to the longer-term goal of open-ended learning agents.
- The bigger picture: The survey organizes ACL by why it is used, what it controls, and which surrogate objective it optimizes.
- Expanding the set of ACL targets: ACL could expand beyond task elements to control agent properties such as memory, body, and capabilities.
- On the need of systematic ACL studies: The field lacks comparative studies, standard benchmark environments, and a theoretical understanding of ACL mechanisms in DRL.
- ACL as a step towards open-ended learning agents: ACL mechanisms could support open-ended agents by organizing exploration and development from simple initial states toward multiple goals.