Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey

1. Introduction

• Progressively harder subgames can build skills and knowledge toward a complex target, motivating curricula for training artificial agents.The Quick Chess example adds pieces, board size, configurations, and objectives across subgames until reaching full chess.Source: §1 · p. 2
• Curriculum learning in RL lacks a consistent definition: curricula may order experience samples, tasks in sequences or directed acyclic graphs, or knowledge from multiple source tasks.The paper also notes that curriculum-generation work is distributed across several research areas and that landmark RL systems have implicitly used curricula, including self-play.Source: §1 · p. 3

Overview

• The article provides an overarching framework for RL curriculum learning, including classification criteria that organize the research landscape across frameworks and settings.Its survey aims to establish a systematic basis for discussing curriculum-learning methods in RL.Source: Overview · p. 3
• The framework defines curriculum representations and methods, extends transfer-learning metrics for evaluation, and examines task construction, sequencing, and knowledge transfer across tasks.Tasks may be generated in advance or dynamically, curricula may be manually or automatically sequenced, and transfer must handle differing state/action spaces, transition functions, and rewards.Source: Overview · pp. 3–4
• The survey reviews RL curriculum-learning methods and related paradigms, then identifies literature gaps, framework limitations, and open problems.Its core survey covers the components of curriculum learning, while later sections discuss supervised learning, human education, and future research directions.Source: Overview · p. 4

2.1 Reinforcement Learning

• An episodic MDP specifies states, actions, transitions, rewards, an initial-state distribution, and terminal states; the agent learns a policy maximizing expected return until termination.At each discrete timestep, the agent observes its state and selects an action according to its policy.Source: §2.1 · pp. 4–5
• RL methods learn optimal policies through value functions, direct policy search, or actor-critic methods combining a policy with a learned value estimate.Value-based methods include SARSA and Q-learning, while policy search optimizes policy parameters directly and actor-critic updates the policy using critic guidance.Source: §2.1 · p. 5

2.2 Transfer Learning

• Transfer learning accelerates difficult target-task training by reusing source-task samples, options, policies, models, or value functions, subject to task-space, mapping, and dynamics assumptions.For example, value-function transfer initializes qtarget(s, a) with parameters learned for qsource(s, a), biasing target exploration and action selection; methods may transfer from multiple source tasks.Source: §2.2 · pp. 5–6

2.3 Evaluation Metrics for Transfer Learning

• Transfer benefits are measured by time to threshold, asymptotic performance, jumpstart, and total reward ratio, comparing transfer with learning from scratch.Weak transfer treats source-task training as sunk cost, whereas strong transfer includes it in learning curves, as illustrated in Figure 2.Source: §2.3 · p. 6, Figure 2 · p. 7

3. The Curriculum Learning Method

• Curriculum learning organizes experience over time to accelerate or improve learning, formalizing curricula as directed acyclic graphs whose edges specify training order and whose paths end at a target sink.Curricula may be created online or offline, and practical simplifications include single-task, task-level, and linear-sequence representations.Source: §3 · p. 7, §3.1 · pp. 7–8

3.2 Curriculum Learning

• Curriculum learning consists of task generation, sequencing, and transfer, using knowledge from simpler or intermediate tasks to reduce exploration and improve performance on final tasks.Task generation supplies candidate intermediate tasks, sequencing orders experience, and transfer passes reusable knowledge across tasks that may differ in state or action spaces, rewards, or transitions.Source: §3.2 · p. 9

3.3 Evaluating Curricula

• Curricula are evaluated against learning from scratch using target-task performance or learning speed, while costs may include curriculum construction and intermediate-task training.Evaluation choices depend on whether the goal is initial performance, asymptotic performance, threshold attainment, or total cost, and whether design effort is treated as sunk.Source: §3.3 · pp. 10–11

3.4 Dimensions of Categorization

• The survey classifies curriculum methods along seven dimensions: task generation, representation, transfer method, sequencing, adaptivity, evaluation metric, and application area.These dimensions distinguish, for example, single curricula, sequences, and graphs; automatic or human design; static or adaptive construction; and weak or strong transfer evaluation.Source: §3.4 · pp. 11–12
• The taxonomy also records whether methods target toy environments, simulated or real robotics, video games, or other domains to clarify their scalability and applicability.Application areas include grid worlds and cart-pole, MuJoCo-style simulated robotics, physical robots, and environments such as Starcraft or Atari.Source: §3.4 · pp. 12–13

4. Curriculum Learning for Reinforcement Learning Agents

• The survey organizes prior work around task generation, sequencing, and transfer learning, categorizing each subproblem according to the framework’s dimensions.It provides a table for each subproblem and devotes most attention to sequencing, the element most commonly associated with curriculum learning.Source: §4 · p. 13

Task Generation

• Task generation creates intermediate tasks relevant to final tasks, with the goal of beneficial transfer and avoiding negative transfer.It differs from lifelong learning, where potentially unrelated tasks may be continually proposed.Source: §4.1 · p. 13
• Formal reinforcement-learning research on task generation is very limited, and known methods represent domains with parameters whose instantiations define different tasks.Narvekar et al. (2016) model domains as sets of MDPs described by task-descriptor vectors, then vary degrees of freedom and impose restrictions to construct task spaces.Source: §4.1 · p. 13
• Narvekar et al. (2016) generate task variants through simplification, promising initialization near high-reward states, mistake learning, and related restrictions.These methods reduce task dimensions, alter initial states, or resume learning from states preceding detected mistakes.Source: §4.1 · p. 13
• Da Silva and Reali Costa (2018) use Object-Oriented MDP attributes to partially automate simpler task generation, but mostly random generation requires designers to ensure solvability.Attribute values determine the kinds and numbers of objects, producing tasks with different difficulty levels.Source: §4.1 · pp. 13–14
• Outside reinforcement learning, Stone and Veloso (1994) use partially completed target-task search trajectories to suggest auxiliary subproblems such as reordered goals, fewer goals, difficult subgoals, or altered operators and objects.Their approach identifies sources of difficulty rather than relying only on static domain-property analysis.Source: §4.1 · p. 14
• Powerplay indefinitely invents tasks and solver modifications that preserve all prior solutions while adding an unsolved task, thereby creating a curriculum without a final task.A limited computational budget favors simple tasks the current solver cannot solve, and the framework has been demonstrated on pattern-recognition and control tasks (Srivastava et al., 2013).Source: §4.1 · p. 14

4.2 Sequencing

• Sequencing methods form a spectrum from reordering final-task experience without changing its MDP to progressively modifying some or all target-task properties.Their applicability depends on the practitioner’s control over environment configuration.Source: §4.2 · pp. 14–15

4.2.1 Sample Sequencing

• Sample sequencing reorders or resamples target-task experience, including PER, HER, CHER, and EBU, to create implicit curricula without modifying the environment.PER prioritizes high-TD-error transitions; HER exploits achieved goals in sparse-reward settings; CHER balances goal diversity and proximity; EBU propagates delayed rewards backward through episodes and matched DQN’s mean and median human-normalized Atari performance with significantly fewer samples in 49 games.Source: §4.2.1 · pp. 15, 17

4.2.2 Co-learning

• Co-learning generates curricula through cooperation or competition among agents, with interactions producing progressively challenging goals, counter-strategies, robustness conditions, or behaviors.Asymmetric self-play uses teacher–student objectives for unsupervised task generation, while adversarial and league-based training expose agents to increasingly difficult opponents; priority functions remain domain-dependent, and broadly applicable priorities are still an open problem.Source: §4.2.1 · p. 16, §4.2.2 · pp. 18–19

4.2.3 Reward and Initial/Terminal State Distribution Changes

• Restricted intermediate-task methods preserve state and action spaces and dynamics while varying rewards or initial and terminal distributions to regulate difficulty.Examples include reverse and forward goal expansion, setter–solver curricula that were more stable and effective than Goal GAN on complex tasks, and SAC-X scheduling auxiliary rewards.Source: §4.2.3 · pp. 19–21

4.2.4 No restrictions

• Unrestricted intermediate-task methods impose no restrictions on task MDPs and treat sequencing as an MDP/POMDP, combinatorial sequence optimization, or directed-acyclic task-graph learning.Because arbitrary intermediate tasks are allowed, these methods typically assume transfer algorithms and task descriptors, and may also address how long to train on each task.Source: §4.2.4 · p. 22

MDP formalization

• Curriculum generation is modeled as interaction between a standard student MDP and a higher-level teacher meta-MDP that selects the student’s next task.The student interacts with tasks, while the teacher controls task sequencing.Source: MDP-based Sequencing · p. 22

CMDP formulation

• In Narvekar et al. (2017), the CMDP state is the student policy, actions are tasks, rewards measure learning time, and the goal is reaching target-task performance quickly.The initial state corresponds to a random policy, and the terminal state achieves a desired performance threshold.Source: MDP-based Sequencing · p. 22

POMDP formulation

• Matiisen et al. (2017) use a partially observed formulation that replaces inaccessible student parameters with scores on intermediate tasks and rewards score improvements.Their objective maximizes cumulative performance across encountered tasks rather than minimizing time to target-task threshold.Source: MDP-based Sequencing · pp. 22–23

Task-selection heuristics

• Because meta-MDP learning is computationally expensive, Narvekar et al. (2017) select tasks by the greatest policy change on target-task samples, whereas Matiisen et al. (2017) prioritize the steepest learning-curve slope.The latter alternates toward tasks showing the most progress or forgetting, then switches when progress plateaus.Source: MDP-based Sequencing · p. 23

Learned curriculum policies

• Narvekar and Stone (2019) learned a curriculum policy over transfer-representation weights, showing that brief task training with reselection can accelerate learning, although per-agent training remains expensive.The curriculum policy maps current learning progress to the next task and can determine how long to train each intermediate task.Source: MDP-based Sequencing · p. 23

Problem formulation

• Optimal permutation search is a computationally difficult black-box problem, so fast approximate solutions are typically preferred.The objective is to find the task ordering that optimizes a curriculum-learning metric over a fixed task set.Source: Combinatorial Optimization and Search · p. 23

Metaheuristic search

• Trajectory-based metaheuristics outperformed population-based methods across time-to-threshold, maximum-return, and cumulative-return objectives on the tested domains.Foglino et al. (2019a) evaluated beam search and tabu search against genetic algorithms and ant colony optimization.Source: Combinatorial Optimization and Search · pp. 23–24

Task-specific heuristic search

• A task-specific anytime heuristic estimates pairwise cumulative-return transferability, sorts tasks by source and target potential, and incrementally chains curricula before eventually exhaustively searching bounded lengths.Foglino et al. (2019b) design the method specifically to optimize cumulative return.Source: Combinatorial Optimization and Search · p. 24

Gray-box optimization

• A gray-box method combines an efficiently solved integer-linear-program scheduling model with black-box parameter optimization to maximize cumulative return.The scheduling model assigns task utilities and pairwise penalties, while the external optimizer tunes its parameters; Foglino et al. (2019c) propose this approach.Source: Combinatorial Optimization and Search · p. 24

• Task-level curricula represent tasks as nodes in a directed acyclic graph, where an edge identifies a source task for another task.Source: Graph-based Sequencing · p. 24
• Existing graph-construction approaches rely on heuristics and additional domain information, including known task sets and task feature descriptors.Source: Graph-based Sequencing · pp. 24–25
• Svetlik et al. (2017) group tasks by binary feature descriptors, connect similar tasks using transfer potential, and link subgraphs from feature subsets to supersets.Transfer potential trades off source-task applicability against learning cost, approximated by state-space size.Source: Graph-based Sequencing · p. 25
• Da Silva and Reali Costa (2018) generate simpler source tasks by selecting subsets of objects from target-task OO-MDPs and compare object sets to sequence the graph.Although sequencing is automated, human input is required to ensure the generated tasks are solvable.Source: Graph-based Sequencing · p. 25

Auxiliary Problems

• Most existing methods train on intermediate tasks until performance plateaus, but Narvekar and Stone (2019) showed that brief training followed by dynamic task reselection can perform better.This challenges plateau-based duration decisions for curriculum tasks.Source: Auxiliary Problems · p. 25
• Progression functions specify task difficulty over time, enabling episode-level task changes while solving how long to spend on each intermediate task.Fixed schedules prescribe linear or exponential increases, whereas an adaptive friction-based schedule increases difficulty with improving performance and slows when performance declines.Source: Auxiliary Problems · pp. 25–26

4.2.5 Human-in-the-Loop Curriculum Generation

• Domain experts manually designed curricula that trained simulated robots to learn sophisticated battle tactics and coordinated related robot-soccer skills through overlapping layered learning.Stanley et al. (2005) used programmer-designed NERO training scenarios, while MacAlpine and Stone (2018) scheduled 19 learned behaviors together.Source: §4.2.5 · p. 26
• Human curriculum design remains poorly understood, despite evidence that human feedback can accelerate reinforcement learning and that human-generated data can provide easier starting states for sparse-reward tasks.Hosu and Rebedea (2016) used checkpoints from human Atari play to supplement online experience, while the survey identifies limited investigation of human design strategies.Source: §4.2.5 · pp. 27–28
• Khan et al. (2011) found that one of three naive-user teaching strategies followed curriculum learning by starting simple and gradually increasing difficulty.They proposed that this strategy minimizes the learner’s per-iteration expected error.Source: §4.2.5 · p. 27
• Non-expert humans designed curricula that improved agent performance over learning the final task from scratch, especially when intermediate tasks contained concepts important to the final task.The resulting algorithm also learned human-generated curricula more efficiently by exploiting this guidance.Source: §4.2.5 · p. 28

Knowledge Transfer

• Multiple-task curricula require transferring reusable knowledge across tasks that may differ in state/action spaces, rewards, or transition functions, whereas single-target-task curricula involve no transfer.Transferred knowledge may be low-level, such as policies, value functions, task models, or samples, or high-level, such as options, skills, shaping rewards, and subtasks.Source: §4.3 · p. 29

Low-Level Transfer

• Policy and value-function transfer initialize learning in new tasks, including manipulator dressing, StarCraft micromanagement, and task-selection methods based on transfer benefits.Clegg et al. (2017) transfer policies across different transition functions, while Sinapov et al. (2015) use task metadata to predict useful source-target transfers when final-task samples are unavailable.Source: §4.3 · pp. 29–30

Transfer Challenges

• Catastrophic forgetting can erase knowledge from earlier tasks when neural-network weights adapt to new objectives; progressive neural networks prevent this with frozen task-specific columns and lateral connections, at the cost of growing parameters and requiring task labels.This challenge matters especially when information from orthogonal tasks must be combined, although final-task performance is often the primary curriculum objective.Source: §4.3 · p. 30

High-Level Transfer

• When state and action spaces differ, higher-level transfer uses partial policies, options, and related abstractions to progressively expand skills or compose controllers across tasks.Yang and Asada (1996) transfer control parameters across increasing motion speeds; Zimmer et al. (2018) expand state and action dimensionality; Fujii et al. (1998) compose staged options.Source: §4.3 · pp. 30–31

Skills and Macro-Actions

• Pre-learned skills can be reused to solve related tasks without additional learning, while temporally abstracted macro-actions remain unchanged and adapt as curriculum goals move progressively farther from the start state.Source: §4.3 · pp. 31–32

Multi-Agent Transfer

• Multi-agent curricula transfer knowledge while scaling agent numbers through dependency removal, replay-buffer reuse, policy distillation, or architectures that combine agent-specific networks with graph-based teammate representations.Yang et al. (2020) first trains a single-agent subspace and then adds cooperation; Wang et al. (2020) propose buffer reuse, KL-based curriculum distillation, and DyAN.Source: §4.3 · p. 32

5.1 Related Paradigms in Reinforcement Learning

• Reinforcement-learning curriculum learning addresses sample complexity differently from imitation, off-policy, and model-based learning, which respectively require demonstrations, broad action coverage, or accurate environment models.Difficult RL problems may require millions of episodes because of sparse rewards or complex dynamics.Source: §5.1 · p. 33
• Curriculum learning differs from multitask, lifelong, active, and meta-learning by controlling task creation or order and optimizing a specific target task rather than all tasks or adaptability.Its primary assumption is that environments can generate subtasks whose reusable knowledge helps solve harder tasks.Source: §5.1 · pp. 33–34

5.2 Curricula in Supervised Machine Learning

• Supervised curriculum learning can improve training speed and performance, with Bengio et al. (2009) interpreting curricula as continuation methods and regularizers.Their case studies used shape recognition and language domains.Source: §5.2 · p. 34
• Graves et al. (2017) use learning-progress measures as rewards for a non-stationary bandit that selects tasks, while noting that easier tasks can provide stronger gradients and faster convergence.The two measures track rates of increase in prediction accuracy and network complexity; random curricula remain a strong baseline, although random training can be worse in RL because policy changes alter encountered states.Source: §5.2 · p. 35
• Fan et al. (2018) formulate “Learning to Teach” as an MDP in which a teacher selects training data, loss functions, and hypothesis spaces for a learner.Teacher-training costs can be amortized across new students with the same or different architectures, but differing RL train-test distributions complicate evaluation.Source: §5.2 · p. 35

5.3 Algorithmically Designed Curricula in Education

• Educational intelligent tutoring systems sequence personalized materials or actions to maximize learning efficiently, and RL can adapt teaching policies from each student’s real-time performance without complex pedagogical rules.This application is domain-independent and supports individualized instruction for differing needs, styles, and capabilities.Source: §5.3 · p. 36
• Educational curriculum methods model tutoring as Q-learning, factored MDP, or POMDP problems, selecting knowledge items, hints, problems, examples, or quizzes from student knowledge states.These models range from observed knowledge states to hidden states inferred through learner models and observations.Source: §5.3 · pp. 36–37
• Some educational systems optimize both learning gains and sustained engagement, using child-specific policies and real-time engagement signals, while others rank lessons from dynamically updated performance-based fitness values.Ramachandran and Scassellati (2014) personalize question ordering by knowledge and engagement; Ballera et al. (2014) combine reversed roulette-wheel selection with linear ranking.Source: §5.3 · pp. 37–38

6.1 Fully Automated Task Creation

• Fully automated task creation remains underexplored because manually specified pools and manually tuned generation rules limit curriculum quality and sequencing efficiency.Future methods should reduce the manual input required to control which tasks are created and how many are produced.Source: §6.1 · p. 38

6.2 Transferring Different Types of Knowledge

• Curriculum methods should select and combine different knowledge types across tasks rather than transferring a fixed representation such as value functions or shaping rewards.The best transferable knowledge may vary by task, including options or models.Source: §6.2 · p. 38

6.3 Reusing Curricula and Sim-to-Real Curriculum Learning

• Curriculum-generation costs could be amortized by reusing curricula across multiple agents or target tasks, including sim-to-real training on physical robots.In sim-to-real transfer, task semantics may remain useful even when simulation-learned policy weights do not apply to real-world dynamics.Source: §6.3 · pp. 38–39

6.4 Combining Task Generation and Sequencing

• Combining task generation and sequencing could directly generate each next task, but requires valid, solvable, appropriately difficult maps and may be computationally expensive.This integration could reduce human intervention in curriculum design; Green et al. (2019) used evolutionary map generation to maximize a deep RL agent’s network loss.Source: §6.4 · p. 39

6.5 Theoretical Results

• Curriculum learning lacks theoretical results explaining when and why curricula help, motivating RL difficulty metrics and convergence guarantees analogous to supervised-learning analyses.Weinshall et al. (2018) and Weinshall and Amir (2018) showed convergence decreases with Ideal Difficulty Score and increases with Local Difficulty Score.Source: §6.5 · p. 39

6.6 Understanding General Principles for Curriculum Design

• Curriculum design still depends heavily on experts or algorithms, while the principles used by non-expert humans to select suitable intermediate tasks remain poorly understood.Understanding these principles could improve automatic task generation and inspire algorithms and interfaces that better accommodate human trainers.Source: §6.6 · pp. 39–40

7. Conclusion

• The survey defines curriculum learning as a three-part reinforcement-learning approach comprising task generation, sequencing, and transfer learning.It focuses particularly on sequencing methods and establishes common terminology for the field.Source: §7 · p. 40
• Sequencing methods are classified into five categories based on assumptions about intermediate tasks: sample sequencing, co-learning, modified-MDP curricula with restricted changes, unrestricted MDP changes, and human-designed sequencing.Sample sequencing reorders final-task samples, co-learning emerges through multiple agents, and MDP-based methods vary state distributions, rewards, dynamics, or other task properties.Source: §7 · p. 40
• The survey concludes by identifying open problems intended to guide future reinforcement-learning curriculum-learning research.It aims to provide a common foundation that promotes discussion and advancement in this emerging area.Source: §7 · p. 40