PaperlayerView exact revision on arXiv

Source-linked AI summary

Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey

Sanmit Narvekar, Bei Peng, Matteo Leonetti, Jivko Sinapov, Matthew E. Taylor, Peter Stone

arXiv:2003.04960v2cs.LGcs.AIstat.ML

TL;DR

RL often requires costly interaction to learn difficult tasks, motivating curricula that sequence tasks or samples for more effective learning. This paper formalizes curriculum learning in RL as task generation, sequencing, and transfer learning, then surveys methods and uses the framework to identify open problems. It concludes that the field has substantial unresolved challenges in task creation, knowledge transfer, and curriculum design.

  • Problem

    RL learning can require substantial environmental interaction, while curriculum-learning research lacks consistent definitions and spans disconnected methods.

  • Method

    The paper defines a three-part RL curriculum-learning framework—task generation, sequencing, and transfer learning—and systematically surveys methods by their assumptions and capabilities.

  • Results

    The survey classifies existing approaches, with particular attention to sequencing methods and their assumptions about intermediate tasks.

  • Takeaways & Limitations

    The framework organizes the RL curriculum-learning landscape and highlights open problems in automated task creation, knowledge transfer, and general curriculum design.

  • Takeaways & Limitations

    Existing methods often rely on manually crafted tasks, human input, or assumptions about parameterized domains and transferable knowledge.

Abstract

from arXiv · show

Reinforcement learning (RL) is a popular paradigm for addressing sequential decision tasks in which the agent has only limited environmental feedback. Despite many advances over the past three decades, learning in many domains still requires a large amount of interaction with the environment, which can be prohibitively expensive in realistic scenarios. To address this problem, transfer learning has been applied to reinforcement learning such that experience gained in one task can be leveraged when starting to learn the next, harder task. More recently, several lines of research have explored how tasks, or data samples themselves, can be sequenced into a curriculum for the purpose of learning a problem that may otherwise be too difficult to learn from scratch. In this article, we present a framework for curriculum learning (CL) in reinforcement learning, and use it to survey and classify existing CL methods in terms of their assumptions, capabilities, and goals. Finally, we use our framework to find open problems and suggest directions for future RL curriculum learning research.

1. Introduction

Curriculum learning organizes tasks or experience to help RL agents learn difficult problems more efficiently. This survey formalizes the area, classifies its methods, and identifies open research questions.

  • Motivation: Transfer learning lets RL agents reuse knowledge across source and target tasks, potentially improving difficult-task performance or convergence time.
  • Motivation: Curricula organize tasks or experience samples into sequences that can leverage previously learned concepts or skills.The literature includes both task-level and sample-level orderings.
  • Survey scope: The survey addresses inconsistent definitions of curricula and disconnected research across curriculum generation, sequencing, and transfer learning.
  • Survey scope: The framework organizes the RL curriculum-learning landscape and uses identified gaps to motivate future research.
  • Survey scope: It formalizes curriculum-learning methods and extends transfer-learning metrics to evaluate their benefits and costs.
  • Survey scope: The survey reviews task construction, automated and human sequencing, and knowledge transfer between tasks with differing state/action spaces, rewards, or transitions.

2. Background

RL learns policies through interaction with episodic environments, while transfer learning reuses source-task knowledge to accelerate target-task learning. Evaluation distinguishes performance and cost, including whether source-task training is counted.

  • Reinforcement learning: An episodic RL task is modeled as an MDP with states, actions, transitions, rewards, an initial-state distribution, and terminal states.
  • Reinforcement learning: The agent seeks an optimal policy that maximizes cumulative reward until the episode ends.
  • Reinforcement learning: RL methods include value-function, policy-search, and actor-critic approaches.
  • Transfer learning: Transfer learning initializes target-task learning with knowledge from source tasks, including samples, options, policies, models, or value functions.
  • Transfer learning: Transfer methods may require shared state/action spaces, task mappings, or unchanged transition and reward functions.
  • Evaluation: Time to threshold compares how quickly transferred and from-scratch agents reach a desired target-task return.
  • Evaluation: Weak transfer treats source-task training as sunk cost, whereas strong transfer includes it in the learning trajectory or cost.

3. The Curriculum Learning Method

Curriculum learning optimizes the order in which an agent acquires experience. The framework represents curricula as directed acyclic graphs and separates task generation, sequencing, and transfer learning.

  • Definition: Curriculum learning organizes accumulated experience to improve performance or training speed on final tasks.
  • Representation: The general curriculum is a directed acyclic graph whose vertices map to subsets of transition samples.
  • Representation: A curriculum can be created online with dynamically added edges or offline from a pre-generated graph.
  • Representation: Single-task curricula organize experience from one task, while task-level curricula associate graph vertices with samples from individual tasks.
  • Representation: Sequence curricula are linear task orderings, whereas task graphs support revisiting tasks and more general connections.
  • Core elements: Curriculum design comprises task generation, sequencing, and transfer learning.
  • Core elements: Task generation supplies intermediate tasks, sequencing orders samples or tasks, and transfer passes reusable knowledge between differing tasks.
  • Categorization: Categorization distinguishes sample sequencing, co-learning, and task-level structures according to what is transferred between tasks.

4. Curriculum Learning for Reinforcement Learning Agents

The survey systematically reviews the three central components of RL curriculum learning, emphasizing the sequencing problem most commonly associated with the field.

  • Survey organization: The survey covers task generation, sequencing, and transfer learning as the three central elements of RL curriculum learning.
  • Survey organization: Its main emphasis is sequencing, while tables categorize surveyed work using the framework’s dimensions.

4.1 Task Generation

Task generation creates intermediate tasks intended to make transfer beneficial for a final task while avoiding negative transfer. In RL, surveyed methods commonly parameterize domains and vary task descriptors or restrictions to produce easier tasks.

  • Task generation creates intermediate tasks specifically to support beneficial transfer to final tasks and avoid negative transfer.
  • Most known RL methods assume a parameterized domain in which task descriptors specify degrees of freedom that can be varied.Examples include board size and piece counts in Quick Chess.
  • Task simplification reduces task dimensions, while promising initialization, mistake learning, and related methods modify states or training context.
  • Object-Oriented MDP methods let a designer instantiate environment attributes to create simpler versions of a final task, but mostly random generation requires solvability checks.
  • Related non-RL approaches suggest auxiliary tasks from partially completed target-task trajectories or invent new problems while preserving solutions to earlier tasks.

4.2 Sequencing

Sequencing methods range from reordering target-task samples to creating unconstrained intermediate tasks, with multi-agent interaction and goal generation providing additional curriculum mechanisms. Their assumptions depend on how much control they have over environments, tasks, and transfer.

  • Sequencing methods span reordering target-task experience, progressively modifying task properties, and creating entirely new intermediate tasks.
  • Sample Sequencing: Prioritized Experience Replay orders transitions by expected learning progress using temporal-difference error, while complexity-based methods combine difficulty selection with coverage penalties.
  • Sample Sequencing: Domain-independent priority design remains an open problem, and learned weighting methods such as ScreenerNet can predict sample significance without storing the sample.
  • Sample Sequencing: Hindsight Experience Replay replays episodes with achieved goals, while Curriculum-guided HER selects goals using curiosity and proximity criteria.
  • Sample Sequencing: Episodic backward update propagates delayed rewards by updating sampled episodes from their final transitions backward, with diffusion reducing cumulative overestimation.
  • Sample Sequencing: Sample-sequencing methods apply directly to the target task and have broad applicability because they do not require altered intermediate environments.
  • Co-learning: Co-learning generates an implicit curriculum through cooperative or adversarial interaction among multiple agents or versions of one agent.
  • Reward and Initial/Terminal State Distribution Changes: Goal-generation methods use models such as Goal GAN to propose state-space regions whose difficulty matches the agent’s current ability.

MDP-based Sequencing

MDP-based approaches model curriculum generation as a higher-level decision process in which a teacher selects tasks for a learning agent. Curriculum policies can learn when and what to train, but solving these meta-processes remains computationally expensive.

  • MDP formulation: Curriculum generation is formulated as interaction between a student MDP and a teacher meta-MDP that selects the student’s next task.The teacher’s actions are tasks, while student learning changes the state of the curriculum process.
  • MDP formulation: The meta-process can be modeled as a POMDP when the teacher observes task scores instead of the learner’s internal parameters.Matiisen et al. use score observations for intermediate tasks rather than direct access to agent parameters.
  • Objectives: The two frameworks optimize different objectives: minimizing target-task time to threshold versus maximizing cumulative performance across encountered tasks.Their reward definitions reflect these distinct curriculum goals.
  • Heuristic solutions: Heuristics select tasks because learning the curriculum POMDPs directly is computationally expensive.Narvekar et al. compare policy changes on target-task samples, whereas Matiisen et al. use a contrasting heuristic approach.
  • Learned curriculum policies: Learning a curriculum policy can avoid heuristic sequencing and improve efficiency by revisiting tasks after only a few episodes.Narvekar and Stone also address how long to train on intermediate tasks, rather than waiting for performance plateaus.

Combinatorial Optimization and Search

Search-based methods treat task sequencing as optimization over curricula, using black-box, heuristic, or partially structured formulations. Evaluations compare objectives such as threshold time, asymptotic return, and cumulative return.

  • Problem formulation: Finding the best permutation of a fixed task set is a computationally difficult black-box optimization problem, so approximate solutions are typically used.The objective is defined by a curriculum-learning metric.
  • Metaheuristic search: Metaheuristic methods apply general-purpose search algorithms such as beam search, tabu search, and genetic algorithms to task sequencing.These methods are not tied to a specific problem domain.
  • Metaheuristic search: Trajectory-based metaheuristics outperformed population-based counterparts on the tested domains across time to threshold, maximum return, and cumulative return.The comparison covered three curriculum objectives.
  • Task-specific search: A task-specific heuristic estimates pairwise transferability with a simulator and uses those estimates to optimize cumulative return.The method evaluates using one task as a source for another.
  • Online search: Online search methods use simulated learning trajectories to choose new tasks, including selection by reward, after fixed training intervals.These approaches assume a simulator is available for evaluating learning on candidate tasks.
  • Gray-box optimization: Gray-box optimization combines a white-box scheduling model with black-box parameter optimization, assigning task utilities and pairwise penalties.The formulation partially models sequence effects while retaining black-box optimization.

Graph-based Sequencing

Graph-based sequencing represents tasks as nodes in a directed acyclic graph, with edges indicating source-target relationships. Existing methods rely on task descriptors and heuristics to construct connections or generate related source tasks.

  • Graph representation: A task-level curriculum graph associates tasks with nodes and uses directed edges to indicate that one task is a source for another.The graph is typically directed and acyclic.
  • Task descriptors: Graph construction methods commonly assume that the task set is known and that tasks have feature descriptors encoding domain properties.In Ms. Pac-Man, descriptors can include the number of ghosts or maze type.
  • Heuristic graph construction: Svetlik et al. group tasks by shared binary features, connect tasks within groups using transfer potential, and link subgraphs by feature inclusion.Transfer potential trades off source-task applicability against the cost of learning it.
  • Object-oriented task generation: Object-oriented MDPs provide another graph-construction basis by representing tasks through their objects and task-specific dynamics.Source tasks can be generated by selecting smaller sets of objects from the domain.

Auxiliary Problems

Auxiliary curriculum problems include deciding training duration, changing task difficulty, and understanding human curriculum design. Proposed methods dynamically adjust tasks, while human studies show that non-experts can improve learning but leave the design process insufficiently understood.

  • Training duration: Training intermediate tasks until performance plateaus is unnecessary; reselection after a few episodes can produce faster learning.Dynamic reselection addresses how long to spend on each intermediate task.
  • Difficulty progression: Progression functions specify how task complexity changes over time and can change the task every episode.A task-generation function maps desired complexity ct ∈[0, 1] to a task, with ct = 1 as the final task.
  • Human curriculum design: Human curriculum design remains poorly understood despite evidence that human teaching and curriculum interfaces can inform machine-learning methods.The survey identifies limited investigation of how humans design curricula.
  • Expert-designed curricula: Domain experts have manually trained agents successfully on sophisticated behaviors, including simulated robot combat and robot soccer.Examples include NERO combat tactics and a 19-behavior robot-soccer schedule.
  • Open problems: Existing curriculum studies rarely examine the strategies people use to sequence tasks, leaving the design process as an open research issue.The limitation applies even where domain experts produce effective curricula.
  • Human feedback: Interactive reinforcement learning shows that human feedback can speed agent learning toward a near-optimal policy.This literature focuses on how teachers teach and how to incorporate feedback into reinforcement learning.
  • Naive-user curricula: Non-expert humans designed curricula that improved overall agent performance on both the full curriculum and the final task relative to learning the final task from scratch.Participants tended to select intermediate commands containing concepts important for the final task, and those curricula performed better.

4.3 Knowledge Transfer

Knowledge transfer enables agents to reuse information across curriculum tasks, but transferred knowledge may differ in abstraction and must handle task changes without losing useful prior knowledge.

  • Curriculum learning transfers knowledge between intermediate tasks so agents can reuse information while learning the target task.Transferred knowledge may initialize the learner directly or guide learning in the target task.
  • Transferred information ranges from low-level policies, value functions, models, and samples to high-level options, skills, shaping rewards, and subtask definitions.
  • Policy transfer can bridge tasks differing in starting states, reward functions, or transition functions.
  • Value-function transfer commonly initializes the value function for the next intermediate task using parameters learned in the preceding task.
  • Catastrophic forgetting can erase earlier knowledge when neural-network parameters are changed for new tasks.Progressive neural networks prevent forgetting by freezing previous task columns, but their parameter count grows with the number of tasks.
  • Higher-level knowledge such as partial policies or options can address differing state and action spaces between tasks.A partial policy need not be defined for every state in an MDP.

5. Related Areas and Paradigms

Curriculum learning relates to several reinforcement-learning and machine-learning paradigms that improve learning efficiency or organize experience, but it remains focused on a specific target task.

  • Reinforcement-learning paradigms related to curriculum learning include multitask, lifelong or continual, active, and meta-learning.
  • Multitask learning co-learns multiple tasks simultaneously, typically using a shared basis space and optimizing performance across all tasks.
  • Lifelong and continual learning present tasks sequentially and reuse shared knowledge, usually optimizing performance over all encountered tasks.
  • Unlike multitask and lifelong learning, curriculum source tasks are designed solely to improve performance on one target task.
  • Meta-learning trains agents across varied tasks for rapid adaptation, whereas curriculum designers typically use task identity when sequencing tasks.
  • Supervised-learning curricula include continuation and regularization ideas, learning-progress-based task selection, teacher-agent formulations, and self-paced ordering.

6. Open Questions

Open questions concern reducing human input and cost, integrating task generation with sequencing, selecting transferable knowledge, and establishing theoretical explanations for when curricula work.

  • 6.1 Fully Automated Task Creation: Automatically generating useful task pools remains underexplored, with existing methods relying on manually crafted tasks or manually tuned creation rules.Reducing manual input is identified as an important future direction.
  • Most surveyed approaches fix the type of knowledge transferred between each pair of tasks, leaving adaptive knowledge selection open.
  • Curriculum-generation time can exceed the time required to learn the target task directly.The survey suggests amortizing this cost across multiple agents, tasks, or sim-to-real training.
  • Joint task generation and sequencing could reduce human intervention but requires valid, solvable, appropriately challenging maps and can be expensive to train.
  • 6.5 Theoretical Results: Despite empirical benefits, curriculum learning lacks theoretical results explaining when and why curricula help or how they should be created.
  • Determining example difficulty and matching examples to current learner ability remains a major challenge, especially because curriculum design still relies heavily on domain experts.

7. Conclusion

The survey formalizes reinforcement-learning curriculum learning as a three-part approach and classifies sequencing methods by their assumptions about intermediate tasks.

  • Curriculum learning consists of task generation, sequencing, and transfer learning.
  • The survey classifies sequencing into sample sequencing, co-learning, and methods that explicitly modify the MDP.The MDP-based methods differ in whether they restrict changes to initial or terminal distributions and rewards or allow broader differences.
  • The survey also discusses human sequencing practices to illuminate manually designed curricula.
Loading 2003.04960v2…