PaperlayerView exact revision on arXiv

Source-linked AI summary

Active Curriculum Refinement for Reinforcement Learning

Zhenya Liu, Yuxin Chen

arXiv:2608.26469v1cs.LG

TL;DR

RL policies can overfit finite environment sets, motivating methods that use prerequisite structure for broader generalization. PATH models environments as a curriculum DAG, explores diverse paths, then concentrates training on unmastered successor regions; experiments report improved robustness and generalization.

  • Problem

    Policies trained on finite environment sets can overfit and fail to transfer robustly across broader distributions, despite many domains having exploitable prerequisite structure.

  • Method

    PATH performs active learning on a curriculum DAG through random path exploration followed by regret-driven expansion of successor subgraphs.

  • Results

    PATH improves robustness and generalization over prior curriculum and regret-based methods across discrete MiniGrid and continuous-control BipedalWalker benchmarks.

  • Takeaways & Limitations

    Explicitly modeling curriculum graph structure supports efficient allocation of training toward reachable environments that remain unmastered.

  • Takeaways & Limitations

    PATH assumes access to a constructible curriculum DAG, requiring additional effort in fully underspecified domains, and relies on hyperparameters such as εes.

Abstract

from arXiv · show

In many reinforcement learning (RL) domains, environments are connected by prerequisite relations, such as difficulty-increasing edits or parameter increments, which induce a directed acyclic curriculum graph (DAG). Although this structure is often exploited only implicitly, explicitly modeling it can improve training. We introduce PATH, a curriculum-learning framework that performs active learning over the curriculum graph. PATH first expands coverage by sampling diverse curriculum paths and then reallocates training toward regions that remain unmastered. Experiments across diverse environments show that PATH explicitly leverages the graph structure to achieve strong robustness and generalization.

1. Introduction

PATH makes curriculum structure explicit by treating training as active learning over a curriculum DAG. It combines broad random path exploration with regret-driven allocation to improve robustness and generalization.

  • Curriculum learning controls which environments are visited and in what order to support reinforcement-learning generalization.
  • Curriculum DAGs can encode progression through parameter increments, complexity-increasing edits, or feature-induced structure.
  • PATH:RANDOM samples curriculum paths to expand coverage, while PATH:ACTIVE reallocates training toward high-potential successor regions using regret signals.
  • PATH formalizes curriculum learning as active path acquisition, seeking mastery of the reachable environment space through relatively few paths.
  • Experiments on MiniGrid and BipedalWalker show that exploiting curriculum graph structure improves robustness and generalization over prior curriculum and regret-based methods.

2. Preliminaries

The paper models learning across environment instances with an underspecified POMDP and represents prerequisite relations as a curriculum DAG. Mastery is defined through low regret across the environment set.

  • An environment is a discounted MDP specified by states, actions, transitions, rewards, an initial-state distribution, and a discount factor.
  • A POMDP adds observations and an observation function while keeping the underlying transition and reward dynamics fixed.
  • An underspecified POMDP represents learning across a set of environment instances whose transition and reward functions depend on instance θ.
  • The objective is to learn a policy with low regret across all environments, with eliminating regret on an instance defined as mastering it.
  • A curriculum DAG has environment instances as nodes, and edges indicate that training on one environment is intended to facilitate learning on another.
  • A maximal directed path defines an ordered curriculum sequence from a source node to a sink node.
  • The prerequisite assumption states that mastering a predecessor makes each successor learnable with at most m effective training interactions.

3. Proposed Methods

PATH exploits an explicit curriculum DAG through randomized path exploration followed by regret-driven active allocation. The two phases expand coverage early, then focus training on residual unmastered regions as random exploration saturates.

  • PATH overview: PATH combines randomized path exploration with regret-driven active learning over a curriculum DAG.PATH:RANDOM explores diverse paths, while PATH:ACTIVE reallocates training toward high-learning-potential regions.
  • PATH:RANDOM: PATH:RANDOM maintains N path pointers, samples current environments for training, and advances pointers through randomly selected successors.Pointers begin at uniformly sampled source nodes, restart after reaching sinks, and can generate paths online without enumerating the full DAG.
  • PATH:RANDOM: Early stopping advances a path when recent reward reaches εes, while sampling patience εp terminates and reinitializes paths that fail to meet the threshold.The patience parameter approximates the budget needed to learn an environment along a curriculum path.
  • PATH:ACTIVE: PATH:ACTIVE samples current path-pointer environments proportionally to regret-based weights updated from temporal-difference errors and positive value loss.Higher S(θ) indicates unexpectedly high returns from recent interactions and therefore stronger learning potential.
  • PATH:ACTIVE: When an environment is solved, PATH:ACTIVE inserts successor pointers, evicts low-weight pointers, and concentrates the fixed-size buffer on high-regret regions.This expands exploration from promising nodes while maintaining |B| = N.
  • PATH transition: PATH switches from random exploration to active allocation after n ≥ Tswitch maximal paths terminate, continuing from the same buffer.The switch is intended to occur after broad exploration, when path terminations and revisits become frequent.
  • Coverage perspective: PATH models progress as coverage of an unknown reachable subgraph, where εes defines the current frontier and generalization extends mastery within graph-local neighborhoods.Training on visited nodes can master an unknown subset G(u) of each node’s t-ball neighborhood.
  • Coverage perspective: Random paths provide large early coverage gains when neighborhood overlap is limited, but later revisit saturated regions and yield little additional coverage.PATH:ACTIVE addresses the residual unmastered mass that random sampling increasingly misses.

4. Experiments

Experiments on MiniGrid and BipedalWalker evaluate PATH against curriculum and regret-based baselines using robustness and generalization tests. PATH consistently achieves stronger training performance, robustness, and generalization, while its random and active components complement one another.

  • Setup: Experiments compare PATH with DR, PLR, and ACCEL using PPO updates to isolate curriculum-design effects.DR randomizes uniformly, PLR replays scored environments, and ACCEL and PATH use active selection within curriculum structure.
  • 4.1. MiniGrid: MiniGrid uses a curriculum DAG of 15×15 layouts whose edges add blocks, with paths progressing from empty rooms to 60-block layouts.Each transition adds a uniformly sampled 1–5 blocks, mirroring ACCEL’s implicit editing curriculum.
  • 4.1. MiniGrid: PATH reaches higher MiniGrid training returns in fewer updates while sampling complex layouts stably.The comparison uses mean training returns and a sampling-complexity metric across training.
  • 4.1. MiniGrid: 98% mean solved rate is achieved by PATH on challenging MiniGrid robustness tests, compared with around 80% for ACCEL and PLR.All methods exceed 95% solved rate on 1000 random layouts, while PATH reaches 99.4%.
  • 4.3. Random Exploration vs. Active Path Allocation: PATH:RANDOM improves early through rapid complexity growth, whereas PATH:ACTIVE improves more slowly before overtaking it by 20k updates.The ablation attributes efficient budget use and broader coverage to combining both components.

5. Related Work

Related work automates RL curricula through teacher–student signals, self-paced reweighting, goal generation, randomized environments, and regret-based replay. PATH differs by explicitly modeling curriculum progression as a DAG and selecting at the path level.

  • Automated curriculum design for RL: Automated curriculum methods use competence signals, self-paced task reweighting, or goal generation to organize progression toward harder tasks.Other work generates diverse tasks or randomizes environments to support generalizable skills and sim-to-real transfer.
  • Unsupervised Environment Design: Unsupervised Environment Design formulates training as a bilevel minimax-regret game, while PLR approximates it by replaying prioritized environments.These approaches select or generate environments without explicitly representing curriculum paths.
  • Unsupervised Environment Design: ACCEL combines evolutionary environment generation with PLR prioritization and is described as a strong replay-based baseline.The related-work comparison positions ACCEL as the closest prior method.
  • Novelty: PATH makes evolution-induced progression explicit as a curriculum DAG and introduces path-level selection rather than only improving environment generation or prioritization.This is the paper’s stated distinction from ACCEL and prior UED approaches.

6. Conclusion

PATH performs active learning over a curriculum DAG, using rapid exploration followed by regret-based concentration. Its main scope boundaries are the need for a constructible DAG, evaluation against UED-family baselines, and future application to more complex settings.

  • PATH combines rapid exploration with regret-based concentration to allocate training over a curriculum graph.The framework operates active learning over the curriculum itself.
  • PATH assumes access to a constructible curriculum DAG, which is more natural in parameterized or observable settings.Automatic DAG construction is identified as a promising future direction for fully underspecified domains.
  • The evaluation focuses on UED-family baselines, while broader curriculum comparisons remain open.
  • Applying PATH to 3D embodied environments and sim-to-real transfer curricula is identified as an important next step.

Impact Statement

The paper anticipates methodological benefits from active path selection, including better sample efficiency and stronger generalization. PATH operationalizes this through a random exploration stage followed by active, regret-based successor expansion.

  • Active path selection is intended to improve sample efficiency and generalization in reinforcement learning.The anticipated impact is reducing interaction and computation needed to train robust policies.
  • PATH first samples random curriculum paths to expand coverage, then uses regret-driven signals to concentrate training on promising successors.
  • During the active stage, buffer weights are set from regret scores and sampling probabilities are proportional to those weights.
  • The algorithm expands sampled nodes through selected successors and maintains a bounded buffer by deleting minimum-weight entries when necessary.

B. Implementation Details

The experiments instantiate explicit curriculum DAGs for MiniGrid and BipedalWalker, align PATH's graph with edit-based curriculum structure, and evaluate robustness and generalization on held-out environments.

  • MiniGrid: MiniGrid uses a partially observable 15 × 15 navigation task with a 5 × 5 egocentric observation and a 250-step episode limit.
  • MiniGrid: PATH's MiniGrid curriculum DAG starts from an empty room and connects layouts differing by one valid clutter insertion.The graph uses fixed start and goal cells and allows at most 60 interior clutters.
  • BipedalWalker: BipedalWalker represents curriculum nodes as terrain-parameter vectors paired with random seeds, with edges formed by single valid coordinate edits.The terrain parameter space has d = 8 coordinates under coordinate-wise bounds.
  • BipedalWalker: BipedalWalker exploration follows outgoing DAG edges uniformly, with one to three successive edits introducing controlled local variation.
  • Evaluation: Robustness evaluation uses fixed MiniGrid layouts and six BipedalWalker terrain classes, while generalization uses 1000 held-out samples in each environment space.MiniGrid reports solve rates over 100 episodes per layout; BipedalWalker uses class-specific terrain suites and randomly sampled held-out terrains.
  • Evaluation: PATH is reported as stable and consistently outperforming baselines on fixed MiniGrid layouts.The supplied passage introduces the corresponding solved-rate results but does not provide the numerical values.

C.3.3. EXPERIMENT CONFIGURATION AND RESULTS

The Leaper experiment trains PATH and three baselines on 300 environments organized into 81 difficulty buckets. PATH achieves the highest average solved rate with the lowest standard error and still outperforms DR despite the small training set.

  • Experiment configuration: The experiment uses 300 training environments across 81 difficulty buckets and compares PATH, ACCEL, PLR, and DR across 5 seeds.Training uses 32 parallel PPO workers, 1000 update steps, and a phase shift after 500 PPO updates.
  • Results: PATH achieves the highest average solved rate with the lowest standard error across all methods.
  • Results: PATH outperforms DR even though the small training set makes uniform sampling a strong baseline.The result is presented as evidence that structured traversal yields gains over unstructured and replay-only methods.

C.4.1. COMPONENT ABLATION: PATH:RANDOM VS. PATH:ACTIVE.

PATH benefits from both curriculum phases: PATH:RANDOM broadens coverage, while PATH:ACTIVE concentrates training on environments with remaining learning potential. Figure 17 compares their training behavior on BipedalWalker robustness environments, including PATH’s switch after 8k updates.

  • PATH:RANDOM improves broad curriculum coverage, while PATH:ACTIVE improves sample efficiency by concentrating training on environments with remaining learning potential.The ablation separates the contributions of the two phases and evaluates robustness curves alongside generalization performance.
  • PATH runs PATH:RANDOM until 8k updates before switching to PATH:ACTIVE.The vertical dashed line in Figure 17 marks this transition.
  • PATH:RANDOM keeps the random phase throughout training, whereas PATH:ACTIVE uses the active phase from the beginning.These variants isolate the effects of starting with or omitting the two-stage schedule.

C.4.2. HYPERPARAMETER ABLATION: EARLY STOPPING AND SAMPLING PATIENCE.

The hyperparameter ablation finds that PATH is generally stable across reasonable early-stopping and patience settings, while removing early stopping substantially harms robustness and generalization. The default patience setting avoids overspending training on mastered nodes.

  • The comparison uses the same PATH:RANDOM phase for the first 8k PPO updates before transferring to PATH:ACTIVE through 20k updates.Robustness and generalization are reported in separate tables for the ablation variants.
  • PATH20 None performs substantially worse on both robustness and generalization than configurations using early stopping.Without early stopping, training repeatedly revisits effectively solved nodes, reducing curriculum coverage and slowing progress toward unmastered regions.
  • Increasing patience from Ep = 20 to Ep = 30 is slightly worse than the default, supporting Ep = 20 as an effective upper bound.The stated design goal is to maintain smooth curriculum progression without overspending on mastered nodes.
  • Aside from the no early stop setting, PATH is relatively insensitive to reasonable choices of Ees and Ep and remains stronger than baselines such as ACCEL.This ablation supports PATH as a stable and effective curriculum learning method.

C.4.3. ROBUSTNESS TO DAG MISSPECIFICATION

PATH remains robust when curriculum edges are randomly replaced, degrading gradually rather than catastrophically. In additional shortened and cropped-subspace experiments, the two-stage design and graph-based training retain measurable or substantial advantages.

  • At 30% perturbation, PATH is comparable to ACCEL on MiniGrid and exceeds ACCEL on BipedalWalker.MiniGrid solved rate is 0.53 versus 0.56, while BipedalWalker score is 168.24 versus 142.31.
  • At 50% perturbation, PATH still substantially outperforms DR and PLR on both benchmarks.Half of all curriculum edges are replaced with random jumps, yet performance degrades gradually rather than catastrophically.
  • The full BipedalWalker parameter space contains many inherently unsolvable configurations, motivating evaluation on a more tractable restricted subspace.The upper parameter limits are reduced from [10, 10, 5, 5, 9] to [6, 6, 3, 3, 6].
  • In the cropped solvable BipedalWalker subspace, PATH achieves a 97% improvement in solved rate and a 66% improvement in mean score over ACCEL.The evaluation uses 1000 uniformly sampled environments at 20k updates, averaged over 5 seeds.
Loading 2608.26469v1…