PaperlayerView exact revision on arXiv

Source-linked AI summary

Scaling Curriculum Learning For Autonomous Driving

Cevahir Koprulu, David Paz, Feng Tao, Yuliang Guo, Xinyu Huang, Ufuk Topcu, Liu Ren

arXiv:2608.22549v1cs.AIcs.LG

TL;DR

Batched autonomous-driving simulators make RL training highly scalable, but uniform scenario sampling remains sample-inefficient. CL4AD frames scenario selection as UED and adds utility functions for success and realism; experiments show substantially faster learning than domain randomization and heuristic curricula, with a documented realism-related limitation.

  • Problem

    Uniform domain randomization wastes interactions on scenarios that are too easy or too difficult to provide useful learning signals, limiting sample efficiency despite high simulator throughput.

  • Method

    CL4AD integrates prioritized level replay into batched autonomous-driving simulation and uses regret-, success-, and realism-based utility functions to adaptively prioritize scenarios.

  • Results

    99% success is reached a billion steps earlier than domain randomization, reducing wall-clock time by 77%; CL4AD also outperforms heuristic curricula except at the largest scale.

  • Takeaways & Limitations

    Curriculum learning scales to batched autonomous-driving simulators and improves training efficiency across large scenario sets, including under limited compute.

  • Takeaways & Limitations

    CL4AD currently relies on variants of PLR and real self-driving datasets such as WOMD because GPUDRIVE uses predefined scenarios.

Abstract

from arXiv · show

Batched simulators for autonomous driving have recently enabled training reinforcement learning (RL) agents at scale, encompassing thousands of traffic scenarios and billions of interactions within a matter of days. Although such high-throughput feeds RL algorithms faster than ever, their sample-efficiency has not kept pace: As the standard training scheme, domain randomization uniformly samples scenarios, thereby consuming a vast number of interactions on cases that contribute little to learning. Curriculum learning offers a remedy by adaptively prioritizing scenarios that matter most to policy improvement. We present CL4AD, the first integration of curriculum learning into batched autonomous driving simulators by framing scenario selection as an unsupervised environment design problem. We introduce utility functions that shape curricula based on success rates and the realism of the agent's behavior, in addition to existing regret-estimation functions. Large-scale experiments in GPUDRIVE demonstrate that curriculum learning achieves a 99% success rate a billion steps earlier than domain randomization, reducing wall-clock time by 77%, and outperforms heuristic curricula with static and dynamic attributes, with only one exception at the largest scale. An ablation under limited compute shows that curriculum learning improves sample efficiency by 67%. We also investigate how utility functions behave at scale, and how prioritized scenarios evolve during training. We release an implementation of CLForAD in GPUDRIVE.

1 Introduction

Batched autonomous-driving simulators provide massive interaction throughput, but uniform scenario sampling wastes interactions on cases that are too easy or too difficult. CL4AD integrates automated curriculum learning through UED to prioritize scenarios contributing to policy improvement.

  • Motivation: Batched simulators train RL agents across hundreds to thousands of scenarios in parallel, enabling billions of interactions within days.Self-play allows a single policy to control all vehicles while scenarios run concurrently.
  • Motivation: Uniform domain randomization wastes interactions on scenarios that provide insufficient learning signal or are currently too difficult for the policy.Curriculum learning instead prioritizes scenarios that contribute most to policy improvement.
  • CL4AD: CL4AD is the first integration of automated curricula into a batched autonomous-driving simulator, framing scenario selection as unsupervised environment design.Its utility functions adaptively shape training toward scenarios at the edge of the agent’s capabilities.
  • Findings: CL4AD accelerates RL training by hundreds of millions of steps compared with domain randomization and heuristic curricula.The implementation is provided in the open-source GPUDRIVE simulator.
  • CL4AD: CL4AD introduces utility functions based on realism and safety-critical success alongside existing regret-based functions.The study also examines prioritized-scenario evolution, utility correlations, self-play learning frontiers, and the distinction between reward optimality and behavioral realism.

2 Related Work

Prior work has expanded curriculum learning from general RL and UED to autonomous driving, while batched simulators have increased throughput but still commonly sample scenarios randomly. Existing AD curricula include heuristic, staged, automated, and UED-based approaches.

  • Autonomous-driving simulators: Waymax, Nocturne, CARLA, and Metadrive support autonomous-driving simulation with combinations of real driving data, procedural generation, and non-data-driven environments.Batched simulators such as Waymax, GPUDRIVE, and GIGAFLOW increase throughput but still rely on random scenario generation or sampling.
  • Curriculum learning for RL: Curriculum-learning research studies goal-conditioned, contextual, and UED settings, using signals such as learning progress or changing success probability.UED methods generate environment levels intended to accelerate policy learning and generalization.
  • UED methods: UED methods differ in how they generate or replay levels, including regret-based generation, replay stabilization, mutation, and prioritized replay.PLR scores and replays encountered levels without training a teacher or mutating parameters.
  • Curriculum learning for AD: AD curriculum methods range from heuristic and multi-stage curricula to automated intersection curricula and recent UED methods such as RE-PAIRED and ACCEL.These approaches target faster training of self-driving policies under varying agents, positions, weather, or urban scenarios.

3 Background

The paper models traffic scenarios as underspecified partially observable stochastic games and formulates curriculum learning as unsupervised environment design. Scenario utilities determine which levels are prioritized, spanning constant, regret-based, success-based, and learnability-oriented criteria.

  • Traffic scenarios as POSGs: Traffic scenarios are modeled as partially observable stochastic games to represent multiple interacting agents, observations, rewards, and stochastic dynamics.Scenario-specific road layouts, traffic rules, collision dynamics, initial states, and agent observations define the game.
  • Traffic scenarios as POSGs: An underspecified POSG represents a set of games through scenario parameters that determine attributes such as dynamics, while WOMD supplies about 100,000 scenarios.Scenarios vary in layouts, speed limits, and numbers of vehicles.
  • Unsupervised environment design: UED generates a sequence of scenarios through a level generator whose utility measures each scenario’s contribution to improving the policy.The generator produces a distribution over scenarios conditioned on the current policy.
  • Utility functions: Domain randomization assigns every scenario a constant utility, whereas UED primarily differs through its chosen utility function.The main utility categories are regret-based and success-based.
  • Utility functions: Regret-based utilities prioritize levels the current policy cannot yet solve, using estimates such as AMGAE, PVL, or MaxMC when optimal returns are unavailable.MaxMC uses the highest return achieved so far to mitigate potential bias from bootstrapped estimates.
  • Utility functions: Success-based utilities evaluate whether agents reach goal states, while learnability measures inconsistency in solving a scenario across controlled agents.In sparse-reward settings, regret utilities may correlate poorly with success rates and identify the learning frontier inaccurately.

4 Curriculum Learning for Autonomous Driving at Scale

CL4AD scales prioritized level replay with utility functions to batched autonomous-driving simulation, combining large-scale concurrent training with asynchronous scenario and episode management. It adds utilities that measure safe success and behavioral realism while adapting replay toward useful scenarios.

  • Scaling CL4AD: CL4AD integrates prioritized level replay into batched autonomous-driving simulation and scales it across hundreds of concurrent scenarios, tens of agents, tens of thousands of scenarios, and billions of steps.The method retains the UED framing while adapting it to the scale of GPU-accelerated self-play.
  • Asynchronous management: CL4AD manages asynchronous episodes by tracking all controlled agents across concurrent scenarios and computing utility from their collective self-play behavior.This accommodates different scenario horizons and agent termination times in GPUDRIVE’s logged-data scenarios.
  • Scenario sampling: PLR samples scenarios from the training set or a rolling replay buffer, scores them with a utility function, and prioritizes high-scoring levels.Its sampling combines utility rank with staleness so that levels not sampled recently receive higher likelihood.
  • Scenario sampling: The replay distribution uses utility and staleness terms to prevent scores of stored scenarios from becoming off-policy while they remain unsampled.The staleness component increases the chance of replaying levels that have waited longer.
  • Novel utilities: CL4AD introduces learnability-hard, GC-ADE, and Act-MAE utilities for safe success and behavioral realism.Learnability-hard excludes goal completions involving collisions or off-road events; GC-ADE and Act-MAE compare agent positions and actions with logged trajectories.
  • Training procedure: Algorithm 1 alternates curriculum sampling, rollout collection, curriculum updates from terminated scenarios, and self-play policy updates.The procedure maintains scenario and experience buffers while sampling scenarios into concurrently simulated worlds.

5 Experimental Results

Experiments in GPUDRIVE show that curriculum learning accelerates training across compute settings and dataset scales, while utility functions shape distinct scenario priorities and exhibit scale-dependent behavior. The results also identify a boundary: realism-based prioritization improves training efficiency but cannot make reward-optimal behavior realistic without corresponding reward terms.

  • Learning acceleration: A billion steps earlier, PLR reaches 99% success than DR in unseen test scenarios, reducing wall-clock time by 77%.PLR also reaches the same success rate 40% and 66% faster than Heuristic-Sparse and Heuristic-Dense, respectively.
  • Scenario selection: Regret-based utilities increasingly prioritize denser-than-average scenarios, whereas different utility functions produce distinct ranking persistence and scenario-selection patterns.U GC-ADE and U MaxMC retain higher rank correlations at lag 5, while success-based utilities and U Act-MAE reshuffle faster.
  • Learning acceleration: 67% faster, PLR reaches 99% success than DR under limited compute, despite the constrained setup taking about four times longer in wall-clock time.The smaller buffer also limits scenario diversity used for updates.
  • Scaling with dataset size: Over 55% fewer interactions are needed to reach 99% success in 10,000 scenarios, while PLR improves sample-efficiency by 72% in 80,000 scenarios.The 10,000-scenario result combines PLR with U MaxMC and U Act-MAE; the 80,000-scenario result combines PLR with U Learn.
  • Scale effects: Multi-agent self-play sustains a learning frontier at scale, while large policy updates make replay-buffer scores rapidly off-policy.Higher score temperatures spread sampling probability across more scenarios to address this effect.
  • Realism boundary: Realism-based utilities accelerate training but cannot steer behavior toward realism because they prioritize scenarios without changing rewards.DR achieves higher map-based WOSAC scores, while PLR variants achieve lower displacement error but diverge from human-like driving.
  • Utility-function behavior: U MaxMC is the most reliable regret-based utility, success-based functions work at every scale, and no single utility dominates across scales.U AMGAE and U PVL improve with dataset scale, while realism-based utilities accelerate training but cannot improve realism.

6 Conclusion

CL4AD integrates curriculum learning into batched autonomous-driving simulators by adaptively prioritizing scenarios with utility functions based on regret, success, and realism. Its experiments support scale-dependent curriculum benefits and provide practitioner guidance, while current use remains tied to PLR and real driving datasets.

  • CL4AD frames scenario selection as unsupervised environment design and combines PLR with regret-, success-, and realism-based utility functions.
  • CL4AD reaches 99% success up to 77% faster than domain randomization and generally outperforms heuristic curricula, except at the largest scale.
  • Limitations and future work: CL4AD currently relies on PLR variants and real self-driving datasets because GPUDRIVE samples predefined scenarios rather than generating them.
  • PLR configuration at scale: A full-size replay buffer and score temperatures β ∈{2, 4} are recommended because low temperatures concentrate sampling on too few scenarios as datasets grow.
  • Utility function selection: No single utility function dominates across all scales, while success-based functions are particularly strong at large scale and realism-based functions accelerate training without improving realism.

B.4 Computational overhead of CL4AD

CL4AD’s curriculum updates and sampling impose little computational overhead relative to simulator rollouts. Across a billion-step run, the added training wall-clock time remains below 1%.

  • Curriculum updates and scenario sampling occur every Tsce = 2,000,000 interactions.
  • ∼4.3s of curriculum updates per inter-sampling segment represent ∼1% of ∼428.5s of evaluation and rollout time.
  • ∼0.66ms per PLR sampling call is small, while ∼6.15s per sampling block is simulator overhead for scenario assignment and resetting.
  • Over one billion steps, CL4AD adds less than 1% to training wall-clock time, with approximately 36 minutes of curriculum updates and 0.3 seconds of PLR sampling.

C.2 Self-play PPO Training

The experiments train autonomous-driving agents with self-play PPO in batched worlds, using a shared decentralized policy and larger throughput settings for the main cases than for the limited-compute ablation.

  • Self-play trains a single shared decentralized policy that controls all vehicles in each scenario in parallel.
  • The limited-compute ablation uses W = 50 worlds, whereas the larger-scale cases use significantly more concurrent worlds and larger experience batches.
  • The PLR search varies staleness coefficient ρ ∈{0.1, 0.3} and score temperature β ∈{2, 4}.
  • A 2-billion-step training run on an NVIDIA H200 takes around 60 hours, while the 1-billion-interaction ablation on an RTX A5000 takes over 110 hours.

E.1 WOSAC Evaluation

Final policies are evaluated with WOSAC metrics under self-play and ego-only settings. Curriculum choice mainly affects map-based metrics, while kinematics and interaction scores remain comparable across methods.

  • WOSAC evaluation covers all controlled agents via self-play and an ego-only setting with other agents replaying logged trajectories.
  • Kinematics and interaction scores are comparable across methods, indicating little effect from curriculum choice on these metric categories.
  • Domain randomization scores higher on map-based Dist Edge and Offroad metrics because WOSAC measures distributional similarity to logged data.

U MaxMC 0.5 2 0.3

The WOSAC evaluation compares final trained policies on 150 test scenarios, while the reported analysis finds that curriculum variants improve task performance but remain less realistic than uniform sampling.

  • 150 test scenarios form the self-play WOSAC evaluation set for final trained policies.
  • Curriculum variants achieve lower displacement error but diverge from human driving patterns in map-based metrics.
  • The realism gap widens in the ego-only setting, while kinematics remain the largest gap from the oracle.

E.2 Quantitative Results

The quantitative results track performance, regret, realism, and learnability across training and test partitions, including an ego-only evaluation on 150 test scenarios.

  • Figures 7–10 report progression of performance, regret, realism, and learnability across training and test partitions.
  • Training-partition regret, learnability, and realism reveal how automated curricula affect training.
  • 150 test scenarios are used for the case 1 ego-only WOSAC evaluation with final trained policies.

E.3 Qualitative Results

The qualitative analyses examine how utility functions prioritize scenarios, how curricula evolve during training, and how sensitive or complementary different utility choices are.

  • Preplay progression: Higher score temperatures produce more uniform replay distributions, reducing the influence of scenario ranking.
  • Sensitivity to sampling interval and buffer size: All tested buffer and sampling configurations exceed domain randomization in return and success rate at every checkpoint.
  • Combining utility functions across categories: All three cross-category combinations converge with their individual components by 1,400 policy updates and exceed domain randomization throughout.
  • Combining utility functions across categories: U Learn-hard is the slowest individual PLR variant, whereas U MaxMC + U Learn-hard matches or exceeds both components early in training.
  • Scale comparisons: Figures 7–9 compare regret, realism, and learnability progression for 1,000, 10,000, and 80,000-scenario experiments.
  • Compute ablation: Figure 10 evaluates performance, regret, realism, and learnability in training and test partitions across three compute-ablation runs.
Loading 2608.22549v1…