Robust Autonomy Emerges from Self-Play

Marco Cusumano-Towner; David Hafner; Alex Hertzberg; Brody Huval; Aleksei Petrenko; Eugene Vinitsky; Erik Wijmans; Taylor Killian; Stuart Bowers; Ozan Sener; Philipp Krähenbühl; Vladlen Koltun

Source-linked AI summary

Robust Autonomy Emerges from Self-Play

Marco Cusumano-Towner, David Hafner, Alex Hertzberg, Brody Huval, Aleksei Petrenko, Eugene Vinitsky, Erik Wijmans, Taylor Killian, Stuart Bowers, Ozan Sener, Philipp Krähenbühl, Vladlen Koltun

arXiv:2502.03349v1Feb 5, 2025cs.LGcs.AIcs.RO

TL;DR

Autonomous driving needs policies that handle diverse, uncertain interactions without relying entirely on human driving data. The paper trains a parameterized policy through massive-scale simulated self-play, achieving state-of-the-art performance across three benchmarks and strong robustness in simulation. The results indicate that self-play can produce human-like, robust driving without human data during training.

Problem
The paper addresses whether simulated self-play can produce naturalistic and robust driving policies without human driving data.
Method
The authors train a shared parameterized policy entirely through self-play, using GIGAFLOW to simulate diverse traffic agents and focus learning on informative transitions.
Results
A single self-play-trained policy achieves state-of-the-art performance on CARLA, nuPlan, and Waymax, with over 3 million km, or 17.5 years, between incidents in simulation.
Takeaways & Limitations
Large-scale self-play can yield naturalistic and robust driving policies without human data during training, potentially reducing reliance on human data collection.
Takeaways & Limitations
Benchmark infractions are partly attributable to benchmark limitations, including externally caused collisions, initialization in collision, and simulator-specific traffic behavior.

Abstract

from arXiv · show

Self-play has powered breakthroughs in two-player and multi-player games. Here we show that self-play is a surprisingly effective strategy in another domain. We show that robust and naturalistic driving emerges entirely from self-play in simulation at unprecedented scale -- 1.6~billion~km of driving. This is enabled by Gigaflow, a batched simulator that can synthesize and train on 42 years of subjective driving experience per hour on a single 8-GPU node. The resulting policy achieves state-of-the-art performance on three independent autonomous driving benchmarks. The policy outperforms the prior state of the art when tested on recorded real-world scenarios, amidst human drivers, without ever seeing human data during training. The policy is realistic when assessed against human references and achieves unprecedented robustness, averaging 17.5 years of continuous driving between incidents in simulation.

1. Introduction

The paper demonstrates that large-scale simulated self-play can produce naturalistic, robust driving without human training data. A single generalist policy surpasses prior benchmark-specific systems across three benchmarks and shows strong realism and robustness.

Self-play yields naturalistic and robust driving policies using a minimalistic reward function and no human data during training.
1.6 billion km of simulated driving enables self-play training at a scale orders of magnitude beyond prior experiments.
The parameterized policy controls varied traffic participants and driving styles, with behavior parameters modifiable at test time without additional training.
A single policy achieves state-of-the-art performance on CARLA, nuPlan, and Waymax after training entirely through self-play without benchmark-specific datasets.
Over 3 million km, or 17.5 years of continuous driving, separate incidents in simulation on average.

2. GIGAFLOW

GIGAFLOW trains a shared, parameterized policy through massive parallel self-play in simple simulated worlds. Its design combines diverse agent conditioning, uncertainty modeling, high-throughput simulation, and advantage-focused learning.

GIGAFLOW: GIGAFLOW trains a generalist policy from the world, the agent’s state, nearby agents, and a behavior-conditioning parameter.
GIGAFLOW: Self-play replaces scripted scenarios and human traces, allowing realistic traffic behaviors to emerge from large-scale interaction.
GIGAFLOW world: A single shared policy controls vehicles, pedestrians, and cyclists while conditioning varies dynamics, reward weights, and driving styles.
GIGAFLOW simulation and training: 4.4 billion state transitions per hour are generated on one 8-GPU node through batched simulation and optimized localization, collision checking, and observation retrieval.
GIGAFLOW simulation and training: Advantage filtering removes up to 80% of low-impact samples, concentrating learning on informative transitions from underexplored parts of the data distribution.

3. Zero-shot evaluation on driving benchmarks

A single GIGAFLOW policy is evaluated zero-shot across three independent driving benchmarks and surpasses specialist state-of-the-art systems despite training only through self-play. Its performance generalizes across actors, maps, observation noise, and differing scoring rules, though benchmark artifacts account for many infractions.

Cross-benchmark performance: A single policy trained purely in self-play surpasses state-of-the-art specialist policies across CARLA, nuPlan, and Waymax without fine-tuning.The policy is evaluated zero-shot in each benchmark environment.
Generalization: The policy generalizes to diverse actor behaviors, including reactive vehicles, pedestrians, cyclists, and different Intelligent Driver Model variants.The benchmarks vary actor behavior and interaction patterns.
Generalization: The policy generalizes to real-world maps and driving situations without training on recorded driving logs or human-designed scenarios.The evaluated settings include merges, unprotected turns, pedestrians, and cyclists.
Generalization: Zero-shot performance extends to real-world observation noise, including occlusions, incorrect or missing traffic-light states, and late-revealed obstacles.GIGAFLOW uses only minimalistic noise modeling during training.
Scoring robustness: The policy remains state of the art across benchmark scoring methodologies despite distinct and sometimes contradictory definitions of good driving.The benchmarks reward and penalize behaviors differently, such as red-light violations, hard braking, and forward progress.
Benchmark limitations: 20% of CARLA infractions arise from pedestrians or cyclists entering the roadway without reacting to evasive maneuvers, while 33% arise from gridlocks caused by CARLA traffic.Additional infractions involve fuzzy stop-sign and red-light checks, accounting for 16% of all infractions.
Collision analysis: In nuPlan, 15 collisions occurred across 1118 scenarios, including nine attributed to invalid initialization or sensor noise.Other cases involved non-reactive pedestrians or traffic-light violations by other agents.
Collision analysis: In Waymax, 187 collisions occurred across 44 097 scenarios, but only 2.7% were judged at-fault and avoidable by GIGAFLOW.Most collisions were attributed to unavoidable benchmark-agent behavior or initialization in a colliding state.

4. Analysis

Analysis shows that large-scale self-play produces increasingly capable driving behaviors, danger-sensitive value estimates, context-dependent decisions, long-horizon responses, and controllable driving styles. The same infrastructure also supports highly robust simulation and human-like zero-shot behavior.

Evolution during training: Complex maneuvers emerge with training scale, including unprotected turns, crowded-road driving, dangerous merges, bottleneck negotiation, and traffic-flow merging.Many skills are mastered after 10^11 to 10^12 training steps, corresponding to 90 to 1600 million km driven.
Value and attention: The value network assigns low value to states involving excessive cornering speed or imminent collision from high relative velocity.Its danger estimates are evaluated by densely sampling driver positions and orientations.
Value and attention: The policy and value networks attend to different actors: value estimates reflect long-term danger, while policy actions change only when mitigation is possible.Attention is measured through action-distribution mutual information and changes in value estimates when actors are removed.
Contingent behavior: The policy executes discrete and routing maneuvers contingent on nearby-vehicle behavior, including lane changes, passing, and circling a block instead of making a three-point turn.These responses differ when a nearby vehicle behaves predictably versus unexpectedly.
Long-horizon behavior: The networks respond to distant salient features, such as an obstruction 150 m down the road, while ignoring nearby irrelevant actors.This behavior reflects long-term return optimization without dedicated planning or search modules.
Conditioning and style: A single trained policy produces a continuum of driving styles by changing conditioning parameters, vehicle dimensions, goals, and reward coefficients without retraining.Examples include squeezing through narrow passages and performing tight turns.
Robust simulation: 17.5 years of driving and over 3 million km elapse on average before an incident in the robust simulation configuration.This configuration reduces dynamics noise, increases control frequency, and prioritizes safety through conditioning parameters.
Human realism: A zero-shot WOSAC realism score of 0.62 exceeds several supervised autoregressive prediction approaches despite no human training data.WOSAC evaluates reproduction of real-world driving behavior for simulation.

5. Discussion

The discussion identifies simulation-to-reality transfer and perception modeling as remaining boundaries, while describing GIGAFLOW’s scalable GPU simulation design and broader implications for self-play.

5. Discussion: Simulation-to-reality transfer is required before claims can be made about self-play policies’ efficacy in the physical world.
5. Discussion: The work largely abstracts perception, so operational deployment would require substantially closer modeling of sensing and perceptual inputs.
5. Discussion: The findings show that human-like, robust policies can be trained without real-world driving traces and tested among human participants in recorded scenarios.
5. Discussion: The authors suggest applying self-play to agents acting with humans in mobile robotics and digital environments, potentially reducing human-data collection needs.
A. Simulator Design: A single GIGAFLOW instance simulates many worlds in parallel using GPU-accelerated batched operators for simulation tasks.
A. Simulator Design: Road localization converts world-frame positions into Frenet coordinates after spatial-hash lookup, avoiding prohibitively expensive O(N × A × P) geometric checks.

B.2. Actions and dynamics

GIGAFLOW uses discrete jerk-based vehicle control, randomized dynamics and rewards, and diverse traffic conditions to produce varied driving behaviors in simulation.

B.2. Actions and dynamics: 12 discrete actions control vehicle acceleration changes through a jerk-actuated bicycle dynamics model.
B.2. Actions and dynamics: Setting lateral acceleration to zero when longitudinal acceleration changes sign makes it easier to wait or maintain constant velocity, producing smoother trajectories.
B.2. Actions and dynamics: Vehicle velocities and accelerations are numerically integrated, clipped, and stabilized with sign-change handling and randomized dynamics coefficients.
B.2. Actions and dynamics: The steering update limits angular changes to δmax = 0.6 rad s−1 and ϕmax = 0.55 rad.
B.3. Reward: The reward combines goal progress, velocity, reversal, stop-line, collision, off-road, comfort, lane-alignment, and lane-centering terms.
B.3. Reward: Randomized reward weights and simulator parameters generate behavioral diversity, including varied driving styles, vehicle sizes, traffic-light patterns, and responses to erratic agents.

C. Training algorithm

The training algorithm combines PPO with adaptive advantage filtering and distributed experience collection to process massive synthetic driving data more efficiently.

C. Training algorithm: PPO trains separate policy and critic networks, with multiple GPUs collecting experience independently before synchronizing gradients.
C. Training algorithm: Population Based Training was used during development to optimize PPO and simulator hyperparameters in a simplified setup.
C. Training algorithm: Advantage filtering discards most low-information transitions, functioning as a prioritized replay variant in which many transitions are sampled zero times.
C. Training algorithm: The adaptive threshold η is set to 1% of the moving-average maximum advantage magnitude, making filtering insensitive to reward scale.
C. Training algorithm: ∼80% of samples are filtered on average, exceeding 90% in early epochs, and training throughput increases 2.3-fold from 0.53 to 1.2 million steps per second.

D. Neural network architecture

The policy and critic use compact fully connected networks, while set-valued observations are encoded with permutation-invariant networks and reconstructed to control memory use.

D. Neural network architecture: The actor and critic each contain 3 million parameters, with a combined 6 million trainable parameters.
D. Neural network architecture: A [1024 × 1024 × 1024] MLP maps concatenated feature embeddings to either an action distribution or scalar value estimate.
D. Neural network architecture: Set-valued observations use per-element fully connected encoders followed by channel-wise max pooling and concatenation.
D. Neural network architecture: Because observation sets contain over 10^8 feature vectors per inference step, the rollout buffer stores world states and reconstructs observations during minibatch training.
D. Neural network architecture: Feature dropout removes 40% of boundary features and 50% of lane features, fitting the system on 40 GB A100 GPUs while modeling sensor noise.

E. Benchmark evaluation

The evaluation uses benchmark-provided simulators alongside GIGAFLOW as a co-simulator, with benchmark-specific rollout procedures.

GIGAFLOW is integrated with each benchmark’s simulation infrastructure by translating observations and running the policy as a co-simulator.Waymax and CARLA use one GIGAFLOW step per evaluation step, while nuPlan additionally requires trajectory predictions.
In nuPlan, GIGAFLOW policies control all agents for 0.3 seconds before the subsequent observed driver trajectory is used for prediction.

E.1. nuPlan benchmark evaluation

The nuPlan evaluation focuses on its most realistic reactive closed-loop challenge and reports a zero-shot state-of-the-art aggregate score, with explicit evaluation constraints.

The nuPlan benchmark contains training, validation, and held-out test sets built from thousands of hours collected across Las Vegas, Boston, Pittsburgh, and Singapore.
Challenge 3 is evaluated because both the driver and traffic vehicles operate in closed loop, making it the most realistic nuPlan setting.The benchmark also includes open-loop and closed-loop non-reactive challenges, but the study focuses only on Challenge 3.
The evaluation uses Val14 as a proxy for nuPlan’s leaderboard because the online evaluation servers are no longer accessible.Val14 was selected because it has been shown to proxy leaderboard evaluation and has publicly reported results.
The GIGAFLOW policy achieves state-of-the-art results zero-shot on nuPlan’s Challenge 3 closed-loop score.The score ranges from 0 to 100 and weights collisions, drivable-area and direction compliance, progress, time-to-collision, speed limits, and comfort.

E.3. Waymax benchmark evaluation

Waymax evaluation uses a distributed runner on the full WOMD validation set and reports standard metrics alongside a study-defined aggregate score.

The Waymax evaluation runs across 44,097 validation scenarios, each 8 seconds long at 10 Hz, using specified object limits and IDMRoutePolicy settings.
The evaluation uses Waymax’s agent-overlap, off-road-rate, log-divergence, and SDC-kinematic-infeasibility metrics, plus a modified route-progress ratio.The modified ratio compares GIGAFLOW’s driven distance with the expert trajectory per scenario and clamps the value at 100% when the expert drove less than 1 m.
Because Waymax lacks an aggregate score, the study proposes one combining progress, collision rates, and off-road rates.This proposed aggregate is the single Waymax score reported in the figures.
The Waymax results are presented in Table A7.

E.4. Waymo Open Sim-Agents Challenge

GIGAFLOW is evaluated zero-shot in multiagent Waymo scenarios while conditioning behavior on reward parameters. It approaches expert-model performance without WOMD map or human training data and produces varied driving styles.

Waymo Open Sim-Agents Challenge: GIGAFLOW is evaluated zero-shot on WOSAC using recorded road-user scenarios from WOMD, with behavior modeled for vehicles and bicycles by GIGAFLOW.Pedestrians use an IDM-like policy, and generated positions include Gaussian noise to account for sensor noise.
Waymo Open Sim-Agents Challenge: A 2-point improvement in the overall metametric results from adding Gaussian position noise with σ = 0.0125.The comparison is against rollouts without added noise.
Waymo Open Sim-Agents Challenge: GIGAFLOW approaches expert-model performance despite never seeing WOMD map or recorded human data during training.It performs strongly on collision and off-road metrics and maintains respectable acceleration metrics through smooth, dynamically feasible rollouts.
Waymo Open Sim-Agents Challenge: WOSAC compares generated trajectories using likelihood measures of ground-truth behavior, while GIGAFLOW explicitly penalizes collisions and off-road outcomes during training.This objective limits expected improvement on likelihood metrics for collision and off-road behavior.
Reward conditioning analysis: Reward conditioning varies driving behavior through 12 reward parameters, whose trajectory effects are analyzed by sampling 200 parameter settings and clustering rollouts.
Reward conditioning analysis: Conditioning on reward parameters makes behavior variation interpretable, including lane-position bias, goal-directed lane changes, and wider turns for comfort.
Reward conditioning analysis: Reward conditioning supports one policy that handles diverse hidden goals and behaviors while permitting alignment toward an ideal logged driver.The same policy can support chaotic self-play for robustness and safe, comfortable control.

F.2. Ablation studies

Ablations examine how algorithmic choices affect benchmark performance and self-play robustness. Advantage filtering improves both training throughput and converged behavior, while long-form evaluation shows nearly accident-free driving under conservative conditioning.

Experimental setup: The ablation suite evaluates algorithmic choices using a reduced budget of 660 GPU-hours, approximately 30% of the final training run.The experiments are designed to isolate the influence of individual algorithmic features on final performance.
Advantage filtering: Without advantage filtering, training plateaus substantially lower on the Carla LAV benchmark and misses up to 5% of goals within the time limit.The unfiltered version also produces timed-out episodes during self-play evaluation.
Long-form evaluation: At 15 Hz decision-making frequency, conservative GIGAFLOW agents average 3 million km before a collision or off-road event.The long-form self-play environment uses 50 drivers and removes dynamics noise and traffic-light randomization.