Building reliable sim driving agents by scaling self-play

Reliability gap

• State-of-the-art imitation models can score highly on distributional metrics while their collision and off-road rates remain appreciable.The cited top WOSAC 2024 models had aggregate collision rates of about 4% and off-road rates of 1–2%.Source: §1 · pp. 1–2

Motivation

• Reliable agents should minimize unintended behavior because unreliable trajectories can obscure whether observed AV outcomes are meaningful or simulator artifacts.This matters especially when agents generate rare safety-critical scenarios for evaluation.Source: §1 · pp. 1–2

Research questions

• The paper tests whether self-play reinforcement learning can improve reliability as data scales, including generalization to unseen and out-of-distribution scenarios.The study evaluates performance across thousands of Waymo Open Motion Dataset scenarios using a semi-realistic human-perception framework.Source: §1 · p. 2

Contribution

• At scale, self-play PPO agents consistently meet criteria for staying on-road, avoiding collisions, and reaching target positions.The authors frame this as supporting nominal and safety-critical traffic simulation.Source: §1 · p. 3
• The work also takes a step toward fine-tuning agents for underrepresented behaviors and releases pretrained agents for reuse and reproduction.The agents are open-sourced at the Emerge-Lab/gpudrive repository.Source: §1 · p. 3

2.1 Dataset and simulator; 2.2.1 Task definition

• GPUDrive supplies 160,147 WOMD traffic scenarios with static road graphs and nine-second joint human-trajectory demonstrations; agents must reach dataset endpoints within 91 steps while avoiding collisions and staying on-road.The task assumes dataset human agents reached their endpoints within the demonstration horizon.Source: §2.1 · p. 3, §2.2.1 · p. 4

2.2.2 Metrics

• Performance is measured by goal achievement, collisions, off-road events, and other failures; agent-based aggregation supplements scene-based metrics, and goal achievement is a binary target-reaching measure.The goal metric is less directly comparable to WOSAC than its route-progress metric because this task targets a particular endpoint rather than logged-trajectory imitation.Source: §2.2.2 · p. 4

2.3 State and observation space

• Agents receive a partial, ego-centric current-timestep view, with a 50-meter observation radius and a reduced local road graph capped at 200 points.The road graph is sparsified for computational efficiency; the authors report this suffices for navigation without collisions or off-road events.Source: §2.3 · pp. 4–5

2.4 Action space and dynamics model

• Each agent selects jointly modeled discrete acceleration and steering actions, yielding 91 possible actions, and the simulator advances them with an Ackermann bicycle model.Acceleration uses seven values over [−4, 4] and steering uses 13 values over [−π, π].Source: §2.4 · p. 5

2.5 Reward function

• Individual rewards combine goal achievement with penalties for collisions and off-road events.The reward is a weighted combination of these criteria.Source: §2.5 · p. 5

2.6 Collision behavior

• Episodes continue after collisions or off-road events, and agents receive a penalty for each event, allowing multiple penalties per episode.Source: §2.6 · p. 5

2.7 Model

• The model processes ego state, road graph, and partner observations independently, then concatenates and max-pools them into a shared embedding for separate actor and critic heads.The network has approximately 50,000 trainable parameters.Source: §2.7 · p. 5

2.8 Training

• Self-play PPO controls up to 64 agents with a shared decentralized policy and trains on batches of 800 scenarios, uniformly resampled every two million steps.The agents initially behave randomly and crash frequently, then develop smoother trajectories with high goal-reaching rates.Source: §2.8 · pp. 5–6

Self-play for agents in games

• Self-play has produced strong agents in two-player games and many-player settings, though most cited successes involve zero-sum games unlike driving's likely general-sum, many-agent interactions.Source: §3 · p. 6

RL for driving agents

• Driving RL commonly trains in simulators built on open datasets, often in log-replay mode with one learning agent and other agents replaying human trajectories or using hand-coded policies.The cited human crash-rate benchmarks range from one police-reported crash per 800,000 km in the United States to one per 24,800 km in challenging San Francisco domains.Source: §3 · p. 6
• The authors distinguish their work by scale and lower crash and off-road rates, while contrasting its roughly one crash per 10 km with Gigaflow's one per 100,000–1,000,000 km.They emphasize complementary open-source training artifacts and log-replay curricula of easy and challenging tasks.Source: §3 · p. 6

4.1 Scaling with data

• 99.81% goal-reaching, 0.44% collision, and 0.31% off-road rates on 10,000 held-out test scenarios approach the benchmark ceiling.The rates are scene-based; collision and off-road incidents may overlap.Source: Table 1 · p. 6, §4.1 · p. 8
• At 2 billion training steps, agents achieve 99.84% goal-reaching, 0.38% collision, and 0.26% off-road rates on the training dataset.Excluding agents that need to travel no more than 2 meters gives 99.40% goal-reaching, 0.5% collision, and 0.6% off-road rates.Source: §4.1 · p. 7
• Training for 24 hours on one NVIDIA A100 GPU generated 2 billion steps, and metrics continued to improve gradually during the final four hours.The training curves report average reward, goal achievement, and collision and road-edge rates.Source: §4.1 · p. 7, Figure 4 · p. 7
• With 10,000 or more training scenarios, results show no significant train-test gap; Figure 3 plots validation performance against training-set size.Experiments use 100, 1,000, 10,000, and 100,000 training scenarios.Source: §4.1 · p. 7, Figure 3 · p. 7

4.2 Extrapolative generalization and fast fine-tuning

• With goals placed behind agents, goal-reaching fell from 100% to 53.5% in hand-designed out-of-distribution scenarios.The 13 scenarios contained 27 agents; only 1.86% of training-set agents had goals behind them.Source: §4.2 · p. 8
• Fine-tuning on 13 hand-designed scenarios restored backward-goal success to 100% within 15 minutes.The fine-tuned agents reliably reached goals behind them, including by executing U-turns.Source: §4.2 · p. 8, Figure 5 · p. 8
• The method achieved a 0.629 WOSAC realism meta-score, above linear extrapolation (0.324) and competitive with VPD-BP (0.63) and GigaFlow (0.6190).The evaluation covered 100 held-out scenes and conditioned on trajectory endpoints rather than the first second of logs.Source: §4.2 · pp. 8–9, Table 2 · p. 9

5 Discussion and conclusions

• Self-play achieves low-collision performance on unseen WOMD scenarios with no train-test gap, reducing collision and off-road rates by approximately 4× versus supervised baselines.The authors report crashes on the order of once every 30 minutes, still below human capabilities.Source: §5 · p. 9
• Rare or out-of-distribution cases, including goals behind agents and roundabouts, remain challenging and produce significant performance drops.The authors identify uncommon situations as a key limitation.Source: §5 · p. 9
• Fine-tuning a pre-trained model for a few minutes on hand-designed cases enables near-perfect goal-reaching rates on previously difficult tasks.The agents are open-sourced for use in other simulators or autonomous-vehicle test cases.Source: §5 · p. 9
• The benchmark uses 9-second WOMD scenarios, excludes pedestrians, cyclists, and traffic lights, and prioritizes specified criteria over realism; its WOSAC meta-score is 0.63.The authors identify balancing reliability with realistic human-like behavior as future work.Source: §5 · p. 9

A Observation features and design choices

• Each agent observation combines ego state, visible road-graph information, and partner-agent observations, with a maximum of 64 agents per scenario.The observation is a flattened vector of about 3,000 elements; the appendix tables list ego-state, road-graph, and partner-agent features and dimensions.Source: A Observation features and design… · p. 14, Table 3 · p. 14, Table 4 · p. 14

B.1 Sample rollouts

• Both figures show example rollouts generated with the best-trained policy.The policy-controlled agents are blue, while static agents are gray.Source: Figure 6 · p. 15, Figure 7 · p. 15
• In Figure 6, policy-controlled agents are blue and static agents are gray.The caption identifies the scenes as example rollouts with the best-trained policy.Source: Figure 6 · p. 15
• Figure 7 uses blue for policy-controlled agents and gray for static agents.The caption identifies the scenes as example rollouts with the best-trained policy.Source: Figure 7 · p. 15

C Distribution of errors and remaining failure modes

• 8.95% of the test dataset is not perfectly solved; most unsolved scenes have only a small error rate, and collision and off-road fractions are not significantly correlated (ρ=0.0135, α=0.05).The analysis counts 896 of 10,000 scenarios and finds most contain just a single failure.Source: C Distribution of errors and remaining… · p. 15, Figure 9 · p. 17
• Rare road structures, including roundabouts, are associated with high off-road rates; tight corners, narrow lane entries, and parking lots are also difficult to navigate.Larger vehicles especially struggle with these layouts, and crowding can contribute to vehicles going off-road.Source: C Distribution of errors and remaining… · pp. 15–16
• High collision rates occur in intersections, fast highways, and crowded scenes requiring agents to coordinate, such as yielding or making room to overtake.Crowding and rare layouts can compound scene difficulty and increase collision and off-road rates.Source: C Distribution of errors and remaining… · p. 16
• With a 91-step horizon, distant goals can force agents to squeeze past other vehicles and narrow lanes, increasing collision and off-road rates.This difficulty can compound with challenging layouts and interactions.Source: C Distribution of errors and remaining… · p. 16

D.1 Collision behavior

• GPUDrive supports ignore, remove, and stop collision behaviors, each with different effects on learned behavior.The appendix outlines considerations for future experiments using these alternatives.Source: §D.1 · p. 16

C Distribution of errors and remaining failure modes

• Figure 9 plots collision, off-road, and other failure distributions for scenes that are not fully solved; nearly all contain a single failure.The other category covers agents that neither failed nor reached their goal.Source: Figure 9 · p. 17

D.1 Collision behavior

• Ignoring collisions allows an agent to continue after contact, but negative signals can make its value function pessimistic and lead it to freeze instead of pursuing the goal.The authors suggest sufficient exploration through a large enough entropy coefficient to provide positive signals alongside negative ones early in training.Source: §D.1 · p. 17
• Removing agents at collisions makes the penalty implicit, avoiding an overly pessimistic value function; accidental goal-reaching can reinforce successful action sequences.The passage describes termination for collisions without assigning negative rewards.Source: §D.1 · p. 17

E.1 Detecting out-of-distribution events

• 0.73% of about 4.2 million controllable agents made a U-turn and 1.13% drove in reverse across approximately 129,000 training scenes.Most reverse-driving cases involved pulling out of park with a goal immediately behind the agent; complex U-turn goals were notably scarce.Source: §E.1 · p. 17

E.2 Placing goals behind agents

• The authors hand-designed 13 test scenes with 27 agents and placed each controlled agent's goal behind it, selecting scenes where the resulting goals remained valid and reachable.Table 7 compares aggregate performance for altered and original goal positions; its caption specifies mean ± standard deviation.Source: §E.1 · p. 17, Table 7 · p. 18

E.1 Detecting out-of-distribution events

• U-turns are detected when an agent's heading differs from its initial heading by more than 150° at a valid timestep.This is the paper's stated event criterion.Source: §E.1 · p. 17
• Reverse driving is identified by a velocity-direction and heading difference above 150°, sustained for more than 10 steps at speeds above 0.5 km/hr.These thresholds filter brief reversals and low-speed movement.Source: §E.1 · pp. 17–18

• Table 8 reports the hyperparameters used for the experiments.Source: §F.1 · p. 18

• Experiments ran on either a single NVIDIA A100 or RTX4080 for 12–36 hours each.Source: G Compute resources · p. 18
• All runs, including tuning and experimentation, took approximately 5 GPU days combined.Source: G Compute resources · p. 18
• Table 8 is titled “PPO Algorithm Hyperparameters.”Source: Table 8 · p. 19