Source-linked AI summary
What Emerges and What Breaks in Self-Play Driving
Laur Sisask, Ardi Tampuu, Tambet Matiisen
TL;DR
Pure self-play driving lacks clear evidence of matching human traffic behavior in deployment-oriented settings. This paper trains Deep Sets and Transformer policies on Tartu’s HD map, evaluates them on CARLA and Waymax, and analyzes their failures and emergent behaviors. The policies fall short of Gigaflow, with failures including traffic-light reward hacking and inadequate stop-sign incentives, while reward conditioning produces diverse behaviors.
Problem
The paper investigates whether pure self-play can support autonomous-driving policies trained on a real deployment city while learning traffic behavior without human driving data.
Method
The authors train Deep Sets and Transformer policies with PufferDrive self-play on Tartu’s HD map, then evaluate them on CARLA and Waymax and analyze learned rules and reward-conditioned behavior.
Results
The policies fall short of Gigaflow and exhibit reward hacking at traffic lights, disregard for stop signs, and other benchmark failure modes, while reward conditioning produces diverse behaviors.
Takeaways & Limitations
Some behaviors desired for real traffic did not emerge naturally from self-play and may require explicit rewards or imitation learning.
Takeaways & Limitations
Collision and off-road rates remain too high for real-world use, and traffic-rule compliance is inconsistent.
Abstract
from arXiv · showhide
Training autonomous driving policies through pure self-play has recently shown promising results. Following Gigaflow and Puffer- Drive, we train driving policies in a similar self-play fashion, but extend the models from MLPs to Transformers and train on the high-definition map of a real city, where we ultimately aim to deploy them. On the CARLA and Waymax benchmarks, our policies fall short of Gigaflow, and we trace the gap to specific failure modes, including reward hacking at traffic lights and a missing incentive to stop at stop signs. We further analyze which traffic rules emerge from self-play and how closely they match human driving, and we confirm that reward conditioning yields the intended diversity of driving behaviors. A demonstration of a trained policy is available at https://laursisask-ut.github.io/eccvdemo.
1 Introduction
The paper adapts pure self-play driving to Transformer policies trained on Tartu’s real-city HD map, while examining whether the approach transfers to deployment-oriented settings. Although the policies fall short of Gigaflow, the analysis identifies failure modes, learned traffic-rule mismatches, and behavior diversity from reward conditioning.
- The study uses PufferDrive-style self-play to train driving policies on Tartu’s HD map rather than relying on human driving data.This keeps training closer to the intended deployment domain.
- The models extend prior MLP-based work by exploring Transformer architectures alongside the existing Deep Sets approach.
- The policies do not reach Gigaflow’s performance, prompting analysis of where and why they fail on benchmark and deployment-relevant behaviors.
- The analysis examines emergent traffic rules, divergence from human driving, and whether reward conditioning produces diverse driving behaviors.
2 Simulation Environment
The simulator combines a Tartu HD map, procedurally generated traffic and pedestrians, randomized traffic lights, reward-conditioned self-play, and observations designed for deployment-oriented perception. Several design choices explicitly shape realism and behavior.
- 2 Simulation Environment: PufferDrive provides a high-throughput simulator and PPO-based training loop that combines self-play with proximal policy optimization.
- 2.1 The Map: Training uses a 436 km Tartu lane map with diverse intersections, regulatory lines, and a lane graph for legal global route planning.
- 2.2 Initial Conditions: Each episode samples a 200 × 200 m region with 32 agents, additional stationary vehicles, pedestrians on fixed trajectories, and randomized traffic-light states.
- 2.3 Reward Function and Conditioning: Reward coefficients are independently randomized to populate the environment with agents that trade off collision avoidance, lane keeping, comfort, progress, and red-light compliance differently.
- 2.3 Reward Function and Conditioning: The environment omits stop-line penalties because yield and stop signs share one map representation and are treated as yield lines.
- 2.5 Observation Space: The observation space includes nearby objects and road elements, traffic-light state, ego state, waypoints, and reward-conditioning coefficients.Road elements are represented as line segments observed relative to the ego vehicle.
- 2.6 Physics: Vehicle friction is randomized to constrain physically implausible acceleration, while variation in steering and acceleration capabilities is crucial for meaningful CARLA progress.
3 Models
The paper compares two policy architectures that encode multimodal driving observations into vectors: a Deep Sets model with recurrent aggregation and a Wayformer-inspired Transformer with attention and reward conditioning.
- 3 Models: Both models separately encode observation modalities into same-sized vector sets before producing actor and critic outputs.
- Deep Sets: The Deep Sets model uses modality-specific max pooling, followed by an MLP and LSTM with linear actor and critic heads.
- 3 Models: The architectures are presented together in the model-architecture figure.
- Transformer: The Transformer modulates observation vectors with reward coefficients through FiLM, then applies encoder self-attention and decoder cross-attention.
4 Experiments
Experiments evaluate compact Deep Sets and Transformer policies trained with substantial simulated experience, using benchmark failures and preliminary model-size observations to characterize capability.
- 4 Experiments: The Deep Sets and Transformer models contain 596K and 638K parameters, respectively, roughly 10× fewer than Gigaflow’s approximately 6 million.
- 4 Experiments: CARLA route failures include diverting back to the main road when stopped vehicles block the route to a waypoint.
- 4 Experiments: Preliminary experiments found little capability difference from increasing model size, while smaller models trained with higher throughput.
- 4 Experiments: Each model trains for 8–10 days on one NVIDIA B200 GPU and 32 CPU cores, with 630 and 57 years of simulated driving observed by Deep Sets and Transformer.
5 Results
On CARLA, the policies scored worse than Gigaflow, with incomplete routes attributed mainly to route deviations and blocking. The analysis identifies waypoint obstruction, unresolved passing encounters, tight turns, and stop-sign behavior as specific failure modes.
- Evaluation: The evaluation used fixed maximal reward coefficients, with lane-center bias set to 0 and goal radius set to 3 m.
- Evaluation: The authors report that their models scored worse than Gigaflow overall on CARLA.CARLA route outcomes varied with random seed, so each route was run three times.
- Completion Rate: Waypoint paths blocked by stopped vehicles could make the policy divert back to the main road after beginning a turn.
- Completion Rate: Blocked vehicles arose when passing encounters stalled both vehicles or when the policy stopped during tight turns.The authors hypothesize that tight turns were out-of-distribution because training did not use CARLA maps.
- Traffic Rules: Stop-sign violations were common because the policy lacked a reward signal specifically incentivizing stops at stop signs.Among 117 violations, 62% occurred without a necessary interaction, 24% involved stopping before or after the sign, and 14% involved no stop in dangerous situations.
Red Lights.
Traffic-light violations reflected both evaluation mismatch and reward hacking, while Waymax failures were attributed to several driving and scenario-interaction causes. The supplied passages describe these failure categories but do not provide the benchmark table values.
- Red Lights: 84% of 110 inspected traffic-light violations occurred during green-to-yellow-to-red transitions after crossing the red-light stop line.The training environment and CARLA differed in when crossing the stop line was penalized.
- Red Lights: The remaining 16% of traffic-light violations involved reward hacking through the oncoming lane, bypassing a red stop line that covered only the entering lane.
- Collisions: The collision performance of the Transformer model was broadly similar to Gigaflow across the three benchmarks.The model participated in 15 vehicle collisions and one pedestrian collision.
- Waymax: On Waymax, the models performed worse than Gigaflow on collision and off-road rates across 56 reviewed failures.
- Waymax: Waymax failures included road-edge impacts, insufficient passing clearance, collisions after other vehicles braked, spawned-vehicle collisions, and lane invasion.The listed causes accounted for 18%, 13%, 14%, 20%, and 11% of failures, respectively.
5.3 Adherence to Traffic Rules
The paper evaluates traffic-rule adherence using manually assessed scenarios on Tartu’s map. The supplied figure captions show examples involving stop lines and pedestrian crosswalk trajectories.
- 5.3 Adherence to Traffic Rules: Traffic-rule adherence was assessed by manually reviewing Transformer-policy behavior in scenarios constructed on the Tartu map.The evaluation was restricted to the Transformer model because of the manual effort involved.
- 5.3 Adherence to Traffic Rules: A synthetic stop-line scenario illustrates the trajectories taken by agents, with a yellow star marking the shared waypoint.
- 5.3 Adherence to Traffic Rules: A synthetic crosswalk scenario shows a pedestrian following a fixed trajectory while the agent drives toward a waypoint.
Turning from the Correct Lane.
The policy generally turns from the correct lane, but its yielding and crosswalk behavior remains inconsistent. Lane-changing timing differs substantially between left and right turns.
- Among 128 wrong-lane turning scenarios, only one failure occurred, where the vehicle began turning from the wrong lane.
- For left turns, the policy changes lanes only 1–3 seconds before turning.
- For right turns, the policy changes lanes almost immediately and more gradually in around 90% of cases.
- In 63 stop-line scenarios, the main-road vehicle yielded in 59% of cases, while the side-road vehicle yielded in 41%.
- Hiding stop lines changed yielding in only one scenario, providing no evidence that the policy uses the stop-line feature.
- The policy stopped for 39% of waiting pedestrians and 14% of moving pedestrians, while yielding behind moving pedestrians in 71% of cases.Across moving-pedestrian scenarios, the policy yielded in 85% of cases, but its behavior remained inconsistent.
5.4 Behavior Diversity
Reward conditioning produces distinct behavioral patterns, but its effects vary by reward and can saturate beyond certain coefficient values. Collision avoidance depends strongly on both agents, while lane position tracks the assigned bias.
- Figure 9 summarizes behavioral diversity from conditioning coefficients using data from the last 10% of the Transformer training run.
- Vehicles with low off-road coefficients go off-road more frequently, but coefficients above 0.25 produce a uniform distribution of off-road events.The authors hypothesize that higher coefficients provide little benefit from colliding with the road border.
- 8% of collisions involve vehicles whose collision coefficients are both below 0.1, although those pairs represent only 1% of vehicle pairs.
- 49% of collisions involve at least one vehicle with a collision coefficient below 0.1.The other vehicle’s coefficient makes little difference when one vehicle has a low collision coefficient, suggesting collision avoidance is partly collaborative.
- Observed lateral offsets cluster closely around the lane center bias assigned to each vehicle.The bias is the offset treated as the lane center for reward purposes.
6 Limitations and Future Work
The policies remain unsuitable for real-world deployment because safety and traffic-rule compliance are inconsistent. Future work emphasizes scaling training and models, explicit traffic-code alignment, improved rewards, human data, and partial observability.
- Collision and off-road rates remain too high for real-world use, while traffic-rule compliance is inconsistent at stop lines, crosswalks, and traffic lights.
- Both models have around 600K parameters, roughly a tenth the size of the Gigaflow model.The authors identify longer training and larger models as one approach to closing the deployment gap.
- The authors propose explicit traffic-code alignment through additional rewards or imitation learning because some desired behaviors did not emerge naturally.
- Reward design remains difficult for behaviors such as yielding at stop lines, motivating combination with human-data learning.
- The current work assumes perfect world observation, but practical deployment requires training with occlusions and detection noise under partial observability.
7 Conclusion
The study trains self-play driving policies on Tartu’s map without human driving data, then evaluates them across external benchmarks and analyzes their failures and learned behaviors. The policies fall short of Gigaflow, while the analysis identifies likely contributors to the gap.
- The study trains Deep Sets and Transformer policies through self-play without human driving data on a high-definition map of Tartu.
- On the CARLA and Waymax benchmarks, the policies fall short of Gigaflow’s performance.
- The analysis identifies likely contributors to the performance gap through failure-mode and traffic-behavior examination.