Source-linked AI summary
SHRED: Self-Play for Heterogeneous Road-Users without Expert Demonstrations
Wei-Jer Chang, Zhouchonghao Wu, Akshay Rangesh, Weixin Li, Masayoshi Tomizuka, Wei Zhan
TL;DR
Realistic simulation and autonomous planning require modeling interactions among road users with different dynamics and behaviors, while existing approaches rely heavily on human driving demonstrations. SHRED trains vehicles, pedestrians, and cyclists through heterogeneous self-play without demonstrations or logged-trajectory replay. It reports strong simulation and planning performance, zero-shot map transfer, and gains from heterogeneous self-play.
Problem
Realistic traffic simulation and planning depend on modeling heterogeneous road-user interactions, but existing approaches predominantly learn interaction behavior from human driving demonstrations.
Source: §1 · p. 3Method
SHRED jointly trains vehicles, pedestrians, and cyclists from random initialization using type-specific dynamics and objectives, procedural scenarios, and per-agent randomization.
Source: §1 · pp. 3–4Results
SHRED achieves strong realism on WOSAC and outperforms all compared planners on InterPlan, while nuPlan-map-trained policies retain comparable realism in zero-shot Waymo evaluation.
Source: §1 · p. 4, §5 · p. 11Takeaways & Limitations
The results suggest heterogeneous procedural self-play is a scalable, demonstration-free paradigm for learning interactive traffic behavior across simulation and planning.
Source: §5 · p. 12SHRED inherits its simulator and reward-design fidelity ceiling, and WOSAC agreement with logged behavior does not establish behavioral validity or real-world transfer.
Source: §5 · p. 12
Abstract
from arXiv · showhide
Realistic traffic simulation and autonomous planning both depend on modeling interactions among heterogeneous road users whose dynamics, constraints, and behaviors differ fundamentally. Existing approaches predominantly learn realistic interaction behavior from large-scale human driving demonstrations. In this work, we ask whether such behavior can instead emerge purely through self-play. We present SHRED, a framework that jointly trains vehicles, pedestrians, and cyclists through large-scale heterogeneous self-play from random initialization, without demonstrations or logged-trajectory replay. SHRED combines type-specific dynamics and objectives with procedural scenario generation and per-agent randomization, allowing diverse behaviors to emerge through closed-loop interaction. On the Waymo Open Sim Agents Challenge (WOSAC), SHRED jointly controls all three road-user types and outperforms both the closest demonstration-free self-play baseline and prior self-play methods that use demonstration-trained reference policies. Policies trained only on nuPlan map geometry retain comparable realism under zero-shot evaluation on Waymo. For planning, SHRED outperforms all compared planners on the interaction-heavy InterPlan benchmark. Controlled ablations show that heterogeneity in self-play improves both traffic realism and planning performance, with particularly large planning gains in interaction-heavy scenarios. Together, these results establish heterogeneous procedural self-play as a scalable, demonstration-free paradigm for learning realistic and interactive traffic behavior.
1 Introduction
SHRED addresses the reliance on human driving demonstrations and the limited treatment of heterogeneous road users in self-play by jointly training vehicles, pedestrians, and cyclists from random initialization. It reports strong simulation and planning results, zero-shot map transfer, and diverse rollout alternatives.
1 Introduction
- Human-data-driven methods remain widely dependent on logged interaction distributions, while prior self-play largely uses vehicle-only or homogeneous policies and rewards.The paper identifies scalable, fully reactive self-play across vehicles, pedestrians, and cyclists without logged data or pretrained imitation anchors as largely unexplored.Source: §1 · p. 3
- SHRED jointly trains vehicles, pedestrians, and cyclists from random initialization using type-specific dynamics, objectives, and action spaces.Procedural scenarios and per-agent randomization expose agents to diverse configurations without demonstration supervision.Source: §1 · pp. 3–4
- Each Fig. 1 panel shows 32 closed-loop rollouts, with road-user types color-coded and the logged trajectory overlaid, illustrating alternative maneuvers.The figure spans scenarios from urban intersections to parking lots.Source: Figure 1 · p. 4, §1 · p. 4
- SHRED achieves strong sim-agent realism on WOSAC, the highest score among compared planners on the official 80-scenario InterPlan split, and zero-shot map transfer from nuPlan to Waymo.The introduction also reports that heterogeneous self-play improves realism and planning, with the largest planning gains in interaction-heavy scenarios.Source: §1 · pp. 3–4
2 Related Work
Prior work models traffic through demonstrations or self-play, but typically relies on trajectory-model priors, vehicle-centric control, or shared dynamics across road-user types. SHRED instead studies procedural, heterogeneous self-play and organizes policies by physical type.
2.1 Data-Driven and Heterogeneous Traffic Simulation
- Data-driven simulation and planning methods use human driving data, while heterogeneous interaction behavior is typically learned implicitly from demonstrations and closed-loop control remains largely vehicle-centric or shares dynamics.SHRED instead studies heterogeneous recurrent self-play without a trajectory-model prior.Source: §2.1 · pp. 4–5
2.2 Self-Play Reinforcement Learning for Driving
- Gigaflow [10] is a demonstration-free self-play precedent across vehicles, pedestrians, and cyclists, but shares jerk-actuated bicycle dynamics and goal generation across agents.Other approaches use logged WOMD initial states [8], behavior-cloned or tokenized policy anchors [9] [5], or a pretrained vision backbone [60]; SHRED uses procedural initialization without these anchors or logged replay.Source: §2.2 · p. 5
2.3 Heterogeneity in Multi-Agent Reinforcement Learning
- Unlike cooperative methods with shared rewards, SHRED’s mixed-motive setting gives each road user its own goal, type-specific rewards, and dynamics.Policies are organized by physical type, sharing perception while using separate recurrent, actor, and value heads for vehicles, pedestrians, and cyclists.Source: §2.3 · p. 5
3 Method
SHRED models vehicles, pedestrians, and cyclists with type-specific dynamics, actions, and rewards, and trains them in procedurally generated heterogeneous self-play. Shared scene encoders feed specialized policies, while per-type normalization and replay address optimization imbalance across agent populations.
3.1 Heterogeneous Agent Modeling
- Vehicles, pedestrians, and cyclists use distinct dynamics and action spaces, with rewards tailored to their different road-use behaviors.Vehicles use jerk-integrated Ackermann dynamics, pedestrians use unicycle dynamics, and cyclists use kinematic bicycle dynamics; vehicle collision penalties double when the other party is a pedestrian.Source: §3.1 · p. 6
3.2 Procedural Heterogeneous Self-Play
- Procedural initialization, type-conditioned goals, and per-agent randomization create diverse closed-loop scenes without replaying logged destinations.Goals are regenerated after completion, extending interaction beyond recorded scenario durations; some agents have goal observations masked during training.Source: §3.2 · pp. 6–7
3.3 Architecture and Algorithm Design Choices
- Shared scene encoders feed separate LSTM cells, actor heads, and value heads for each road-user type.The architecture trains on mixed transitions, with per-type rewards and value normalization to accommodate distinct reward scales.Source: Figure 2 · p. 6, §3.3 · p. 7
- Independent PopArt statistics, per-type advantage standardization, GAE with V-trace, and advantage-weighted prioritized replay address optimization imbalance in mixed heterogeneous rollouts.Segments are sampled in proportion to estimated advantage magnitude to increase high-signal transitions while preserving balanced learning across agent populations.Source: §3.3 · p. 7
- On WOSAC 2023, SHRED’s advantage over Gigaflow depends on type-specific rewards and dynamics; without them, SHRED falls below the baseline.Gigaflow uses a scripted IDM-like pedestrian policy, while SHRED learns all three road-user types; both use map-sampled goals without logged destinations.Source: Table 1 · p. 8
4 Experiments
SHRED is evaluated for heterogeneous traffic simulation, interactive planning, and emergent behavior. It outperforms compared methods on planning and improves traffic realism through heterogeneous modeling, while pedestrian group metrics rise with crowd density without an explicit group objective.
4.1 Heterogeneous Traffic Simulation
- SHRED scores 0.631 versus Gigaflow’s 0.619 in WOSAC 2023 overall realism, and 0.894 versus 0.808 in time-to-collision likelihood.Gigaflow scores higher on distance-to-object likelihood; SHRED learns policies for all three road-user types, unlike Gigaflow’s scripted pedestrian policy.Source: §4.1 · p. 8
- On WOSAC 2024, the full heterogeneous policy scores 0.631 versus 0.604 for the homogeneous variant, with the largest per-type realism drop for vehicles.Homogenizing either dynamics or rewards modestly reduces realism; removing both causes the largest drop, and removing random initialization and goals lowers overall realism by 0.010.Source: §4.1 · pp. 8–9, Table 2 · p. 9
- SHRED achieves competitive realism and greater trajectory diversity without demonstrations or a pretrained imitation-policy anchor, while nuPlan-trained policies retain comparable realism zero-shot on Waymo.The comparison is on WOSAC 2024 against self-play methods HR-PPO and SPACeR.Source: §4.1 · p. 9, Table 3 · p. 9
4.2 Interactive Ego Planning
- 67.50 on InterPlan’s official 80-scenario split and 70.14 on its full 335-scenario evaluation are the highest scores among compared planners.SHRED uses a 6.7M-parameter policy; it scores 67.50 versus 53.00 for HybridLLMPlanner on the official split.Source: Table 4 · p. 10, §4.2 · p. 10
4.3 Behavioral Analysis
- In-group fraction, local alignment, and mean group size rise with pedestrian density despite the absence of an explicit group objective.The stationary reference stays near its floor across densities, indicating that static crowding alone does not produce the measured structure.Source: §4.3 · p. 11, Figure 3 · p. 11
5 Conclusion
SHRED jointly trains vehicles, pedestrians, and cyclists through heterogeneous self-play, and its ablations show that heterogeneity improves realism and planning, especially in interaction-heavy scenarios. The conclusion also identifies fidelity limits tied to simulator and reward design, while the dynamics appendix specifies distinct motion models for each road-user type.
5 Conclusion
- SHRED jointly trains vehicles, pedestrians, and cyclists from random initialization without trajectory supervision or logged-trajectory replay.The conclusion positions heterogeneous self-play as complementary to data-driven approaches for simulation and planning.Source: §5 · pp. 11–12
- Heterogeneous self-play improves both traffic realism and planning, with particularly large planning gains in interaction-heavy scenarios.Policies trained on nuPlan maps retain comparable realism under zero-shot evaluation on Waymo, while imitation-based methods retain an advantage in overall WOSAC distributional realism.Source: §5 · p. 11
- The simulator and reward design bound what self-play can produce; richer right-of-way conventions and traffic rules may not emerge.The paper also notes that WOSAC measures agreement with logged behavior, not behavioral validity, and that real-world driving transfer is unevaluated.Source: §5 · p. 12
A Per-Type Dynamics
- The appendix describes per-type continuous-time dynamics operating on a shared kinematic state at Δt = 0.1 s.It identifies jerk-integrated Ackermann dynamics for vehicles, unicycle dynamics for pedestrians, and kinematic bicycle dynamics for cyclists.Source: A Per-Type Dynamics · p. 16, §A.1 · p. 16, §A.2 · p. 16
A.1 Vehicle: Jerk-Integrated Ackermann Arc
- Vehicle control selects longitudinal and lateral jerk primitives, and the pose advances along a circular arc after steering constraints are applied.The steering angle is subject to a slew-rate limit and hard cap; nearly straight motion reduces to a line segment.Source: §A.1 · p. 16
A.2 Pedestrian: Unicycle
- Pedestrians use a unicycle model with integrated heading and velocity constrained to the heading direction, plus a dedicated stop action.The action grid combines acceleration and heading-rate choices with a STOP action that forces speed to zero.Source: §A.2 · pp. 16–17
A.3 Cyclist: Kinematic Bicycle
- Cyclists use a kinematic bicycle model with acceleration and steering controls, and a logarithmic grid concentrates finer resolution near zero.The cyclist model allows velocity direction to differ from heading by a slip angle during turns.Source: §A.3 · pp. 17–18
A.4 Domain Randomization Parameters
Dynamics coefficients are randomized by agent and episode, with ranges tailored to each road-user type. Random initialization places agents and generates goals from map topology rather than replaying logged futures.
A.4 Domain Randomization Parameters
- Per-episode dynamics coefficients are sampled from type-specific ranges, and all four coefficients are observation features so policies can condition on their kinematic regime.Vehicle ranges are [0.5, 1.5], cyclist ranges are [0.7, 1.3], and pedestrian coefficients are fixed at 1.0.Source: §A.4 · p. 18
- Table 5 covers per-type kinematic parameters and randomization ranges, distinguishing episode-sampled coefficients from fixed scalars and hard clip bounds.Bounding-box ranges are sampled at spawn and calibrated to Waymo Open Motion Dataset distributions; vehicle bodywork includes a truck/car draw.Source: Table 5 · p. 18
B Random Initialization Details
- Random initialization samples vehicles and cyclists from lane segments and pedestrians from crosswalks or road edges, rejecting collisions and off-road placements.Each candidate has at most 100 placement attempts per slot.Source: B Random Initialization Details · pp. 18–19
- Agents receive map-topology-based goals that differ by type: vehicles and cyclists follow lane topology, while pedestrians cross or follow road edges.Agents without a valid goal remain in the scene and receive an observation feature indicating that no active goal exists.Source: B Random Initialization Details · p. 19
C Per-Type Reward Details
SHRED uses weighted, type-specific reward terms for safety, progress, comfort, and behavioral constraints. It also applies PopArt statistics and weight corrections to keep the value function on its real scale as return statistics change.
C Per-Type Reward Details
- Per-type rewards combine safety, progress, comfort, and type-specific constraints, with weights sampled independently from type-specific uniform ranges at episode reset.This is reward-domain randomization.Source: C Per-Type Reward Details · p. 19
- Comfort counts kinematic channels exceeding type-specific thresholds; the thresholds were calibrated to the 95th–97.5th percentiles of Waymo motion magnitudes.Pedestrian and cyclist thresholds are calibrated but unused in the reported run, where their comfort weights are zero.Source: C Per-Type Reward Details · p. 19, Table 7 · p. 20
- Pedestrians receive a per-step penalty for being on the road outside crosswalks, while vehicles and cyclists do not receive this penalty.Source: C Per-Type Reward Details · p. 19
- Vehicles have lane-alignment, lane-centering, velocity-alignment, and reverse penalties; cyclists have softer lane alignment and an edge-preference bonus, while pedestrians and cyclists receive speed-limit penalties.Pedestrian and cyclist speed limits are sampled per agent from type-specific ranges.Source: C Per-Type Reward Details · p. 19
D PopArt Weight Correction
- Per-type PopArt statistics are updated by exponential moving averages over epoch returns, and value-layer weights are corrected so shifts in those statistics leave the real-scale value function unchanged.The correction is intended to avoid policy-gradient discontinuity at the commit step.Source: D PopArt Weight Correction · p. 20
E.1 WOSAC 2024 Validation
The WOSAC evaluation uses procedural training without logged-trajectory replay and compares the GPU evaluator with the official evaluator. Their realism meta-scores agree closely on 50 scenarios.
E.1 WOSAC 2024 Validation
- Training remains procedural and does not use logged-trajectory replay.Source: §E.1 · p. 20
- 3.5 × 10−5 is the maximum reported difference between GPU and official evaluator realism meta-scores across 50 WOSAC scenarios.Source: §E.1 · p. 20
- Realism meta-scores use the official weighted definition, and logged history is prepended when computing kinematic features.Source: §E.1 · p. 20
E.2 WOSAC 2025 Test Evaluation
The section reports a WOSAC 2025 test evaluation and treats its metric separately from the 2023 and 2024 metrics.
E.2 WOSAC 2025 Test Evaluation
- The evaluation covered the WOSAC 2025 test set.Source: §E.2 · p. 21
- The WOSAC 2025 metric differs from the 2023 and 2024 metrics.Source: §E.2 · p. 21
- The authors report the 2025 evaluation separately because its metric differs from the earlier years’ metrics.Source: §E.2 · p. 21
E.3 Full InterPlan Comparison
The full InterPlan comparison includes all baselines and listed variants, and SHRED achieves the highest score among the compared planners on both evaluation sets.
E.3 Full InterPlan Comparison
- The expanded comparison includes all baselines and their listed variants.Source: §E.3 · p. 21
- SHRED achieves the highest score among compared planners on both InterPlan (80) and Full Scale (335).Source: Table 10 · p. 22
- The vehicle-only self-play baseline uses scripted pedestrians and no cyclists.Source: Table 10 · p. 22