The Waymo Open Sim Agents Challenge

1 Introduction

• Simulation supports fast, lower-risk evaluation of autonomous-driving behavior, including safety validation with Monte Carlo trials involving realistic traffic participants.WOSAC focuses on agents’ mid-level object representations, such as trajectories, rather than sensor simulation to reduce computational burden and complexity.Source: §1 · p. 1
• Under arbitrary AV behavior, replayed agents can drift from realistic behavior and appear too aggressive, while heuristic agents such as IDM can be overly accommodating and reactive.WOSAC seeks agents that match the full distribution of human behavior.Source: §1 · pp. 1–2
• WOSAC addresses the lack of an existing simulation-agent benchmark with a closed-loop evaluation and realism metrics suited to long-term futures, building on WOMD.The benchmark models only a subset of perception outputs; traffic-light states and vehicle attributes are left for future work.Source: §1 · p. 2
• The challenge provides an approximate negative log-likelihood evaluation framework for autoregressive traffic agents and an online leaderboard for submissions.The paper also reports empirical evaluations of baseline methods and external submissions.Source: §1 · p. 2

2 Related Work

• CARLA, SUMO, and Flow use heuristic sim-agent policies, while Nocturne uses a simplified world of a fixed roadgraph and moving agent boxes.These simulators have nevertheless enabled progress in autonomous-vehicle motion planning.Source: §2 · p. 2
• Simulation relates to trajectory forecasting and planning, but differs from both in objectives, outputs, and constraints.Open-loop forecasting has limited ability to recover from out-of-domain predictions in closed-loop simulation, and few forecasting methods produce consistent scene-level joint samples.Source: §2 · p. 3
• Learned sim agents vary in coordination, dynamics, observability, inputs, and training; TrafficSim is identified as the only closed-loop learned method using a joint scene-centric actor policy.Other approaches include decentralized agent-centric inference, different dynamics models and input formats, goal-conditioned formulations, and imitation, adversarial, or reinforcement-learning objectives.Source: §2 · p. 3
• Distribution-based evaluation includes FID and simulation-specific distances, but dataset-level aggregate statistics can be insensitive and IS is unsuitable for multimodal simulation.Prior work also uses KDE-based average negative log likelihood of ground-truth trajectories, the metric family WOSAC builds on; some likelihood-based models can produce likelihoods directly.Source: §2 · pp. 3–4
• Existing multi-agent simulation evaluation lacks consensus, and collision rate alone cannot measure realism because static policies can drive it to zero.Other approaches compare logged data using dynamics distributions, offroad rates, coverage, diversity, or progress to goal; goals are difficult to extract reliably, while direct trajectory reconstruction is limited by multiple realistic actions.Source: §2 · pp. 3–4

3 Traffic Simulation as Conditional Generative Modeling

• Driving is modeled as a hidden Markov process in which observations depend on unobservable world states and include the poses of non-AV agents.The model conditions on map, traffic signals, and an observation history of length H.Source: §3 · p. 4
• The world model must generate futures autoregressively for T steps and factorize generation into an AV policy and environment dynamics.The AV policy is defined over next observations, rather than current actions.Source: §3 · p. 4
• The required world-model distribution is evaluated by matching its generated outcomes to logged outcomes, and may be multimodal when agent behavior is uncertain.The AV policy and environment dynamics are left structurally flexible; a multi-agent option factorizes environment dynamics into a sequence of per-agent observation-space policies.Source: §3 · p. 5
• Valid simulation repeatedly applies the AV policy and environment dynamics to updated observations, whereas the illustrated open-loop procedure generates future AV behavior before environment rollouts.The required factorization is represented as autoregressively interleaved distributions over T steps.Source: §3 · p. 5, Figure 2 · p. 5

4.1 Dataset

• WOMD v1.2.0 provides 9-second, 10 Hz scenarios with 1.1 seconds of history and 8 seconds of simulated futures, covering vehicles, cyclists, and pedestrians present at handover.Each scenario requires simulating up to 128 agents; object dimensions are fixed to their last historical measurements.Source: §4.1 · pp. 5–6

4.2 Evaluation

• Approximate NLL measures how well the agent-induced distribution accounts for logged real-world scenarios, using histograms fitted to 32 submitted future samples.The evaluation replaces full-scenario NLL with a composite of lower-dimensional component metrics because the scenario is high-dimensional and submissions need only provide samples.Source: §4.2 · p. 6

4.2.1 Component Metrics

• Component NLLs cover nine kinematic, object-interaction, and map-based measurements, averaged over time independently to allow behavior at different intervals.Figure 3 visualizes baseline simulations on two validation scenes, separating input history from full-duration trajectories and color-coding AV and environment agents.Source: §4.2.1 · pp. 6–7, Figure 3 · p. 7

4.2.2 Composite Metric

• The composite metric is a manually weighted average of nine component metrics, with collision and road-departure NLL weights set twice as high as the others.The component and composite metrics are parameterized by 32 samples.Source: §4.2.2 · p. 7

5.1 Baselines

• Baselines range from random and constant-velocity policies to autoregressive Wayformer variants; external entries included closed-loop MVTA/MVTE and hybrid MTR+++.MVTE samples from a pool of MVTA variants, while MTR+++ searches for a dense subgraph of non-colliding future trajectories.Source: §5.1 · p. 8, §5.2 · p. 8

5.3 Learnings from the 2023 Challenge

• MVTA/MVTE was the only submission using closed-loop training; submissions also showed a benefit from adding planning techniques to motion-prediction frameworks.Open-loop-trained methods such as MTR+++ and the Wayformer-derived baseline needed slower replanning rates to obtain high composite scores; all submissions used agent-centric coordinates.Source: §5.3 · pp. 8–9
• 11% better: 32 diverse Wayformer rollouts per scene outperformed 32 identical rollouts on the evaluation metrics.The authors report that their likelihood-based metrics reward models producing diverse futures.Source: §5.3 · p. 9
• Collision-minimization components can trim the tail of realistic driving behavior, and post-hoc pruning can remove AV-caused collisions, making AV performance appear misleadingly optimistic.The authors note that WOMD contains few examples from the tail of real-driving distributions.Source: §5.3 · p. 9
• Test-set rankings by ADE and minADE disagree with the NLL composite ranking; lower minADE tends to accompany higher composite scores, unlike ADE.Source: §5.3 · p. 9
• Learned stochastic sim agents outperform heuristic and learned deterministic agents in the composite ranking.Figure 4 presents composite metric results for baselines on the WOMD test split.Source: Figure 4 · p. 8, §5.3 · p. 9
• 0.253 vs. 0.222: CAD outperformed MVTE on linear acceleration likelihood, while MVTE scored highest on all other component metrics except logged oracle.MVTE’s angular acceleration likelihood was 0.481 vs. logged oracle’s 0.489; its collision likelihood (0.893 vs. 1.000) and distance-to-nearest-object likelihood (0.383 vs. 0.485) leave room for improvement in interactive metrics.Source: §5.3 · pp. 9–10, Table 3 · p. 9

5.4 Qualitative Results

• Intersection behavior exceeds simple heuristics’ predictive capability in the qualitative examples, where constant-velocity baselines show collisions.The paper presents two WOMD scenarios in Figure 3.Source: §5.4 · p. 10

6 Discussion

• Submission validity required manual verification because factorization and closed-loop rules were subtle.Several open-loop or hybrid submissions may have limited applicability for some simulation applications.Source: §6 · p. 10
• Even Docker submissions would be difficult to enforce automatically; strict stateless input/output APIs would reduce ambiguity about closed-loop behavior.User-maintained arbitrary state prevents the system from enforcing every detail of a closed-loop function call.Source: §6 · p. 10
• Evaluating submissions with held-out AV planning policies and new scenarios would allow further valuable analysis.The benchmark allowed users to control the AV through an independent policy.Source: §6 · p. 10
• The first challenge excluded object insertion and deletion to reduce complexity, with future realism metrics planned for those behaviors.The WOMD data distribution already includes such insertions and deletions, while planners must exercise caution around occlusions where objects may emerge.Source: §6 · p. 10
• The first WOSAC iteration assumed time-invariant object dimensions, although dimensions change in WOMD, and future iterations may evaluate their prediction.The paper treats time-varying dimensions as a vehicle attribute.Source: §6 · p. 10
• One proposed approach to better represent rare collisions is to add synthetic challenging initial conditions or mine close calls and collisions from real driving data.The discussion raises these options after noting collision-minimization components among submissions.Source: §6 · p. 10

A Appendix

• The appendix adds replanning-rate ablations, competition learnings, submission-method descriptions, qualitative results, metric implementation details, dataset splits, and leaderboard instructions.It covers methods from external challenge submissions and all nine component metrics.Source: A Appendix · p. 16
• The appendix includes further analysis of the 2023 CVPR competition and additional descriptions and qualitative results for external submissions.It also provides implementation details for each of the nine component metrics.Source: A Appendix · p. 16
• V1 scores reflect a December 2023 improvement to collision and offroad likelihood calculations, unlike the earlier V0 scores presented at the CVPR 2023 workshop.The update improved most collision-likelihood, offroad-likelihood, and composite-metric scores; both leaderboards remain available online.Source: A Appendix · p. 16

A.1 Replanning Rate Ablation Results

• Figure 5 adds data points spanning replanning intervals from 100 ms to 1100 ms for the same identical-sample agent.Source: §A.1 · p. 16
• 41.2% relative performance drop (composite scores of 0.575 vs. 0.338) occurs for the identical-sample agent when replanning rises from 2 Hz to 10 Hz.The appendix reports that more frequent replanning negatively impacts Wayformer-based agents whether they sample diverse rollouts or produce 32 identical rollouts.Source: Figure 5 · p. 16, §A.1 · p. 16
• Performance degrades smoothly as the identical-sample Wayformer baseline replans more frequently, from 1 Hz through 2 Hz to 10 Hz.The ablation evaluates composite metric performance on the WOMD test set.Source: Figure 5 · p. 16, §A.1 · p. 16

A.2 Additional Learnings from the 2023 CVPR Competition

• Most challenge participants used GMMs; no 2023 CVPR submission used normalizing flows, GANs, VAEs, or denoising diffusion models.The authors expect model-family participation may change as more entrants join.Source: §A.2 · p. 17
• The top four methods, run fully closed-loop or in hybrid fashion, outperformed the two open-loop submissions, especially on collision likelihood.The top methods were MVTE, MVTA, MTR+++, and Wayformer; the open-loop submissions were JointMultiPath++ and CAD.Source: §A.2 · p. 17
• Constant velocity achieved angular speed and angular acceleration likelihoods of 0.02 and 0.04, versus MVTE's 0.54 and 0.38, respectively.The paper notes that the constant-velocity model does not account for yaw rate.Source: §A.2 · p. 17

• The section compares WOSAC with other benchmarks used to evaluate behavior models for autonomous driving.Source: §A.3 · p. 17
• The supplied text states the comparison's scope but does not report individual benchmark characteristics or results.Source: §A.3 · p. 17, Table 4 · p. 17
• Table 4 is identified as a comparison of existing benchmarks for evaluating autonomous-driving behavior models.Source: Table 4 · p. 17

A.4 Additional Information about Methods from External Challenge Submissions

• MVTA is a closed-loop, autoregressive method using a receding-horizon policy, while MVTE randomly selects among three MVTA variants to generate 32 simulations.MVTA predicts one second of motion but uses only the next 100 ms; MVTE is intended to increase rollout diversity.Source: §A.4 · p. 17
• MTR+++ replans every 0.5 seconds over a 2-second horizon and uses a dense-subgraph heuristic to select non-colliding futures from predicted trajectories.It does not meet WOSAC's 10 Hz closed-loop or world-versus-AV policy-factorization requirements.Source: §A.4 · pp. 17–18
• CAD resamples the entire open-loop future when predicted agent collisions occur, using separate MTR checkpoints for world and AV agents.Non-evaluated agents use a constant-velocity model, and CAD does not meet WOSAC's closed-loop requirement.Source: §A.4 · p. 18
• Joint-Multipath++ generates 32 scene-centric, length-80 rollouts in one pass but does not separate AV and world policies as WOSAC requires.Its decoder transforms per-agent embeddings into n × 32 × 80 × 3 rollout tensors.Source: §A.4 · p. 18
• SBTA-ADIA uses a two-stage approach: a GNN scene encoder predicts multi-agent goals, followed by a closed-loop planning policy that tries to achieve them.Source: §A.4 · p. 18

A.5 Additional Qualitative Results from External Challenge Submissions

• Figure 6 shows an MVTE collision and an MTR+++ near-miss, while Figure 7 visualizes four possible futures for one test scene.The figures show scene histories and simulated rollouts; their captions identify the AV and environment-agent trajectory colors.Source: Figure 6 · p. 18, §A.5 · p. 18, Figure 7 · p. 19

A.6 Component Metrics Implementation Details

• The nine component metrics cover kinematics, object interactions, and map-based behavior, including speeds, accelerations, collisions, time-to-collision, and road departures.Angular speed uses the minimal angular difference between headings; collisions are counted when signed distance to the nearest object becomes negative.Source: §A.6 · p. 20

A.6.1 Evaluation Source Code References

• The evaluation code references group the metrics into kinematic, interaction-based, and map-based features, and provide a separate implementation for time-series NLL computation.Source: §A.6.1 · p. 20

A.6.2 Evaluation Code License and Dependencies

• WOMD is non-commercially licensed, while WOSAC evaluation code uses a BSD+limited patent license and lists its software dependencies.Listed dependencies include NumPy, TensorFlow, TensorFlow Probability, Matplotlib, and Protocol Buffers.Source: §A.6.2 · pp. 20–21

A.7 Additional Information about WOMD Splits Used

• Evaluation excludes 401 run segments affected by object-count discrepancies and nine test segments missing maps; Table 5 is identified as reporting WOMD split statistics.Source: §A.7 · p. 21, Table 5 · p. 21

A.8 Submission Format

• Each scenario submission must contain 32 eight-second rollouts, and validation or test submissions may be uploaded to the evaluation server.Submissions are serialized protocol-buffer data and uploaded as one compressed .tar.gz archive.Source: §A.8 · p. 21