PaperlayerView exact revision on arXiv

Source-linked AI summary

Large Scale Interactive Motion Forecasting for Autonomous Driving : The Waymo Open Motion Dataset

Scott Ettinger, Shuyang Cheng, Benjamin Caine, Chenxi Liu, Hang Zhao, Sabeek Pradhan, Yuning Chai, Ben Sapp, Charles Qi, Yin Zhou, Zoey Yang, Aurelien Chouard, Pei Sun, Jiquan Ngiam, Vijay Vasudevan, Alexander McCauley, Jonathon Shlens, Dragomir Anguelov

arXiv:2104.10133v1cs.CVcs.LGcs.RO

TL;DR

Motion forecasting for autonomous-driving planning lacks large, high-quality data focused on interactive multi-agent situations. The paper addresses this gap with the WAYMO OPEN MOTION DATASET, high-quality tracking, joint and individual prediction metrics, and baseline models; its analysis reports substantially stronger interactivity signals in interactive validation data than in standard data.

  • Problem

    Interactive driving situations require joint prediction of multiple objects, while large-scale high-quality data with interaction annotations has been limited.

  • Method

    The paper releases an interaction-mined dataset with explicit interacting-object annotations, offboard automatic labeling, joint-prediction metrics, and marginal and joint baseline models.

  • Results

    Interactive validation contains larger conditional-versus-unconditional KL divergences than standard validation: 73% versus 48% exceed 10, and 45% versus 28% exceed 50.

  • Takeaways & Limitations

    The dataset and evaluation framework provide benchmarks for developing models that predict individual-agent and interactive multi-agent motion.

Abstract

from arXiv · show

As autonomous driving systems mature, motion forecasting has received increasing attention as a critical requirement for planning. Of particular importance are interactive situations such as merges, unprotected turns, etc., where predicting individual object motion is not sufficient. Joint predictions of multiple objects are required for effective route planning. There has been a critical need for high-quality motion data that is rich in both interactions and annotation to develop motion planning models. In this work, we introduce the most diverse interactive motion dataset to our knowledge, and provide specific labels for interacting objects suitable for developing joint prediction models. With over 100,000 scenes, each 20 seconds long at 10 Hz, our new dataset contains more than 570 hours of unique data over 1750 km of roadways. It was collected by mining for interesting interactions between vehicles, pedestrians, and cyclists across six cities within the United States. We use a high-accuracy 3D auto-labeling system to generate high quality 3D bounding boxes for each road agent, and provide corresponding high definition 3D maps for each scene. Furthermore, we introduce a new set of metrics that provides a comprehensive evaluation of both single agent and joint agent interaction motion forecasting models. Finally, we provide strong baseline models for individual-agent prediction and joint-prediction. We hope that this new large-scale interactive motion dataset will provide new opportunities for advancing motion forecasting models.

1. Introduction

The paper motivates joint motion forecasting for interactive driving situations and introduces a large-scale dataset with interaction labels, high-quality tracking, and metrics for both marginal and joint prediction.

  • Motivation: Interactive situations such as merges, lane changes, and unprotected turns require jointly predicting multiple interacting objects.Existing datasets have focused primarily on single-agent representation, leaving large-scale interaction modeling less developed.
  • Dataset goals: The dataset targets diverse interactive behaviors across road geometries and locations, with explicit interaction-group annotations for training and evaluation.These annotations support both multi-agent joint prediction and individual-agent prediction models.
  • Data quality: High-quality offboard automatic labeling provides accurate object tracks without the real-time constraints of onboard perception systems.The resulting tracks are intended to reduce perception noise and focus modeling on object behavior complexity.
  • Evaluation: The paper proposes joint-prediction metrics, including mAP, to evaluate performance across object types, prediction horizons, and trajectory-shape buckets.The metric is designed to capture precision-recall performance in a balanced way across semantic driving behaviors.
  • Contributions: The work releases the WAYMO OPEN MOTION DATASET with specifically labeled interactive behaviors and provides benchmarks for marginal and joint prediction.The dataset combines diverse scenes, multiple cities, rich annotations, and high-quality perception output.

2. Related Work

Prior motion-forecasting datasets differ in scale, taxonomy, horizon, and interaction complexity, while most prediction methods still produce independent per-object futures rather than explicitly joint trajectories.

  • Motion forecasting datasets: Existing datasets vary in scenes, duration, roadway coverage, tracked objects, and time segments, but many were not collected for diverse complex interactions.Lyft Level 5 has the most hours, while NuScenes provides rich object taxonomy; neither was designed to capture broad interactive driving scenarios.
  • Motion forecasting datasets: The INTERACTION dataset increases interaction complexity through manually selected locations and times, but uses a small set of specific driving environments.The paper contrasts this targeted collection strategy with its own interaction-mining methodology.
  • Jointly consistent multi-agent forecasting: Most existing models output independent future distributions per object, encouraged by per-object metrics and datasets requiring prediction for only one agent.Some methods model interactions for better performance, but explicitly jointly consistent futures remain less common.

3. Dataset

The dataset combines mined interactive scenarios with high-quality tracks, maps, and annotations across diverse agent types and trajectories. Its validation and test variants explicitly support both marginal and joint prediction of interacting objects.

  • Dataset construction: The dataset provides 10 Hz object tracks with 3D state attributes and static and dynamic map features for road-environment context.Tracks include 3D center, heading, dimensions, and velocity; some states may be absent because of sensor range or occlusion.
  • Dataset construction: Interaction scenarios are mined using composable semantic predicates and annotate the interacting pair across merges, turns, lane changes, and vehicle-agent encounters.The mining framework queries a large driving corpus using SQL and relational operations.
  • Dataset statistics: 57% of scenes contain at least one pedestrian and 16% contain at least one cyclist, while all scenes contain at least one vehicle.The dataset also includes scenes with many agents: 46% contain more than 32 agents and 11% more than 64.
  • Dataset statistics: 38.3% of validation scenes require predictions for more than one agent type, and 4.9% require all three classes: vehicles, pedestrians, and cyclists.On the interactive validation set, 77.5% of scenes involve two vehicles, 14.9% a vehicle and pedestrian, and 7.6% a vehicle and cyclist.
  • Dataset statistics: The selected agents span diverse trajectory shapes, speeds, and behaviors, addressing the dominance of straight, constant-velocity motion in real-world data.The dataset is designed to cover varied behaviors and trajectory shapes across object types.
  • Data quality: Offboard automatic labeling generates temporally consistent, accurate 3D bounding boxes without the real-time constraints of onboard perception.The system uses detection, multi-object tracking, and object-centric refinement across each track.

4. Metrics

The paper extends trajectory metrics to joint multi-agent predictions and introduces mAP to evaluate accuracy across confidence, behavior, and trajectory-shape variation. It reports marginal metrics on standard splits and joint metrics on interactive splits.

  • Metric formulation: The metric suite includes minADE, minFDE, Miss Rate, overlap rate, and Average Precision for trajectory-set and joint-prediction evaluation.Joint metrics select the best joint configuration rather than independently minimizing each agent’s error.
  • Metric formulation: Joint prediction samples contain K joint trajectories, each with a confidence and trajectories for A agents; single-agent prediction is the special case A = 1.This formulation supports evaluating marginal and joint forecasting within one framework.
  • Mean average precision: mAP averages precision-recall performance over eight trajectory-shape buckets to balance common and infrequent driving behaviors.The buckets include straight, turns, u-turns, and stationary trajectories.
  • Evaluation settings: Table 2 reports marginal validation and test metrics at 8s, distinguishing road-graph, traffic-signal, and high-order-interaction information.The constant-velocity baseline uses K = 1, while other models use K = 6.

5. Experiments

Experiments evaluate baseline forecasting models, feature encodings, joint prediction, dataset interactivity, perception quality, and metrics. Results show that rich context and joint modeling improve forecasting, while interactive scenes exhibit stronger agent influence and mAP better reflects confidence quality than minADE alone.

  • Baseline models and feature encodings: Baseline models use constant velocity or LSTM encoders with agent history, while variants add road graphs, traffic signals, and high-order interactions.The models decode K=6 trajectories using concatenated feature embeddings in agent-centered coordinates.
  • Baseline models and feature encodings: Constant velocity achieves double-digit vehicle minADE, indicating that the dataset contains nontrivial future trajectories.
  • Feature ablations: Vehicle minADE improves from 2.63 to 1.34 and mAP from 0.07 to 0.23 as map, signal, and interaction information are incrementally added.The same improvement trend is reported for pedestrian trajectory prediction.
  • Joint prediction: Joint evaluation reuses marginal models by selecting top joint predictions from K² possibilities, but interactive performance remains low because joint prediction is difficult and lacks mature baselines.The selection uses products of predicted probabilities across the two interactive agents.
  • Interactivity: Interactive validation contains stronger conditional influence: 73% of agent pairs exceed KL divergence 10 and 45% exceed 50.KL divergence compares unconditional target predictions with predictions conditioned on a query agent’s future.
  • Perception and metrics: Offboard perception has accuracy and distance-error distributions similar to human labels, whereas the baseline detector has substantially higher mean distance error.The increased perception noise from the baseline detector implies a higher lower-bound minADE for behavior models.
  • Perception and metrics: MinADE improves with more predictions regardless of confidence, while mAP peaks at 3 or 6 predictions depending on forecast horizon and penalizes confident false positives.mAP peaks at 3 predictions for 3s and 5s, and at 6 predictions for 8s.

6. Discussion

The discussion presents the Waymo Open Motion Dataset as a large-scale resource centered on interactive behaviors, rich annotations, and evaluation of both individual-agent and joint-agent forecasting.

  • The dataset mines interactive behaviors across diverse road geometries and multiple cities, providing rich 3D object states and high-definition map information.Object tracks are generated by a high-fidelity offboard automatic labeling system.
  • The evaluation framework covers both per-agent and joint-agent motion forecasting models.

A. Motion Forecasting Metrics

This section explains distance-error metrics for comparing predicted trajectories with future object tracks. Because futures are stochastic and multimodal, minADE reports the minimum error among a constrained set of trajectory hypotheses.

  • Distance-error metrics measure Euclidean distance between a predicted trajectory and the future object track.
  • MinADE reports the minimum displacement error across a constrained set of weighted trajectory hypotheses.The metric addresses multimodal futures by selecting the lowest error among model outputs.

B. Dataset Splits

The dataset creates mutually exclusive training, validation, and test subsets from 20-second scenarios, then extracts shorter windows for examples.

  • Scenario splits allocate 70% to training, 15% to validation, and 15% to testing using hashes of capture date and vehicle ID.
  • Examples are generated by extracting 9.1-second windows from the original 20-second scenarios.

C. Metrics Details

The metrics define overlaps and misses for joint trajectory predictions using rotated 5-dof bounding boxes, agent-relative thresholds, and speed-dependent scaling.

  • Overlap rate: Overlap rate flags whether predicted agent boxes overlap visible environmental objects at any timestep within the prediction interval.The highest-scoring joint prediction is evaluated, while agents appearing only later are excluded from potential overlaps.
  • Bounding-box construction: Each waypoint is converted into a 5-dof bounding box containing position, width, length, and heading for intersection-over-union comparison.Predicted headings come from the derivative to the previous waypoint, while ground-truth box sizes are reused.
  • Miss rate: Miss rate uses an indicator function with separate longitudinal and lateral thresholds defined relative to each agent’s initial heading.A 2D rotation matrix aligns the thresholds with the agent heading at timestamp 0.
  • Threshold scaling: Because initial speeds differ, the longitudinal and lateral thresholds are scaled by speed to avoid over-penalizing faster agents.The scaling function is clamped between 0 and 1, with υH set to 11 m/s and υL to 1.4 m/s.

D. Overlap Metric

The overlap metric measures trajectory collisions with other objects, and the interactive split is empirically more challenging for simple models than the regular split.

  • Split comparison: The interactive split contains more overlap cases despite requiring predictions for only two agents instead of up to eight in the regular dataset.Table 5 reports this pattern for constant-velocity and LSTM models that do not model other agents.
  • Baseline context: The baseline architecture combines an agent-state LSTM with detachable roadgraph, traffic-state, and high-order interaction encoders.Trajectories are predicted through an MLP trained with min-of-k loss.
  • Metric definition: Overlap rate is the fraction of predicted agents whose trajectories overlap a ground-truth box at any timepoint.The metric counts overlapping agents relative to the total number of predicted agents.
  • Split comparison: 38.4% of predicted vehicles overlap ground truth in the regular split versus 44.2% in the interactive split for the constant velocity model.The comparison uses a marginal overlap-based metric.

E. Conditional Model Details

The conditional model extends the baseline architecture with conditional encoding, while precision–recall analyses examine prediction-count effects across horizons and trajectory shapes.

  • Conditional model: The conditional behavior model uses the baseline LSTM encoder plus roadgraph, traffic-signal, and global interaction-graph enhancements.It adds an early-fusion conditional encoder for conditional prediction.
  • Qualitative examples: The included videos visualize selected dataset scenarios, including the pedestrian–vehicle interactions shown in Figure 1.They provide qualitative examples rather than additional model metrics.
  • Evaluation: Precision–recall curves evaluate increasing prediction counts K for vehicle trajectories across shape buckets at 3, 5, and 8 seconds.The evaluations use the standard validation dataset and the polyline model.
  • Evaluation: At 3 seconds, recall increases with K while area under the precision–recall curve decreases.The reported comparison is across trajectory shape buckets for vehicles.
Loading 2104.10133v1…