Source-linked AI summary
What-If Motion Prediction for Autonomous Driving
Siddhesh Khandelwal, William Qi, Jagjeet Singh, Andrew Hartnett, Deva Ramanan
TL;DR
Forecasting actor motion is challenging because futures are multimodal and depend on social, road, and AV-plan context. WIMP uses graph-based attention to condition diverse forecasts on lane goals and social context; it reports strong results on Argoverse and NuScenes and supports planner-relevant counterfactual queries.
Problem
Actor futures are multimodal and depend on other actors, road structures, and the AV’s intended motion plan, making forecasting a central challenge for autonomous vehicles.
Source: §1 · p. 1Method
Results
WIMP outperforms prior published work on Argoverse and reports state-of-the-art single- and multi-modal results on NuScenes without architecture changes or hyperparameter tuning.
Source: §4.2 · pp. 7–8, §5 · p. 8Takeaways & Limitations
Conditioning on specific lane goals and adjustable social context lets a planner query trajectories relevant to its intended route, including hypothetical interactions.
Source: §1 · p. 2Multi-agent prediction for all actors did not improve performance; the authors suggest parked actors may require different representations or greater model capacity.
Source: §3.5 · p. 6
Abstract
from arXiv · showhide
Forecasting the long-term future motion of road actors is a core challenge to the deployment of safe autonomous vehicles (AVs). Viable solutions must account for both the static geometric context, such as road lanes, and dynamic social interactions arising from multiple actors. While recent deep architectures have achieved state-of-the-art performance on distance-based forecasting metrics, these approaches produce forecasts that are predicted without regard to the AV's intended motion plan. In contrast, we propose a recurrent graph-based attentional approach with interpretable geometric (actor-lane) and social (actor-actor) relationships that supports the injection of counterfactual geometric goals and social contexts. Our model can produce diverse predictions conditioned on hypothetical or "what-if" road lanes and multi-actor interactions. We show that such an approach could be used in the planning loop to reason about unobserved causes or unlikely futures that are directly relevant to the AV's intended route.
1 Introduction
Motion forecasting must account for multimodal futures shaped by other actors, road structure, and the AV’s intended plan. WIMP uses road-network and social context to support counterfactual forecasts relevant to the planner.
1 Introduction
- Actor futures are multimodal and depend on other actors, road structures, and even the AV’s intended motion plan.Large-scale AV testing and public driving datasets and maps have stimulated data-driven feedforward approaches.Source: §1 · p. 1
- Rasterized bird’s-eye-view models can require many parameters and are less suited to coordinate spaces that represent vehicle physics, lane behavior, and social interactions.Euclidean space suits vehicle motion, curvilinear road coordinates suit lane interactions, and topological graphs can represent interactions among actors.Source: §1 · p. 1
- Because real-world urban driving has a large space of potential futures, planners may need to probe only relevant futures, and scene reprocessing may be too inefficient for real-time use.Safety-critical applications may also require top-down feedback and causal explainability.Source: §1 · pp. 1–2
- WIMP combines a recurrent model, road-network attention, and a dynamic interaction graph to support counterfactual forecasts conditioned on topological goals and manipulated social context.The planner can query trajectories for a lane intersecting its path or add hypothetical actors.Source: §1 · p. 2
2 Related Work
Prior motion-forecasting work spans recurrent social models, scene-context fusion, rasterized representations, and graph methods. WIMP is related to conditional and graph-based approaches but differs in its lane-polyline conditioning and support for diverse counterfactual forecasts.
2 Related Work
- WIMP builds on graph attention networks [41] and is related to sequence-model and feature-learning approaches; its work is most related to tracking-output methods, while sensor-input approaches are not explored in depth.The paper also points to seq2seq [38] and ResNet [19] as theoretically and practically important architectures.Source: §2 · pp. 2–3
- Social-LSTM introduced social pooling in an RNN encoder-decoder, followed by methods including SoPhie [34] and DESIRE [23] that incorporate physical-environment features, attention, adversarial training, or scene-context fusion.Source: §2 · p. 2
- Prior conditional methods use anchor trajectories [7] [30], maneuver classification [13] [14], or hypothetical focal-agent rollouts [39]; PRECOG [32] conditions other actors’ futures on an actor’s goal.WIMP is described as requiring no rasterized input and allowing efficient social-context changes.Source: §2 · pp. 2–3
- Rasterized methods combine histories with map semantics, while rasterization-free methods encode social interactions; WIMP attends to lane polylines as well as the social graph.The cited rasterized approaches include both simultaneous and recurrent trajectory prediction.Source: §2 · p. 3
- VectorNet represents road components and agent histories as vectors, but conditions on the whole local neighborhood and uses a deterministic decoder limited to one trajectory.WIMP instead conditions on specific map elements and uses a multimodal decoder for diverse predictions.Source: §2 · p. 3
3 Method
WIMP models actor motion with a recurrent, graph-based attentional architecture that combines actor histories, road polylines, and social context. Its decoder produces diverse predictions conditioned on reference polylines, while its learning procedure encourages multimodal outputs.
3.2 Geometric Context via Polyline Attention
- Reference paths use concatenated lane-center polylines, and learned attention selects current and goal indices using the actor’s position and a coarse waypoint predicted from kinematic history.The waypoint predictor does not use polyline information and is trained with ground-truth future trajectories as labels.Source: §3 · p. 3, §3.2 · p. 4
3 Method
- WIMP combines a graph-based encoder for scene context and higher-order social interactions, a multimodal decoder, and polyline attention over relevant road regions.The model represents each actor’s future conditional on observed histories and reference polylines.Source: §3 · pp. 3–4
3.1 Historical Context via Recurrence hn
- A shared recurrent encoder captures each actor’s history, normalized by translation and rotation so the focal actor’s heading aligns with the +x-axis.The focal actor is placed at the origin in this reference frame.Source: §3.1 · p. 4
3.3 Social Context via Graph Attention
- Graph attention incorporates other actors’ latent representations as a learned residual change to each actor’s socially unaware hidden state.The model applies graph attention over actor representations rather than supplying all other actors’ planar coordinates as direct inputs.Source: §3.3 · pp. 4–5
3.4 Decoding
- An LSTM-based decoder generates diverse, multimodal predictions conditioned on each actor’s reference polyline and graph-encoder social embeddings.The prediction function can use different output distributions, including a bivariate Gaussian or mixture of Gaussians.Source: §3.4 · p. 5
3.5 Learning
- Training uses trajectory-history, future, and map-context triplets for a focal actor, with winner-takes-all mixtures and evolving WTA to encourage diverse predictions.Multi-agent prediction did not improve performance in the authors’ experiments; they suggest parked actors may require different representations or greater model capacity.Source: §3.5 · pp. 5–6
4 Experiments
The experiments evaluate WIMP on Argoverse and NuScenes, including challenging blind-turn scenarios, component ablations, and counterfactual queries. Results report strong benchmark performance and show how predictions respond to road and social context.
4 Experiments
- The experiments assess accurate, interpretable, and controllable predictions, including how scene context influences the distribution of predicted modes.The study describes scene-attention embeddings combined with multi-modal decoders and counterfactual reasoning experiments.Source: §4 · p. 6
4.1 Experimental Setup
- The experiments use Argoverse’s more than 300,000 scenarios and NuScenes’ approximately 40,000 scenarios, with displacement-error and miss-rate metrics across multiple prediction counts.Argoverse forecasts three seconds from a two-second history; its blind-turn subset contains about 350 challenging cases, while NuScenes includes geographically diverse, more complex scenarios.Source: §4.1 · pp. 6–7
4.2 Quantitative Results
- On the Argoverse test set, WIMP outperforms prior published work and performs similarly to concurrent work.The comparison includes miss rate and minimum FDE/ADE for K = 1 and K = 6 predictions.Source: Table 1 · p. 7, §4.2 · p. 7
- On blind turns, WIMP outperforms non-map-based approaches by a larger margin than on the full dataset and performs well in both blind-turn and general settings.The authors also report improved accuracy with higher-quality reference polylines, including gains from oracle-selected polylines.Source: Table 2 · p. 7, §4.2 · p. 7
- The best ablation results use both map and social context with L1-based EWTA; NLL significantly degrades performance, while standard L1 cannot support multiple predictions.The ablation reports K = 1 and K = 6 results on the Argoverse validation set.Source: §4.2 · pp. 7–8, Table 3 · p. 8
- On NuScenes, WIMP achieves state-of-the-art results across single- and multi-modal scenarios, outperforming previous methods in miss rate and displacement error.The evaluation used the Argoverse model without hyperparameter tuning or architecture changes; the authors associate especially strong results with NuScenes’ many intersection and turn scenarios.Source: Table 4 · p. 8, §4.2 · p. 8
4.3 Counterfactual Validation
- Counterfactual queries support testing predictions under hypothetical or unlikely scenarios and allocating planner computation to relevant futures.The authors also propose using such queries to investigate social influences from occluded road regions and evaluate models beyond distance-based metrics.Source: §4.3 · p. 8
5 Discussion
WIMP combines historical, social, and geometric information for multi-actor multimodal forecasting and supports counterfactual context. Its Argoverse performance and visualizations illustrate benchmark performance and responses to hypothetical lane and actor changes.
5 Discussion
- WIMP supports joint multimodal prediction over an arbitrary number of scene actors and outperforms all previous methods on the Argoverse forecasting dataset.The framework injects counterfactual contextual states using interpretable geometric and social relationships.Source: §5 · p. 8
- Future work could make polyline selection trainable and explore pedestrian or human-action prediction beyond autonomous driving.The authors state that they will release code to support such work.Source: §5 · p. 8
- When a counterfactual reference polyline intersects other actors, WIMP learns to ignore the map input and rely on social and historical context.Some diverse predictions also proceed independently of the conditioning polyline.Source: Figure 4 · p. 9
- Injecting a stopped vehicle beyond an intersection removes the right-turn mode, while replacing a leading actor with a stopped vehicle yields a representative final velocity of 3.3m/s versus 10.3m/s originally.The second scenario produces trajectories with aggressive deceleration.Source: Figure 5 · p. 9
Broader Impact
Motion forecasting conditioned on dynamic agents is presented as central to safe and efficient AV operation. The discussion connects forecasting progress to deployment and highlights planner utility, novel-scenario response, and interpretability as practical requirements.
Broader Impact
- Conditioning plans on dynamic agents’ future states is a central challenge for safe and efficient AV operation.Source: Broader Impact · p. 9
- Motion-forecasting progress has downstream implications for AV deployment timeline, scale, and performance in paratransit, long-haul freight, and local delivery.Source: Broader Impact · p. 9
- Prediction models should serve planners, respond prudently to extreme or perturbed scenarios, and provide interpretable decisions for safety cases and public trust.Source: Broader Impact · p. 9
A Implementation Details
The implementation uses recurrent modules and attention to process trajectories and map polylines, with scenario coordinates normalized to the focal agent. Training and evaluation details specify the Argoverse setup and forecasting metrics.
A Implementation Details
- WIMP abstracts learned components and transformations so configurations can support varied prediction tasks; reported results use a specified Argoverse implementation.Source: A Implementation Details · p. 13
- Scenario points, including trajectories and reference polylines, are transformed into a focal-agent local coordinate frame with the positive X-axis aligned to its heading.Source: A Implementation Details · p. 13
- Separate encoder and decoder polyline-attention modules use four-layer LSTMs; the hidden state is 4 × 512, and learned Q, K, V matrices are 512 × 512.The distance metric is the L2-norm, and dropout of 0.5 is applied over the first three layers during training.Source: A Implementation Details · p. 13
- The shared actor-history encoder is a four-layer LSTM with a 512-dimensional hidden state, while graph attention uses four heads and parameters sized 2048 × 512 and 1024 × 1.The decoder is configured like the encoder, and its prediction component is a linear layer.Source: A Implementation Details · p. 13
- Training uses ADAM with minibatches of 100 scenarios, gradient clipping at 1.0, an annealed learning rate, and an EWTA-based loss; training averages about 100 epochs and 28 hours on four V100 GPUs.Validation is checked every three epochs, and training stops after 30 consecutive epochs without improvement.Source: A Implementation Details · p. 13
- Evaluation transforms predictions back to global coordinates and computes minFDE, minADE for the trajectory with lowest endpoint error, and miss rate at a 2m minFDE threshold.Source: A Implementation Details · p. 13
B Polyline Selection
A heuristic proposal procedure finds and ranks candidate lane polylines using the observed focal-agent history at evaluation or the ground-truth trajectory during training. It combines geometric overlap and trajectory alignment scores, with inference allowing a diversity–accuracy trade-off.
B Polyline Selection
- Candidate polylines are retrieved from the Argoverse map API and ranked against the focal actor’s 0–2s history at evaluation or full 0–5s ground-truth trajectory during training.Source: B Polyline Selection · p. 14
- The procedure searches for lane nodes within 2.5m of the query endpoint, expands the radius by a factor of two if needed, then traverses predecessors and successors up to twice the query-trajectory length.Source: B Polyline Selection · p. 14
- After removing significantly overlapping candidates, the method ranks polylines by point-in-polygon score, the number of query points inside their lane-region polygon.The score ranges from zero to the number of trajectory points.Source: B Polyline Selection · p. 14
- The alignment score ranks a candidate by the maximum tangential distance reached along its polyline, with longer travel indicating better alignment.Source: B Polyline Selection · p. 14
- PIP and alignment scores are combined because either alone can favor semantically or directionally misleading lanes; proposals are drawn from their rankings in alternating order.Source: B Polyline Selection · p. 14
- Training uses only the top-ranked proposal, while inference can trade diversity against accuracy by changing how predictions are distributed across polylines.For example, inference can generate six predictions from one polyline or one prediction from each of six polylines.Source: B Polyline Selection · p. 15
C Improving Maps via Prediction
Repeated disagreement between accurate polyline-based predictions and observed traffic can reveal locations where lane proposals miss dominant behavior. Explicit path-conditioning could also support map maintenance through disagreement-weighted priors that adapt to environmental changes.
C Improving Maps via Prediction
- Repeated disagreement between accurate polyline-based predictions and observed traffic identifies map locations where proposed lane polylines miss dominant traffic modes.Figure 6 shows three such intersections.Source: C Improving Maps via Prediction · p. 15, Figure 6 · p. 15
- Disagreement rates could inversely weight map-polyline priors, letting proposal modules select high-quality reference trajectories and update them as environmental conditions change.Examples of changing conditions include construction, weather, and potholes.Source: C Improving Maps via Prediction · p. 15
D Argoverse Challenge Entries
WIMP outperformed previously published and publicly released methods on evaluated MR- and DE-based metrics, but later leaderboard entries improved further. The section compares leaderboard snapshots from March 23 and June 8, 2020.
D Argoverse Challenge Entries
- WIMP outperformed previously published and publicly released methods on evaluated MR- and DE-based metrics, while several newer leaderboard entries later surpassed its results.The comparison reports leaderboard snapshots from March 23 and June 8, 2020.Source: D Argoverse Challenge Entries · p. 15
E Dynamic Visualizations
Dynamic visualizations are provided because static 2D images can miss temporal context in trajectory prediction. The examples span varied driving events, while leaderboard captions report WIMP's changing relative standing.
E Dynamic Visualizations
- Dynamic visualizations address the difficulty of capturing temporal context from static 2D images alone.They accompany the BEV scenarios shown in the main text.Source: E Dynamic Visualizations · p. 15
- Additional visualizations cover events from acceleration and braking to lane changes and protected turns, intended to demonstrate accuracy, diversity, and generalization across geographic and semantic settings.Source: E Dynamic Visualizations · p. 15
- WIMP ranked second on K=6 FDE/ADE and third on MR and K=1 metrics on the May 23, 2020 leaderboard.The table reports Argoverse test-set MR and minimum FDE/ADE for single- and multi-modal predictions.Source: Table 5 · p. 16
- On the June 8, 2020 leaderboard, WIMP ranked second for K=6 ADE/FDE as rankings shifted with new and refined entries.The table reports Argoverse test-set MR and minimum FDE/ADE for single- and multi-modal predictions.Source: Table 6 · p. 16