Source-linked AI summary
PRECOG: PREdiction Conditioned On Goals in Visual Multi-Agent Settings
Nicholas Rhinehart, Rowan McAllister, Kris Kitani, Sergey Levine
TL;DR
Autonomous vehicles need to reason about uncertain human-driver behavior in interacting traffic scenes. The paper proposes probabilistic multi-agent forecasting with goal-conditioned queries using perceptual observations, and reports improved forecasting relative to prior methods, including when conditioning on the AV’s goal. It also applies conditional forecasts to offline multi-agent imitative planning.
Problem
Autonomous vehicles need to reason about uncertain intentions and decisions of human-driven vehicles in multi-agent environments.
Method
The paper proposes ESP, a factorized flow-based generative model that forecasts joint agent trajectories from past positions and rich scene observations, and PRECOG for conditioning forecasts on agent goals.
Results
The model outperforms state-of-the-art multi-agent forecasting methods on real nuScenes and simulated CARLA data, while conditional forecasts improve joint-agent and per-agent predictions over unconditional forecasts.
Takeaways & Limitations
Goal-conditioned forecasting can model how agents respond to a robot’s intended goal and can support offline planning with a multi-agent imitative planning objective.
Takeaways & Limitations
Executing planned decisions on the robot is outside the scope of this work, and the setup assumes confidence that a positional goal can be achieved.
Abstract
from arXiv · showhide
For autonomous vehicles (AVs) to behave appropriately on roads populated by human-driven vehicles, they must be able to reason about the uncertain intentions and decisions of other drivers from rich perceptual information. Towards these capabilities, we present a probabilistic forecasting model of future interactions between a variable number of agents. We perform both standard forecasting and the novel task of conditional forecasting, which reasons about how all agents will likely respond to the goal of a controlled agent (here, the AV). We train models on real and simulated data to forecast vehicle trajectories given past positions and LIDAR. Our evaluation shows that our model is substantially more accurate in multi-agent driving scenarios compared to existing state-of-the-art. Beyond its general ability to perform conditional forecasting queries, we show that our model's predictions of all agents improve when conditioned on knowledge of the AV's goal, further illustrating its capability to model agent interactions.
1. Introduction
PRECOG addresses multi-agent driving by forecasting interactions from perceptual data and conditioning predictions on agents’ goals. Its ESP model and goal-conditioned forecasting improve prediction and support offline planning in multi-agent environments.
- Autonomous driving requires predicting interacting agents’ future behaviors across situations such as stop signs, roundabouts, crosswalks, parking, and merging.
- ESP is a multi-agent forecasting model using exact likelihood inference to outperform three state-of-the-art methods on real and simulated vehicle datasets.
- PRECOG forecasts how other agents respond to different decisions or goals of a controlled robot agent.
- Conditioning one agent’s goal alters predictions of other agents after modeling their interactions.
- The planning objective balances reaching a goal with the probability that expert demonstrators would execute the same plan, improving forecasting performance.
2. Related Work
Multi-agent forecasting is difficult because agents react concurrently, making faithful interaction models important for safe control. Related approaches include game-theoretic planning and forecasting methods that must account for uncertainty and dependencies among agents.
- Motivation: Concurrent reactions make multi-agent modeling and forecasting challenging for control applications.Faithful models help anticipate dangerous situations before they occur.
- Motivation: Modeling dependencies between agents is especially critical in tightly coupled scenarios such as intersections.
- Game-theoretic planning: Game-theoretic planning generalizes decision-process frameworks to multiple decision makers and explicitly models agents’ policies or internal states.These frameworks support reasoning about collaboration strategies.
- Game-theoretic planning: State-space explosion makes multi-agent planning and game-theory approaches intractable unless interactions are sparse or hierarchically decomposable.
- Multi-agent forecasting: For controlled-agent planning, multi-agent forecasting requires planning over a distribution of possible paths because other drivers’ future actions are uncertain.The method models co-influence between the robot and human drivers.
3. Deep Multi-Agent Forecasting
The paper develops a likelihood-based generative model for forecasting trajectories of interacting vehicles from past states and perceptual observations. Factorized latent variables support flexible agent-specific reactions, conditional forecasting, and planning through controlled robot latents.
- Likelihood-Based Forecasting: The forecasting process represents a variable number of interacting vehicles using past multi-agent positions and robot-observed LIDAR or camera information.The context includes a history of positions and a high-dimensional scene observation, with LIDAR represented as a spatial point histogram.
- Likelihood-Based Forecasting: ESP models T-step multi-agent trajectories with an invertible, likelihood-based generative mapping from latent variables and scene context.The model computes exact probabilities of joint trajectories, supporting planning with the learned distribution.
- Factorized Interactions: Latent variables factorize across agents and time, allowing the model to represent decoupled agent decisions while retaining coupled agent dynamics.This factorization enables queries about how other agents respond to different robot decisions at different times.
- Conditional Forecasting: Robot control is implemented by assigning values to robot latent variables, while human latent variables remain stochastic and uncontrollable but respond through the evolving joint history.Conditional forecasts fix the robot latent values, sample human reactions, and transform the combined latent state into trajectories.
- Planning: Planning optimizes robot latent controls using a goal likelihood together with the learned probability of safe multi-agent trajectories.The resulting posterior combines reaching the goal with the expert-demonstration prior represented by ESP.
4. Experiments
The experiments evaluate multi-agent forecasting against prior methods, analyze interaction-aware and multimodal predictions, and test goal-conditioned planning on simulated and real driving data.
- Experimental setup: The evaluation compares ESP with SocialGAN, DESIRE, and R2P2-MA on CARLA and nuScenes multi-agent forecasting settings.CARLA models train on Town01 and test on Town02 without Town02 training data.
- Multi-agent forecasting: In the didactic interaction test, R2P2-MA produces crashes 50% of the time because it does not condition either agent on the other’s future positions.ESP can react to the human’s decision during generation by turning when the human turns.
- Multi-agent forecasting: ESP achieves the best ˆm and ˆe scores across all 10 evaluation settings, while visual context uniformly improves performance.The ESP no-LIDAR ablation is evaluated with inputs matched to SocialGAN.
- Multi-agent forecasting: ESP forecasts multimodal speeds and paths while capturing social behaviors such as one vehicle waiting for another before accelerating.Qualitative examples include alternative turns, waiting or moving jointly, overtaking, and intersection waiting.
- PRECOG experiments: Planning the robot toward its observed final state significantly improves joint trajectory quality, with the largest gains for the planned agent.Forecasting performance for non-planned agents also improves across all datasets and agents, often most for the closest vehicle.
5. Conclusions
The paper concludes that ESP outperforms state-of-the-art forecasting methods, while PRECOG enables goal-conditioned forecasts and supports multi-agent planning.
- ESP outperforms state-of-the-art multi-agent forecasting methods on real nuScenes and simulated CARLA driving data.
- PRECOG conditions forecasts on agent goals, allowing one agent’s goal to alter predictions for other agents.
- Conditional forecasts improve joint-agent and per-agent predictions compared with unconditional forecasts.
- Goal-conditioned forecasting is performed by planning the controlled agent’s latent behavior, then sampling other agents’ stochastic behaviors.
C. Architecture and Training Details
The architecture processes agent trajectories and LIDAR-derived spatial features, augmenting position features with multi-radius arc sampling and signed-distance inputs.
- A convolutional network processes the spatial input into a feature grid, while agent trajectories are represented in local coordinate frames.
- “Whiskers” interpolate features at seven points along arcs across six radii to represent forecasted agents’ future fields of view.
- The resulting whisker feature has size 8·7·6, and the approach was used in R2P2-MA and all proposed methods.
- In nuScenes, a signed-distance transform replaces the original spatial input and was found to augment performance.
D. Baseline Implementations
The baselines include KDE, DESIRE, SocialGAN, and an extended R2P2 model, with differences in joint modeling, likelihood computation, and planning support.
- SocialGAN uses a public implementation, but improved performance required parameters from its run script rather than default training settings.
- DESIRE is reimplemented as a conditional VAE, whereas the proposed model jointly models multiple agents with exact likelihood.
- The graphical-model comparisons show prior work, R2P2-MA, ESP, and PRECOG under shared notation and agent counts.
- R2P2 is extended to the multi-agent setting as R2P2-MA, with likelihood computed by assuming independence across agents.
- Because R2P2 does not model future inter-agent influence, it cannot be used for multi-agent planning.
E. CARLA Dataset Details
The CARLA dataset was generated from long Town01 autopilot episodes containing many other vehicles, with trajectories and LIDAR recorded for train, validation, and test sets.
- The dataset uses over 900 Town01 episodes of 100 seconds each with the autopilot driving among 100 other vehicles.
- Each episode records every vehicle’s trajectory and the autopilot’s LIDAR observation.
- Episodes are randomized into training, validation, and test sets.
F.1. Robustness to Agent Localization Errors
The appendix evaluates robustness to noisy agent localization and reports that training with injected noise mitigates sensitivity to test-time perturbations. It also summarizes forecast distributions across datasets and horizons.
- Localization robustness: Noise-injected training is less sensitive to test-time localization perturbations than training without noise at all perturbation scales.The comparison uses nuScenes with A = 2 and contrasts Mϵ=0.0 with Mϵ=0.1.
- Localization robustness: Noise injection also improves performance when test data are not perturbed.
- Forecast distributions: The reported median ˆmK=12 is 0.09 on CARLA Town02 and 1.31 on nuScenes under their respective A = 5, four-second or two-second settings.CARLA Town02 uses T = 20 at 10Hz, while nuScenes uses T = 20 at 5Hz.
F.2. Additional CARLA and nuScenes Evaluations.
Additional evaluations cover CARLA and nuScenes forecasting, including multi-agent comparisons, planning-conditioned results, and longer-horizon or distributional analyses.
- CARLA evaluation: Variants of the ESP method mostly outperform prior work in the multi-agent CARLA setting.Table 4 reports mean sample quality ˆm (12) and log likelihood ˆe (11), with standard errors.
- Planning-conditioned evaluation: Planning the robot to a goal position generates better predictions for all agents on CARLA Town01 Test and nuScenes Test data.Table 6 reports means and standard errors.
- Additional analyses: The appendix includes single-agent CARLA results, forecast histograms, longer-horizon forecasting, and plots of mean ˆmK versus K.
F.3. Full Conditional Forecasting Experiments
Full conditional-forecasting experiments extend the evaluation across agent counts and forecasting configurations. The reported trends show improved predictions for all agents when the ego-agent goal is known.
- Conditional forecasting: For A = {3, 4}, PRECOG improves predictions of all agents’ future trajectories.
- Conditional forecasting: Knowledge of the ego-agent’s goal improves predictions for closer agents more than for farther agents.
- Evaluation settings: Table 5 reports CARLA A = 1 performance, where the model is identical to R2PA-MA.
- Evaluation settings: Table 7 evaluates CARLA Town01 Test with T = 40 at 10Hz, corresponding to four seconds of future.This setting has larger dimensionality than the shorter CARLA and nuScenes configurations described in the table caption.
G. Additional Visualizations
Additional visualizations show forecasting and planning examples from CARLA and nuScenes, including joint samples, qualitative quality levels, and planning-criterion landscapes.
- Forecasting visualizations: Figures 16 and 17 show 12 joint forecast samples from held-out nuScenes and multi-agent scenes, respectively.LIDAR colors are discretized into near-ground and above-ground.
- Planning visualizations: Figures 18 and 19 show planned multi-agent forecasting examples from CARLA and nuScenes.Conditioning the robot on its true final position makes robot predictions more accurate and often improves predictions of the other agent.
- Qualitative quality: Figures 20 and 21 order qualitative CARLA and nuScenes results by ˆmK=12 across approximately 100%, 50%, and 0% quality levels.The captions emphasize that ˆm is a joint-agent statistic based on the closest joint trajectory to the true future.
- Planning criterion: Figures 22 and 23 plot the planning criterion ˆL over candidate positions for four-second spatio-temporal goals in CARLA and nuScenes.The criterion prefers locations within the vehicle’s lane unless turning is uncertain.