Source-linked AI summary
Multimodal Trajectory Predictions for Autonomous Driving using Deep Convolutional Networks
Henggang Cui, Vladan Radosavljevic, Fang-Chieh Chou, Tsung-Han Lin, Thi Nguyen, Tzu-Kuo Huang, Jeff Schneider, Nemanja Djuric
TL;DR
Uncertain and multimodal traffic behavior makes it difficult for autonomous vehicles to anticipate surrounding actors’ future motion safely and efficiently. The paper uses CNNs on rasterized actor context to generate multiple trajectories with probabilities, and reports improved prediction errors plus successful closed-course SDV testing.
Problem
Uncertain traffic behavior and diverse road situations require autonomous vehicles to anticipate multiple possible future trajectories of surrounding actors.
Method
A CNN encodes each actor’s rasterized surrounding context and outputs multiple future trajectories with associated probabilities.
Results
MTP methods reduce average and 6s-horizon prediction errors across metrics, perform best with M = 3, and are successfully tested onboard SDVs in closed-course tests.
Takeaways & Limitations
Modeling multimodal vehicle movement provides practical benefits for predicting traffic actors’ uncertain future behavior.
Abstract
from arXiv · showhide
Autonomous driving presents one of the largest problems that the robotics and artificial intelligence communities are facing at the moment, both in terms of difficulty and potential societal impact. Self-driving vehicles (SDVs) are expected to prevent road accidents and save millions of lives while improving the livelihood and life quality of many more. However, despite large interest and a number of industry players working in the autonomous domain, there still remains more to be done in order to develop a system capable of operating at a level comparable to best human drivers. One reason for this is high uncertainty of traffic behavior and large number of situations that an SDV may encounter on the roads, making it very difficult to create a fully generalizable system. To ensure safe and efficient operations, an autonomous vehicle is required to account for this uncertainty and to anticipate a multitude of possible behaviors of traffic actors in its surrounding. We address this critical problem and present a method to predict multiple possible trajectories of actors while also estimating their probabilities. The method encodes each actor's surrounding context into a raster image, used as input by deep convolutional networks to automatically derive relevant features for the task. Following extensive offline evaluation and comparison to state-of-the-art baselines, the method was successfully tested on SDVs in closed-course tests.
I. INTRODUCTION
The paper frames autonomous driving as a safety-critical prediction problem because traffic behavior is uncertain and multimodal. It proposes predicting multiple actor trajectories with probabilities using rasterized context and deep convolutional networks, followed by offline evaluation and closed-course testing.
- Traffic driving remains dangerous, with human error implicated in the vast majority of crashes.
- Safe autonomous operation requires predicting surrounding actors’ movement while accounting for multiple possible future behaviors.
- The proposed method predicts multiple trajectories and their probabilities rather than inferring only a single trajectory.
- The method was evaluated extensively offline and successfully tested onboard self-driving vehicles in closed-course tests.
- Actor-specific rasterized context, including maps and other actors, is used to predict movement in dynamic environments.
II. RELATED WORK
Related work contrasts engineered motion-prediction systems with approaches that incorporate environmental context. Established physical-state propagation works for short horizons but degrades over longer horizons when surrounding context is ignored.
- The paper reviews actor-motion prediction research from the perspectives of engineered, machine-learning, and deep-learning methods.
- Deployed self-driving systems commonly propagate object states with Kalman-filter-based physical models.
- These engineered approaches perform well for short-term prediction but degrade over longer horizons because they omit roads, other actors, and traffic rules.
B. Machine-learned prediction models
Machine-learned prediction methods broaden motion modeling beyond manually engineered physical assumptions, while multimodal approaches represent alternative futures. The reviewed literature includes probabilistic, recurrent, convolutional, and multi-hypothesis strategies.
- Manually engineered models motivated alternatives including Hidden Markov Models, Bayesian networks, Gaussian Processes, and inverse reinforcement learning.
- Figure 1 illustrates multimodal six-second vehicle trajectories with predicted paths shown in blue and probabilities marked at trajectory endpoints.
- Recurrent approaches use LSTMs or GRUs to predict pedestrian or vehicle trajectories from sequence data and interactions.
- Mixture Density Networks model multimodal regression with Gaussian mixtures but can be difficult to train in high-dimensional spaces.
- Multi-hypothesis networks produce M alternative outputs and train using the prediction closest to the ground-truth label.
III. PROPOSED APPROACH
The proposed approach section introduces the problem formulation, notation, convolutional architecture, and loss functions used for multimodal trajectory prediction.
- The method section defines the prediction setting and notation before presenting the CNN architecture and considered loss functions.
A. Problem setting
The problem setting assumes real-time sensor streams, tracked state estimates for surrounding actors, and detailed high-definition maps of the operating area.
- The system receives real-time data from lidar, radar, or cameras mounted on the self-driving vehicle.
- The detection and tracking system outputs each actor’s bounding box, position, velocity, acceleration, heading, and heading change rate.
- Tracker outputs occur at discrete, uniformly spaced times, while the number of tracked actors can vary as actors enter or leave sensor range.
- The vehicle is assumed to access detailed high-definition maps containing roads, crosswalks, lane directions, and other relevant map information.
B. Modeling multimodal trajectories
The model rasterizes each actor’s map and neighboring-actor context, then uses a CNN to predict multiple future position sequences and their probabilities.
- Each actor’s map surroundings and neighboring actors are rasterized into an actor-specific bird’s-eye-view image for CNN input.
- The CNN predicts M possible future state sequences over H consecutive time steps and assigns each sequence a probability.
- The task predicts future x- and y-positions, with remaining states derived from the current state and predicted positions.
- The network takes a 300 × 300 RGB raster at 0.2m resolution plus current velocity, acceleration, and heading change rate as inputs.
- It outputs (2H + 1)M values per actor: 2H position outputs and one probability scalar for each of M modes.
- A softmax layer normalizes mode probabilities to sum to 1, with MobileNet-v2 used as the base CNN architecture.
C. Multimodal optimization functions
The paper compares multimodal trajectory losses and distance functions, proposing Multiple-Trajectory Prediction (MTP) loss to represent distinct behavioral modes and avoid mode collapse.
- The single-mode loss measures average displacement between ground-truth and predicted trajectory points using their 2-D positions.The trajectory points are represented as x- and y-position vectors at each prediction horizon.
- The MTP loss produces multiple trajectories, selects the mode closest to ground truth, and combines its trajectory loss with classification loss.The selected mode is identified using an arbitrary trajectory distance function before the final loss is applied.
- Mixture-of-Experts loss is described as an alternative, but evaluation finds it unsuitable for trajectory prediction because of mode collapse.The MTP loss is motivated as an explicit model of the multimodal trajectory space.
- During training, position outputs update only for the winning mode while probability outputs update for all modes, encouraging behavioral specialization.The authors report that this addresses mode collapse, with modes specializing in behaviors such as going straight or turning.
- The proposed angle-based distance measures the angle between trajectory endpoints from the actor’s position, improving handling of intersection scenarios.The authors motivate this function because average displacement does not model multimodal behavior well at intersections.
- The multimodal loss functions are agnostic to the per-mode loss and can be extended to predict trajectory-point uncertainties using negative Gaussian log-likelihood.This extension follows the uncertainty modeling approach proposed in earlier work.
D. Lane-following multimodal predictions
The method uses lane information encoded through an additional rasterization layer to generate multiple trajectories for lane-following vehicles. It generalizes across intersection and straight-road scenarios and produces paths that follow intended lanes.
- D. Lane-following multimodal predictions: The method adds a rasterization layer encoding possible lanes and trains the network to implicitly output multiple trajectories.A lane-scoring system filters unlikely lanes before the lane information is encoded.
- D. Lane-following multimodal predictions: A 5° angle threshold identifies modes sufficiently close to the ground truth, with displacement distance breaking ties among potential matches.The authors report that this optimization generalizes across intersection and straight-road scenarios.
- D. Lane-following multimodal predictions: Table I compares prediction errors for competing methods in meters.The supplied table caption identifies the comparison metric and unit but does not provide row values.
- D. Lane-following multimodal predictions: Using two alternative following lanes, the method outputs trajectories that follow intended straight and left-turn paths well.The examples use the same scene with two different following lanes.
IV. EXPERIMENTS
The experiments use real-world SDV data with sensor-fused actor tracking and compare the proposed approach with several prediction baselines. Models were trained end-to-end and deployed for onboard batch inference.
- IV. EXPERIMENTS: The dataset contains 240 hours of manually driven SDV data collected in Pittsburgh and Phoenix under varied traffic conditions.Actors were tracked from camera, lidar, and radar using a UKF with a kinematic vehicle model at 10Hz.
- IV. EXPERIMENTS: The considered methods include UKF forward propagation, the single-trajectory predictor from [16], and an MDN modeling trajectories as a Gaussian mixture.These methods form the comparison set described for the experiments.
- IV. EXPERIMENTS: The network infers a 2 × 2 covariance matrix measuring uncertainty at each prediction horizon.This covariance represents uncertainty associated with the predicted trajectory quantity.
- IV. EXPERIMENTS: Models were trained end-to-end from scratch on 16 Nvidia Titan X GPUs for about 24 hours and deployed for onboard batch inference in about 10ms.Training used Horovod and Adam with an initial learning rate of 10^-4.
A. Empirical results
The empirical evaluation shows that MTP captures multimodal traffic behavior more effectively than competing approaches, particularly at longer horizons. Performance is best with three modes, angle-based mode selection improves turning behavior, and predicted mode probabilities are well calibrated.
- Comparative performance: Single-mode models perform reasonably at 1s but produce substantially worse 6s predictions because they represent multimodal futures by their distribution mean.Longer horizons expose discrete behavioral choices, such as alternative maneuvers at intersections.
- Comparative performance: MDN and ME remain similar to STP across mode counts because mode collapse leaves only one non-degenerate prediction.This causes affected multimodal methods to fall back to unimodal behavior.
- Comparative performance: MTP methods significantly reduce average and 6s-horizon errors versus competing approaches, with larger benefits at longer horizons.The results also report lower errors at both 1s and 6s horizons.
- Mode selection: Angle-based mode selection improves handling of turning trajectories while slightly degrading going-straight cases.The authors therefore use the angle policy for the remainder of the evaluation.
- Mode analysis: Increasing M separates straight, right-turn, and left-turn behaviors, while M = 4 additionally splits straight motion into fast and slow modes.The same velocity-related splitting appears on straight roads far from intersections.
- Probability calibration: Predicted mode probabilities are well calibrated, with empirical mode-matching likelihoods closely following the y = x reference line.Calibration is evaluated by bucketing trajectories according to predicted probability.
V. CONCLUSION
The paper models uncertain vehicle motion by generating multiple future trajectories and their probabilities from rasterized surrounding context using a CNN. Extensive offline evaluation suggested practical benefits, and the method was tested onboard SDVs in closed-course tests.
- V. CONCLUSION: The method rasterizes each vehicle's surrounding context and uses a CNN to output several possible trajectories with their probabilities.The raster encodes the surrounding context for each vehicle actor.
- V. CONCLUSION: Extensive offline evaluation and comparison with state-of-the-art methods strongly suggested practical benefits of the proposed approach.The conclusion does not specify a single headline metric.
- V. CONCLUSION: The method was successfully tested onboard self-driving vehicles in closed-course tests.This follows the reported offline evaluation.