Source-linked AI summary
Rules of the Road: Predicting Driving Behavior with a Convolutional Model of Semantic Interactions
Joey Hong, Benjamin Sapp, James Philbin
TL;DR
The paper targets the lack of entity-state prediction methods that exploit mature 3D perception and semantic maps in complex driving scenes. It combines these inputs in a unified top-down grid, predicts multimodal future-state distributions, and evaluates them on a large, rich vehicle dataset. The authors report strong performance up to 5 seconds ahead and improvement over baselines without scene context.
Problem
Entity-state prediction lacks methods that leverage mature high-fidelity 3D perception outputs and semantic maps, despite the need to predict distributions over actors’ possible futures in driving.
Method
The model encodes static and dynamic world history with semantic-map information in a unified top-down spatial grid and uses convolutional modeling with multimodal output distributions.
Results
The authors report strong vehicle-behavior prediction up to 5 seconds ahead and improved performance over baselines that do not encode scene context.
Takeaways & Limitations
The framework provides diverse future samples and jointly represents entity dynamics, interactions, and semantic scene context for complex driving scenarios.
Takeaways & Limitations
The non-parametric framework is used only during inference, while incorporating it into a learned end-to-end system remains future work.
Abstract
from arXiv · showhide
We focus on the problem of predicting future states of entities in complex, real-world driving scenarios. Previous research has used low-level signals to predict short time horizons, and has not addressed how to leverage key assets relied upon heavily by industry self-driving systems: (1) large 3D perception efforts which provide highly accurate 3D states of agents with rich attributes, and (2) detailed and accurate semantic maps of the environment (lanes, traffic lights, crosswalks, etc). We present a unified representation which encodes such high-level semantic information in a spatial grid, allowing the use of deep convolutional models to fuse complex scene context. This enables learning entity-entity and entity-environment interactions with simple, feed-forward computations in each timestep within an overall temporal model of an agent's behavior. We propose different ways of modelling the future as a distribution over future states using standard supervised learning. We introduce a novel dataset providing industry-grade rich perception and semantic inputs, and empirically show we can effectively learn fundamentals of driving behavior.
1. Introduction
The paper addresses limited entity-state prediction research by combining mature 3D perception outputs, semantic maps, and a larger realistic dataset in a unified spatial representation. It predicts multimodal future-state distributions and reports strong vehicle-behavior performance up to 5 seconds ahead.
- 1. Introduction: Entity-state prediction remains underexplored despite its importance for modeling actors’ unobserved intents in real-world robotic systems.The motivating driving application also requires understanding static and dynamic environments, traffic laws, conventions, and interactions.
- 1. Introduction: Previous work often relied on raw sensors, whereas industrial perception systems provide mature, high-fidelity 2D and 3D entity states and attributes.
- 1. Introduction: Public prediction datasets were too small or unrealistic, and prior research omitted semantic maps despite their importance in industry self-driving systems.The paper identifies diversity of locations, many unique 3D tracks, and meaningful time intervals as dataset requirements for generalization and long-horizon prediction.
- 1. Introduction: The introduced dataset contains 9,659 unique vehicles across 83,880 prediction scenarios, 173 hours, and 88 physically distinct locations, with semantic map information.
- 1. Introduction: The model encodes static and dynamic world history plus semantic maps in a unified top-down grid, enabling convolutional modeling of dynamics, interactions, and scene context jointly.
- 1. Introduction: The approach predicts future-state distributions rather than point estimates and shows strong vehicle-behavior performance up to 5 seconds ahead while leveraging road information and other agents’ states.The paper presents multiple parametric and non-parametric output distribution representations.
2. Related Work
Related work covers forecasting, behavior cloning, semantic-grid models, and multimodal prediction. The paper distinguishes its approach by directly predicting time sequences of multimodal distributions in a one-shot feed-forward framework.
- Activity/motion forecasting: Earlier forecasting work framed activity prediction through partially observable decision processes, inverse optimal control, or one-step policy distributions.
- Activity/motion forecasting: Behavior-cloning approaches commonly regress trajectories while jointly handling detection, tracking, motion forecasting, or semantic road-map inputs.IntentNet predicts single trajectories per entity out to 3 seconds.
- Activity/motion forecasting: Egocentric forecasting adds the challenge of recovering ego position or velocity when predicting the ego entity or other entities from a moving camera frame.
- Modeling entity interactions: IntentNet and ChaufferNet encode static and dynamic scene context in top-down rasterized grids, similar to this paper’s spatial representation.
- Modeling entity interactions: Unlike one-step stochastic-policy methods such as DESIRE and R2P2, this work directly predicts a time sequence of multimodal distributions.Policy-based alternatives require future rollout and sampling, creating computational trade-offs and uncertainty about the number of samples needed for confidence.
3. Method
The method represents a target entity, other entities, world history, and semantic road context in a unified spatial grid, then predicts multimodal future-state distributions with neural models.
- Input representation: The entity-centric model encodes a target entity and its surrounding world context, while incorporating other entities into the prediction.It maps past and present world representations to future behavior using output forms intended for robot planning.
- Input representation: Road-network data are rendered as top-down geometric primitives, including lanes, junction connectivity, crosswalks, traffic-light permissibility, and stop or yield lines.Rendering establishes a common spatial coordinate system, but loses the road network’s true graph structure.
- Input representation: Tracked entities contribute positions, velocities, accelerations, dimensions, orientations, classes, and perception uncertainty to spatial tensors aligned with the road representation.Additional dynamic context is encoded through oriented bounding boxes and traffic-light permissibility rendered in an RGB tensor.
- Temporal modeling: Past scene tensors are concatenated over time in a coordinate system centered on the target entity at prediction time.The representation can be augmented with features such as brake lights, turn signals, pose, gestures, and audio cues.
- Output distributions: Future outputs are designed to be probabilistic, multimodal, and one-shot, rather than single point estimates, to represent uncertain and diverse possible actions.The paper predicts future xy-displacements and explores several parametric and non-parametric representations of P(Y | X).
- Output distributions: A bivariate Gaussian output predicts future-position means, standard deviations, and correlations, while mixtures of Gaussian trajectories add multiple possible futures and trajectory weights.The naive mixture formulation has exchangeability and mode-collapse problems; a CVAE with a categorical latent variable addresses the mixture construction.
- Output distributions: The GMM-CVAE models the discrete Gaussian-mixture choice with a categorical latent variable trained using a conditional variational autoencoder and Gumbel-Softmax reparameterization.This formulation uses P(z|X) as the discrete mixture distribution over future states.
- Output distributions: The non-parametric trajectory-selection framework is used only during inference to maximize scores while enforcing separation between selected trajectories.The paper identifies incorporating this framework into an end-to-end learned system as future work.
4. Experiments
The experiments evaluate a large, intersection-focused driving dataset and compare multimodal prediction methods using scene context, probabilistic outputs, and qualitative trajectory behavior. Results show that other agents and road maps improve prediction, especially at longer horizons, while some sampling approaches remain limited.
- Dataset: The dataset contains more than 6.25 million frames, with 72,878 training and 10,473 test events split across non-overlapping intersections.Events last 7.5 seconds and are concentrated near intersections to emphasize non-trivial behavior.
- Quantitative method comparison: Best-in-top-5 trajectory evaluation outperforms single MAP evaluation across all metrics, supporting multiple predicted modes.The comparison uses all input features and evaluates multimodal prediction methods.
- Qualitative analysis: Mixtures of sampled Gaussian trajectories underperform the other proposed methods because some samples are implausible.The authors leave better techniques for directly obtaining diverse trajectory samples to future work.
- Quantitative method comparison: Linear and Industry baselines perform better at small offsets but worse at larger offsets than the proposed methods.The paper attributes near-future performance to classical physics and distant-future prediction to semantic understanding.
- Qualitative analysis: Qualitative trajectories exhibit lane keeping, traffic-light compliance, following, maneuvering around vehicles, and lane changes, but Grid Map can predict a turn into oncoming traffic.The failure is attributed to traffic rules that may be difficult to infer from only a road map.
5. Conclusion
The paper presents a unified framework for multimodal future-state prediction that combines static and dynamic scene context with high-definition road maps. Evaluation shows improved performance over baselines without scene context and diverse samples in complex driving scenarios.
- 5. Conclusion: The framework combines sensor-derived static and dynamic scene context with high-definition road maps for multimodal future-state prediction.It supports both continuous and discrete output representations.
- 5. Conclusion: Empirical and qualitative evaluations show improvements over baselines without scene context and diverse samples in complex driving scenarios.The conclusion summarizes the paper’s reported performance and qualitative behavior.