Source-linked AI summary
Multimodal Trajectory Prediction Conditioned on Lane-Graph Traversals
Nachiket Deo, Eric M. Wolff, Oscar Beijbom
TL;DR
Predicting surrounding vehicles requires handling uncertainty in both route choices and motion profiles, while generic multimodal decoders can violate scene structure. PGP combines probabilistic lane-graph policy traversals with latent-variable decoding over selectively attended graph context, achieving state-of-the-art nuScenes results and diverse, scene-compliant trajectories.
Problem
Trajectory prediction must represent uncertain lateral route choices and longitudinal motion profiles, but single-context-vector approaches can produce off-road or traffic-rule-violating trajectories.
Method
PGP samples discrete policy traversals through a lane graph and conditions an attention-based decoder on the traversed context and a sampled latent variable.
Results
PGP achieves state-of-the-art nuScenes results, with diverse routes, diverse motion profiles, and the lowest off-road rates reported in the conclusion.
Takeaways & Limitations
Lane-graph traversal conditioning improves route diversity and scene compliance, while latent variables add motion-profile diversity.
Takeaways & Limitations
The formulation assumes access to past trajectories from onboard detectors and multi-object trackers.
Abstract
from arXiv · showhide
Accurately predicting the future motion of surrounding vehicles requires reasoning about the inherent uncertainty in driving behavior. This uncertainty can be loosely decoupled into lateral (e.g., keeping lane, turning) and longitudinal (e.g., accelerating, braking). We present a novel method that combines learned discrete policy rollouts with a focused decoder on subsets of the lane graph. The policy rollouts explore different goals given current observations, ensuring that the model captures lateral variability. Longitudinal variability is captured by our latent variable model decoder that is conditioned on various subsets of the lane graph. Our model achieves state-of-the-art performance on the nuScenes motion prediction dataset, and qualitatively demonstrates excellent scene compliance. Detailed ablations highlight the importance of the policy rollouts and the decoder architecture.
1 Introduction
Trajectory prediction is difficult because future driving behavior is uncertain, while existing graph-based methods often compress the scene into one context vector before generating multiple trajectories. PGP instead uses discrete lane-graph policy traversals and latent variables to model route and motion-profile variability.
- Motivation: Future vehicle motion is uncertain, but lane structure and traffic rules provide exploitable regularities for prediction.HD maps encode road topology and traffic rules, supporting trajectory prediction in complex traffic scenes.
- Motivation: Single-vector multimodal decoders must map entire scenes to multiple trajectories, which can produce off-road or traffic-rule-violating predictions.They must jointly represent lateral route choices and longitudinal motion profiles.
- Method: PGP explicitly models route variability by sampling path traversals from a learned behavior-cloning policy and selectively aggregating the corresponding graph context.This reduces the representational demands on the output decoder while producing diverse route samples.
- Method: Sampled latent variables model longitudinal variability, allowing distinct trajectories even when the sampled path traversal is identical.The authors report that this produces greater diversity in motion profiles.
- Contributions: The method combines discrete policy roll-outs with a lane-graph subset decoder and reports state-of-the-art nuScenes performance with ablations of lateral and longitudinal variation.The stated contributions include benchmark performance and analyses of both motion-variation axes.
2 Related Work
Prior work represents maps and multimodal futures in several ways, including graph encoders, stochastic or latent trajectory models, and goal-conditioned prediction. PGP differs by conditioning on probabilistically sampled lane-graph paths, which encode reachability and route context beyond endpoint goals or deterministically selected centerlines.
- Graph-based map encoding: VectorNet and LaneGCN encode structured HD maps as graphs, but aggregate scene context into learned representations for prediction.VectorNet uses attention, while LaneGCN uses a dilated graph-convolution variant.
- Multimodal prediction: Multimodal prediction methods include trajectory distributions, occupancy grids, stochastic policy roll-outs, and latent-variable mappings.These approaches represent multiple possible futures through different output or sampling formulations.
- Goal-conditioned prediction: Goal-conditioned methods restrict predictions near driver goals but may leave multiple connecting paths or unreachable goals unresolved.The cited limitation arises because reachability can depend on scene constraints away from the goal location.
- PGP distinction: PGP conditions on lane-graph paths, ensuring inferred goals are reachable and providing stronger route context than goal locations alone.The traversed path supplies an inductive bias that endpoint-only conditioning lacks.
- PGP distinction: Unlike centerline-goal methods requiring deterministic route selection, PGP probabilistically samples both routes and motion profiles.This supports different routes and different motion profiles along a route within the prediction set.
3 Formulation
The formulation represents agent histories and HD-map lane segments in an agent-centric bird’s-eye-view frame, then predicts K future trajectories as modes. Lane-graph nodes encode geometry and traffic-control elements, while graph paths constrain predictions to legal routes.
- 3 Formulation: The model predicts target-vehicle futures from agent histories, nearby agents and pedestrians, and the HD map in an agent-centric bird’s-eye-view frame.The frame is aligned with the target agent’s instantaneous direction of motion.
- 3.1 Trajectory representation: Past trajectories are assumed to come from onboard detectors and multi-object trackers.This is an explicit input-data assumption of the formulation.
- 3.1 Trajectory representation: Each agent state represents BEV position, speed, acceleration, yaw rate, and a pedestrian-versus-vehicle indicator.The target vehicle is nominally indexed as 0 and prediction time as timestamp 0.
- 3.2 Representing HD maps as lane graphs: The HD map is represented as a directed graph whose lane-centerline snippets are nodes, with each snippet discretized into poses.The graph captures traffic-flow direction and legal routes, while fixed-length snippets standardize lane segments.
- 3.2 Representing HD maps as lane graphs: Node features encode pose geometry together with binary indicators for stop lines and crosswalks.Thus, lane nodes include both geometric and traffic-control information.
- 3.2 Representing HD maps as lane graphs: Graph edges are constrained so traversals correspond to legal vehicle routes, using successor connections and additional proximal connections for lane changes.Successor structure also represents branching and merging lanes.
- 3.3 Output representation: The output is a set of K future x-y trajectories, with each mode intended to represent a plausible route or motion profile.Modes are defined over a prediction horizon of tf time steps.
- 3.3 Output representation: PGP combines a graph encoder, a discrete-policy header for path sampling, and an attention decoder conditioned on traversed paths and a latent variable.The three modules are trained end-to-end.
4 Proposed Model
The proposed model combines a lane-graph encoder, a learned policy for sampling likely routes, and a traversal-conditioned decoder with latent variables for diverse motion profiles.
- Graph encoder: The graph encoder represents map and agent information as lane-graph node encodings, updates nodes with nearby-agent attention, and aggregates neighboring context using GNN layers.It experiments with graph convolution and graph attention while treating successor and proximal edges as bidirectional.
- Discrete policy for graph traversal: The policy header assigns probabilities to outgoing lane-graph edges, including end-state edges, so sampled roll-outs represent likely future routes.Edge scores use target motion plus local scene and agent context, then are normalized with a softmax; behavior cloning trains the policy from ground-truth visited edges.
- Decoding trajectories conditioned on traversals: Each trajectory prediction selectively aggregates map and agent context along a sampled node traversal, producing a route-specific context vector for decoding.Multi-head attention uses the target motion encoding as query and node features as keys and values.
- Decoding trajectories conditioned on traversals: A sampled continuous latent vector conditions predictions alongside the traversal context, allowing distinct motion profiles even for identical routes.The latent variable models longitudinal variation such as braking, acceleration, and different velocities or accelerations.
- Evaluation: The model is evaluated against state-of-the-art methods on the nuScenes prediction benchmark in Table 1.The supplied table passage identifies the comparison, while the reported evaluation context concerns the nuScenes benchmark.
- Decoding trajectories conditioned on traversals: The decoder outputs future locations, after which K-means clustering reduces potentially redundant samples to K final trajectory modes.Training uses winner-takes-all average displacement error and combines losses from the policy and decoder objectives in an end-to-end multi-task loss.
5 Experiments
Experiments on nuScenes evaluate standard prediction, off-road, and diversity metrics, alongside encoder and decoder ablations. Results show that combining lane-graph traversals with latent variables improves route and motion-profile diversity while maintaining scene compliance.
- Comparison to the state of the art: The model achieves state-of-the-art results on almost all nuScenes benchmark metrics, significantly outperforming P2T on MinADEK and MissRate while obtaining a comparable off-road rate.The authors interpret this as better coverage of trajectory-distribution modes with scene-compliant predictions.
- Encoder ablations: Adding proximal edges and agent-node attention improves all encoder metrics, while GCN layers are slightly worse and GAT layers perform on par with omitting GNN layers.The authors suggest that attention across the traversed path may make additional GNN layers redundant.
- Decoder ablations: Latent-only and traversal-only decoders both underperform the combined decoder on MinADE and MissRate because each captures only one major dimension of multimodal variation.Latent-only predictions usually follow one route, whereas traversal-only predictions span routes but lack motion-profile diversity.
- Decoder ablations: The Goals + LV decoder performs worse than Traversals + LV, especially on off-road rate, due to unreachable goals and inaccurate paths to otherwise correct goals.Lane-graph traversals constrain predictions to reachable paths rather than merely selecting goal locations.
- Qualitative comparison: Figure 3 qualitatively shows that the combined model produces scene-compliant trajectories across diverse routes and reallocates its K predictions toward longitudinal diversity when few routes are plausible.Other decoders respectively exhibit off-road trajectories, limited lateral diversity, limited longitudinal diversity, or spurious goals.
6 Conclusions
The paper concludes that conditioning on lane-graph traversals and sampled latent variables supports multimodal trajectory prediction. Together, these choices produce route-diverse, motion-profile-diverse, precise, scene-compliant predictions and state-of-the-art MinADE and MissRate results.
- The method conditions trajectories on discrete-policy lane-graph traversals and a sampled latent variable.
- Lane-graph traversals provide route diversity and precise, scene-compliant predictions with the lowest off-road rates.
- Sampled latent variables add diversity in motion profiles, including when predictions share the same traversal.
- Combining both conditioning mechanisms yields state-of-the-art MinADE and MissRate results.
A Implementation details
The implementation uses PyTorch and provides architectural, ablation, and training details.
- The model is implemented using PyTorch, with the paper providing details of its architecture, ablations, and training.
A.1 Map representation
The map is represented with lane snippets as graph nodes, using fixed spatial and pose resolutions chosen to balance informativeness against encoder complexity.
- A.1 Map representation: Lane centerlines are split into snippets of at most 20m and discretized at 1m resolution.Each snippet becomes one graph node.
- A.1 Map representation: The selected 20m node resolution and 1m pose resolution were chosen experimentally.
- A.1 Map representation: Finer lane-node resolution provides more informative inputs but increases the number of graph nodes and poses, raising encoder complexity.
A.2 GRU encoders
Agent and map-node features are embedded with small linear layers and processed by one-layer GRUs with 32-dimensional hidden states.
- A.2 GRU encoders: Agent and node features are embedded using linear layers of size 16 followed by leaky ReLU.
- A.2 GRU encoders: Both the agent and node encoders use depth-1 GRUs with hidden state dimension 32.
A.3 Agent-node attention
Agent-node attention updates node encodings by projecting agent and node representations, applying single-head scaled dot-product attention, and combining the result with original node features.
- A.3 Agent-node attention: Agent-node attention uses scaled dot-product attention with a single attention head.
- A.3 Agent-node attention: The model uses 32 × 32 matrices to project node and agent encodings into queries, keys, and values.
- A.3 Agent-node attention: Attention outputs are concatenated with original node encodings and passed through a size-32 linear layer with leaky ReLU.
A.4 GNN layers
The model uses graph layers and policy or decoder components to generate trajectories from lane-graph context, sampled traversals, and latent variables, with staged training.
- A.4 GNN layers: The graph adjacency includes bidirectional successor and proximal edges plus self-loops, while GCN and GAT outputs remain 32-dimensional.
- A.4 GNN layers: The policy header scores each graph edge with a two-hidden-layer, size-32 MLP producing a scalar output.
- A.4 GNN layers: The trajectory decoder aggregates traversed-node context, samples a 5-dimensional standard-normal latent variable, and outputs 24 trajectory coordinates.The coordinates cover x and y over six seconds at 2 Hz.
- A.4 GNN layers: The complete model is pretrained with ground-truth traversals for 100 epochs and then finetuned with policy-sampled paths for 100 epochs.
- A.4 GNN layers: The MTP baseline aggregates the entire graph and predicts 10 trajectories with their probabilities.
- A.4 GNN layers: The LV-only decoder conditions trajectories on whole-graph context, motion features, and sampled latent variables.
- A.4 GNN layers: The traversal-only decoder conditions trajectories on traversed-path context and motion features without a sampled latent variable.
- A.4 GNN layers: The Goals + LV decoder combines node-level goal probabilities with goal-conditioned trajectories using context, motion, and latent features.