Source-linked AI summary
Learning to Predict Vehicle Trajectories with Model-based Planning
Haoran Song, Di Luan, Wenchao Ding, Michael Yu Wang, Qifeng Chen
TL;DR
Autonomous-driving prediction needs multimodal vehicle trajectories that remain accurate while respecting physical and environmental constraints. PRIME combines a constrained model-based generator with a learning-based evaluator, and outperforms state-of-the-art methods in accuracy, feasibility, and robustness. Its deployment scope remains bounded by tracking quality and fixed generator parameters.
Problem
Vehicle trajectory prediction must model multimodal futures for planning, but many learning-based methods generate trajectories without explicit kinematic and environmental constraints.
Method
PRIME uses a model-based generator to produce feasible trajectories under explicit constraints and a learning-based evaluator to model interactions and select predictions.
Results
PRIME outperforms state-of-the-art methods in prediction accuracy, feasibility, and robustness, including the best reported performance on Argoverse Miss Rate (K = 6).
Takeaways & Limitations
By regularizing trajectory space with continuous-state, constraint-compatible predictions, PRIME is presented as useful for real-system deployment and downstream motion planning.
Takeaways & Limitations
The generator uses fixed parameters, while serious tracking noise can make target heading and velocity estimation difficult and leave ground truth outside the generated prediction space.
Abstract
from arXiv · showhide
Predicting the future trajectories of on-road vehicles is critical for autonomous driving. In this paper, we introduce a novel prediction framework called PRIME, which stands for Prediction with Model-based Planning. Unlike recent prediction works that utilize neural networks to model scene context and produce unconstrained trajectories, PRIME is designed to generate accurate and feasibility-guaranteed future trajectory predictions. PRIME guarantees the trajectory feasibility by exploiting a model-based generator to produce future trajectories under explicit constraints and enables accurate multimodal prediction by utilizing a learning-based evaluator to select future trajectories. We conduct experiments on the large-scale Argoverse Motion Forecasting Benchmark, where PRIME outperforms the state-of-the-art methods in prediction accuracy, feasibility, and robustness under imperfect tracking.
1 Introduction
Trajectory prediction must support safe autonomous-driving planning by modeling multimodal futures while respecting vehicle, road, and semantic constraints. PRIME combines constrained model-based generation with learned evaluation to improve accuracy, feasibility, and robustness.
- Accurate and reasonable vehicle trajectory predictions provide future states for safe, efficient, and comfortable self-driving motion planning.
- Existing learning-based methods improve accuracy but commonly generate point-based trajectories without explicit kinematic or environmental constraints.
- PRIME uses a model-based generator for feasibility-guaranteed trajectories and a neural evaluator for multimodal selection through learned scene interactions.
- PRIME outperforms state-of-the-art methods in prediction accuracy and also improves trajectory feasibility and robustness under imperfect tracking.
2 Related Work
Prior prediction methods either use learning-based representations and multimodal generation without guaranteed constraint compliance or couple prediction with planning signals. PRIME instead combines model-based feasibility guarantees with learned interaction modeling.
- Prediction and planning share trajectory reasoning but differ because planning selects constraint-compliant ego trajectories while prediction infers surrounding vehicles’ probable futures.
- Existing planning-inspired prediction methods infer goals or condition trajectories on ego intentions and motion plans, but do not ensure imposed constraints during generation.
- Rasterized and vectorized representations encode scene context, while PRIME uses a hierarchical structure combining local path-conditioned generation with global scene aggregation.
- PRIME addresses stochastic inference concerns by using a model-based planner to provide feasible trajectory sets that a learning-based component evaluates.
3 Overview
PRIME is a two-stage architecture: a constrained model-based generator constructs a feasible trajectory space, and a learning-based evaluator scores and selects multimodal predictions from it.
- The prediction objective is to produce K multimodal trajectories with probabilities while ensuring every trajectory satisfies environmental and target-vehicle kinematic constraints.
- The generator maps the target state, HD map, and constraints to reachable paths and a finite set of feasible future trajectories.
- The evaluator receives paths, feasible trajectories, and observed tracks, then learns implicit interactions across varying numbers of paths, agents, and trajectories.
- The evaluator scores feasible trajectories and selects the final predicted set together with their probabilities.
4 Model-based Generator
The model-based generator searches map-reachable paths, samples longitudinal and lateral motions in a Frenét frame, and filters trajectories against environmental and kinematic constraints. The resulting feasible trajectory space is passed to PRIME’s evaluator.
- Path Search: Potential paths are searched on the HD map, then unreachable paths are removed during constrained trajectory generation to retain reachable paths.
- Trajectory Generation: PRIME adapts a planning-style Frenét generator to prediction by sampling many trajectories for targets with unknown intentions.
- Trajectory Generation: The Frenét representation expresses Cartesian position using path arc length s and perpendicular offset d relative to a reference path.
- Trajectory Generation: Longitudinal and lateral motions are generated independently with polynomial curves, combined across sampled endpoints, and projected into global coordinates.
- Constraint Filtering: Candidate trajectories are filtered for static-obstacle collisions and violations of velocity, acceleration, or curvature constraints before forming the feasible trajectory space.
- Framework Integration: The framework overview connects map-based path search and feasible trajectory generation with learned interaction encoding and final trajectory ranking.
5 Learning-based Evaluator
The learning-based evaluator encodes tracks, paths, and candidate futures to model scene interactions, score feasible trajectories, and select diverse multimodal predictions.
- The evaluator encodes observed tracks, reference paths, and candidate future trajectories as discretized sequences before modeling their interactions.
- Four attention modules—P2T, P2F, A2A, and F2F—fuse spatial-temporal information across paths, agents, and futures.
- F2F aggregates future encodings across paths to characterize reachable-space differences between candidate trajectories.
- The evaluator uses a maximum entropy model to score all candidate trajectories from their full scene-context descriptions.
- Training applies cross entropy between predicted and distance-based trajectory scores, while inference uses NMS to retain K high-scoring, non-duplicate trajectories.
6 Experiments
Experiments evaluate PRIME on Argoverse using accuracy, multimodal probability, feasibility, ablation, and robustness analyses against baselines and state-of-the-art methods.
- Argoverse contains over 324K urban-driving sequences, with 2 seconds of observation used to predict a target vehicle’s locations 3 seconds ahead.
- Evaluation uses minADEK, minFDEK, MRK, p-minADEK, and p-minFDEK, with MR6 as the official benchmark ranking metric.
- PRIME outperforms the compared methods on Miss Rate at K = 6 and achieves the best p-minFDE6 on the Argoverse test set.
- Ablations show that both Frenét representation and F2F improve performance, with Frenét representation contributing more; the complete network uses 1.02M parameters.
- The feasibility analysis evaluates curvature because higher-order vehicle states cannot be accurately estimated from discrete predicted locations.
7 Conclusion
PRIME combines model-based planning with learning-based evaluation to produce feasible, accurate, and robust vehicle trajectory predictions. Its constrained trajectory space and continuous-state predictions support potential real-system deployment.
- PRIME uses a model-based generator under explicit constraints and a learning-based evaluator that scores trajectories to capture implicit scene interactions.
- The framework is reported to outperform state-of-the-art methods in prediction accuracy, feasibility, and robustness.
- The conclusion identifies regularized trajectory space, continuous-state prediction, and compatibility with on-road information as useful for real-system deployment.
A Implementation Details
Implementation estimates the target vehicle state, searches reachable HD-map paths, generates constrained trajectories, and evaluates them using discretized scene representations.
- State Estimation: Kalman filtering processes noisy track data to estimate the target vehicle’s current velocity and heading before generation.
- Path Search: Depth-First Search explores predecessor and successor lane segments on the HD map to construct potential reachable paths.
- Trajectory Generation: The trajectory generator connects the initial state to sampled longitudinal and lateral end states using parametric curves under explicit limits.
- Trajectory Generation: Longitudinal sampling uses 35 velocity targets, while lateral sampling uses 9 offsets across a fixed 5-meter lane width.
- Learning-based Evaluator: The evaluator encodes history tracks, paths, and future trajectories at fixed temporal and spatial intervals before scoring candidate futures.
- Learning-based Evaluator: Qualitative visualizations show blue feasible trajectory sets generated under environmental and dynamic constraints, with six red evaluator-selected predictions.
B.1 Results Under Diverse Traffic Scenarios
PRIME generates feasible, multimodal trajectories by enforcing kinematic and environmental constraints, producing realistic predictions across diverse traffic scenarios. Qualitative results also show robustness to defective ground truth, although failures remain under inaccurate state estimation and high-speed motion.
- Diverse traffic scenarios: PRIME’s constrained trajectory space covers diverse speeds, maneuvers, and road scenarios while the evaluator selects accurate multimodal predictions.The examples include overtaking, braking, lane changing, turning, U-turns, straight roads, T-junctions, and crossroads.
- Constraint effects: Kinematic constraints prevent sudden high-speed turns and reversed motion, while environmental constraints prevent crossing lane boundaries or entering reverse lanes.These cases contrast PRIME’s constrained predictions with LaneGCN’s infeasible trajectories.
- Downstream planning: Comparable minADE6 and minFDE6 can conceal different planning impacts because infeasible predictions increase uncertainty and burden downstream decision-making.The negative impact is especially aggravated when many surrounding vehicles must be predicted in dense traffic.
- Defective ground truth: Defective Argoverse ground truth, including position oscillation and identity switches, can worsen ADE/FDE metrics even when PRIME’s smooth predictions are more realistic.The annotation defects arise from tracking issues and can make metric-based evaluation misrepresent qualitative prediction quality.
- Failure cases: Failures mainly arise from inaccurate current-state estimation and the larger future prediction space in high-speed scenarios.Noisy centroid histories can make heading and velocity estimation difficult, while high-speed motion increases the chance of missed predictions.
C Runtime Analysis
PRIME’s runtime depends on scene complexity, sampling density, and computing power, while its independently sampled generator and lightweight evaluator support real-time operation.
- Runtime: The generator takes 0.1 ∼0.2 ms per trajectory on one Intel i7-7820X thread, and its independent samples enable high parallelism.Parallel generation is intended to provide broad future-space coverage with satisfactory real-time performance.
- Runtime: The evaluator has 1.02 million parameters and runs in 8 ∼12 ms on an NVIDIA 2080TI, while the full framework satisfies autonomous-driving real-time requirements.Overall inference frequency still depends on scene complexity, sampling density, and computing power.
D Limitation and Future Work
The framework’s future improvements target fixed generator parameters, computational cost, and richer scene encoding while preserving the lightweight evaluator design.
- Model-based generator: Future work will adapt path-search thresholds and pruning to target state and lane connectivity, and adjust trajectory sampling to speed limits, lane width, and target impact.These changes are intended to alleviate computational cost in the model-based generator.
- Learning-based evaluator: The evaluator remains lightweight at 1.02M parameters, leaving room for network optimization and extending scene encoding from reachable paths to a lane graph.The proposed lane-graph extension is intended to provide more complete contextual representation.