Trajectory Prediction for Autonomous Driving: Progress, Limitations, and Future Directions

1. Introduction

• Trajectory prediction supports collision-avoiding planning, yet must capture heterogeneous agents, interactions, behavioral randomness, traffic-rule violations, multimodal futures, and scene context under computational constraints.These challenges affect vehicles, cyclists, pedestrians, and motorbikes, with pedestrian forecasting additionally requiring social norms, destination intent, and contextual cues.Source: §1 · pp. 1–2
• The review frames its agenda around identifying existing trajectory-prediction solutions, assessing whether the problem remains unsolved, and recognizing emerging research trends.These questions target modeling approaches, limitations affecting real-world deployment, research gaps, and challenges currently guiding development.Source: §1 · pp. 2–3
• The review surveys prediction pipelines, extends existing taxonomies, examines active research areas, and releases pipelines for experimentation and benchmarking.Its scope includes input and output modalities, modeling features, evaluation metrics, prediction paradigms, uncertainty, driving knowledge, planning-conditioned prediction, VLM guidance, and vehicle collaboration.Source: §1 · p. 3

1.1. Related Surveys

• Earlier surveys classify methods by representations, modeling strategies, outputs, sensors, pedestrian behaviors, driving knowledge, uncertainty, and interaction awareness, while identifying gaps in robustness, observability, rules, and generalization.The reviewed literature spans vehicle and pedestrian prediction, including physics-based, learning-based, stochastic, planning-based, and mixed approaches.Source: §1.1 · pp. 4–6

1.2. Scope and Structure

• The paper is structured into pipeline and taxonomy foundations, performance and robustness topics, and active areas covering planning coupling, VLMs, collaboration, research gaps, and future directions.Sections 2–3 provide general knowledge, Sections 4–5 address driving knowledge and uncertainty, and Sections 6–9 cover active topics and conclusions.Source: §1.1 · p. 6, §1.2 · p. 6

• The overview starts with trajectory prediction problem formulation and input modalities, then discusses modeling features and approaches.Source: §2 · p. 6
• It surveys possible output modalities and the metrics used to evaluate trajectory forecasting models.Source: §2 · p. 6
• The section lists benchmark datasets and leaderboard solutions before concluding with existing trajectory prediction paradigms.Source: §2 · p. 6

Problem formulation

• A traffic scenario contains an ego vehicle and agents whose trajectories are represented as time-indexed state vectors, minimally encoding positions (x, y).The observation length is t, the state dimension is m, and the minimal representation uses m = 2.Source: §2.1 · p. 6
• The predictor estimates each agent’s future trajectory over horizon ∆t from historical traffic-agent states, optionally incorporating orientation, speed, road geometry, and interaction features.The predictor can therefore depend on past trajectories T together with additional inputs such as G and I.Source: §2.1 · p. 6

Coordinate systems

• Local coordinates avoid transformations but require ego-motion compensation, whereas global coordinates require ego-pose information and precise localization, as in.Without compensation, stationary vehicles may appear to move and turning maneuvers may be distorted in the ego-vehicle’s local frame.Source: §2.1 · pp. 6–7
• Frenet coordinates represent motion as longitudinal arc length s and lateral offset d relative to the lane centerline, improving behavior learning across structurally different road layouts.Cartesian coordinates remain tied to underlying road geometry, which can hinder cross-scene generalization.Source: §2.1 · p. 7

2.2. Input Modalities

• Input modalities comprise past agent trajectories, LiDAR, radar, cameras, and HD maps, all providing information from which traffic dynamics can be extracted.Modalities may be used individually or combined according to sensing capabilities, operational constraints, and design choices.Source: §2.2 · p. 7

2.2.1. Past Trajectories

• Past-trajectory models are computationally efficient and can capture interactions by incorporating surrounding agents, but noisy or incomplete observations can degrade real-world performance.Robustness should be evaluated under conditions such as simulated tracker errors; velocity, acceleration, and orientation can supplement coordinate histories.Source: §2.2.1 · pp. 7–8

2.2.2. Sensors Data

• LiDAR provides reliable geometric perception across lighting conditions and supports obstacle, vehicle, lane-marker, and forecasting tasks, but rain, snow, reflective surfaces, cost, and pedestrian-recognition limitations constrain adoption.On KITTI, the top pedestrian detection score is 56%, compared with 93% vehicle detection by.Source: §2.2.2 · p. 8
• Cameras provide semantic perception for traffic participants, signs, and lights; monocular systems are lightweight and inexpensive, whereas stereo cameras enable 3D perception through depth estimation.RGB performance degrades in low light and adverse weather, motivating complementary thermal or infrared sensing.Source: §2.2.2 · pp. 8–9
• Radar estimates target velocity, distance, and direction, providing motion cues that commonly complement LiDAR and camera inputs in state-of-the-art datasets.FMCW radar derives distance from beat frequency, velocity from Doppler shift, and direction using beam-forming or multi-antenna setups.Source: §2.2.2 · p. 9
• Sensor fusion is facilitated by rasterizing LiDAR, radar, and camera data into unified bird’s-eye-view representations containing geometry, dynamics, semantics, and temporal information.BEV representations align positions and velocities with the ego vehicle and can encode motion using stacked frames or brightness variations.Source: §2.2.2 · p. 9

2.2.3. HD Maps

• HD maps encode static features such as lanes, crossings, layouts, and signs within locally extracted regions, but require precise localization, costly updates, and additional preprocessing.Combining maps with sensor data can address localization errors and environmental changes, although it affects inference time.Source: §2.2.3 · pp. 9–10
• Map information reduces displacement error from 1.90 to 1.35 meters at 6 seconds for LaneGCN, while VectorNet reports a 28% reduction at 3 seconds and QCNet improves from 0.82 to 0.78 meters.Benefits depend on map representation and integration with agent dynamics.Source: §2.2.3 · p. 10

2.3. Modelling Features

• Feature design spans static scene components and dynamic motion or interaction components, with longer histories and interaction modeling especially relevant for dense traffic and intersections.Semantic attributes may require annotated-data subnetworks or pretrained networks, creating risks of error propagation and added inference time; hand-crafted interaction features can miss real-world complexity, motivating social fields, game theory, graph methods, and social pooling.Source: §2.3 · p. 10, §2.3.2 · p. 11, §2.3.3 · p. 11

2.3.1. Environment Features

• Static environment features encode topology, boundaries, traffic controls, and semantic obstacles so forecasts better respect scene geometry.Examples include crossroads-boundary distances, traffic lights, stop signs, zebra crossings, intersection distances, and occupancy grids integrating predicted destinations.Source: §2.3.1 · p. 10

2.3.2. Target Agents

• Dynamic target-agent features include past trajectories, velocity, heading, orientation, object category, behavioral history, and visual posture or appearance cues.Category-specific dynamics can suffer from detector misclassification, while posture is commonly estimated from skeleton key-points or pretrained CNNs.Source: §2.3.2 · p. 11

2.3.3. Interaction Features

• Interaction-aware models improve socially acceptable forecasting, but their effectiveness depends on integration design and the prevalence of densely interactive scenarios; isolating social or temporal cues degrades AgentFormer performance.Social-LSTM and Social-GAN improve trajectories in crowded environments, while Social-STGCNN and Social-BiGAT improve accuracy and reduce inference latency through graph convolutions or attention.Source: §2.3.3 · p. 11

• Traditional models such as Constant-Velocity and Interactive Multiple Model explicitly encode motion constraints, producing physically plausible trajectories with simple implementation and interpretation.Source: §2.4 · pp. 11–12
• Sampling-based methods model future-trajectory distributions conditioned on observed paths, then generate diverse predictions through parametric distributions or learning-based latent spaces.Makansi et al. use modified EWTA sampling and refinement, while Gilles et al. produce interpretable likelihood maps for socially aware multi-agent sampling.Source: §2.4 · p. 12
• Diffusion models add noise and iteratively denoise trajectories conditioned on past motion and scene context, enabling diverse, multimodal, and controllable forecasts.MotionDiffuser supports collision-avoidance constraints, BeLFusion conditions on behavioral latents, and PhysDiff uses physics-guided denoising to remove motion artifacts.Source: §2.4 · p. 12
• Classical machine learning provides interpretable probabilistic modeling for simpler environments, whereas deep learning handles complex dynamics but depends strongly on diverse, high-quality data and greater computational cost.Reviewed deep-learning families include sequential, convolutional, graph, transformer, autoencoder, and adversarial models.Source: §2.4 · p. 12
• Self-supervised learning leverages unlabeled data and proxy objectives to learn transferable motion representations that improve forecasting generalization, often through multitask training.Contrastive objectives bring similar trajectories closer and dissimilar trajectories farther apart; the survey refers to [152] for detailed SSL coverage.Source: §2.4 · p. 12
• Interaction-aware models support socially compatible prediction in dense mixed traffic, while intention-aware models use lateral and longitudinal maneuvers to inform future states.Intention awareness introduces an intermediate planning-based reasoning layer between observed behavior and predicted states.Source: §2.4 · pp. 12–13

Object-centric outputs

• Object-centric models use unimodal or multimodal trajectory outputs, with multimodal distributions representing multiple plausible paths and their probabilities.Multimodal forecasting is important because driving behavior can remain ambiguous before an agent reveals its intent, affecting collision avoidance during intersections and lane merging.Source: §2.5 · p. 13

Maneuver outputs

• Maneuver-oriented models predict actions such as lane changes, lane keeping, straight travel, and turns, either jointly with trajectories through shared features or separately with maneuvers guiding trajectory prediction.The two joint-prediction strategies are multi-task modeling and sequential maneuver-then-trajectory prediction.Source: §2.5 · p. 13

Occupancy-centric outputs

• Occupancy-centric models predict future cell occupancy, while occupancy flow additionally predicts motion dynamics through joint occupancy and flow fields.Grid resolution trades precision against computational cost, motivating finer grids in dense urban traffic and coarser grids on highways; these methods learn directly from frame sequences without tracking.Source: §2.5 · p. 13

• Open-loop evaluation directly compares predicted trajectories with ground-truth future paths, without measuring effects on downstream modules.Source: §2.6 · p. 13
• RMSE, ADE, FDE, and NLL evaluate trajectory errors or probabilistic likelihood, while MinADE_k and MinFDE_k extend displacement metrics to multimodal predictions.Source: §2.6 · p. 13
• When object identifiers are unavailable, predicted and ground-truth objects require matching before metric computation; Miss Rate and inference time should also be reported.Matching typically uses IoU thresholds or nearest-centroid distance, while Miss Rate captures agents missed by the perception-prediction pipeline and inference time reflects real-time suitability.Source: §2.6 · pp. 13–14
• Closed-loop evaluation embeds prediction within the autonomy stack, measuring downstream safety effects, unnecessary rerouting, and conservatism that can cause unnecessary stopping.Open- and closed-loop evaluation are further discussed in [152].Source: §2.6 · p. 14
• Cross-dataset evaluation and deployment require sampling-rate adjustment through downsampling, resampling, or temporal alignment, while continuous stochastic processes can avoid discretization challenges.Source: §2.6 · p. 14

Dataset landscape

• Benchmark datasets cover varied environments, traffic densities, modalities, and locations, with Table 2 summarizing collection modalities, descriptions, and leaderboard solutions.Scenarios include intersections, highways, urban streets, public spaces, and residential areas; some dataset details require consultation of official websites.Source: §2.7 · p. 14, Table 2 · p. 15
• KITTI lacks an official trajectory-prediction split, motivating a standardized 6-second split with 8,613 training and 2,907 testing trajectories, plus top-view maps.KITTI-360 extends coverage across 70 kilometers with OpenStreetMap geolocation, supporting global-map-based navigation without HD-map granularity.Source: §2.7 · p. 14
• Daimler, Ko-PER Intersection, ApolloScape, PIE, and NuScenes provide complementary pedestrian, intersection, camera-LiDAR, urban-behavior, and multimodal traffic-participant data.Their settings range from sub-2-second pedestrian prediction and three intersection sequences to Beijing traffic, Toronto pedestrian behavior, and 1,000 NuScenes urban scenes with LiDAR, radar, cameras, and HD maps.Source: §2.7 · p. 14
• WAYMO, Argoverse I [68], Argoverse II, and Lyft Level 5 offer large-scale interactive benchmarks with long sequences, complex interactions, detailed annotations, and high-definition or semantic maps.WAYMO contains around 100,000 20-second scenes; Argoverse I and II contain 324,557 5-second and 250,000 11-second scenarios, while Lyft Level 5 contains 170,000 25-second scenes from over 1,000 hours of driving.Source: §2.7 · pp. 14–15

2.8. Prediction Paradigms

• The three paradigms are Detect-Track-Predict, Joint Learning, and Query-Based prediction, as summarized in Figure 6.They differ in how detection, tracking, scene encoding, and forecasting are integrated.Source: §2.8 · p. 15, Figure 6 · p. 16

2.8.1. Detect-Track-Predict

• Detect-Track-Predict sequentially detects objects, maintains tracklets, and forecasts trajectories, offering modular debugging but propagating errors and limiting end-to-end optimization.Robust tracklets are essential because prediction models rely on trajectories from the previous 10–20 timestamps; sensor fusion can improve perception but may increase inference time, such as BEVFusion’s 190 ms detection speed.Source: §2.8.1 · pp. 15–17

2.8.2. Joint Learning Models

• Joint Learning models optimize detection and forecasting together, often without trackers, using pooled representations and compound losses, but their tight coupling complicates debugging and responsibility attribution.Integration ranges from shared BEV features in FIERY and shared gradient flow in PnPNet to co-optimization with limited gradient sharing in FaF.Source: §2.8.2 · p. 17

2.8.3. Query-Based Prediction Models

• Query-Based models use learnable spatial-temporal queries to jointly produce detections and forecasts, reducing subsystem error propagation while incurring computational and input-quality challenges.Examples include DenseTNT, UniAD, DeTra, and scene-centric BEV-query models; iterative refinement can hinder real-time deployment in dense scenarios, and reliability depends on scene encoding, HD maps, and LiDAR.Source: §2.8.3 · pp. 17–18

3.1. Taxonomy

• The taxonomy separates physics-based methods from learning-based methods, with the latter divided into classical machine learning, deep learning, and reinforcement learning approaches.Physics-based methods explicitly define motion equations; learning-based methods learn motion patterns from observations, including Gaussian Processes, Hidden Markov Models, Dynamic Bayesian Networks, Gaussian Mixture Models, sequential models, CNNs, GNNs, Transformers, autoencoders, GANs, and reinforcement-learning variants.Source: §3.1 · p. 18, Figure 8 · p. 19

3.2. Physics-based Methods

• Physics-based methods use kinematic or dynamic equations, Kalman filtering, Monte Carlo sampling, and multiple-model estimators to predict motion and represent uncertainty or mode variation.Kinematic models include CV, CA, CTRV, and CTRA formulations; Kalman filters model Gaussian state uncertainty, Monte Carlo methods avoid fixed distributional assumptions, and multiple-model methods combine predefined motion models.Source: §3.2 · pp. 18–19

3.2.1. Strengths and Limitations of Physics-based Models

• Physics-based models are computationally efficient for short-term prediction and abrupt maneuvers, but limited histories constrain flexibility, while Monte Carlo methods require substantial computation and careful tuning.Constant-motion models omit uncertainty, Kalman variants assume Gaussian noise, and hybrid approaches combine physical constraints with learning-based models.Source: §3.2.1 · pp. 19–20

3.3. Machine Learning-based Methods

• Classical machine-learning methods fit probabilistic function approximators, using GPs, HMMs, GMMs, and DBNs to model smooth trajectories, temporal states, multimodal futures, and agent interactions.GPs produce posterior means and variances, HMMs represent latent states and transitions, GMMs combine plausible paths with uncertainty, and DBNs model hidden variables across interacting agents.Source: §3.3 · p. 20, §3.3.1 · p. 20, §3.3.2 · p. 20

3.3.5. Strengths and Limitations of ML Methods

• ML-based methods provide interpretable structures, uncertainty estimates, and useful performance with limited data, but restricted parameterization, scalability, and manual feature design limit complex behavior modeling.DBNs in particular struggle as the number of agents or observations grows and cannot learn hierarchical representations as flexibly as deep-learning models.Source: §3.3.5 · pp. 22–23

3.4. Deep Learning-based Methods

• Deep-learning approaches learn motion dynamics across recurrent, convolutional, graph, transformer, autoencoder, and adversarial architectures, with model summaries organized by architecture.The surveyed architectures target temporal dependencies, spatial context, interactions, latent representations, and multimodal future trajectories.Source: §3.1 · p. 18, Figure 10 · p. 21, §3.4 · p. 23

3.4.1. RNNs and LSTMs

• RNNs and LSTMs model causal temporal dependencies through hidden states, while extensions combine recurrent groups, attention, context reasoning, maneuver conditioning, and probabilistic multimodal decoding.Representative designs separately model individual motion and interactions, use encoder-decoder structures for GMM parameters or occupancy, and employ maneuver-specific decoders.Source: §3.4.1 · p. 23

3.4.2. CNNs

• CNN-based methods extract spatial context from images or model trajectory sequences with temporal convolutions, while ConvLSTMs integrate spatial and temporal processing for interaction-aware prediction.TCNs use causal and dilated convolutions for parallel, long-range sequence modeling; ConvLSTM systems process localized interactions, risk maps, and past motion jointly.Source: §3.4.2 · pp. 23–24

3.4.3. Autoencoders

• Autoencoder-based methods encode multimodal state information and use stochastic latent variables, MPC, or stacked autoencoders to generate trajectory distributions and handle irrational behavior.CVAE models integrate ego and neighboring states with latent variable z; other frameworks sample joint futures, reweight them toward rational behavior, or decode maneuver-conditioned distributions.Source: §3.4.3 · p. 25

3.4.4. GANs

• GANs use adversarial generator-discriminator training to produce diverse, realistic trajectories conditioned on motion history, social interactions, physical constraints, and scene context.Social pooling, attention, CNN/LSTM encoders, and conditional generation are used to model interactions, avoid mode collapse, and enforce physical or social plausibility.Source: §3.4.4 · pp. 25–26

3.4.5. GNNs

• GNNs represent agents as graph nodes and interactions as edges, using static or dynamic graphs with GCNs, GATs, TGNs, temporal modules, and multimodal decoders.Dynamic graphs adapt to changing agent counts, whereas static graphs use fixed nodes and masking; TGN-based temporal memory requires consistent object IDs, otherwise re-identification or retrieval is needed.Source: §3.4.5 · pp. 26–27

3.4.6. Attention and Transformers

• Attention mechanisms weight influential agents and historical states, while Transformers unify spatial-temporal dependencies through self-attention, positional encodings, cross-attention, and hierarchical or graph-based processing.Reviewed Transformer designs combine motion, map, and social information; hierarchical models separate local context extraction from global interaction modeling to improve computational efficiency.Source: §3.4.6 · pp. 30–31

3.4.7. Strengths and Limitations of DL Methods

• Deep learning architectures trade off temporal, spatial, interaction, uncertainty, and long-range modeling: LSTMs lack spatial reasoning, CNNs enable fast spatial processing, VAEs support multimodality, and GNNs add interaction-aware complexity.GANs require substantial data and stable training, may generalize poorly under distribution shift, dynamic GNNs depend on tracking quality, and Transformers require stable training, velocity inputs, tuning, and warm-up.Source: §3.4.7 · p. 31

3.5. Reinforcement Learning-based Methods

• Reinforcement learning treats trajectory prediction as policy optimization, with inverse reinforcement learning inferring rewards from demonstrations and imitation learning directly learning policies from expert trajectories.IRL includes maximum-margin, maximum-entropy, hierarchical, and Transformer-integrated approaches for socially aware, risk-aware, or decision-conditioned prediction.Source: §3.5 · p. 32, §3.5.1 · pp. 32–33

3.5.2. Imitation Learning

• Imitation learning bypasses explicit reward estimation and models expert-like futures using latent-variable policies, probabilistic dynamics, GAIL, recurrent policies, latent decisions, and POMDPs.These methods represent uncertainty and multimodality while incorporating spatial reasoning, temporal driver behavior, latent intent, and partial observability.Source: §3.5.2 · p. 33

3.5.3. Strengths and Limitations of RL Methods

• RL methods can reproduce realistic multimodal behavior under partial observability, but require extensive expert data, are computationally demanding, and face difficult reward formalization and interpretability problems.Learned rewards must capture safety, social dynamics, and ambiguous traffic-agent intent, while reward and policy representations can be difficult to debug.Source: §3.5.3 · p. 33

4. Incorporating Driving Knowledge

• Driving knowledge improves trajectory forecasting by constraining plausible motion through road topology, lane directions, traffic rules, physical limits, behavioral priors, and regional driving styles.Modular knowledge can adapt to deployment regions and improve generalizability.Source: §4 · pp. 33–34

4.1. Types of Driving Knowledge

• The taxonomy distinguishes map-related information, agent-related kinematics and mobility constraints, and traffic-related rules and interaction patterns.Map knowledge covers static infrastructure; agent knowledge covers vehicle and pedestrian behavior; traffic knowledge covers road-user interactions.Source: §4.1 · p. 34

4.1.1. Map-related knowledge

• Map encoders preserve structured road geometry and semantics using GNNs; VectorNet and LaneGCN model vectorized map elements, lane topology, and actor-map interactions.These representations avoid information loss associated with rasterized map grids and support interaction-aware prediction features.Source: §4.1.1 · p. 34

4.1.2. Agent-related knowledge

• Agent-related knowledge constrains trajectories using vehicle kinematics and behavioral priors, including turning radius, braking distance, acceleration capacity, and driving-style tendencies.Historical trajectories can identify patterns such as aggressive overtaking or forcing trailing vehicles to brake.Source: §4.1.2 · p. 34

4.1.3. Tra

• Traffic-related knowledge encodes rules and interactions, enabling models to represent stopping, yielding, right-of-way, follow, and conflict-free behaviors.Implicit social norms, such as drivers signaling pedestrians to pass, can improve prediction in heterogeneous dense traffic.Source: §4.1.3 · p. 34

4.2. Injecting Driving Knowledge

• Models inject driving knowledge through kinematic constraints, cost functions and auxiliary losses, STL rules, and map-based priors.These mechanisms enforce dynamics, collision and road compliance, lane directionality, rule satisfaction, feasible maneuvers, and lane-aware context.Source: §4.2 · p. 35

5. Uncertainty Estimation and Mitigation

• Because uncertainty must be modeled throughout the autonomy stack, the section synthesizes its sources, quantification techniques, and mitigation methods for safer downstream planning.The paper recommends Yang et al. and Wang et al. [152] for broader treatment across the full autonomy pipeline.Source: §5 · p. 35

5.1. Sources of Uncertainty

• Uncertainty arises from environmental ambiguity, model limitations, sensor noise, multimodal behavior, domain gaps, and errors propagated from perception and tracking.External uncertainty includes occlusions, adverse conditions, rule violations, and rare events, while internal uncertainty includes poor generalization, biased data, limited capacity, and cascading module errors.Source: §5.1 · p. 36, Figure 11 · p. 36

5.2. Mitigating Uncertainty

• Mitigation strategies address distribution shifts, OOD inputs, incomplete observations, causal confounding, and cascaded errors through testing, detection, physics-based recovery, disentanglement, and joint modeling.Laurent et al. propose structured scenario-coverage testing, while OOD detection can use label-based, Bayesian, or generative methods; other methods recover missing histories, generate diverse hypotheses, predict out-of-sight agents, separate causal factors, or jointly optimize detection, tracking, and forecasting.Source: §5.2 · p. 37

5.3. Quantifying Uncertainty

• Uncertainty quantification replaces single deterministic forecasts with multimodal trajectory distributions, commonly using Gaussian or Gaussian Mixture Models to represent possible futures and predictive confidence.These methods capture ambiguity in agent intentions by modeling both mean predictions and associated uncertainty; Table 15 summarizes reviewed deep learning-based uncertainty-quantification models.Source: §5.3 · p. 37, Table 15 · p. 37

6. Planning-conditioned Trajectory Prediction

• Planning-conditioned methods couple trajectory prediction with observed or inferred motion plans, intentions, destinations, or potential future paths.They extend beyond past motion, context, and interaction features by modeling agents as following planned trajectories or pursuing specific destinations.Source: §6 · p. 38

6. Planning Methods for Trajectory Prediction

• Planning-based prediction methods are categorized as model-based, learning-based, or combined, while anchor-based methods use goal or trajectory anchors.Goal anchors estimate tentative destinations; trajectory anchors construct possible paths using environmental and physical constraints.Source: §6 · p. 38, §6.1 · p. 38

6.1.1. Model-based planning approaches

• Model-based planning combines dynamics models, cost functions, and constrained optimization with behavioral, control, and road-structure information.These components produce interpretable and physically feasible trajectories under objectives such as energy, travel time, or collision-risk minimization.Source: §6.1.1 · p. 38

6.1.2. Learning-based planning approaches

• Learning-based planning uses data-driven, reinforcement, and inverse-reinforcement learning to infer strategic decisions, reactive behaviors, cost functions, and multi-agent interactions.Examples include maneuver-probability prediction, pedestrian cost-function learning, and joint modeling of goals and social interactions.Source: §6.1.2 · p. 38

6.2. Anchor-based Trajectory Prediction

• Anchor-based prediction encodes past trajectories and scene context, then decodes future trajectories conditioned on predefined or learned goal and trajectory anchors.These anchors represent potential destinations, paths, or waypoints within a coupled planning-prediction pipeline.Source: §6.2 · pp. 38–39

6.2.1. Goal Anchor-Based Prediction Methods

• Goal anchor-based methods predict feasible destinations from context, behavior, and static structure, then generate trajectories conditioned on those goals.Examples include multi-goal CVAE forecasting in BiTraP, multiscale goal estimation in SGNet, and latent endpoint sampling in PECNet.Source: §6.2.1 · p. 39

6.2.2. Trajectory Anchor-Based Prediction

• Trajectory anchor-based methods select or refine predefined or learned path templates, including polynomial, reachable, and hierarchically decomposed trajectories.PRIME combines planning-based path search with learning-based prediction while enforcing environmental and kinematic constraints.Source: §6.2.2 · pp. 39–40

6.3. Summary Discussion and Future Directions

• Model-based methods provide interpretability and constraint satisfaction but lack adaptability, whereas learning-based methods handle complex behavior yet may ignore physical limits; anchor generation also faces scalability costs.Future directions include hybrid methods such as PRIME, differentiable safety control, intent- and graph-based multi-agent planning, and efficient online anchor pruning as agent counts grow.Source: §6.3 · p. 40

7. Vision-Language Models for Prediction

• VLM prediction pipelines combine visual and textual encoders through cross-modal attention, producing trajectory forecasts informed by visual observations and semantic guidance.Inputs may include camera images, multi-camera feeds, LiDAR-range images, instructions, scene descriptions, object queries, or navigation commands.Source: §7 · p. 41, Figure 13 · p. 42

7.1. Trajectory Prediction

• Recent trajectory-prediction models use VLMs or LLMs for difficulty-ranked data selection, multimodal motion prediction, text-based scene representations, explainable lane-change forecasting, and global traffic-context understanding.These approaches integrate visual prompts, structured lane descriptions, human gazes and gestures, road conditions, textual guidance, and conventional prediction pipelines.Source: §7.1 · p. 41, Table 18 · p. 42

7.2. Traffic Scene Understanding

• Scene-understanding systems apply VLMs to intentions, spatial reasoning, query refinement, corner cases, localization, long-term video memory, and hierarchical planning for complex driving environments.Methods use components including DINO detection, DepthAnything depth estimation, retrieval, prompt optimization, space-aware pre-training, and time-aware token selection.Source: §7.2 · p. 41, Table 19 · p. 43

7.3. Benchmark Datasets

• Language-enhanced datasets extend autonomous-driving benchmarks with referring, action explanation, multi-object tracking, question answering, BEV reasoning, and decision-support tasks across varied driving conditions.Examples build on nuScenes, BDD100K, Waymo, and ONCE, supporting camera- and BEV-centered language interaction.Source: §7.3 · p. 42, Table 20 · p. 43

7.4. Summary and Discussion

• Integrating VLMs into autonomous driving remains unresolved because large-model inference, prompt design, and non-deterministic outputs affect latency, computation, interpretability, adaptability, and safety.Quantization, pruning, distillation, hardware or cloud deployment, few-shot and Chain-of-Thought prompting, temperature control, voting, and top-k averaging offer possible mitigations but involve trade-offs.Source: §7.4 · pp. 42–43

8. Collaboration between AVs for Prediction

• Collaborative perception aggregates multi-vehicle information to overcome sensor limitations and support detection, tracking, trajectory prediction, planning, and other downstream tasks.The review categorizes collaboration by fusion strategy rather than collaboration type and surveys models, datasets, limitations, and future directions.Source: §8 · pp. 43–44

8.1. Type of Fusion

• Early, intermediate, and late fusion differ in where information is integrated, with trade-offs among complementary context, information loss, bandwidth, communication overhead, noise, and incomplete predictions.Early fusion combines raw data, intermediate fusion shares deep semantic features, and late fusion aggregates independently generated decisions.Source: §8.1 · p. 44, Figure 14 · p. 45

8.2. Communication Redundancy

• Communication-redundancy research selects complementary information and compresses messages, including sparse high-confidence features, codebook representations, information filling, and compact object-level representations [643] [644].These methods aim to reduce message sizes while preserving essential information that complements ego-vehicle observations.Source: §8.2 · pp. 44–45

8.3. Transmission Delays

• Transmission delays complicate temporal and spatial alignment, prompting synchronization, temporal adjustment, feature-flow prediction, and motion-based warping methods [646].These approaches align exchanged information with the ego vehicle’s current timeframe despite latency and localization or transformation inaccuracies.Source: §8.3 · p. 45

8.4. Unified Fusion Space

• Unified fusion spaces address heterogeneous sensor suites and model architectures through feature resizing, domain adaptation, confidence calibration, backward alignment, domain-invariant distillation, and lightweight cooperative modules [623].These methods remove assumptions of uniform sensors or architectures and harmonize representations across collaborating agents.Source: §8.4 · p. 46

8.5. Benchmark Datasets

• Collaborative datasets span simulated and real-world settings, supporting perception, tracking, motion forecasting, and trajectory prediction across vehicle-to-vehicle and vehicle-to-infrastructure scenarios.Table 21 summarizes sensor types, agent counts, communication modes, frame counts, and annotation tasks across available datasets.Source: §8.5 · p. 46, Table 21 · p. 47

8.6. Collaborative Trajectory Prediction Models

• Collaborative trajectory-prediction models use intermediate, BEV-centered, graph-based, hybrid, and heterogeneous-graph fusion to combine observations, maps, and trajectories for future-motion prediction [657] [658] [659].Table 22 summarizes each model’s fusion type and evaluation dataset.Source: §8.6 · p. 46, Table 22 · p. 47

8.7. Summary and Discussion

• The field remains limited by perception-level bandwidth demands, fully connected-network assumptions, time sensitivity, missing inference-time analysis, unverified information, and insufficient standardized testing platforms [660].Task-specific fusion at the trajectory-prediction stage could use structured low-dimensional inputs more efficiently; practical systems also require synchronization, reliability checks, and broader platforms such as OpenCDA.Source: §8.7 · p. 47

9.1. On Research Question 1

• Existing solutions evolved from physics-based methods through machine learning and recurrent networks to generative, graph-based, attention-based, and hybrid architectures.Current systems commonly combine CNNs, attention, encoder–decoders, graph-enhanced Transformers, CNN–LSTM pipelines, and physics-based constraints.Source: §9.1 · pp. 47–48

9.2. On Research Question 2

• Top-model improvements are diminishing: Waymo’s top two solutions differ by only 0.01 minADE, while averaged accuracy can conceal severe edge-case failures.The cited comparison is MTR_v3: 0.55 versus ModeSeq: 0.56; deployment readiness also requires qualitative failure analysis, consistency mechanisms, and cross-dataset testing.Source: §9.2 · p. 48

9.3. On Research Question 3

• Emerging research prioritizes robustness and generalization through uncertainty modeling, collaborative perception, planning-conditioned prediction, and contextual reasoning with VLMs.Planning-conditioned methods provide interpretable, constraint-compliant predictions, whereas VLM deployment still requires investigation of latency, predictability, and cloud-hosting configurations.Source: §9.3 · p. 48

9.4. Future Directions

• Autonomous-driving predictors should balance accuracy and speed under a 100 ms safe-response limit, with inference-time testing and hardware-aware model selection.Parallelizable or hardware-accelerated models can provide faster updates and error correction, but deployment feasibility on edge devices remains under-evaluated.Source: §9.4 · pp. 48–49
• Standardized cross-dataset benchmarking should characterize road geometry, interaction density, rare maneuvers, motion patterns, and contextual inputs to evaluate transfer and failure cases.Existing evaluations usually rely on ADE and FDE on isolated datasets, while unified scenario parametrization and rare-behavior sampling remain limited.Source: §9.4 · p. 49
• Prediction must become robust to noisy perception and support integrated detect-track-predict, prediction-informed perception, and prediction-informed tracking pipelines.Most models assume clean trajectories despite detection and tracking errors propagating into prediction during deployment.Source: §9.4 · p. 49
• Future models should support multiple horizons, out-of-distribution adaptation, heterogeneous agents, and group-aware pedestrian forecasting through specialized or hierarchical designs.Short horizons support abrupt-event safety, long horizons support strategic planning, while Sim2Real, continual learning, category priors, and crowd-level modeling address broader generalization needs.Source: §9.4 · pp. 49–50
• Tighter integration with planning and collaboration requires uncertainty-aware multimodal outputs, credible shared information, efficient V2X protocols, and real-world validation.These directions support collision avoidance, task-specific fusion, communication scalability, and proactive safety interventions across autonomous vehicles, ADAS, and traffic-management systems.Source: §9.4 · p. 50

• The review classifies trajectory-prediction methods into non-learning and learning-based approaches, including machine learning, deep learning, and reinforcement learning.It also covers input and output modalities, benchmark datasets, and evaluation metrics.Source: §10 · p. 50
• Robustness and generalizability require uncertainty mitigation, cross-dataset evaluation, and qualitative assessment across diverse scenarios.Collaborative perception can address occlusions, sensor noise, and restricted sensing capabilities affecting individual perception.Source: §10 · pp. 50–51
• Emerging directions include planning-conditioned prediction, goal-anchor methods for interpretable destinations, and VLM integration for contextual reasoning in uncertain conditions.VLM-based systems still require investigation of latency and scalability.Source: §10 · p. 51
• The field has progressed from physics-based models to Graph Transformers, while physics-based hybrid methods seek to balance interpretability with predictive capacity.The review addresses its research questions and identifies trajectory forecasting as an open area requiring further exploration.Source: §10 · p. 51