PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Learning-based Vehicle Behaviour Prediction For Autonomous Driving Applications: A Review

Sajjad Mozaffari, Omar Y. Al-Jarrah, Mehrdad Dianati, Paul Jennings, Alexandros Mouzakitis

arXiv:1912.11676v2cs.CVcs.LGcs.ROstat.ML

TL;DR

Vehicle behaviour prediction must account for interactions, environmental conditions, and multiple possible futures. This paper reviews and classifies deep learning approaches to the problem. The reviewed methods often outperform constant-velocity baselines, while key adoption challenges remain.

  • Problem

    Vehicle prediction must account for interactions among vehicles, environmental and traffic-rule effects, and multiple plausible futures from the same motion history.

    Source: A. Challenges · p. 2

  • Method

    The paper reviews deep learning-based prediction studies and classifies their input representations, including vehicle track histories and bird’s-eye-view or raw sensor data.

    Source: §III · p. 3, A. Input Representation · p. 3

  • Results

    Most reviewed deep learning methods surpass the constant-velocity baseline, but multimodal trajectory prediction does not always yield lower RMSE than unimodal prediction.

    Source: B. Performance of Existing Methods · p. 12

  • Takeaways & Limitations

    The review identifies environment conditions, traffic rules, sensor impairments, and limited onboard computation as challenges not fully addressed by existing solutions.

    Source: §V · p. 12

    Practical adoption remains constrained by sensor impairments and limited computational resources, which existing solutions have not fully taken into account.

    Source: §V · p. 12

Abstract

from arXiv · show

Behaviour prediction function of an autonomous vehicle predicts the future states of the nearby vehicles based on the current and past observations of the surrounding environment. This helps enhance their awareness of the imminent hazards. However, conventional behaviour prediction solutions are applicable in simple driving scenarios that require short prediction horizons. Most recently, deep learning-based approaches have become popular due to their superior performance in more complex environments compared to the conventional approaches. Motivated by this increased popularity, we provide a comprehensive review of the state-of-the-art of deep learning-based approaches for vehicle behaviour prediction in this paper. We firstly give an overview of the generic problem of vehicle behaviour prediction and discuss its challenges, followed by classification and review of the most recent deep learning-based solutions based on three criteria: input representation, output type, and prediction method. The paper also discusses the performance of several well-known solutions, identifies the research gaps in the literature and outlines potential new research directions.

I. INTRODUCTION

Vehicle behaviour prediction must anticipate interdependent, multimodal futures despite environmental and sensing constraints. This review addresses a gap in systematic comparative coverage of deep learning approaches and outlines their challenges and research directions.

I. INTRODUCTION

  • The review addresses the lack of a systematic comparative review of deep learning-based vehicle behaviour prediction, classifying and comparing approaches and identifying research gaps.It also discusses practical implementation limitations and potential research directions.Source: §I · p. 1

II. BASICS AND CHALLENGES OF VEHICLE BEHAVIOUR PREDICTION

  • The review introduces the prediction problem and its challenges, then compares deep learning solutions and discusses evaluation, research gaps, and future directions.It also provides a generic problem definition and terminology.Source: §I · p. 1, §II · p. 2

A. Challenges

  • Vehicle prediction is challenging because surrounding vehicles interact, traffic rules and road geometry shape behaviour, and one motion history can have multiple possible futures.Reliable prediction should identify possible future motions.Source: A. Challenges · p. 2
  • Onboard sensing is partial, while many studies assume unobstructed infrastructure views that require sufficient-capacity communication and cannot be assumed across all road sections.Unobservable vehicles can contribute to inaccurate predictions, including when a hidden vehicle changes lanes ahead of the target vehicle.Source: A. Challenges · p. 2, Figure 1 · p. 2

B. Terminology

The paper defines vehicle roles and formulates prediction probabilistically: future states of target vehicles are inferred from observations available to the ego vehicle.

B. Terminology

  • A target vehicle (TV) is the vehicle whose behaviour is predicted; the ego vehicle (EV) observes the surroundings, while surrounding vehicles (SVs) may affect the TV and non-effective vehicles (NVs) are assumed not to.The paper notes that studies may select SVs using different criteria; distance is one example for separating SVs from NVs.Source: B. Terminology · p. 2

C. Generic Problem Formulation

  • Behaviour is represented as the future states a target vehicle will traverse, and the paper uses a probabilistic formulation to address uncertainty.In this formulation, behaviour may mean a manoeuvre or a trajectory, depending on its representation.Source: C. Generic Problem Formulation · p. 2
  • The generic problem is to compute the conditional distribution of future target-vehicle states given ego-vehicle observations; predicting each target vehicle separately reduces computational requirements by dropping future interdependencies.The formulation first describes a joint distribution over interdependent vehicles' state sequences, which may be intractable.Source: C. Generic Problem Formulation · p. 3

III. CLASSIFICATIONS OF EXISTING WORKS

The review classifies deep learning vehicle-behaviour prediction by input representation, output type, and prediction method, then divides input representations into four classes. It contrasts conventional target-vehicle track history with approaches that incorporate surrounding vehicles.

III. CLASSIFICATIONS OF EXISTING WORKS

  • The review classifies deep learning approaches by input representation, output type, and prediction method, excluding physics-based approaches as no longer state of the art.The input, output, and prediction-method classifications provide more detail than the earlier physics-based, manoeuvre-based, and interaction-aware grouping.Source: §III · p. 3

A. Input Representation

  • Input representations comprise target-vehicle history, target-vehicle and surrounding-vehicle histories, simplified bird’s-eye views, and raw sensor data.The last three classes are sub-classes of interaction-aware approaches.Source: A. Input Representation · p. 3

1) Track history of the TV:

  • Target-vehicle track history uses state features such as position, velocity, acceleration, and heading, but relying on it alone can be inaccurate in crowded, long-term prediction.Some studies exclude observable surrounding vehicles, although their behaviour can affect the target vehicle’s behaviour.Source: 1) Track history of the TV · p. 3

2) Track history of the TV and SVs:

Input approaches represent surrounding traffic either through selected vehicle tracks or through simplified bird’s-eye-view images. Track-only methods face observability and scene-context constraints, while bird’s-eye-view designs vary in temporal detail and sensor fusion.

2) Track history of the TV and SVs:

  • Studies select surrounding vehicles using fixed neighbours, lane-based reference vehicles, distance thresholds, or weighted attention over observable agents.One study notes that more vehicles can improve prediction; for example, an accelerating second vehicle ahead may signal an upcoming speed increase for the target vehicle in a traffic jam.Source: 2) Track history of the TV and SVs · p. 4
  • Track-history methods often assume surrounding vehicles are observable, while realistic systems must account for occlusion and noise and include environment conditions and traffic rules.Source: 2) Track history of the TV and SVs · p. 4

3) Simplified Bird’s Eye View:

  • Simplified bird’s-eye views depict objects, lanes, and other scene elements with polygons and lines, preserving size, location, and road geometry while omitting texture.Source: 3) Simplified Bird’s Eye View · p. 4
  • Bird’s-eye-view inputs range from stacked radar-camera channels and semantic scene images to social tensors that encode vehicle dynamics and spatial relations.Other approaches add scene-context channels or combine social pooling with a semantically segmented camera view transformed to bird’s-eye view.Source: 3) Simplified Bird’s Eye View · p. 4
  • Dynamic occupancy grid maps fuse sensor data into occupancy-probability and velocity channels, but do not retain complete object history; simplified views can flexibly fuse sensors, yet inherit perception errors.Representation complexity can be matched to computational constraints, while errors or omissions in object-state estimates can cascade into prediction.Source: 3) Simplified Bird’s Eye View · p. 4

4) Raw sensor data:

Raw-sensor approaches offer the prediction model the available sensory data but require more computation. The review distinguishes output types and notes that manoeuvre-intention classes provide high-level behaviour descriptions whose detail and breadth demand more data and model capacity.

4) Raw sensor data:

  • Raw sensor data lets prediction models learn features from available sensory inputs, but its larger dimension increases computational demands and can hinder onboard implementation.Sharing parameters with an object-detection model is suggested because that module also processes raw sensor data and commonly has millions of parameters.Source: 4) Raw sensor data · p. 4

B. Output Type

  • Vehicle behaviour outputs are grouped into manoeuvre intention, unimodal trajectory, multimodal trajectory, and occupancy map.Source: B. Output Type · p. 5

1) Manoeuvre Intention:

  • Manoeuvre-intention prediction estimates an upcoming manoeuvre, with classes such as lane changes and lane keeping on highways or turns at intersections.Reviewed examples include junction intentions and highway lane-change, lane-keeping, and cut-in predictions.Source: 1) Manoeuvre Intention · p. 5
  • A small manoeuvre set gives only high-level behaviour and may be specific to one driving environment; finer, broader classes require diverse data with sufficient examples per class and greater model capacity.Source: 1) Manoeuvre Intention · p. 5

2) Unimodal trajectory:

Unimodal trajectory predictors output one future path, but averaging possible behaviours can produce an invalid trajectory, while intention-conditioned paths depend on predefined manoeuvre labels. Multimodal methods represent multiple paths, with different trade-offs between predefined and dynamically learned modes.

2) Unimodal trajectory:

  • Unimodal predictors independent of manoeuvre may estimate positions, displacements, lateral position and longitudinal velocity, heading, or trajectory uncertainty.Some methods predict Gaussian-distribution parameters or standard deviations for future positions.Source: 2) Unimodal trajectory · p. 6
  • Intention-conditioned unimodal models predict the path for the most likely predefined manoeuvre, but can miss unlisted intentions and require manual intention labels.Source: 2) Unimodal trajectory · p. 6
  • Unimodal predictors output one trajectory even when a situation permits multiple futures, and averaging those modes can yield an invalid path that causes a collision.Trajectory outputs are more precise than discrete intentions but add complexity because they are continuous.Source: Figure 3 · p. 6, 2) Unimodal trajectory · p. 6

3) Multimodal trajectory:

  • Multimodal models predict trajectories and probabilities for behaviour modes, either using explicitly defined static modes or dynamically learned modes.Static-mode methods can model uncertainty separately for each manoeuvre; dynamic-mode methods predict or sample trajectory sequences and rank them by likelihood.Source: 3) Multimodal trajectory · p. 6
  • Static multimodal modes retain predefined-set and manual-labelling drawbacks, whereas dynamic modes avoid those requirements but may collapse to one mode or miss existing modes.Source: 3) Multimodal trajectory · p. 6

4) Occupancy map:

The section reviews occupancy-map outputs and classifies prediction methods, emphasizing recurrent networks and their uses for temporal sequences, multimodality, and interaction.

4) Occupancy map:

  • Occupancy-map approaches estimate future cell occupancy rather than trajectories directly; smaller cells can improve spatial resolution but increase computational cost.The reviewed models assign occupancy likelihoods across cells and prediction time steps, enabling multiple trajectory modes.Source: 4) Occupancy map · pp. 6–7

C. Prediction Method

  • The prediction-method classification covers recurrent neural networks, convolutional neural networks, and other methods.Table III summarizes studies by prediction method and the advantages and disadvantages of each class.Source: C. Prediction Method · p. 7, Table III · p. 8

1) Recurrent neural networks:

  • Vanilla RNNs combine current inputs with prior hidden-state memory, while gated RNNs such as LSTMs and GRUs address the difficulty of training on long sequences.LSTMs are reported as the most-used deep models in vehicle behaviour prediction.Source: 1) Recurrent neural networks · p. 7
  • Single-RNN models support intention or unimodal trajectory prediction, and some add secondary models for interaction awareness or multimodal output.Examples include LSTM sequence classifiers, regression models, and encoder-decoder systems for multimodal prediction.Source: 1) Recurrent neural networks · p. 9
  • Multiple-RNN architectures are used to model vehicle interactions or produce multimodal predictions, including maneuver-conditioned trajectory distributions.Some designs also combine instance-level movement and interaction modeling with category-level reasoning.Source: 1) Recurrent neural networks · p. 9
  • RNNs have difficulty modeling spatial relationships and image-like driving context, so sophisticated solutions often add methods to compensate.Source: 1) Recurrent neural networks · p. 9

2) Convolutional neural networks:

CNNs process image-like inputs and outputs while preserving spatial relationships, whereas other reviewed approaches combine neural architectures or represent vehicle interactions as graphs.

2) Convolutional neural networks:

  • Reviewed CNN applications range from intention prediction and future occupancy maps to joint object detection and behaviour prediction.Examples use BEV representations, temporal convolutions on voxelized lidar, and separate processing of lidar and rasterized maps.Source: 2) Convolutional neural networks · p. 9
  • CNNs preserve spatial relationships and handle image-like inputs and outputs, but 2D CNNs lack a mechanism for modeling temporal dependencies among vehicle states.Source: 2) Convolutional neural networks · p. 9

3) Other Methods:

  • Feed-forward networks can use current vehicle states when history is unavailable or a first-order Markov assumption is made, and can be competitive in some scenarios with faster processing than RNNs.One reviewed model predicts a Gaussian mixture distribution over arrival time and final location.Source: 3) Other Methods · p. 10
  • Hybrid RNN-CNN models use recurrent networks for temporal features and CNNs for spatial features, including scene context and vehicle interactions.Reviewed designs combine vehicle-specific LSTM encodings with convolutional processing and recurrent trajectory decoders.Source: 3) Other Methods · p. 10
  • Graph neural networks represent vehicles as nodes and their interactions as edges, but current graph-based approaches usually neglect static scene context.A reviewed graph-convolutional interaction model feeds its output to an LSTM encoder-decoder for multiple target vehicles.Source: 3) Other Methods · p. 10

IV. EVALUATION

The evaluation section separates intention and trajectory metrics by task type, then reviews classification measures including accuracy, precision, recall, F1, NLL, and prediction time.

IV. EVALUATION

1) Intention Prediction Metrics:

  • Accuracy can mislead on imbalanced datasets, since always predicting the common lane-keeping class may score highly despite missing lane changes.The paper also notes precision, recall, and F1 as measures used in existing studies.Source: 1) Intention Prediction Metrics · p. 10
  • Precision measures the share of predictions for a class that are correct, while recall measures the share of that class's samples correctly identified.F1 balances precision and recall.Source: 1) Intention Prediction Metrics · p. 10
  • NLL can compare uncertainty across intention models, while average prediction time measures how early a positive class is first predicted correctly.One cited approach measures the start of consistent correct lane-change predictions to improve metric robustness.Source: 1) Intention Prediction Metrics · pp. 10–11

2) Trajectory Prediction Metrics:

The trajectory-metric discussion covers prediction errors, computation time, multimodal best-of-K evaluation, and cross entropy for comparing predicted and ground-truth distributions.

2) Trajectory Prediction Metrics:

  • FDE measures the distance between predicted and true final locations at the prediction horizon, without accounting for errors at earlier time steps.Source: 2) Trajectory Prediction Metrics · p. 11
  • MAE and RMSE measure trajectory prediction error, with RMSE more sensitive to large errors because it uses squared error.The error can be defined as displacement between the predicted trajectory and ground truth.Source: 2) Trajectory Prediction Metrics · p. 11
  • Minimum-of-K evaluation scores only the best of K predicted trajectories, leaving the quality of the other K−1 trajectories unexamined.A model can therefore score highly while producing mostly poor predictions.Source: 2) Trajectory Prediction Metrics · p. 11
  • Cross entropy evaluates a modeled trajectory distribution against ground truth, while symmetrized cross entropy also penalizes predicted modes absent from the ground-truth distribution.The ground-truth distribution's PDF cannot be evaluated directly, so the symmetric measure uses an approximation.Source: 2) Trajectory Prediction Metrics · p. 11
  • Computation time matters because complex trajectory-prediction models may be impractical for on-board autonomous-vehicle implementation.Source: 2) Trajectory Prediction Metrics · p. 11

B. Performance of Existing Methods

Across the selected studies, most deep learning models outperform the constant-velocity baseline, and more complex models often outperform single RNNs. Multimodal prediction does not consistently yield lower RMSE than unimodal prediction.

B. Performance of Existing Methods

  • The review compares studies using common public datasets and metrics, adjusting some reported RMSE values for consistent units, position measures, or dataset averages.The selected studies report errors across prediction horizons from 1.0 to 5.0 s on NGSIM I-80 and US-101.Source: B. Performance of Existing Methods · p. 12
  • Most reviewed deep learning models outperform the constant-velocity baseline by a high margin, while complex models often outperform single RNNs.The comparison uses reported RMSE errors for 1.0–5.0 s horizons on NGSIM I-80 and US-101; the review also reports a constant-velocity Kalman Filter baseline.Source: B. Performance of Existing Methods · p. 12
  • GRIP [29] and ST-LSTM outperform multimodal M-LSTM and CS-LSTM, showing that multimodal prediction does not always lower RMSE.The authors suggest limited model capacity or training data may explain this pattern.Source: B. Performance of Existing Methods · p. 12

C. Research Gaps and New Opportunities

The review identifies evaluation, observability, traffic-rule, input-data, and driving-scenario gaps in vehicle behaviour prediction. It concludes that practical adoption also requires accounting for sensor impairments and limited computational resources.

C. Research Gaps and New Opportunities

  • Only seven reviewed works share a unified evaluation method, and few report computation time, limiting fair comparison across approaches.The authors identify a shared benchmark as a future research direction.Source: C. Research Gaps and New Opportunities · p. 12
  • Full observability is often assumed but impractical: infrastructure cannot cover every road, while occlusion and noise can reduce observable vehicles and prediction accuracy.Connected vehicles could share sensor information through V2V and V2I communication to provide more information and reduce occlusion.Source: C. Research Gaps and New Opportunities · p. 12, Figure 4 · p. 12
  • Traffic rules are rarely explicit model inputs, although they can reshape vehicle behaviour; some studies include road direction or traffic lights only.The cited examples are and [30].Source: C. Research Gaps and New Opportunities · p. 12
  • Vehicle signalling lights and horns are additional visual and auditory cues that could help infer future behaviour.Source: C. Research Gaps and New Opportunities · p. 12
  • Many existing models target specific settings such as roundabouts, intersections, or T-junctions, while autonomous vehicles need prediction across varied driving scenarios.The authors identify models applicable to a variety of driving environments as a future direction.Source: C. Research Gaps and New Opportunities · p. 12

V. CONCLUSION

  • Despite promising performance in complex scenarios, adoption challenges include unmodeled environment conditions and traffic rules, sensor impairments, and limited computational resources.Source: §V · p. 12
Loading 1912.11676v2…