Source-linked AI summary
Survey of Deep Reinforcement Learning for Motion Planning of Autonomous Vehicles
Szilárd Aradi
TL;DR
Autonomous-vehicle motion planning spans hierarchical decisions, trajectory generation, and control, creating complex settings for learning-based methods. The paper surveys DRL formulations, modeling and representation choices, and applications across driving tasks, while highlighting computational and training challenges. Its supported conclusion is that DRL can be used across different motion-planning levels, but substantial design and efficiency questions remain.
Problem
Autonomous-vehicle motion planning involves complex, hierarchical problems with varied state, observation, and control requirements that motivate evaluating DRL methods.
Method
The paper surveys DRL for autonomous-vehicle motion planning, covering vehicle and environment models, observations, action spaces, algorithms, and task-level applications.
Results
The surveyed achievements show that different DRL techniques can be effectively used for different levels of autonomous-vehicle motion planning.
Takeaways & Limitations
DRL is applicable to motion-planning tasks involving unstructured radar or camera information, but system design requires choices about abstractions, rewards, networks, and training.
Takeaways & Limitations
Training complex DRL systems can require millions of steps, and one hyper-parameter or reward setup may take hours or days to evaluate.
Abstract
from arXiv · showhide
Academic research in the field of autonomous vehicles has reached high popularity in recent years related to several topics as sensor technologies, V2X communications, safety, security, decision making, control, and even legal and standardization rules. Besides classic control design approaches, Artificial Intelligence and Machine Learning methods are present in almost all of these fields. Another part of research focuses on different layers of Motion Planning, such as strategic decisions, trajectory planning, and control. A wide range of techniques in Machine Learning itself have been developed, and this article describes one of these fields, Deep Reinforcement Learning (DRL). The paper provides insight into the hierarchical motion planning problem and describes the basics of DRL. The main elements of designing such a system are the modeling of the environment, the modeling abstractions, the description of the state and the perception models, the appropriate rewarding, and the realization of the underlying neural network. The paper describes vehicle models, simulation possibilities and computational requirements. Strategic decisions on different layers and the observation models, e.g., continuous and discrete state representations, grid-based, and camera-based solutions are presented. The paper surveys the state-of-art solutions systematized by the different tasks and levels of autonomous driving, such as car-following, lane-keeping, trajectory following, merging, or driving in dense traffic. Finally, open questions and future challenges are discussed.
I. INTRODUCTION
The paper surveys Deep Reinforcement Learning for autonomous-vehicle motion planning, framing driving as a hierarchical and often partially observable decision problem. It reviews environment modeling, observation and action representations, DRL methods, and unresolved training challenges.
- I. INTRODUCTION: DRL combines reinforcement learning with deep neural networks and has gained popularity for complex autonomous-vehicle motion-planning problems.The surveyed application area includes optimization, control, artificial intelligence, and machine-learning approaches.
- A. The Hierarchical Classification of Motion Planning for Autonomous Driving: Autonomous driving can be decomposed into route planning, behavioral decisions, motion planning, and local feedback control.The behavioral layer selects short-term policies, while local control minimizes deviation from a prescribed path or trajectory.
- B. Reinforcement Learning: DRL systems must choose environment and observation abstractions, because vehicle-model fidelity, structured versus unstructured inputs, and observability affect learning and computation.Vehicle dynamics models trade accuracy against simulation cost, while image, lidar, and grid inputs typically require CNN processing.
- B. Reinforcement Learning: Unknown intentions of surrounding drivers make behavioral planning partially observable and motivate a POMDP formulation.The POMDP represents states, actions, transitions, rewards, observations, and the observation function.
- B. Reinforcement Learning: Value-based methods learn action values, whereas policy-based methods directly select behavior; continuous control may require specialized DRL or discretized commands.DQN uses Q values and Bellman updates, while policy networks predict action preferences; classical finite-action methods do not directly handle continuous control.
- B. Reinforcement Learning: Complex DRL tasks can require long or unsuccessful training, prompting approaches such as curriculum learning, adversarial learning, and model-based action choice.Curriculum learning increases task difficulty gradually, while model-based action choice can reduce distant-reward effects.
A. Vehicle modeling
Vehicle modeling balances simulation fidelity against computational cost, so DRL studies select models and simulators according to task requirements and training speed.
- A. Vehicle modeling: Vehicle-model selection trades accuracy against computational resources because RL requires many environment episodes.The environment step time depends strongly on evaluating vehicle dynamics, ranging from simple kinematic models to complex tire models.
- A. Vehicle modeling: Kinematic models neglect tire slip and are generally limited to low-speed applications, whereas 3DoF dynamic models include longitudinal and lateral movement.The cited passage describes 3DoF models using Vx, Vy, and yaw-rate dynamics with a linear tire model.
- A. Vehicle modeling: A kinematic bicycle model is appropriate around 0.5g lateral acceleration or less on dry roads, but higher accelerations require more accurate dynamics.Above this limit, the model cannot handle vehicle dynamics near handling limits.
- A. Vehicle modeling: 3DoF models can require 10 ... 50 times the kinematic-model calculation time, while precise 9DoF nonlinear-tire models can require 100 ... 300 times more.These computational costs help explain the field’s preference for low abstraction levels during RL training.
- A. Vehicle modeling: Traffic simulators include SUMO, VISSIM, and custom cellular-automata, MOBIL, or IDM environments, while CARLA, AirSim, and Udacity provide richer sensor-oriented simulation.TORCS is used chiefly for vehicle dynamics and includes tracks, competing agents, and sensor models.
- A. Vehicle modeling: SUMO is fast and easy to use for traffic-agent training but provides no sensor model beyond ground-truth vehicle states.Professional tools such as CarSIM and CarMaker are more precise but are expensive or resource-intensive for repeated RL episodes.
C. Action Space
DRL action spaces span direct vehicle control and higher-level behavioral decisions, with continuous actuation creating a mismatch for algorithms requiring finite actions.
- C. Action Space: Action spaces depend on the vehicle model and task, covering direct steering, braking, and acceleration or strategic behavioral commands.Behavioral commands include lane changes, lane keeping, and adaptive-cruise-control reference settings.
- C. Action Space: Only a few studies operate at the motion-planning layer, where agents define spline trajectory knots between specified endpoints.Some other approaches generate actions by stepping through a grid.
- C. Action Space: Behavioral actions are finite choices selected by the neural network, while low-level controllers convert them into actual trajectories.Some studies combine layers by using direct longitudinal acceleration with strategic lane-changing decisions.
- C. Action Space: Continuous steering, throttle, and braking are incompatible with finite-action methods such as DQN and PG, whereas DDPG supports continuous actions.Many studies discretize steering and acceleration into 3 to 9 possibilities per channel.
- C. Action Space: Discretizing control can move solutions away from realistic actuation and create slip, jerk, or yaw-rate issues, while many choices slow POMDP learning exponentially.Kinematic models sometimes mask these vehicle-dynamics problems.
D. Rewarding
Reward design determines how DRL agents receive feedback across an episode, balancing learning speed against the risk of shaping behavior toward the designer’s intended strategy.
- D. Rewarding: Episodes end after task completion, a predefined step limit, or a terminal failure such as collision or leaving the lane or track.Terminal conditions represent states from which the task is impossible or an unacceptable mistake has occurred.
- D. Rewarding: Rewarding evaluates the agent’s choices during an episode and provides feedback used to improve the policy.Designers must choose among reward-timing strategies with different advantages and disadvantages.
- D. Rewarding: End-of-episode rewards minimize human-driven policy shaping but can slow learning because feedback is discounted backward.Immediate rewards generally learn faster but make the chosen reward strongly influential on the resulting strategy.
- D. Rewarding: Intermediate rewards can be delivered at predefined periods, travel distances, or good and bad decisions.Motion-planning rewards commonly reflect task completion or failure and factors such as time, speed, yaw, trajectory deviation, and overtaking.
- D. Rewarding: Rewards based on datasets or reference models can produce favorable results but depart from reinforcement learning’s original trial-and-error philosophy.A previously known strategy guides the learning in these approaches.
E. Observation Space
Observation design supplies the information needed for action selection, ranging from basic ego-state variables to richer descriptions of topology, traffic, and vehicle dynamics.
- E. Observation Space: Observation spaces may include ego position, speed, and yaw; road topology, signs, and rules; and surrounding vehicles or obstacles.These components describe the world from which the agent selects actions.
- E. Observation Space: An ego-centric reference frame concentrates visited positions, headings, and velocities around the vehicle and simplifies observation of nearby participants.The alternative is an absolute frame fixed to world coordinates.
- E. Observation Space: The simplest common ego observation is (|e|, v, θe): lateral position from lane center, vehicle speed, and yaw angle.This is the minimum information for classical kinematic car-like models under a no-skidding assumption.
- E. Observation Space: Fig. 4 presents an observation for the basic vehicle state.The supplied figure text identifies the observation but does not specify axes or a further comparison.
- E. Observation Space: The basic vehicle-state observation is insufficient when complex dynamics or vehicle stability matter, requiring yaw, pitch, roll, tire dynamics, and slip.Such conditions include racing situations and other tasks where vehicle dynamics are inevitable.
2) Environment observation:
The survey distinguishes sensor, intermediate, and ground-truth observation abstractions, each shaping the neural-network architecture. It contrasts unstructured image, lidar, and grid inputs with structured vehicle-state representations that can simplify learning.
- Observation abstractions: Observation models span sensor-level camera, lidar, or radar inputs; intermediate idealized sensor information; and ground-truth information.The sensor-model structure determines whether agents use 2D or 1D CNNs, dense networks, or mixed input layers.
- Image-based observations: Image observations encode state in C x W x H matrices, but their unstructured content often requires large samples and long training to converge.Examples include down-sampled inputs of 1x48x27 and 3x84x84.
- Observation preprocessing: Preprocessing can simplify unstructured observations; using difference images required approximately 10 times fewer training steps to reach the same performance in one study.The difference image was intended to emphasize foreground motion over background features.
- Grid-based observations: Grid representations encode occupancy and may add vehicle velocity, but simple equidistant grids omit lateral movement information.Lane-based grids instead align rows with road lanes and discretize longitudinal position.
- Grid-based observations: A multilayer observation tensor can encode on-road occupancy, relative velocity, lateral displacement, and heading across a field of view.The cited representation has size 4 x 3 x (2 x FoV + 1), with FoV denoting observation distance in cells.
- Structured observations: Structured inputs place relevant surrounding-vehicle parameters in fixed fields, reducing input size and allowing simple DNNs instead of CNN-based feature extraction.For car following, examples use leading-vehicle distance and speed, or distance, speed, and acceleration.
III. SCENARIO-BASED CLASSIFICATION OF THE APPROACHES
The survey organizes reinforcement-learning research around scenario-specific autonomous-driving subtasks rather than a single end-to-end solution. It highlights state-space growth as a limitation of classic Q-learning and motivates DRL for unstructured observations.
- Classic reinforcement learning: Classic reinforcement learning cannot directly handle unstructured camera, radar, or lidar data.This limitation is one motivation for combining reinforcement learning with deep neural networks.
- Classic reinforcement learning: Q-learning requires a table over all state-action pairs, causing space complexity to grow with the product of state and action class sizes.In the cited overtaking example, discretization produced 6 ∗10 ∗8 ∗9 ∗3 = 12960 Q-table elements.
- Problem decomposition: Multiple-goal reinforcement learning can reduce the overall problem by dividing driving into subtasks such as collision avoidance, target seeking, and lane-related behaviors.The survey presents this decomposition as a possible reduction for more complex problems.
- Path planning: A classic Q-learning path-planning agent for an Ackermann-steering ground vehicle used speed, position, and heading as its state representation.Reinforcement learning served as an optimizer for obstacle-aware path planning.
- Scenario-based classification: Recent reinforcement-learning studies generally examine dedicated autonomous-driving scenarios and subtasks rather than solving automated driving end to end.The surveyed problems vary in complexity and include specific driving situations.
A. Car following
Car following is formulated as a longitudinal-control task in which a following vehicle adjusts acceleration to maintain a safe distance from a leading vehicle.
- A. Car following: Car following involves a leading and following vehicle remaining in lane while the follower adjusts longitudinal speed to maintain a safe distance.The observation is (v, dv, ds), representing agent speed, speed difference to the lead, and headway distance; the action is acceleration.
B. Lane keeping
The survey covers lateral control, ramp merging, and dense-traffic behavior as progressively richer autonomous-driving scenarios. These studies vary from ground-truth or camera observations to recurrent models and curriculum learning.
- B. Lane keeping: Lane keeping and trajectory following focus on lateral control using either ground-truth lane position and angle or front-facing camera images.Image-based studies use external simulators such as TORCS and GAZEBO/ROS.
- C. Merging: Ramp merging requires the ego vehicle to identify an acceptable gap between two vehicles before entering the highway.Some approaches learn longitudinal control, while others use full steering and acceleration control.
- C. Merging: Dense-traffic training with cooperative and non-cooperative drivers used curriculum learning by gradually increasing traffic density.The study compared the learned agents with three differently built MCTS planners.
- C. Merging: A policy that does not leverage other drivers’ cooperativeness avoids collisions but fails to achieve the maneuver.This outcome is reported for direct training in dense traffic.
- C. Merging: One ramp-merge formulation uses an LSTM to incorporate historical and interactive driving behavior before Deep Q-learning selects actions through a Q-function approximator.The Q-network is updated with experience replay and a second target Q-network.
D. Driving in traffic
Driving in traffic is addressed through hierarchical decisions, varied observation and action spaces, and increasingly complex vehicle models. The surveyed approaches include cooperative behavior, end-to-end control, and substantial training demands.
- Behavioral decisions: Traffic-driving agents use discrete strategic actions such as keeping the lane or changing left or right.One study observed ego speed and lane position plus the relative positions and speeds of eight surrounding vehicles.
- Hierarchical control: Hierarchical designs separate lane-change decisions from longitudinal control and can combine DQN policies with classic controllers.A two-lane setup first decides whether to change lanes, then selects acceleration; Pure Pursuit Control adjusts vehicle position.
- Cooperative behavior: Cooperative highway behavior can be induced by rewarding both ego-vehicle performance and the speed of surrounding traffic.The cited setup uses a three-lane highway, a lane-based grid observation, and four actions: left, right, speedup, and none.
- Vehicle modeling: Simplified holonomic vehicle models reduce control complexity by replacing nonholonomic steering geometry with fixed-speed lateral lane changes.The cited actions are acceleration, deceleration, changing lanes left or right, and taking no action, using Dueling DQN with prioritized replay.
- End-to-end driving: End-to-end traffic driving with nonholonomic kinematics requires quantized steering and acceleration commands and 300’000 training episodes.By comparison, simple lane-keeping tasks required only a few hundred episodes.
IV. FUTURE CHALLENGES
Future work must address the computational, modeling, validation, transferability, and standardization challenges of DRL for autonomous-vehicle motion planning. The paper points toward mixed hierarchical systems and extended learning techniques while retaining uncertainty about automotive suitability.
- Open challenges: DRL methods can address multiple autonomous-vehicle motion-planning levels, including systems handling raw or lightly processed radar and camera data.The paper nevertheless states that many questions remain unanswered.
- Validation and standards: Deep-learning function development remains difficult to verify and validate because automotive standards do not yet provide dedicated deep-learning statements.The cited standards include Automotive SPICE and ISO 26262.
- Computational requirements: Trained networks have relatively low computational requirements, but complex reinforcement-learning tasks may require millions of training steps and hours or days per setup.Repeated iteration over the environment, network, reward scheme, or algorithm makes system design time-consuming.
- Sensor modeling: Many studies use ground-truth or ideal sensor representations, while only a few model sensor noise, raising feasibility questions for real-world transfer.The paper also notes that noisy or erroneous models could produce more robust agents.
- Environment realism: Highway environments commonly use fixed road topologies and limited surrounding-vehicle behavior, while validation often reuses the training setup.The paper calls for more realistic and diverse environments and vehicle-dynamics models, although these increase numerical complexity.
- Future directions: The paper proposes mixed or hierarchical systems combining classic control with deep RL, alongside curriculum learning, transfer learning, and planning agents.These approaches are presented as future directions for improving project efficiency.
- Overall assessment: Unresolved issues include environment and sensor modeling, computational requirements, real-world transferability, robustness, and validation.Because of these issues, the paper says it is hard to predict whether reinforcement learning is appropriate for automotive applications.