PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Reinforcement Learning for Autonomous Driving: A Survey

B Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A. Al Sallab, Senthil Yogamani, Patrick Pérez

arXiv:2002.00444v2cs.LGcs.AIcs.RO

TL;DR

Real-world autonomous driving requires methods for sequential, interacting-agent decisions, but RL applications remain emerging and face computational and deployment challenges. This review synthesizes DRL algorithms, organizes their use across driving tasks, distinguishes adjacent learning paradigms, and discusses simulators, validation, and robust deployment. It concludes with a theoretical overview, comprehensive task survey, open challenges, and implementation resources for further research.

  • Problem

    RL for autonomous driving remains an emerging area with little literature or large-scale public datasets despite scenarios involving interaction, negotiation, and dynamic decision-making.

  • Method

    The review presents RL theory, surveys RL applications across autonomous driving tasks, discusses deployment challenges, and describes open-source RL frameworks.

  • Results

    The paper organizes RL applications for autonomous driving and identifies computational, validation, implementation, and real-world deployment challenges.

  • Takeaways & Limitations

    The review provides an organized foundation for evaluating and extending RL algorithms for autonomous driving through theoretical background, task literature, and implementation resources.

Abstract

from arXiv · show

With the development of deep representation learning, the domain of reinforcement learning (RL) has become a powerful learning framework now capable of learning complex policies in high dimensional environments. This review summarises deep reinforcement learning (DRL) algorithms and provides a taxonomy of automated driving tasks where (D)RL methods have been employed, while addressing key computational challenges in real world deployment of autonomous driving agents. It also delineates adjacent domains such as behavior cloning, imitation learning, inverse reinforcement learning that are related but are not classical RL algorithms. The role of simulators in training agents, methods to validate, test and robustify existing solutions in RL are discussed.

I. INTRODUCTION

Autonomous driving combines perception with sequential decision-making, where classical supervised learning is limited because actions alter future observations and supervisory signals can be difficult to specify. The review organizes RL applications, contributions, and deployment challenges across autonomous driving systems.

  • Motivation: Autonomous driving includes perception tasks and decisions whose actions change future sensor observations, limiting classical supervised learning.The passage gives optimal urban driving speed as an example.
  • Scope: The review formalizes RL applications across driving policy, predictive perception, path and motion planning, and low-level controller design.
  • Contributions: The paper provides a self-contained RL background, a detailed literature review of autonomous driving tasks, and discussion of real-world challenges and opportunities.
  • System context: A modern autonomous driving pipeline uses sensor streams to create intermediate environment representations for decision-making and driving policy generation.Sensors include cameras, radars, LIDARs, GPS-GNSS, and IMUs; perception uncertainty propagates through the information chain.
  • System context: The pipeline’s key problems are scene understanding, decision, and planning.

A. Scene Understanding

Scene understanding maps heterogeneous sensor data into an abstract, uncertainty-aware representation that supports decision-making and planning. The broader driving pipeline also depends on maps and trajectory generation under vehicle constraints.

  • A. Scene Understanding: Scene understanding fuses heterogeneous sensors to produce a higher-level, sensor-agnostic representation that robustly generalizes as scene content becomes more abstract.The representation models sensor noise and detection uncertainties across LIDAR, camera, radar, and ultrasound.
  • A. Scene Understanding: Mapping localizes the vehicle within a mapped area, while semantic object detection and HD maps support reliable disambiguation and object detection.
  • A. Scene Understanding: Trajectory planning converts route-level plans into motion-level commands, while classical planning faces difficulties with non-holonomic vehicle constraints.

D. Control

Vehicle control uses temporal dynamics and classical optimal-control foundations, while RL provides frameworks for sequential decision-making through value, policy, and reward-based formulations. The paper also situates RL within broader machine-learning and algorithmic components.

  • D. Control: Trajectory tracking models vehicle dynamics over sequential waypoints, whereas controllers specify speed, steering, and braking actions along a path.
  • D. Control: Classical vehicle control minimizes a cost function over states and control actions within a feasible state space, using methods such as PID and MPC.
  • III. Reinforcement Learning: RL agents balance exploration and exploitation by trying new actions while using known high-reward actions; ε-greedy is one example.
  • III. Reinforcement Learning: An MDP formalizes sequential decision-making with states, actions, transitions, and rewards, while a policy maps states to action distributions.
  • III. Reinforcement Learning: Value-based methods estimate cumulative reward for action selection, while policy-based methods estimate the optimal policy directly.
  • III. Reinforcement Learning: The discount factor controls how strongly agents value future rewards, with lower values encouraging short-term behavior and higher values longer-term behavior.

A. Value-based methods

Value-based methods estimate action values and derive policies from them, while policy-based methods estimate policies directly. Policy-gradient extensions constrain updates to improve stability and sample efficiency.

  • A. Value-based methods: Q-learning is a model-free temporal-difference algorithm that estimates the utility of state-action pairs.Its convergence guarantee requires discrete state-action values and infinitely frequent sampling of all actions in all states.
  • A. Value-based methods: Q-learning can converge to optimal state-action values under its stated sampling and representation conditions, regardless of initial Q values.Initial Q estimates may be optimistic, pessimistic, or informed by problem knowledge.
  • A. Value-based methods: Value-based methods evaluate cumulative reward and derive actions from value estimates, whereas policy-based methods estimate the optimal policy directly.Policy-based methods may produce stochastic or deterministic policies, often parameterized as neural networks.
  • A. Value-based methods: Deterministic policy gradients support reinforcement learning with continuous actions through a model-free gradient of the action-value function.They avoid integrating over both state and action spaces under the stated MDP conditions.
  • A. Value-based methods: TRPO limits policy changes using KL divergence, while PPO clips its surrogate objective to keep updates relatively small.The review describes TRPO as producing monotonic performance improvements and PPO as simpler to implement with better sample complexity.

C. Actor-critic methods

Actor-critic methods combine a policy-selecting actor with a value-estimating critic. Their variants use baselines, asynchronous or synchronous updates, advantage estimates, and entropy to support learning and exploration.

  • C. Actor-critic methods: Actor-critic methods combine policy-based and value-based learning through an actor that selects actions and a critic that evaluates them.Both networks learn using gradients, with the critic assessing whether outcomes were better or worse than expected.
  • C. Actor-critic methods: A well-chosen baseline reduces policy-gradient variance and can make learning more stable; DDPG applies this framework to continuous action spaces.DDPG is described as model-free, off-policy, and actor-critic, using deep neural-network function approximation.
  • C. Actor-critic methods: DDPG evaluates Q after the policy has selected an action, then updates separate estimates of the optimal value function and policy.This differs from Q-learning, which chooses actions according to the Q-function.
  • C. Actor-critic methods: A3C replaces experience replay with asynchronous agents operating on parallel environment instances and uses value estimates as baselines.The review associates this design with reduced variance and improved convergence time.
  • C. Actor-critic methods: Advantage estimates compare action values with state values, indicating how much better actions were than expected and supporting more stable training.A2C is the synchronous counterpart to A3C, and their performance is described as comparable.
  • C. Actor-critic methods: Entropy-based methods use uncertainty to encourage exploration, including energy-based policies and Soft Actor Critic.The passage describes entropy as an uncertainty quantity included in many A3C implementations.

D. Model-based (vs. Model-free) & On/Off Policy methods

Model-based methods learn environment dynamics to reduce real-environment interaction, while policy choices and deep representations address large or continuous state-action spaces. Architectural extensions improve value estimation and partial-observation handling.

  • D. Model-based (vs. Model-free) & On/Off Policy methods: Learning an environment model can reduce costly or unsafe real-world interactions and enable exploration on the learned dynamics.The review gives Dyna-Q as an example of a model-based approach.
  • D. Model-based (vs. Model-free) & On/Off Policy methods: Long-term memory stores general domain knowledge, short-term memory captures the current situation, and the value function combines both.This memory decomposition is presented within the discussion of learned environment models.
  • D. Model-based (vs. Model-free) & On/Off Policy methods: On-policy methods update from trajectories generated by the current policy, whereas off-policy methods can learn from another policy’s behavior.The review contrasts SARSA as on-policy with Q-learning as off-policy.
  • D. Model-based (vs. Model-free) & On/Off Policy methods: Tabular representations scale exponentially with additional state features, creating memory and computation problems in large real-world state-action spaces.Fine discretization may require impractical numbers of samples, while coarse discretization can limit performance.
  • D. Model-based (vs. Model-free) & On/Off Policy methods: DQN learns control policies from raw video using stacked frames and deep neural networks, with target and online networks supporting training stability.Its input is an 84×84×4 tensor of four consecutive frames, and a forward pass predicts values for valid actions.
  • D. Model-based (vs. Model-free) & On/Off Policy methods: Double DQN reduces overestimation by selecting actions with the online network and evaluating them with the target network.The passage reports more accurate value estimates and higher scores on several games.
  • D. Model-based (vs. Model-free) & On/Off Policy methods: Dueling networks separately estimate state value and advantage before combining them into action values, improving state-value approximation.The value stream is updated with every update, which is relevant to temporal-difference methods such as Q-learning.
  • D. Model-based (vs. Model-free) & On/Off Policy methods: DRQN combines an LSTM with DQN to integrate information across frames and generalizes better than DQN from complete to flickering observations.The reported capability includes detecting information such as object velocity.

IV. EXTENSIONS TO REINFORCEMENT LEARNING

Extensions to reinforcement learning broaden applicability and can improve scalability, learning speed, or converged performance. The review discusses reward shaping, multi-agent formulations, and challenges arising from sparse rewards and multiple interacting agents.

  • IV. EXTENSIONS TO REINFORCEMENT LEARNING: Extensions to single-agent RL paradigms have been introduced to broaden applicability and improve scalability, learning speed, or converged performance.The section presents these extensions as developments for complex problem domains.
  • IV. EXTENSIONS TO REINFORCEMENT LEARNING: Reward design defines the optimal policy because RL agents maximize return from the reward function.Sparse or delayed rewards can make learning difficult, motivating additional shaping rewards.
  • IV. EXTENSIONS TO REINFORCEMENT LEARNING: Reward shaping adds rewards for behavior leading toward desired behavior to improve learning speed and converged performance.The review identifies D-shaped rewards and potential-based reward shaping as common approaches with theoretical guarantees.
  • IV. EXTENSIONS TO REINFORCEMENT LEARNING: Multi-agent reinforcement learning requires a stochastic-game formulation when multiple autonomous agents act simultaneously in a shared environment.The single-agent MDP framework is described as inadequate for this setting.

C. Multi-objective reinforcement learning

This section situates multi-objective reinforcement learning and related learning approaches within autonomous-driving applications. It also covers state representation learning and learning from demonstrations as adjacent techniques.

  • Multi-objective reinforcement learning: Multi-objective reinforcement learning represents performance across conflicting objectives with a vector-valued reward.This extends standard decision processes to multi-objective formulations.
  • State representation learning: State representation learning extracts and compresses high-dimensional observations into a latent state used for action selection.The representation can be decoded to estimate the original observation, and training requires no labels.
  • Learning from demonstrations: Learning from demonstrations supports initial exploration when rewards are sparse or input spaces are too large to cover.Demonstrations usually take the form of state-action pairs.
  • Learning from demonstrations: Demonstrations can initialize or improve learning through expert-policy divergence, prioritized replay, and combined demonstration-and-environment training.The cited approaches include DQfD and methods that reduce the need for explicit exploration after an initial demonstration.
  • Autonomous-driving applications: Autonomous-driving applications for reinforcement learning include policy learning, planning, controller optimization, inverse-reward learning, and safety-oriented risk estimation.The listed tasks span high-level driving behavior, motion, prediction, and control.

A. State Spaces, Action Spaces and Rewards

Autonomous-driving DRL depends on designing suitable state, action, and reward spaces. The review contrasts compact scene representations and action discretization with continuous control, while noting unresolved reward-design challenges and simulator-based training.

  • State spaces: Suitable state spaces, action spaces, and reward functions are essential for applying DRL to autonomous driving.Common state features include ego position, heading, velocity, and nearby obstacles.
  • State spaces: A 2D bird’s-eye-view representation preserves spatial road layout while reducing complexity relative to raw sensor inputs.It can combine occupancy, trajectories, and semantic information, and is available in simulators such as Carla and Flow.
  • Action spaces: Action discretization enables discrete-action DRL methods but may produce jerky trajectories and creates a bin-count trade-off.Continuous actuator values can instead be handled directly by algorithms such as DDPG.
  • Rewards: Reward design remains an open question, with criteria including progress, speed, collisions, lane keeping, comfort, stability, and traffic-rule adherence.The review lists both task objectives and penalties for unsafe or uncomfortable behavior.
  • Applications: A full-sized autonomous vehicle learned lane following after simulation and real-time onboard training, completing a 250 metre real-world road trial.The result is reported as an application of DDPG for autonomous driving.
  • Simulation: Reinforcement learning training and validation require simulators that model vehicle and environment dynamics, stochasticity, and recoverable state-action interactions.This differs from supervised datasets built around image-label pairs.

D. LfD and IRL for AD applications

This section distinguishes behavior cloning and related imitation-learning approaches from reinforcement learning, and reviews simulation, validation, and domain-adaptation issues relevant to autonomous driving.

  • Behavior cloning: Behavior cloning learns to imitate expert demonstrations through supervised learning but is difficult to adapt to unseen situations.Early driving systems learned from demonstrations, including end-to-end mappings from camera pixels.
  • Real-world deployment: Real-world reinforcement-learning deployment requires addressing computational challenges and risks before autonomous-driving systems are used outside controlled settings.The review frames these issues as deployment challenges and discusses related research approaches.
  • Validation: Validation can be distorted by differing implementations, hyperparameters, and unprincipled top-k rollout estimates, complicating benchmark reproduction and generalization assessment.The cited work recommends either a well-defined common setup or evaluation on real-world tasks.
  • Validation: Adversarial scenario generation in high-fidelity simulators can create rare, challenging driving situations for evaluation and training.Adding these scenarios to imitation-learning training was shown to increase safety.
  • Simulation-to-real transfer: Simulation-to-real transfer uses domain adaptation or randomized simulator dynamics to reduce dependence on real-world data.One cited vision-based grasping system achieved comparable performance with 50 times fewer real-world samples, though it is outside autonomous driving.
  • Simulation-to-real transfer: Unsupervised image translation can support imitation learning from simulated ground truth when simulated and real images lack pairwise correspondences.Related methods either translate simulated images toward reality or adapt real camera streams toward synthetic images during deployment.

C. Sample efficiency

Sample efficiency is a central challenge for reinforcement learning in autonomous driving because experience can be costly or risky to collect. The review surveys reward shaping, imitation initialization, replay, transfer, meta-learning, and compact state representations as responses.

  • Challenge: RL sample efficiency is difficult in autonomous driving because rewards are delayed and sparse, observations are unbalanced, and valuable experience is expensive or risky.These conditions occur in large state spaces and make learning a reasonable policy require many samples.
  • Efficiency approaches: Reward shaping supplies more frequent intermediate feedback to encourage faster learning from fewer samples.A trauma replay memory can separately retain collision situations for use during training.
  • Efficiency approaches: Imitation learning can bootstrap RL by initializing an offline policy from expert roll-outs before environment-based self-improvement.This combines demonstration-based initialization with subsequent reinforcement learning.
  • Efficiency approaches: Experience replay improves sample use by enabling multiple gradient updates from each sampled experience.ACER combines replay with a sample-efficient policy-gradient approach and trust-region optimization.
  • Efficiency approaches: Transfer learning reuses policies from source tasks, while policy composition combines learned basis policies for faster learning on novel tasks.The cited work frames reuse as an approach to sample efficiency.
  • Efficiency approaches: Meta-learning trains agents to adapt rapidly to new tasks using prior knowledge and small amounts of experience.Examples include recurrent task-structure learning, MAML initialization, and Reptile.
  • Efficiency approaches: World models compress spatial and temporal environment information with VAEs before learning a policy from the compact representation.The approach is presented as an efficient state-representation strategy.

D. Exploration issues with Imitation

Imitation learning can fail when expert trajectories do not cover the states encountered during testing, motivating methods that address distribution shift and safety. The review also identifies reward design, safe deployment, and multi-agent interaction as central autonomous-driving concerns.

  • Imitation-learning limitations: Expert trajectories may not cover all states encountered by the trained agent during testing, limiting imitation learning.Imitation learning also assumes actions are independently and identically distributed; DAgger is presented as one response.
  • Reward design: Reward shaping supplies additional designed rewards to encourage learning toward the optimal policy in real-world autonomous driving.Unlike controlled game environments, real-world driving requires designing an appropriate reward function.
  • Safe deployment: Direct deployment after training can be dangerous, motivating safety policies and combinations of deep reinforcement learning with safety-based control.Safe DAgger predicts primary-policy error without querying a reference policy; the cited combination performs well in most scenarios.
  • Safe deployment: Survival-Oriented Reinforcement Learning favors survival over total reward by formulating autonomous driving as a constrained MDP and learning from previous failures.The approach is intended to escape local optima, accelerate training, and avoid danger conditions or accidents.
  • Multi-agent interaction: Autonomous driving is fundamentally multi-agent because pedestrians, cyclists, and other vehicles interact with the controlled ego vehicle.The review identifies explicit multi-agent learning as an important future direction, including coordination and adversarial testing.

VII. CONCLUSION

The review organizes reinforcement learning for autonomous driving and surveys its applications, while emphasizing unresolved deployment and reproducibility challenges. It also highlights open-source frameworks and multi-agent reinforcement learning as areas for practical support and future development.

  • Conclusion: The paper presents a comprehensive literature survey and detailed theoretical treatment of reinforcement learning for autonomous-driving tasks.It formalizes and organizes an emerging literature on real-world autonomous-driving applications.
  • Challenges and opportunities: The review identifies validation, simulation-reality gaps, sample efficiency, reward design, and safety as deployment challenges.These topics are discussed as challenges, future research directions, and opportunities.
  • Reproducibility: Table III catalogs open-source frameworks and packages for state-of-the-art RL/DRL algorithms and evaluation.The table supports the review’s focus on implementation and evaluation resources.
  • Reproducibility: Reinforcement-learning results are difficult to reproduce and highly sensitive to hyper-parameter choices that are often insufficiently reported.The paper points to reliable, documented, and tested implementations as a needed starting point.
  • Future directions: Explicit multi-agent reinforcement learning remains an important future challenge that has received limited attention.The review notes potential applications in coordination, high-level decision making, and safety-policy testing.
  • Future directions: The paper presents examples of documented open-source frameworks that enable using, evaluating, and extending different reinforcement-learning algorithms.Implementation is described as challenging for both researchers and practitioners.
Loading 2002.00444v2…