PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey of Deep RL and IL for Autonomous Driving Policy Learning

Zeyu Zhu, Huijing Zhao

arXiv:2101.01993v1cs.RO

TL;DR

Autonomous-driving policy learning must support safe, efficient, and harmonious behavior in complex scenarios. This survey organizes DRL/DIL research by system architecture, task formulation, and application challenges, identifying five integration modes and emphasizing unresolved architectural, formulation, interaction, uncertainty, and validation issues.

  • Problem

    Autonomous-driving agents still face substantial challenges in complex scenarios, while existing surveys do not simultaneously address DRL/DIL policy learning from system, task-driven, and problem-driven perspectives.

  • Method

    The paper comprehensively surveys DRL/DIL autonomous-driving policy learning across system integration, task-specific model formulations, and critical application issues.

  • Results

    The survey identifies five DRL/DIL integration modes and finds that research has focused more on accomplishing autonomous-driving tasks than optimizing embedded system architectures.

  • Takeaways & Limitations

    The findings support further investigation of architecture optimization, theoretically supported formulations, systematic interaction studies, and stronger validation and benchmarks for real-world deployment.

Abstract

from arXiv · show

Autonomous driving (AD) agents generate driving policies based on online perception results, which are obtained at multiple levels of abstraction, e.g., behavior planning, motion planning and control. Driving policies are crucial to the realization of safe, efficient and harmonious driving behaviors, where AD agents still face substantial challenges in complex scenarios. Due to their successful application in fields such as robotics and video games, the use of deep reinforcement learning (DRL) and deep imitation learning (DIL) techniques to derive AD policies have witnessed vast research efforts in recent years. This paper is a comprehensive survey of this body of work, which is conducted at three levels: First, a taxonomy of the literature studies is constructed from the system perspective, among which five modes of integration of DRL/DIL models into an AD architecture are identified. Second, the formulations of DRL/DIL models for conducting specified AD tasks are comprehensively reviewed, where various designs on the model state and action spaces and the reinforcement learning rewards are covered. Finally, an in-depth review is conducted on how the critical issues of AD applications regarding driving safety, interaction with other traffic participants and uncertainty of the environment are addressed by the DRL/DIL models. To the best of our knowledge, this is the first survey to focus on AD policy learning using DRL/DIL, which is addressed simultaneously from the system, task-driven and problem-driven perspectives. We share and discuss findings, which may lead to the investigation of various topics in the future.

I. INTRODUCTION

The survey examines how DRL and DIL can learn autonomous-driving policies across system architecture, task formulation, and application challenges. It identifies five integration modes and highlights unresolved issues in architecture optimization, empirical formulation design, safety, interaction, and uncertainty.

  • I. INTRODUCTION: Autonomous-driving systems generate policies from online perception through behavior planning, motion planning, and control.
  • I. INTRODUCTION: Deep learning enables RL and IL to scale to high-dimensional state spaces and complex autonomous-driving tasks.The survey therefore focuses on deep reinforcement learning and deep imitation learning.
  • I. INTRODUCTION: The survey reviews DRL/DIL integration into autonomous-driving architectures and identifies five integration modes.It also compares the studies, methods, upstream perception modules, targeted tasks, and architectural advantages and disadvantages.
  • I. INTRODUCTION: The survey comprehensively reviews model state and action spaces and reinforcement-learning reward designs for specified autonomous-driving tasks.It finds that these formulations rely heavily on empirical designs and can produce substantially different driving policies when designs or parameters change.
  • I. INTRODUCTION: Safety, interaction with traffic participants, and environmental uncertainty are reviewed as critical autonomous-driving application issues.Safety is well studied through combinations of learned policies and traditional hard-constraint methods, while balancing optimality and guarantees remains non-trivial.
  • I. INTRODUCTION: RL models represent sequential decision making with states, actions, transitions, rewards, and a discount factor, seeking a policy that maximizes expected return.The framework includes trajectories, value functions, action-value functions, and Bellman recursion.

B. Value-based Methods

Value-based methods learn action values and recover policies from them. The section contrasts tabular and deep value-function approaches, emphasizing Q-learning, DQNs, and extensions that improve training stability or value representation.

  • B. Value-based Methods: Value-based RL identifies an optimal action-value function and recovers the optimal policy from learned state-action values.
  • B. Value-based Methods: Q-learning estimates Q values through temporal-difference updates and is off-policy, whereas SARSA is on-policy.Q-learning updates can use experiences not necessarily generated by the derived policy; SARSA uses the derived policy to generate experiences.
  • B. Value-based Methods: Tabular value methods suffer from the curse of dimensionality as increasing state features causes exponential growth in stored state-action pairs.Deep Q-networks address this by approximating a parameterized value function with a neural network.
  • B. Value-based Methods: DQN stabilizes training with a target network and experience replay.The target network is held fixed for specified iterations, while replay stores transition tuples for updates.
  • B. Value-based Methods: Dueling-DQN estimates value and advantage functions separately, while QR-DQN learns the full value distribution rather than only its expectation.

C. Policy-based Methods

Policy-based methods directly optimize a parameterized policy for expected return, using policy gradients and actor-critic variants to improve learning stability, exploration, and sample efficiency.

  • Policy optimization: Policy-based methods optimize a parameterized policy πθ to maximize expected return.Policy gradients provide the optimization basis for updating policy parameters.
  • Policy gradients: REINFORCE estimates policy gradients with discounted returns, but empirical returns can produce high-variance gradients.Baselines or advantage functions can replace returns to reduce variance and accelerate learning.
  • Stable updates: TRPO constrains policy updates using a quadratic approximation of KL divergence to guarantee monotonic performance improvement.PPO uses an adaptive KL penalty and clipped surrogate objective, requiring only first-order gradients.
  • Actor-critic methods: Actor-critic methods combine an actor that selects actions with a critic that estimates action or state values.The actor and critic are typically updated alternately according to policy-gradient and value-learning objectives.
  • Exploration: A3C uses parallel independent environments for stabilization and exploration, while SAC adds entropy to encourage better exploration.A2C is the synchronous version of A3C and can use a single agent or wait for multiple trajectories.

III. PRELIMINARIES OF IMITATION LEARNING

Imitation learning learns driving policies from expert demonstrations, offering a simpler alternative when interaction with real environments is dangerous and human driving data are available.

  • Learning from demonstrations: Imitation learning learns policies from demonstrations collected under an expert policy πE.A demonstration dataset contains trajectories represented as sequences of state-action pairs.
  • Problem formulation: A common optimization strategy learns a policy π* that mimics the expert policy according to a policy-similarity measure.The reviewed methods are divided into three groups.
  • Behavior cloning: Behavior cloning formulates imitation as supervised learning that matches the learned policy πθ to the expert policy πE.Its objective is typically realized by minimizing an L2 loss.
  • Deep imitation learning: Deep imitation learning uses neural networks as policy approximators and has succeeded in end-to-end autonomous-driving systems.Behavior cloning performs well on states covered by training data but generalizes poorly to new states because of compounding action errors.

C. Inverse Reinforcement Learning

Inverse reinforcement learning infers reward functions that make expert behavior optimal, but traditional formulations can be computationally expensive and depend on system-dynamics knowledge.

  • IRL formulation: Inverse reinforcement learning identifies a reward function rθ for which expert behavior is optimal.The problem was first formulated by Ng et al.
  • Reward learning: Maximum-entropy IRL addresses reward ambiguity by maximizing the posterior probability of observing expert trajectories.Trajectory probability is proportional to exp(rθ(ξi)).
  • Reward representations: Neural-network reward functions extend IRL from low-dimensional features to raw high-dimensional state representations.Earlier nonlinear formulations used Gaussian processes or boosting.
  • Computational cost: Traditional IRL repeatedly solves a forward reinforcement-learning problem to evaluate rewards, making the approach inefficient and computationally expensive.Many methods solve an MDP inside each optimization iteration.
  • Practical limitations: These methods require accurate system dynamics and an efficient offline solver, limiting application in complex real-world settings.Guided cost learning addresses unknown dynamics and learns neural-network cost functions through sample-based approximation.
  • Adversarial imitation: GAIL learns policies directly from demonstrations without hand-designed RL rewards or an expensive IRL inner loop.Its generator imitates expert state-action distributions, while the discriminator provides a surrogate reward.
  • Adversarial reward learning: AIRL uses adversarial reward learning to recover reward functions that are robust to changes in dynamics.This work builds on the mathematical connection among GANs, IRL, and energy-based models.

B. Mode 2. Extension of Mode 1 with High-level Command

Mode 2 extends Mode 1 by incorporating high-level planning outputs to guide control, while Mode 3 integrates DRL/DIL into motion planning and predicts future trajectories or paths.

  • Mode 2. Extension of Mode 1 with High-level Command: Mode 2 extends Mode 1 by using high-level planning outputs to guide the control module.Control may use one general model or separate models for distinct behaviors.
  • Mode 2. Extension of Mode 1 with High-level Command: Mode 2 methods encode environmental vehicles and routing on a bird-view map or use branched networks for behavior-specific control.These designs condition control on detected surroundings and high-level commands.
  • Mode 2. Extension of Mode 1 with High-level Command: Mode 2 may require more training data and computation because behavior-specific models and predetermined high-level planning modules add complexity.A general model may not capture diverse behaviors, whereas separate models increase data demand.
  • C. Mode 3. DRL/DIL Integrated Motion Planning: Mode 3 integrates DRL/DIL into motion planning, combining high-level routes or decisions with current perception to predict future trajectories or paths.DIL models are the mainstream implementation choice, while control typically uses PID or MPC.
  • C. Mode 3. DRL/DIL Integrated Motion Planning: DNN-planned trajectories can imitate human trajectories, but their safety and feasibility are not guaranteed.This is a major disadvantage of Mode 3.

D. Mode 4. DRL/DIL Integrated Behavior Planning

Mode 4 integrates DRL/DIL into behavior planning to derive high-level driving policies, typically leaving motion planning and control to traditional methods. The literature highlights a trade-off between behavioral-action diversity and training cost, alongside an imbalance toward control-focused architectures.

  • D. Mode 4. DRL/DIL Integrated Behavior Planning: Mode 4 uses DNNs to decide behavioral actions while subsequent motion-planning and control modules typically use traditional methods.
  • D. Mode 4. DRL/DIL Integrated Behavior Planning: Many Mode 4 studies build on DQN and its variants, while DIL studies for high-level policies remain limited.
  • D. Mode 4. DRL/DIL Integrated Behavior Planning: Fewer behavioral actions reduce training cost but limit control precision and driving-style diversity; sophisticated action spaces increase training cost.
  • D. Mode 4. DRL/DIL Integrated Behavior Planning: Mode 5 uses single or hierarchical DNNs to output actions at multiple planning levels, but hierarchical networks can increase training cost and reduce convergence speed.
  • D. Mode 4. DRL/DIL Integrated Behavior Planning: Most studies integrate DRL/DIL into control, followed by behavior planning; DRL is more common for high-level decisions, whereas DIL is more common for low-level control.
  • D. Mode 4. DRL/DIL Integrated Behavior Planning: Future work may address the imbalance among integration modes and identify new architectures.

V. TASK-DRIVEN METHODS

Task-driven studies formulate DRL/DIL policies through state and input, action and output, and reward designs. Input choices vary by learning paradigm and scenario, while dynamic inputs and empirically designed rewards remain important challenges.

  • V. TASK-DRIVEN METHODS: Task-driven DRL/DIL formulations comprise state and input design, action and output design, and reinforcement-learning reward design.
  • A. State Space and Input Design: Ego-vehicle and road-environment information are more popular inputs than task information and prior knowledge.
  • A. State Space and Input Design: DRL models prefer object data, whereas DIL models prefer pixel data; low-dimensional object inputs make DRL more tractable and accelerate training.
  • A. State Space and Input Design: Urban scenarios more often use task and prior-knowledge inputs, while highway scenarios more often use point and object data.
  • A. State Space and Input Design: Dynamic input sizes are needed because nearby-car and pedestrian counts vary, but standard DRL/DIL methods generally require fixed-size inputs.
  • C. Reinforcement Learning Reward Design: Empirical reward functions commonly combine factors linearly, but expert dependence and reward-balancing difficulty can affect trained-policy performance.

VI. PROBLEM-DRIVEN METHODS

Problem-driven studies examine how DRL/DIL addresses safety, interaction, and uncertainty in autonomous driving. Safety methods modify, combine, or hybridize learning with traditional techniques, while hybrid-method evidence can remain setting-limited.

  • VI. PROBLEM-DRIVEN METHODS: Safety methods are grouped into modified methods, combined methods, and hybrid methods that integrate DRL/DIL with traditional approaches.
  • Safety: Modified methods enhance safety by constraining exploration through safety model checkers based on reachability, probabilistic prediction, or prior constraints.
  • Safety: Combined methods retain standard DRL/DIL learning while adding traditional rule-based methods for safety.
  • Safety: Hybrid methods integrate DRL/DIL with heuristic search or POMDP planning to combine learned policies with traditional planning.
  • Safety: A hybrid method combining DRL and approximate POMDP planning achieved collision-free autonomous navigation in simulated critical traffic scenarios, but experiments were limited to stationary environments.
  • Interaction: Interaction-aware methods model other traffic participants through interactive environment encoding or interactive learning strategies.

1) Interaction Modeling:

Interaction modeling spans MDP, POMDP, and Markov-game formulations, with methods encoding other agents explicitly or through relational neural architectures. Interactive policies also use staged learning, while uncertainty-aware work distinguishes environmental and deep-learning uncertainty.

  • 1) Interaction Modeling:: Interaction scenarios are modeled with MDPs, POMDPs, and Markov games, with additional categorization based on agents’ relative importance.
  • 1) Interaction Modeling:: Explicit interaction methods model other agents’ intentions or cooperation levels as unobservable states that must be inferred.
  • 1) Interaction Modeling:: Relational neural methods represent agents and their interactions through architectures such as graph convolutional reinforcement learning.
  • 1) Interaction Modeling:: Curriculum learning can stage interactive-policy training from decentralized critics that learn individual behavior to centralized critics that enable joint success and smooth traffic.
  • 1) Interaction Modeling:: Increasing the number of agents may require learning new models in these interactive-policy methods.
  • Uncertainty Modeling: Uncertainty-aware methods address AD and deep-learning uncertainty through uncertainty estimation and multimodal driving-behavior learning.
  • Uncertainty Modeling: Traffic-environment uncertainty arises from stochastic, dynamic interactions among agents with distinct behaviors, while driving behavior itself can be multimodal and stochastic.
  • Uncertainty Modeling: Aleatoric uncertainty reflects incomplete environmental knowledge and cannot be reduced through more data, whereas epistemic uncertainty originates from insufficient datasets.

2) Uncertainty Estimation Methods:

The survey reviews uncertainty estimation and multimodal imitation methods for autonomous-driving policy learning. It also emphasizes that DRL/DIL remains far from real-world readiness because of unresolved architecture, task, and algorithm challenges.

  • Aleatoric uncertainty is modeled with heteroscedastic loss, which relaxes prediction-error penalties when estimated data uncertainty is large.The model predicts uncertainty alongside the regression output.
  • Epistemic uncertainty is commonly estimated with MC-dropout or ensembles, trading sampling time against training and storage costs.Both methods use probabilistic reasoning over network weights, with output variance serving as the uncertainty estimate.
  • One policy-learning method unrolls a learned dynamics model while penalizing both policy cost and divergence from the training dataset, estimating aleatoric and epistemic uncertainty.
  • Mixture density networks estimate uncertainty analytically from Gaussian-mixture variance using a single forward pass rather than repeated sampling.
  • Multi-modal imitation methods infer latent behavior modes using mutual information or variational auto-encoding, but unlabeled demonstrations can make the inferred labels ambiguous.
  • DRL and DIL remain far from ready for real-world applications, with challenges at the architecture, task, and algorithm levels.
  • Architectural integration remains challenging because many studies focus on AD-task potential rather than optimizing DRL/DIL-embedded systems for deployment.

B. Formulation of driving tasks

The survey identifies empirical formulation, safety, interaction, uncertainty, and validation as central challenges in DRL/DIL-based autonomous driving. It finds that safety receives substantial attention, while systematic architectural design, interactive behavior, uncertainty handling, and benchmarking remain insufficiently developed.

  • B. Formulation of driving tasks: DRL/DIL task formulations use case-specific states and inputs plus hand-tuned rewards, but lack theoretical support and can produce substantially different policies after design changes.The survey calls for rigorous comparative experiments to identify optimal designs, scene adaptability, and boundary conditions.
  • C. Safe driving policy: Safety is difficult because deep models are hard to interpret and must handle unseen or rarely seen scenarios; traditional methods are commonly combined to provide functional safety.
  • D. Interaction with traffic participants: Interaction-aware research needs systematic study, while interaction-aware DIL is scarce and interaction-aware DRL remains limited to simplified scenarios with few agents.
  • E. Uncertainty of the environment: Uncertainty modeling remains challenging in complex traffic environments, and current approaches often follow deep-learning predictive-uncertainty methods without deeper investigation.
  • E. Uncertainty of the environment: Ignoring multimodal demonstrations can make models learn an average policy rather than scenario-dependent multimodal driving behavior.
  • F. Validation and benchmarks: Validation and benchmarking are insufficient, including comparisons with traditional architectures, systematic comparisons among DRL/DIL architectures, and exhaustive pre-deployment policy validation.Simulation can reduce field testing but provides performance and safety verification only in a statistical sense.
  • VIII. CONCLUSIONS: The survey organizes AD policy-learning research across system architecture, task formulation, and application problems, identifying five integration modes and reviewing safety, interaction, and environmental uncertainty.
  • VIII. CONCLUSIONS: Architecture design remains less investigated than DRL/DIL task capability, while empirical formulations may create stability and robustness challenges during real-world deployment.
Loading 2101.01993v1…