PaperlayerView exact revision on arXiv

Source-linked AI summary

Comprehensive Comparisons of Reinforcement Learning Algorithms in Autonomous Driving

Dechen Xie, Mushuang Liu

arXiv:2610.19429v1cs.LGcs.SYeess.SY

TL;DR

Autonomous vehicles must make safe and efficient decisions in complex, dynamic traffic, but RL methods have rarely been compared under identical conditions. The paper compares four methods on the same unsignalized-intersection problem, finding distinct trade-offs in optimality, safety, stability, and data efficiency.

  • Problem

    Safe and efficient autonomous-vehicle decisions in complex, dynamic traffic remain challenging, and the strengths and limitations of RL methods have rarely been compared under identical conditions.

    Source: §I · p. 1

  • Method

    The paper compares FVI, RCPI, DQN, and D-DDPG on the same unsignalized-intersection decision problem across optimality, safety, training stability, and training and decision cost.

    Source: §I · p. 1

  • Results

    RCPI is closest to the reference and most stable but least data-efficient; D-DDPG ranks second in optimality, leads data efficiency, and is least stable; DQN is the most stable model-free method but has the highest collision rate.

    Source: 3) Training and Decision Cost · p. 8, §VI · p. 9

  • Takeaways & Limitations

    RCPI is suggested when a model is available and training budget is unconstrained, while D-DDPG is suggested when no reliable model is available or training budget is limited.

    Source: C. Method Selection · p. 9

    D-DDPG is not suitable when its policy is updated with newly collected deployment data, since the policy in use can degrade unpredictably.

    Source: C. Method Selection · p. 9

Abstract

from arXiv · show

This paper presents a comprehensive comparison of four representative reinforcement learning (RL) methods in autonomous vehicle (AV) decision-making, including: Fitted Value Iteration (FVI), Rollout Classification Policy Iteration (RCPI), Deep Q-Network (DQN), and a discrete-action adaptation of Deep Deterministic Policy Gradient (D-DDPG). The four methods are applied to the same AV decision-making problem at an unsignalized intersection and are compared in terms of optimality and safety, training stability, and training and decision cost. RCPI is the closest to a reference solution and has the most stable training process. However, it is the least data-efficient. D-DDPG reaches the second-smallest gap to a reference solution with the highest data efficiency, but has the least training stability. DQN is the most stable model-free method but has the highest collision rate. FVI achieves a lower collision rate than DQN, but is less data-efficient and less stable. Based on these results, we provide suggestions for selecting a method according to whether a model is available, the budget for training, and whether stable training is required.

I. INTRODUCTION

RL methods for autonomous-vehicle decision-making have rarely been compared under identical conditions, leaving their relative strengths and limitations unclear. The paper compares four methods at an unsignalized intersection and formulates vehicle decision-making as a deterministic Markov decision process.

I. INTRODUCTION

  • RL represents sequential decision-making by learning a policy to maximize cumulative reward, and driving decisions matter because they affect future traffic situations.Methods can be distinguished by whether they use an environment model and whether they represent the policy explicitly.Source: §I · p. 1
  • FVI, RCPI, DQN, and D-DDPG are compared on the same unsignalized-intersection problem across optimality, safety, training stability, and training and decision cost.FVI and RCPI are model-based, while DQN and D-DDPG are model-free; FVI and DQN are value-based, while RCPI and D-DDPG are policy-based.Source: §I · p. 1

II. PROBLEM FORMULATION

  • The problem considers a controlled vehicle and surrounding vehicles, with a continuous joint state space and a finite action space.The controlled vehicle follows a deterministic policy; its action determines the next state and immediate reward through the transition and reward functions.Source: §II · p. 2
  • The objective is to find a policy that maximizes discounted cumulative reward at every state in the deterministic Markov decision process.The process is Markov because the next state is determined by the current state and action alone.Source: §II · p. 2

III. PRELIMINARIES OF REINFORCEMENT LEARNING

The preliminaries distinguish model-based from model-free learning and value-based from policy-based methods. They describe experience-driven updates and introduce the four neural-network methods compared in the paper.

III. PRELIMINARIES OF REINFORCEMENT LEARNING

  • Policy-based methods commonly alternate policy evaluation using Vπ or Qπ with policy improvement toward actions having higher value.Value-based methods instead estimate an optimal value function and derive a policy from it.Source: §III · p. 2
  • An experience is the state, action, reward, and next-state information used to correct value-function or policy estimates during iterative updates.The paper represents an experience as (s, a, r, s′), with reward r(s, a) and next state f(s, a).Source: §III · p. 2
  • Model-based methods obtain experiences by querying a model, whereas model-free methods execute actions and observe outcomes; in the deterministic setting, model queries yield the same experiences as execution.The paper compares data use by counting experiences.Source: §III · pp. 2–3

IV. REINFORCEMENT LEARNING METHODS

  • The paper introduces FVI, DQN, RCPI, and D-DDPG as neural-network function-approximation methods, using a parameter subscript to identify each network’s parameters.The parameters are ω for FVI, ψ for DQN, η for RCPI, and θ and ϕ for the D-DDPG actor and critic.Source: §IV · p. 3

A. FVI

FVI uses a neural network to estimate the optimal state-value function and relies on an environment model to construct training targets and select actions.

A. FVI

  • FVI is a model-based, value-based method that approximates the optimal state-value function with an online network and a delayed target network.It samples states from a replay buffer and obtains experiences for every action at each sampled state by querying the model.Source: A. FVI · p. 3
  • FVI computes its target by maximizing immediate reward plus discounted target-network value over actions, requiring |A| model-query experiences for each target.Because actions are finite, the maximum is obtained by enumerating them.Source: A. FVI · p. 3
  • FVI derives its policy by maximizing over actions and requires the environment model both during training and at each decision.The policy calculation also evaluates the model-dependent reward and transition outcomes for the available actions.Source: A. FVI · p. 3

B. DQN

DQN is a model-free, value-based method that learns action values from interaction with the environment and uses replay and a target network during training.

B. DQN

  • DQN is a model-free, value-based method that represents action values for every available action with a neural network.It maintains online and target networks, with the target network updated as a delayed copy of the online network.Source: B. DQN · p. 3
  • DQN gathers experiences through ε-greedy interaction, stores them in a replay buffer, and updates its online network using sampled batches.At each training iteration, it executes one action, stores the resulting experience, and samples a batch from the buffer.Source: B. DQN · p. 3
  • A DQN target uses one experience because its next-state action values come from the target network, so the method needs no environment model.The experience can be collected by executing the action in the environment.Source: B. DQN · p. 3

C. RCPI

RCPI learns a policy by repeatedly comparing actions using model-generated rollouts, then deploys the learned policy network without the model.

C. RCPI

  • At each iteration, RCPI samples states from a replay buffer and trains its policy network to predict a target action for each state.The policy selects the action with the highest network score.Source: C. RCPI · p. 4
  • For every sampled state and possible action, RCPI simulates a trajectory that begins with that action and then follows the current policy to estimate its value.With a finite action space, each target requires |A|H experiences, where H is the trajectory length.Source: C. RCPI · p. 4
  • RCPI is model-based during training, but the final policy is obtained from the network alone and needs no model at deployment.Source: C. RCPI · p. 4

D. D-DDPG

D-DDPG adapts an actor-critic method to discrete actions by using a critic to learn action values and a straight-through estimator to train the actor. It learns from environment interactions and deploys the actor without a model.

D. D-DDPG

  • D-DDPG is a model-free actor-critic adaptation of DDPG for discrete actions, with a policy network as actor and a critic that learns action values.Source: D. D-DDPG · p. 4
  • The straight-through Gumbel-Softmax estimator passes actor gradients through a softmax over action scores, while one-hot action vectors provide critic inputs.Training adds independent Gumbel noise to action scores, selects the largest perturbed score, and uses the tempered softmax gradient in the backward pass.Source: D. D-DDPG · p. 4
  • D-DDPG updates online actor and critic networks using replayed experience batches and target networks that move toward online parameters by soft updates.Source: D. D-DDPG · pp. 4–5
  • Each target requires one environment experience, and the learned policy comes from the actor alone, so D-DDPG needs no model at deployment.Source: D. D-DDPG · p. 5

V. NUMERICAL RESULTS

The numerical-results section compares four RL methods in a simulated unsignalized intersection across optimality and safety, training stability, and training and decision cost.

V. NUMERICAL RESULTS

  • The comparison evaluates four RL methods in a simulated unsignalized intersection using optimality and safety, training stability, and training and decision cost.Source: §V · p. 5
  • Optimality is measured by the discounted-reward gap from a reference solution, while safety is measured by the learned policy’s collision rate.Source: §V · p. 5
  • Training stability is measured by the reward drop from its peak to the end of training, training cost by experiences needed to reach reward levels, and decision cost by time per state.Source: §V · p. 5
  • The section proceeds from simulation setup to method comparisons and then recommendations for selecting a method.Source: §V · p. 5

1) Environment Setup:

The environment models five vehicles moving straight through an unsignalized intersection, with vehicle 1 controlled by the policy. The task is to maximize discounted reward under a finite acceleration action space.

1) Environment Setup:

  • The simulation has five vehicles crossing the intersection straight; vehicle 1 is controlled, while vehicles 2–5 move at constant speed.The objective is to maximize discounted reward with γ = 0.99.Source: 1) Environment Setup · p. 5
  • Vehicle states use a fixed Cartesian frame centered at the intersection, with x pointing east and y north; each vehicle’s state includes position and velocity.The joint state comprises the states of all five vehicles and lies in R20.Source: 1) Environment Setup · p. 5
  • Simulation steps are 0.25 s and end when the controlled vehicle clears the intersection, a collision occurs, or 20 s elapse; collision means distance below 0.5 m.Source: 1) Environment Setup · p. 5
  • The controlled vehicle moves along the positive y-axis, and its finite action is longitudinal acceleration from −4 to 4 m/s2 in nine possible values.Source: 1) Environment Setup · pp. 5–6
  • The reward penalizes deviation from a desired speed of 5 m/s and proximity to surrounding vehicles, weighted by α = 6 and β = 100.A 0.01 offset prevents division by zero.Source: 1) Environment Setup · p. 6

2) Training Setup:

The methods are trained across four intersection layouts of increasing conflict complexity, using method-appropriate replay data and a shared network backbone. Evaluation compares learned policies with a heuristic reference on optimality and safety, while tracking training stability, experience use, and decision time.

2) Training Setup:

  • A single policy is trained across C1–C4, whose one-to-four conflicting vehicles progressively reduce the controlled vehicle’s feasible safe decisions.Vehicle 1 is controlled; surrounding vehicles with crossing future paths are classified as conflicting.Source: 2) Training Setup · p. 6, Figure 2 · p. 7
  • Model-based FVI and RCPI replay states from uniformly randomized trajectories, while model-free DQN and D-DDPG replay action experiences collected by their respective policies.All methods use a FIFO replay buffer and uniformly sample 64 states or experiences per iteration.Source: 2) Training Setup · p. 6
  • All methods share a two-hidden-layer, 256-unit ReLU backbone, while method-dependent input and output layers encode their value, Q, policy, or actor-critic functions.The D-DDPG critic additionally takes a one-hot action input.Source: 2) Training Setup · p. 6, Figure 3 · p. 7
  • Each method is trained for 2.5 × 10^6 network updates across five random seeds.For D-DDPG, one update includes an update to both critic and actor parameters.Source: 2) Training Setup · p. 6

3) Evaluation Setup:

  • Policies are selected by their best mean discounted reward on a 500-state validation set, then evaluated on a separate 4,000-state test set sampled evenly across layouts.The selected checkpoint is chosen separately for each method and seed.Source: 3) Evaluation Setup · p. 6
  • Optimality is measured by reward gap to a Differential Evolution reference, while safety is measured by the learned policy’s collision rate.The reference searches 80-step action sequences and is near-optimal rather than exact; surrounding-vehicle collisions are excluded from the collision rate.Source: 3) Evaluation Setup · p. 6
  • Training stability is assessed from validation learning curves and the normalized drop from the selected peak reward to the mean reward of the final ten checkpoints.A large drop indicates that the training peak is not retained.Source: 3) Evaluation Setup · p. 6
  • Training and decision costs are compared using experience counts at specified reward thresholds and peak performance, plus time per decision.Decision time is measured on 100 states across the layouts; the report gives mean and standard deviation, with Differential Evolution timing based on its action-sequence optimization.Source: 3) Evaluation Setup · pp. 6–7

B. Comparison Results

RCPI leads on optimality and safety and has the steadiest training, while the methods differ in sensitivity to layout complexity and in how their training targets affect stability.

1) Optimality and Safety:

  • From C1 to C4, gap increases are 1.71 for RCPI, 11.06 for D-DDPG, 20.04 for FVI, and 28.02 for DQN.FVI’s gap grows approximately linearly with conflicting vehicles, while DQN’s accelerates; DQN’s gap is smaller than FVI’s in C1–C3 but larger in C4.Source: 1) Optimality and Safety · p. 7
  • Across 20,000 trajectories, RCPI, FVI, D-DDPG, and DQN have about 1, 20, 26, and 79 collisions, respectively, so a smaller gap does not imply a lower collision rate.Collisions are rare, and their effect is diluted in mean discounted reward.Source: 1) Optimality and Safety · p. 7

2) Training Stability:

  • RCPI’s trajectory-summed target is exact in the deterministic environment, whereas the other methods’ network-estimated targets carry value errors forward; D-DDPG’s actor and critic errors can reinforce each other.This may explain D-DDPG’s large oscillations and later decline.Source: 1) Optimality and Safety · p. 7, 2) Training Stability · p. 8
  • RCPI approaches the DE reference within about 0.5×10^6 updates and maintains that level; D-DDPG later declines and oscillates, with the largest drop of 0.946.The figure tracks discounted reward against network updates, and Table III reports peak-to-end reward drops over five seeds.Source: 2) Training Stability · p. 7, Figure 4 · p. 8, Table III · p. 8

1) Optimality and Safety:

2) Training Stability:

  • DQN begins with the lowest discounted reward and oscillates strongly early, while FVI improves quickly before plateauing below the reference and declining from its peak across seeds.DQN then improves slowly but continuously and has the second-smallest peak-to-end drop.Source: 2) Training Stability · pp. 7–8

3) Training and Decision Cost:

The methods trade off experience use, decision cost, safety, and training stability, so the paper recommends choosing according to model availability and training constraints. RCPI is most data-intensive, while D-DDPG is most data-efficient; no method is collision-free.

3) Training and Decision Cost:

  • RCPI is least data-efficient, using 3.62×10^10 experiences to reach its peak—about 660 times as many as D-DDPG.D-DDPG reaches each threshold with the fewest experiences, followed by DQN and FVI; RCPI requires the most at every threshold.Source: 3) Training and Decision Cost · p. 8
  • D-DDPG uses each experience to update both actor and critic, while FVI requires |A| experiences per target and RCPI requires |A|H.DQN also requires one experience per target, but an experience updates only the Q estimate for the action it contains.Source: 3) Training and Decision Cost · p. 9
  • FVI takes longer to select an action than RCPI, DQN, and D-DDPG because it enumerates actions and evaluates the next-state value for each.All four methods require much less decision time than DE; RCPI, DQN, and D-DDPG have similar decision times.Source: 3) Training and Decision Cost · p. 9

C. Method Selection

  • RCPI has the smallest optimality gap, lowest collision rate, most stable training, and least sensitivity to layout complexity, but needs a model and the most experiences.It is suggested when a model is available and the training budget is not constrained.Source: C. Method Selection · p. 9
  • D-DDPG ranks second in optimality and is most data-efficient but least stable; DQN is the most stable model-free method but has the highest collision rate.D-DDPG is suggested when no reliable model is available or training resources are limited; DQN when stable training is required without a reliable model.Source: C. Method Selection · p. 9

VI. CONCLUSIONS

  • FVI is a compromise when a model is available during training and deployment but the budget does not allow RCPI; none of the four methods is collision-free.The comparison uses a deterministic environment with the exact model; stochastic traffic and approximate models are left for future work.Source: C. Method Selection · p. 9, §VI · p. 9

Related papers

Loading 2610.19429v1…