PaperlayerView exact revision on arXiv

Source-linked AI summary

Autonomous Highway Driving using Deep Reinforcement Learning

Subramanya Nageshrao, Eric Tseng, Dimitar Filev

arXiv:1904.00035v1cs.ROcs.LGcs.SYeess.SYstat.ML

TL;DR

Diverse, unforeseen driving scenarios challenge fixed maneuver rules and a-priori optimization, while unsafe exploration can demand many samples. The paper trains a DDQN highway-driving agent with a short-horizon safety check; with that check it converged to an optimal policy in the reported task, whereas standard DDQN always collided.

  • Problem

    Diverse, unforeseen driving scenarios make fixed maneuver rules or an a-priori cost function potentially unsuitable, and an RL agent without explicit safety may need many samples before finding a reliable policy.

    Source: §I · p. 2, C. Need for safety · p. 3

  • Method

    The paper trains a DDQN highway-driving agent through interaction with simulated traffic and augments it with a short-horizon safety check that can replace an unsafe action when a safe alternative exists.

    Source: §I · p. 1

  • Results

    With an explicit safety check, DDQN converged to an optimal policy with average reward per decision around −0.025; standard DDQN without the check could not learn a decent policy and always collided.

    Source: C. Decision making for ego vehicle · p. 6

  • Takeaways & Limitations

    The authors report that the learned controller performed under varying traffic density and that the adaptation framework reduced the number of safety triggers.

    Source: §V · p. 7

Abstract

from arXiv · show

The operational space of an autonomous vehicle (AV) can be diverse and vary significantly. This may lead to a scenario that was not postulated in the design phase. Due to this, formulating a rule based decision maker for selecting maneuvers may not be ideal. Similarly, it may not be effective to design an a-priori cost function and then solve the optimal control problem in real-time. In order to address these issues and to avoid peculiar behaviors when encountering unforeseen scenario, we propose a reinforcement learning (RL) based method, where the ego car, i.e., an autonomous vehicle, learns to make decisions by directly interacting with simulated traffic. The decision maker for AV is implemented as a deep neural network providing an action choice for a given system state. In a critical application such as driving, an RL agent without explicit notion of safety may not converge or it may need extremely large number of samples before finding a reliable policy. To best address the issue, this paper incorporates reinforcement learning with an additional short horizon safety check (SC). In a critical scenario, the safety check will also provide an alternate safe action to the agent provided if it exists. This leads to two novel contributions. First, it generalizes the states that could lead to undesirable "near-misses" or "collisions ". Second, inclusion of safety check can provide a safe and stable training environment. This significantly enhances learning efficiency without inhibiting meaningful exploration to ensure safe and optimal learned behavior. We demonstrate the performance of the developed algorithm in highway driving scenario where the trained AV encounters varying traffic density in a highway setting.

I. INTRODUCTION

Autonomous driving decisions can be computationally demanding under conventional path-planning optimization, while diverse traffic motivates a learning-based approach. The paper develops a DDQN highway-driving agent augmented with safety checks, constrained exploration, and adaptation.

I. INTRODUCTION

  • A short-horizon safety check replaces an unsafe DDQN action with an alternate safe action when one is feasible.The check is also available as an override during inference.Source: §I · p. 1
  • Safe and non-safe transitions are stored in separate buffers, supporting constrained exploration and more stable, efficient learning.The two-buffer approach is described as a simpler version of prioritized experience replay [12].Source: §I · pp. 1–2
  • The inference-phase safety controller supports continuous adaptation by allowing the learned agent to be retrained with new information.The new information could potentially come from real-world data collected by deploying the initial agent in multiple vehicles.Source: §I · p. 2
  • The algorithm is demonstrated in simulated highway driving with varying traffic density.The paper also presents the work as applying RL to scenarios not easily addressed by fixed decision rules or an a-priori cost function.Source: §I · pp. 1–2
  • The highway-driving architecture combines a DDQN decision maker with standard state-feedback controllers.The agent learns by interacting directly with diverse simulated traffic.Source: §I · pp. 1–2

II. PRELIMINARIES: DEEP LEARNING FOR CONTROL

The preliminaries frame reinforcement learning as policy learning through interaction, then distinguish high-level decisions from low-level actuation. The paper assigns those roles to DDQN and classical feedback control, respectively.

A. Reinforcement learning

  • An RL problem is represented as an MDP with state space, action space, transition function, and user-defined reward function.The agent interacts with the system to learn behavior with respect to the reward or cost.Source: A. Reinforcement learning · p. 2
  • At each time step, the policy selects an action from the current state; the system transitions, returns a reward, and episodes end at a terminal state or sample limit.The description assumes the system has the Markovian property.Source: A. Reinforcement learning · p. 2
  • RL seeks a policy that maximizes accumulated discounted reward, with discrete-action optimal policies obtained through the optimal Q function and Bellman equation.The supplied passage identifies γ as the discount factor.Source: A. Reinforcement learning · p. 2

B. Decision making and low-level actuation

  • The paper separates high-level decision making from low-level actuation: RL with modified DDQN handles decisions, while classical feedback control such as PID handles actuation.The authors say learning both simultaneously may require substantial training data; they describe their hierarchy as overlapping with hierarchical RL [16].Source: B. Decision making and low-level… · p. 2

C. Need for safety

Unguided RL exploration can produce collisions or near-misses, wasting learning effort and potentially requiring resets. The paper adds a short-horizon safety check to constrain risky actions during both learning and deployment.

C. Need for safety

  • Unguided exploration can lead to highway collisions or near-misses, simulation resets, and slower learning; physical-platform exploration may also be costly.Function approximation can leave a trained Q-network with a small probability of choosing an unsafe maneuver even after convergence.Source: C. Need for safety · p. 3
  • A standard DDQN may need many samples to discover dangerous state-action pairs, such as changing right from the right-most lane beside a barricade.The paper says this can waste learning effort on irrelevant regions of the state and action space.Source: C. Need for safety · p. 3
  • The short-horizon safety check evaluates the agent’s action and supplies an alternative safe action whenever feasible, using common-sense road rules to avoid unsafe actions in dangerous situations.The authors compare its role to a teacher providing corrective action, while noting that the safety controller may not be optimal or expert.Source: C. Need for safety · p. 3

III. SAFE AND ADAPTIVE DECISION MAKING FOR AV

The highway-driving agent combines DDQN maneuver decisions with feedback control and a short-horizon safety check. Its state represents nearby traffic from the ego vehicle’s perspective, and safety rules can override unsafe actions during learning and deployment.

III. SAFE AND ADAPTIVE DECISION MAKING FOR AV

  • The architecture pairs DDQN decisions with a short-horizon safety check and low-level feedback control.The safety check validates DDQN actions and can provide an alternate safe action when feasible.Source: §III · pp. 3–4, Figure 2 · p. 4
  • The state uses 24 indicators for the six nearest traffic vehicles, plus the ego vehicle’s longitudinal velocity, lateral position, and lateral velocity.The traffic indicators encode relative distances and velocities for nearby front and rear vehicles in each of three lanes.Source: §III · p. 3
  • DDQN selects among 12 actions formed from four longitudinal choices and three lateral choices.Longitudinal choices are maintain, accelerate, brake, and hard brake; lateral choices are lane keep or change lanes left or right.Source: §III · p. 4
  • The safety check enforces minimum gaps based on relative velocity and time to collision, and can substitute a safe longitudinal action when feasible.It also blocks lane changes at lane boundaries and monitors vehicles in the target lane, aborting a lane change when the safety condition fails.Source: §III · p. 4
  • During deployment, new safety-violation data can be added to the collision buffer and used to retrain or adapt the learned agent.The paper describes this as continuous adaptation using new information, potentially from real-world data collected across multiple vehicles.Source: §III · p. 4
  • During learning, unsafe actions and collisions are stored in a collision buffer, while safe transitions go to a separate buffer for DDQN training.The algorithm replaces an unsafe action with a safe alternative when available and samples minibatches from both buffers.Source: §III · p. 5

IV. USE OF DRL FOR DECISION MAKING

This section applies the DRL decision-making algorithm to autonomous highway driving, introducing the training vehicle model and simulation environment before evaluating the learned policy.

IV. USE OF DRL FOR DECISION MAKING

  • The section presents the vehicle dynamical model used for training, the training environment, and an evaluation of the learned policy.The passage previews these topics without reporting evaluation results.Source: §IV · p. 5

A. Vehicle dynamics

Vehicles use a computationally efficient point-mass model, with discrete-time longitudinal and lateral motion models. Longitudinal acceleration and lane-change timing are specified for training.

A. Vehicle dynamics

  • Each vehicle uses a computationally efficient point-mass model with discrete-time longitudinal double-integrator and lateral kinematic motion.The longitudinal model uses position and velocity, while the lateral model uses lateral position and velocity.Source: A. Vehicle dynamics · p. 5
  • Longitudinal acceleration takes four values, including nominal acceleration, zero, braking, and hard braking; lane changes take five seconds and may be aborted at each sampling instance.The specified accelerations are a1 = 2 m/s2 and a2 = 4 m/s2, with hard braking reserved for emergencies; sampling is at 1 Hz.Source: A. Vehicle dynamics · p. 5

B. Simulation environment

Training uses a three-lane circular-loop simulation to approximate an infinite straight highway. Traffic vehicle controllers and randomized parameters are used to create diverse scenarios.

B. Simulation environment

  • The training environment is a three-lane circular loop, with 1 to 30 cars placed randomly within 250 m of the ego car at each episode’s start.The 30-car maximum is the setting used in this example.Source: B. Simulation environment · p. 5, Figure 3 · p. 5
  • Traffic vehicles use a combination of controllers, may change lanes randomly, and have randomized system parameters to create diverse training and evaluation scenarios.The simulated traffic vehicles account for the relative distance and velocity to the preceding vehicle and are assumed not to rear-end it in the same lane.Source: B. Simulation environment · p. 5

C. Decision making for ego vehicle

The section defines a reward over speed, lane, and safe following distance, then evaluates DDQN training and highway speed against safety and model-based baselines.

D. Continuous adaptation

During implementation, the learned policy replaces exploratory action selection, and retraining uses additional data from safety-check failures. Continuous adaptation reduces safety triggers, but they do not disappear.

  • With continuous adaptation, the mean number of safety triggers decreased monotonically to a smaller value, unlike the no-adaptation case, but never reached zero.Source: D. Continuous adaptation · p. 7
  • The authors suggest persistent triggers may reflect function approximation selecting non-safe actions or rigid, static safety rules.Source: D. Continuous adaptation · p. 7

V. CONCLUSIONS

The paper presents a control-architecture-based deep RL framework for safe AV decision-making, augmenting a Q-network with short-horizon safety control. The learned controller was evaluated under varying traffic density, and continuous adaptation reduced safety triggers.

  • The framework augments the Q-network with short-horizon safety control that embeds some common rules of the road in AV decision-making.Source: §V · p. 7
  • Continuous adaptation was shown to reduce the number of safety triggers.Source: Figure 8 · p. 7, §V · p. 7
  • The learned controller was evaluated under varying traffic density, and the results demonstrated its superior capabilities.Source: §V · p. 7
  • Future work is planned on mandatory lane changes and highway on-ramp-to-off-ramp driving.Source: §V · p. 7

APPENDIX

The appendix identifies Table I as listing the parameter values used in the work.

Loading 1904.00035v1…