Source-linked AI summary
Automated Speed and Lane Change Decision Making using Deep Reinforcement Learning
Carl-Johan Hoel, Krister Wolff, Leo Laine
TL;DR
Autonomous vehicles need decision-making methods that can handle complex and changing traffic situations without manually coding every case. The paper trains a DQN in simulation to control speed and lane changes, achieving performance on par with or better than an IDM–MOBIL reference in highway driving and applying the same method to oncoming traffic. The study also introduces a CNN design for interchangeable-object inputs and reports collision-free handling in both cases.
Problem
Existing autonomous-driving methods commonly target specific driving cases, motivating a general method that can learn decision behavior across different environments.
Method
A DQN agent is trained in a simulated environment to generate speed and lane-change decisions, using a CNN applied to high-level input representing interchangeable objects.
Results
The method performed on par with or better than the IDM–MOBIL reference in highway driving, also handled an oncoming-traffic case, and trained CNN agents completed episodes without collisions.
Takeaways & Limitations
The results support using one learned decision-making method across the presented driving cases and combining lateral and longitudinal decisions for better performance.
Takeaways & Limitations
The agent can solve only situations represented in its training simulations, and functional safety is difficult to guarantee without an underlying safety layer.
Abstract
from arXiv · showhide
This paper introduces a method, based on deep reinforcement learning, for automatically generating a general purpose decision making function. A Deep Q-Network agent was trained in a simulated environment to handle speed and lane change decisions for a truck-trailer combination. In a highway driving case, it is shown that the method produced an agent that matched or surpassed the performance of a commonly used reference model. To demonstrate the generality of the method, the exact same algorithm was also tested by training it for an overtaking case on a road with oncoming traffic. Furthermore, a novel way of applying a convolutional neural network to high level input that represents interchangeable objects is also introduced.
I. INTRODUCTION
The paper addresses the difficulty of manually coding decisions across complex driving situations by training a DQN agent in simulation. It targets a general-purpose function for speed and lane-change decisions across highway and oncoming-traffic cases.
- Manual coding of all possible traffic situations would be time-consuming and error-prone, motivating behavior learned from experience.
- Many existing autonomous-driving methods target one specific driving case, requiring a different method for settings such as roads with oncoming traffic.
- A previous genetic-algorithm approach handled different cases but still required manually defined features to adapt its rules and actions.
- The proposed DQN learns a decision-making function in simulation without hand-crafted features and is intended to generalize across driving cases.
- In highway driving, the method outperformed the IDM and MOBIL reference combination, while the same untuned method was applied to an oncoming-traffic setting.
A. Reinforcement learning
Reinforcement learning models decision making as an agent learning actions from states and rewards, while DQN approximates optimal action values with a deep neural network. Double DQN further separates action selection from evaluation to address value overestimation.
- Core concepts: Reinforcement learning trains an agent to learn a policy that selects actions from states to maximize cumulative reward.The environment transitions to a new state and returns a reward after each action.
- Q-learning: Q-learning learns the optimal action value function Q∗(s, a), representing the maximum expected return for an action followed by the optimal policy.
- Q-learning: The Bellman equation supports selecting the action with the highest expected optimal action value.
- Deep Q-Networks: DQN approximates Q∗(s, a) with a deep neural network and updates its weights by minimizing Bellman-equation error on sampled experiences.Training typically uses stochastic gradient descent with mini-batches.
- Deep Q-Networks: DQN stabilizes learning with periodically fixed target-network parameters and balances exploration and exploitation through an ϵ-greedy policy.Random actions are selected with probability ϵ; otherwise the highest-valued action is chosen.
- Double DQN: Double DQN aims to reduce action-value overestimation by decoupling action selection from action evaluation.
C. Agent implementation
The vehicle agents use Double DQN control in a simulated environment, with a shared state representation and different action responsibilities. Their reward encourages distance traveled while penalizing collisions, near collisions, and leaving the road.
- Implementation overview: The implemented Double DQN agents were applied to vehicle control in two test cases described later in the paper.
- State representation: Because other road users’ intentions are unobservable, the decision problem is modeled as a POMDP and approximated using the latest observation with k = 1.
- State representation: Both agents receive the same 27-element state vector containing ego-vehicle speed, available lanes, and the states of eight surrounding vehicles.
- Action spaces: Agent1 controls lane changes while IDM controls speed, enabling direct comparison with MOBIL lane-change decisions under the same speed-control model.
- Action spaces: Agent2 controls lane changes and speed by selecting among four acceleration options, including full brake, medium brake, maintain speed, and acceleration.
- Reward function: The reward normally scales distance traveled as ∆d/∆dmax, while collisions, leaving the road, and near collisions receive −10 penalties.Collision or leaving-road events terminate the episode; near-collision events do not.
2) Neural network design:
The study compares fully connected and convolutional network designs for mapping vehicle-state inputs to action values. The CNN architecture applies convolutions and max pooling to interchangeable surrounding-vehicle inputs before combining them with the remaining state.
- Architecture overview: Both architectures use 27 input neurons, while Agent1 and Agent2 produce 3 and 6 action-value outputs, respectively.Each output neuron represents Q(s, ai) for one available action.
- Fully connected architecture: The first design is a fully connected network with two 512-neuron hidden layers using ReLU activations.Its final output layer uses linear activation.
- Convolutional architecture: The second design applies CNNs to high-level state features representing identical, interchangeable surrounding vehicles.The vehicle features encode relative position, speed, and lane.
- Convolutional architecture: Convolutional filters process each vehicle’s three features separately, aggregate them, and max-pool across vehicles to create translational invariance.The first convolution uses filter size 3 and stride 3; the second uses filter size 1 and stride 1.
- Output integration: The pooled vehicle representation is concatenated with the remaining input and passed through a 64-unit fully connected layer before the 3- or 6-unit output layer.The fully connected layer uses ReLU activation and the output layer uses linear activation.
3) Training details:
Training uses Double DQN mechanisms to stabilize learning from replayed experience, while IDM and MOBIL provide the reference driving model. The setup also specifies assumptions about hyperparameter selection and episode-end handling.
- DQN training: The agents are trained with Double DQN, ϵ-greedy exploration, discounted future rewards, a periodically cloned target network, replay memory, mini-batches, RMSProp, and error clipping.The target network is updated by cloning the online parameters at scheduled iterations.
- Training choices: Hyperparameters were selected through an informal search based on prior work rather than a systematic grid search.The authors cite computational complexity as the reason systematic grid search was not performed.
- Episode handling: The state omits longitudinal episode position because the goal is highway driving of infinite length, and the final successful experience is excluded from replay memory.This prevents learning from the terminal return of zero.
- Reference model: The IDM models longitudinal speed using the distance to the vehicle ahead and the speed difference, with parameters taken from the original IDM paper.The MOBIL model uses IDM-predicted accelerations for lane-change decisions.
- Reference model: MOBIL first requires the induced acceleration of the following vehicle in the target lane to satisfy an > −bsafe, then applies a collective acceleration-gain threshold.Its politeness factor controls how effects on other vehicles are valued.
1) Highway case:
The main evaluation uses a simulated three-lane highway with a truck-trailer ego vehicle surrounded by eight cars. Randomized speed trajectories create situations requiring overtaking while accounting for faster vehicles approaching from behind.
- Evaluation: The highway case is the paper’s main test of the proposed method and is evaluated against the IDM–MOBIL reference model.The simulated highway parameters are summarized in Table V.
- Scenario: The highway simulation contains a three-lane road, a 16.5 m truck-semitrailer ego vehicle, and eight 4.8 m passenger cars.Surrounding vehicles remain in their initial lanes and follow IDM longitudinally; overtaking is allowed on either side.
- Traffic generation: Training includes occasional hard braking and rapid acceleration so the agent encounters quick speed changes while learning safe inter-vehicle distances.These changes are included because normal highway traffic is usually more constant but can contain abrupt maneuvers.
- Traffic generation: Six example trajectories use solid lines for fast vehicles behind the ego vehicle and dashed lines for slow vehicles ahead.The trajectories are randomly generated for different highway positions.
- Traffic generation: Vehicles ahead receive slower speed trajectories and vehicles behind receive faster ones, producing overtaking demands alongside approaching rear traffic.Episodes with unavoidable collisions caused by overly close placement and large speed differences are deleted.
2) Overtaking case:
A secondary scenario tests the same decision-making approach on a two-way road where the ego truck overtakes a slower vehicle while accounting for oncoming traffic. Vehicle motion and performance are evaluated through simulated trajectories and distance-speed indices.
- Scenario: The overtaking scenario starts the ego vehicle in the right lane at 25 m/s, with a slower vehicle 50 m ahead and two oncoming vehicles 300–1100 m ahead.All surrounding vehicles follow slow speed profiles.
- Vehicle control: Both scenarios simulate vehicle motion with kinematic models and use a two-point visual-control model for lateral lane following.A lane-change decision changes the lateral controller’s desired-lane setpoint, and lane changes normally take 2 to 3 s.
- Performance measurement: In the highway case, the performance index multiplies normalized driven distance by the ratio of agent average speed to reference-model average speed.A value greater than 1 indicates performance better than the reference model.
- Performance measurement: For overtaking, the performance index compares normalized driven distance and ego average speed against IDM control without overtaking the preceding vehicle.The IDM-only mean speed supplies the overtaking-case reference.
- Results reporting: Table VI summarizes results for the different agents across both the highway and overtaking cases.The supplied passage identifies the table as covering both tasks but gives no numerical entries.
IV. RESULTS
The CNN agents learned highway speed and lane-change decisions from evaluation experience. Agent1CNN eventually matched the reference model, while Agent2CNN achieved collision-free performance with a stabilized performance index of 1.1.
- Evaluation setup: The study evaluated four agent variants across five random-seed runs, training them for 2 million iterations and evaluating 1,000 unseen random episodes every 50,000 iterations.Agent2FCNN was trained for 3 million iterations.
- CNN agents: Agent1CNN solved all episodes after 100,000 iterations by staying in its lane, then learned necessary lane changes and performed similarly to the reference model after about 600,000 iterations.Its initial performance index was below 1 because slower vehicles blocked it.
- CNN agents: Agent2CNN reached the reference performance index of 1 at about 250,000 iterations and solved all evaluation episodes without collisions at 400,000 iterations.With further training, its performance index increased and stabilized at 1.1 while remaining collision-free.
- Final performance: For the final Agent1CNN and Agent2CNN models, most collision-free episodes had average speeds close to the reference model, with both faster and slower outliers.Because all episodes were collision-free, the performance index was the speed ratio v̄/v̄ref.
- Learned behavior: Agent1CNN initially selected staying in its lane almost 100% of the time, later choosing lane changes about 1% of the time as it learned when they were safe.Agent2CNN first accelerated for immediate reward, causing rear-end collisions, then learned braking, idling, and necessary lane changes; both agents changed left and right equally often.
B. Agents using a FCNN
The fully connected agents were less reliable than the CNN agents in highway evaluation, while the same algorithm generalized to overtaking with oncoming traffic. The CNN overtaking agents ultimately completed every episode without collisions and overtook the slower vehicle in every episode.
- FCNN highway results: Both Agent1FCNN and Agent2FCNN failed to complete all highway evaluation episodes without collisions.After 20 million iterations, their results were unchanged.
- FCNN highway results: Agent1FCNN outperformed Agent2FCNN because it only decided when to change lanes, whereas Agent2FCNN also controlled speed.Agent2FCNN collided in 14% of episodes by the end of training.
- Overtaking case: The overtaking evaluation measured successfully completed episodes and the modified performance index for Agent1CNN and Agent2CNN.These evaluations tested the method in a different setting from highway driving.
- Overtaking case: By the end of overtaking training, both CNN agents solved all episodes without collisions, and the ego vehicle overtook the slower vehicle in every episode.Their performance indexes were above 1 in all episodes.
V. DISCUSSION
The method outperformed or matched the reference in highway driving, generalized to oncoming-traffic overtaking, and benefited from CNNs for interchangeable high-level inputs. Its behavior depends on reward design and exposure to representative simulated situations, motivating broader evaluation.
- Performance and decision coupling: Both Agent1CNN and Agent2CNN completed all episodes without collisions; Agent1CNN matched the reference, while Agent2CNN outperformed it by jointly controlling speed and lane changes.The comparison suggests lateral and longitudinal decisions should not be completely separated for better performance.
- CNN architecture: CNN architectures significantly outperformed FCNN architectures by providing translational invariance and reducing network complexity through shared weights.The paper extends CNN use from image pixels to high-level inputs representing interchangeable objects.
- Reward design: A simple reward function worked in this study, but other cases may require more careful design because reward choices strongly affect resulting behavior.Without a lane-change penalty, the agent repeatedly demanded opposite lane changes and drove between lanes.
- Generality: Unlike the earlier genetic-algorithm method, the presented method uses measured state directly and requires no hand-crafted features when adapting to different driving cases.The earlier approach remained general but required manually defined features for its rules and actions.
- Scope and safety: The trained agent can solve only situations represented in its simulations, so the simulated traffic environment must cover the intended driving case.The paper also notes that machine-learned decision functions make functional safety difficult to guarantee, motivating an underlying safety layer.
- Future work: Future work should test additional cases such as crossings and roundabouts and systematically study parameters and network architectures.Prioritized experience replay is also proposed as a possible way to improve and accelerate learning.