Automated Lane Change Decision Making using Deep Reinforcement Learning in Dynamic and Uncertain Highway Environment

• Tactical lane changing has become a focus as autonomous-driving research moves beyond steady-state maneuvers, and is required in both fully and partially autonomous systems.Source: §I · p. 1
• Highway lane changes require tactical-to-strategic decisions that account for complex traffic, goals such as reaching an exit, congestion avoidance, and safety.Local control approaches can overlook highway traffic.Source: §I · p. 1
• Machine learning can handle unforeseen situations after training, while rule-based methods tune predefined parameters to a specific environment.Source: §I · p. 1
• Prior deep RL lane-changing work used an occupancy grid but omitted vehicle dynamics, while another approach included vehicle dynamics and a lateral controller but evaluated deterministic, simple single-lane traffic.The paper identifies these simplifications as limits on realism and practical validity.Source: §I · p. 1

A. Our contribution

• The deep RL agent is designed to make safe lane changes in dynamic highway traffic while minimizing estimated time of arrival.Source: A. Our contribution · p. 1
• The simulation combines vehicle dynamics, a lateral controller, IDM adaptive cruise control, and MOBIL safe lane changing across varied scenarios and uncertainty settings.Source: A. Our contribution · p. 2
• The approach is reported to perform robustly across uncertain, noisy traffic scenarios, while MOBIL is sensitive to measurement noise and uncertainty.Source: A. Our contribution · p. 2
• Multiple cars can make lane- and speed-change decisions while the agent receives noisy observations, enabling richer traffic scenarios than settings without surrounding-vehicle behavior.Source: A. Our contribution · p. 2

A. Deep Q-Learning with real-time validation

• Deep Q-learning was selected for its model-free, off-policy formulation and discrete action space, with prior promise in planning and autonomous-system control.Source: A. Deep Q-Learning with real-time… · p. 2
• Validation periodically evaluates the agent on unseen scenarios and records the network weights when mean reward exceeds the previous maximum.Two validation periods with different episode counts are used to balance faster training and model generalization.Source: A. Deep Q-Learning with real-time… · p. 2

III. SIMULATION SETUP

• The simulation setup describes initialization, vehicle kinematics, a two-point steering model, and the IDM and MOBIL algorithms.Source: §III · p. 2

A. Initialization

• Random initial positions, speeds, desired speeds, and episode length define the highway setup, with a minimum inter-vehicle distance between vehicles.The middle vehicle is selected as the ego vehicle; parameter values are listed in Table I.Source: A. Initialization · p. 2, Table I · p. 2, Figure 1 · p. 3

B. Vehicle and Steering Control Model

• A nonlinear kinematic bicycle model uses IDM for acceleration and a two-point visual controller for the steering angle.The controller computes steering from near- and far-point angles using PI tuning parameters.Source: B. Vehicle and Steering Control Model · p. 3
• MOBIL selects the reference target lane, whose position guides the low-level steering controller during lane changes.For an occupied target lane, the far-point distance is set by the spacing between lead and current vehicles.Source: B. Vehicle and Steering Control Model · p. 3

C. Observation states and action spaces

• The ego vehicle has a fully observable state representation designed to accommodate different numbers of surrounding cars.The states are presented from the ego vehicle's perspective in Table III.Source: C. Observation states and action spaces · p. 3, Table III · p. 3
• The ego vehicle's RL action space is specified in Table IV, though the supplied passage does not enumerate its actions.Source: Table IV · p. 3, C. Observation states and action spaces · p. 3
• After an RL lane-change decision, a low-level controller moves the vehicle from its current lane to the target lane.Source: C. Observation states and action spaces · p. 3

D. Intelligent-Driver Model (IDM)

• IDM generates vehicle acceleration as a continuous-time car-following model, with longitudinal dynamics calculated separately for each lane.The model uses current and desired speed, speed difference, and the gap to the leading vehicle.Source: D. Intelligent-Driver Model (IDM) · p. 4
• Table V lists the IDM parameters used to tune the simulation.Source: Table V · p. 4
• A condition limits IDM's minimum deceleration, which is otherwise unlimited in the original model.The simulation uses a desired or comfortable deceleration of 1.7 m/s2.Source: D. Intelligent-Driver Model (IDM) · p. 4

E. Minimizing Overall Braking Induced by Lane Changes (MOBIL)

• MOBIL first checks whether a lane change would make the new follower decelerate beyond the safety criterion.The criterion uses the new follower’s acceleration after the change and the maximum safe deceleration.Source: E. Minimizing Overall Braking Induced… · p. 4
• After the safety check passes, MOBIL evaluates a second criterion based on acceleration gains among surrounding vehicles.The algorithm propagates vehicle dynamics for a few steps to make its decision.Source: E. Minimizing Overall Braking Induced… · p. 4
• The politeness factors p and q are chosen heuristically for safety and performance, and the applied USA rules allow overtaking on either side.The MOBIL parameters are listed in Table VI.Source: E. Minimizing Overall Braking Induced… · p. 4, Table VI · p. 4

IV. HYPER-PARAMETERS

• Neural-network architecture and reward function are the hyper-parameters prioritized because they have the highest impact on overall performance.Source: §IV · p. 4

A. Neural network architecture

• The selected dense MLP has four hidden layers with 64, 128, 128, and 64 neurons, each using tanh activation.The architecture was selected through trial and error and a simple grid search.Source: A. Neural network architecture · p. 4
• The network maps an observation vector to an estimated Q-value for each action; its simplicity accelerates training and prevents overfitting.Source: A. Neural network architecture · p. 4

B. Reward function:

• The reward objective is to reach maximum allowable highway speed while respecting safety criteria and reducing lane changes.Source: B. Reward function · pp. 4–5
• The reward assigns a −1 lane-change penalty, −20 for an unnecessary action, −50 for a collision, −5 for a TTC violation, and +50 for reaching the goal.Source: B. Reward function · p. 5
• Speed progress is rewarded as (vcurrent − vinitial)/vd, where vd is the maximum allowable highway speed.Unnecessary actions leading out of road are masked, and the safe TTC region is set to 1.8 seconds.Source: B. Reward function · p. 5

V. RESULTS

• Observation noise degraded agent II’s training rewards in the mid-level noisy environment, with MOBIL evaluated on the same noisy scenarios.The reward comparison is shown in Fig. 4.Source: §V · p. 5, Figure 4 · p. 5
• The agents were evaluated in identical traffic scenarios with autonomous actors using IDM car-following and MOBIL lane-changing algorithms.Agent I was trained for 500,000 time steps; observation noise levels were noise-free, 5%, and 15%, with percentages denoting Gaussian standard deviation from ground-truth states.Source: §V · p. 5
• Agent I significantly outperforms MOBIL in the uncertain test cases despite training in a mid-level stochastic environment.The authors report that it performs considerably well in the more uncertain environment; Table VII presents the comparison.Source: §V · p. 5, Table VII · p. 5
• Across 100 episodes, mean reward was comparable, but MOBIL had 8 collisions while agent I had none.The comparison is shown in Fig. 5, and Fig. 6 illustrates a MOBIL weakness in highway traffic.Source: §V · p. 5, Figure 5 · p. 6, Figure 6 · p. 6
• Training an RL agent in a static environment did not capture dynamics and uncertainty from surrounding vehicles making rational decisions such as lane changes.The comparison of agent II with agent I is reported in Table VIII.Source: Table VIII · p. 5, §V · p. 6
• Agent I performs better than MOBIL in noisy environments, while MOBIL performs better than both RL agents in noise-free conditions.The authors attribute agent I’s noisy-environment performance to training in a stochastic, dynamic environment.Source: §V · p. 6

VI. CONCLUSIONS

• The deep RL agent balances ego-vehicle safety and agility on the fly in uncertain, stochastic highway environments.The authors describe this as indicating adaptive behavior.Source: §VI · p. 6
• The authors report superior deep RL capability over rule-based baselines in uncertain and stochastic environments.Source: §VI · p. 6
• Future work is planned on a collaborative multi-agent environment where agents cooperate toward shared goals such as increasing traffic flow.Reducing each agent’s time of arrival is another example goal.Source: §VI · p. 6