Automated Lane Change Strategy using Proximal Policy Optimization-based Deep Reinforcement Learning

I. INTRODUCTION

• Dynamic interactions make effective and safe automated driving decisions difficult, while imitation-learning data collection and labeling can be costly and incomplete.The required demonstrations may not cover all complex real-world driving situations.Source: §I · p. 1
• RL can learn by trial and error without labeling each sample, but lane-change learning faces slow rates and reward maximization alone does not guarantee safety.Improper lane changes can contribute to highway crashes and traffic jams,,, [10].Source: §I · pp. 1–2
• The proposed method combines PPO with a safety intervention module and adds an abort action that can return the vehicle to its original lane during a maneuver.For longitudinal decisions, the vehicle also chooses which leading vehicle to follow for speed adjustments before changing lanes.Source: §I · p. 2

II. POLICY-BASED REINFORCEMENT LEARNING

• Policy-based RL directly optimizes an advantage quantity, whereas value-based methods approximate a value function using neural networks in an off-policy way.The paper focuses on policy-based methods such as REINFORCE [20].Source: §II · p. 2
• Repeated optimization of the basic policy-gradient loss can cause high variance and destructively large policy updates that affect later observation and reward distributions.Source: §II · p. 2
• PPO uses an actor-critic structure and a clipped surrogate objective combined with value-function error and entropy terms.The clipped objective uses a probability ratio bounded by 1−ε and 1+ε and takes the minimum of clipped and unclipped objectives to avoid large policy updates [17].Source: §II · p. 2
• PPO optimizes its loss using mini-batch SGD for K epochs on data from fixed-length trajectory segments collected by N parallel actors.The section describes a truncated generalized advantage estimate for the collected trajectories.Source: §II · p. 2

III. PROBLEM DESCRIPTION

• The task focuses on mandatory lane changes with route-planner intentions already specified, leaving the policy to decide when and how to maneuver.Source: §III · p. 3
• After the policy makes a decision, a low-level controller regulates longitudinal and lateral movement to execute it.Source: §III · p. 3
• The learned policy is intended to avoid collisions, achieve high driving efficiency, and execute smooth maneuvers.Source: §III · p. 3

IV. METHODOLOGY

• The proposed methodology uses PPO-based reinforcement learning to learn a mandatory lane-change strategy.Source: §IV · p. 3
• The section presents the design of the decision strategy’s state and action spaces.Source: §IV · p. 3
• The methodology also presents the design of the reward functions for the lane-change strategy.Source: §IV · p. 3

A. System Architecture

• The learner uses PPO to train a high-level decision policy while interacting with traffic in a SUMO simulation environment through TraCI.Vehicle states pass through the policy network, and its actions are returned to SUMO to model movement and compute reward.Source: A. System Architecture · p. 3

B. State and Action Space

• The state represents five vehicles and contains 21 continuous variables: five for the ego-vehicle and four for each surrounding vehicle.The five vehicles are the ego-vehicle and the leaders and followers in the current and target lanes.Source: B. State and Action Space · p. 3, Figure 2 · p. 4
• The policy chooses lateral actions to keep the lane, change lanes immediately, or abort, and longitudinal actions to follow either lane’s leader.The abort action allows the vehicle to discontinue a lane change when another vehicle may be merging into the same space.Source: B. State and Action Space · p. 4
• Six combined actions are executed by low-level longitudinal and lateral controllers, with a modified car-following IDM used for longitudinal control.The IDM modification is intended to generate a human-like smooth trajectory.Source: B. State and Action Space · p. 4

C. Reward Function

• Comfort is evaluated through lateral and longitudinal jerk, penalizing sudden acceleration or deceleration that may cause occupant discomfort.The two jerk terms are weighted separately.Source: C. Reward Function · p. 4
• Efficiency rewards reaching the target lane promptly while respecting the speed limit, using time, lateral-position, and speed sub-rewards with corresponding weights.The total efficiency reward is the weighted sum of those three sub-rewards.Source: C. Reward Function · p. 4
• Near-collision penalties depend on the action-state space and relative vehicle positions, allowing the ego-vehicle to learn to abort when nearby distances fall below a threshold.The formulation penalizes near-collision risk rather than waiting only for an actual collision.Source: C. Reward Function · p. 4
• A safety intervention labels actor-network actions as catastrophic or safe, replaces catastrophic actions with safer ones, and returns a negative penalty reward.The overall reward design also includes penalties for collision and near-collision.Source: C. Reward Function · pp. 4–5

A. Simulation Setup

• The modeled highway segment extends 800 m to the ramp exit, uses 3.75 m lanes, and simulates dense traffic with vehicle counts generated from a probability model.Maximum acceleration is 2.5 m/s2, emergency braking deceleration is −4.5 m/s2, and the near-collision distance threshold is 10 m.Source: A. Simulation Setup · p. 5
• Other vehicles and the ego-vehicle’s low-level longitudinal control use IDM, while lane changes can be aborted in challenging dense traffic and resumed when safe.Lane-change kinematics were tested in normal and challenging situations.Source: A. Simulation Setup · p. 5
• PPO uses Adam with a 1×10−4 learning rate, horizon T = 512, mini-batches of 64, discount factor γ = 0.99, and clipping parameter ε = 0.2.Source: A. Simulation Setup · p. 5

B. Evaluation Metrics

• Collision rate counts rear-end and side-impact collisions, while task success requires reaching the target lane before the exit without colliding.Source: B. Evaluation Metrics · p. 5
• Training metrics are computed over 10 rollouts with horizon T = 512.Source: B. Evaluation Metrics · p. 5
• Testing metrics are computed over 50 rollouts with horizon T = 1024.Source: B. Evaluation Metrics · p. 5

• After IDM-based speed adjustment narrows the gap to its leader, the ego vehicle executes the lane-change decision and follows the target-lane leader.Source: C. Results · p. 5, Figure 4 · p. 5
• Training reward curves rise across comfort, efficiency, and safety, though safety reward drops early during exploration–exploitation balancing.Source: C. Results · p. 5, Figure 5 · p. 6
• The average cumulative loss shows a decreasing trend, indicating that it converges after sufficient epochs.Source: C. Results · p. 5, Figure 6 · p. 6
• During training in dense traffic, the agent achieved a 95% success rate as collision rate fell from 20% to 2%.Source: C. Results · p. 6, Figure 6 · p. 6
• In testing, the safety-intervention agent achieved 0.5% average collision rate and 99.15% average success rate, while the best TTC rule-based agent recorded 6.31% and 92.62%, respectively.Source: C. Results · p. 6

VI. CONCLUSIONS AND FUTURE WORK

• The PPO policy selects when and how to change lanes from current driving situations, while a predefined model executes lower-level control.Source: §VI · pp. 6–7
• 95%–99% success rate in dense traffic is reported for the PPO-trained ego vehicle, demonstrating the proposed strategy’s effectiveness.The conclusion describes the vehicle as taking appropriate actions to maximize accumulated reward.Source: §VI · p. 7
• Future work includes incorporating human/expert inference with probability modeling and exploring interdisciplinary approaches to improve generalization.The paper also points to transfer learning as one example of a technique to explore.Source: §VI · p. 7