Source-linked AI summary
Reinforcement Learning-Based Control of CAV Platoon Joining Maneuvers in Mixed Traffic
Biao Yin, Abderrahmane Kasmi, Nadir Farhi
TL;DR
CAV platoon joining in mixed traffic requires safe, efficient control despite heterogeneous human driving, while existing evaluations lack comprehensive comparisons of safety strategies. This paper develops a SUMO-based deep-RL framework and finds PPO achieves about 98% successful joinings with below-1% failures and collisions, at the cost of more decision steps.
Problem
Existing studies lack realistic mixed-traffic evaluations comparing penalty-based rewards and external safety controllers for CAV platoon joining.
Method
A SUMO-based agent framework models mixed and dedicated CAV lanes and compares DQN, DDQN, and PPO with risk penalties or an external safety controller.
Results
About 98% of PPO-controlled joinings succeeded, while failures and collisions each remained below 1%, despite about 10% more decision steps.
Takeaways & Limitations
PPO provides safer and more effective joining across tested traffic loads, but its more conservative control increases decision-step requirements.
Takeaways & Limitations
The discrete speed-adjustment action space is not optimized, and joining efficiency is not explicitly confirmed through optimization of the ego vehicle’s speed trajectory.
Abstract
from arXiv · showhide
Connected and automated vehicle (CAV) platooning offers a promising approach to improving road safety and traffic capacity. However, platoon control in real-world traffic is challenging due to uncertainty and heterogeneous driving behaviors. Reinforcement learning (RL) has strong potential for addressing such control problems, but its practical deployment raises challenges related to safety and learning efficiency. This paper proposes a generic modeling and simulation framework for investigating CAV platoon joining maneuvers and comparing deep reinforcement learning (DRL)-based control algorithms. The problem is particularly challenging in mixed-traffic environments, where CAVs coexist with human-driven vehicles exhibiting heterogeneous longitudinal and lateral behaviors. The objective is to achieve safe and efficient joining maneuvers by either incorporating penalties for risky behaviors into the learning process or using an external safety controller to constrain the learned policy. An agent-based modeling framework coupled with the Simulation of Urban MObility (SUMO) simulator is used to evaluate Deep Q-Network (DQN), Double Deep Q-Network (DDQN), and Proximal Policy Optimization (PPO). Results show that PPO outperforms DQN and DDQN, achieving a joining success rate of approximately 98 % and a collision rate below 1 %, largely due to risk-related penalties incorporated into the reward function. However, this improved performance requires more decision steps to complete the maneuver, revealing a trade-off between safety, joining effectiveness, and decision efficiency. An external safety controller effectively prevents collisions, although its interventions may reduce joining efficiency. The results highlight the importance of jointly considering safety and efficiency when designing RL-based controllers for CAV platoon joining in mixed traffic.
1. Introduction
The introduction motivates CAV platoon joining as a safety- and efficiency-critical phase that remains insufficiently studied in realistic mixed traffic. It proposes a framework for comparing deep reinforcement learning approaches and analyzing joining performance under diverse traffic conditions.
- CAV platoons can improve road capacity, traffic stability, and energy efficiency, while cooperative communication helps mitigate undesirable human driving behavior [2].
- CAV platoon joining is a fundamentally important operational phase because it enables a CAV to merge into a platoon at a designated position.
- Existing platoon-joining studies mainly use rules, optimization models, or conceptual techniques, but rule-based performance can degrade in highly dynamic environments,,.
- Most existing frameworks use oversimplified environments, while systematic study of transitions from mixed-traffic lanes to dedicated CAV lanes remains limited.
- The paper develops a mixed-to-dedicated-lane joining framework, compares three deep reinforcement learning algorithms, and evaluates performance under diverse traffic conditions.
2. Background Knowledge
Reinforcement learning models decision-making as an agent interacting with an environment through actions and feedback, commonly formalized as a Markov decision process. The section introduces Q-learning, DQN, DDQN, and PPO, emphasizing their distinct value-estimation, action-selection, and policy-update mechanisms.
- RL models agent decision-making as an MDP defined by state and action spaces, transition probabilities, rewards, and a discount factor.The agent learns through environmental interaction, taking actions and receiving feedback.
- Q-learning maximizes an objective by iteratively updating Q-value estimates according to the Bellman equation.
- DQN uses separate online and target networks to approximate the Q-function, minimizing a loss while periodically updating target-network weights for stability.Deep RL uses neural networks to approximate value functions or policies.
- DDQN reduces max-operator overestimation by having the online network select the action and the target network evaluate it [21].Both DQN variants use replay buffers, while action selection follows a decaying ε-greedy policy that increasingly favors the best action.
- PPO updates its policy with an advantage function, often computed using GAE, and maximizes a surrogate objective through importance sampling without a Q-target network.
3. Methodology
The methodology combines SUMO-based mixed-traffic simulation with an agent-based RL controller for CAV platoon joining. It defines a phased joining procedure, a 15-variable state, six discrete actions, and reward penalties for unsafe or inefficient behavior.
- System architecture: The system couples a SUMO two-lane environment with an RL agent that receives Traci-based traffic states and returns longitudinal or lateral control actions.The inner lane is reserved for CAV platoons, while the outer lane mixes HDVs and CAVs with potentially different speed limits.
- Joining workflow: The joining procedure selects a joiner and target platoon, executes a lane change or speed adjustment, and uses PLEXE-based CACC for gap closing.The joiner enters and leaves at the rear of the target platoon, reducing internal reordering and traffic disturbance.
- State representation: The state is a 15-variable vector containing ego, neighboring-vehicle and platoon speeds, intervehicle distances, and lane-related variables.It represents surrounding vehicles on both the mixed and platoon lanes, including the target and following platoons.
- Action space: The action space has six discrete actions: one lane change at current speed and five no-lane-change accelerations from 2 m/s2 to −2 m/s2.Each decision is applied over five simulation steps, totaling 0.5 s with a 0.1 s simulation timestep.
4. Simulation and Results
Across mixed-traffic simulations, PPO achieves the strongest platoon-joining performance, with high success and low collision rates, while safety enhancements trade some efficiency for protection. PPO’s superior safety and effectiveness are accompanied by more decision steps in some traffic conditions.
- Training performance: PPO outperforms DQN and DDQN in reward evolution, particularly before 4,000 and after 8,000 iterations, with more stable and larger late-period mean rewards than DQN.Small negative rewards indicate that collisions, failures, or risky joining occur only occasionally.
- Operational performance: PPO converges fastest at around 3,000 iterations and shows increasing success with decreasing failure and collision rates over training.The operational-rate trends are reported for success, collision, and failure across the three algorithms.
- Operational performance: PPO achieves more than 97.5% success and less than 1.0% collision rates across traffic loads, outperforming DQN and DDQN.These results use the TTC penalty in the reward function and demonstrate PPO’s strongest overall joining performance.
- Joining efficiency: PPO requires more decision steps for joining than DQN and DDQN under asymmetric traffic loads, whereas higher symmetric loads generally reduce decision-step counts.The reduction is attributed to lower vehicle speeds caused by high traffic density on both lanes.
- External safety protection: After applying a safety shield, collision rates fall below 0.001 but are converted into abandonment, while PPO retains success rates of at least 96.8%.Some previously successful but dangerous joinings with TTC < 2 s are also abandoned; under higher traffic density, DDQN shows the best joining efficiency.
- Joining quality: Average joining distances remain stable between 10 and 20 meters under symmetric loads but increase slightly when platoon-lane traffic is low under asymmetric loads.The observed distances align with the stable-maneuver setting, where no penalty is assigned in this range.
5. Conclusion
The proposed deep RL framework compares DQN, DDQN, and PPO for CAV platoon joining in mixed traffic, with PPO performing best but requiring slightly more decision steps and joining distance. The study also identifies action-space and trajectory-optimization limitations and outlines future extensions.
- Conclusion: PPO outperforms DQN and DDQN in training and testing, but requires about 10% more decision steps and 0.9–1.2 m longer average distances to target platoons.The comparison concerns value-based DQN and DDQN versus policy-based PPO within the proposed deep RL framework.
- Limitations: The discrete speed-adjustment action space is not optimized, so joining efficiency remains unconfirmed despite jerk-penalty consideration for driving comfort.The ego vehicle’s speed trajectory is not explicitly optimized.
- Future Work: Future work will optimize energy-efficient and time-efficient platoon joining globally and develop strategies for CAVs to exit platoons safely and effectively.
Appendix A. Configuration
Appendix A specifies the configurations used for DQN/DDQN, PPO, and the traffic scenarios through three dedicated tables.
- DQN and DDQN configuration: Table A.1 defines the configurations for the DQN and DDQN algorithms.
- PPO configuration: Table A.2 defines the configuration for the PPO algorithm.
- Traffic-scenario configuration: Table A.3 defines the configurations for the traffic scenarios.