Source-linked AI summary
Multi-agent Reinforcement Learning for Cooperative Lane Changing of Connected and Autonomous Vehicles in Mixed Traffic
Wei Zhou, Dong Chen, Jun Yan, Zhaojian Li, Huilin Yin, Wanchen Ge
TL;DR
Multi-AV lane changing in mixed traffic remains relatively underexplored, while coordination amid uncertain HDV behavior is challenging. The paper develops a cooperative MARL approach with shared parameters and a multi-objective reward; across tested traffic densities and HDV aggressiveness levels, it outperforms several state-of-the-art benchmarks on efficiency, safety, and driver comfort.
Problem
Most RL lane-changing studies focus on a single vehicle, leaving multi-AV coordination with HDVs in mixed traffic relatively underexplored.
Source: §1 · pp. 2–3Method
The paper formulates multi-AV lane changing as cooperative MARL, using MA2C with parameter sharing and a local multi-objective reward for safety, efficiency, and comfort.
Source: §1 · p. 3, §3.2 · p. 6Results
Across three traffic densities and different levels of HDV aggressiveness, the proposed framework consistently outperforms several state-of-the-art benchmarks in efficiency, safety, and driver comfort.
Source: §5 · pp. 15–16Takeaways & Limitations
The reported performance is stable under aggressive and courteous HDV behavior across the tested traffic densities.
Source: §4.3.4 · p. 12The model-free design assumes no prior knowledge about transition probabilities.
Source: §3.2 · p. 8
Abstract
from arXiv · showhide
Autonomous driving has attracted significant research interests in the past two decades as it offers many potential benefits, including releasing drivers from exhausting driving and mitigating traffic congestion, among others. Despite promising progress, lane-changing remains a great challenge for autonomous vehicles (AV), especially in mixed and dynamic traffic scenarios. Recently, reinforcement learning (RL), a powerful data-driven control method, has been widely explored for lane-changing decision makings in AVs with encouraging results demonstrated. However, the majority of those studies are focused on a single-vehicle setting, and lane-changing in the context of multiple AVs coexisting with human-driven vehicles (HDVs) have received scarce attention. In this paper, we formulate the lane-changing decision making of multiple AVs in a mixed-traffic highway environment as a multi-agent reinforcement learning (MARL) problem, where each AV makes lane-changing decisions based on the motions of both neighboring AVs and HDVs. Specifically, a multi-agent advantage actor-critic network (MA2C) is developed with a novel local reward design and a parameter sharing scheme. In particular, a multi-objective reward function is proposed to incorporate fuel efficiency, driving comfort, and safety of autonomous driving. Comprehensive experimental results, conducted under three different traffic densities and various levels of human driver aggressiveness, show that our proposed MARL framework consistently outperforms several state-of-the-art benchmarks in terms of efficiency, safety and driver comfort.
1 Introduction
Lane changing in mixed traffic is challenging because AVs must coordinate safe, efficient maneuvers amid uncertain HDV behavior. The paper formulates multi-AV lane changing as cooperative MARL and proposes shared policies and rewards that include safety, efficiency, and comfort.
1 Introduction
- Lane changing is a major high-level AV decision challenge with significant implications for traffic safety and efficiency.Source: §1 · p. 2
- In the considered two-lane highway scenario, AVs must make necessary lane changes to overtake slow vehicles while adapting to uncertain HDV behavior.Source: §1 · p. 2, Figure 1 · p. 2
- Prior single-agent lane-changing approaches treat other vehicles as environment, but multi-AV mixed traffic requires collaboration and coordination among AVs.Source: §1 · pp. 2–3
- Existing MARL lane-change rewards often omit passenger comfort and assume uniform HDV behavior, despite variation between aggressive and mild drivers.Source: §1 · p. 3
- The proposed MA2C uses a local reward incorporating safety, efficiency, and passenger comfort, with parameter sharing to foster collaboration among AVs.Source: §1 · p. 3
- The paper formulates multi-AV highway lane changing in mixed traffic as decentralized cooperative MARL and evaluates it across three traffic densities and two driver behavior modes.The study compares the models with state-of-the-art approaches on safety, efficiency, and driver comfort.Source: §1 · p. 3
2 Related Work
Related work spans rule- and model-based planning as well as data-driven RL and MARL. The paper positions its approach against limitations in mixed-traffic coordination, passenger comfort, and assumptions about human-driver behavior.
2 Related Work
- The related-work survey distinguishes nondata-driven methods from data-driven methods for autonomous-driving decision tasks.Source: §2 · p. 4
2.1 Non-data-driven Methods
- Rule- and model-based lane-changing methods use predefined rules, safe trajectory planning, or constrained predictive control, but complex and stochastic mixed traffic makes explicit criteria difficult to formulate.Source: §2.1 · p. 4
2.2 Data-Driven Methods
- Single-agent RL methods treat other vehicles as environment, whereas the considered multi-agent setting requires AV collaboration and coordination.Source: §2.2 · p. 4
- Prior MARL lane-changing approaches commonly omit passenger comfort and assume constant universal HDV behavior, limiting their fit to varied human driving styles.Source: §2.2 · p. 5
- The paper formulates multi-AV highway lane changing as MARL, combining a reward for safety, efficiency, and passenger comfort with parameter sharing; experiments cover three traffic densities and two driver-aggressiveness levels.Source: §2.2 · p. 5
3 Problem Formulation
The paper formulates multi-AV highway lane changing as decentralized cooperative MARL and develops MA2C with a multi-objective reward, shared network parameters, and local rewards. Agents act from partial observations of nearby traffic, with rewards designed around safety, efficiency, and comfort.
3.1 Preliminary of RL
- In standard RL, agents seek discounted future reward; partial observations yield a POMDP, and the action-value function is expected return for an action followed by policy π.The actor-critic setup uses a critic’s value estimate to guide policy updates, while the advantage function is introduced to reduce sample variance.Source: §3.1 · pp. 5–6
3.2 Lane Changing as MARL
- The lane-changing problem is modeled as a decentralized multi-agent POMDP in which vehicles communicate with neighbors and choose actions from partial observations.Each agent follows a decentralized policy over its observation and state.Source: §3.2 · p. 6
- Each agent observes detected vehicles’ relative positions and speeds and can choose to speed up, slow down, cruise, or change lanes.The observation includes longitudinal and lateral position and speed; the action space includes left and right turns.Source: §3.2 · p. 6
- The reward combines collision safety, safe headway, speed, and driving comfort, including smooth acceleration and avoiding frequent lane changes.Safety receives a large weighting coefficient because it is treated as the most important driving criterion.Source: §3.2 · pp. 6–7
- The comfort design penalizes rapid acceleration or deceleration and lane changes, while the model-free method assumes no prior transition probabilities.Necessary lane changes remain promoted through safety and speed evaluations.Source: §3.2 · pp. 7–8
3.3 MA2C for AVs
- MA2C uses shared actor-critic parameters and averages rewards over each agent and its close neighbors, mitigating remote agents’ impact.The shared network processes extracted features to update policy and value networks.Source: §3.3 · p. 8
- The actor-critic backbone processes states separated by physical units through separate 64-neuron layers, then combines them before a 128-neuron layer and shared actor-critic networks.The parameter-sharing scheme is described as improving learning efficiency.Source: §3.3 · p. 8
- The MA2C training procedure updates actions from each agent’s observation and updates network parameters after episodes; completed or collided episodes trigger a reset.The algorithm lists discount factor, learning rate, politeness coefficient, and epoch length as hyperparameters.Source: §3.3 · pp. 8–9
4 Experiments and Discussion
The experiments evaluate the proposed MARL algorithm’s training efficiency, safety, and driving comfort in the considered highway lane-changing scenario.
4 Experiments and Discussion
- The evaluation considers training efficiency, safety, and driving comfort in the highway lane-changing scenario.The scenario is the one introduced in Fig. 1.Source: §4 · p. 9
4.1 HDV Models
The HDV models use IDM for longitudinal control and MOBIL for lateral control; MOBIL’s politeness coefficient varies how much follower effects enter lane-changing decisions.
4.1 HDV Models
- HDV longitudinal dynamics follow IDM, a deterministic continuous-time model of vehicle position and speed that accounts for expected speed, inter-vehicle distance, and driving habits.MOBIL is used for lateral control and takes vehicle acceleration as input.Source: §4.1 · p. 10
- A lane change is allowed when the new-follower braking condition in Eqn. 8 is satisfied.The passage identifies the new follower’s acceleration and the maximum braking imposed on it.Source: §4.1 · p. 10
- MOBIL’s politeness coefficient ranges from considerate decisions that account for blocked followers at p = 1 to selfish speed-gain decisions at p = 0.The paper discusses performance at different p values in Section 4.3.3.Source: §4.1 · p. 10
4.2 Experimental Settings
The experimental setting uses a modified highway-env simulator, three traffic-density levels, and repeated training and evaluation runs.
4.2 Experimental Settings
- The simulator uses a 520 m highway, randomly spawned vehicles starting at 25−30 m/s, and 5 Hz control sampling.Vehicles beyond the road are ignored.Source: §4.2 · p. 10
- Three traffic-density levels represent low, middle, and high congestion, with vehicle counts reported in Table 1.The supplied table caption identifies it as traffic density modes.Source: §4.2 · p. 10, Table 1 · p. 11
- HDVs use IDM and MOBIL with safety deceleration bsafe = −9 m/s2, politeness p = 0, and lane-changing threshold ∆ath = 0.1 m/s2.The initial vehicle speeds are randomly drawn from the stated range.Source: §4.2 · p. 10
- We train MARL algorithms for 1 million steps (10,000 epochs), applying two different random seeds shared among agents, and evaluate each model three times every 200 training episodes.Source: §4.2 · p. 10
- The discount factor is 0.99, the learning rate is 5 × 10−4, and reward weights are ωs = 200, ωd = 4, ωv = 1, and ωc = 1.Experiments ran on a macOS server with a 2.7 GHz Intel Core i5 and 8 GB of memory.Source: §4.2 · p. 11
4.3 Results & Analysis
MA2C performs favorably against the evaluated benchmarks, while the experiments also support its local reward design, shared actor-critic parameters, and comfort measurement. Tests across traffic densities and HDV behaviors, along with a lane-change example, examine its performance and learned cooperative behavior.
4.3.1 Local v.s. Global Reward Designs
- Across all three traffic modes, local rewards yield higher evaluation rewards and smaller variance than global rewards, with gaps widening as vehicle counts increase.Global rewards perform better before 2000 epochs, but have relatively large variance; the authors attribute the disadvantage to credit-assignment issues.Source: §4.3.1 · p. 11, Figure 3 · p. 11
4.3.2 Sharing v.s. Separate Actor-critic Network
- Sharing actor-critic parameters produces higher rewards and lower variance than separate networks in all three modes.The authors explain that shared state representations let the actor benefit from the critic during training.Source: §4.3.2 · p. 11, Figure 4 · p. 12
4.3.3 Verification of Driving Comfort
- With comfort measurement, AV acceleration has a lower average deviation of 0.455m/s2 than the 0.582m/s2 without the comfort term.The acceleration profile is also described as smoother with comfort measurement.Source: §4.3.3 · p. 12, Figure 5 · p. 12
4.3.4 Adaptability of the Proposed Method
- MA2C achieves scalable and stable performance across traffic densities when HDVs behave aggressively or courteously.The politeness coefficient p ranges from 0 for the most aggressive behavior to 1 for the most polite behavior.Source: §4.3.4 · p. 12, Figure 6 · p. 13
4.3.6 Policy Interpretation
- In the illustrated maneuver, one AV slows to make space for another AV’s safe lane change, then accelerates after the lane change is complete.The sequence is shown in snapshots and AV/HDV speed traces.Source: §4.3.6 · pp. 13–14, Figure 8 · p. 15, Figure 9 · p. 15
4.3.5 Comparison with the state-of-the-art benchmarks
- MA2C outperforms MAACKTR and MAPPO in evaluation reward and reward standard deviations, while showing promising results across the three density scenarios.MADQN has higher average reward in some comparisons but larger reward deviations; MA2C performs best in density1. The comparison is summarized in Table 2 and training curves in Fig. 7.Source: §4.3.5 · p. 13, Table 2 · p. 14, Figure 7 · p. 14
5 Conclusion
The conclusion presents an on-policy multi-agent extension of A2C for mixed-traffic lane changing, with local rewards and parameter sharing. Across varied traffic densities and HDV aggressiveness, the framework outperforms state-of-the-art benchmarks on efficiency, safety, and driver comfort.
5 Conclusion
- The approach extends A2C to on-policy multi-agent lane-changing with a novel local reward design and parameter-sharing schemes.Its multi-objective reward promotes driving efficiency, comfort, and safety.Source: §5 · p. 15
- The empirical evaluation spans three traffic densities and different levels of HDV aggressiveness.Source: §5 · p. 15
- The framework consistently outperforms several state-of-the-art benchmarks in efficiency, safety, and driver comfort.The experiments cover three traffic densities and different levels of HDV aggressiveness.Source: §5 · pp. 15–16
Authors’ Contribution.
The end matter records funding support and states that the paper has no conflicts of interest.
Authors’ Contribution.
- The end matter acknowledges grant, institutional, and scholarship support for the research.Source: Funding · p. 16
Code availability.
- The paper reports that there are no conflicts of interest.Source: Competing interests · p. 16