PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Multi-agent Reinforcement Learning for Highway On-Ramp Merging in Mixed Traffic

Dong Chen, Mohammad Hajidavalloo, Zhaojian Li, Kaian Chen, Yongqiang Wang, Longsheng Jiang, Yue Wang

arXiv:2105.05701v3eess.SY

TL;DR

Mixed-traffic highway on-ramp merging requires AVs to coordinate with HDVs while maintaining safe and efficient traffic flow. The paper develops a scalable decentralized MARL framework with local cooperation and safety supervision, and reports consistent improvements over benchmark algorithms in training efficiency and collision rate.

  • Problem

    Highway on-ramp merging is difficult in mixed traffic because multiple AVs must adapt to HDV behavior while coordinating decisions in a dynamic environment.

  • Method

    The paper develops decentralized MARL using parameter sharing, local rewards, action masking, curriculum learning, and a priority-based safety supervisor.

  • Results

    The proposed approach consistently outperforms benchmark approaches in training efficiency and collision rate, with the safety supervisor achieving zero collisions for all traffic modes when Tn >= 7.

  • Takeaways & Limitations

    The framework supports safe and efficient cooperative merging across traffic densities while remaining scalable to dynamic multi-agent traffic.

Abstract

from arXiv · show

On-ramp merging is a challenging task for autonomous vehicles (AVs), especially in mixed traffic where AVs coexist with human-driven vehicles (HDVs). In this paper, we formulate the mixed-traffic highway on-ramp merging problem as a multi-agent reinforcement learning (MARL) problem, where the AVs (on both merge lane and through lane) collaboratively learn a policy to adapt to HDVs to maximize the traffic throughput. We develop an efficient and scalable MARL framework that can be used in dynamic traffic where the communication topology could be time-varying. Parameter sharing and local rewards are exploited to foster inter-agent cooperation while achieving great scalability. An action masking scheme is employed to improve learning efficiency by filtering out invalid/unsafe actions at each step. In addition, a novel priority-based safety supervisor is developed to significantly reduce collision rate and greatly expedite the training process. A gym-like simulation environment is developed and open-sourced with three different levels of traffic densities. We exploit curriculum learning to efficiently learn harder tasks from trained models under simpler settings. Comprehensive experimental results show the proposed MARL framework consistently outperforms several state-of-the-art benchmarks.

I. INTRODUCTION

Highway on-ramp merging becomes especially difficult when AVs must coordinate with HDVs in dynamic mixed traffic. The paper formulates this setting as decentralized MARL and introduces safety and scalability mechanisms to improve cooperative merging.

  • Problem: Mixed-traffic on-ramp merging requires on-ramp vehicles to merge efficiently without collision while through-lane vehicles coordinate by adjusting speed.The scenario includes AVs and HDVs on both merge and through lanes, making coordination central to avoiding deadlock.
  • Related work: Prior RL approaches for highway merging control a single AV and treat all other vehicles as part of the environment.Existing rule-based methods become impractical in complex scenarios, while prior RL and MPC-RL approaches were designed for one AV.
  • Problem: Multiple AVs must adapt to HDVs and cooperatively maximize traffic throughput, but dynamic connectivity, coupled vehicle dynamics, and complex decisions make MARL challenging.The presence of human drivers further increases this complexity.
  • Contributions: The paper formulates mixed-traffic on-ramp merging as decentralized MARL with time-varying connectivity and provides an open-sourced gym-like simulator with three traffic-density levels.The formulation covers AVs and HDVs on both ramp and through lanes.
  • Contributions: The proposed framework combines parameter sharing, local rewards, action masking, curriculum learning, and a priority-based safety supervisor.The safety supervisor is designed to reduce collision rates and improve learning efficiency, while the experiments compare safety and efficiency against state-of-the-art algorithms.

B. Multi-agent Reinforcement Learning (MARL)

The paper reviews cooperative MARL and formulates the on-ramp merging task as a decentralized POMDP solved with an actor-critic framework. Its design combines local observations, high-level actions, reward shaping, and model-based traffic simulation.

  • MARL background: Cooperative MARL methods encourage agents to achieve a common objective, while prior approaches include independent learning, centralized critics, communication protocols, and parameter sharing.Parameter sharing supports homogeneous agents and changes in the number of participating agents; centralized supervision can create scalability concerns.
  • Problem formulation: The proposed problem is modeled as a decentralized MARL framework in which agents communicate with neighboring vehicles through a potentially changing topology.Each agent observes only nearby vehicles, with the five nearest vehicles within 150 m providing the best reported performance.
  • State, action, and transition design: Agents select high-level actions—turning, cruising, speeding up, or slowing down—while lower-level controllers generate steering and throttle commands.HDV behavior is simulated with IDM and MOBIL, AV trajectories use a kinematic bicycle model, and PID control tracks AV decisions.
  • Reward function: The reward combines collision, stable-speed, time-headway, and merging-cost evaluations, with collision weighting prioritized for safety.The time-headway threshold is 1.2 s, while the merging cost penalizes waiting near the end of the ramp to reduce deadlocks.
  • Reward function: The merging cost penalizes waiting on the merge lane, and its plotted penalty increases as the vehicle approaches the merging end.This reward component is designed to avoid deadlocks during merging.

B. MA2C for CAVs

The MA2C design uses local rewards and shared network parameters to support cooperative learning while reducing communication and credit-assignment difficulties. Actor and critic representations are shared during optimization.

  • Local reward assignment: The cooperative objective maximizes the global reward, but each agent receives a neighborhood-based reward rather than an identical global average.The design reflects the limited influence of distant vehicles on an ego vehicle.
  • Local reward assignment: Local rewards replace a global reward because global aggregation increases latency and communication overhead and creates a credit-assignment problem.Each agent’s reward is based on itself and its neighboring vehicles, whose behavior is most directly related to task success or failure.
  • Network and loss design: The actor and critic networks share low-level representations, and their policy and value losses are combined into a single loss function.Entropy regularization is used to encourage exploration.
  • Network and loss design: The advantage function uses reward and discounted next-state value relative to current-state value, while the value network is updated through its own loss.These updates implement the actor-critic training procedure.
  • Parameter sharing: A single policy network with shared parameters is updated from separate experience replay buffers for each agent.This supports a general policy for both on-ramp and through-lane AVs.

C. DNN Settings

The DNN processes locally observed vehicle features by physical units before producing actor and critic outputs. Invalid-action masking removes actions that are infeasible in the current traffic state.

  • Network architecture: The observation is divided into presence, position, and speed groups, each encoded separately before concatenation.The concatenated representation is passed through a 128-neuron fully connected layer shared by the actor and critic.
  • Action masking: Standard softmax sampling can select invalid or unsafe actions, producing misleading experiences and invalid policy updates during training.These actions may cause undesirable behavior or system breakdown.
  • Action masking: Action masking replaces invalid-action logits with large negative values, making their softmax probabilities nearly zero.The resulting distribution effectively renormalizes over valid actions.
  • Action masking: The mask excludes lane changes toward nonexistent lanes and speed changes beyond the predefined minimum or maximum speeds.Other unsafe actions are handled by the priority-based safety supervisor.

IV. PRIORITY-BASED SAFETY ENHANCEMENT

The priority-based safety enhancement predicts potential collisions and sequentially regulates AV actions under real-time constraints. Priorities favor vehicles with more urgent merging or safety needs.

  • Safety supervision: The supervisor predicts collisions over a horizon T_n using HDV motion models and corrects unsafe exploratory AV actions.IDM predicts HDV longitudinal acceleration, while MOBIL predicts HDV lane changes.
  • Motivation: Joint-action safety checking becomes computationally intractable as the number of AVs grows because the action space scales as |A_i|^N.This motivates a sequential rather than exhaustive safety-correction scheme.
  • Safety supervision: Sequential coordination prioritizes AVs with smaller safety margins, making the safety scheme computationally efficient for real-time implementation.Vehicles near the merge-lane end or safety boundary receive higher priority.
  • Priority assignment: Priority assignment favors merge-lane vehicles, vehicles closer to the merge-lane end, and vehicles with smaller time headway.These criteria reflect time-critical merging, collision and deadlock risk, and proximity to preceding vehicles.
  • Priority assignment: The priority metric combines merging status, distance to the ramp end, and time headway using positive weights and a small random perturbation.The perturbation prevents two vehicles from receiving identical priority indices.

B. Priority-based Safety Supervisor

The priority-based safety supervisor orders AVs by priority, predicts multi-step trajectories, and replaces exploratory actions when conflicts are detected. It sequentially checks vehicles and selects safe alternatives using safety margins, with prediction-horizon choice balancing foresight against uncertainty.

  • The supervisor ranks AVs by priority and checks the highest-priority vehicle’s exploratory action against predicted neighboring trajectories over Tn steps.HDV motions use driver decision and kinematic models, while lower-priority AVs initially use their previous actions for prediction.
  • Conflicting predicted trajectories mark an exploratory action unsafe and trigger replacement by a safe action.A conflict example occurs when vehicle 1’s motion primitive overlaps an HDV trajectory at prediction steps 4 and 5.
  • Candidate actions are evaluated using safety margins based on distances to relevant surrounding vehicles.Lane changes use minimum distances to preceding and following vehicles on current and target lanes; speed actions use minimum distance headway.
  • After deciding one vehicle’s action, the supervisor regenerates its trajectory, removes it from the priority list, and checks the next vehicle using higher-priority motion primitives.This sequential procedure updates the priority list as each vehicle is processed.
  • The prediction horizon Tn trades off foresight and conservatism: short horizons can become short-sighted, while long horizons propagate HDV-motion uncertainty.The supervisor can be implemented through vehicle-to-infrastructure communication, with infrastructure coordinating priority scores and safe actions.

V. NUMERICAL EXPERIMENTS

The numerical experiments evaluate training efficiency and collision rate in an on-ramp merging simulator across three traffic-density modes. The setup uses randomized vehicle counts and initial conditions, repeated training seeds, and standardized evaluation intervals.

  • The simulator models a 520 m road with a 100 m merge lane and 12 spawn points distributed across through and ramp lanes.Vehicles beyond the road remain kinematically updated after being removed from display.
  • Easy mode contains 1–3 AVs and 1–3 HDVs, while Medium mode contains 2–4 AVs and 2–4 HDVs.
  • Hard mode contains 4–6 AVs and 3–5 HDVs with randomized spawn noise and initial speeds between 25 and 27 m/s.
  • Algorithms are trained for 2 million steps using three random seeds and evaluated every 200 training episodes over three episodes.Episodes use a 100-step horizon, and control is sampled at 5 Hz.
  • The environment is modified from gym-based highway-env, uses default IDM and MOBIL parameters, and is open-sourced.

A. Reward Function Designs

The experiments compare local and global reward designs and examine how the safety penalty weight affects collision rate and traffic speed. Local rewards converge faster and perform better across traffic levels, while stronger safety weighting eliminates collisions at a speed cost.

  • Local rewards outperform averaged global rewards in evaluation reward and convergence speed across Easy, Medium, and Hard traffic scenarios.The global design fails the Hard-mode control tasks with evaluation reward below 0, whereas it remains reasonable in easier modes.
  • Global rewards suffer from credit-assignment problems and weaker correlation between individual actions and assigned rewards as agent counts increase.Averaging rewards can also increase communication overhead and latency in real-time AV systems.
  • No collisions occur for safety penalty weights wc ≥100 in Medium traffic, while average traffic speed decreases as wc increases further.The experiments choose wc = 200 as a trade-off between safety and efficiency.

B. Curriculum Learning

Curriculum learning transfers models trained on easier traffic modes to Hard mode rather than starting from scratch. This accelerates convergence and improves final performance, including higher average vehicle speed.

  • B. Curriculum Learning: Curriculum learning initializes Hard-mode training from models trained on Easy and Medium modes.The approach is motivated by reducing risky blind exploration in safety-critical autonomous-driving tasks.
  • B. Curriculum Learning: Training from easier-task models greatly accelerates convergence and improves final Hard-mode model performance compared with starting from scratch.
  • B. Curriculum Learning: 22 m/s average speed is achieved with curriculum learning versus 18 m/s for the baseline during training.The authors use curriculum learning in subsequent Hard-mode experiments.

C. Performance of the Priority-based Safety Supervisor

The priority-based safety supervisor improves learning efficiency and traffic efficiency while reducing collisions across traffic densities. Compared with benchmark MARL methods, the proposed approach consistently performs better during training and testing.

  • Supervisor effectiveness: Faster convergence and higher evaluation rewards are achieved across all three traffic densities with the priority-based safety supervisor.Replacing unsafe exploratory actions with safe actions avoids early terminations, especially during early exploration.
  • Supervisor effectiveness: 26 m/s, 24 m/s and 22 m/s are the average speeds for Easy, Medium and Hard traffic densities, respectively, during training with safety supervision.Average vehicle speed is used as an indication of traffic throughput and decreases as traffic density increases.
  • Safety and efficiency trade-offs: With reasonable Tn values such as 7 or 8, the safety supervisor produces higher average speed and lower collision rate in all traffic modes.Increasing Tn excessively can increase collision rate and reduce speed because of uncertainty propagation and more cautious behavior.
  • Benchmark comparison: The proposed method, baseline + Tn = 8, consistently outperforms MAA2C, MAPPO and MAACKTR in training across all traffic levels.Its advantage in sample efficiency and training performance is greater in Hard mode; higher training speeds also improve training efficiency.
  • Benchmark comparison: After training, the proposed method runs without collisions and achieves higher efficiency than the benchmark algorithms.Testing compares average collision rates and vehicle speeds over 3 random seeds and 30 epochs for each traffic density.

E. Policy Interpretation

The learned policy coordinates merging through speed adjustments and can be extended to multiple through lanes. The revised reward adds a lane-changing penalty to discourage unnecessary oscillatory behavior.

  • Learned behavior: Vehicle 2 slows to make space, while vehicle 3 accelerates, maintains headway, merges successfully, and then speeds up.The demonstrated behavior shows coordinated longitudinal adjustments between through-lane and on-ramp vehicles.
  • Multiple through-lane case: The multiple through-lane case extends the action state without changes and demonstrates that the proposed approach can be applied when through-lane vehicles change lanes.The approach achieves good performance in this more challenging setting.
  • Reward design: The revised reward adds lane-changing evaluation rl to penalize unnecessary and frequent lane changes that can produce unsafe driving.The reward combines collision, stable-speed, headway-time, merging-cost, and lane-changing evaluations.
  • Multiple through-lane case: The approach is extended to multiple through-lane traffic, where the reported results show good performance.The scenario includes vehicles changing lanes in the through lanes.
  • Overall findings: The paper reports consistent benchmark improvements in training efficiency and collision rate for the proposed MARL approach.The conclusion attributes the framework to action masking, local rewards, curriculum learning, parameter sharing, and a priority-based safety supervisor.

APPENDIX

The appendix describes the improved nonlinear MPC baseline used for comparison. Its formulation models vehicle dynamics, HDV behavior, control inputs, costs, and hard input constraints.

  • Vehicle and traffic modeling: The improved MPC formulation uses a kinematic bicycle model for vehicle kinematics instead of the point-mass model.This change is made to simulate vehicle dynamics in the highway environment.
  • Vehicle and traffic modeling: HDV longitudinal behavior is modeled with IDM, while lateral lane changes are modeled with MOBIL.These models specify the HDV dynamics used in the MPC comparison environment.
  • Control and cost formulation: CAV steering angle and acceleration are the MPC control inputs, translated to high-level decisions comparable to the MARL action space.The high-level decisions include faster, slower, idle, and left-lane actions.
  • Control and cost formulation: The MPC cost function includes collision avoidance, moving-area bounds, a nominal merging path, and tracking of desired speed.These components are combined with hard constraints on acceleration and steering angle in a nonlinear MPC problem.
  • Implementation: A nonlinear MPC problem is implemented from the specified dynamics, cost terms, and hard constraints, with results reported in Table III.The detailed implementation is open-sourced for the highway on-ramp merging comparison.
Loading 2105.05701v3…