PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey on Autonomous Vehicle Control in the Era of Mixed-Autonomy: From Physics-Based to AI-Guided Driving Policy Learning

Xuan Di, Rongye Shi

arXiv:2007.05156v1cs.AIcs.RO

TL;DR

Mixed-autonomy traffic poses unresolved questions about scalable AV control, human behavior, uncontrollable-AV behavior, and human–AV interaction. This paper surveys transportation and AI methodologies across AV deployment phases, synthesizes models including game theory and learning, and identifies open questions and research directions. Its supported outcome is a cross-disciplinary overview intended to guide AI applications toward safe and efficient mixed traffic.

  • Problem

    Mixed autonomy is understudied because interactions among human-driven and autonomous vehicles are unknown and complex, leaving key modeling and control questions unresolved.

  • Method

    The paper comprehensively surveys transportation-engineering and AI literature on mixed traffic across the HV-dominated, AV-dominated, and pure-AV deployment phases.

  • Results

    The survey summarizes models spanning game theory, deep reinforcement learning, and imitation learning, and identifies open questions and promising research directions.

  • Takeaways & Limitations

    The paper calls for transportation, robotics, and machine-learning communities to collaborate on safe and efficient mixed traffic.

  • Takeaways & Limitations

    Large-player deterministic control can become mathematically intractable, while decentralized multi-AV control is considered under a stochastic environment assumption.

Abstract

from arXiv · show

This paper serves as an introduction and overview of the potentially useful models and methodologies from artificial intelligence (AI) into the field of transportation engineering for autonomous vehicle (AV) control in the era of mixed autonomy. We will discuss state-of-the-art applications of AI-guided methods, identify opportunities and obstacles, raise open questions, and help suggest the building blocks and areas where AI could play a role in mixed autonomy. We divide the stage of autonomous vehicle (AV) deployment into four phases: the pure HVs, the HV-dominated, the AVdominated, and the pure AVs. This paper is primarily focused on the latter three phases. It is the first-of-its-kind survey paper to comprehensively review literature in both transportation engineering and AI for mixed traffic modeling. Models used for each phase are summarized, encompassing game theory, deep (reinforcement) learning, and imitation learning. While reviewing the methodologies, we primarily focus on the following research questions: (1) What scalable driving policies are to control a large number of AVs in mixed traffic comprised of human drivers and uncontrollable AVs? (2) How do we estimate human driver behaviors? (3) How should the driving behavior of uncontrollable AVs be modeled in the environment? (4) How are the interactions between human drivers and autonomous vehicles characterized? Hopefully this paper will not only inspire our transportation community to rethink the conventional models that are developed in the data-shortage era, but also reach out to other disciplines, in particular robotics and machine learning, to join forces towards creating a safe and efficient mixed traffic ecosystem.

1. Introduction

The paper surveys AI-guided and transportation models for AV control during the transition from human-driven to autonomous traffic. It focuses on mixed autonomy, where interactions among human-driven and autonomous vehicles create substantial modeling complexity.

  • 1. Introduction: The survey reviews AI-guided methods alongside transportation-engineering models to address scalable AV policies, human behavior estimation, uncontrollable-AV behavior, and human–AV interactions.Its scope spans literature from transportation engineering and AI, including game theory, learning, and control-oriented approaches.
  • 1.1. Modeling complexity: Mixed autonomy receives less attention than single-AV and pure-AV scenarios despite representing the realistic transition between them.Existing research has largely studied one AV among human drivers or AV platoons with negligible human interaction.
  • 1.1. Modeling complexity: AV deployment is divided into pure-HV, HV-dominated, AV-dominated, and pure-AV phases, with the survey focusing on the latter three.The HV-dominated and AV-dominated phases are identified as the most challenging because vehicle interactions are unknown and complex.
  • 1.2. Divergence in the communities: Transportation and robotics communities pursue different scales and goals: system-level effects such as congestion versus vehicle-level policies learned in stochastic environments.These differing objectives lead to different AV decision-making frameworks and modeling assumptions.
  • 1.2. Divergence in the communities: Transportation models often treat AVs as physics-based particles or faster, better-informed human drivers, whereas robotics models treat AVs as AI agents exploring and optimizing actions.The communities also differ in how they represent human and autonomous behavior and their interactions.
  • 1.4. AI for decision-making of AVs: The paper advocates AI-guided multi-agent learning and adaptation because future AVs should act as utility-optimizing agents while using richer information than human drivers.The authors also call for collaboration across transportation, robotics, and machine learning to pursue safe and efficient mixed traffic.

2. AI-guided driving policy learning for AVs

The paper frames mixed-autonomy AV control as a multi-agent problem involving controllable AVs, human drivers, and uncontrollable AVs. It organizes the review around scalable control, behavior estimation, environment modeling, and human–AV interaction.

  • Mixed traffic is a multi-agent system because agents’ driving actions are interdependent through rewards, shared states, or constraints.
  • The mixed-autonomy control problem seeks distributed strategies for N controllable AVs selecting optimal acceleration or steering controls over a planning horizon.
  • The review asks how to scale AV policies, estimate human behavior, model uncontrollable AVs, and characterize human–AV interactions.
  • Studies are categorized by the numbers of AVs and HVs involved, including 1 AV + 1 HV, 1 AV + m HVs, n AVs + m HVs, n AVs + 1 HV, and n AVs.
  • Multi-AV control is further organized by cooperation and whether control accounts for uncertainty in the external environment.

3. 1 AV + 1 HV, 1 AV + 1 AV: General-sum Game-Based Control

Game-based control models interacting vehicles as strategic agents with potentially conflicting objectives. Static games address one-step actions, while dynamic and hierarchical formulations extend control across time.

  • Game theory models non-cooperative strategic interactions among AVs or between an AV and an HV, treating vehicles as agents optimizing individual objectives.
  • One-shot two-person games represent driving, merging, lane-changing, and unprotected left-turning decisions at a single step.
  • One-shot games cannot represent vehicles’ dynamic driving actions, motivating dynamic optimal control, MPC, rolling-horizon control, and differential games.
  • In a simultaneous differential game, each vehicle optimizes a reward over shared system dynamics while selecting a time-varying driving policy.
  • Leader–follower games simplify differential games by having the AV act first and the HV respond using predictions based on the AV’s historical actions.
  • Hierarchical game-theoretic planning separates long-horizon strategic planning with approximate dynamics from tactical trajectory optimization with high-fidelity vehicle dynamics.

4. 1 AV + m HV

For one AV among many HVs, the literature combines stability-oriented control, game-based methods, and reinforcement learning. The AV problem is formulated as sequential decision-making under partial observability, with actions optimized against safety, efficiency, and emissions rewards.

  • Most studies use reinforcement learning for one-AV control because game-theoretic control faces the high dimensionality of coupled game systems.
  • Linear and nonlinear controllers, supported by field experiments, show that one AV can stabilize an HV platoon under deterministic conditions.
  • Game-based methods include iterative best-response control, Nash-equilibrium prediction of HV actions, and SVO-based modeling of human driving aggressiveness.Social preference learning improved AV performance by 25% in highway merging and unprotected left-turn validation.
  • A sequential game can combine multiple HVs into one effective human, with the AV using safe control constraints to incorporate human intentions.
  • An AV in HV-dominated traffic selects acceleration and steering policies from environmental information in a partially or fully observable random environment.
  • Policy learning spans behavioral cloning, deep reinforcement learning, and MCTS-assisted planning, with BC relying on driving data and DRL commonly trained in simulation.

5. n AVs: A Driverless World

The pure-AV literature organizes multi-vehicle control by cooperation and environmental determinism, using centralized, distributed, game-theoretic, and mean-field approaches. Scalability is a central concern as the number of controllable AVs grows.

  • Multi-AV control is classified as cooperative or non-cooperative according to whether vehicles optimize a common or individual objective.
  • Cooperative control: Most cooperative models assume communication, full observability, and full controllability to coordinate vehicles toward system-level goals.Reported goals include traffic-flow stabilization, driving comfort, and fuel-efficiency improvement.
  • Cooperative control: Centralized control is difficult because vehicle states and controls are coupled, while longer horizons introduce dimensionality and disturbance challenges.Distributed algorithms and short-horizon MPC are used to address these computational and prediction issues.
  • Non-cooperative control: Game-based control is generally limited to finite AV populations because computing dynamic equilibria becomes intractable as the number of coupled agents increases.The survey identifies scalable, computationally efficient algorithms as necessary for large AV populations.
  • Non-cooperative control: Mean field game models provide a scalable formulation for large AV populations and connect behavioral multi-agent dynamics with macroscopic traffic-flow theory.The LWR model is characterized as a myopic mean field game with a specially designed objective function.
  • Non-cooperative control: Compared with LWR, MFG mitigates traffic oscillations faster by using traffic density across the entire horizon rather than only at the vehicle’s location.MFG-controlled vehicles slow farther upstream before a jam and speed up immediately after leaving it.

6. n AV + m HV: controllable AVs navigating the HV-dominated traffic

When multiple controllable AVs navigate among human-driven vehicles, human behavior makes the environment stochastic and complicates scalable control. The literature combines stability analysis, model-free MARL, and decentralized or partially observable formulations.

  • Mixed-traffic multi-AV control is studied using vehicle cooperation as the primary category and environmental stochasticity as the secondary category.
  • Mixed-vehicle platooning: Stabilizing a platoon containing multiple AVs and HVs remains largely unsolved because stability depends on the topology of the mixed platoon.
  • Mixed-vehicle platooning: Head-to-tail stability approximations avoid enumerating mixed-platoon topologies, while subsystem decomposition provides a topology-aware alternative.
  • Mixed-vehicle platooning: P-step MPC is used to mitigate uncertainty in human-driver trajectories.
  • Mixed-vehicle platooning: PDE approximations and MFG-based continuum models support scalable mixed-traffic stability analysis involving HVs modeled by ARZ and AVs by MFG.Linear stability analysis quantifies the effects of AV penetration rate and controller design on ring-road traffic stability.
  • MARL-based control: Model-free MARL learns multi-AV policies from simulation transitions without explicitly defining a state-transition matrix.SUMO is used to simulate human drivers and uncontrolled AVs with IDM models in the cited studies.

MARL algorithms

MARL extends reinforcement learning to multiple AVs operating under partial observability and interacting with other agents. The surveyed approaches distinguish independent from joint learning and face trade-offs between local information, coordination, and scalability.

  • Independent learners use local observations to learn policies separately, whereas joint learners use shared global information to coordinate policies.
  • Independent learners: In decentralized multi-agent settings, each agent’s value estimates depend on the environment state and the joint actions of all agents.
  • Independent learners: Independent-agent learning loses standard convergence guarantees when other agents make the environment non-Markovian and non-stationary.Information exchange is one proposed response to this challenge.
  • Joint learners: Existing multi-AV MARL studies are few and mainly assume fully observable cooperative systems with a shared objective.The cited work uses TRPO policy gradients and transfers policies from multi-lane ring roads to highway merging scenarios.
  • Joint learners: Centralized learning with decentralized execution stabilizes training using global information but remains constrained by scalability as agent size grows.
  • Human driving behavior may adapt around AVs, but the direction and evolution of that adaptation remain unclear.The survey notes that low AV penetration prevents validating proposed moral-hazard hypotheses in the existing market.

7. Data-driven policy learning

Data-driven policy learning uses increasingly rich traffic and vehicle-sensor data to estimate human behavior and learn driving policies, while addressing limitations of conventional models.

  • Motivation: AV control must operate in dynamic, uncertain traffic containing human drivers and uncontrollable AVs.Human trajectories are also needed for safety-critical motion planning and behavior estimation.
  • Datasets: Trajectory data come from aggregate sensors, naturalistic driving systems, simulators, and emerging connected-vehicle and onboard sensing platforms.Real-time individual trajectory collection can be costly and raise privacy concerns, while simulators do not reproduce multi-vehicle interactions realistically.
  • Physics-based calibration: Car-following models calibrated from leader-follower pairs may miss disturbance propagation and fail to capture traffic instability.Common calibration approaches include maximum likelihood, Bayesian estimation, regression, and heuristics.
  • Policy learning: Machine learning learns driving policies from observations such as environmental states, acceleration, and steering angle.Vehicle detection and tracking provide a foundation for end-to-end perception-to-control policy training.
  • Human behavior estimation: Human intent estimation is commonly framed as classification, while future trajectories can be estimated with probabilistic and state-space models.Lane-change intent is more challenging than car-following partly because it requires identifying intended actions.
  • Limitations: Physics-based models simplify human decision-making and may have limited predictive power because parameters are estimated through an open-loop procedure.Their predefined heuristics may not capture strategic planning or generalize across diverse driving behaviors.

Imitation learning

Imitation learning models driving behavior from demonstrations, with approaches differing in whether they learn actions, rewards, or policies directly. The survey contrasts their error behavior, computational cost, realism, and applicability to AV data and interactions.

  • Imitation learning methods: Behavior cloning learns actions from expert demonstrations but can suffer cascading errors when deployed in unseen states.The problem arises because corrective actions for rarely visited states are scarce in the training data.
  • Imitation learning methods: DAgger mitigates behavior-cloning error propagation by augmenting the training data with expert demonstrations for missing states.One application maps ego and surrounding states plus a reasoning level to the corresponding action.
  • Imitation learning methods: Inverse reinforcement learning infers the expert’s underlying reward function instead of directly learning actions, helping address missing-state behavior.IRL assumes demonstrations arise from an approximately optimal policy under an unknown reward function.
  • Imitation learning methods: GAIL learns expert behavior through direct policy optimization, avoiding explicit reward recovery and offering faster learning than IRL in the survey’s comparison.The survey describes GAIL as more generalizable than behavior cloning across limited driving scenarios.
  • Limitations: GAIL-generated policies may produce unstable steering behavior and newer variants lacked thorough evaluation for driving-behavior modeling.The survey therefore presents GAIL as promising while identifying realism and evaluation concerns.
  • AV datasets: Public AV datasets are difficult to interpret because many datasets labeled as AV-related were collected by human-driven vehicles.Waymo and Lyft provide acceleration records suitable for AV policy training, whereas some datasets require acceleration inference.
  • AV datasets: Waymo and Lyft are identified as the only two public datasets describing Level-5 automated vehicles driving and interacting with road users on public roads.Their data combine accelerometer, camera, and LiDAR measurements and offer insights into AV driver models.
  • Waymo AV model: An LSTM model trained on Waymo data predicts acceleration in car-following scenarios, including acceleration after the lead truck turns away.The reported example shows predicted and ground-truth acceleration curves climbing simultaneously.

8. Model Summary

The survey summarizes physics-based and AI-based mixed-traffic models across control, efficiency, safety, and vehicle-interaction objectives.

  • Model categories: The model summary is organized into physics-based and AI-based mixed-traffic categories.The accompanying tables are labeled as physics-based and AI-based mixed traffic model summaries.
  • Objectives: The summarized models address objectives including safety, control, efficiency, travel efficiency, and kinetic energy.The extracted entries also include collision avoidance, task completion, lane changes, and route-related objectives.
  • Interactions and behaviors: The summary includes car-following and lane-change models, with cooperative and competitive settings represented in the table terminology.The table defines CF as car-following and LC as lane-change, and distinguishes cooperative from competitive approaches.
  • AI-based models: A listed GRU neural-network model is associated with system-level velocity and collision outcomes.The available table text reports that the GRU NN is close to desirable system-level velocity and includes collision as an outcome.

9. Conclusions and Open Questions

The paper identifies unresolved challenges in multi-agent autonomous driving and calls for research directions addressing cooperation, regulation, and social welfare in mixed autonomy.

  • Research Directions: The paper identifies unanswered questions and proposes promising research directions for AI-based mixed-autonomy modeling.Its stated aim is to guide future work rather than report a single empirical result.
  • Open Questions: Few successful MARL applications address complex multi-AV driving scenarios, with prior work emphasizing either centralized cooperative efficiency or decentralized collision-free driving.These approaches differ in computational burden and whether cooperation is explicitly modeled.
  • Open Questions: AV designers often optimize individual welfare, while city planners seek socially beneficial traffic outcomes, creating an unexplored upper-level control problem.The paper calls for socially optimal control schemes that address these competing goals.

9.2. Human Driving Policy Learning

Human driving policy learning is constrained by limited behavioral understanding, sparse and biased data, heterogeneous drivers, and the differing strengths of physics-based and AI-guided models.

  • Modeling Challenges: Physics-based models may capture limited real-world dynamics, whereas AI-guided models require massive data and may still miss observed traffic phenomena.The paper presents physics-informed deep learning as a way to mitigate these limitations.
  • Physics-Informed AI Models: Physics-informed deep learning is presented as a promising direction because existing traffic models constrain AI solutions with prior knowledge and can improve prediction with smaller datasets.This is particularly relevant when human-driver data are sparse and sample-biased.
  • Data and Calibration: Driving-model predictive power depends heavily on training and test datasets, while emergent stop-and-go waves or oscillations may arise across datasets.The paper notes that optimal data or experiment selection for robust calibration remains understudied.
  • Human Behavior: Human driving behavior remains poorly understood, even though robust mixed-traffic models must represent drivers’ differing capabilities, risk profiles, preferences, experience, and errors.The same environment and information can produce different maneuvers across drivers.

9.3. Multi-Scale Human-Machine Ecosystem Modeling

The paper frames mixed-autonomy modeling as a multi-scale problem linking microscopic vehicle behavior with macroscopic traffic patterns, especially in complex urban environments.

  • Multi-Scale Ecosystem: Understanding large-scale traffic patterns requires modeling both microscopic AV behavior and the evolving driving behavior of human drivers.The goal is to characterize the systematic effects of AVs on traffic safety and efficiency.
  • Multi-Scale Ecosystem: Urban mixed traffic extends beyond highway vehicle interactions to include traffic lights, pedestrians, cyclists, scooters, and other road users.This multimodal setting further complicates AV control.

9.4. Accountable, Fair, and Ethical AVs

Accountable, fair, and ethical AV decision-making raises engineering questions about responsibility, bias, privacy, and ethical behavior that existing qualitative studies have not yet integrated into engineering decisions.

  • Accountability: AV development must address accountability for damages or injuries caused by autonomous vehicles.The paper identifies responsibility assignment as a central question for automated systems making life-or-death decisions.
  • Fairness: AVs should make unbiased decisions, making fairness an explicit requirement for algorithmic decision-making.This concern is presented alongside accountability and ethics.
  • Ethics: Ethical AV development requires privacy-preserving data collection and use as well as ethical decisions by the vehicles.The paper treats data practices and vehicle behavior as connected ethical concerns.
  • Engineering Integration: How to integrate accountability, fairness, and ethics into engineering decisions remains unanswered despite qualitative studies on these topics.The paper identifies collaboration with legal experts and social scientists as key to progress.

9.5. A Pathway to Artificial General Intelligence (AGI)

The paper frames progress toward human-level intelligence in AVs as requiring broad cognitive capabilities and convergence across AI, engineering, cognitive science, and social science. It also emphasizes analytical tools and scalable control algorithms for managing the growing AV fleet and its societal impact.

  • Human-level intelligence in AVs requires reasoning, knowledge representation, planning, learning, natural-language communication, and integration toward common goals.
  • Reaching this capability requires bridging AI-tool gaps and converging engineering, cognitive science, and social science.
  • Rapid AV-fleet growth makes analytical tools for mixed traffic crucial to understanding transportation-system performance.
  • These tools can support scalable autonomous-driving control and help planners, policymakers, and lawmakers manage AVs for social good.
Loading 2007.05156v1…