Source-linked AI summary
Multi-Agent Imitation Learning for Driving Simulation
Raunak P. Bhattacharyya, Derek J. Phillips, Blake Wulfe, Jeremy Morton, Alex Kuefler, Mykel J. Kochenderfer
TL;DR
Simulation can support vehicle-safety validation, but realistic scenes require driver models that interact well over time. The paper extends GAIL with parameter sharing and a training curriculum for multi-agent driving. Compared with single-agent GAIL, PS-GAIL better matches human trajectories and supports more stable interactions, including over longer horizons.
Problem
Single-agent imitation-learned driver models have difficulty generalizing to multi-agent scenes, where realistic interactions are needed for driving simulation.
Source: §I · p. 1Method
PS-GAIL extends GAIL using parameter sharing and a training curriculum to train human-driver models in a multi-agent setting.
Source: B. Parameter Sharing GAIL · p. 3, §VI · p. 6Results
PS-GAIL better matches expert driving behavior and supports more stable multi-agent interactions, with fewer collisions, extreme decelerations, and off-road events than single-agent policies.
Source: §I · p. 1, B. Results and Discussion · p. 5Takeaways & Limitations
The learned multi-agent model exhibits more realistic behavior than single-agent models, particularly over longer time horizons.
Source: §VI · p. 6The feed-forward critic cannot account for partial observability of the scene.
Source: C. Critic · p. 4
Abstract
from arXiv · showhide
Simulation is an appealing option for validating the safety of autonomous vehicles. Generative Adversarial Imitation Learning (GAIL) has recently been shown to learn representative human driver models. These human driver models were learned through training in single-agent environments, but they have difficulty in generalizing to multi-agent driving scenarios. We argue these difficulties arise because observations at training and test time are sampled from different distributions. This difference makes such models unsuitable for the simulation of driving scenes, where multiple agents must interact realistically over long time horizons. We extend GAIL to address these shortcomings through a parameter-sharing approach grounded in curriculum learning. Compared with single-agent GAIL policies, policies generated by our PS-GAIL method prove superior at interacting stably in a multi-agent setting and capturing the emergent behavior of human drivers.
I. INTRODUCTION
Simulation offers a safer, faster, and more economical route to vehicle-safety validation, but it depends on realistic models of surrounding drivers. Single-agent imitation policies fail to behave realistically in multi-agent traffic; PS-GAIL extends GAIL and parameter sharing to control multiple vehicles, with more human-like trajectories and stable long-horizon interactions.
I. INTRODUCTION
- Simulation can evaluate vehicle performance across many scenes quickly, safely, and economically, unlike costly and potentially dangerous real-world tests.The passage also notes that a statistically significant safety case based solely on real-world testing is likely infeasible,.Source: §I · p. 1
- Useful driving simulations require realistic human-driver models to govern non-autonomous vehicles on the roadway.Source: §I · p. 1
- Single-agent imitation policies fail to exhibit realistic behavior in multi-agent driving, making them inadequate for simulation.The paper attributes this deterioration to covariate shift when transitioning from single-agent to multi-agent settings.Source: §I · p. 1
- PS-GAIL extends GAIL and PS-TRPO to generate policies that control multiple vehicles simultaneously.This enables simulation of complex roadway scenes.Source: §I · p. 1
- PS-GAIL policies produce trajectories that better match human drivers and interact more stably over long horizons than single-agent GAIL policies.They are also less prone to collisions and off-road events during interactions.Source: §I · p. 1
A. Markov Decision Processes
The background introduces Markov decision processes and imitation learning, then explains why distribution shift limits learned driver policies. GAIL addresses that issue when training and testing environments match, but single-agent training does not match multi-agent deployment.
A. Markov Decision Processes
- An infinite-horizon MDP is defined by state and action spaces, a transition model, a reward function, and a discount factor.The transition model gives next-state probabilities, while the discount factor weights future rewards relative to immediate ones.Source: A. Markov Decision Processes · p. 1
- A partially observable MDP adds an observation space and an observation model that specifies the probability of observing o in state s [9],.Source: A. Markov Decision Processes · p. 2
B. Imitation Learning
- Imitation learning learns a policy from expert demonstrations, while behavioral cloning minimizes a loss on demonstrations but can suffer covariate shift,.Behavioral cloning trains on expert-occupied states but runs on states occupied by its learned policy, where errors can grow beyond recovery.Source: B. Imitation Learning · p. 2
- GAIL uses a discriminator to distinguish expert from generated state-action pairs, with its output serving as a surrogate reward for the policy.TRPO updates the policy using these rewards, which guide it toward state-action regions explored by experts.Source: B. Imitation Learning · p. 2
- GAIL addresses covariate shift when training and testing environments match, but single-agent training differs from multi-agent deployment, where nearby vehicles act differently.The paper therefore formulates driver-model learning as a multi-agent problem.Source: B. Imitation Learning · p. 2
C. Multi-agent reinforcement learning
Multi-agent reinforcement learning offers a general alternative to hand-engineered multi-agent control, but centralized policies scale poorly with agent count. For homogeneous agents, parameter sharing offers a decentralized policy approach that shares parameters and experience.
C. Multi-agent reinforcement learning
- Centralized multi-agent policies map joint observations to joint actions, and their computational complexity grows rapidly as the number of agents increases.Source: C. Multi-agent reinforcement learning · pp. 2–3
- For homogeneous agents, the joint policy distribution can factor into per-agent policies, scaling linearly with the number of agents.Homogeneity means each policy performs essentially the same observation-to-action mapping.Source: C. Multi-agent reinforcement learning · p. 3
- PS-TRPO combines parameter sharing with TRPO to produce decentralized policies that can exhibit emergent cooperation without explicit communication.Sharing reduces the number of parameters by a factor of M and pools experience across agents, while unique observations still allow different behaviors.Source: C. Multi-agent reinforcement learning · p. 3
- PS-TRPO updates policy parameters through a constrained optimization problem whose KL-divergence bound limits the maximum policy change per step.The advantage function compares an action's value with the baseline for its observation.Source: C. Multi-agent reinforcement learning · p. 3
III. APPROACH
The approach extends GAIL to control multiple human-driver models simultaneously. It frames the multi-agent driving problem and describes methods for addressing its challenges.
III. APPROACH
- The proposed approach extends GAIL to enable simultaneous control of multiple human-driver models.Source: §III · p. 3
- The approach targets multi-agent driving, where multiple human-driver models must be controlled together.Source: §III · p. 3
- The following subsections describe the multi-agent driving problem and the approach to combating its challenges.Source: §III · p. 3
A. Problem Formulation
The paper formulates multi-agent driving as a Markov game with homogeneous agents, independent rewards, and an identical reward function. It notes these assumptions are idealizations that can apply approximately and support learning realistic policies.
A. Problem Formulation
- The formulation assumes agents share observation and action spaces, receive rewards independent of other agents’ actions and next states, and use the same reward function.These are described as simplifying assumptions for a Markov game of M agents.Source: A. Problem Formulation · p. 3
- PS-GAIL takes expert trajectories, shared policy parameters, discriminator parameters, and a trust-region size as inputs, then updates the policy and critic using trajectories from all agents.The algorithm scores rollout trajectories with a critic, uses a TRPO policy step, and updates critic parameters.Source: A. Problem Formulation · p. 3
- The assumptions do not fully match real driving, where vehicle capabilities, lane-change decisions, and driver preferences can differ.The authors say the assumptions often apply approximately and allow realistic policies to be learned.Source: A. Problem Formulation · p. 3
B. Parameter Sharing GAIL
Single-agent training can reintroduce covariate shift when its policy is deployed across many interacting vehicles. PS-GAIL uses shared policies and a training curriculum to support multi-agent driving under changing environment dynamics.
B. Parameter Sharing GAIL
- A policy trained with one controlled vehicle can fail when deployed across many vehicles because introducing it to other agents reintroduces covariate shift.Small errors in one vehicle can destabilize neighbors and ultimately lead to failures across the scene.Source: B. Parameter Sharing GAIL · p. 3
- PS-GAIL shares one policy and critic across agents, using their trajectories for critic rewards and policy and critic updates.The approach combines GAIL with PS-TRPO and is described as a special case of algorithms by Song et al..Source: B. Parameter Sharing GAIL · p. 3
- The policy is recurrent because observations are high-dimensional, the observation-to-action mapping is nonlinear, and driving scenes are partially observable.Partial observability arises from sensor noise, occlusions, and unobserved driver traits and intended maneuvers.Source: B. Parameter Sharing GAIL · p. 3
- Because multi-agent policies change environment dynamics during training, the curriculum gradually shifts toward episodes with more controlled agents.In practice, the number of controlled agents increases by a fixed amount every K iterations.Source: B. Parameter Sharing GAIL · pp. 3–4
IV. IMPLEMENTATION
The implementation section describes the simulator used to train multi-agent policies and the implementation and training of policy and critic networks.
IV. IMPLEMENTATION
- The implementation section covers the simulator for training multi-agent policies and the implementation and training of policy and critic networks.Source: §IV · p. 4
A. Simulator
The simulator replays real trajectories while advancing policy-controlled vehicles from their selected actions. The implementation uses recurrent policies and a two-phase training procedure, with the multi-agent curriculum increasing the number of controlled vehicles.
A. Simulator
- Episodes start from NGSIM trajectory states, select vehicles for policy control, and repeat policy observations and actions for 200 timesteps at 10 Hz.The episode horizon corresponds to 20 seconds of driving.Source: A. Simulator · p. 4
B. Policies
- The policy is a 64-GRU recurrent network, and Table I describes the features extracted for its observations.A single shared policy selects actions for all vehicles in the multi-agent setting.Source: B. Policies · p. 4, Table I · p. 4, A. Simulator · p. 4
- Training uses 1,000 initial iterations followed by 200 fine-tuning iterations, while multi-agent training adds 10 controlled agents every 200 initial-phase iterations and uses 100 agents during fine-tuning.The initial phase uses discount 0.95 and batches of 10,000 observation-action pairs; fine-tuning uses discount 0.99 and batches of 40,000.Source: B. Policies · p. 4
C. Critic
The section describes the critic and evaluation setup for comparing learned driving policies. It reports that multi-agent training better matches human driving as prediction horizons grow.
C. Critic
- The critic scores each vehicle's observation-action pair and supplies that scalar as its reward.It is a WGAN-GP feed-forward network; critic training uses a replay memory containing samples from the three most recent epochs.Source: C. Critic · p. 4
- The feed-forward critic cannot account for scene partial observability, while recurrent alternatives pose reward-timing and credit-assignment challenges.The authors leave recurrent critics to future work because the feed-forward critic was sufficient for good driving policies.Source: C. Critic · pp. 4–5
- Multi-agent policies more closely match human drivers, especially at longer prediction horizons.The comparison is root weighted square error versus prediction horizon.Source: Figure 1 · p. 5
V. EXPERIMENTS
- Three models of each type are evaluated using averages over 10,000 rollouts in a 100-agent environment.The models are GAIL and multi-agent GAIL policies for two-dimensional highway driving.Source: §V · p. 5
A. Evaluation Metrics
- Evaluation measures trajectory RMSE for global position, lane offset, and speed through 20 s, alongside off-road duration, collision rate, and hard-brake rate.Off-road means more than 1 m beyond a road edge; the hard-brake threshold is −3 m s−1.Source: A. Evaluation Metrics · p. 5
B. Results and Discussion
PS-GAIL more faithfully matches expert trajectories, with the clearest advantage at longer horizons. Its performance also degrades more gradually than single-agent GAIL as the number of controlled vehicles increases.
B. Results and Discussion
- PS-GAIL captures expert behavior more faithfully than single-agent GAIL, especially at longer prediction horizons where single-agent errors accumulate rapidly.The comparison covers prediction horizons up to 20 s.Source: B. Results and Discussion · p. 5, Figure 3 · p. 5
- PS-GAIL policies are less likely to lead vehicles into collisions, extreme decelerations, or off-road driving, indicating stabler agent interactions.These outcomes are reported as validation results illustrated by Fig. 2.Source: B. Results and Discussion · p. 5
- As more vehicles are controlled, single-agent GAIL performance deteriorates rapidly, while PS-GAIL declines more gradually.Fig. 3 reports average RMSE across episode timesteps; at each setting, controlled vehicles replace that many vehicles in the NGSIM scene.Source: Figure 3 · p. 5, B. Results and Discussion · p. 6
VI. CONCLUSIONS
The work targets realistic human-driver models for simulated scenes involving multiple vehicles. It uses parameter sharing and a training curriculum; experiments report more realistic behavior than single-agent models, particularly over longer time horizons.
VI. CONCLUSIONS
- Real-world autonomous-vehicle safety validation is costly, dangerous, and time-consuming, motivating the use of simulated environments.Simulation requires highly realistic driver models for interacting vehicles.Source: §VI · p. 6
- The multi-agent model exhibits significantly more realistic behavior than existing single-agent models, particularly over longer time horizons.Experiments compared the multi-agent model with existing single-agent models.Source: §VI · p. 6
- The models are trained in a multi-agent setting using parameter sharing to address scaling issues and a curriculum to address learning instability.The work focuses on driver models that interact with other controlled vehicles in a manner representative of real data.Source: §VI · p. 6
- Future work will investigate improving model performance and applying learned driver models to autonomous-vehicle validation in simulation.Proposed improvement directions include reward augmentation, more diverse behavior, and a recurrent critic for partial observability.Source: §VI · p. 6