PaperlayerView exact revision on arXiv

Source-linked AI summary

Modeling Human Driving Behavior through Generative Adversarial Imitation Learning

Raunak Bhattacharyya, Blake Wulfe, Derek Phillips, Alex Kuefler, Jeremy Morton, Ransalu Senanayake, Mykel Kochenderfer

arXiv:2006.06412v2cs.AI

TL;DR

Reliable autonomous-driving simulation needs human-behavior models despite unknown costs, stochasticity, and multi-agent interactions. The paper applies GAIL to real-world driving demonstrations and develops extensions for interaction, driving knowledge, and latent styles. Experiments on NGSIM show that these modifications model highway behavior and produce realistic traffic interactions.

  • Problem

    Autonomous-vehicle safety validation needs reliable simulation models of human driving despite unknown costs, complex behavior, and multi-agent interactions.

  • Method

    The paper learns neural driving policies with GAIL and introduces PS-GAIL, RAIL, and Burn-InfoGAIL for multi-agent interaction, domain knowledge, and latent variability.

  • Results

    Experiments on NGSIM show that GAIL variants capture desirable driving properties, with recurrent GAIL improving single-agent performance and multi-agent policies degrading more gradually as controlled-agent count increases.

  • Takeaways & Limitations

    The modifications support realistic highway-driving behavior and emergent traffic-flow behavior arising from interactions among learned driving agents.

Abstract

from arXiv · show

An open problem in autonomous vehicle safety validation is building reliable models of human driving behavior in simulation. This work presents an approach to learn neural driving policies from real world driving demonstration data. We model human driving as a sequential decision making problem that is characterized by non-linearity and stochasticity, and unknown underlying cost functions. Imitation learning is an approach for generating intelligent behavior when the cost function is unknown or difficult to specify. Building upon work in inverse reinforcement learning (IRL), Generative Adversarial Imitation Learning (GAIL) aims to provide effective imitation even for problems with large or continuous state and action spaces, such as modeling human driving. This article describes the use of GAIL for learning-based driver modeling. Because driver modeling is inherently a multi-agent problem, where the interaction between agents needs to be modeled, this paper describes a parameter-sharing extension of GAIL called PS-GAIL to tackle multi-agent driver modeling. In addition, GAIL is domain agnostic, making it difficult to encode specific knowledge relevant to driving in the learning process. This paper describes Reward Augmented Imitation Learning (RAIL), which modifies the reward signal to provide domain-specific knowledge to the agent. Finally, human demonstrations are dependent upon latent factors that may not be captured by GAIL. This paper describes Burn-InfoGAIL, which allows for disentanglement of latent variability in demonstrations. Imitation learning experiments are performed using NGSIM, a real-world highway driving dataset. Experiments show that these modifications to GAIL can successfully model highway driving behavior, accurately replicating human demonstrations and generating realistic, emergent behavior in the traffic flow arising from the interaction between driving agents.

I. INTRODUCTION

The paper frames human-driving simulation as an imitation-learning problem under stochastic, multimodal, latent, and nonlinear behavior, using GAIL and driving-specific extensions on real-world demonstrations.

  • I. INTRODUCTION: Reliable simulation requires representative human-driver models because autonomous-driving safety validation depends on accurate behavioral simulation.
  • I. INTRODUCTION: Human driving is difficult to model because it is stochastic, multimodal, influenced by latent factors, and governed by high-dimensional nonlinear dynamics.
  • I. INTRODUCTION: Imitation learning addresses unknown driving costs by learning policies from expert demonstrations rather than requiring an explicitly specified cost function.
  • I. INTRODUCTION: GAIL replaces restricted hand-designed cost classes with expressive learned functions and directly optimizes policies using TRPO.
  • I. INTRODUCTION: The paper trains neural policies from demonstrations and extends GAIL with parameter sharing, reward augmentation, and mutual-information maximization for multi-agent, domain-aware, and style-sensitive modeling.
  • I. INTRODUCTION: The study applies the approach to NGSIM highway data, whose varied densities, congestion, and vehicle interactions support behavioral evaluation and generalization testing.

IV. IMITATION LEARNING

Imitation learning seeks policies that reproduce expert behavior from demonstrations, but behavioral cloning can fail when the policy encounters states unlike those in training. The section formalizes policy trajectories and occupancy distributions to motivate interactive approaches.

  • Policy Formulation: A stochastic policy specifies the probability of each action in each state, while expert behavior is provided as trajectory samples rather than an explicit policy.
  • The imitation-learning pipeline feeds NGSIM demonstration data into an imitation-learning module to produce driving policies.
  • Occupancy Distributions: The state-action occupancy distribution ρπ(s, a) represents the state-action distribution encountered by policy π and expresses expected trajectory cost for any cost function.It is formed from the policy’s action probabilities and state-occupancy distribution.
  • Imitation learning learns a policy π that imitates an expert policy πE from state-action demonstrations.A demonstration is a sequence of state-action pairs generated as a policy interacts with the environment.
  • Behavioral Cloning: Behavioral cloning minimizes a loss over demonstrations, typically cross-entropy for discrete actions or negative log likelihood for continuous actions.
  • Behavioral Cloning: Covariate shift occurs because training samples expert states ρπE, whereas deployment samples states from the learned policy ρπsup, causing increasingly large errors.
  • Interactive training addresses covariate shift but requires an explicit or implicit reward function for states absent from the demonstrations.

A. Apprenticeship Learning

Apprenticeship learning seeks a policy that performs no worse than an expert despite an unknown true cost function. It obtains this guarantee by optimizing against the worst cost in a restricted cost-function class, provided the true cost belongs to that class.

  • Apprenticeship learning seeks a policy that performs no worse than the expert under the unknown true cost function.
  • The guarantee depends on assuming that the unknown true cost function ctrue belongs to the restricted cost-function class C.
  • The method finds the worst possible cost function for a candidate policy and then optimizes the policy against that worst-case function.
  • The resulting policy is guaranteed to perform no worse than the expert on the true cost function when the true cost belongs to the restricted class C.
  • Equation (9) provides a general framework requiring a cost-function class C and an optimization algorithm.
  • Classic apprenticeship learning restricts C to convex sets formed from linear combinations of known basis cost functions.If the true cost lies outside this class, the method cannot guarantee performance no worse than the expert.

V. GENERATIVE ADVERSARIAL IMITATION LEARNING

GAIL reformulates imitation learning to avoid restricting the cost function to a small hand-designed basis, instead matching expert and learned occupancy distributions. This supports imitation in large state-action spaces while retaining entropy regularization.

  • V. GENERATIVE ADVERSARIAL IMITATION LEARNING: GAIL is derived from cost-regularized maximum causal entropy inverse reinforcement learning, with entropy and a convex-conjugate regularizer shaping policy optimization.The convex conjugate assigns high cost to state-action pairs visited more often by the learner than by the expert.
  • V. GENERATIVE ADVERSARIAL IMITATION LEARNING: Under a constant cost regularizer, the dual objective seeks exact matching between learner and expert occupancy distributions.The corresponding optimization maximizes occupancy entropy subject to equality for every state-action pair.
  • V. GENERATIVE ADVERSARIAL IMITATION LEARNING: The exact occupancy-matching formulation is intractable for large MDPs because it imposes constraints across the full state-action space.Many occupancy entries would need to be zero because expert demonstrations cover only a limited dataset.
  • V. GENERATIVE ADVERSARIAL IMITATION LEARNING: Traditional apprenticeship learning can lose expert information when the true cost function lies outside its restricted basis-function space.This loss partially explains failures to imitate the expert well.
  • V. GENERATIVE ADVERSARIAL IMITATION LEARNING: GAIL replaces small basis-function cost restrictions with a regularizer based on divergence between expert and learner occupancy distributions.The formulation is intended to scale to large state-action spaces without manually specifying basis cost functions.

B. Connection to Generative Adversarial Networks

GAIL connects imitation learning to binary classification by distinguishing expert from learner state-action pairs. Because the environment is a black box, optimization alternates discriminator updates with TRPO policy updates rather than using ordinary GAN gradient descent.

  • B. Connection to Generative Adversarial Networks: GAIL converts classification of expert versus learner state-action pairs into an imitation-learning objective through the logistic loss.The resulting objective establishes the connection between GAIL and GANs.
  • B. Connection to Generative Adversarial Networks: The discriminator classifies state-action pairs drawn from expert or learner occupancy distributions, providing the bridge from binary classification to imitation learning.The classification setup supplies the cost-regularizer connection used by GAIL.
  • B. Connection to Generative Adversarial Networks: Unlike standard GANs, GAIL cannot differentiate through the environment, so it alternates discriminator gradient steps with TRPO updates to the policy.The environment is treated as a black box, making simultaneous gradient descent unsuitable.

C. Information Maximizing GAIL

Information-maximizing extensions introduce latent variables to represent distinct driving styles in otherwise similar demonstrations. Burn-InfoGAIL addresses a consistency problem by inferring the latent style from a burn-in demonstration before the learned policy takes over.

  • C. Information Maximizing GAIL: InfoGAIL introduces a latent variable that selects among a mixture of expert policies to represent variability such as aggressive versus passive driving.The latent variable is sampled from a prior and determines the policy component generating the trajectory.
  • C. Information Maximizing GAIL: InfoGAIL encourages the generated trajectory to reveal its latent code by maximizing mutual information between the code and the trajectory.The mutual information measures information about the latent variable obtained from the trajectory.
  • C. Information Maximizing GAIL: Because the true posterior over latent codes is difficult to access, InfoGAIL uses a variational lower bound with an approximate posterior Q(z | τ).The approximation is incorporated into the policy objective through an information-regularization term.
  • C. Information Maximizing GAIL: InfoGAIL may assign an inconsistent style when a policy starts from the end of a recorded human trajectory, limiting its use with playback-initialized driving scenes.A randomly sampled latent code need not match the demonstrator’s preceding behavior.
  • C. Information Maximizing GAIL: Burn-InfoGAIL conditions latent-code inference on a burn-in demonstration and takes over when the expert trajectory ends.This design targets consistency with the driver’s past behavior.

D. Extension to Multiple Agents

PS-GAIL extends GAIL to multi-agent driving by sharing policy parameters across agents while preserving distinct behavior through agent-specific observations. It addresses interaction-driven covariate shift and combines shared policies with TRPO for traffic-scene simulation.

  • D. Extension to Multiple Agents: The multi-agent formulation uses parameter sharing under assumptions of homogeneous agents, identical costs, and cooperative behavior.The paper explicitly notes that these assumptions are idealizations that do not fully hold in real driving scenes.
  • D. Extension to Multiple Agents: PS-GAIL enables imitation learning to scale from one driver to interacting vehicles by collecting trajectories from all agents under a shared policy.Agents can still exhibit different behavior because each receives unique observations.
  • D. Extension to Multiple Agents: Single-agent imitation policies can fail in multi-agent simulation because interaction-induced covariate shift compounds small behavioral errors over time.Agents encounter nearby vehicles behaving differently from those seen during single-agent training.
  • D. Extension to Multiple Agents: Centralized training scales poorly with agent count, whereas parameter sharing offers centralized learning with decentralized execution and fewer policy parameters.PS-TRPO combines parameter sharing with direct policy search for continuous-action multi-agent learning.
  • D. Extension to Multiple Agents: PS-GAIL combines GAIL with PS-TRPO to learn shared policies that control multiple vehicles and stabilize simulation of entire road scenes.The shared policy generates trajectories for all agents, while a critic supplies rewards and TRPO updates the policy.

VI. EXPERIMENTS

The experiments use a trajectory-based simulator to train policies from driving scenes, with observations built from vehicle and surrounding-scene information. Neural policies address nonlinear, high-dimensional, partially observable, and stochastic driving behavior.

  • VI. EXPERIMENTS: The simulator initializes scenes from real trajectories, selects vehicles for control, and feeds extracted scene features to the policy.Single-agent training controls one vehicle, whereas multiagent training controls M vehicles.
  • VI. EXPERIMENTS: The observation design is documented as a feature table describing the scene information provided to the policy.The supplied table passage identifies the observation-feature specification but does not enumerate its entries.
  • VI. EXPERIMENTS: Driving policies use neural networks to represent nonlinear mappings from high-dimensional scene states to actions.The state representation includes the ego vehicle, surrounding cars, and road conditions.
  • VI. EXPERIMENTS: The policies model stochastic driving by interpreting network outputs as the mean and diagonal covariance of a Gaussian action distribution.Actions are sampled from the policy conditioned on the current state.
  • VI. EXPERIMENTS: Feedforward and recurrent architectures are evaluated because sensor errors and occlusions make real-world driving partially observable.The recurrent policies add a GRU layer to the feedforward architecture.

C. Metrics

Performance is assessed through local trajectory imitation and undesirable traffic phenomena in simulator rollouts. GAIL produces more stable long-horizon behavior and generally outperforms behavioral-cloning baselines on safety-related metrics.

  • C. Metrics: The evaluation combines RMSE between generated and recorded trajectories with metrics for collisions, hard braking, and off-road driving.The latter metrics are compared with those extracted from NGSIM driving data.
  • C. Metrics: BC models perform competitively at short horizons but accumulate error over longer horizons, whereas GAIL produces more stable trajectories.The ego vehicle is simulated with acceleration and turn rate sampled from the policy network.
  • C. Metrics: GAIL policies outperform other methods on position, velocity, and lane-offset RMSE across prediction horizons.The figure caption reports the overall comparison without supplying numeric RMSE values.
  • C. Metrics: GAIL policies outperform BC on undesirable traffic metrics, while the GAIL GRU policy most closely matches the data except for hard brakes.The GAIL GRU policy rarely takes extreme actions, and only GAIL and IDM + MOBIL remain on the road for extended stretches.
  • C. Metrics: GAIL-based models achieve the lowest collision and off-road driving rates except for the rule-based controller, outperforming baseline and similarly structured BC models.The results also report improved performance from recurrent GAIL policies.

E. Multi Agent Imitation

PS-GAIL trains a shared policy across multiple vehicles while accounting for changing environment dynamics as more agents are controlled. A curriculum gradually increases the number of policy-controlled agents during training.

  • E. Multi Agent Imitation: PS-GAIL learns a shared-parameter policy from batched observations and actions across multiple vehicles, whose policies change the environment dynamics.The training procedure must therefore address non-stationary multiagent dynamics.
  • E. Multi Agent Imitation: A curriculum increases task difficulty by shifting training toward episodes with larger numbers of controlled agents.The simplified curriculum increments the number of controlled agents by a fixed number every K iterations.
  • E. Multi Agent Imitation: Recurrent policies with 64 GRUs are used to address partial observability from occluded vehicles, with one shared policy selecting actions for all vehicles.The observation is passed directly into the recurrent network without an initial dimensionality reduction.
  • E. Multi Agent Imitation: Training uses an initial low-discount, small-batch phase followed by higher-discount fine-tuning with a larger batch, while adding agents during the first phase.The fine-tuning phase uses 100 agents for the multiagent GAIL models.
  • E. Multi Agent Imitation: The critic supplies a scalar surrogate reward for each vehicle from its observation-action pairs and is implemented with a WGAN-GP architecture.The critic network uses (128,128,64) ReLU units.
  • E. Multi Agent Imitation: Figure 4 evaluates the single-agent and multiagent models as the number of policy-controlled agents increases in an NGSIM environment.Controlled vehicles are randomly sampled and replaced, while remaining vehicles retain their recorded trajectories.

1) Reward Augmentation:

Reward augmentation adds driving-specific penalties to imitation learning, improving long-horizon fidelity and reducing undesirable traffic outcomes. The broader experiments also show that parameter sharing and latent-style modeling address multi-agent interaction and behavioral variability.

  • 1) Reward Augmentation:: RAIL combines imitation-based critic costs with externally supplied penalties for off-road driving, hard braking, and collisions.The penalties provide domain knowledge and discourage states that could lead to collisions.
  • 1) Reward Augmentation:: Binary reward augmentation assigns the largest penalty R to collisions and off-road driving, while hard braking receives R/2.Hard braking is defined as acceleration below −3 m/s2.
  • 1) Reward Augmentation:: Smooth penalties increase gradually as vehicles approach off-road or hard-braking thresholds, providing earlier warning for reinforcement-learning credit assignment.The paper motivates this design as a way to address credit assignment difficulties.
  • 1) Reward Augmentation:: Reward augmentation improves local imitation performance, while PS-GAIL and RAIL outperform single-agent GAIL especially at longer prediction horizons.The comparison evaluates position, lane-offset, and speed RMSE across prediction horizons up to 20 s.
  • 1) Reward Augmentation:: RAIL produces fewer collisions, off-road events, and hard-braking events than the PS-GAIL baseline.These behaviors are explicitly penalized as undesirable traffic phenomena.
  • F. Disentangling Driving Styles: Burn-InfoGAIL achieves the lowest validation error over the longest driving period by disentangling latent driving-style variability.GAIL captures style differences for about 10 s before drifting toward an average policy, while the VAE suffers cascading errors without a simulator.

VII. CONCLUSION

The paper models uncertain human driving with imitation learning and introduces three GAIL modifications for multi-agent interaction, driving knowledge, and latent behavioral variability. Experiments demonstrate these modifications on real-world driving demonstrations, while future work targets broader models, methods, datasets, and applications.

  • VII. CONCLUSION: Human driving is modeled as sequential decision making under uncertainty with continuous states and actions, nonlinear stochastic behavior, and an unknown cost function.These properties motivate learning policies from demonstrations rather than specifying the driving cost directly.
  • VII. CONCLUSION: The paper introduces PS-GAIL, RAIL, and Burn-InfoGAIL to address multi-agent interaction, domain knowledge, and latent variability in human driving demonstrations.All three modifications are demonstrated in driver-modeling experiments using real-world driving demonstration data.
  • VII. CONCLUSION: Future work includes centralized interaction modeling with graph-based methods, alternative imitation-learning methods, larger datasets and domains, and applying learned models to autonomous-vehicle simulation validation.The paper presents these as directions for improving performance and applying learned driver models.
Loading 2006.06412v2…