Source-linked AI summary
Imitating Driver Behavior with Generative Adversarial Networks
Alex Kuefler, Jeremy Morton, Tim Wheeler, Mykel Kochenderfer
TL;DR
Human driving models must remain realistic over long horizons despite cascading errors in behavioral cloning. The paper extends GAIL to recurrent policies and evaluates it in realistic highway simulations. GAIL models reduce collision and off-road rates while reproducing human-like emergent behavior, including lane-change rates.
Problem
Behavioral cloning can accumulate small prediction errors into poorly covered states, causing inaccurate behavior during long-horizon driving simulations.
Method
The paper extends Generative Adversarial Imitation Learning to recurrent policies for modeling highway driving from real-world trajectories.
Results
GAIL policies achieve low collision and off-road rates while matching human driving more closely in lane change rate than compared methods.
Takeaways & Limitations
GAIL-based recurrent driver models support realistic long-horizon highway simulation while capturing microscopic human-like behavior.
Abstract
from arXiv · showhide
The ability to accurately predict and simulate human driving behavior is critical for the development of intelligent transportation systems. Traditional modeling methods have employed simple parametric models and behavioral cloning. This paper adopts a method for overcoming the problem of cascading errors inherent in prior approaches, resulting in realistic behavior that is robust to trajectory perturbations. We extend Generative Adversarial Imitation Learning to the training of recurrent policies, and we demonstrate that our model outperforms rule-based controllers and maximum likelihood models in realistic highway simulations. Our model both reproduces emergent behavior of human drivers, such as lane change rate, while maintaining realistic control over long time horizons.
I. INTRODUCTION
Human driver modeling supports realistic driving simulation, but behavioral cloning can suffer cascading errors. This paper applies GAIL with recurrent policies to model highway driving from real-world trajectories.
- Accurate human driver models are important for realistic driving simulation and automotive-safety research.
- Behavioral cloning learns policies from fixed datasets of expert state-action pairs using supervised learning.
- Small prediction inaccuracies in behavioral cloning can compound, driving the policy into underrepresented and ultimately invalid states.
- Inverse reinforcement learning can generalize to unseen states by recovering a reward function and using reinforcement learning to find a matching policy.
- The paper extends GAIL to recurrent neural policies and evaluates them in a realistic highway simulator using real-world NGSIM trajectories.
A. Behavioral Cloning
Behavioral cloning maximizes likelihood on observed actions, but limited state coverage makes it vulnerable to cascading errors during sequential driving. These errors can produce collisions and off-road behavior, especially beyond short prediction horizons.
- Behavioral cloning obtains its policy by maximizing the likelihood of actions in the training data.
- Small action-prediction errors can compound into rarely observed states where poorer predictions create a feedback cycle.
- In highway driving, cascading errors can lead to off-road driving and collisions because datasets rarely cover such situations.
- Behavioral cloning predicts accurately for only a few seconds when learning nuanced general driving behavior with potential to leave the lane.
- Reinforcement learning provides signals across all state-action pairs and optimizes global trajectory return, supporting robustness against cascading errors.
III. POLICY REPRESENTATION
The policy representation uses stochastic neural networks to map complex driving observations to continuous actions. Recurrent policies add memory to address partial observability and are compared with feedforward alternatives under both BC and GAIL.
- Neural networks represent nonlinear, high-dimensional, and stochastic mappings from driving states to actions.
- Policy outputs parameterize the mean and logarithm of diagonal covariance for a Gaussian action distribution, from which actions are sampled.
- Feedforward policies can be limited by partial observability when sensor error or occlusion hides relevant driving information.
- Recurrent policies use memory of past observations to disambiguate perceptually similar driving states.
- The experiment trains GAIL GRU, GAIL MLP, BC GRU, and BC MLP policies using recurrent and feedforward architectures.
IV. POLICY OPTIMIZATION
The paper addresses unknown human rewards by training policies with reinforcement learning and GAIL. A discriminator distinguishes expert from policy state-action pairs, producing surrogate rewards for TRPO updates.
- Because human drivers’ reward function is unknown, the paper separates policy optimization from learning a reward signal.
- Policy-gradient optimization uses simulated rollouts, while TRPO constrains policy updates to limit damage from noisy gradient estimates.
- A hand-crafted surrogate reward can encode avoiding collisions and off-road driving while favoring smooth, centered driving.
- GAIL trains a policy to produce expert-like behavior by rewarding it for deceiving a discriminator that separates policy and expert state-action pairs.
- The discriminator objective is optimized by minibatch gradient ascent, while the policy is optimized through reinforcement learning because the objective is nondifferentiable in policy parameters.
- GAIL’s surrogate rewards increase as policy state-action tuples become indistinguishable from expert tuples and guide policies toward expert-explored regions.
V. DATASET
The study uses public NGSIM trajectories from US Highway 101 and Interstate 80, covering diverse traffic conditions and substantial vehicle interaction.
- V. DATASET: NGSIM provides 45 minutes of driving at 10 Hz for each roadway, covering US Highway 101 and Interstate 80.Highway 101 spans approximately 640 m with five mainline lanes and one auxiliary lane; Interstate 80 spans approximately 500 m with six mainline lanes.
- V. DATASET: The datasets transition from uncongested traffic to full congestion and include frequent merging and interactions among vehicles.These varied conditions make the trajectories useful for studying driver behavior across changing traffic regimes.
- V. DATASET: The recorded trajectories were smoothed with an extended Kalman filter and projected onto lanes using extracted centerlines.
VI. EXPERIMENTS
The experiments train and evaluate highway-driving policies using GAIL and behavioral cloning in a simulated environment with recurrent or feedforward neural architectures and rich vehicle-context features.
- VI. EXPERIMENTS: GAIL and BC learn policies for two-dimensional highway driving, with performance evaluated against baseline models.
- A. Environment: Simulations use NGSIM 80 and 101 road networks, run for 100 steps at 10 Hz, and terminate after collision, off-road driving, or reverse travel.
- B. Features: The feedforward policy network outputs mean µ and covariance parameters ν that define a Gaussian distribution over driver actions.
- B. Features: Each model receives a 51-element feature vector combining core vehicle and lane-state features, LIDAR-like range measurements, and undesirable-state indicators.The LIDAR-like representation uses 20 range and range-rate beams with 100 m maximum range and complete 360° coverage.
- B. Features: Previous actions were excluded because including them caused policies to over-rely on action history and harmed performance even after mitigation attempts.
C. Baseline Models
The evaluation compares learned policies with a static maximum-likelihood Gaussian, a mixture-regression behavioral-cloning model, and a rule-based highway controller.
- C. Baseline Models: The static Gaussian baseline is an unchanging distribution π(a | s) = N(a | µ, Σ) fitted using maximum likelihood.
- C. Baseline Models: The mixture-regression baseline is a behavioral-cloning model trained with expectation maximization over a Gaussian mixture of actions and features.Its stochastic policy conditions the weighted Gaussian components on features, with greedy feature selection minimizing the Bayesian information criterion.
- C. Baseline Models: The rule-based controller uses IDM for longitudinal motion and MOBIL plus proportional lane-centerline tracking for lateral motion.
D. Validation
Validation simulates many repeated highway scenes and measures how closely each model’s stochastic trajectories match recorded driver behavior over multiple prediction horizons.
- D. Validation: Each model was tested on 1,000 ten-second scenes simulated 20 times each in the training environment.The rollouts produced several metrics quantifying how well models simulate human driver behavior.
- 1) Root-Weighted Square Error:: RWSE measures deviation between a model’s probability mass and real-world trajectories by comparing recorded paths with sampled simulated traces.The procedure samples n = 20 simulated traces per recorded trajectory.
- 1) Root-Weighted Square Error:: RWSE is computed for global position, centerline offset, and speed across time horizons extending to 5 s.The metric compares the true value for each trajectory and horizon with the corresponding simulated variable across samples.
2) Kullback-Leibler Divergence:
The paper evaluates imitation using distributional similarity and emergent driving metrics, while cautioning that simulator collisions may be artificially high.
- KL divergence measures how closely simulated and real-world distributions match for speed, acceleration, turn-rate, jerk, and inverse time-to-collision.The distributions use 100 evenly spaced piecewise-uniform bins.
- Emergent metrics include lane change rate, offroad duration, collision rate, and hard brake rate.These metrics complement distributional comparisons with trajectory-level behavioral indicators.
- Lane change rate counts average lane changes within a 10-second trajectory, while offroad duration counts steps spent more than 1 m outside the outer road marker.
- Collision rate is the fraction of trajectories intersecting another participant, and hard brake rate measures braking harder than −3 m/s2.
- Pre-recorded non-ego trajectories can produce artificially high collision counts when they fail to respond to ego deviations.The hard brake rate is also extracted to quantify dangerous driving situations.
VII. RESULTS
GAIL models produce more stable and realistic long-horizon driving behavior than behavioral cloning and other baselines, while recurrent GAIL improves fidelity and captures human-like emergent behavior.
- Feedforward behavioral cloning performs best at short horizons but accumulates error over longer prediction horizons, whereas GAIL produces more stable trajectories.The controller’s nearly constant lane-offset error also indicates that human drivers do not always closely follow the nearest lane centerline.
- GAIL GRU performs well on iTTC, speed, and acceleration, but poorly on turn-rate and jerk because it oscillates between small actions more than humans.Its average actions resemble human actions, but alternating small positive and negative turn-rates increases mismatch in these metrics.
- GAIL policies outperform behavioral cloning on emergent variables, with GAIL GRU closest to the data except for hard brakes.GAIL GRU rarely takes extreme actions, while mixture regression remains susceptible to cascading errors.
- Only GAIL and IDM + MOBIL stay on the road for extended stretches, while SG’s lack of hard braking contributes to its high collision rate.The collision rate of GAIL GRU is roughly the same as IDM + MOBIL in this simulation, possibly because other vehicles cannot fully react to the ego vehicle.
- GAIL achieves lower collision and off-road driving rates than baseline and similarly structured behavioral cloning models, while matching human lane-change rates more closely than other compared methods.The hand-coded controller is the exception to GAIL’s lowest collision and off-road driving rates.
- Extending GAIL to recurrent policies improves performance, unlike recurrence in behavioral cloning, which largely does not improve results.Recurrence alone is insufficient to address cascading errors in behavioral cloning policies.
VIII. CONCLUSIONS
The paper extends Generative Adversarial Imitation Learning to recurrent policies for realistic, long-horizon highway driving. It reports improved long-term stability and human-like behavior, while behavioral cloning remains stronger over short horizons.
- Generative Adversarial Imitation Learning enables long-term, stable trajectories by overcoming cascading errors.Behavioral cloning’s greedy behavior prevents realistic driving over extended periods.
- The recurrent-policy model captures microscopic, human-like driving behavior while performing realistically over long time horizons.
- Behavioral cloning outperforms Generative Adversarial Imitation Learning on short (∼2 s) horizons.
- Future work may combine learned surrogate rewards with engineered rewards to model driver style or penalize acceleration and turn-rate oscillations.