PaperlayerView exact revision on arXiv

Source-linked AI summary

Human-Like Autonomous Car-Following Model with Deep Reinforcement Learning

Meixin Zhu, Xuesong Wang, Yinhai Wang

arXiv:1901.00569v1cs.LGcs.AIstat.ML

TL;DR

The paper addresses limitations in traditional car-following models, particularly their accuracy and generalization across drivers and situations. It trains deep deterministic policy-gradient models through trial-and-error interaction with a simulation environment and historical driving data. DDPGvRT, using speed deviation as reward and a 1s reaction delay, outperforms tested alternatives in trajectory reproduction and generalization while adapting to different drivers.

  • Problem

    Traditional car-following models have limited accuracy and generalization, motivating models that better reproduce human driving across drivers and situations.

  • Method

    Deep deterministic policy gradient models learn car-following policies from historical driving data through trial-and-error interaction with a simulation environment and reward function.

  • Results

    DDPGvRT, using speed deviation as reward and a 1s reaction delay, outperforms tested traditional and data-driven models in trajectory-reproducing accuracy and generalization.

  • Takeaways & Limitations

    Reinforcement learning can support human-like car-following models that adapt to different drivers through continuous learning.

  • Takeaways & Limitations

    The approach does not address absolute safety or human errors in historical driving data, which require complementary safety and anomaly-detection systems.

Abstract

from arXiv · show

This study proposes a framework for human-like autonomous car-following planning based on deep reinforcement learning (deep RL). Historical driving data are fed into a simulation environment where an RL agent learns from trial and error interactions based on a reward function that signals how much the agent deviates from the empirical data. Through these interactions, an optimal policy, or car-following model that maps in a human-like way from speed, relative speed between a lead and following vehicle, and inter-vehicle spacing to acceleration of a following vehicle is finally obtained. The model can be continuously updated when more data are fed in. Two thousand car-following periods extracted from the 2015 Shanghai Naturalistic Driving Study were used to train the model and compare its performance with that of traditional and recent data-driven car-following models. As shown by this study results, a deep deterministic policy gradient car-following model that uses disparity between simulated and observed speed as the reward function and considers a reaction delay of 1s, denoted as DDPGvRT, can reproduce human-like car-following behavior with higher accuracy than traditional and recent data-driven car-following models. Specifically, the DDPGvRT model has a spacing validation error of 18% and speed validation error of 5%, which are less than those of other models, including the intelligent driver model, models based on locally weighted regression, and conventional neural network-based models. Moreover, the DDPGvRT demonstrates good capability of generalization to various driving situations and can adapt to different drivers by continuously learning. This study demonstrates that reinforcement learning methodology can offer insight into driver behavior and can contribute to the development of human-like autonomous driving algorithms and traffic-flow models.

1 INTRODUCTION

The study motivates human-like autonomous car following as a way to support safer, more predictable interaction with human drivers. It proposes deep RL to address accuracy and generalization limitations in existing models using historical driving data and simulation-based learning.

  • Human-like driving can improve passenger comfort and confidence while making autonomous vehicles’ behavior more understandable to surrounding drivers.
  • Car-following models reproduce longitudinal driving trajectories and support microscopic traffic simulation and autonomous car-following systems.
  • Traditional models sacrifice flexibility and accuracy through simplified parameterizations and may generalize poorly across traffic scenarios and drivers.
  • Deep RL may improve approximation of complex stimulus–reaction relationships, generalization, and adaptation to regular drivers through continuous learning.
  • The proposed framework trains an RL agent in a simulation environment using historical data and rewards based on deviations from empirical driving.
  • The models were trained and tested on real-world data from the 2015 Shanghai Naturalistic Driving Study and compared for accuracy, generalization, and adaptivity.

2 BACKGROUND

Car-following research includes traditional stimulus-based, safety-distance, psycho-physical, and desired-measures models alongside newer data-driven approaches. This study frames deep reinforcement learning as a further data-driven approach using sequential agent–environment interactions and experience replay.

  • Car-following models describe a following vehicle’s motion relative to the lead vehicle and include GHR, IDM, optimal velocity, Helly, Gipps, and Wiedemann models.
  • Stimulus-based models determine acceleration from relative distance and relative speed, whereas safety-distance models maintain a gap sufficient for safe stopping.
  • Psycho-physical models represent driving states such as free driving, approaching, stable following, and emergency braking using relative-speed and relative-distance boundaries.
  • Desired-measures models continuously reduce differences between actual and preferred situations, and IDM incorporates both desired speed and desired following distance.
  • Data-driven car-following methods use high-fidelity traffic data and include nonparametric regression, conventional neural networks, recurrent neural networks, and deep neural networks.
  • The study introduces deep reinforcement learning for car-following, where an agent maps observed states to actions through rewards, trajectory-based state updates, and replay-buffer sampling of historical transitions.The state includes following-vehicle speed, inter-vehicle spacing, and relative speed; the action is following-vehicle longitudinal acceleration constrained between -3 m/s2 and 3 m/s2.

3 DATA PREPARATION

The study used SH-NDS naturalistic driving data, filtered and video-validated car-following periods, then sampled 2,000 periods from 20 drivers for analysis and style comparison.

  • Data source: SH-NDS collected real-world data from 60 Chinese drivers across 161,055 km using five instrumented vehicles.The acquisition systems recorded vehicle, control, radar, and video-related measurements at 10–50 Hz.
  • Period extraction: Car-following periods required a constant radar target, longitudinal distance below 120 m, lateral distance below 2.5 m, and duration exceeding 15 s.These filters selected sustained same-lane following rather than free-flow conditions.
  • Period validation: Analysts confirmed automatically extracted periods by reviewing video and retained only periods on limited-access expressways and freeways.The roadway restriction matched the study’s focus on limited-access driving behavior.
  • Analysis sample: 2,000 periods from 20 randomly selected drivers represented 827 minutes, with 70 periods per driver for calibration and 30 for validation.The periods were randomly sampled because training the models required substantial time and computing resources.
  • Driving styles: K-means clustering identified aggressive and conservative driving styles from speed, gap, and relative-speed features.Aggressive drivers maintained shorter mean space and time gaps than conservative drivers at similar mean following speeds.

4 APPROACH TO THE PROPOSED MODEL

The proposed model uses DDPG to generate continuous following acceleration in a simulated environment, rewarding agreement between simulated and observed behavior. Variants test spacing versus speed rewards and a 1s reaction-time delay.

  • Model choice: DDPG was selected because vehicle acceleration is continuous and DDPG performs in continuous action spaces.The method models car-following through deep deterministic policy gradient reinforcement learning.
  • Simulation environment: The simulation initializes following-vehicle speed, spacing, and velocity differences from empirical data, then iteratively updates states after agent-selected acceleration.The lead vehicle supplies externally controlled input, and the simulated trajectory is compared with SH-NDS observations.
  • Reward design: The reward function minimizes disparity between simulated and observed behavior, with both spacing and speed disparities tested.The DDPGs variant uses spacing disparity, whereas DDPGv uses speed disparity.
  • Network roles: The actor maps state inputs to continuous follower acceleration, while the critic evaluates each state–action pair with a scalar Q-value.The networks use the follower’s speed, relative speed, and spacing-related state variables.
  • Network architecture: The actor and critic each contain input, hidden, and output layers, with 30 hidden neurons and RLU activations; the actor output uses tanh to bound actions.Deeper networks were unnecessary for the three- or four-variable input setting.
  • Reaction time: DDPGvRT and DDPGsRT incorporate a 1s reaction delay using time-series inputs and expand hidden layers to 100 neurons.The larger layers accommodate 30 or 40 input variables in the reaction-time models.
  • Learning procedure: Training uses replay memory, critic loss minimization, actor policy gradients, slowly updated target networks, and temporally correlated Ornstein-Uhlenbeck exploration noise.The exploration process used θ = 0.15 and σ = 0.2.

5 TRAINING AND TEST

Models were trained and calibrated per driver, evaluated within-driver and across drivers, and compared against traditional and data-driven alternatives using held-out periods.

  • Calibration and validation: For each driver, 70 of 100 periods were used for calibration and 30 for intra-driver validation.Intra-driver error measured trajectory-reproducing accuracy on periods excluded from calibration.
  • Calibration and validation: Inter-driver validation applied each calibrated model to all 100 periods of each of the other 19 drivers.This error measured generalization beyond the driver used for calibration.
  • Adaptation: Retraining a DDPG model on another driver’s data quantified its ability to adapt to different drivers.The study used error reduction after retraining as the adaptation measure.
  • Models investigated: Seven models were compared: DDPGs, DDPGv, DDPGvRT, IDM, NNa, RNN, and Loess.The proposed models vary by reward type and reaction-time delay; DDPGsRT and NNaRT were omitted because they performed worse than corresponding no-delay models.
  • Baseline calibration: IDM calibration minimized spacing RMSPE using a genetic algorithm, with twelve stochastic optimization runs per driver.The parameter set with minimum RMSPE was selected.
  • DDPG training: DDPG training simulated the 70 calibration periods sequentially for 60 episodes, selecting the agent with the smallest combined training and test errors.The state was reinitialized from empirical data for each new period, and spacing RMSPE was tracked during training.

6 RESULTS

DDPGvRT achieved the strongest intra-driver validation performance among the seven models and tracked observed gap and speed variations closely in both stable and dynamic held-out situations.

  • Trajectory-reproducing accuracy: DDPGvRT had the lowest mean and standard deviation for both spacing and speed intra-driver validation errors among all seven models.The comparison used RMSPE of spacing and speed on validation datasets.
  • Trajectory examples: In one stable and one dynamic held-out car-following period, DDPGvRT tracked empirical gap and speed variations more closely than IDM and RNN.The dynamic case involved approaching a standing vehicle.

(a) Stable Following

The DDPGvRT model generalizes across drivers, showing lower spacing and speed validation errors than the investigated alternatives. Errors are lower within similar driver groups, and retraining enables adaptation to new drivers.

  • 20 ×20 error matrices evaluated models calibrated on one driver and applied to the remaining 19 drivers.
  • DDPGvRT achieved the lowest mean and standard deviation errors for both spacing and speed among the investigated models.
  • Inter-driver validation errors were higher than intra-driver errors, indicating substantial behavioral differences between drivers.
  • Intra-group validation errors were lower than inter-group errors, showing better generalization to drivers with similar driving styles.
  • During retraining, error curves initially remained stable until the replay buffer contained new experience data and learning began.

7 DISCUSSION AND CONCLUSION

The study concludes that DDPGvRT can reproduce human-like driving with strong generalization and driver adaptation, while acknowledging limits in safety handling, network depth, and experience replay. The model supports more predictable interaction with human-driven vehicles but does not address absolute safety.

  • DDPGvRT uses speed deviation as its reward and a 1s reaction delay, outperforming tested traditional and data-driven models in accuracy and generalization.
  • Reinforcement learning may improve generalization by learning decision-making mechanisms rather than estimating parameters through data fitting.
  • DDPGvRT incorporates reaction time and value functions, combining instantaneous traffic conditions with historical behavior and long-term rewards.
  • Inter-driver errors exceeded intra-driver errors, supporting continuous adaptation and the use of diverse driver archetypes in traffic simulation.
  • The study’s one-hidden-layer network is not truly deep but can be extended when additional input variables become available.
  • Prioritized experience replay is proposed as future work because uniform sampling ignores the potential significance of individual transitions.
  • After learning an initial policy, real-time acceleration generation requires only a forward pass, while policy updates need not occur in real time.
  • The approach does not address absolute safety or human errors in historical data; collision avoidance and anomaly detection are assigned to coupled systems.

A. Loess car-following model

The Loess model estimates car-following responses through locally weighted regression, using nearby observations of relative speed, gap, and follower speed. It selects weights by neighborhood distance and minimizes weighted residual errors.

  • The Loess model represents follower acceleration as a regression function of relative speed, gap, and following vehicle speed.The response variable is typically following vehicle acceleration, while predictor vectors contain the three car-following state variables.
  • Local regression estimates the response near x=x0 by fitting a regression surface to observations within a neighborhood controlled by span α.The span specifies the percentage of data used for each local fit; Papathanasopoulou and Antoniou used span = 0.4.
  • Observations in the neighborhood are weighted according to their Euclidean distance from the neighborhood center.The weighting uses the observation vector, neighborhood center, and distance to the most distant predictor value within the neighborhood.
  • A tri-cube weight function is used to assign local weights to observations.The variable u is used to estimate the weights for each observation.
  • The Loess parameters are obtained by minimizing the weighted residual sums of squares.The optimization incorporates the calculated observation weights.

B. Conventional neural network car-following model

The conventional feedforward neural network model predicts follower acceleration from car-following state variables and updates the simulated state after each prediction. It is trained to reduce differences between observed and predicted acceleration.

  • The NNa model predicts following vehicle acceleration from car-following gap and speed using a feedforward neural network.Its network architecture matches the actor network used by the DDPGv model.
  • At time step t, the actor network receives relative speed, follower speed, and spacing as its state input.The state contains the relevant car-following variables at the current time step.
  • The network outputs a continuous action representing the following vehicle’s acceleration.The predicted acceleration is then used to update the car-following simulation.
  • After acceleration is calculated, the following gap, speed, and relative speed are updated according to Equation (3).Training minimizes differences between observed acceleration values and network predictions.

C. Recurrent neural network (RNN) based car-following model

The RNN comparison model uses sequence processing and internal memory to predict follower acceleration from car-following states. Its recurrent updates learn weights and biases by back-propagating an objective through time.

  • The RNN maintains an internal state representing the current driving situation and uses internal memory for subsequent predictions.This allows the model to process sequences of inputs rather than isolated observations.
  • The recurrent computation combines the current input with the previous hidden state through input and hidden-state weights, followed by a ReLU activation.The hidden state represents internal memory at time t, and the output is generated from the recurrent computation.
  • The RNN output equation maps the hidden representation to an output using output weights and biases.The prose defines Wo and bo as output weights and biases and Ot as the output at time t.
  • The RNN takes gap, relative speed, and vehicle speed at time step t and outputs follower acceleration for the next time step.The follower’s position and velocity are then updated based on Equation (3), with 60 hidden units.
  • The RNN objective compares simulated spacing and speed with empirical observations and is optimized by back-propagating updates through time.The model gradually adjusts its weights and biases in the direction that optimizes the objective.
Loading 1901.00569v1…