PaperlayerView exact revision on arXiv

Source-linked AI summary

Driving Behavior Modeling using Naturalistic Human Driving Data with Inverse Reinforcement Learning

Zhiyu Huang, Jingda Wu, Chen Lv

arXiv:2010.03118v4cs.RO

TL;DR

The paper addresses interpretable, personalized modeling of human driving behavior for autonomous vehicles. It infers internal reward functions with maximum entropy IRL under discrete latent driving intentions and interaction-aware trajectory evaluation. Personalized modeling achieves lower human-likeness errors than general modeling and the stated baselines, while learned rewards remain robust in testing.

  • Problem

    Human-like autonomous driving requires models that capture interpretable decision motivations, surrounding-vehicle interactions, and diverse personalized preferences, while maximum entropy IRL is difficult in continuous, large driving spaces.

  • Method

    The paper infers interpretable internal reward functions using maximum entropy IRL, discrete latent intentions, candidate trajectory sampling, and an environment model with interactive vehicle behavior.

  • Results

    Personalized modeling reduced human-likeness error to Mean = 2.066 m, compared with general modeling at Mean = 2.681 m, IDM+MOBIL at Mean = 4.504 m, and constant velocity at Mean = 4.986 m.

  • Takeaways & Limitations

    Learned reward functions explicitly express different drivers’ preferences, and personalized reward functions improve modeling accuracy over general and baseline approaches.

Abstract

from arXiv · show

Driving behavior modeling is of great importance for designing safe, smart, and personalized autonomous driving systems. In this paper, an internal reward function-based driving model that emulates the human's decision-making mechanism is utilized. To infer the reward function parameters from naturalistic human driving data, we propose a structural assumption about human driving behavior that focuses on discrete latent driving intentions. It converts the continuous behavior modeling problem to a discrete setting and thus makes maximum entropy inverse reinforcement learning (IRL) tractable to learn reward functions. Specifically, a polynomial trajectory sampler is adopted to generate candidate trajectories considering high-level intentions and approximate the partition function in the maximum entropy IRL framework. An environment model considering interactive behaviors among the ego and surrounding vehicles is built to better estimate the generated trajectories. The proposed method is applied to learn personalized reward functions for individual human drivers from the NGSIM highway driving dataset. The qualitative results demonstrate that the learned reward functions are able to explicitly express the preferences of different drivers and interpret their decisions. The quantitative results reveal that the learned reward functions are robust, which is manifested by only a marginal decline in proximity to the human driving trajectories when applying the reward function in the testing conditions. For the testing performance, the personalized modeling method outperforms the general modeling approach, significantly reducing the modeling errors in human likeness (a custom metric to gauge accuracy), and these two methods deliver better results compared to other baseline methods.

I. INTRODUCTION

The paper models human-like and personalized driving by inferring interpretable internal reward functions rather than directly learning policies. It makes maximum entropy IRL tractable through discrete latent intentions, while addressing interaction awareness and driver diversity.

  • Motivation: Human-like, predictable, and personalized driving is important for autonomous vehicles operating safely among human drivers.The paper links current AV shortcomings to conservative or unnatural decisions and insufficient reasoning about surrounding agents.
  • Motivation: Internal reward functions are used because they represent action motivations and expose interpretable weights that can reflect different drivers’ preferences.The approach assumes rational drivers choose actions optimizing internal rewards.
  • Contribution: Discrete latent intentions convert continuous driving behavior into a tractable maximum entropy IRL problem.Drivers are modeled as generating, evaluating, and selecting candidate trajectories over long-term intentions.
  • Contribution: The study evaluates personalized and general modeling, interaction-aware environment simulation, and interaction features using naturalistic highway driving data.The reported contributions include qualitative and quantitative evaluation of accuracy, robustness, and planning with learned rewards.
  • Challenges: Maximum entropy IRL faces continuous, large state spaces, expensive reinforcement-learning evaluation, and an intractable partition function.Prior trajectory-based approaches may rely on local optimality, a single optimal trajectory, or static environments, limiting behavioral coverage.
  • Contribution: Personalized modeling assigns each driver a distinct reward function, addressing prior approaches that assume all vehicles share one common cost function.The paper contrasts this with general modeling, which uses a common cost function for all drivers.

III. METHODOLOGY

The methodology represents driving scenes, actions, trajectories, and rewards using a linear feature-based formulation. Maximum entropy IRL then infers reward weights from demonstrations under a latent-intention trajectory framework.

  • Problem formulation: A driving state contains the ego and surrounding vehicles’ positions, orientations, and velocities, while actions contain ego speed and steering controls.A finite sequence of states and actions forms a trajectory over the decision horizon.
  • Problem formulation: The structural assumption models latent high-level intentions through trajectory generation, evaluation, and selection rather than directly modeling low-level controls.This assumption is illustrated in the paper’s framework figure.
  • Reward formulation: The state reward is a weighted sum of selected features, with consistent weights assumed for the settled highway-driving scenario.The weight vector is θ and the feature vector characterizes the observed state.
  • Reward formulation: Trajectory reward accumulates feature contributions across the trajectory.The accumulated feature vector f_ζ is used to express the trajectory-level reward.
  • Reward inference: Given human demonstration trajectories, maximum entropy IRL infers reward weights that generate a policy matching those demonstrations.The structural assumption supports the reward-weight inference procedure.

B. Maximum entropy inverse reinforcement learning

Maximum entropy IRL models demonstrated driving trajectories with a Boltzmann distribution and adjusts reward parameters to maximize their likelihood. Because continuous trajectory spaces make the partition function intractable, sampled feasible trajectories approximate it and enable gradient-based optimization.

  • Trajectory probability follows a Boltzmann distribution based on trajectory returns, corresponding to maximum entropy IRL.
  • The partition function is intractable in continuous, high-dimensional trajectory spaces because it requires integrating over all possible trajectories.
  • A limited set of feasible generated trajectories approximates the partition function and converts trajectory probabilities into an easier-to-compute probability mass.
  • The objective maximizes the likelihood of expert demonstration trajectories under the modeled trajectory distribution.
  • The gradient compares feature expectations from human demonstrations with those from generated trajectories, and gradient ascent updates reward parameters until convergence.
  • L2 regularization is added to the reward-parameter objective to reduce overfitting.

C. Trajectory generation

The trajectory generator represents short-term driving plans through longitudinal and lateral target states. Polynomial trajectories satisfy boundary conditions for smooth, dynamically feasible motion and sample maneuvers spanning lane changes, lane keeping, and desired speed.

  • Drivers are modeled as selecting short-term plans toward end targets, with target speed longitudinally and lane-changing or lane-keeping tactically laterally.
  • Longitudinal position uses a fourth-order polynomial, while lateral position uses a fifth-order polynomial in time.
  • Start and target states, together with the required time T, define boundary conditions that determine the polynomial coefficients.
  • The trajectory horizon is 5 seconds, with position, velocity, and acceleration computed every 0.1 seconds.
  • Polynomial trajectories are generated by sampling target states Φ = {vxe, axe, ye, vye, aye}, covering possible maneuvers.
  • The sampled candidates span lane-changing, lane-keeping, and different desired longitudinal speeds.

D. Environment model

The environment model simulates generated ego trajectories and surrounding-vehicle responses on a reconstructed multilane highway. It combines recorded vehicle motion with reactive safe-distance behavior to estimate interactive outcomes.

  • The model simulates generated trajectories and surrounding-agent reactions, serving as a mental-world approximation for anticipating outcomes.
  • It reconstructs the NGSIM highway structure and spawns vehicles from recorded dataset states at a selected instant.
  • A pure-pursuit controller tracks the generated ego trajectory, while vehicle kinematics are propagated with a bicycle model.
  • Surrounding vehicles initially follow recorded trajectories but switch to IDM responses when gaps fall below desired safe distances.
  • The model assumes human drivers anticipate other agents’ reactions accurately, while acknowledging this setting may bias transition-function estimates.

E. Summary of the IRL algorithm

The IRL algorithm precomputes sampled trajectories in the environment model, stores their feature vectors, and uses them with naturalistic driving demonstrations. The evaluation uses NGSIM US-101 data containing detailed, interactive vehicle trajectories.

  • The algorithm initializes reward parameters, computes demonstration feature expectations, and generates candidate trajectories for each driving scene.
  • The environment model captures interactive cases including yielding during lane changes, rear-vehicle braking, and collisions caused by generated trajectories.
  • Each generated trajectory is rolled out in the environment model to obtain its feature vector before being added to a reusable buffer.
  • The trajectory-set size equals the number of longitudinal targets multiplied by the number of lateral targets.
  • The evaluation uses a 640-meter US-101 segment with five main lanes and an auxiliary lane, recorded at 10 frames per second.

B. Feature selection

The paper represents driving-state features through four aspects important to human drivers, including speed and ride comfort. Comfort is quantified using longitudinal and lateral acceleration and longitudinal jerk.

  • Driving-state features are mappings from state to real values that capture important properties of the state.
  • The speed feature reflects a human driver’s desire to reach the destination as fast as possible.
  • Ride comfort is measured using longitudinal acceleration, lateral acceleration, and longitudinal jerk.

3) Risk aversion:

The risk-aversion features model how drivers maintain safety relative to surrounding vehicles and potential collisions. Front- and rear-vehicle risks depend on time headway, while collision risk captures impacts with vehicles or curbs.

  • 3) Risk aversion:: Front-vehicle risk is modeled as an exponential function of the ego vehicle’s time headway to the nearest front vehicle.The formulation assumes constant-speed movement and uses the relative longitudinal positions and ego speed.
  • 3) Risk aversion:: Rear-end risk is likewise modeled as an exponential function of the time headway from the nearest rear vehicle to the ego vehicle.
  • 3) Risk aversion:: Collision risk is included because generated trajectories may collide with other vehicles or road curbs in the environment model.

4) Interaction:

The interaction feature represents how an ego vehicle’s behavior affects surrounding vehicles. The model estimates these effects through predicted deceleration reactions and evaluates the learned reward in testing scenes.

  • 4) Interaction:: The interaction feature sums predicted decelerations of surrounding vehicles affected by the ego vehicle’s behavior.It represents chain deceleration reactions caused by the ego vehicle changing its original course.
  • 4) Interaction:: An affected vehicle contributes the magnitude of its acceleration when that acceleration is negative.
  • 4) Interaction:: For real-world application, a prediction module can estimate surrounding agents’ reactions using driving models such as IDM.
  • 4) Interaction:: Trajectory features are accumulated over time, normalized to [0, 1], and assigned a fixed collision-feature weight of -10.Normalization removes unit and scale differences, while the fixed collision weight improved modeling accuracy relative to making it learnable.
  • 4) Interaction:: Testing evaluates whether learned reward functions retain similarity to human policies in scenes excluded from training.

4) Interaction factors:

The study evaluates interaction-aware driving behavior modeling through candidate trajectory sampling, training convergence, and testing on held-out trajectory segments. It also examines personalized versus general reward functions and uses representative driver cases to interpret learned preferences.

  • 4) Interaction factors:: Interaction-factor analysis compares including the interaction feature with simulating surrounding vehicles’ reactions to ego-vehicle course changes.
  • 4) Interaction factors:: Candidate trajectories vary longitudinal speed from v − 5 to v + 5 m/s in 1 m/s intervals and use initial, left-lane, or right-lane lateral positions.
  • 4) Interaction factors:: Training hyperparameters λ, α, and E are selected by grid search using the average likelihood of demonstration trajectories from 10 drivers.
  • 4) Interaction factors:: 35 of 50 five-second trajectory segments are used for reward learning, while the remaining 15 provide testing conditions.Each vehicle trajectory spans approximately 50 to 70 seconds and is partitioned into 50 segments.
  • 4) Interaction factors:: Smaller human likeness indicates better modeling accuracy when comparing the three highest-probability selected trajectories with the human ground truth.
  • 4) Interaction factors:: During training, log-likelihood increases and converges, feature difference decreases, and human likeness declines as trajectories closer to human behavior become more probable.
  • 4) Interaction factors:: Representative US-101 cases display candidate probabilities, human trajectories, and surrounding-vehicle context across lane-keeping, left-change, and right-change groups.Most drivers prioritize reducing front and rear risk, while speed, comfort, and interaction preferences vary.

B. Testing of accuracy and robustness

Testing shows that learned reward functions remain robust in untrained conditions, with personalized modeling matching human driving more closely than general and baseline models. Interaction-aware modeling improves naturalistic behavior accuracy and generalization, while inaccurate surrounding-vehicle forecasts degrade personalized performance.

  • Robustness testing: The learned reward functions show only slight deterioration in human likeness when selecting trajectories under testing conditions.This indicates robustness in untrained driving scenes.
  • Modeling accuracy testing: Mean human-likeness error was 2.066 m for personalized modeling, 2.681 m for general modeling, 4.504 m for IDM+MOBIL, and 4.986 m for constant velocity.Personalized and general reward modeling both outperformed the baseline models.
  • Modeling accuracy testing: The personalized method significantly outperformed general modeling in human likeness, with the difference reaching p < 0.001 in a t-test.General modeling still encoded basic requirements and common human-driving preferences.
  • Interaction factors: Removing interaction awareness from the reward function impaired modeling accuracy, indicating that interaction or courtesy factors matter for naturalistic driving behavior.The interaction factor was evaluated in the reward function and environment model.
  • Interaction factors: Not simulating surrounding-vehicle reactions could improve training performance but reduced testing generalization, likely because fixed paths bias partition-function estimation.Reactive surrounding vehicles make risky-looking sampled trajectories possible in real driving because other drivers can adapt.
  • Interaction factors: With an inaccurate forecasting model, personalized reward performance degraded, although it remained better than general modeling; general modeling stayed nearly unchanged.Forecasting errors affected estimated risk features, which often received high weights for individual drivers.

D. Discussions

The method supports personalized AV planning and surrounding-vehicle prediction by learning internal reward functions from naturalistic highway driving data. Its interaction-aware modeling improves accuracy and generalization, while broader deployment remains bounded by reward-function and trajectory-sampling assumptions.

  • Applications: Learned personalized cost functions can be integrated into AV trajectory planning to provide personalized driving experiences.The paper also identifies surrounding-vehicle motion prediction as another application.
  • Reward-function interpretation: The recovered reward function scores generated candidate trajectories rather than directly driving an agent interacting with the environment.Its applicability in the classical reinforcement-learning sense requires further investigation.
  • Limitations: Linear time-invariant rewards and hand-crafted features may not represent all real-world driving factors.The authors propose neural-network reward parameterization to model nonlinear rewards from raw sensory states.
  • Limitations: The trajectory sampling space may omit possible maneuvers, limiting estimation of the partition function and behavioral diversity.Suggested remedies include more target states, diversified planning horizons, and refined sampling.
  • Evaluation and interaction awareness: Personalized planning accuracy depends on the forecasting model but still exceeds planning with a general reward function.The method models surrounding-vehicle responses and incorporates ego-induced speed loss into trajectory evaluation.
  • Evaluation and interaction awareness: Removing response-action simulation can improve training results while compromising generalization, whereas removing interaction awareness can reduce modeling accuracy.Interaction awareness represents speed loss imposed on other vehicles by the ego vehicle's action.
Loading 2010.03118v4…