Source-linked AI summary
Efficient Sampling-Based Maximum Entropy Inverse Reinforcement Learning with Application to Autonomous Driving
Zheng Wu, Liting Sun, Wei Zhan, Chenyu Yang, Masayoshi Tomizuka
TL;DR
Human-driving reward functions are difficult to specify, yet they matter for safe, transparent interaction between autonomous vehicles and people. SMIRL applies maximum-entropy IRL with continuous-domain trajectory sampling to learn rewards from driving data. It reports improved prediction metrics across interactive and non-interactive scenarios and better generalization than GCL in unseen environments.
Problem
Learning from real driving demonstrations must accommodate high-dimensional continuous trajectories and demonstrations that may be uncertain or sub-optimal, while reward misspecification can undermine safe, transparent interaction.
Source: §I · p. 1Method
SMIRL combines maximum-entropy IRL with an efficient sampler that generates feasible driving trajectories to estimate the partition term.
Source: §I · p. 2, A. Maximum-entropy IRL · p. 2, Step I - Path Sampling via Discrete ED · p. 3Results
SMIRL achieved smaller MEDs and variances than CIOC, Opt-IRL, and GCL, and higher test-trajectory likelihoods than all three in both driving scenarios; model-based methods generalized better than GCL in unseen environments.
Source: 2) MED · p. 6, 3) Probabilistic Metric · p. 6, B. Performance on Test Sets in Unseen… · p. 7Takeaways & Limitations
Abstract
from arXiv · showhide
In the past decades, we have witnessed significant progress in the domain of autonomous driving. Advanced techniques based on optimization and reinforcement learning (RL) become increasingly powerful at solving the forward problem: given designed reward/cost functions, how should we optimize them and obtain driving policies that interact with the environment safely and efficiently. Such progress has raised another equally important question: \emph{what should we optimize}? Instead of manually specifying the reward functions, it is desired that we can extract what human drivers try to optimize from real traffic data and assign that to autonomous vehicles to enable more naturalistic and transparent interaction between humans and intelligent agents. To address this issue, we present an efficient sampling-based maximum-entropy inverse reinforcement learning (IRL) algorithm in this paper. Different from existing IRL algorithms, by introducing an efficient continuous-domain trajectory sampler, the proposed algorithm can directly learn the reward functions in the continuous domain while considering the uncertainties in demonstrated trajectories from human drivers. We evaluate the proposed algorithm on real driving data, including both non-interactive and interactive scenarios. The experimental results show that the proposed algorithm achieves more accurate prediction performance with faster convergence speed and better generalization compared to other baseline IRL algorithms.
I. INTRODUCTION
The introduction frames reward recovery from human driving data as an important counterpart to optimizing a specified reward, since misspecification can undermine safety and transparency. SMIRL uses maximum-entropy IRL and trajectory sampling to estimate rewards from continuous driving demonstrations.
I. INTRODUCTION
- A misspecified reward can make autonomous vehicles too conservative or aggressive and reduce human trust by producing behavior misaligned with human expectations.Source: §I · p. 1
- Human-driving IRL must handle high-dimensional continuous trajectories, vehicle kinematics, uncertain demonstrations, and interpretable, generalizable reward features.Source: §I · p. 1
- Prior approaches leave gaps: continuous IOC relies on optimal demonstrations and costly derivatives, deep IOC [12] has uninterpretable features, and sampling-based IRL omits maneuver sub-optimality uncertainty.Source: §I · pp. 1–2
- SMIRL combines maximum-entropy IRL with an efficient sampler informed by vehicle motion planning to estimate the partition term from feasible long-horizon trajectories.Source: §I · p. 2
A. Maximum-entropy IRL
- Maximum-entropy IRL recovers a reward that maximizes demonstration likelihood, with β controlling assumed demonstration optimality; the learned reward is averaged across demonstrations under shared-dynamics and reward-consistency assumptions.Source: A. Maximum-entropy IRL · p. 2
B. The Sampler
The sampler generates feasible spatiotemporal trajectories through path sampling, smoothing, and speed sampling. These stages incorporate collision safety, vehicle kinematics, physical speed constraints, and alternative interactive decisions.
B. The Sampler
- The sampler integrates feasible-trajectory sampling with non-conservative defensive motion planning to estimate the maximum-entropy partition term.Source: B. The Sampler · p. 3
- Discrete elastic-band path sampling searches for collision-free paths, using obstacle and lane forces while accounting for moving objects’ predicted sweep volumes.Source: Step I - Path Sampling via Discrete ED · p. 3
- Pure-pursuit smoothing converts piecewise-linear paths into smooth trajectories suitable for vehicle kinematics and partition-term estimation.Source: Step II - Path Smoothing · p. 3
- Speed sampling explores polynomial curves around physically constrained, time-optimal profiles, including separate yield and pass decisions in interactive scenarios.Source: Step III - Speed Sampling · p. 3
C. Re-Distribution of Samples
SMIRL redistributes trajectory samples in feature space to reduce bias when estimating trajectory probabilities.
C. Re-Distribution of Samples
- The method bins samples by Euclidean feature-space similarity and resamples within each bin so that bins contain roughly equal numbers of samples.Source: C. Re-Distribution of Samples · p. 4, Figure 2 · p. 4
D. Summary of the SMIRL algorithm
The paper summarizes SMIRL in an algorithm listing that begins with parameter initialization and sample redistribution.
D. Summary of the SMIRL algorithm
- Algorithm 1 summarizes the proposed sampling-based maximum-entropy IRL procedure for driving.Source: D. Summary of the SMIRL algorithm · p. 4
- The listed procedure initializes θ_0 and iteration counter k, then computes the expert feature count.Source: D. Summary of the SMIRL algorithm · p. 4
- The procedure redistributes samples according to their feature-space similarities to form a new sample set.Source: D. Summary of the SMIRL algorithm · p. 4
III. EXPERIMENTS ON DRIVING BEHAVIOR
The experiments use real roundabout traffic data to study independent and interactive driving, with manually selected features intended to represent drivers’ preferences. The feature set includes speed and comfort-related measures, plus predicted-distance measures for interactions.
III. EXPERIMENTS ON DRIVING BEHAVIOR
- The experiments cover independent driving and merging interactions, where the ego vehicle merges into a roundabout while another vehicle is already circulating.The NID case captures preferences during free driving; the ID case examines interaction in merging traffic.Source: §III · p. 4, A. Dataset · p. 4
A. Dataset
- The NID data contain 113 trajectories (80 for training and 33 for testing), while the ID data contain 233 interactive pairs (150 for training and 83 for testing).The trajectories come from the INTERACTION dataset’s DR USA Roundabout SR and DR USA Roundabout FT subsets, respectively.Source: A. Dataset · p. 4
B. Feature Selection
- The linear reward uses non-interactive features for speed, longitudinal and lateral acceleration, and longitudinal jerk, alongside interactive features for future distance and future interaction distance.The features are intended to capture properties people care about while driving; the interactive features represent mutual influence between drivers.Source: B. Feature Selection · p. 4, Non-interactive features · p. 5
- Speed is represented relative to the speed limit, while acceleration and longitudinal jerk are included as features related to vehicle power consumption and driver comfort.The speed feature also represents drivers’ incentives to drive fast and the influence of traffic rules.Source: Non-interactive features · p. 5
- Features with different physical meanings and units are normalized to (0,1) by dividing each by its maximum value in the dataset.Source: Non-interactive features · p. 5
C. Baseline Methods
The comparison includes three maximum-entropy IRL baselines that estimate the partition term Z differently. CIOC uses a Laplace approximation, Opt-IRL uses an optimal trajectory, and GCL estimates Z from policy rollouts.
C. Baseline Methods
- CIOC estimates Z in the continuous domain using a Laplace approximation of the reward around a demonstration trajectory.The approximation uses a second-order Taylor expansion, treating the integral as a Gaussian integral that can be calculated directly.Source: C. Baseline Methods · p. 5
- Opt-IRL approximates Z using an optimal trajectory obtained by minimizing the updated reward at each training iteration.It uses Z≈e^βR(θ,ξopt) as the approximation.Source: C. Baseline Methods · p. 5
- GCL uses policy-network rollouts to estimate Z and jointly trains neural networks for the reward and policy, learning features rather than relying on manual feature selection.Source: C. Baseline Methods · p. 5
D. Evaluation Metrics
The evaluation uses feature deviation, mean Euclidean distance, and ground-truth trajectory likelihood. Feature deviation compares predicted test trajectories with their ground-truth trajectories.
D. Evaluation Metrics
- The evaluation uses three metrics: feature deviation from ground truth, mean Euclidean distance to ground truth, and likelihood of the ground-truth trajectories.The paper states that the metrics are defined in the following subsections.Source: D. Evaluation Metrics · p. 6
1) Feature Deviation:
- Feature deviation measures the difference between predicted trajectories generated from a learned reward and the ground-truth trajectories in the test set.The paper’s definition also accounts for the number and lengths of test trajectories.Source: 1) Feature Deviation · p. 6
2) Mean Euclidean Distance (MED):
The section defines ground-truth trajectory likelihood as a probabilistic evaluation of learned rewards and specifies the feature inputs and test settings. Testing includes both the training environment and a new merging environment.
3) Probabilistic Metrics:
- Among the four algorithms, the one assigning the highest likelihood to ground-truth test trajectories is treated as the winner.The likelihood uses the set of samples generated by the proposed approach; model-based methods use a linear feature reward, while GCL uses a neural-network reward.Source: 3) Probabilistic Metrics · p. 6
E. Experiment Conditions
- For model-based methods, the non-interactive case uses four manually selected features and the interactive case uses six; GCL takes trajectories directly as input.The four are speed, longitudinal acceleration, lateral acceleration, and longitudinal jerk; the interactive case adds future distance and future interaction distance.Source: E. Experiment Conditions · p. 6
IV. RESULTS AND DISCUSSION
- The algorithms are evaluated on test sets from the training environment and on test sets in a new merging environment.Source: §IV · p. 6
A. Performance on Test Sets in Seen Environments
On seen-environment test sets, SMIRL performed better than the compared IRL methods on MED, MED variance, and test-trajectory likelihood. Its feature deviations were smaller on most features in the non-interactive scenario and on all features in the interactive scenario.
1) Feature Deviation:
- SMIRL achieved smaller EFD on most features in non-interactive driving, except a_lat, and on all features in interactive driving; EFD variations were consistently smaller across features and scenarios.The learned a_lat weight was relatively small, particularly in non-interactive driving, which the authors link to its larger feature deviation there.Source: 1) Feature Deviation · p. 6
2) MED:
- Across both scenarios, SMIRL achieved smaller MEDs and variances than CIOC, Opt-IRL, and GCL, and higher test-trajectory likelihood than all three.The results are reported in Tables I and II for the non-interactive and interactive scenarios, respectively.Source: 2) MED · p. 6, 3) Probabilistic Metric · p. 6, Table I · p. 7
3) Probabilistic Metric:
- Ground-truth test trajectories had higher likelihood under SMIRL's reward function than under those of the other three algorithms in both scenarios.The comparison is shown in Tables I and II.Source: 3) Probabilistic Metric · p. 6, Table I · p. 7, Table II · p. 7
- Learned weights indicate that drivers cared more about longitudinal acceleration in both scenarios; during interaction, they weighted future interaction distance more and longitudinal jerk and lateral acceleration less.The learned weights are summarized in Table III.Source: 3) Probabilistic Metric · pp. 6–7, Table III · p. 7
B. Performance on Test Sets in Unseen Environments
SMIRL outperformed the three baseline IRL algorithms on deterministic and probabilistic metrics, generalized better, and converged faster. The section also reports benefits from sample re-distribution and limits the method’s current design to mobile robot systems such as ground vehicles.
B. Performance on Test Sets in Unseen Environments
- On unseen merging scenarios, all model-based methods generalized better than GCL, with MED and likelihood degrading less.The models were trained on roundabout merging and tested on merging scenarios with different road structures.Source: B. Performance on Test Sets in Unseen… · p. 7, Table IV · p. 7, Table V · p. 7
C. Computation Complexity
- SMIRL converged significantly faster than CIOC, Opt-IRL, and GCL, generating all training samples once rather than regenerating them each iteration.Opt-IRL repeatedly solves an optimization problem, while CIOC’s gradient and Hessian calculations contribute computational load.Source: C. Computation Complexity · p. 7, Table VI · p. 7
D. The Effect of Sample Re-Distribution
- Sample re-distribution improved SMIRL’s feature-count deviations, MED, and test-set demonstration probabilities in both interactive and non-interactive scenarios.The most significant reported improvements were in “Win Count,” which evaluates ground-truth demonstration probabilities.Source: D. The Effect of Sample Re-Distribution · p. 8, Table VII · p. 8, Table VIII · p. 8
V. CONCLUSION
- SMIRL achieved better deterministic and probabilistic results, better generalization, and significantly faster convergence than the three baseline IRL algorithms.Experiments used the INTERACTION dataset in interactive and non-interactive driving scenarios.Source: §V · p. 8
- The current algorithm is designed specifically for mobile robot systems such as ground vehicles; the authors also identify Euclidean distance as a potentially improvable re-sampling metric.They propose extending the method to higher-dimensional robotic systems and exploring better metrics in future work.Source: §V · p. 8