Source-linked AI summary
Imitation Is Not Enough: Robustifying Imitation with Reinforcement Learning for Challenging Driving Scenarios
Yiren Lu, Justin Fu, George Tucker, Xinlei Pan, Eli Bronstein, Rebecca Roelofs, Benjamin Sapp, Brandyn White, Aleksandra Faust, Shimon Whiteson, Dragomir Anguelov, Sergey Levine
TL;DR
Imitation-only driving policies can underrepresent safety and reliability in rare, challenging scenarios, while reinforcement learning alone depends heavily on difficult reward design. The paper combines imitation learning with reinforcement learning and simple safety rewards using over 100k miles of real-world urban driving data, reducing safety events by more than 38% in the most difficult scenarios. The results support combining human-like imitation behavior with reinforcement-learning safety optimization for challenging driving cases.
Problem
Imitation learning can fail to account sufficiently for safety and reliability in rare or challenging scenarios, while reinforcement learning alone is heavily dependent on reward design.
Method
BC-SAC combines imitation learning with reinforcement learning and a simple safety reward, training on over 100k miles of real-world urban driving data.
Results
More than 38% reduction in safety events is reported for the most difficult scenarios compared with imitation-learning-only and reinforcement-learning-only baselines.
Takeaways & Limitations
Combining imitation and reinforcement learning improves safety and reliability in challenging scenarios while retaining human-like driving behavior from imitation.
Takeaways & Limitations
The proposed future work includes reactive simulated agents and enforcing safety as an explicit constraint, potentially with methods to mitigate distributional shift.
Abstract
from arXiv · showhide
Imitation learning (IL) is a simple and powerful way to use high-quality human driving data, which can be collected at scale, to produce human-like behavior. However, policies based on imitation learning alone often fail to sufficiently account for safety and reliability concerns. In this paper, we show how imitation learning combined with reinforcement learning using simple rewards can substantially improve the safety and reliability of driving policies over those learned from imitation alone. In particular, we train a policy on over 100k miles of urban driving data, and measure its effectiveness in test scenarios grouped by different levels of collision likelihood. Our analysis shows that while imitation can perform well in low-difficulty scenarios that are well-covered by the demonstration data, our proposed approach significantly improves robustness on the most challenging scenarios (over 38% reduction in failures). To our knowledge, this is the first application of a combined imitation and reinforcement learning approach in autonomous driving that utilizes large amounts of real-world human driving data.
I. INTRODUCTION
Imitation learning scales with human driving data but can behave unpredictably in rare, dangerous scenarios where demonstrations are scarce or sub-optimal. The paper combines imitation learning and reinforcement learning with simple safety rewards to improve robustness on difficult driving scenarios.
- Motivation: Rare or dangerous scenarios may cause imitation policies to respond unpredictably because demonstrations provide limited explicit guidance about risky behavior.Complex interactions further challenge imitation when similar human driving examples are scarce or sub-optimal.
- Motivation: Reinforcement learning adds explicit safety rewards and closed-loop training, helping policies account for outcomes and reduce vulnerability to covariate shifts.The reward function can encode safety considerations that remain implicit in demonstrations.
- Approach: IL and RL provide complementary strengths: imitation supports realistic behavior and reduces reward-design burden, while RL improves safety and robustness in rare scenarios.The paper frames the combination as especially useful when challenging scenarios lack abundant data.
- Approach: BC-SAC combines imitation learning and reinforcement learning with a simple reward function, focusing training on scenarios estimated to have high collision or near-miss likelihood.Difficulty is estimated by replaying scenarios with a pretrained planning policy.
- Results: Over 38% reduction in safety events is reported for the most difficult scenarios when combining IL and RL over imitation alone.The approach uses over 100k miles of real-world urban human driving data and a simple reward function.
II. RELATED WORK
Prior autonomous-driving research uses imitation, reinforcement learning, closed-loop simulation, and hybrid methods to address control, exploration, and safety. This paper positions its contribution as incorporating safety awareness directly into model learning through a reward while leveraging real-world driving data.
- Learning-based approaches: Imitation learning derives driving policies by mimicking expert behavior, while controllable variants accept high-level goals or control signals for route-conditioned control.The related work distinguishes imitation learning from reinforcement learning and other learning-based planning approaches.
- Learning-based approaches: Imitation learning suffers from covariate shift in open-loop settings and lacks explicit knowledge of desirable driving outcomes such as collision avoidance.Closed-loop training and reinforcement learning are presented as ways to address these limitations.
- Simulation-based RL: Reinforcement-learning driving methods commonly train in simulation, whose realism depends on the behavior of simulated agents and how scenarios are generated.Examples include expert log playback, rule-based systems, machine-learning agents, and procedural generation.
- Combining IL and RL: Existing combinations of imitation and reinforcement learning use demonstrations to overcome exploration challenges or regularize value learning against out-of-distribution estimates.The paper states that its goal is to apply this general combination to autonomous-driving challenges rather than propose a novel algorithmic combination.
- Safety-critical driving: Unlike fallback-layer and ensemble approaches, this work incorporates safety awareness directly into model learning through a reward and remains compatible with a fallback layer.The comparison concerns methods for addressing challenging and safety-critical autonomous-driving scenarios.
A. Markov Decision Processes (MDPs)
The paper formulates autonomous-driving policy learning as a Markov decision process and defines imitation learning as learning a policy that matches expert trajectories. The formulation specifies states, actions, transitions, rewards, discounting, and initial-state distribution.
- Markov Decision Processes: An autonomous-driving Markov decision process is represented by the tuple {S, A, T, R, γ, ρ0}.These components describe the state and action spaces, transition model, reward function, discount factor, and initial-state distribution.
- Comparison: Table I compares learning-based approaches for robotic control and autonomous driving.The supplied table caption identifies the comparison scope but does not specify individual rows or entries.
- Markov Decision Processes: The policy π maps states to actions and is optimized to maximize the expected discounted sum of rewards.The objective is defined over the transition model, policy, and initial-state distribution.
- Imitation Learning: Imitation learning trains a policy to reproduce actions from expert trajectories collected through interaction with the environment.Behavioral cloning uses a log-likelihood objective, while inverse and adversarial imitation learning provide alternative approaches.
C. Reinforcement Learning (RL)
The paper uses reinforcement learning as an iterative trial-and-error framework for continuous-control policies. Its actor-critic method alternates between value-function and policy updates, using entropy-regularized Soft Actor-Critic updates.
- Reinforcement Learning: Reinforcement learning learns an optimal policy through iterative online trial and error, with value-based methods estimating future returns for state-action pairs.The paper identifies Q-learning as an off-policy, value-based reinforcement-learning approach.
- Actor-Critic Methods: For continuous control, the actor-critic method alternates between training a critic to reduce Bellman error and an actor to maximize value.The paper uses entropy-regularized updates from Soft Actor-Critic.
- Actor-Critic Methods: A target critic network is a copy of the critic through which gradients do not pass.This target network is part of the actor-critic architecture described in the paper.
IV. LEARNING TO DRIVE WITH RL-AUGMENTED BC
The method combines imitation and reinforcement learning so demonstrations guide behavior where data are available, while rewards guide learning in out-of-distribution states. An actor-critic implementation adds the imitation objective to the SAC value-based objective.
- A weighted mixture uses demonstration learning where data are abundant and reward learning where data are scarce.
- Actor-critic training incorporates the imitation objective into the expected Q-function value, building on SAC with entropy regularization.
- With suitable λ, the policy primarily mimics expert data in-distribution and relies on rewards in out-of-distribution states.
- For in-distribution states, both IL and RL provide learning signals, whereas the RL objective dominates for out-of-distribution states.
B. Reward Function
The reward design avoids encoding all aspects of good driving by using imitation to guide natural behavior and simple rewards to enforce safety constraints. Collision and off-road terms are combined additively.
- The reward function encodes safety constraints through collision and off-road distances, while imitation guides good driving behavior.
- The collision term uses the distance between the ego vehicle and the nearest other-vehicle bounding box, with an offset encouraging separation.
- The off-road term uses distance to the nearest road edge, and the total reward is R = Rcollision + Roff-road.
C. Forward and Inverse Vehicle Dynamics Models
The system updates vehicle state with a kinematic bicycle model and infers expert actions from logged trajectories using inverse dynamics. Its actor-critic networks encode vehicle, road, traffic-light, and route information with Transformer observation encoders.
- The kinematic bicycle model computes the next vehicle pose from steering and acceleration actions.
- Inverse dynamics estimates expert actions by minimizing corner-position MSE between inferred and logged next states.
- The dual actor-critic architecture contains an actor, double Q-critic, and target double Q-critic network.
- Transformer encoders represent vehicle states, road-graph points, traffic-light signals, and route goals; the actor outputs a tanh-squashed diagonal Gaussian policy.
- Because performance depends strongly on long-tail training distributions, the study examines how training-data difficulty affects method performance.
A. Experimental Setup
Experiments use large-scale urban human-driving data and evaluate failure rates across challenging scenario sets. BC-SAC is reported to achieve the lowest failure rates and degrade least as difficulty increases.
- The dataset contains over 100k miles of expert trajectories, partitioned into 6.4 million training and 10k testing segments.
- On the most challenging Top1 and Top10 evaluation sets, BC-SAC consistently achieves the lowest error rates.
- Across 50%-100% difficulty scenarios, BC-SAC performs best and shows the least degradation as evaluation data become more challenging.
- The baselines include open-loop BC and closed-loop MGAIL, which uses closed-loop training and differentiable simulation.
- Failure rate is the percentage of run segments containing at least one collision or off-road event.
B. Results
BC-SAC performs best when trained on challenging scenarios and balances imitation with reinforcement learning, improving safety while preserving progress. Its remaining failures differ qualitatively from those of imitation-only methods.
- Training-data distribution: BC-SAC performs best when trained on the Top10 subset of challenging scenarios, while BC trained on Top1 performs significantly worse than when trained on All or Top10.BC-SAC remains robust when trained on Top1, reflecting different sensitivity to training-data volume than imitation-only BC.
- Reward design: Dense rewards lead to fewer safety events than binary rewards, while balanced off-road and collision weights provide the best performance.The binary reward is −1 for a safety event and 0 otherwise; the off-road and collision weights sum to 2.0.
- Objective balance: A balanced choice of imitation and RL terms leads to the best performance.
- Reward design: A small amount of progress reward produces significantly more progress without major safety regressions, whereas large progress rewards degrade performance.
- Failure analysis: BC-SAC generally has fewer direct collisions and off-road events than MGAIL, but more failures from being hit by other objects.The failure analysis categorizes events into clipping, off-road, bad-lane, collision, red-light, and log-divergence failures.
- Failure analysis: MGAIL tends to produce more clipping collisions and off-road events, while BC-SAC avoids several illustrated collisions through greater clearance and more appropriate speed profiles.The examples include a pedestrian exiting a double-parked car, an incoming vehicle, and an intersection rear collision.
VI. CONCLUSIONS
The paper presents BC-SAC, which combines imitation learning with reinforcement learning and a simple safety reward for challenging driving scenarios. It reports more than a 38% reduction in safety events in the most difficult scenarios, while identifying reactive agents and distributional shift as future concerns.
- BC-SAC combines imitation learning with RL, uses a simple safety reward, and trains on large datasets of real-world driving.
- BC-SAC achieves more than a 38% reduction in safety events in the most difficult scenarios compared with IL-only and RL-only baselines.
- Imitation supplies implicit human-like driving behavior, while RL acts as a fail-safe for out-of-distribution safety scenarios.
- The approach does not account for unexpected behavior of other agents in response to out-of-distribution situations.
- Future work includes reactive simulation agents, explicit safety constraints, and methods to mitigate distributional shift.
APPENDIX
The appendix describes a distributed actor-learner implementation, Transformer-based actor-critic components, and alternating reinforcement-learning and imitation-learning updates. It also summarizes baseline implementations for SAC, BC, and MGAIL.
- Training architecture: The distributed architecture adds demonstration rollout workers and a demonstration replay buffer, and the learner samples from both replay buffers off-policy.
- Model architecture: The actor, double Q-critic, and target double Q-critic each use a separate Transformer observation encoder followed by a fully connected head.
- Optimization: BC-SAC alternates reinforcement-learning and imitation-learning updates, using one imitation-learning update for every eight RL updates.
- Baselines: SAC uses the same network design and hyperparameters as BC-SAC but omits the imitation-learning step.
- Baselines: BC discretizes steering and acceleration into 217 actions and trains a Softmax prediction head with cross-entropy loss.
- Baselines: MGAIL follows the network design and hyperparameter settings reported in prior work.