Imitation Is Not Enough: Robustifying Imitation with Reinforcement Learning for Challenging Driving Scenarios

I. INTRODUCTION

• Imitation can scale with data, [2],, but rare edge cases and complex interactions with scarce or suboptimal demonstrations [4] can prompt unpredictable responses.The concern is that imitation policies are not explicitly instructed about risky or inappropriate responses.Source: §I · p. 1
• Closed-loop RL can learn observation-action-outcome relationships and encode safety through rewards, reducing vulnerability to covariate shifts and spurious correlations,.Rewards can make safety considerations explicit, including outcomes such as collisions.Source: §I · p. 1
• RL alone depends on difficult reward design [7],,, [10] and can produce unnatural behavior or poor progress, whereas IL adds realism and RL adds safety and robustness.The authors argue this complementarity matters especially in rare, challenging scenarios with little data and interactions requiring human-like coordination and driving conventions.Source: §I · p. 1
• BC-SAC combines IL and RL with a simple safety reward, trains on difficult scenarios, and uses over 100k miles of real-world urban human-driving data.A classifier estimates difficulty from collision or near-miss likelihood when scenarios are re-simulated with a pretrained planning policy.Source: §I · pp. 1–2
• 38% and 40% improvements over pure IL and RL baselines are reported, with over 38% fewer safety events than imitation-only policies in the most difficult bucket.The authors report these gains without compromising human-like behavior.Source: §I · p. 2

II. RELATED WORK

• IL was an early, popular driving-policy approach, [2],,, [26],, while controllable IL or RL models accept high-level route commands, [28],.Source: §II · p. 2
• IL can suffer covariate shift and lack explicit safety objectives, whereas reward-trained RL has addressed lane keeping, intersections, and lane changes; this work studies aggregate and safety-critical performance.Source: §II · p. 2
• Prior closed-loop driving RL commonly trains in simulation with varied agents and scenario initialization; this work instead develops and evaluates on real-world data with other agents following logs.Source: §II · p. 2
• Prior combinations use IL to aid sparse-reward RL, [43], [44] or regularize offline Q-learning,; this work applies the established combination to driving at scale, not a new algorithm.Source: §II · p. 2
• Prior work targets long-tail cases with IL ensembles [4], rule-based fallbacks [45],, negative examples, or curricula; here rewards encode safety and RL most improves hardest scenarios.Source: §II · p. 2

A. Markov Decision Processes (MDPs)

• The driving problem is modeled as an MDP {S, A, T, R, γ, ρ0}, with a policy mapping states to actions to maximize expected discounted rewards.S and A are the state and action spaces; T is the transition model, R the reward, γ the discount factor, and ρ0 the initial-state distribution.Source: A. Markov Decision Processes (MDPs) · pp. 2–3

B. Imitation Learning (IL)

• In behavioral cloning, the learner maximizes the log-likelihood of expert actions conditioned on observed states.The objective is E_{s,a∼D}[log π(a|s)].Source: B. Imitation Learning (IL) · p. 3
• IL assumes an expert policy generates trajectories, and estimates expert actions from observed states using inverse dynamics.The learner’s goal is to train a policy that imitates the expert policy.Source: B. Imitation Learning (IL) · p. 3
• Closed-loop IRL and adversarial IL methods, including GAIL and MGAIL, match policy and expert occupancy measures, potentially addressing open-loop covariate shift.The passage cites IRL, GAIL, MGAIL [17], and covariate shift.Source: B. Imitation Learning (IL) · p. 3

C. Reinforcement Learning (RL)

• RL learns policies through iterative online trial and error; this work uses off-policy, value-based algorithms such as Q-learning.The state-action value function is the expected future return from a particular starting state and action.Source: C. Reinforcement Learning (RL) · p. 3
• For continuous control, the work trains policies with actor-critic methods that alternate between minimizing critic Bellman error and maximizing actor value.The actor-critic method uses entropy-regularized Soft Actor-Critic updates.Source: C. Reinforcement Learning (RL) · p. 3
• Soft Actor-Critic uses entropy-regularized updates.The method is identified as SAC.Source: C. Reinforcement Learning (RL) · p. 3
• The target network Q̄ copies the critic, and gradients do not pass through that copy.Source: C. Reinforcement Learning (RL) · p. 3

• Imitation learning supplies abundant learning signal without requiring reward design.Source: §IV · p. 3
• Reinforcement learning addresses imitation learning’s weaknesses in rare, challenging scenarios where data is scarce.Source: §IV · p. 3
• The approach formulates a weighted mixture of imitation-learning and reinforcement-learning objectives.Source: §IV · p. 3

A. Behavior Cloned Soft Actor-Critic (BC-SAC)

• BC-SAC combines an imitation-learning objective with the critic’s expected Q-value in an SAC actor objective that also includes entropy regularization.The actor-critic formulation is presented as a convenient choice for efficiently optimizing Eq. 4, similar to DAPG or TD3+BC.Source: A. Behavior Cloned Soft Actor-Critic… · p. 3
• The critic update remains unchanged from SAC, while λ controls how strongly the policy mimics expert data within the data distribution D.Source: A. Behavior Cloned Soft Actor-Critic… · p. 3
• In out-of-distribution states, the reward objective dominates learning, whereas both imitation and reinforcement-learning objectives provide signals in-distribution.Source: A. Behavior Cloned Soft Actor-Critic… · p. 3, Figure 2 · p. 4

B. Reward Function

• Imitation loss primarily guides the policy, while simple collision and off-road rewards encode safety constraints.The authors use this approach to sidestep the challenge of designing a reward that captures good driving behavior.Source: B. Reward Function · p. 4
• The collision reward uses the distance between the ego vehicle and the nearest vehicle’s bounding box, with a default 1.0 offset to encourage clearance.The distance is the Euclidean distance in meters between the closest points.Source: B. Reward Function · p. 4
• The off-road reward uses distance to the nearest road edge, with negative values on-road and positive values off-road, plus a default 1.0 offset to encourage edge clearance.The distance is measured in meters.Source: B. Reward Function · p. 4
• The total reward is the sum of the collision and off-road rewards.R = Rcollision + Roff-road.Source: B. Reward Function · p. 4

C. Forward and Inverse Vehicle Dynamics Models

• A kinematic bicycle dynamics model updates the vehicle’s next pose (x, y, θ) from steering and acceleration actions.The action is a = (asteer, aaccel).Source: C. Forward and Inverse Vehicle… · p. 4
• An inverse dynamics model infers expert actions for logged trajectories by minimizing corner-position MSE between the inferred and ground-truth next states.The inferred state is T (st, at), and it is compared with the logged next state st+1.Source: C. Forward and Inverse Vehicle… · p. 4

D. Model Architecture

• The policy uses a dual actor-critic architecture similar to TD3 and SAC,, comprising an actor network π(a|s), a double Q-critic network Q(s, a), and a target double Q-critic network ¯Q(s, a).Source: D. Model Architecture · p. 4
• Each network’s Transformer encoder represents vehicle states, road-graph points, traffic-light signals, and route goals; the actor outputs a tanh-squashed diagonal Gaussian parameterized by mean µ and variance σ.Source: D. Model Architecture · p. 4

E. Training on Difficult Examples

• Because driving is safety-critical and long-tailed, the study explores how training-data distributions affect performance, building on prior findings that harder examples can improve performance.Most scenarios are mundane, while a sizable minority raise critical safety concerns.Source: E. Training on Difficult Examples · p. 4

A. Experimental Setup

• The All dataset contains over 100k miles of San Francisco expert driving, split into 6.4 million training and 10k testing segments, with same-vehicle, same-day trajectories kept together.Segments are 10 seconds long and sampled at 15 Hz; difficulty-ranked subsets Top1, Top10, and Top50 contain 40k, 400k, and 2 million training segments, respectively, and 1.2k, 19k, and 66k test segments. The difficulty model from estimates collision or near-miss likelihood using an internal planner.Source: A. Experimental Setup · p. 4
• Simulation uses 2D bicycle vehicle dynamics and replays other agents from logs, while short 10-second segments and imitative losses limit pose divergence and unrealistic agent behavior.Log-playback follows [34], [35],.Source: A. Experimental Setup · pp. 4–5
• BC-SAC has the lowest failure rates on the Top1 and Top10 evaluation sets, both when trained on All and on Top10.Lower failure rates are better; the figure caption provides no numerical values.Source: Figure 3 · p. 5
• BC-SAC performs best across scenarios from 50% to 100% difficulty and shows the least degradation as evaluation scenarios become harder.All methods perform worse as the evaluation dataset becomes more challenging.Source: Figure 4 · p. 5
• The baselines are open-loop BC, closed-loop MGAIL [17], and SAC; evaluation reports failure rate and route progress ratio.Failure rate is the share of segments with any collision or off-road event; route progress ratio compares the policy’s route distance with the expert demonstration.Source: A. Experimental Setup · p. 5

Training and evaluation subsets

• All methods perform best when trained on Top10; BC trained on Top1 performs significantly worse than on All or Top10, while BC-SAC remains robust on Top1.The authors attribute BC’s weakness on Top1 to imitation learning’s reliance on large datasets to infer driving preferences.Source: B. Results · p. 5

BC-SAC comparison to imitation methods

• σ = 0.37 for BC-SAC, versus 1.29 for BC and 0.78 for MGAIL; BC-SAC also performs better overall, especially on challenging evaluation slices.Both imitation baselines degrade substantially on harder slices, while BC-SAC has the lowest variance across scenarios of varying difficulty.Source: B. Results · p. 5

BC-SAC comparison to RL-only training

• BC-SAC outperforms SAC on safety metrics across configurations, and its BC loss keeps actions similar to logs rather than SAC’s more extreme, unnatural actions.SAC’s boundary-valued actions produce more swerves and abrupt acceleration.Source: B. Results · p. 5, Figure 5 · p. 6, Table II · p. 6

Reward shaping and RL / IL weights

• Dense shaped rewards yield fewer safety events than binary rewards, and balanced off-road and collision weights perform best; small offsets also improve overall performance.The proposed shaped reward outperforms the simpler sparse reward with appropriate parameters.Source: B. Results · p. 5, Figure 6 · p. 6, Figure 7 · p. 6
• A balance between imitation and RL terms performs best; a small progress reward substantially increases progress without major safety regressions, while large progress rewards degrade performance.The ablations use a dataset sampled from 10% of Top10 data.Source: B. Results · pp. 5–6, Figure 8 · p. 6

In-depth failure analysis

• Across six failure categories, BC-SAC generally has fewer direct collisions and off-road events than MGAIL, but is hit by other objects more often.MGAIL tends toward clipping collisions and off-road events; BC-SAC’s collisions more often follow divergence from logs and being hit by other vehicles, with some traffic-rule failures. The analysis hypothesizes that MGAIL lacks an explicit collision penalty, while BC-SAC is not explicitly rewarded for traffic-rule compliance.Source: Table III · p. 6, B. Results · p. 6

VI. CONCLUSIONS

• More than 38% fewer safety events occurred in the most difficult scenarios with BC-SAC than with IL-only and RL-only baselines.The method was trained on large datasets of real-world driving.Source: §VI · p. 7
• BC-SAC inherits implicit human-like driving behaviors from imitation, while RL acts as a fail-safe for out-of-distribution safety scenarios.The method combines imitation learning and RL with a simple safety reward.Source: §VI · p. 7
• Training on the top 10% most challenging scenarios yields the most robust performance in the combined imitation and RL setting.Experiments examined the roles of training datasets, reward shaping, and IL/RL objective terms.Source: §VI · p. 7
• The approach focuses on safety rewards, does not model other agents’ unexpected responses to out-of-distribution ego actions, and requires a heuristic IL/RL objective tradeoff.The objective could be extended to include progress, traffic-rule adherence, and passenger comfort.Source: §VI · p. 7
• Future work could enable reactive simulation agents for training and evaluation and enforce safety as an explicit constraint, potentially with distribution-shift mitigation.The conclusion presents these as promising directions.Source: §VI · p. 7

A. IL + RL Distributed Actor-Learner Training Architecture

• The distributed IMPALA architecture adds demonstration rollout workers and a demo replay buffer; the learner samples both replay buffers for off-policy updates.Source: Figure 10 · p. 9

B. Additional Details on Model Architectures and Hyper-parameters Settings

• BC-SAC uses separate Transformer encoders and (256, 256) heads for its actor, double Q-critic, and target double Q-critic; its actor outputs a tanh-squashed diagonal Gaussian.Source: B. Additional Details on Model… · p. 9
• BC-SAC uses actor and critic learning rates of 1e-4, an imitation learning rate of 5e-5, batch size 64, reward discount 0.92, and replay sample-to-insert ratio 8.Source: B. Additional Details on Model… · p. 9
• Rather than combine IL and RL objectives in one gradient step, BC-SAC alternates updates, applying one IL-loss update for every eight RL updates; hyperparameters are grid-searched.Source: B. Additional Details on Model… · p. 9
• SAC uses BC-SAC’s network design and hyperparameters but omits the IL step.Source: B. Additional Details on Model… · p. 9
• BC discretizes steer and acceleration into 217 actions, uses a Softmax actor head and cross-entropy loss, and trains with learning rate 1e-4 and batch size 256.Source: B. Additional Details on Model… · p. 9
• MGAIL follows the network design and hyperparameter settings presented in [55].Source: B. Additional Details on Model… · p. 9