PaperlayerView exact revision on arXiv

Source-linked AI summary

Efficient Deep Reinforcement Learning with Imitative Expert Priors for Autonomous Driving

Zhiyu Huang, Jingda Wu, Chen Lv

arXiv:2103.10690v3cs.RO

TL;DR

Deep RL for autonomous driving is limited by low sample efficiency and difficult reward design. The paper addresses both through an uncertainty-aware imitative expert policy that regularizes RL, reporting better efficiency and success across simulated driving scenarios.

  • Problem

    RL’s low sample efficiency and difficult reward-function design hinder practical autonomous-driving applications.

  • Method

    The framework derives an imitative expert policy from demonstrations using behavioral cloning and uncertainty estimation, then guides RL through KL-divergence regularization.

  • Results

    60% improvement in sample efficiency over SAC is reported, alongside higher success rates in the evaluated driving scenarios.

  • Takeaways & Limitations

    The results support using human demonstrations to improve RL sample efficiency and achieve human-like driving with sparse reward feedback.

Abstract

from arXiv · show

Deep reinforcement learning (DRL) is a promising way to achieve human-like autonomous driving. However, the low sample efficiency and difficulty of designing reward functions for DRL would hinder its applications in practice. In light of this, this paper proposes a novel framework to incorporate human prior knowledge in DRL, in order to improve the sample efficiency and save the effort of designing sophisticated reward functions. Our framework consists of three ingredients, namely expert demonstration, policy derivation, and reinforcement learning. In the expert demonstration step, a human expert demonstrates their execution of the task, and their behaviors are stored as state-action pairs. In the policy derivation step, the imitative expert policy is derived using behavioral cloning and uncertainty estimation relying on the demonstration data. In the reinforcement learning step, the imitative expert policy is utilized to guide the learning of the DRL agent by regularizing the KL divergence between the DRL agent's policy and the imitative expert policy. To validate the proposed method in autonomous driving applications, two simulated urban driving scenarios (unprotected left turn and roundabout) are designed. The strengths of our proposed method are manifested by the training results as our method can not only achieve the best performance but also significantly improve the sample efficiency in comparison with the baseline algorithms (particularly 60\% improvement compared to soft actor-critic). In testing conditions, the agent trained by our method obtains the highest success rate and shows diverse and human-like driving behaviors as demonstrated by the human expert.

I. INTRODUCTION

The paper addresses RL’s low sample efficiency and difficult reward design by incorporating human knowledge through an imitative expert policy. The framework uses demonstrations, uncertainty estimation, and KL-based regularization to guide learning while preserving exploration.

  • Motivation: RL requires massive environment interaction and carefully designed reward functions, creating challenges for increasingly demanding autonomous-driving tasks.Poorly designed rewards can produce unintended behaviors, while tuning suitable rewards is time-consuming.
  • Proposed framework: The proposed framework transfers human prior knowledge to RL agents using a small amount of expert demonstration data.The framework aims to improve sample efficiency and reduce the effort required to design sophisticated reward functions.
  • Proposed framework: Expert demonstrations are distilled into an imitative expert policy through imitation learning and uncertainty estimation.The resulting policy represents the expert’s action distribution and can be queried during RL training.
  • Proposed framework: KL-divergence regularization guides the RL policy toward expert behavior while high expert uncertainty preserves the agent’s exploration ability.The framework integrates expert priors through value-penalty and policy-constraint approaches.
  • Validation: The method is validated in challenging autonomous-driving scenarios, where improved sample efficiency and human-like behavior are reported with sparse rewards.The paper studies expert policies learned from different demonstration sample sizes and uncertainty-estimation methods.

C. Reinforcement learning with demonstrations

The paper places expert-guided learning within actor-critic RL, where policies maximize long-term reward while SAC adds entropy for exploration. Its method instead regularizes the learned policy toward an informative expert behavior policy.

  • Reinforcement learning: RL models sequential control as an MDP with states, actions, transitions, rewards, and a discount factor, seeking a policy that maximizes discounted cumulative reward.The policy is a distribution over actions conditioned on states.
  • Reinforcement learning: Actor-critic methods alternate value-function learning through Bellman errors with policy learning through Q-value maximization.The approach learns V-functions and Q-functions to recover an effective policy.
  • Soft actor-critic: SAC adds policy entropy to the objective to facilitate exploration with a stochastic policy.Its entropy term is equivalent to KL divergence from a uniformly random action prior.
  • Expert-guided RL: The proposed framework regularizes the learned policy toward a non-uniform expert behavior policy or directly constrains their divergence.The divergence metric is chosen as KL divergence in the described formulation.

B. Behavioral cloning

Behavioral cloning approximates an inaccessible expert policy from state-action demonstrations, but deterministic outputs are insufficient for stochastic RL regularization. The paper therefore models expert actions probabilistically and trains the policy by Gaussian likelihood.

  • Demonstration-based policy learning: The imitative expert policy is learned from demonstration trajectories containing sequences of state-action pairs.The policy is obtained by maximizing likelihood over the demonstration dataset.
  • Behavioral cloning: Standard behavioral cloning maps state inputs to deterministic actions and commonly minimizes mean squared error against expert actions.This setup produces a single point estimate for each state.
  • Behavioral cloning: A deterministic point estimate cannot provide the action distribution needed to regularize a stochastic RL policy.The paper identifies this as a limitation of the basic behavioral-cloning setup.
  • Stochastic expert policy: The stochastic expert policy represents human behavioral variation by outputting parameters of a Gaussian action distribution.This accounts for policy uncertainty arising from potentially different feasible human actions in the same state.
  • Stochastic expert policy: The imitative expert policy is trained by maximum likelihood, equivalently minimizing the Gaussian negative log-likelihood using predictive mean and variance.The policy parameters are θ, while the predictive mean and variance are ˆµθ and ˆσ2θ.

D. Model uncertainty

The framework estimates uncertainty in the imitative expert policy because RL may encounter states absent from demonstrations. It uses uncertainty to regulate expert guidance and combines the resulting policy with actor-critic learning.

  • Uncertainty estimation: Model uncertainty reflects unreliable predictive mean and variance in state-space regions lacking training data.It indicates how confident the imitative expert policy is about its action outputs.
  • Policy derivation: The framework derives the imitative expert policy from state-action demonstrations and queries its action distribution for states encountered during RL training.The overall pipeline consists of expert demonstration, policy derivation, and reinforcement learning.
  • Uncertainty-aware guidance: When expert-policy variance is small, the RL action should remain close to the expert policy; higher uncertainty allows greater exploration.This uncertainty-aware behavior avoids unnecessary exploration where expert predictions are confident.
  • Actor-critic integration: The modified actor-critic algorithm learns Q-functions, a value function, and a stochastic policy while incorporating a pretrained imitative expert policy.A target value network is maintained through Polyak averaging.
  • Actor-critic integration: Expert priors are integrated either by adding a divergence penalty to the value function or by constraining divergence between agent and expert policies.The divergence is measured with KL divergence in the described formulation.

C. Expert priors as value penalty

The framework encourages the learned policy to remain close to the imitative expert policy by adding KL-divergence regularization to the reward and value objectives.

  • C. Expert priors as value penalty: A penalty term is added to the reward to encourage the learned policy to stay close to the expert policy.The divergence metric is the Kullback–Leibler divergence, scaled by temperature parameter α.
  • C. Expert priors as value penalty: The KL penalty is equivalently incorporated directly into the value function.
  • C. Expert priors as value penalty: The KL divergence is estimated from the learned and expert policies, both modeled as Gaussian action distributions conditioned on states.
  • C. Expert priors as value penalty: The actor-critic policy loss is modified while Q-function updates remain unchanged.This tailors the actor-critic algorithm to behavior regularization through a value-function penalty.

D. Expert priors as policy constraint

A second integration strategy constrains the learned policy's KL divergence from the imitative expert policy during optimization, using a Lagrangian dual procedure.

  • D. Expert priors as policy constraint: Policy optimization can explicitly constrain the deviation between the learned and expert policies to remain within a small KL-divergence tolerance.The tolerance ϵ represents the permitted closeness to the imitative expert policy.
  • D. Expert priors as policy constraint: The constrained policy-learning problem is transformed into a corresponding Lagrangian dual problem.
  • D. Expert priors as policy constraint: Dual gradient descent alternates updates to the policy parameters and the nonnegative Lagrange multiplier when the constraint is violated.
  • D. Expert priors as policy constraint: The Q-learning value-function and Q-function updates remain unchanged under the constrained formulation.

V. EXPERIMENTAL SETUP

The method is evaluated in two challenging urban-driving scenarios built on the SMARTS simulation platform: an unprotected left turn and a roundabout.

  • V. EXPERIMENTAL SETUP: Two urban-driving tasks are designed in SMARTS: an unprotected left turn and a roundabout.
  • V. EXPERIMENTAL SETUP: The left-turn task requires crossing a two-way four-lane road in heavy traffic without traffic-light regulation and reaching the rightmost lane.
  • V. EXPERIMENTAL SETUP: The expert observes the environment from the ego vehicle's first-person view while demonstrating the driving task.
  • V. EXPERIMENTAL SETUP: Traffic flows mix vehicles with varied driving-behavior parameters to simulate different types of drivers and real-world situations.Parameters include speed distribution, maneuver imperfection, impatience at intersections, and willingness to cooperate.

1) State space:

The experimental setup represents driving scenes with a bird's-eye-view image, controls vehicles through target speed and lane-change actions, and uses sparse or shaped reward variants.

  • 1) State space:: The bird's-eye-view state encodes a 32 × 32 meter square around the ego vehicle.
  • 1) State space:: The RGB scene representation places the ego vehicle at the center and depicts roads and surrounding vehicles with distinct colors.It encodes road structure and interaction situations while supporting an arbitrary number of objects.
  • 1) State space:: Vehicle control uses continuous target speed from 0 to 10 m/s and discrete lane-change actions for left, keep, and right.
  • 3) Reward function:: The sparse reward emits meaningful feedback only at episode end, assigning −1 for collision and 1 for reaching the goal.These reward components are otherwise zero.
  • 3) Reward function:: Sparse rewards make baseline RL algorithms struggle to achieve acceptable performance, motivating reward shaping for fair comparison.
  • 3) Reward function:: The shaped reward adds 0.001rspeed to collision and goal rewards and is used only for other RL algorithms.The restriction prevents the shaping term from disturbing the proposed method.
  • 1) State space:: Human demonstrations consist of keyboard-selected actions for changing speed and lateral movement during the designed urban-driving tasks.

D. Comparison baselines

The paper benchmarks its method against SAC, PPO, GAIL, and behavioral cloning, while describing shared network architectures and experimental training settings. Behavioral cloning is evaluated only during testing because its learning mechanism differs from reinforcement-learning training.

  • D. Comparison baselines: The proposed method is benchmarked against SAC, PPO, GAIL, and behavioral cloning.SAC and PPO represent reinforcement-learning baselines, GAIL uses expert demonstrations with adversarial training, and behavioral cloning reproduces expert actions through supervised learning.
  • D. Comparison baselines: Behavioral cloning is compared with the proposed method only during testing because its learning mechanism differs from reinforcement-learning training.
  • E. Implementation details: The networks share a state encoder with four convolutional layers and global average pooling, followed by network-specific fully connected layers.The architecture and layer hyperparameters are provided in Fig. 3.
  • E. Implementation details: Five single policy networks with different weight initializations and dataset shuffles form the ensemble expert policy model, with each trained for 100 epochs.The ensemble networks use the same structure as the policy network in Fig. 3.
  • E. Implementation details: The simulation takes roughly one hour for 100,000 interaction steps, with the RL agent making decisions every 0.1 seconds.

VI. RESULTS AND DISCUSSIONS

Across two simulated urban-driving scenarios, the proposed imitative-expert framework improves training efficiency and final performance over baseline methods, while producing higher testing success rates and more human-like behaviors.

  • Experimental setup: The evaluation trains autonomous-driving agents in unprotected left-turn and roundabout scenarios, including aggressive and conservative expert styles.Ten trials with different random seeds are used for each method; testing uses 50 novel traffic flows and episodes.
  • Training results: The proposed methods achieve the best final training performance in the unprotected left-turn scenario, with value penalty and policy constraint performing equally well.Training curves show mean success rates with 95% confidence intervals.
  • Training results: 70% sample-efficiency improvement lets the proposed method reach SAC’s performance using about 30% of the interaction steps in the unprotected left-turn scenario.The imitative expert policy provides a reasonable search direction, while SAC’s training is unstable across trials.
  • Training results: In the roundabout scenario, the proposed method achieves higher success rates and better sample efficiency than other methods, with a 60% sample-efficiency improvement over SAC.Value penalty is more favorable than policy constraint when rewards are sparse and the task involves longer, more complicated interactions.
  • Testing results: The proposed value penalty method obtains the highest testing success rate across all three combinations of scenario and demonstration behavior.Testing performance generally follows training performance, and the method outperforms SAC with the same number of training steps.
  • Driving behavior: Human-like behaviors include nudging forward or waiting for gaps at intersections, changing lanes for speed, and decelerating to reduce risk.Compared with the proposed method, SAC makes less human-like maneuvers, while PPO and GAIL perform poorly in testing.

C. Effects of imitative expert policy

The framework compares expert-policy uncertainty estimators and finds ensemble-based estimation of both policy and model uncertainty most beneficial, especially for harder driving tasks.

  • Uncertainty estimation: Figure 6 reports average success-rate training processes for aggressive and conservative left turns and roundabouts under different uncertainty estimators.The three panels correspond to aggressive left turn, conservative left turn, and roundabout scenarios.
  • Training sample size: Figure 7 reports corresponding training processes for different expert-policy training sample sizes across the same three driving settings.The figure organizes results by aggressive left turn, conservative left turn, and roundabout scenarios.
  • Uncertainty estimation: A Gaussian policy with fixed uncertainty uses a constant standard deviation of 0.2, whereas the single-network alternative models only policy uncertainty.These are the two comparison methods used against the ensemble model.
  • Uncertainty estimation: The ensemble expert policy estimates both policy and model uncertainty, outperforming alternatives as task difficulty increases.It performs better in roundabouts and improves testing performance according to the discussion and Table IV reference.
  • Uncertainty estimation: Larger uncertainty in uncertain states can enhance RL exploration by representing combined policy and model uncertainty.The ensemble method also avoids the fixed method’s inability to provide a state-sensitive search range.

2) Training sample size:

The study evaluates how demonstration quantity affects expert-policy and RL-agent training, finding that larger datasets help more as driving tasks become complex.

  • Training sample size: The experiments train ensemble expert policies with 10, 20, or 40 demonstration trajectories and use value penalty to guide RL agents.This design isolates demonstration quantity while holding the expert-policy derivation and RL integration methods fixed.
  • Training sample size: Increasing expert-policy training sample size helps train both the expert policy and the subsequently guided RL agent.The evaluation uses 10, 20, and 40 demonstration trajectories.
  • Training sample size: The sample-size difference becomes more significant as task complexity increases.In the relatively simple aggressive left-turn scenario, even a small number of trajectories can guide RL training to a satisfactory level.
  • Discussions: The broader framework improves sample efficiency by 60% compared with the state-of-the-art RL algorithm and uses only sparse reward feedback.These discussion-level results motivate using human prior knowledge for autonomous-driving RL.
  • Discussions: A major limitation is reliance on human experts’ high-level decisions, which does not scale when real-world data contain only trajectories.The authors also identify additional hyperparameters and time-consuming tuning as drawbacks.
Loading 2103.10690v3…