PaperlayerView exact revision on arXiv

Source-linked AI summary

HG-DAgger: Interactive Imitation Learning with Human Experts

Michael Kelly, Chelsea Sidrane, Katherine Driggs-Campbell, Mykel J. Kochenderfer

arXiv:1810.02890v2cs.RO

TL;DR

Behavioral cloning and DAgger face data-mismatch, compounding-error, safety, and human-label-quality challenges in interactive imitation learning. HG-DAGGER gives human experts direct control over intervention and learns a safety threshold for a risk metric; across simulated and real-world driving, it improves performance over DAgger and behavioral cloning.

  • Problem

    Behavioral cloning suffers from data mismatch and compounding errors, while interactive DAgger methods can limit human control and compromise safety and action-label quality.

  • Method

    HG-DAGGER trains novice policies with expert-controlled gating and learns a safety threshold from human intervention data for a model-uncertainty-based risk metric.

  • Results

    HG-DAGGER outperforms DAgger and behavioral cloning in sample efficiency, training stability, and similarity to human behavior in simulated and real-world autonomous driving tasks.

  • Takeaways & Limitations

    HG-DAGGER provides a more suitable framework for interactive imitation learning from human experts in real-world systems.

Abstract

from arXiv · show

Imitation learning has proven to be useful for many real-world problems, but approaches such as behavioral cloning suffer from data mismatch and compounding error issues. One attempt to address these limitations is the DAgger algorithm, which uses the state distribution induced by the novice to sample corrective actions from the expert. Such sampling schemes, however, require the expert to provide action labels without being fully in control of the system. This can decrease safety and, when using humans as experts, is likely to degrade the quality of the collected labels due to perceived actuator lag. In this work, we propose HG-DAgger, a variant of DAgger that is more suitable for interactive imitation learning from human experts in real-world systems. In addition to training a novice policy, HG-DAgger also learns a safety threshold for a model-uncertainty-based risk metric that can be used to predict the performance of the fully trained novice in different regions of the state space. We evaluate our method on both a simulated and real-world autonomous driving task, and demonstrate improved performance over both DAgger and behavioral cloning.

I. INTRODUCTION

The introduction frames behavioral cloning and DAgger as responses to imitation-learning challenges, while emphasizing that human-in-the-loop DAgger variants can compromise safety and label quality. HG-DAGGER addresses these concerns by giving humans direct control over intervention and learning a data-driven safety threshold.

  • Behavioral cloning trains novice policies from expert demonstrations but often suffers from data mismatch and compounding errors.
  • DAgger improves robustness by sampling corrective expert labels from states influenced by the novice policy.This robot-centric sampling exposes the novice to perturbations from nominal expert trajectories.
  • Robot-centric sampling can compromise training safety because an incompletely trained novice partially or completely controls data-gathering rollouts.Shared control may also alter human behavior, degrade action-label quality, and destabilize the combined system.
  • Switching control authority between novice and expert can degrade human feedback, create perceived actuator lag, and lead experts to change their behavior.Such effects may reduce learning stability and produce substantially different novice behaviors.
  • Increasing DAgger’s β gives the human more control authority and may improve safety and label quality, but excessive β approaches behavioral cloning and its compounding-error problems.
  • HG-DAGGER is designed for effective human-expert imitation learning, learns a meaningful safety threshold for its risk metric, and improves sample efficiency, training stability, and human-like behavior relative to DAgger and behavioral cloning.

II. METHODS

HG-DAGGER gives the human expert direct control over when to intervene, collects uninterrupted expert labels, and learns a doubt threshold from interventions to assess novice-policy risk.

  • II. METHODS: HG-DAGGER lets the expert take control when the novice enters an unsafe state and return control after recovery to a safe region.The gating function is controlled directly by the expert rather than switching control stochastically at each time-step.
  • II. METHODS: The method aggregates expert labels collected during human-controlled interventions with the existing training data across repeated rollouts.Each epoch’s collected data is added to the training set, which is initialized with behavioral-cloning samples.
  • II. METHODS: HG-DAGGER is motivated by uninterrupted human control, which is expected to improve action-label quality along nominal and recovery trajectories.Unlike stochastic handoffs or retroactive labeling, the expert maintains exclusive control until manually returning it to the novice.
  • II. METHODS: The novice’s doubt is the ℓ2-norm of the main diagonal of an ensemble-output covariance matrix, serving as a model-uncertainty risk metric.The method uses an ensemble of neural networks to approximate policy confidence from the covariance matrix of its outputs.
  • II. METHODS: HG-DAGGER learns the doubt threshold τ from intervention data by averaging the final 25% of entries in the doubt intervention logfile.This emphasizes interventions from later, more-data-trained novice policies while retaining more observations than using only the latest intervention.
  • II. METHODS: The learned risk metric is used to assess the fully trained policy and is proposed for future test-time handoff to a safer controller.The current work evaluates and understands final-policy performance rather than using the threshold for test-time control switching.
  • II. METHODS: HG-DAGGER is unsuitable where human experts cannot quickly identify and react to unsafe situations.Training-time safety depends on timely human intervention.
  • II. METHODS: HG-DAGGER differs from Confidence-Based Autonomy by using neural-network ensembles and learning its doubt threshold from expert interventions.Confidence-Based Autonomy instead uses Gaussian mixture models and an ad-hoc threshold based on average nearest-neighbor distance.

III. EXPERIMENTAL SETUP

The study evaluates HG-DAGGER for human-in-the-loop autonomous driving and compares it with DAGGER and behavioral cloning in simulation and on a physical vehicle.

  • III. EXPERIMENTAL SETUP: The application is autonomous driving learned from human drivers, extending prior behavioral-cloning and DAGGER work on lane-keeping and end-to-end driving.The study specifically targets the human-in-the-loop setting.
  • III. EXPERIMENTAL SETUP: Performance is compared across HG-DAGGER, DAGGER, and behavioral cloning using both simulated and real-world experiments.The experiments collect data and train policies in simulation and on a physical test vehicle.

A. Experimental Task

The task requires an ego vehicle to weave safely through stationary cars on a two-lane one-way road, with performance measured by road departures and collisions.

  • A. Experimental Task: The ego vehicle must weave between stationary cars while remaining safely within the roadway.Obstacle placement varies in lane and spacing during training, with cars initialized at 30-meter intervals and spacing randomized by ±5 meters.
  • A. Experimental Task: The novice observes vehicle pose, speed, lane-edge distances, and distances to the nearest obstacle in each lane, then outputs steering and speed commands.The observation includes y, θ, s, l_l, l_r, d_l, and d_r.
  • A. Experimental Task: Performance is evaluated primarily using road-departure and collision rates calculated per meter.A road departure occurs when the ego vehicle’s center of mass leaves the road under novice control.
  • A. Experimental Task: All methods begin with behavioral-cloning initialization from 10,000 expert labels and then receive five additional epochs of 2,000 labels each.The same initial policy is refined for each tested method.

1) Training:

The real-world setup uses an MG-GS vehicle with LiDAR and high-fidelity localization, while simulated vehicles serve as stationary obstacles on a simulated two-lane road.

  • 1) Training:: The physical platform is an MG-GS vehicle equipped with LiDAR and high-fidelity localization, tested against simulated stationary obstacles.A safety driver monitors the vehicle during both training and testing.

2) Vehicle Experiments:

The vehicle experiments use simulation to evaluate imitation-learning methods and visualize road-departure performance across training. The setup also accounts for the need to initialize policies with expert data and limits expert-label collection in the final HG-DAGGER epoch.

  • Simulation supplements physical-car experiments for evaluations that would be time-consuming or dangerous on the vehicle.
  • Policies are initialized with behavioral cloning using expert data because novice-shaped sampling requires a trained novice to induce realistic state distributions.
  • HG-DAGGER’s final training epoch uses fewer expert labels because the novice can avoid unsafe state-space regions without human intervention.
  • Figure 3 tracks mean road departure rate per meter over training epochs, with error bars showing standard deviation.

IV. RESULTS

HG-DAGGER outperforms DAGGER and behavioral cloning in the driving task, with faster, more stable learning measured by road-departure and collision rates. Learning curves compare methods as the number of expert labels increases.

  • HG-DAGGER outperforms DAGGER and behavioral cloning in sample efficiency, training stability, and similarity to human behavior.
  • Learning curves evaluate each method using the same eight obstacle configurations and initializations while varying the number of expert labels.
  • HG-DAGGER learns faster and more stably than DAGGER and behavioral cloning according to road departure rate and collision rate, both measured per meter.
  • Figure 4 reports mean collision rate per meter over training epochs with standard-deviation error bars.

B. Simulation: Safety and Risk Evaluation

The safety evaluation tests whether a learned novice-doubt threshold identifies safer regions of the state space. Policies initialized inside the estimated permissible set perform substantially better than those initialized outside it.

  • The estimated permissible set ˆP is derived from novice doubt and the learned threshold τ, while its complement represents the estimated unsafe set.
  • Initializations are sampled from intersections with a conservative set S to compare similar, realistically encountered states inside and outside ˆP.
  • 12 times lower mean collision rate and 20 times lower mean road departure rate occur when the novice starts inside ˆP rather than outside it.
  • Average road-departure duration inside ˆP is less than half the duration observed outside the permitted set.
  • The results support novice doubt as a model-free risk approximator and show that HG-DAGGER learns a threshold distinguishing similarly situated states by riskiness.

C. Test Vehicle: Driving Performance

On-vehicle tests found that HG-DAGGER produced the strongest driving outcomes among the evaluated methods, with safer trajectories and more human-like steering. The limited test data makes these findings primarily heuristic.

  • HG-DAGGER had the fewest collisions and road departures among the three evaluated methods.Policies were evaluated on five fixed random obstacle configurations.
  • 21.1% closer steering-angle distributions to human driving data distinguished HG-DAGGER from DAGGER by Bhattacharyya distance.
  • The limited amount of on-vehicle test data restricts the statistical significance of the results.The authors interpret the findings primarily as a heuristic for further real-world evaluation.
  • HG-DAGGER maintained safer road-edge distances than DAGGER and stayed on the roadway unlike behavioral cloning.These trajectory differences were reported qualitatively from the on-vehicle tests.

D. Test Vehicle: Safety and Risk Evaluation

The learned doubt threshold produced a useful risk partition: it approximately separated safe free space from dangerous occupied space. Across 40 obstacle configurations, the learned threshold was near-optimal on all examined classification metrics.

  • The evaluation compared estimated permitted and excluded regions with free and occupied workspace using pixelwise binary classification metrics.Novice doubt was sampled along constant-curvature trajectories and linearly interpolated to assign pixels.
  • The learned threshold τ approximately characterized workspace risk, while nearby thresholds became unsafe or overly conservative.Increasing τ misclassified dangerous regions as safe; decreasing it overestimated risk.
  • 40 randomly generated obstacle configurations supported the threshold’s near-optimal performance across all examined pixelwise classification metrics.

V. CONLCUSION

HG-DAGGER addresses human-in-the-loop safety and control problems in interactive imitation learning by learning both a novice policy and a test-time risk threshold. The method showed improved sample efficiency and training stability relative to DAgger and behavioral cloning, while its free-space safety proxy has limitations.

  • HG-DAGGER was designed to improve human-expert imitation learning when interactive methods limit control authority and compromise safety.
  • HG-DAGGER learns a novice policy and a doubt-based safety threshold for estimating risk across state-space regions at test time.
  • Free space and occupied space only approximate safe and dangerous regions because some free-space points remain within inevitable-collision regions.
  • Simulated and real-world autonomous-driving experiments showed improved sample efficiency and increased training stability relative to DAgger and behavioral cloning.
  • Future work will use doubt-based risk metrics for automated gating and investigate more sophisticated uncertainty-to-execution-risk estimates.
Loading 1810.02890v2…