PaperlayerView exact revision on arXiv

Source-linked AI summary

EnsembleDAgger: A Bayesian Approach to Safe Imitation Learning

Kunal Menda, Katherine Driggs-Campbell, Mykel J. Kochenderfer

arXiv:1807.08364v3cs.LGcs.AI

TL;DR

Imitation learning can suffer when novice policies encounter states absent from expert demonstrations, while DAgger does not explicitly account for safety. EnsembleDAgger uses ensemble-based uncertainty and action discrepancy to decide when the novice may act, aiming to increase novice control while constraining failure probability. The method reports improved safety and learning performance across inverted pendulum and MuJoCo HalfCheetah evaluations.

  • Problem

    Imitation-learning policies can encounter inadequately represented states, and DAgger aggregates expert and novice data without guaranteeing safety.

  • Method

    EnsembleDAgger approximates a Gaussian Process with neural-network ensembles and uses novice action variance together with expert–novice discrepancy to decide when the novice acts.

  • Results

    The method improves performance and safety in an inverted-pendulum case study and demonstrates scalability on MuJoCo HalfCheetah.

  • Takeaways & Limitations

    The doubt rule constrains novice actions to familiar states and supports faster attainment of expert-level control while avoiding severely compromising states under the stated risk assumptions.

  • Takeaways & Limitations

    The approach conflates safety with familiarity, may not fit bottleneck states near unsafe regions, and lacks a method for choosing χ and τ.

Abstract

from arXiv · show

While imitation learning is often used in robotics, the approach frequently suffers from data mismatch and compounding errors. DAgger is an iterative algorithm that addresses these issues by aggregating training data from both the expert and novice policies, but does not consider the impact of safety. We present a probabilistic extension to DAgger, which attempts to quantify the confidence of the novice policy as a proxy for safety. Our method, EnsembleDAgger, approximates a Gaussian Process using an ensemble of neural networks. Using the variance as a measure of confidence, we compute a decision rule that captures how much we doubt the novice, thus determining when it is safe to allow the novice to act. With this approach, we aim to maximize the novice's share of actions, while constraining the probability of failure. We demonstrate improved safety and learning performance compared to other DAgger variants and classic imitation learning on an inverted pendulum and in the MuJoCo HalfCheetah environment.

I. INTRODUCTION

Imitation learning can fail when novice policies encounter states absent from expert data, making safe exploration difficult. EnsembleDAgger extends DAgger with probabilistic confidence estimates to increase novice autonomy while constraining failure risk.

  • Motivation: Novice policies can fail in states inadequately represented in expert demonstrations, although such failures matter especially in real-world operation.Imitation learning uses expert supervision to guide exploration, but dataset coverage remains limited.
  • Motivation: DAgger aggregates data from expert and novice policies but does not guarantee safety, while SafeDAgger adds a decision rule for safer exploration.These methods aim to train the novice efficiently while minimizing expert intervention.
  • Objective: The paper aims to maximize the novice’s share of actions while constraining the probability of failure.Expert control is preferred when novice actions may lead to unsafe or failure states.
  • Approach: EnsembleDAgger estimates novice uncertainty with an ensemble of neural networks that approximates a Gaussian Process in complex, high-dimensional spaces.The resulting variance serves as a confidence or doubt measure for deciding when the novice should act.
  • Contributions: The method introduces probabilistic safety and improves performance and safety on an inverted pendulum while scaling to MuJoCo HalfCheetah.These experiments compare EnsembleDAgger with existing DAgger variants.

II. BACKGROUND

This section situates the paper within approaches that approximate Gaussian Processes using neural networks. Such approximations address the computational difficulty of traditional GPs in high-dimensional settings and large datasets.

  • Bayesian approximation methods: Neural-network approximations of Gaussian Processes target uncertainty modeling that is computationally expensive with traditional GPs on high-dimensional data.Ensemble methods and Monte-Carlo dropout are identified as two approximation strategies.

A. DAgger and SafeDAgger

DAgger alternates expert and novice control through a decision rule, enabling exploration but risking unsafe states. VanillaDAgger uses randomized expert intervention, whereas SafeDAgger uses action discrepancy and EnsembleDAgger motivates uncertainty-aware decisions.

  • DAgger framework: DAgger initializes a novice from expert data, collects trajectories under a decision rule, labels observations with expert actions, and aggregates the resulting datasets.The decision rule selects expert or novice actions at each time step.
  • DAgger framework: Allowing novice actions explores states beyond expert trajectories and improves robustness, but unrestricted novice control can encounter costly unsafe states.Decision rules balance exploration against safety and expert intervention.
  • VanillaDAgger: VanillaDAgger selects the expert with probability β_i, which decreases across epochs so the novice receives increasing autonomy.Its decision rule does not compare novice and expert actions, so unsafe novice actions may still be selected.
  • SafeDAgger: SafeDAgger* allows the novice to act when the novice–expert action distance is at most τ; its fixed threshold can be too conservative far from danger or too relaxed near it.SafeDAgger approximations can also reduce expert queries by predicting likely policy deviation.
  • Motivation: An ideal rule should tolerate larger expert-action perturbations far from unsafe states and smaller perturbations near them.This motivates incorporating state-dependent uncertainty rather than relying only on action discrepancy.

B. Bayesian Approximation Methods

Neural networks provide scalable alternatives for approximating Gaussian Processes and estimating predictive uncertainty. The paper focuses on ensembles, including approaches that predict both means and variances and combine outputs as Gaussian mixtures.

  • Motivation: Traditional Gaussian Processes become computationally expensive for high-dimensional feature spaces and large datasets, motivating neural-network approximations.Ensembles and Monte-Carlo dropout are presented as approximation methods.
  • Ensembles: An ensemble trains multiple neural networks for the same task and combines their outputs into a single prediction.This approach is used because it offers a straightforward way to estimate predictive uncertainty.
  • Uncertainty estimation: Proper scoring rules such as negative log-likelihood can train networks to predict Gaussian means and variances, with predictions combined as a mixture of Gaussians.The approach is intended to improve the calibration of predictive uncertainty.

III. ENSEMBLEDAGGER

EnsembleDAgger allows the novice to act only when its action is both close to the expert’s and sufficiently certain. The decision rule uses discrepancy and doubt thresholds as a low-order, model-free approximation to an ideal safety rule.

  • Decision rule: The novice acts only when expert–novice action discrepancy is below τ and novice-action variance is below χ.These are the discrepancy and doubt rules, respectively.
  • Safety rationale: The method assumes epistemic doubt provides a model-free proxy for proximity to failure states because unfamiliar states are likely closer to failure regions.The safety rationale relies on expert trajectories avoiding failure states and novice uncertainty indicating unfamiliarity.
  • Decision rule: The decision rule is parameterized by doubt (χ) and discrepancy (τ) bounds and approximates the ideal rule without a system model.Figure 3 describes this as a low-order, model-free approximation.
  • Decision rule: As χ approaches infinity, the rule converges to SafeDAgger*, while as τ approaches infinity it ignores discrepancy and uses confidence alone.The two limits isolate the discrepancy-based and doubt-based decision behaviors.
  • Decision rule: Expert queries are needed only when the doubt rule is satisfied, enabling query efficiency.The supplied passage frames this as a consequence of the decision rule.

IV. EXPERIMENTS

The experiments test whether discrepancy alone allocates permitted actions poorly, whether variance reflects familiarity, and whether doubt-based and combined rules improve safety and novice performance. The inverted pendulum supports visualization, while HalfCheetah tests the latter claims at scale.

  • Experimental claims: The experiments evaluate whether fixed discrepancy thresholds are overly conservative in some state-space regions and insufficiently conservative in others.This is one of the explicitly stated experimental claims.
  • Experimental claims: They test whether novice-policy output variance measures dissimilarity between a query state and states in the training dataset.The claim concerns variance as a familiarity signal.
  • Experimental claims: The study tests whether a fixed doubt rule trains a better novice policy at the same safety compromise as the combined system.This claim is evaluated alongside the discrepancy-rule comparison.
  • Experimental claims: The combined decision rules are evaluated for improving novice performance while making the combined system strictly safer.The inverted pendulum provides visualization, and MuJoCo HalfCheetah supports the latter claims.

A. Inverted Pendulum domain

The inverted-pendulum experiments compare how doubt and discrepancy rules allocate novice action across the state space under matched permitted-set budgets. Doubt concentrates action near familiar training states and lets the expert take over in unfamiliar regions without trajectories entering dangerous territory.

  • Environment: The inverted pendulum is a deterministic nonlinear control problem with state [θ, ˙θ] and action u.The controller’s basin of attraction and sampled initial-condition region are also characterized.
  • Setup: The novice policy uses an ensemble of ten multilayer perceptrons, each with hidden-layer sizes [64, 64, 32, 32].Each network is trained with MSE because the deterministic expert provides noise-free labels.
  • Comparison protocol: Permitted-set volume is the fraction of grid-sampled states where a decision rule selects the novice action, and its budget grows linearly across epochs.Thresholds χ and τ are selected before each episode to match the desired volume.
  • Permitted sets: For a fixed permitted-set volume, the doubt rule allocates action near states represented in the training data, whereas discrepancy allocates it more haphazardly.The discrepancy rule can permit action where novice and expert outputs happen to agree in potentially dangerous regions.
  • Trajectory behavior: Under the doubt rule, the expert takes control at the edge of familiar regions, producing a larger novice basin of attraction without trajectories entering dangerous territory.Under discrepancy, the novice rarely moves away from expert trajectories, yielding data little more informative than behavior cloning.

B. MuJoCo HalfCheetah domain

The HalfCheetah experiments compare doubt, discrepancy, and combined decision rules across novice learning and combined-system performance. The doubt rule Pareto dominates discrepancy, while some combined settings are more conservative and slightly improve novice performance.

  • Experimental setup: The combined decision rule is evaluated across parameter settings (τ, χ) = [(0.2, 0.02), (0.5, 0.05), (1.0, 0.1), (2.0, 0.2), (5.0, 0.5)].The sampled parameter pairs lie on a line chosen so the two component rules are approximately equally responsible for preventing novice action initially.
  • Results: The doubt rule Pareto dominates the discrepancy rule, achieving better novice performance for the same compromise in combined-system performance.The comparison uses Pareto frontiers because the hyperparameter trades off lone-novice and combined-system performance.
  • Results: The combined rule is strictly more conservative than either component rule and always improves combined-system performance.Along the sampled parameter line, strict improvement over the independent rules is not observed in every case.

V. CONCLUSION

The paper extends DAgger with probabilistic safety by combining bounded action deviation with low novice-action variance. Experiments find the doubt rule superior to discrepancy alone, while some conjunction settings improve on both individual rules; the approach depends on safety–familiarity assumptions and hyperparameter selection.

  • Contribution: EnsembleDAgger replaces VanillaDAgger’s weighted coin flip with a novice action bounded in deviation from the expert and required to have low variance.The method uses low variance as a model-free safety signal tied to familiarity with states in the dataset D.
  • Limitations: The method assumes that safety risk correlates with familiarity and expects this assumption to hold when the expert maintains a safety margin.Future work targets environments with bottleneck states and methods that relax the conflation of safety and familiarity.
  • Findings: The doubt rule lets the novice act mainly in states near labeled states, whereas the discrepancy rule allows action haphazardly where expert and novice actions happen to agree.Under the paper’s risk assumptions, this distinction supports faster novice attainment of expert-level control and avoidance of severely compromising states.
  • Findings: Some hyperparameter settings make the conjunction of doubt and discrepancy rules better than either rule individually.The conclusion reports this as a setting-dependent result rather than a universal improvement.
  • Limitations: The paper does not provide a method for choosing χ and τ, motivating future heuristic strategies for safely selecting them.The authors note that a recent method has already demonstrated hyperparameter selection on real vehicles.

1) Empirical Evaluation:

The empirical evaluation compares Gaussian-process approximations on sparse samples of a nonlinear function. Vanilla ensembles and ensembles with predictive uncertainty most resemble the GP, and the authors therefore use a vanilla ensemble with MSE loss for the novice policy.

  • Evaluation design: Four methods are evaluated on f(x) = sin(πx)+0.2 sin(4πx) using only eight samples.The comparison includes a traditional GP, vanilla ensemble, predictive-uncertainty ensemble, and MC-dropout approach.
  • Evaluation design: The GP baseline uses a squared-exponential kernel with length-scale 10, while neural-network alternatives differ in loss or dropout treatment.The vanilla ensemble uses MSE loss for 300 epochs, whereas the predictive-uncertainty ensemble uses NLL loss for 2400 epochs.
  • Evaluation design: Each model is queried for a mean and standard deviation over x ∈ [−1.5, 1.5], with model standard deviations scaled to match the GP’s summed standard deviation.The resulting predictive distributions are shown in Figure 9.
  • Results: The vanilla ensemble and predictive-uncertainty ensemble have the most visual similarity to a GP.The authors select the vanilla ensemble because it achieves this performance in a small fraction of the training epochs used by the other models.
  • Resulting design: The novice policy therefore uses an ensemble of neural networks trained with MSE loss.The implementations did not use adversarial training, and noiseless data does not highlight potential benefits of predictive-uncertainty ensembles or MC-dropout.

B. The Inverted Pendulum domain

The inverted-pendulum experiment uses a two-dimensional state and one-dimensional saturated control system stabilized by feedback linearization. Its basin of attraction and permitted-set volume provide the setting for comparing decision rules.

  • System: The inverted pendulum has state [θ, ˙θ] and one-dimensional action u.The system dynamics and control law define the learning environment used for the decision-rule comparison.
  • System: Feedback linearization can drive the system to equilibrium at [0, 0] using a control law based on residual linear terms v1 and v2.The resulting controller is deterministic but sufficiently nonlinear to create a learning problem.
  • Evaluation measures: Figure 10 compares learning performance, failure rate, and permitted-set volume for the doubt and discrepancy rules.The χ and τ parameters are chosen a priori.
  • Experimental setup: The experiment sets a = 10, b = 2, c = 10 and saturates u to [−1, 1].The gains are K = [0.316, 0.175], obtained with a linear quadratic regulator.
  • Experimental setup: Control saturation prevents convergence from arbitrary initial conditions, producing the basin of attraction shown in Figure 5.The same figure identifies the state-space region from which initial conditions are sampled during successive DAgger epochs.

C. Inverted Pendulum: Selecting Hyperparameters

The experiment evaluates learning performance, failure rate, and permitted-set volume across six epochs and repeated trials. The doubt rule learns faster with no observed failures, while discrepancy-rule conservatism trades lower failures for poorer learning.

  • Metrics: Learning performance measures the fraction of grid-sampled states in the expert and novice basins of attraction.
  • Metrics: Failure rate measures repetitions whose trajectories leave the expert basin of attraction, averaged over 30 repetitions across six epochs.
  • Results: The doubt rule with χ = 10^-3 achieved superior learning performance and no failures across all six epochs and repeated experiments.
  • Results: Permitted sets grew monotonically; discrepancy rules allowed much larger volumes, whereas the doubt rule generated more informative trajectories despite being less permissive.
  • Results: Discrepancy-rule instances experienced failures, while a more conservative τ reduced failure rate at the expense of learning performance.
Loading 1807.08364v3…