PaperlayerView exact revision on arXiv

Source-linked AI summary

DART: Noise Injection for Robust Imitation Learning

Michael Laskey, Jonathan Lee, Roy Fox, Anca Dragan, Ken Goldberg

arXiv:1703.09327v2cs.LG

TL;DR

Behavior Cloning can suffer compounding errors when robot execution leaves the supervisor’s demonstration distribution, while on-policy correction introduces supervisor, computation, and safety burdens. DART injects optimized noise into supervisor demonstrations to provide recovery examples off-policy, matching DAgger in locomotion and improving grasping performance over Behavior Cloning.

  • Problem

    Off-policy Behavior Cloning suffers compounding errors from covariate shift, while on-policy correction can burden supervisors, increase computation, and expose robots to dangerous states.

  • Method

    DART injects parameterized noise into the supervisor’s control stream and optimizes its level to approximate the robot policy’s errors during demonstration collection.

  • Results

    Across MuJoCo locomotion and Toyota HSR grasping, DART reaches parity with DAgger in locomotion and yields a 62% average performance increase over Behavior Cloning in cluttered grasping.

  • Takeaways & Limitations

    DART provides robust off-policy imitation learning while reducing the correction, computation, and state-exploration burdens associated with on-policy methods.

  • Takeaways & Limitations

    The theoretical noise-injection bound requires surrogate losses bounded between 0 and 1, such as discrete or normalized bounded continuous controls.

Abstract

from arXiv · show

One approach to Imitation Learning is Behavior Cloning, in which a robot observes a supervisor and infers a control policy. A known problem with this "off-policy" approach is that the robot's errors compound when drifting away from the supervisor's demonstrations. On-policy, techniques alleviate this by iteratively collecting corrective actions for the current robot policy. However, these techniques can be tedious for human supervisors, add significant computation burden, and may visit dangerous states during training. We propose an off-policy approach that injects noise into the supervisor's policy while demonstrating. This forces the supervisor to demonstrate how to recover from errors. We propose a new algorithm, DART (Disturbances for Augmenting Robot Trajectories), that collects demonstrations with injected noise, and optimizes the noise level to approximate the error of the robot's trained policy during data collection. We compare DART with DAgger and Behavior Cloning in two domains: in simulation with an algorithmic supervisor on the MuJoCo tasks (Walker, Humanoid, Hopper, Half-Cheetah) and in physical experiments with human supervisors training a Toyota HSR robot to perform grasping in clutter. For high dimensional tasks like Humanoid, DART can be up to $3x$ faster in computation time and only decreases the supervisor's cumulative reward by $5\%$ during training, whereas DAgger executes policies that have $80\%$ less cumulative reward than the supervisor. On the grasping in clutter task, DART obtains on average a $62\%$ performance increase over Behavior Cloning.

1 Introduction

DART addresses compounding errors in off-policy imitation learning by injecting optimized noise into supervisor demonstrations, retaining off-policy collection while providing corrective examples. Across simulated locomotion and physical grasping, it matches DAgger in locomotion and improves grasping performance over Behavior Cloning.

  • Motivation: Behavior Cloning suffers compounding error because robot execution drifts into states unlike those in the demonstrations.This distribution mismatch is identified as covariate shift and can lead to dangerous states.
  • Motivation: DAgger reduces covariate shift by collecting supervisor corrections on states visited by the robot, but human correction, safety, and computation remain concerns.The method iteratively rolls out the current policy and requests labels or corrections.
  • Approach: DART injects optimized noise into the supervisor’s control stream so demonstrations include recovery from errors while remaining off-policy.The noise is intended to approximate the trained robot policy’s error distribution.
  • Evaluation: DART is evaluated with algorithmic supervisors on MuJoCo locomotion tasks and with human supervisors on Toyota HSR grasping in clutter.The physical task requires pushing occluding objects away to reach a goal object from an eye-in-hand camera perspective.
  • Results: 3x faster computation and only 5% supervisor-reward reduction are reported for Humanoid, while DAgger policies have 80% less cumulative reward than the supervisor.These figures are reported for high-dimensional Humanoid training.
  • Results: 62% average performance increase over traditional Behavior Cloning is reported for grasping in clutter.The result comes from experiments with four human supervisors.
  • Contributions: The paper contributes DART, a theoretical noise-injection analysis, and experiments with algorithmic and human supervisors.The experiments characterize noise injection as a way to reduce covariate shift.

2 Related Work

Related work frames imitation learning around off-policy and on-policy methods: off-policy learning risks covariate shift, while on-policy correction addresses it but burdens supervisors, computation, and safety. Noise injection draws on robust-control ideas to expose systems to disturbances during demonstrations.

  • Imitation Learning: Off-policy imitation learning can compound errors because the robot visits a state distribution different from the supervisor’s during execution.This phenomenon is called covariate shift.
  • On-Policy Methods: DAgger trains the robot to identify and fix small errors after they occur, reducing compounding errors through iterative corrective feedback.Its on-policy data collection can require human supervisors to correct behavior in potentially dangerous state-space regions.
  • Limitations: Proposed efficiency improvements for on-policy learning include gradient updates, but local updates have been reported to underperform full policy retraining.A classifier-based alternative still inherits human-supervision and computation limitations.
  • Robust Control: Robust-control work identifies a related shift when optimized control laws drive systems into unstable state-space regions.Persistence excitation is proposed as a condition requiring informative training data for robust model learning.
  • Noise Injection: Isotropic Gaussian noise can provide persistence excitation because its full-rank covariance exposes systems to any disturbance with positive probability.DART applies the related idea to supervisor demonstrations to simulate test-time robot errors.

3 Problem Statement

The paper formulates imitation learning as minimizing expected surrogate control loss under the robot-induced trajectory distribution. It models stochastic Markovian dynamics and parameterized policies, and distinguishes Behavior Cloning as an off-policy baseline.

  • Objective: Imitation learning seeks a policy matching the supervisor’s policy without assuming access to a reward function.The supervisor is assumed to achieve a desired, though not necessarily optimal, task performance.
  • Assumptions: The system is modeled with Markovian stochastic dynamics, a distribution over initial states, known state and action spaces, and access to a robot or simulator and supervisor.These assumptions support sampling state sequences induced by policies.
  • Formalization: A trajectory is a finite sequence of visited states and corresponding controls over a time horizon T, while a policy maps states to controls.Parameterized policies induce probability densities over trajectories.
  • Policies: The applied policy may be stochastic, with πθ(ut|xt) represented as a user-defined distribution around the policy’s deterministic output.An ϵ-greedy policy is given as an example.
  • Objective: Expected surrogate loss is evaluated along trajectories induced by the robot policy, coupling the trajectory distribution and cumulative loss.The paper uses a surrogate loss over controls and focuses on minimizing covariate shift in the performance decomposition.
  • Behavior Cloning: Behavior Cloning samples from the supervisor’s trajectory distribution and performs expected risk minimization on demonstrations.Unlike on-policy approaches, it does not collect data from the robot’s current policy.

4 Off-Policy Imitation Learning with Noise Injection

Noise-injected demonstrations expose the robot to recovery situations while keeping data concentrated around the supervisor’s policy. DART trains on these demonstrations and optimizes the noise parameter to reduce covariate shift under bounded surrogate loss.

  • Noise Injection: Noise injection simulates test-time errors and forces the supervisor to provide corrective actions for successful task completion.These corrections teach the robot recovery when it deviates from the supervisor’s distribution.
  • Noise Injection: DART widens the supervisor’s trajectory distribution while remaining an off-policy method.Figure 1 contrasts ordinary off-policy demonstrations, on-policy robot rollouts, and DART’s noise-widened supervisor distribution.
  • Noise Model: The noise parameter ψ defines the injected distribution over trajectories, such as Gaussian noise with covariance Σ around supervisor actions.The parameterized distribution is denoted p(ξ|πθ∗, ψ).
  • Training: DART applies standard supervised learning to demonstrations sampled from the noise-injected supervisor.This parallels Behavior Cloning while changing the trajectory distribution used for demonstrations.
  • Optimization: The noise level is optimized through a bound on covariate shift rather than selected arbitrarily.The bound applies when surrogate loss values lie between 0 and 1.

[See Appendix for Proof]

DART iteratively tunes demonstration noise to approximate the robot’s expected final error, producing corrective examples near the supervisor’s policy. Its analysis predicts reduced covariate shift relative to Behavior Cloning when robot error is nonzero, while identifying assumptions and boundary conditions.

  • Noise optimization: DART optimizes the noise distribution’s sufficient statistics to reduce the negative log-likelihood of the robot’s control during data collection.The objective seeks a noise parameter that makes the supervisor’s distribution closer to the final robot’s distribution.
  • Noise optimization: DART uses anticipated final robot error to scale simulated supervisor error, with a closed-form expected deviation for Gaussian noise under squared Euclidean loss.For Gaussian noise with covariance matrix Σ, the expected deviation has a closed-form solution involving tr(Σ).
  • DART algorithm: DART first collects noisy demonstrations, trains an intermediate policy, optimizes noise using that policy, scales the noise to an anticipated final error, then retrains on aggregated data.The procedure uses an initial noise parameter, empirical risk minimization, sample-based optimization, and subsequent data collection with the optimized noise.
  • Theoretical analysis: DART’s theoretical result states that injected Gaussian noise reduces an upper bound on covariate shift more than Behavior Cloning when the robot has nonzero error relative to the supervisor.The comparison concerns the distance between the supervisor’s and robot’s state distributions.
  • Theoretical analysis: Noise injection provides no improvement when the robot perfectly represents the supervisor and collects sufficient data.The paper notes that practical applications are unlikely to satisfy both conditions, motivating DART’s use under limited representation or data.

5 Experiments

Experiments compare DART with Behavior Cloning, DAgger, DAgger-B, and isotropic noise in MuJoCo locomotion and Toyota HSR grasping. DART matches DAgger in locomotion while reducing computation and improves grasping success over Behavior Cloning.

  • MuJoCo Locomotion Environments: The MuJoCo evaluation compares Behavior Cloning, DAgger, DAgger-B, isotropic Gaussian noise, and DART.Behavior Cloning uses supervisor-distribution data, DAgger retrains after every demonstration, and DAgger-B retrains intermittently.
  • MuJoCo Locomotion Environments: DART achieves parity with DAgger across Walker, Hopper, Humanoid, and Half-Cheetah.Behavior Cloning and DAgger-B perform below this level in Walker and Humanoid.
  • MuJoCo Locomotion Environments: 3x faster computation and a 5% supervisor-reward decrease characterize DART on Humanoid, while DAgger policies have over 80% less supervisor reward.DAgger’s computation reflects repeatedly updating the current robot policy and forcing the robot into sub-optimal states.
  • MuJoCo Locomotion Environments: Naive isotropic noise performs poorly and produces unsafe execution policies, supporting optimization of the injected-noise level.Reducing DAgger’s policy-update frequency lowers computation but can significantly deteriorate performance.
  • Robotic Grasping in Clutter: The grasping study uses four robotics-experienced human supervisors and compares Behavior Cloning with DART after 10 and 30 additional demonstrations.The experiment is within-subject and counter-balanced, with each supervisor performing all three methods described in the study.
  • Robotic Grasping in Clutter: 79% success for DART with α = 3 exceeds Behavior Cloning’s 49%, while DART with α = 6 reaches 72%.The results suggest that α = 6 may provide too much noise for human supervisors.

6 Conclusion

DART injects noise into supervisor demonstrations to reduce covariate shift while keeping demonstrations near the supervisor’s policy. It matches on-policy methods in simulated locomotion and outperforms Behavior Cloning on a real robot.

  • Conclusion: DART provides corrective examples near the boundary of the supervisor’s policy without visiting highly sub-optimal states.The method is also described as easier for human supervisors tele-operating the robot.
  • Conclusion: DART achieves parity with on-policy methods in simulated MuJoCo domains and is significantly better than traditional Behavior Cloning on a real robot.These conclusions cover both the simulated locomotion and physical grasping evaluations.

8 Appendix

The appendix presents theoretical and experimental analyses of noise injection, including Gaussian and ϵ-greedy optimization, covariance scaling, and comparisons across MuJoCo domains. Results show that DART and DAgger reduce covariate shift, while randomly sampled noise performs well only when its simulated error is carefully chosen.

  • Noise Optimization: DART optimizes a noise parameter using the current robot policy’s likelihood, then scales the parameter according to a prior over final policy error.The appendix also derives Gaussian and ϵ-greedy maximum-likelihood solutions.
  • Noise Optimization: The appendix analyzes Gaussian supervisor policies with covariance matrices and discrete ϵ-greedy policies over finitely many controls.For ϵ-greedy policies, the loss uses an indicator for action disagreement.
  • Experimental Setup: The MuJoCo experiments use 20 iterations for Walker, Hopper, and HalfCheetah, and 200 iterations for Humanoid.Evaluation occurs at selected intermediate iterations, with different initial-demonstration schedules for Humanoid.
  • Reducing Covariate Shift: DART and DAgger reduce covariate shift, reflected by smaller disparities between supervisor-distribution and robot-distribution losses.Behavior Cloning is less effective, showing slower convergence in the corresponding figure.
  • Random Covariance Matrices: In Gaussian noise injection, the expected squared L2 error simulated by the noisy supervisor equals the covariance matrix’s trace.Thus, the trace provides a way to tune the amount of simulated error.
  • Random Covariance Matrices: Random covariance matrices can match DART when the simulated error is carefully chosen, but inappropriate noise levels cause drastically higher losses.The experiments use trace-scaled covariance matrices to control simulated error.
Loading 1703.09327v2…