PaperlayerView exact revision on arXiv

Source-linked AI summary

ThriftyDAgger: Budget-Aware Novelty and Risk Gating for Interactive Imitation Learning

Ryan Hoque, Ashwin Balakrishna, Ellen Novoseller, Albert Wilcox, Daniel S. Brown, Ken Goldberg

arXiv:2109.08273v1cs.ROcs.AI

TL;DR

Interactive imitation learning must balance corrective feedback for robot learning against the human time required to provide interventions. ThriftyDAgger uses novelty and risk to decide when to request help within a desired intervention budget. Experiments report reduced burden alongside strong task and human-study performance across simulated and physical settings.

  • Problem

    Interactive imitation learning needs to balance interventions that support learning with the significant human time those interventions require.

    Source: §1 · pp. 1–2

  • Method

    ThriftyDAgger learns a switching policy that requests human interventions at sufficiently novel or risky states and sets thresholds from a desired context switching rate.

    Source: §1 · p. 2

  • Results

    Experiments suggest ThriftyDAgger reduces supervisor burden while enabling challenging tasks in simulation, physical cable routing, and three-robot fleet supervision.

    Source: §1 · p. 2, §5.4 · p. 7, §5.5 · p. 8

  • Takeaways & Limitations

    ThriftyDAgger’s novelty-and-risk criteria can regulate interventions during both training and execution across simulated and physical tasks.

    Source: §1 · p. 2, §6 · p. 8

    The formulation assumes human and robot share an action space and that success is reaching a specified goal set within a finite horizon.

    Source: §3 · p. 3

Abstract

from arXiv · show

Effective robot learning often requires online human feedback and interventions that can cost significant human time, giving rise to the central challenge in interactive imitation learning: is it possible to control the timing and length of interventions to both facilitate learning and limit burden on the human supervisor? This paper presents ThriftyDAgger, an algorithm for actively querying a human supervisor given a desired budget of human interventions. ThriftyDAgger uses a learned switching policy to solicit interventions only at states that are sufficiently (1) novel, where the robot policy has no reference behavior to imitate, or (2) risky, where the robot has low confidence in task completion. To detect the latter, we introduce a novel metric for estimating risk under the current robot policy. Experiments in simulation and on a physical cable routing experiment suggest that ThriftyDAgger's intervention criteria balances task performance and supervisor burden more effectively than prior algorithms. ThriftyDAgger can also be applied at execution time, where it achieves a 100% success rate on both the simulation and physical tasks. A user study (N=10) in which users control a three-robot fleet while also performing a concentration task suggests that ThriftyDAgger increases human and robot performance by 58% and 80% respectively compared to the next best algorithm while reducing supervisor burden.

1 Introduction

Interactive imitation learning can improve robot policies through corrective interventions, but those interventions consume human time and make it difficult to balance learning with supervisor burden. ThriftyDAgger combines state novelty and risk to request interventions selectively, and the paper reports benefits across simulation, physical manipulation, and fleet supervision.

1 Introduction

  • Interactive imitation learning uses corrective interventions to address brittle policies caused by mismatches between demonstration and robot state distributions, but interventions require significant human time.The burden is especially consequential when a small number of supervisors must oversee many robots.Source: §1 · pp. 1–2
  • Human-gated approaches require supervisors to monitor robots continuously, imposing burden and limiting scalability to settings with few humans overseeing many robots.Robot-gated approaches instead let the robot request interventions when its criterion indicates they are needed.Source: §1 · pp. 1–2
  • ThriftyDAgger combines state novelty and a new risk metric to estimate task success and request interventions only when needed.It also sets intervention thresholds from a supervisor-specified context switching rate, rather than requiring careful parameter tuning.Source: §1 · p. 2
  • Experiments and a three-robot user study suggest ThriftyDAgger reduces supervisor burden while supporting challenging tasks in simulation and physical cable routing.The user study reports improved robot and human task performance, fewer context switches and intervention actions, and lower mental load and frustration than prior algorithms.Source: §1 · p. 2

2 Related Work

Prior work addresses imitation learning with offline demonstrations and online human feedback, including interventions initiated by either the supervisor or the robot. ThriftyDAgger builds on robot-gated methods by combining novelty with a risk measure and automatically tuning key parameters.

2 Related Work

  • Offline imitation learning can suffer from trajectory distribution mismatch, while online feedback may be unreliable when robots visit states unlike those a supervisor would visit.Offline approaches include behavior cloning [31] [32] and learning a reward for reinforcement learning [27].Source: §2 · pp. 2–3
  • Human-gated methods such as HG-DAgger [15, 16, 24] require continuous monitoring and ask supervisors to decide when intervention is needed, which can be burdensome and unreliable.This is particularly impractical when a small number of humans supervise many robots.Source: §2 · p. 3
  • Robot-gated algorithms such as EnsembleDAgger [26], SafeDAgger [19], and LazyDAgger [18] let the robot actively request interventions using estimated performance or uncertainty signals.Prior criteria include state novelty [26] and predicted action discrepancy [19] [18].Source: §2 · p. 3
  • Novelty alone can miss familiar but difficult bottleneck states, while action discrepancy may be acceptable in low-precision states but problematic when precision matters.ThriftyDAgger combines novelty with a risk metric and automatically tunes key parameters.Source: §2 · p. 3

3 Problem Statement

The problem is to train a robot to imitate a human supervisor within a specified context-switching budget. The formulation models imitation loss and supervisor burden, then motivates intervention criteria that limit the number of interventions.

3 Problem Statement

  • The setting is a discrete-time MDP with continuous states and actions, where the robot lacks a shaped reward and transition dynamics but can cede control to a supervisor.Tasks have a goal set and finite horizon, and may be challenging and long-horizon for direct reinforcement learning.Source: §3 · p. 3
  • The formulation assumes human and robot share an action space and task success is reaching a specified goal set within the time horizon.An indicator function identifies whether a state belongs to the goal set.Source: §3 · p. 3
  • The imitation objective minimizes a surrogate loss between robot and supervisor actions over states induced by the robot policy.The loss is an action-discrepancy measure, such as mean squared error, and the state distribution is marginal at each timestep.Source: §3 · p. 3
  • Supervisor burden includes expected context switches, intervention actions, and latency for switching between control modes.The context-switch count tracks switches from autonomous to supervisor control; latency sums switching time in both directions.Source: §3 · pp. 3–4
  • Because directly optimizing the burden constraint is challenging, the paper proposes intervention criteria that limit total interventions to a user-specified budget.When switching latency is sufficiently high, limiting interventions directly limits supervisor burden.Source: §3 · p. 4

4 ThriftyDAgger

ThriftyDAgger switches between robot and supervisor control using estimates of state novelty and task-failure risk, then sets switching thresholds from a desired intervention budget. Its procedure combines novelty and risk estimation, threshold selection, and alternating data collection and policy updates.

4 ThriftyDAgger

  • The method uses novelty and risk estimates to decide when to switch between autonomous and supervisor control.The section describes estimates for novelty and task-failure risk, their integration into switching, and an online procedure for setting thresholds.Source: §4 · p. 4

4.1 Novelty Estimation

  • Novelty is the average across action components of their variance across an ensemble of policies trained on bootstrapped supervisor transitions.In supervisor mode, action discrepancy between supervisor and robot actions provides a more precise correlate of novelty.Source: §4.1 · p. 4

4.2 Risk Estimation

  • The risk estimator uses a learned Q-function for discounted probability of successful convergence to the goal and is trained on supervisor transitions.It need only distinguish risky states from others, rather than make the fine-grained distinctions needed for policy learning.Source: §4.2 · p. 4

4.3 Regulating Switches in Control Modes

  • ThriftyDAgger initiates supervisor control when a state is sufficiently novel or the robot policy has low estimated task-success probability.It returns control to the robot when action discrepancy and risk are both below thresholds, using stricter risk criteria while in supervisor mode to encourage longer interventions and fewer context switches.Source: §4.3 · p. 5

4.4 Computing Risk and Novelty Thresholds from Data

  • A desired intervention proportion sets the high novelty and risk thresholds to their respective 1 − αh quantiles over states previously visited by the robot.The return-to-robot action-discrepancy threshold is the mean on visited supervisor states, while its risk threshold is the median over previously visited robot states.Source: §4.4 · p. 5

4.5 ThriftyDAgger Overview

  • ThriftyDAgger initializes the robot policy with behavior cloning, then alternates robot and supervisor control while updating the policy and risk estimator after episodes.Robot-mode transitions are aggregated separately from supervisor-mode transitions; the policy is updated on supervisor data and the Q-function on robot and supervisor data.Source: §4.5 · p. 5

5 Experiments

The experiments compare ThriftyDAgger with interactive imitation-learning baselines across simulated peg insertion, a three-robot user study, and physical cable routing, measuring both task performance and supervision burden.

5.1 Evaluation Metrics

  • Evaluation tracks successful-episode action and intervention counts, execution success, and— in the user study—fleet throughput, robot idle time, Concentration performance, and user ratings.Successful episodes are used for action and intervention counts to avoid bias from episodes reaching the time horizon.Source: §5.1 · p. 6

5.2 Comparisons

  • The comparisons include Behavior Cloning, human-gated HG-DAgger, robot-gated SafeDAgger and LazyDAgger, and ThriftyDAgger ablations without novelty or risk.The experiments examine simulated peg insertion, a three-robot user study, and physical cable routing; an additional block-stacking simulation is reported in the supplement.Source: §5 · p. 5, §5.2 · p. 6

5.3 Peg Insertion in Simulation

  • The simulated peg-insertion task includes grasping a randomly posed ring and threading it over a fixed cylinder, with grasping and placement as intervention-motivating bottlenecks.Figure 2 also depicts the physical cable-routing domain and examples categorized as novel, risky, or neither.Source: §5.3 · p. 6, Figure 2 · p. 7
  • ThriftyDAgger achieved the highest autonomous and intervention-aided success rates in simulated peg insertion, including higher autonomous success than human-gated HG-DAgger.It requested a similar number of interventions to other robot-gated methods but required more human actions; removing either novelty or risk reduced autonomous success.Source: §5.3 · p. 6, Table 1 · p. 7

5.4 User Study: Controlling A Fleet of Three Robots in Simulation

  • In the three-robot user study, ThriftyDAgger achieved higher throughput than prior algorithms while requiring fewer interventions and human actions, with lower robot idle time and higher Concentration scores.Users reported lower mental load and frustration with ThriftyDAgger and LazyDAgger than with HG-DAgger and SafeDAgger.Source: §5.4 · pp. 7–8, Table 2 · p. 8

5.5 Physical Experiment: Visuomotor Cable Routing

  • On physical cable routing, ThriftyDAgger achieved autonomous success comparable to HG-DAgger with fewer interventions, and 100% intervention-aided success at execution.Behavior Cloning never completed the task.Source: §5.5 · p. 8, Table 3 · p. 8

6 Discussion and Future Work

ThriftyDAgger combines learned estimates of state novelty and task-failure risk to reduce supervisor burden during training and execution.

6 Discussion and Future Work

  • Experiments suggest the approach supports long-horizon manipulation in simulation, on a physical robot, and in a three-robot fleet while limiting supervisor burden.Future work includes interactive reinforcement learning, larger physical-robot fleets, and studying different or time-varying target supervisor burdens.Source: §6 · p. 8

7 Appendix

The appendix contains algorithm details, implementation and hyperparameter information, experiment-domain details, and the user-study protocol and results.

7 Appendix

  • Appendix 7.1 covers algorithm details, followed by implementation and hyperparameters, experiment domains, and the user-study protocol and detailed results.Source: §7 · p. 12

7.1 Algorithm Details

ThriftyDAgger initializes a robot policy from offline supervisor demonstrations, then alternates robot and supervisor control, aggregates their data, and updates the policy and task-success estimates after episodes.

7.1.1 ThriftyDAgger

  • ThriftyDAgger initializes its robot policy by Behavior Cloning, gathers robot data, tunes switching parameters, alternates control, and updates the policy and task-success estimate after each episode.Robot and supervisor transitions are aggregated separately; episodes end at a valid goal or the time horizon.Source: §7.1.1 · p. 12, §7.1.4 · p. 13

7.1.2 Behavior Cloning

  • Behavior Cloning uses offline expert demonstrations only, remains frozen, and receives additional offline data in simulation to approximate the online data provided to interactive methods.Source: §7.1.2 · p. 12

7.1.3 SafeDAgger

  • SafeDAgger switches to supervisor control when a classifier predicts the robot’s action differs from the supervisor’s by more than threshold βh.The classifier is trained and updated using supervisor-intervention data.Source: §7.1.3 · p. 12

7.1.4 LazyDAgger

  • LazyDAgger builds on SafeDAgger with asymmetric switching that requires a lower action-discrepancy threshold to return to robot control, encouraging longer interventions.It also checks true action discrepancy during supervisor control and injects noise into supervisor actions.Source: §7.1.4 · pp. 12–13

7.1.5 HG-DAgger

  • HG-DAgger is human-gated: the supervisor determines intervention timing and duration, while the robot policy learns from aggregated supervisor transitions.Source: §7.1.5 · p. 13

7.2 Hyperparameter and Implementation Details

The appendix specifies ThriftyDAgger’s training configurations and the implementation choices used for its comparison algorithms. It also reports additional peg-insertion metrics and describes how baseline thresholds were set.

7.2.1 ThriftyDAgger

  • Peg-insertion training initializes the robot policy from 2,687 transitions from 30 demonstrations and uses an ensemble of five neural networks.Each network has two 256-unit hidden layers; it is trained for five epochs with 500 gradient steps per epoch and batch size 100.Source: §7.2.1 · p. 13
  • For one Q estimator, training batches contain 50 transitions, with 10% drawn from goal states; Adam uses learning rate 1e-3 and discount factor γ = 0.9999.The associated configuration collects 10 test episodes every 2,000 environment steps and uses two 256-unit hidden layers with a sigmoid output.Source: §7.2.1 · p. 14

7.2.2 Behavior Cloning

  • Behavior Cloning receives more offline demonstrations than ThriftyDAgger in peg insertion and block stacking, while cable routing uses the same architecture and procedure but only the initial offline data.The configurations use 4,004 transitions from 45 demonstrations for peg insertion and 3,532 transitions from 60 demonstrations for block stacking.Source: §7.2.2 · p. 14

7.2.3 SafeDAgger

  • SafeDAgger uses a fixed switching threshold βh = 0.008, recommended in [19], which marks about 20% of states as unsafe in practice.Unlike ThriftyDAgger, SafeDAgger does not automatically set thresholds from an intervention budget.Source: §7.2.3 · p. 14

7.2.4 LazyDAgger

  • LazyDAgger uses βh = 0.015 and βr = 0.25βh, with action noise of 0.02N(0, I); its thresholds were tuned to balance burden and policy performance.Unlike ThriftyDAgger, LazyDAgger does not automatically set thresholds from an intervention budget.Source: §7.2.4 · pp. 14–15
  • ThriftyDAgger solicits fewer training interventions than prior algorithms while achieving success rate and throughput comparable to HG-DAgger; at execution, its longer interventions accompany more task success.Table 4 reports total interventions, human and robot actions, and mixed-control training success; execution-time human-action counts exclude offline actions.Source: Table 4 · p. 15

7.2.5 HG-DAgger

  • HG-DAgger uses Behavior Cloning’s architecture and hyperparameters without the extra offline demonstrations, and the supervisor determines intervention timing and length.Source: §7.2.5 · p. 15

7.3 Environment Details and Additional Metrics

The additional experiments cover simulated peg insertion and block stacking, as well as physical cable routing. Across these settings, the reported comparisons describe task performance alongside intervention and action costs.

7.3.1 Peg Insertion in Simulation

  • For peg insertion, Table 4 reports that ThriftyDAgger requests fewer training interventions than prior algorithms while achieving success rate and throughput comparable to HG-DAgger.At execution, its interventions are longer than prior algorithms’ and it succeeds more often, as discussed in the main text.Source: §7.3.1 · p. 15
  • The peg-insertion simulation uses Robosuite and evaluates a robot grasping a ring from a random initial pose and placing it over a fixed cylinder.The experiment abstracts 10 simulator timesteps into one environment step and terminates episodes after success or 175 actions.Source: §7.3.1 · p. 15

7.3.2 Block Stacking in Simulation

  • In simulated block stacking, the robot must grasp a cube at a randomized pose and place it atop a second cube at another randomized pose.Figure 3 illustrates an example start and goal state, with the red block to be placed on the green one.Source: §7.3.2 · p. 15, Figure 3 · p. 16
  • ThriftyDAgger achieves block-stacking autonomous and training success rates comparable to HG-DAgger, outperforms other baselines and ablations, and requests fewer interventions but generally more human actions.The task’s randomized placement, small placement region, and geometric symmetries make it harder than peg insertion, as reflected in lower autonomous success rates for all algorithms.Source: Table 5 · p. 16, §7.3.2 · p. 16

7.3.3 Physical Cable Routing

  • The physical cable-routing task uses camera images and a 7DOF teleoperation interface, with end-effector rotation disabled and actions projected onto a 2D plane.The workspace is approximately 10 cm × 10 cm, and each action component is limited to 1 cm in magnitude.Source: §7.3.3 · pp. 16–17
  • In physical cable routing, ThriftyDAgger’s training success rate is similar to HG-DAgger’s; at execution it uses the same number of interventions but fewer human and robot actions.Table 6 reports additional intervention, action, and mixed-control training-success metrics.Source: Table 6 · p. 17, §7.3.3 · p. 17

7.4 User Study Details

The user study had participants supervise three simulated robots while alternating between interventions and a Concentration game. ThriftyDAgger was associated with lower reported mental demand and frustration than HG-DAgger and SafeDAgger, and required the least human time.

7.4.1 User Study Interface

  • The study interface used the peg-insertion task with three robots in parallel, and doubled action magnitude to shorten trajectories while adding Gaussian noise with covariance 0.02N(0, I).In robot-gated conditions, participants played Concentration until an intervention request, then switched to teleoperation; in HG-DAgger they monitored all three robots.Source: §7.4.1 · p. 17

7.4.3 Wall Clock Time

  • ThriftyDAgger required the least human time in the user study, while total experiment time was relatively consistent across conditions.HG-DAgger took longer overall because rendering all three robot views required more compute; autonomous robots could continue making progress during human interventions.Source: Table 7 · p. 18, §7.4.3 · p. 18

7.4.4 Detailed Protocol

  • During robot-gated trials, intervention requests entered a FIFO queue and participants served them sequentially before returning to the 24-card Concentration game.For each condition, participants were scored on successful robot task completions and matching pairs found across games.Source: §7.4.1 · p. 18, §7.4.4 · p. 19

7.4.2 NASA TLX Survey Results

  • Participants reported less mental demand and frustration with ThriftyDAgger and LazyDAgger than with HG-DAgger and SafeDAgger.Participants found it cumbersome to monitor all robots in HG-DAgger; frequent SafeDAgger context switches impeded game progress and frustrated participants.Source: §7.4.2 · p. 18

7.4.4 Detailed Protocol

  • The study recruited 10 participants aged 18–37, including people without knowledge or experience in robotics or AI; participants received randomly selected user IDs and were instructed to play a 12-card Concentration game.Source: §7.4.4 · p. 18
Loading 2109.08273v1…