LazyDAgger: Reducing Context Switching in Interactive Imitation Learning

I. INTRODUCTION

• Interactive imitation learning lets a supervisor periodically control the robot, but reducing the resulting human burden remains a key challenge.Interventions are described as an intuitive form of feedback.Source: §I · p. 1
• Context switches cost time because supervisors must interrupt other work, take control, and regain situational awareness before intervening.For 10 interventions of two actions each, the example totals 20L + 20 time units; two interventions of 10 actions each total 4L + 20.Source: §I · p. 1
• LazyDAgger adds asymmetric switching criteria and noise to supervisor actions, reducing context switches while maintaining task performance on three simulated continuous-control tasks.The introduction reports 60% fewer context switches than SafeDAgger and task performance on par with DAgger.Source: §I · p. 1

II. BACKGROUND AND RELATED WORK

• Offline demonstrations require no further supervisor input during learning, but cannot directly address specific learning bottlenecks or policy errors from covariate shift.The cited passage contrasts this with the limited online feedback available from a batch of demonstrations.Source: §II · p. 2
• DAgger queries for supervisor actions in every learner-visited state, which is costly in human time and can be unintuitive feedback.Other forms of online advice can also be hard to provide when learners visit states far from those a human would visit.Source: §II · p. 2
• Prior intervention methods may require continuous human monitoring or guessing when to hand control back; LazyDAgger instead favors sustained interventions and demonstrations in novel states.SafeDAgger and EnsembleDAgger learn to request interventions, but the passage notes sample inefficiency and frequent context switching as concerns.Source: §II · p. 2

III. PROBLEM STATEMENT

• The setting assumes shared human-robot action spaces and that task execution can pause during control transfer.The environment is modeled as a discrete-time MDP, and the supervisor executes a deterministic policy.Source: §III · p. 2
• The robot policy minimizes a surrogate loss measuring its action discrepancy from the supervisor over states induced by the robot policy.The discrepancy measure may be squared loss or 0-1 loss.Source: §III · p. 2
• Interactive imitation learning uses a meta-controller to choose between robot and supervisor policies, with context switches and supervisor actions defining burden components.The formulation counts switches in either direction and expected supervisor actions under the meta-controller.Source: §III · pp. 2–3
• The objective seeks a policy matching supervisor performance while constraining expected supervisor time burden to a supervisor-set threshold.Burden combines context-switch latency with time spent performing interventions.Source: §III · p. 3

IV. PRELIMINARIES: SAFEDAGGER

• SafeDAgger trains a binary discrepancy classifier to distinguish states below versus above the safety threshold βH.The classifier is trained with binary cross-entropy on supervisor-trajectory data.Source: §IV · p. 3
• Classifier labels mark states unsafe when robot-supervisor action discrepancy is at least βH; the loss penalizes misclassifying safe and unsafe states.A sigmoid output below 0.5 predicts a safe state.Source: §IV · p. 3
• SafeDAgger’s meta-policy selects robot control or supervisor control based on the classifier’s prediction of safety.The algorithm passage describes collecting supervisor data, fitting the robot and classifier, then updating them during training.Source: §IV · p. 3
• LazyDAgger uses the same classifier trigger as SafeDAgger but enforces additional intervention properties to lengthen interventions and increase state diversity.The supplied algorithm text specifies noise in supervisor actions and a return-to-autonomy threshold.Source: §IV · p. 3

V. LAZYDAGGER

• LazyDAgger trains a robot policy and safety classifier offline, then evaluates and updates the policy while ceding control when predicted action discrepancy is high.The interactive learning phase runs for N epochs.Source: §V · p. 3

A. Action Discrepancy Prediction

• SafeDAgger’s classifier can cause frequent context switches and prematurely end interventions when it predicts that the robot can match the supervisor.In supervisor mode, LazyDAgger instead computes the action discrepancy from the observed supervisor action and the robot’s proposed action.Source: A. Action Discrepancy Prediction · pp. 3–4
• LazyDAgger switches back to autonomous mode only when ground-truth action discrepancy falls below βR, with βR < βH creating a hysteresis band.The βH threshold governs switching from autonomous mode to supervisor mode; LazyDAgger adjusts βH to reduce switches and βR to lengthen interventions.Source: Figure 2 · p. 3, A. Action Discrepancy Prediction · p. 4

B. Noise Injection

• LazyDAgger injects isotropic Gaussian noise into supervisor actions to diversify visited states where predicted action discrepancy is large.The noise variance σ2 is a scalar hyperparameter.Source: B. Noise Injection · p. 4

A. Simulation Experiments: MuJoCo Benchmarks

• The MuJoCo evaluation covered HalfCheetah-v2, Walker2D-v2, and Ant-v2, measuring task progress and supervisor burden across learning curves.The reported burden measures include supervisor actions, context switches, and total supervisor burden; comparisons were averaged over three random seeds.Source: A. Simulation Experiments: MuJoCo… · pp. 4–5

VI. EXPERIMENTS

• In Gym-Cloth fabric smoothing, LazyDAgger achieved higher final coverage than Behavior Cloning and SafeDAgger with fewer switches but more supervisor actions than SafeDAgger.During execution, it achieved similar coverage to SafeDAgger with fewer context switches.Source: Figure 4 · p. 5

A. Simulation Experiments: MuJoCo Benchmarks

• LazyDAgger matched or outperformed all baselines in task reward and reduced context switches by 79% on HalfCheetah, 56% on Walker2D, and 46% on Ant.It requested a similar number of supervisor actions as SafeDAgger across domains; both methods used an order of magnitude fewer supervisor actions than DAgger.Source: A. Simulation Experiments: MuJoCo… · p. 5
• Cutoff latencies were 0.0 for HalfCheetah, 4.3 for Walker2D, and 7.6 for Ant; above each threshold, LazyDAgger had lower total supervisor burden than SafeDAgger.For extremely low latency, SafeDAgger may be the better option.Source: A. Simulation Experiments: MuJoCo… · p. 5
• LazyDAgger outperformed both ablations on all tasks except that removing ground-truth switching performed similarly on Walker2D.The ablations removed ground-truth discrepancy for returning to autonomous mode or removed noise injection; LazyDAgger consistently requested more supervisor actions than either.Source: A. Simulation Experiments: MuJoCo… · p. 5

B. Fabric Smoothing in Simulation

• The simulation used an initially crumpled fabric and evaluated adaptation after transferring a policy to altered dynamics and a higher-fidelity fabric grid.Rollouts ended after 10 time steps, at 92% coverage, or when fabric moved more than 20% out of bounds; coverage measures the percentage of background plane covered by fabric.Source: B. Fabric Smoothing in Simulation · pp. 5–6
• The physical task required smoothing, aligning fabric corners into view, and making a triangular fold; SafeDAgger ended intervention prematurely, while LazyDAgger returned to robot control when confident in task completion.The rollouts show human actions in red, autonomous actions in green, and task stages in blue, red, and green, respectively.Source: Figure 5 · p. 6
• LazyDAgger-Execution achieved more successes with fewer context switches than the baselines in the physical three-stage fabric manipulation task.The table reports success and supervisor burden summed across 10 trials, with L*=0.28.Source: Table I · p. 6
• During fabric-smoothing learning, LazyDAgger achieved coverage comparable to SafeDAgger with fewer context switches, though it required more supervisor actions.The reported cutoff latency was L*=1.5; during execution, LazyDAgger achieved similar final coverage with significantly fewer context switches.Source: B. Fabric Smoothing in Simulation · p. 6

C. Physical Fabric Manipulation Experiments

• The ABB YuMi task began with crumpled fabric and required smoothing, precise corner alignment, and a triangular fold using image observations and point-and-click supervisor actions.Evaluation trained behavior cloning from 20 offline demonstrations augmented to 10 times as many examples, with rollouts terminated at success, specified failure conditions, or 15 actions.Source: C. Physical Fabric Manipulation… · pp. 6–7
• LazyDAgger-Execution achieved much higher success than both baselines, with all 10 trials reaching folding and 2.1 context switches per trial versus SafeDAgger-Execution’s 5.3.It requested more supervisor actions than SafeDAgger-Execution, but reduced supervisor burden for context-switch latencies L > L*=0.28.Source: C. Physical Fabric Manipulation… · p. 7
• SafeDAgger-Execution often failed because interventions ended before the policy could navigate bottlenecks, while irregular fabric states could also impede policy progress.LazyDAgger and SafeDAgger experienced Failure Mode D at similar rates, but SafeDAgger also failed from premature intervention termination.Source: C. Physical Fabric Manipulation… · p. 7

VII. DISCUSSION AND FUTURE WORK

• Across its evaluations, LazyDAgger reduced context switching by up to 79% over SafeDAgger while improving task performance.The discussion identifies broader intervention criteria and robot-gated fleet control as subsequent work.Source: §VII · p. 7

IX. APPENDIX

• The appendix supplies further details on MuJoCo experiments, hyperparameter sensitivity, simulated fabric experiments, and physical fabric experiments.Source: §IX · p. 9

A. MuJoCo

• The appendix describes MuJoCo environments, expert rewards, neural-network configurations, dataset partitioning, and LazyDAgger hyperparameters.The expert rewards for HalfCheetah-v2, Walker2D-v2, and Ant-v2 were 5330.78 ± 117.65, 3492.08 ± 1110.31, and 4492.88 ± 1580.42, respectively.Source: A. MuJoCo · p. 9

B. LazyDAgger Switching Thresholds

• Threshold tuning begins with the SafeDAgger procedure, which sets approximately 20% of the initial dataset as unsafe, then adjusts βH and βR to balance supervision and task performance.The authors note that parameter changes require rerunning experiments, adding supervisor burden.Source: B. LazyDAgger Switching Thresholds · p. 9, Figure 6 · p. 9
• Across MuJoCo environments, reward was relatively stable as βH decreased but could suffer at high βH; burden rose sharply at βH = 5×10−4 and increased as βR decreased.The heatmaps show test reward, supervisor actions, and context switches across threshold settings.Source: B. LazyDAgger Switching Thresholds · p. 9, Figure 6 · p. 9

• The simulator models fabric as a grid of point masses with self-collision forces and 100×100×3 RGB observations; actions specify a pick point and a place displacement.Episodes end after 10 steps, sufficient coverage, excessive out-of-bounds motion, or spring tearing; the supervisor pulls the corner furthest from its goal.Source: 1) Fabric Simulator · pp. 9–10
• Domain randomization varies fabric and background colors, gamma correction, and pixel noise to broaden visual conditions.Fabric colors are centered around blue, and pixel noise ranges uniformly from -15 to 15.Source: 1) Fabric Simulator · p. 10
• Camera position and rotation are randomized, with position offsets sampled from N(0,0.04) meters and Euler angles from N(0,90°).Source: 1) Fabric Simulator · p. 10

2) Actor Policy and Discrepancy Classifier:

• The actor policy uses four convolutional layers with 32 3×3 filters, followed by four fully connected layers.Source: 2) Actor Policy and Discrepancy… · p. 10
• The discrepancy classifier reuses the actor’s convolutional layers, freezes them after actor training, and adds three fully connected layers.Gradients are not backpropagated through the reused layers during classifier training.Source: 2) Actor Policy and Discrepancy… · p. 10

3) Training:

• For fabric smoothing, the actor is pretrained in simulation and fine-tuned in a new environment, while the discrepancy classifier is trained from scratch.Source: 3) Training · p. 10
• DAgger training follows 100 behavior-cloning epochs on 20,232 state-action pairs, then collects expert labels across 20 parallel environments and takes 240 gradient steps per epoch.The sim-to-sim setup adapts a policy from a blue 16×16 fabric grid with spring constant 10,000 to a pink 25×25 grid with spring constant 2,000.Source: Figure 7 · p. 10, Table III · p. 10, D. Fabric Manipulation with the ABB YuMi · p. 10

1) Experimental Setup:

• Physical experiments manipulate a 10” by 10” fabric square with a parallel-jaw gripper fitted with reverse tweezers.Robot grasps use camera calibration and top-down picks at fixed depth; tweezers can reliably pick only the top fabric layer.Source: 1) Experimental Setup · p. 10, Table IV · p. 11

2) Image Processing Pipeline:

• Workspace images are square-cropped, resized to 100 × 100, and denoised before entering the replay buffer and neural networks.The crop defines the region to smooth and align against, with the fabric assumed to start inside it.Source: 2) Image Processing Pipeline · p. 11

3) User Interface:

• The intervention interface lets a human drag on the scaled-up observation to specify a pick point and pull vector, converted into the environment’s action space.The action space is a ∈[−1,1]4.Source: 3) User Interface · p. 11

4) Data Augmentation:

• Each state-action pair is copied 10 times before replay-buffer insertion to reduce overfitting to the small amount of real data.The listed transformations are applied in random order.Source: 4) Data Augmentation · p. 11
• Augmentations vary contrast by 85–115%, brightness by 90–110%, saturation by 95–105%, and add per-channel pixel values from −10 to 10.They also apply Gaussian blur with σ from 0 to 0.6 and Gaussian noise with σ from 0 to 3.Source: 4) Data Augmentation · p. 11
• With probability 0.8, augmentation independently scales each axis by 98–102%, translates each by −2% to 2%, and rotates by −5 to 5 degrees.The affine transform’s scaling, translation, and rotation are applied together.Source: 4) Data Augmentation · p. 11
• Table V lists physical fabric execution hyperparameters; N, T, and σ2 do not apply at execution time.Source: Table V · p. 11