PaperlayerView exact revision on arXiv

Source-linked AI summary

Policy-Calibrated DAgger: Offline Calibrated Noise Injection for Imitation Learning

Jenny Wang, George Kantor

arXiv:2609.30462v1cs.RO

TL;DR

Imitation policies can drift from their training distribution as errors compound, while collecting expert data at policy-failure states can be unsafe or require choosing a noise level. Policy-Calibrated DAgger estimates policy noise offline from its action distribution and uses that estimate to augment expert interventions. It improves on dataset aggregation without noising and matches the best fixed noise level chosen in hindsight without a noise-level sweep.

  • Problem

    Imitation policies can accumulate errors that move the robot outside the training distribution, while existing collection approaches can involve unsafe conditions or require choosing a noise distribution.

    Source: §I · p. 1

  • Method

    Policy-Calibrated DAgger estimates policy noise offline from the policy’s action distribution and uses calibrated noise to augment expert interventions with targeted recovery data.

    Source: §I · pp. 1–2

  • Results

    Pooled calibrated noise improves on HG-DAgger by 2.8 points and is equivalent within a ±2 point margin to the best fixed noise level chosen in hindsight.

    Source: §V · p. 7

  • Takeaways & Limitations

    The method achieves higher success rates than behavior cloning and HG-DAgger and matches the best hindsight noise level without an expensive noise-level sweep with an expert in the loop.

    Source: §VI · p. 7

Abstract

from arXiv · show

Policies trained with imitation learning can accumulate errors over time, causing the robot to drift outside the training distribution. Existing methods mitigate this covariate shift by collecting additional data where the policy fails or is likely to fail. The first places the robot in unsafe conditions and the second requires choosing an appropriate noise distribution to collect new expert demonstrations under that noise. We propose Policy-Calibrated DAgger, a method that makes use of the properties of recent generative policies to estimate the policy's noise offline by using its own predicted action distribution. We measure a diffusion policy's spread of predicted actions at observations along the expert trajectory and measure its closed-loop error relative to a recorded trajectory. To address issues with measuring error in a multimodal action space, we guide the policy towards the trajectory during closed-loop control through partial denoising, and use properties of a diffusion model to unnormalize the measured error as if we did not guide it. We experiment in a scenario where a robot is tasked to reach an engine lever in a cluttered and narrow environment and show results in a 3D photorealistic simulator and a 2D planar reacher environment. We show that our method surpasses policies trained with dataset aggregation without noising and matches the performance of the best noise level in hindsight, without requiring a sweep over noise levels.

I. INTRODUCTION

Imitation policies can compound errors and drift from their training distribution, while conventional DAgger requires expert guidance in policy-induced states. The paper proposes estimating policy noise offline to create targeted recovery data without requiring experts to control a noisy system.

I. INTRODUCTION

  • Small behavior-cloning errors can compound into state-distribution drift, and DAgger’s alternating expert-policy control can be unintuitive for human experts.Other approaches can burden experts with matching the policy’s induced distribution, which is labor-intensive.Source: §I · p. 1
  • Policy-Calibrated DAgger uses the policy to adapt expert interventions to its learned action distribution, without requiring expert control of a noisy system or modifying policy training.It uses partial diffusion to simulate policy-noise-expert interactions probabilistically.Source: §I · pp. 1–2
  • Policy-calibrated data augmentation noises expert interventions to generate recovery state-action pairs relevant to the policy’s induced state distribution.The method targets task failures without an oracle policy or additional real-world data collection.Source: §I · p. 2
  • The method is evaluated on a long-horizon cluttered goal-reaching task in a photorealistic simulator and a 2D planar simulator representing narrow passageways.The photorealistic simulator models the real engine-lever-reaching task mockup.Source: §I · p. 2

II. RELATED WORK

Related work addresses policy-induced data mismatch through interactive expert aggregation, learning from noisy or suboptimal data, and augmentation of limited demonstrations. The paper positions its approach as using policy errors to target recovery trajectories without an oracle policy or additional expert collection.

A. Interactive Imitation Learning

  • DAgger aggregates expert recovery actions in states the policy would visit, while interactive variants include human-gated interventions and force-feedback methods that keep interventions closer to the policy distribution.These approaches can still place the burden of matching the policy distribution on the expert.Source: A. Interactive Imitation Learning · p. 2

B. Learning from Suboptimal Data

  • Prior methods shape expert interventions with uncertainty queries or injected noise; this paper uses partial diffusion to estimate policy error relative to expert trajectories and calibrate targeted augmentation.The paper states that the partial-diffusion bound allows this estimation while constraining sampled policy behavior toward the expert during offline simulation.Source: B. Learning from Suboptimal Data · p. 2

C. Data Augmentation for Imitation Learning

  • Demonstration-augmentation methods use semantic or language priors, heuristics, or oracle planners, whereas this method generates recovery trajectories in states the policy is likely to visit without an oracle policy or additional inductive biases.The paper reports that calibrated targeted augmentation performs as well as the result of a noise-level hyperparameter sweep.Source: C. Data Augmentation for Imitation… · p. 2

III. PRELIMINARIES

The section frames noise injection as minimizing covariate shift between policy and noised-expert trajectories. DART estimates a Gaussian noise parameter iteratively, but requires closed-loop expert queries and tuning of the final error estimate.

A. Noise Injection for Robust Imitation Learning

  • The stated objective is to minimize KL divergence between trajectories visited by the trained policy and those visited by the noised expert.DART optimizes the sufficient statistics of the noise distribution for this objective.Source: A. Noise Injection for Robust… · p. 2
  • The illustrated DAgger-style loop trains each round on the base dataset, aggregated interventions, and noised interventions, with calibrated noise pooled across data.The figure depicts open-loop policy sampling, partially denoised blended trajectories, and per-timestep noise estimates as inputs to the pooled estimate.Source: Figure 2 · p. 3
  • DART derives an iterative estimate of the Gaussian noise covariance, requiring expert queries at noised states in closed-loop control.The method’s estimate of the robot’s final error α also requires hyperparameter tuning, so the process must be repeated.Source: A. Noise Injection for Robust… · p. 3

IV. POLICY-CALIBRATED DAGGER

Policy-Calibrated DAgger predicts augmentation-noise parameters to bring expert demonstrations closer to the final robot policy’s state distribution, without injecting noise into the expert’s control loop.

IV. POLICY-CALIBRATED DAGGER

  • Policy-Calibrated DAgger predicts augmentation-noise parameters so expert demonstrations cover states closer to the final robot’s distribution.The method computes sufficient statistics offline in a simulator rather than injecting noise into the expert’s control loop.Source: §IV · p. 3
  • Unlike DART, the method does not inject noise into the expert’s control loop.The expert can provide clean demonstrations without interruption.Source: §IV · p. 3
  • The method computes sufficient statistics offline and in a simulator, placing the burden of corrections on the policy rather than the expert.Source: §IV · p. 3

A. Method Design

Policy-Calibrated DAgger augments expert intervention data with targeted recovery trajectories, then trains and fine-tunes policies on aggregated original and augmented data.

A. Method Design

  • The method generates targeted recovery trajectories that match policy failures during evaluation using a noise metric computed entirely offline.Source: A. Method Design · p. 3
  • Each DAgger round aggregates expert interventions, noises recorded intervention states to generate trajectories returning to the expert trajectory, and co-trains on original and augmented data.Source: A. Method Design · p. 3
  • For final evaluation, the initial policy is fine-tuned on all rounds’ aggregate and noised datasets; interventions use expert-gated control and a deterministic RRT-based oracle.The oracle uses simulator privileged information and heuristics to decide when control returns to the policy.Source: A. Method Design · p. 3

B. Partial Denoising as a Learned Action Interpolator

Partial denoising uses a diffusion policy’s learned priors to blend expert and policy actions, preserving meaningful action structure where linear interpolation may fail.

B. Partial Denoising as a Learned Action Interpolator

  • Lower blend ratios β produce action chunks closer to the sampled expert chunk, while the unguided policy at β = 1.0 may travel in a different direction.Source: Figure 3 · p. 4
  • The method measures error after interpolating expert and policy actions with partial denoising, then uses the policy’s noise schedule to estimate unguided error.Source: B. Partial Denoising as a Learned… · p. 4
  • Linear interpolation can yield an invalid action even when both expert and policy actions are valid, such as blending opposing obstacle-avoidance directions into a collision course.Source: B. Partial Denoising as a Learned… · p. 4
  • Partial denoising blends expert and policy actions through the policy’s learned priors rather than directly interpolating between actions.A diffusion model adds noise in a forward process and denoises in a reverse process.Source: B. Partial Denoising as a Learned… · p. 4
  • Partial diffusion runs the forward and reverse processes for ksw − 1 steps, with ksw set from the forward diffusion ratio β and total steps K.The original action’s preservation is bounded under the stated normalized-action and predicted-noise assumptions.Source: B. Partial Denoising as a Learned… · p. 4
  • A larger switching timestep ksw empirically produces larger differences between the expert action and the partially diffused action.Source: B. Partial Denoising as a Learned… · p. 4

C. Extending Partial Denoising to Action Chunks

The method extends partial denoising to expert action chunks by flattening each time-horizon matrix into a vector, with offline processing used to accommodate future-spanning expert actions.

C. Extending Partial Denoising to Action Chunks

D. Estimating Policy Calibration Parameters

The method estimates policy-calibrated noise from expert trajectories without requiring an expert in the loop. It estimates the policy’s settling error by observing partially denoised rollouts across blend ratios.

D. Estimating Policy Calibration Parameters

  • The method empirically measures the unknown quantities in Equation 2 in simulation without requiring an expert in the loop.Figure 4 presents the fitted curve for estimating W in the planar reacher environment.Source: D. Estimating Policy Calibration… · p. 4, Figure 4 · p. 5
  • Policy observations at expert-trajectory segment starts are used to compare predicted action chunks with the expert’s future trajectory, approximating the unknown action-distribution spread open loop.The approximation uses recorded expert trajectories rather than noised expert trajectories because it is found to be close to the true value in practice.Source: D. Estimating Policy Calibration… · pp. 4–5
  • Closed-loop policy error is difficult to measure directly because an unsupervised policy can diverge from the expert, making L2 error ill-posed in multimodal tasks.Partial denoising keeps sampled trajectories closer to the expert while retaining policy influence, modeling a balance between expert guidance and policy error.Source: D. Estimating Policy Calibration… · p. 5
  • The method models the expert as settling at zero error and the policy as settling toward an alternate goal W, with isotropic gains for both.These assumptions define the competing-force model used to interpret the measured error.Source: D. Estimating Policy Calibration… · p. 5
  • Policy deviations are sampled from partially denoised blended rollouts at 13 blend ratios from 0.05 to 0.90, then Equation 12 is fitted to estimate W.Rollouts that diverge from the expert are discarded; the fitted W is the current policy’s settling error and tends to remain stable across rounds.Source: D. Estimating Policy Calibration… · p. 5
  • Per-step error estimates are pooled to reduce sample noise and better match DART’s per-round constant.Error is measured in median-step-size units from the demonstration data.Source: D. Estimating Policy Calibration… · p. 5

E. Generating Corrective Labels

Corrective labels are generated by pairing perturbed states with smooth action chunks that return to the expert intervention trajectory. The section situates evaluation in cluttered reaching tasks and describes the evaluation scope of the reported tables.

E. Generating Corrective Labels

  • Each perturbed state is labeled with an action chunk that returns it to the original expert intervention trajectory, generated using the demonstration data’s velocity and acceleration statistics.The state is associated with an expert-trajectory timestep by monotone closest-point projection; a critically damped PD controller creates a smooth return trajectory with accelerations clamped to demonstration values.Source: E. Generating Corrective Labels · p. 5
  • Table I reports mean±SE success rates across 100 planar-reacher scenarios, with most cells averaged over five DAgger lineages.Policy variants are fine-tuned from a BC checkpoint for each round, and the table compares swept isotropic noise with calibrated noise estimated before training.Source: Table I · p. 6
  • Table II reports mean±SE success rates across 100 photorealistic engine lever-reaching scenarios, using the Table I protocol with fine-tuning to 95k steps.BC is trained on 500 demonstrations.Source: Table II · p. 6
  • The evaluated tasks have cluttered narrow passageways; the photorealistic 3D physics simulation recreates the real mockup, while the 3-link planar robot is used for ablations.The robot may need to untangle its links to progress toward the goal.Source: Figure 5 · p. 6

V. EXPERIMENTS

Across planar and 3D tasks, policy-calibrated noise improves performance over aggregation without noise, while matching the best fixed noise level in hindsight on the planar task. Experiments test whether augmenting intervention data with noise matched to policy errors improves learning with the same number of expert interventions.

Experimental setup

  • The evaluation compares offline BC, human-gated DAgger, fixed-noise variants, and policy-predicted noise on a narrow, cluttered engine-lever task and a 2D obstacle-avoiding reacher task.The engine task uses a photorealistic simulator and joint-space actions; a RRT-based oracle intervenes near collisions or stalls and returns control after executing part of its plan.Source: §V · p. 6

Data and training protocol

  • Experiments pair policies across lineages and fine-tune them for the same number of steps from the same base checkpoint; noised variants use base, intervention, and augmented data.The augmentations target policy-induced errors at evaluation time, and planar results report that even σ = 2 improves success over HG-DAgger across rounds when action labels are relabeled alongside observation noise.Source: §V · pp. 6–7

Planar reacher results

  • Pooled calibrated noise improves planar reacher success over HG-DAgger by 2.8 points and is equivalent within a ±2 point margin to the best fixed noise level chosen in hindsight.BC reaches 68.0% success, while HG-DAgger rises from 79.9% to 87.3% over five collection iterations.Source: §V · p. 7

3D engine lever results

  • In the 3D engine lever task, Pooled ¯Σα scores higher than BC and HG-DAgger with the same intervention data, while HG-DAgger performs similarly to BC.Most HG-DAgger failures are collisions within the first second, limiting intervention coverage; noised observations reuse the original image and noise only joint-position labels.Source: §V · p. 7

Noise-level results

  • Across five lineage reruns, the best fixed noise level falls from σ = 8–16 at K = 1 to σ = 2–4 by K = 4, and large noise can hurt late-round success.Calibrated noise performs similarly to the best fixed isotropic level without sweeping noise magnitudes; measured per-joint σ is 3.9–5.1 in the planar task and 1.3–4.2 in 3D.Source: §V · p. 7

VI. CONCLUSION

Policy-Calibrated DAgger applies policy-calibrated noise to iterative imitation-learning data without additional expert supervision. It achieves higher success than BC and HG-DAgger and matches the best hindsight noise level without an expert-in-the-loop sweep.

Conclusion

  • Policy-Calibrated DAgger applies policy-calibrated noise to iterative imitation-learning data, producing more robust policies without additional expert supervision.The method trades expert time for compute and targets policy errors during evaluation; increased success rates are consistent with reduced covariate shift.Source: §VI · p. 7
Loading 2609.30462v1…