PaperlayerView exact revision on arXiv

Source-linked AI summary

Reward-DAgger: Robot-Gated Interactive Imitation Learning with General-Purpose Progress-Based Reward Models

Ryan Li, Yigit Korkmaz, Erdem Bıyık

arXiv:2610.04054v1cs.ROcs.AI

TL;DR

Robot policies can fail in deployment, while existing runtime monitors often require task- or policy-specific training and calibration. Reward-DAgger uses general-purpose progress estimates to gate human intervention and policy updates. Across simulated and real-world tasks, it improves autonomous success and achieves strong return on human effort, with the same gate configuration transferring across tasks and settings.

  • Problem

    Robot policies can fail during deployment, while existing monitoring often requires task- and policy-specific training or calibration.

    Source: §I · p. 1

  • Method

    Reward-DAgger uses progress from a general-purpose reward model to trigger human takeover and uses the resulting corrections to iteratively fine-tune the policy.

    Source: §I · pp. 1–2, A. Gating Criterion · p. 2

  • Results

    Across simulated and real-world tasks, Reward-DAgger improves autonomous success through interactive learning and outperforms baselines in most settings for success rate and return on human effort.

    Source: §I · p. 2, Q3: Policy Improvement with… · p. 7

  • Takeaways & Limitations

    A single gate configuration transfers across tasks, environments, and policy architectures without task-specific tuning or per-round recalibration.

    Source: §I · p. 2, §V · p. 8

Abstract

from arXiv · show

Recent advances in robot learning have enabled generalist control policies capable of completing a wide range of tasks. However, their performance degrades when deployed in unseen environments, making it critical to detect failures and teach recovery behaviors. Existing runtime monitoring methods often require task- and policy-specific training or hyperparameter tuning, limiting cross-task deployment and introducing additional overhead during iterative policy updates. We present Reward-DAgger, a robot-gated interactive imitation learning framework that uses dense progress signals from a general-purpose reward model to determine when human intervention is needed. Our approach is agnostic to the underlying policy architecture, requires no access to policy internals, and can be applied across tasks without retuning the gating mechanism. Our results show that Reward-DAgger achieves a better failure-detection accuracy-latency tradeoff than existing runtime monitoring baselines. Across eight simulated and real-world tasks, Reward-DAgger consistently improves the downstream policy's autonomous success rate throughout interactive learning and achieves strong return on human effort, outperforming the baselines in most settings. Importantly, the same gating configuration is used across tasks without task-specific hyperparameter tuning, demonstrating transfer across tasks, environments, and policy architectures. Code and videos are available at https://liralab.usc.edu/reward-dagger.

I. INTRODUCTION

Reward-DAgger uses progress from a general-purpose reward model to gate human intervention, addressing the burden and calibration limits of existing monitoring. Across simulated and real-world tasks, it improves policy success and transfers the same gate across tasks, environments, and policy architectures.

I. INTRODUCTION

  • Existing robot gates use failure predictors trained on explicit failure data or policy-dependent uncertainty scores requiring task- and policy-specific calibration.Policy updates can force those gates to be retrained or recalibrated, limiting scalability.Source: §I · p. 1
  • A scalable gate should detect failures early while avoiding unnecessary interruptions, persist across learning updates, work with black-box policies, and transfer across tasks without separate calibration.These four properties motivate the paper's gate design.Source: §I · p. 1
  • Reward-DAgger uses progress estimates from a general-purpose reward model because they describe task accomplishment independently of how a policy generates actions.The paper evaluates four simulated and four real-world manipulation tasks, including task-specific diffusion policies and large pretrained VLAs.Source: §I · pp. 1–2
  • The two-timescale progress gate detects abrupt failures and stalled behavior, then triggers human takeover so resulting corrections can inform policy updates.The reward model and gate configuration remain fixed throughout interactive learning, without per-round retraining or recalibration.Source: §I · pp. 1–2
  • Across simulation and real-world tasks, Reward-DAgger improves autonomous success through interactive learning and outperforms baselines in most settings for success rate and return on human effort.The same simulation-selected gate configuration is used on four real-world tasks without task-specific tuning.Source: §I · p. 2

II. RELATED WORK

Prior robot-gating and runtime-monitoring approaches often depend on task-specific criteria or policy-dependent signals. Reward-DAgger instead uses an external task-progress signal to gate interventions and connect deployment-time monitoring with iterative policy updates.

II. RELATED WORK

  • Robot-gated methods automate intervention decisions, but commonly rely on task-specific criteria or policy-dependent signals that need extra training or calibration across tasks or policy updates.Reward-DAgger instead uses an external task-progress signal applicable zero-shot across tasks and policy architectures.Source: §II · p. 2
  • Runtime-monitoring methods calibrate failure thresholds from scores based on action consistency, observation-distribution deviation, action-chunk entropy, or internal VLA features.They are primarily studied for deployment without updating the monitored policy; Reward-DAgger uses gate firings to collect corrections for subsequent updates.Source: §II · p. 2
  • General-purpose reward models estimate progress from visual observations and language instructions, but directly optimizing imperfect learned rewards can lead to reward hacking.Reward-DAgger uses progress as a monitoring signal rather than directly optimizing it.Source: §II · p. 2

III. METHOD

  • Reward-DAgger uses dense progress from general-purpose reward models to determine when to transfer control to an expert, then incorporates the criterion into policy fine-tuning.Source: §III · p. 2

A. Gating Criterion

The gating criterion monitors progress estimates from a general-purpose reward model at two timescales. A short window detects sharp drops, while a longer window detects plateaus, with rank correlations and a drop-magnitude threshold defining when the gate fires.

A. Gating Criterion

  • The general-purpose progress model produces a scalar estimate from visual observations and a natural-language task instruction; larger values indicate greater predicted progress.The model can be constructed by prompting or fine-tuning a VLM.Source: A. Gating Criterion · pp. 2–3
  • Rapid progress decreases can indicate lost prior progress, while plateaus can indicate stalled behavior such as repeated missed grasps or oscillation near a target.Source: A. Gating Criterion · p. 3
  • The short window examines recent progress for sharp decreases, and the longer window, where Ll > Ls, examines it for plateaus.Each gate is evaluated only after its corresponding window is full.Source: A. Gating Criterion · p. 3
  • Spearman rank correlation measures the trend in each window, but because it captures direction rather than change magnitude, the short-window gate also uses a maximum-drop threshold.The combined criterion fires when either the short-window or long-window gate condition is met.Source: A. Gating Criterion · p. 3

B. Reward-DAgger

Reward-DAgger iteratively collects expert corrections when its reward gate fires, aggregates them with the existing dataset, and fine-tunes the policy without changing the gate configuration.

B. Reward-DAgger

  • The robot executes its policy until the reward gate fires, then an expert provides corrections that are aggregated with the existing dataset to fine-tune the next policy.The process repeats for a fixed number of iterations and a fixed correction budget, while the gate configuration remains unchanged.Source: B. Reward-DAgger · p. 3

IV. EXPERIMENTS

The experiments evaluate Reward-DAgger's failure detection, cross-task transfer, and interactive policy improvement across simulated and real-world tasks.

IV. EXPERIMENTS

  • The evaluation asks whether the gate detects failures with low latency, transfers across tasks without calibration, and improves policy performance while minimizing human burden.The study covers four simulation and four real-world manipulation tasks, with task-specific diffusion policies and pretrained VLAs.Source: §IV · pp. 3–4
  • Baselines include novelty-and-risk gating, diffusion-policy uncertainty methods, and learned failure or observation scores.The compared methods are ThriftyDAgger, Diff-DAgger, UCF, SAFE-LSTM, and logpZO.Source: §IV · p. 4
  • Simulation uses LIBERO-10, Robometer-4B progress estimates, a frozen π0.5 expert, and a diffusion-policy student.Robometer-4B produces dense, frame-level task-progress estimates on a normalized [0, 1] scale.Source: §IV · p. 4
  • Real-robot evaluation uses DROID with a Franka FR3 and Robotiq gripper on four tasks, with π0.5 as student and a human teleoperator as expert.Tasks include block stacking, placing a marker in a cup, moving bread to a bowl, and removing objects from a basket.Source: §IV · p. 4

Q1: Failure Detection

Reward-DAgger's gate is evaluated for failure-detection accuracy and latency against monitoring baselines using accuracy–latency Pareto frontiers.

Q1: Failure Detection

  • Balanced accuracy averages the gate's true-positive rate on failed trajectories and true-negative rate on successful trajectories, accommodating class imbalance.The gate should fire on failures but not on successful trajectories.Source: Q1: Failure Detection · p. 4
  • Latency is mean normalized trigger time on failed trajectories; a failure with no trigger is assigned the episode horizon.The trigger time is normalized by episode horizon because precise failure onset is difficult to label manually.Source: Q1: Failure Detection · p. 4
  • Each of four LIBERO tasks contributes 200 student rollouts, with 50 reserved for learned-gate training and 150 for evaluation; methods are compared across Pareto frontiers.Configurations are evaluated with repeated five-fold cross-validation on the 150 evaluation rollouts.Source: Q1: Failure Detection · p. 4
  • At least 0.96 maximum balanced accuracy on every task; Reward-DAgger's frontier is above or on par with baselines, with large improvements on Tasks 0 and 1.logpZO remains competitive at some early-trigger operating points, but no baseline performs consistently as well across all four tasks.Source: Figure 2 · p. 5, Q1: Failure Detection · p. 5

Q2: Transfer of Gating Configurations

Joint selection across multiple source tasks improves transfer stability, preserving average balanced accuracy over a range of tolerances with relatively small latency penalties.

Q2: Transfer of Gating Configurations

  • One-to-three transfer tunes on one task and evaluates on the other three; three-to-one transfer tunes jointly on three tasks and evaluates on the remaining task.The study examines transfer on LIBERO-10 Tasks 0, 1, 5, and 8.Source: Q2: Transfer of Gating Configurations · p. 5
  • A selected configuration minimizes average trigger time while staying within tolerance ϵ of the highest balanced accuracy on every source task.The transfer metrics compare target-task balanced accuracy and matched-accuracy latency penalty with the target task's native configuration.Source: Q2: Transfer of Gating Configurations · p. 5
  • One-to-three transfer is not consistently reliable, whereas joint multi-source selection preserves average balanced accuracy across tolerances with relatively small latency penalties.The authors attribute single-task transfer's unreliability to possible overfitting of task-specific sampling noise when choosing among many configurations.Source: Table I · p. 5, Q2: Transfer of Gating Configurations · p. 5
  • With ϵ = 0.02, configurations tuned on three of Tasks 0, 1, 5, and 8 are also evaluated on the omitted task and held-out Tasks 4 and 6.A configuration tuned jointly on all four source tasks is evaluated on Tasks 4 and 6; the reported conclusion is reliable transfer with preserved balanced accuracy and a small, controllable latency penalty.Source: Q2: Transfer of Gating Configurations · p. 5

Q3: Policy Improvement with Reward-DAgger

Reward-DAgger improves autonomous policy success in simulation and on real robots, with strong return on human effort. Its full gate combines short- and long-window signals, and the same configuration is reused across tasks without additional tuning.

Interactive-learning procedure

  • Each of three interactive-learning rounds collects 3,000 successful expert state-action pairs, then fine-tunes with 30% offline demonstrations and 70% aggregated interventions.Updated policies are evaluated over 100 autonomous rollouts; ROHE is measured during expert-gated data collection.Source: Q3: Policy Improvement with… · p. 6

Simulation results

  • Reward-DAgger improves autonomous success on all four simulated tasks, with highest final success on Tasks 0 and 6 and competitive results on Tasks 1 and 4.Tasks 4 and 6 were not used to select the gate configuration.Source: Q3: Policy Improvement with… · p. 6, Figure 4 · p. 7

Real-world results

  • Reward-DAgger achieves the highest ROHE in most real-world settings and improves autonomous success across rounds, while rarely triggering during successful behavior.The comparison also finds comparable performance in the other settings.Source: Table III · p. 6, Q3: Policy Improvement with… · p. 7

Cross-task evaluation

  • The same reward model and gate configuration work without additional tuning across eight simulated and real-world tasks and different policy architectures.The paper describes the resulting interventions as informative for downstream improvement and efficient in their use of human supervision.Source: Q3: Policy Improvement with… · p. 7

Gate design ablation

  • The full gate has the lowest or tied trigger time in 11 of 16 task-accuracy combinations and fires earlier than the absolute-progress gate in 14 of 16 comparisons.The short-window-only gate detects some failures earlier but misses high balanced accuracy on Tasks 0, 4, and 6; the long-window-only gate covers more broadly but detects later.Source: Ablations · p. 7, Table IV · p. 7

Reward-model backbone ablation

  • With RynnValue replacing Robometer-4B, the gate achieves a comparable accuracy-latency frontier and the second-highest balanced accuracy on both evaluated simulation tasks.On real-world Stack and Pick experiments, RynnValue yields smaller autonomous-success gains but comparable ROHE overall.Source: Ablations · pp. 7–8, Figure 5 · p. 8

V. CONCLUSION

Reward-DAgger uses a general-purpose reward model to detect progress drops and plateaus, enabling a policy-independent gate that remains fixed across learning rounds. Experiments show effective detection, transfer, and policy improvement, while the reactive design can delay intervention or limit use in high-risk settings.

Framework and scope

  • Reward-DAgger detects progress drops and plateaus from general-purpose reward-model estimates, and decouples the gate from policy checkpoints and architectures.The gate remains fixed without retraining or recalibration between DAgger rounds; the framework can incorporate more accurate general-purpose reward models without changing its pipeline.Source: §V · p. 8

Conclusions

  • In simulated and real-robot experiments, Reward-DAgger achieves effective failure detection, cross-task transfer, and downstream policy improvement.Source: §V · p. 8

Limitations and future work

  • The long-window gate can take several seconds to trigger on sustained plateaus, and the reactive framework may be unsuitable when failures are high-risk or difficult to recover from.The authors identify predictive world models as a possible way to anticipate failure outcomes and request intervention before execution.Source: §V · p. 8

Additional Experiments

Additional experiments examine whether gate configurations transfer across tasks and whether the gating mechanism works with a different reward-model backbone. The real-world backbone comparison reports smaller autonomous-success gains but comparable ROHE with RynnValue.

Frontier Transfer

  • For frontier transfer, configurations on each source task’s Pareto frontier are applied to the other three tasks, and Pareto-optimal transferred points form a new frontier.Figure 6 compares these transferred frontiers with each target task’s task-tuned frontier and reports hypervolume ratios.Source: Additional Experiments · p. 10, Figure 6 · p. 10
  • Figure 6 shows each target task’s native frontier as a solid orange curve and transferred configurations as dashed curves, with hypervolume ratios indicated for transferred curves.The dashed grey line marks the mean normalized length of successful trajectories.Source: Figure 6 · p. 10

Different Reward Model Backbone in Real World Tasks

  • On real-world Stack and Pick experiments, RynnValue yields smaller autonomous-success gains but comparable ROHE overall when using Robometer-selected gating parameters without additional tuning.Source: Additional Experiments · p. 10, Table V · p. 10

Related papers

Loading 2610.04054v1…