PaperlayerView exact revision on arXiv

Source-linked AI summary

Max-Q Selective Imitation for Human-in-the-Loop Online Robot Learning

Zihang Wang, Yishan Wang

arXiv:2608.15088v1cs.ROcs.AI

TL;DR

Human-in-the-loop robot reinforcement learning must rapidly absorb human corrections while improving beyond the human prior under costly, safety-constrained interaction. This paper combines MC Q-chunk evaluation with max-Q selective imitation, achieving 99% success in 30 minutes on a real USB pick-and-insertion task versus about 5 hours for HIL-SERL, while reaching at least 96% success in simulation within roughly half an hour.

  • Problem

    HIL robot learning needs to absorb useful human corrections quickly while continuing beyond demonstration-level performance under costly, safety-constrained interaction.

  • Method

    The method combines an MC Q-chunk critic for sample-average evaluation with hard max-Q selective imitation between policy and buffer actions.

  • Results

    Across simulated assembly tasks and a real robot task, Q-chunk variants outperform HIL baselines on success–time efficiency, with ACT QChunk-MCBC reaching 99% success in 30 minutes versus HIL-SERL’s roughly 5 hours.

  • Takeaways & Limitations

    The approach supports rapid intervention learning while shifting toward on-policy self-improvement as the autonomous policy strengthens.

  • Takeaways & Limitations

    Performance depends on critic quality, while Monte Carlo targets can have higher variance and the chunk horizon remains task-dependent.

Abstract

from arXiv · show

Human-in-the-loop (HIL) online reinforcement learning for real robots must absorb human interventions quickly while continuing to improve beyond the human prior. We present a training method for this setting based on two components. First, an \emph{MC Q-chunk} critic regresses chunk-level action values onto Monte Carlo returns from the replay buffer, performing sample-average (behavior) policy evaluation so that intervention trajectories are credited directly rather than diluted by current-policy TD backups. Second, \emph{max-Q selective imitation} updates the actor by imitating, at each state, the higher-$Q$ action between the current policy action and a buffer sample under a hard winner-take-all rule. This rule automatically switches between learning from interventions and on-policy self-improvement: when the autonomous policy is stronger, targets align with the policy distribution, reducing the policy--target-sample gap that otherwise induces execution-time distribution shift. In practice we score candidates with a standard critic ensemble mean to reduce comparison noise, without softening targets or introducing score-gap thresholds. On a real USB pick-and-insertion task with 20 demonstrations, ACT QChunk-MCBC attains 99\% success within 30 minutes of HIL training, whereas HIL-SERL requires about 5 hours to converge. In simulation on Peg Insertion and Square, ACT/Flow Q-chunk variants similarly reach $\ge$96\% success within roughly half an hour of effective training, outperforming HIL-SERL, EXPO, and E2HiL on the success--time frontier.

1 Introduction

The paper introduces MC Q-chunk evaluation and max-Q selective imitation for HIL robot learning, directly crediting intervention returns while switching toward on-policy self-improvement. The method reduces policy–target mismatch and achieves faster high success than established baselines in simulation and on a real USB task.

  • Method: MC Q-chunk regresses chunk-level values onto logged Monte Carlo returns, providing sample-average evaluation that credits intervention trajectories.This improves intervention-return utilization relative to critics using π-bootstrapped TD backups.
  • Method: Max-Q selective imitation compares the policy action with a buffer action and imitates the higher-Q candidate using a hard winner-take-all rule.The rule switches from intervention learning to on-policy self-improvement when the autonomous action has higher Q.
  • Method: When the policy action wins, target samples align with execution under π, reducing the policy–target-sample gap without soft switching or score-gap thresholds.A standard critic-ensemble mean reduces comparison noise while preserving the discrete switching rule.
  • Results: ≥96% success is reached within roughly half an hour on simulated Peg Insertion and Square by ACT/Flow Q-chunk variants, outperforming HIL-SERL, EXPO, and E2HiL on the success–time frontier.The reported simulation results use effective training time.

2 Related Work

The method builds on human-in-the-loop robot RL, offline-to-online policy improvement, action/Q-chunking, and max-over-candidates policy improvement. It combines MC Q-chunk behavior evaluation with max-Q selective imitation between live-policy and replay actions.

  • Human-in-the-loop robot RL: HIL-SERL combines demonstrations, interventions, and off-policy SAC-style learning, whereas this method uses MC Q-chunk behavior evaluation and max-Q selective imitation.The proposed replacements are π-conditioned TD evaluation with Monte Carlo Q-chunk evaluation, and likelihood-oriented actor updates with hard winner imitation.
  • Offline and offline-to-online RL: Offline RL constrains policies to data support, while IQL evaluates in-sample values and extracts policies with AWR.For online HIL fine-tuning, AWR targets buffer actions only; this method also treats the live policy action as a candidate, enabling on-policy targets when π dominates under Q.
  • Action chunking and Q-chunking: Action chunking supports temporally coherent imitation, and Q-chunking lifts TD actor–critic learning into a chunked action space.This method retains chunk-level Q(s, a) but fits it with Monte Carlo returns, preserving sample-average evaluation without bootstrapping under the current π.
  • Max-over-candidates policy improvement: BCQ and EMaQ maximize Q over actions near behavior support, while this HIL variant compares the live policy action with a replay sample and imitates the hard winner.The replay sample is often an intervention, and improvement is realized through supervised imitation.

3 Method

The method combines Monte Carlo chunk-level behavior-policy evaluation with hard max-Q selection between policy and buffer actions. This supports label-free switching between intervention imitation and autonomous self-improvement while limiting out-of-distribution critic queries and policy–target-sample mismatch.

  • HIL online protocol: The replay buffer starts with demonstrations and grows through autonomous rollouts and human interventions, without requiring intervention labels.Each transition is treated as an ordinary state–action pair under a limited robot interaction budget.
  • MC Q-chunk critic: The MC Q-chunk critic regresses chunk values onto logged-trajectory Monte Carlo returns, evaluating the buffer’s mixed autonomous-intervention behavior policy.Intervention actions receive direct MC credit, and chunk-level scoring matches the temporal structure of demonstrations and interventions.
  • Implementation: An optional ensemble mean can provide the ranking score to reduce early Q-comparison noise, while a single critic preserves the same learning rule.The ensemble is an implementation variance-reduction trick rather than a methodological contribution.
  • Max-Q selective imitation: At each sampled state, the actor imitates the higher-Q candidate between the current policy action and a buffer action under a hard winner-take-all rule.The target remains one discrete action; soft mixing and hand-tuned score-margin gating are not used.
  • Max-Q selective imitation: The hard max restricts scored actions to policy or data support, moves the actor toward the higher-Q candidate, and switches label-free between intervention and self-imitation.When interventions win they are imitated; when the policy is stronger, policy actions win more often and reduce the policy–target-sample gap.

4 Experiments

Across simulated Peg Insertion and Square tasks and a real USB insertion task, MC Q-chunk critics with max-Q selective imitation achieved high success substantially faster than TD-based HIL baselines. ACT QChunk-MCBC was the fastest variant in both simulation and the real-robot evaluation.

  • Experimental setup: The simulation comparison included HIL-SERL, plain MCBC, EXPO, and E2HiL, with ACT QChunk-MCBC and Flow QChunk-MCBC differing only in their chunked action heads.Success was measured through autonomous evaluation episodes at saved checkpoints.
  • Peg Insertion: ACT QChunk-MCBC reached 99% success on Peg Insertion by 0.32 h, while Flow QChunk-MCBC reached 96% by 0.50 h.Plain MCBC reached 96% by 0.67 h under the same MC critic.
  • Peg Insertion: HIL-SERL reached 100% on Peg Insertion only after ≈3.2 h and 100k steps, whereas E2HiL peaked at 90% near 2.4 h and EXPO at 30%.The Q-chunk plus max-Q stack delivered the best success–time trade-off on this task.
  • Square: On Square, ACT QChunk-MCBC reached 99% success by 0.38 h, while Flow QChunk-MCBC reached 96% at 2,200 steps and 0.60 h.Plain MCBC reached 96% by 0.80 h, while HIL-SERL eventually reached 100% only after several hours.
  • Real-robot evaluation: On the real USB pick-and-insertion task, ACT QChunk-MCBC attained 99% success by 0.8 hours, compared with approximately 5 hours for HIL-SERL to reach ceiling success.Plain MCBC reached 96% by about 1 hour; E2HiL peaked near 90% after several hours and EXPO saturated at 30%.

5 Limitations

The method’s switching and target construction depend on critic quality, stable Monte Carlo returns, and task-specific chunk-horizon selection. Its broader multi-task validation and formal analysis remain incomplete.

  • Noisy early Q estimates can select suboptimal targets, while critic ensembles only partially mitigate this limitation.The ensemble remains a heuristic rather than a complete fix.
  • Monte Carlo returns typically have higher variance than TD backups and may require sufficient episode completion for stable targets.
  • The chunk horizon h remains a task-dependent hyperparameter, limiting how uniformly the method can be configured across tasks.
  • Broader multi-task evaluation and a formal characterization of the two-candidate improvement remain open.

6 Conclusion

The paper concludes that MC Q-chunk behavior evaluation combined with hard max-Q selective imitation enables HIL robot learning to absorb beneficial interventions and transition toward autonomous self-improvement. ACT/Flow Q-chunk variants achieve ≥96% success in simulation on Peg Insertion and Square.

  • Conclusion: MC Q-chunk evaluation and hard max-Q selective imitation form the proposed HIL online robot learning method.The critic uses sample-average behavior evaluation, while the actor selects between policy and buffer actions using Q-values.
  • Conclusion: The updates absorb useful interventions when they improve Q and shift toward on-policy self-improvement as the autonomous policy strengthens.This transition reduces the policy–target-sample gap at execution time.
  • Conclusion: ≥96% success is reached by ACT/Flow Q-chunk variants in simulation on Peg Insertion and Square.The reported simulation result supports the method’s effectiveness across both tasks.
Loading 2608.15088v1…