PaperlayerView exact revision on arXiv

Source-linked AI summary

BEE: Intervention-Adaptive Real-World Reinforcement Learning with Vision-Language-Action Models

Weihui Zhao, Xiaohan Yan, Zunian Wan, Xuan Du, Zhaozhan Chi, Jianbo Mao, Ruipu Wu, Rushuai Yang, Houlin Li, Shukai Yang, Jing Wu, Yuxiang Yan, Yongcheng Liu, Chuankang Li, Guanghui Ren, Wei Shan, Maoqing Yao

arXiv:2609.27450v1cs.ROcs.AI

TL;DR

Precision-critical manipulation and costly real-robot exploration make human corrections valuable, but existing VLA RL methods lack correction interfaces or treat corrections uniformly. BEE uses a proposal-conditioned model of correction residuals to constrain policy updates by action dimension. It reports the highest success rate on each evaluated task and the lowest intervention rate on real-world tasks, with 91.2% average success versus 57.5% for RLT and 42.1% for DSRL.

  • Problem

    VLA manipulation can fail at precision-critical phases, while costly rollouts limit exploration and existing online RL methods either omit human corrections or do not distinguish their reliability by action dimension.

    Source: §I · pp. 1–2

  • Method

    BEE models human corrections as residuals from retained VLA proposals and uses their predicted means and dimension-wise uncertainties to constrain RL policy optimization.

    Source: §I · p. 2

  • Results

    91.2% average success for BEE, versus 57.5% for RLT and 42.1% for DSRL; BEE also has the highest success rate on every task and the lowest intervention rate on all real-world tasks.

    Source: §I · p. 2, §VI · p. 7

  • Takeaways & Limitations

    Across the evaluated tasks, BEE improves success while requiring fewer human interventions on the real-world tasks.

    Source: §I · p. 2

Abstract

from arXiv · show

Vision-language-action (VLA) models handle long-horizon manipulation, yet success hinges on a few precision-critical phases where millimeter-scale errors undo all prior progress. Online reinforcement learning (RL) can optimize exactly these actions, but free exploration is far too costly on real robots, which makes human corrections indispensable. However, existing online RL methods for VLAs either cannot incorporate such corrections or fold them into undifferentiated supervision. Yet human corrections are not uniformly noisy but reliable along some action dimensions and variable along others. Building on this, we introduce BEE, an intervention-adaptive framework for real-world RL on a frozen VLA that lets the policy go BEyond Expert imitation. We formulate human corrections not as actions to reproduce but as evidence about a constraint: a Correction Model predicts how a human would correct a given VLA proposal and how consistent the correction is along each action dimension. This predicted consistency sets the per-dimension tightness of a constraint on policy optimization. Where corrections are consistent the policy stays close to the human, and where they vary, the constraint relaxes. We evaluate BEE on three real-world manipulation tasks and one LIBERO-Pro simulation task at a matched online-data budget. BEE attains the highest success rate on every task, 91.2% on average against 57.5% for RLT and 42.1% for DSRL, and the lowest human intervention rate on all real-world tasks.

I. INTRODUCTION

VLA policies can handle long-horizon manipulation, but precision-critical phases and costly real-robot exploration make targeted human corrections valuable. BEE uses proposal-conditioned, dimension-wise correction uncertainty to constrain RL, and outperforms the compared methods in the reported evaluation.

I. INTRODUCTION

  • Precision-critical phases make small action errors consequential, while costly real-robot rollouts make targeted human corrections valuable but scarce.Online RL can optimize actions through environment interaction, but sparse rewards and high-dimensional action chunks limit unconstrained exploration.Source: §I · p. 1
  • Existing VLA online RL methods either lack a human-correction interface or use corrections without distinguishing their reliability across action dimensions.RLT replaces stored VLA proposals with human corrections on intervention samples; SiLRI uses state-wise uncertainty but treats action dimensions uniformly.Source: §I · p. 2, B. Human-in-the-Loop RL · p. 2
  • BEE models each human correction as a residual from its retained VLA proposal and uses predicted correction means and uncertainties to constrain policy optimization dimension-wise.Consistent dimensions keep the policy close to the corrected action, while variable dimensions allow the constraint to relax.Source: §I · p. 2
  • BEE achieves the highest success rate on every evaluated task and the lowest human intervention rate on all real-world tasks.The evaluation covers three real-world manipulation tasks and one LIBERO-Pro simulation task.Source: §I · p. 2

A. RL for Vision-Language-Action Models

III. PRELIMINARIES

The preliminaries frame learning as chunk-level RL over a frozen VLA interface, then describe how human interventions enter off-policy training. BEE builds on that setup to turn interventions into a policy constraint.

A. Chunk-Level RL over a VLA Interface

  • At each chunk boundary, the agent emits a chunk of K primitive actions and seeks to maximize expected discounted return.The chunk-level formulation uses discount factor γ in (0,1).Source: A. Chunk-Level RL over a VLA Interface · p. 3
  • The frozen VLA supplies a proposal and latent token, while the RL policy conditions on both the resulting state and the proposal.The critic evaluates action chunks at the state; transitions are trained off-policy with a K-step Bellman target.Source: A. Chunk-Level RL over a VLA Interface · p. 3

B. Learning from Human Interventions

  • Human corrections override the policy action; corrected and autonomous transitions enter replay, while corrected chunks also enter a demonstration buffer.Training batches draw equally from the replay and demonstration buffers, and the VLA and encoder remain frozen.Source: B. Learning from Human Interventions · p. 3

IV. METHOD

BEE keeps the frozen VLA proposal as an input to a residual policy and models human corrections relative to that proposal. The predicted correction distribution centers and shapes a constraint on policy optimization.

A. Problem Formulation

  • The frozen VLA–RL interface supplies a state–proposal pair, and gradients are not propagated through the VLA.The state combines the VLA-derived latent token with proprioceptive state.Source: A. Problem Formulation · p. 3
  • A deterministic residual policy adds its predicted action residual to the VLA proposal, while a proposal-conditioned Gaussian Correction Model predicts human correction residuals.The model predicts a mean residual and diagonal covariance for the correction distribution.Source: A. Problem Formulation · p. 3
  • The correction mean centers the feasible set and predicted variances determine its dimension-wise geometry, keeping consistent dimensions narrow while leaving variable ones freer for return-driven improvement.Policy optimization is constrained on state–proposal pairs sampled from the training distribution.Source: A. Problem Formulation · p. 3

B. Learning the Correction Distribution

BEE retains both the VLA proposal and human correction to model their residual distribution, then uses correction information to shape policy constraints rather than treating corrections as direct targets.

B. Learning the Correction Distribution

  • Human-corrected samples retain the VLA proposal and correction, and the Correction Model is trained on the correction buffer by negative log-likelihood.The proposal remains an explicit model input so the model need not recover the high-dimensional action from the compact RL state.Source: B. Learning the Correction Distribution · p. 3

C. State-Dependent Primal–Dual Optimization

  • The actor objective combines value maximization with a penalty for distance from the VLA proposal.A state-dependent multiplier weights the proposal-distance term in the population Lagrangian.Source: C. State-Dependent Primal–Dual… · p. 3
  • The state-dependent multiplier is trained by dual ascent, implemented as minimization of its objective.The optimization is posed as a saddle problem over the policy and nonnegative multiplier.Source: C. State-Dependent Primal–Dual… · p. 3
  • Human corrections shape how the policy is constrained rather than serving as direct policy targets.The multiplier uses a softplus output, and sampled states are normalized by 1+λω(st) to stabilize large-multiplier updates.Source: C. State-Dependent Primal–Dual… · p. 3
  • Each optimization iteration samples equally from the RL and human-correction buffers, updates the critic, and then updates the policy and multiplier.The Correction Model is also updated periodically after new intervention samples.Source: C. State-Dependent Primal–Dual… · p. 4

D. Training Procedure

The Correction Model is pretrained on the initial correction buffer and updated periodically as new interventions arrive.

D. Training Procedure

  • The Correction Model is pretrained on seed corrections and updated only after every M new interventions, using NM gradient steps at a fixed update-to-data ratio.The conservative schedule is intended to reduce variance collapse on the small intervention buffer.Source: D. Training Procedure · p. 4

V. EXPERIMENTS

The experiments compare BEE with frozen-VLA baselines on three real-world manipulation tasks and one LIBERO-PRO simulation task under a matched online-data budget. The supplied results passage reports an online-data requirement for RLT, while the table caption specifies the success and intervention-rate measures.

A. Experimental Setup

  • The task suite includes phone charging, snack hanging, and cloth aligning on real hardware, plus bowl placing in LIBERO-PRO simulation.The real-world tasks end in precision-critical contact phases, and bowl placing runs over a comparable horizon.Source: A. Experimental Setup · p. 4, Figure 2 · p. 4
  • Online training updates lightweight residual-policy and auxiliary components while keeping the fine-tuned VLA and latent-token interface frozen.At deployment, the robot uses the residual policy on the frozen interface without interventions or exploration noise; the longest-horizon tasks adapt only their precision-critical phase.Source: A. Experimental Setup · p. 4
  • RLT requires roughly 3× the online data to reach 96.7±5.8.Source: A. Experimental Setup · p. 5
  • Methods are compared at a matched robot-data budget, with success measured over three rounds of 20 trials per task and intervention rate measured during online training.Phone charging and cloth aligning score only their precision-critical phase; snack hanging and bowl placing score full-task success.Source: A. Experimental Setup · p. 5, Table I · p. 5

B. Results and Analysis

At the matched online-data budget, BEE outperforms the compared methods, while its correction model estimates variation across action dimensions and uses that variation to allocate constraint strength.

Overall comparison

  • At the matched online-data budget, BEE achieves the highest success rate on every task and the lowest intervention rate on all real-world tasks and overall.Its gains hold across different Base-Policy starting points; other methods improve less under the same budget, and RLT has a substantially higher intervention rate.Source: B. Results and Analysis · p. 5

Sample efficiency

  • BEE stays above RLT at every online-episode budget on phone charging and snack hanging.On phone charging, BEE matches the strong Base Policy within the first few episodes and then saturates; on snack hanging, it explores away from a non-successful Base Policy before converging to a high success rate within the budget.Source: Figure 3 · p. 5, B. Results and Analysis · p. 5

Per-dimension reliability of human corrections

  • Across q1–q7, correction spread varies about 3×; the Correction Model reduces error to human corrections in every dimension, and its predicted uncertainty follows empirical spread.The analysis is restricted to the seven joint dimensions of the intervened arm, since the other arm and end-effector receive essentially no human correction.Source: Figure 4 · p. 6, B. Results and Analysis · p. 6

Dimension-wise constraint allocation

  • Mahalanobis weighting shifts constraint share toward consistently corrected q2, cuts q5’s share by more than half, and assigns the least share to variable q7.The state-wise form instead gives the largest share to q5, one of the least consistently corrected dimensions.Source: Figure 5 · p. 6, B. Results and Analysis · p. 6

C. Ablations

BEE leads the reported benchmarks at a matched online-data budget, and ablations indicate that proposal-conditioned correction modeling and dimension-wise constraint geometry matter. The conclusion also identifies sparse corrections as a boundary for uncertainty reliability.

C. Ablations

  • On snack hanging, removing the Correction Model or its VLA proposal yields the two lowest success rates; removing the proposal is worse than not modeling corrections at all.Raw corrections are too inconsistent as policy targets, while omitting the proposal makes predicted variances reflect VLA variability rather than human correction consistency.Source: C. Ablations · p. 7

C. Ablations; VI. CONCLUSIONS

  • The dimension-wise constraint sharply raises success and cuts intervention on snack hanging versus a single state-wise constraint, while the encoder ablation stays close to BEE on phone charging.The conclusion notes that predicted uncertainty is less reliable where human corrections are sparse and names multi-phase tasks, multiple operators, and broader embodiments as next steps.Source: C. Ablations · p. 7, §VI · p. 7

VI. CONCLUSIONS

  • At a matched online-data budget, BEE achieves 91.2% average success against 57.5% for RLT and 42.1% for DSRL, with the highest success on every task and lowest intervention rate on all real-world tasks.Ablations show that both the Correction Model and dimension-wise constraint geometry are necessary.Source: §VI · p. 7

APPENDIX

The appendix describes three real-world manipulation tasks and LIBERO-Pro bowl placing, then specifies their platforms, action spaces, and reward functions.

A. Task and Environment Details

  • The evaluated tasks are phone charging, snack hanging, cloth aligning, and bowl placing in LIBERO-Pro.The first three are real-world manipulation tasks; bowl placing is the simulated task.Source: A. Task and Environment Details · p. 8

1) Robot Platform, Observations, and Actions:

  • Real-world tasks use a dual-arm robot with three 480×640 RGB cameras; the learner receives the frozen extractor’s latent token and proprioceptive state.The VLA consumes camera views, language instruction, and proprioceptive state; cloth-aligning reward is computed from the head-camera view.Source: 1) Robot Platform, Observations, and… · p. 8
  • Phone charging and cloth aligning use joint-space control, with online RL editing only the right-arm joint dimensions; snack hanging and bowl placing use end-effector commands.For the latter two tasks, online RL edits pose dimensions while the gripper remains at the VLA proposal.Source: 1) Robot Platform, Observations, and… · p. 8

2) Simulation Task: Bowl Placing:

B. Reward Functions

  • Phone charging, snack hanging, and bowl placing receive zero reward until task completion, when reward becomes 1 and the episode terminates.These tasks use sparse task-completion rewards.Source: B. Reward Functions · p. 8
  • Cloth aligning uses the increment of a vision-based alignment potential as reward, and success requires mask intersection-over-union above 0.85 at some point.The same perception module defines the reward and success criterion.Source: B. Reward Functions · p. 8

C. Implementation Details

Implementation fixes a task-specific frozen VLA and extractor interface, alongside per-task action and budget specifications. Evaluation matches total robot interaction while varying how episodes are allocated and how interventions and phase switching are handled.

C. Implementation Details

  • The phone-charging configuration is detailed in Table III, while task-varying control rates, action dimensions, chunk lengths, and budgets are listed in Table II.The reported hyperparameter distances, thresholds, and noise scales use normalized action units.Source: C. Implementation Details · p. 8, Table II · p. 9, Table III · p. 10
  • Table II specifies per-task action dimensions, editable dimensions, chunk lengths, and online budgets for methods seeded with human corrections.For phone charging and cloth aligning, online RL refines only the precision-critical phase, and success is reported on that phase.Source: Table II · p. 9
  • The frozen VLA is task-specific, and its frozen extractor keeps the latent-token and proposal interface fixed during online training.Experiments start from π0.5 weights fine-tuned separately on each task; the extractor is an RLT-style latent-token encoder.Source: C. Implementation Details · p. 9

D. Intervention Protocol and Evaluation Details

  • Each task uses the same total robot-interaction budget across methods: 90 episodes for real-world tasks and 45 for bowl placing.For methods with demonstration buffers, about 20 episodes seed the buffer, leaving 70 real-world or 25 bowl-placing episodes online; other methods use all episodes online.Source: D. Intervention Protocol and… · p. 9
  • Operators take over when the robot is about to fail unrecoverably, concentrating corrections on precision-critical contact phases.In simulation, the operator uses a 6-DoF SpaceMouse with two gripper buttons.Source: D. Intervention Protocol and… · p. 9
  • For phone charging and cloth aligning, BEE and RLT switch from VLA approach to online RL when a progress-head prediction crosses a threshold.The progress head is trained during supervised fine-tuning.Source: D. Intervention Protocol and… · p. 9

E. Baseline Implementations

Baselines share a controlled evaluation setup, with stated exceptions for methods that do not assume a pretrained VLA. The section also reports limitations of imitation and from-scratch baselines and describes state-dependent constraint behavior during training.

E. Baseline Implementations

  • Baselines share tasks, reward source, intervention interface, action normalization, robot-data budget, and evaluation protocol; SiLRI and HIL-SERL retain their from-scratch visual-policy setup.Methods built on frozen VLAs start from the same fine-tuned π0.5 checkpoint and, when applicable, the same frozen extractor.Source: E. Baseline Implementations · p. 10
  • The Base Policy uses 200–900 task demonstrations for supervised fine-tuning, while its online budget is a small fraction of that training data.It consumes none of the online budget and initializes the frozen-VLA methods.Source: E. Baseline Implementations · p. 10
  • RLT is reimplemented on BEE’s chunk-level actor–critic backbone and shared frozen-VLA interface, replacing the stored proposal with the human correction at intervention transitions.This setup is intended to isolate how RLT and BEE use human corrections.Source: E. Baseline Implementations · p. 10
  • Table IV compares BEE with DAgger, SiLRI, and HIL-SERL on tasks where each comparison is informative, but intervention rates are not directly comparable across methods.All comparisons use the main paper’s intervention protocol and matched robot-data budget.Source: Table IV · p. 10, E. Baseline Implementations · p. 11
  • DSRL uses its chunk-10 configuration: it samples noise for ten steps, repeats the final noise vector to fill the 50-step chunk, and executes the first ten actions.It acts on the same frozen, fine-tuned VLA.Source: E. Baseline Implementations · p. 10
  • DAgger falls below both BEE and its Base Policy on phone charging despite receiving 20 additional online episodes instead of seed corrections.The authors report that aggregating and imitating operator corrections degrades the policy it was intended to improve.Source: E. Baseline Implementations · p. 10
  • SiLRI and HIL-SERL require operator control for almost the entire episode at this budget and make no progress without continuous intervention.Both train task-specific visuomotor policies from scratch; the authors state that tens of episodes are far below what these roughly minute-long tasks require.Source: E. Baseline Implementations · p. 11

F. Training Dynamics of the State-Dependent Multiplier

  • The state-dependent multiplier stays near zero where policy actions agree with corrections and grows where the constraint is active; violation remains nonzero as new corrected states arrive.The per-dimension constraint allocation seen in a snapshot persists throughout online training.Source: F. Training Dynamics of the… · p. 11
Loading 2609.27450v1…