PaperlayerView exact revision on arXiv

Source-linked AI summary

VLA-Precision: Asymmetric Co-Bootstrapping for Efficient Real-World Online RL of Vision-Language-Action Models

Chenyu Su, Zhaolong Shen, Yuan Qian, Chen Qian, Rui Zhang, Feng Yan, Weixing Chen, Fei Zhang, Jiamin Wang, Shuang Cong, Weiwei Shang

arXiv:2609.04355v3cs.ROcs.AIcs.HCcs.LG

TL;DR

Pretrained VLAs can struggle with precision, while real-world online RL faces unreliable value signals and large-model overhead. VLA-Precision pairs cross-timescale behavioral learning and value calibration with an efficient streaming architecture, achieving 98.3% mean success across nine chemistry tasks.

  • Problem

    Demonstrations can limit VLA performance, while real-world online RL faces value-estimation errors that risk policy drift and computational overhead that limits large-VLA throughput.

    Source: §I · p. 2, §V · p. 15

  • Method

    ACoB couples intervention-guided behavioral learning with progressive global-return and local-preference value calibration, while ACoB-Stream manages experience and policy state through invariant-state decoupling and on-demand streaming.

    Source: §I · p. 2

  • Results

    98.3% mean success was achieved across nine high-precision chemistry tasks in 45.8 min/task, with 27.6 s episodes at 1.2× and 1.8× the speeds of VLA and RL baselines.

    Source: §I · p. 2

  • Takeaways & Limitations

    Real-world RL can improve an already capable VLA while preserving its task-specific competence in high-precision tasks.

    Source: §V · p. 15

Abstract

from arXiv · show

Pretrained vision-language-action (VLA) models enable broad manipulation but remain unreliable in tasks demanding precision and repeatability. Applying real-world online reinforcement learning (RL) to VLA post-training enables autonomous trial-and-error improvement beyond demonstrations alone, but exposes two bottlenecks: 1) unreliable value signals can induce policy drift; 2) large-VLA overhead constrains throughput and sample efficiency. To address these challenges, we present VLA-Precision, an efficient real-world online RL framework featuring the Asymmetric Co-Bootstrapping (ACoB) algorithm and the ACoB-Stream architecture. Specifically, ACoB establishes asymmetric co-bootstrapping across timescales: early intervention-guided behavioral learning rapidly improves policy performance while enhancing online experience quality. As autonomous experience accumulates, global return propagation and local preference ranking progressively calibrate value estimates, yielding relative action advantages for reference-regularized policy improvement while suppressing drift. To enable ACoB on large VLAs, we develop ACoB-Stream, a closed-loop experience--policy architecture that establishes invariant-state decoupling and on-demand streaming as design principles, delivering up to 10.9$\times$ improvements in throughput and computational efficiency. Extensive evaluations on nine high-precision chemistry tasks across four categories and four robot embodiments show that VLA-Precision achieves 98.3\% mean success rate in 45.8 min/task, with 27.6 s episodes running at 1.2$\times$ and 1.8$\times$ the speeds of VLA and RL baselines. Resources are available at https://vla-precision.github.io.

I. INTRODUCTION

High-precision VLA tasks expose the limits of demonstrations and the risks of unreliable value estimates and large-model training overhead. VLA-Precision pairs cross-timescale value calibration with an efficient online-training architecture, against a backdrop of prior simulation, real-world, and systems approaches.

I. INTRODUCTION

  • ACoB couples rapid behavioral learning with progressive value calibration to suppress policy drift, while ACoB-Stream reduces the overhead of online RL for large VLAs.The introduction frames these methods against demonstration-limited precision, value-estimation errors, and replay-intensive computation.Source: §I · pp. 1–2
  • 98.3% mean success across nine chemistry tasks, with 45.8 min/task and 27.6 s episodes at 1.2× and 1.8× the speeds of VLA and RL baselines.Evaluations covered four chemistry-task categories and four robotic platforms.Source: §I · p. 2

A. Simulation-Based RL for VLA Post-Training

  • Simulation-based VLA RL offers scalable parallel interaction, but sim-to-real gaps in visual observations and contact dynamics limit reliable transfer to precision manipulation.Prior work also explores physical transfer, while real-world RL directly adapts to physical feedback.Source: A. Simulation-Based RL for VLA… · pp. 2–3

B. Real-World RL for VLA Post-Training

  • Real-world RL adapts policies to deployed sensing and dynamics, but prior compact-policy and large-VLA methods trade off success, policy stability, feedback efficiency, or rollout requirements.Examples include Q-maximization-related drift, progress-model OOD hallucinations, and high rollout counts.Source: B. Real-World RL for VLA Post-Training · p. 3

C. Efficient Online RL Systems for VLA Models

  • Existing systems improve simulation throughput or coordinate real-world workloads, but large-VLA computation, state management, and synchronization remain costly in a closed-loop online setting.Some methods externalize reward-driven updates into auxiliary policies rather than accumulating them within one unified policy.Source: C. Efficient Online RL Systems for VLA… · p. 3

III. METHOD

VLA-Precision initializes real-world online RL from a demonstration-trained policy prior and formulates post-training as a closed-loop experience–policy optimization process. An asynchronous actor–learner loop generates experience and deploys action-expert updates.

III. METHOD

  • The method presents ACoB for policy improvement and drift suppression, ACoB-Stream for a closed-loop experience–policy stream, and an end-to-end training and deployment protocol.The stream manages experience-context formation, persistence, access, and policy-state synchronization.Source: §III · p. 3

A. VLA-Precision Problem Formulation

  • Stage I uses full-parameter imitation learning on demonstrations to establish a task-specific policy prior; Stage II initializes online RL from that prior.The Stage II loop is asynchronous: actors generate trajectories and experience, while learners optimize and redeploy action-expert updates.Source: A. VLA-Precision Problem Formulation · pp. 3–4, Figure 2 · p. 4
  • Physical interaction is modeled as an MDP in which each VLA action is an H-step action chunk that induces the next state and cumulative reward.The task-conditioned policy maps states to action-chunk distributions; online optimization trains action-expert LoRA parameters while freezing the multimodal prefix and inherited action-expert parameters.Source: A. VLA-Precision Problem Formulation · p. 4

B. Asymmetric Co-Bootstrapping for the Action Expert

ACoB couples rapid behavioral improvement with progressively calibrated values so that relative action advantages can guide policy updates. Its critic combines long-horizon return learning with local preference ranking to address different credit-assignment needs.

B. Asymmetric Co-Bootstrapping for the Action Expert

  • ACoB couples fast behavioral learning, which improves the action expert and online experience, with critic calibration as autonomous experience accumulates.The resulting relative action advantages guide further action-expert improvement.Source: B. Asymmetric Co-Bootstrapping for the… · p. 4

1) Progressive Value Calibration:

  • Progressive value calibration combines global return propagation and local preference ranking to provide long-horizon estimates and finer-grained policy-improvement signals.The critic uses an ensemble of task-specific critics with a value–advantage decomposition.Source: 1) Progressive Value Calibration · pp. 4–5
  • Temporal-difference optimization recursively propagates long-horizon returns along executed trajectories.The TD objective uses transitions from the executed action, whether autonomous or human-corrected.Source: 1) Progressive Value Calibration · p. 5
  • State-matched ranking places an effective human correction above the original proposal, which is excluded from TD learning when it lacks an observed successor.The ranking loss acts on action-dependent advantages, addressing local credit ambiguity that long-horizon TD propagation cannot resolve.Source: 1) Progressive Value Calibration · p. 5

2) Relative-Advantage Policy Improvement:

ACoB compares actions by relative advantage rather than pursuing absolute critic values, then combines early intervention-guided behavioral learning with progressive value calibration and reference regularization. This design is intended to improve actions while limiting value-induced policy drift.

2) Relative-Advantage Policy Improvement:

C. ACoB-Stream Architecture

Large-VLA costs in computation, memory, I/O, and synchronization can delay new experience from affecting the deployed policy. ACoB-Stream organizes experience and policy state lifecycles around invariant-state decoupling and on-demand streaming.

C. ACoB-Stream Architecture

  • Large-VLA forward passes, context storage and access, and full-model synchronization add costs that can prevent new experience from promptly affecting the deployed policy.Source: C. ACoB-Stream Architecture · pp. 6–7
  • ACoB-Stream decouples invariant state and streams on demand, managing experience-context and policy state through formation, persistence, access, and synchronization.Source: C. ACoB-Stream Architecture · p. 7
  • The architecture links actor and learner through a closed-loop stream spanning four lifecycle stages for experience-context and policy state.Source: C. ACoB-Stream Architecture · p. 7

1) Experience-Context Formation:

ACoB-Stream reuses frozen-prefix contexts, manages their persistence and retrieval, and synchronizes only trainable policy state. The accompanying real-world collection setup combines teleoperation interfaces with a bounded task-space action representation.

1) Experience-Context Formation:

  • Frozen-prefix contexts are cached across updates and referenced by transition identifiers, reducing prefix computation from once per sampled item to once per new observation.Source: 1) Experience-Context Formation · p. 7

2) Experience-Context Persistence:

  • Deduplicated disk-backed context storage and page-cache-bounded sliding-window replay reduce redundant storage and cold-disk access while retaining full history for checkpoint recovery.Source: 2) Experience-Context Persistence · p. 7

3) Experience-Context Access:

  • Value optimization retrieves successor contexts, while action-expert optimization also retrieves current contexts; background batch preparation overlaps with network updates.Source: 3) Experience-Context Access · p. 7

4) Policy-State Synchronization:

  • Asynchronous synchronization publishes the trainable action-expert state instead of the full VLA, while rollouts continue on the last valid policy during preparation.Source: 4) Policy-State Synchronization · p. 7

D. Real-World Training and Deployment

1) Teleoperation and Data Collection:

  • Isomorphic master teleoperation supports long sequences, while keyboard increments suit fine adjustments; repeated keystrokes are cumbersome for long sequences or large reorientations.Source: 1) Teleoperation and Data Collection · pp. 7–8
  • The six-DoF isomorphic master arm follows the slave and maps operator input for takeover; graded actuation and a load-decoupled structure address following, resistance, backlash, and wear challenges.Source: 1) Teleoperation and Data Collection · p. 8
  • Step-wise delta task-space actions use bounded end-effector increments; recorded state includes TCP pose and twist, force and torque feedback, and gripper state.Source: 1) Teleoperation and Data Collection · p. 8

2) Two-Stage Real-World Training:

VLA-Precision uses demonstration-driven fine-tuning followed by online refinement of the action expert. Its real-world chemistry task suite spans multiple manipulation categories.

2) Two-Stage Real-World Training:

3) Robot Deployment:

Robot-specific Cartesian impedance control converts desired end-effector poses into force or torque commands. UR5e uses force-mode wrench control, while Franka uses torque-level control.

3) Robot Deployment:

  • Robot-specific impedance controllers are used for safe, high-precision execution under physical contact.UR5e uses force-mode wrench control; Franka uses torque-level control.Source: 3) Robot Deployment · p. 9
  • UR5e force-mode control computes a Cartesian PD wrench and applies it through the forceMode interface.The law converts position and velocity errors into force, and rotation error and angular velocity into moment.Source: 3) Robot Deployment · p. 9
  • Franka torque-mode control uses a Cartesian impedance law to produce a seven-joint torque command.The command combines task-space pose control, nullspace posture regulation, and Coriolis compensation.Source: 3) Robot Deployment · p. 9

IV. EXPERIMENTS

The experiments evaluate VLA-Precision on chemistry manipulation tasks using four robot platforms and comparisons with real-world RL and pretrained VLA baselines. The setup spans varied precision demands and reports both training dynamics and test outcomes.

A. Experimental Setup

  • Four platforms include two single-arm UR5e gripper configurations, a dual-UR5e gripper setup, and a Franka Research 3 gripper setup.Single-arm platforms use one wrist and one external camera; the dual-arm platform uses two wrist cameras and one external camera.Source: A. Experimental Setup · p. 9, Figure 6 · p. 10
  • The baselines comprise HIL-SERL, ConRFT, Robo-Dopamine, and fine-tuned π0 and π0.5 VLAs.The large-VLA baselines are fine-tuned without Stage-II online RL.Source: A. Experimental Setup · p. 10
  • The evaluation covers nine chemistry tasks across contact-rich, contact-light, contact-free, and bimanual categories, with positional tolerances ranging from submillimeter contact tasks to ±3.25 mm bimanual alignment.The tasks vary object poses and initial arm poses to assess generalization.Source: A. Experimental Setup · pp. 10–11
  • Training dynamics track rolling autonomous and effective success, cumulative interventions, and intervention rate; test results report success rates and successful-trial completion times.The training metrics use 20-episode rolling means; test success rates use 20 trials per method-task pair and Wilson 95% confidence intervals.Source: Figure 7 · p. 11, Figure 8 · p. 12

B. Real-World Evaluation of VLA-Precision

Across nine chemistry manipulation tasks, VLA-Precision outperforms the listed baselines in success and execution speed. Its online training results and comparisons on representative tasks also show stronger autonomous performance than the baselines.

Overall Performance

  • 98.3% average success after 45.8 min of online training per task, with 27.6 s successful trials, is VLA-Precision’s reported result across nine tasks.It succeeds in 177 of 180 held-out trials, reaches 100% on seven tasks, and maintains at least 90% success on every task.Source: B. Real-World Evaluation of… · p. 12
  • 88.3% higher average success and 61.2% higher average execution speed than Robo-Dopamine are reported for VLA-Precision.Against π0.5 and π0, average success improves by 30.5% and 38.9%, respectively.Source: B. Real-World Evaluation of… · p. 12
  • HIL-SERL, ConRFT, and Robo-Dopamine average 2.8%, 7.8%, and 10.0% success, while fine-tuned π0 and π0.5 average 59.4% and 67.8%.The text attributes the online RL methods’ vulnerability to optimistic value errors and policy drift under direct absolute-Q maximization.Source: B. Real-World Evaluation of… · p. 12

Online RL Training Performance

  • Across five representative tasks, VLA-Precision averages 98.0% autonomous success and 99.0% effective success, with 2,517.4 interventions and a 0.024% intervention rate.Relative to Robo-Dopamine, autonomous and effective success rise by 92.0 and 49.5 percentage points, and intervention rate falls by 40.46 percentage points.Source: B. Real-World Evaluation of… · p. 12
  • All three baselines end at 0% autonomous success on pipette transfer and bimanual tube brushing; the analyses describe accumulated errors and absolute-Q policy drift.ACoB combines intervention-guided behavior cloning, value calibration through global returns and state-matched preferences, and relative-advantage updates with a frozen reference.Source: B. Real-World Evaluation of… · p. 13

Offline VLA Test Performance

  • On five offline-tested tasks, VLA-Precision succeeds in 99 of 100 trials, versus 61 for π0.5 and 50 for π0, and averages 26.65 s per successful trial.It reaches 100% on all four single-arm tasks and 95% on bimanual tube brushing.Source: B. Real-World Evaluation of… · p. 13

C. Throughput and Efficiency Evaluation of ACoB-Stream

ACoB-Stream improves online-RL update efficiency for large VLAs, while ACoB’s behavioral-learning, critic-preference, and relative-advantage mechanisms each contribute to autonomous success across the ablated tasks.

C. Throughput and Efficiency Evaluation of ACoB-Stream

  • 10.95× throughput and 90.9% lower mean latency versus No KV Cache; Disk ACoB-Stream reaches 1.7407 CTA/s with 0.574 s per cycle.It completes 15,666 CTA cycles in 150 min.Source: C. Throughput and Efficiency… · p. 13
  • 1.53× throughput and 34.5% lower mean latency versus Disk Full Random; 1.80× throughput and 44.4% lower mean latency versus Naive Context Access.Against Dynamic CPU RAM, Disk ACoB-Stream delivers 1.09× throughput and 8.0% lower latency.Source: C. Throughput and Efficiency… · pp. 13–14

D. Ablation Experiments for the ACoB Algorithm

  • 96.25% final autonomous success with a 0.03% intervention rate; removing critic preference, actor BC, or relative advantage lowers success to 26.25%, 20.00%, or 8.75%.Across the four tasks, interventions rise from 10,080 to 16,580, 19,581, or 28,806, respectively.Source: D. Ablation Experiments for the ACoB… · p. 14, Figure 10 · p. 14
  • Without critic preference, failed policy proposals overwritten by human corrections can remain overvalued, weakening subsequent action comparisons.The reported 70.00-point success drop is attributed to unresolved local credit ambiguity and weaker action-dependent value calibration.Source: D. Ablation Experiments for the ACoB… · pp. 14–15
  • Removing actor BC prevents successful and human-corrected actions from being directly absorbed, leaving slower critic guidance and degrading experience for value calibration.The reported success drop is 76.25 points.Source: D. Ablation Experiments for the ACoB… · p. 15
  • Direct Q maximization produces an 87.50-point success drop and a 24.93% intervention rate, exposing the actor to absolute-value scale and optimistic critic errors.The ablation summary describes actor BC as improving behavior and experience, critic preference as calibrating values, and relative advantages as stabilizing policy improvement.Source: D. Ablation Experiments for the ACoB… · p. 15

V. DISCUSSION

The discussion highlights how ACoB’s behavioral improvement and value calibration address the challenge of precise real-world RL, while ACoB-Stream manages accumulating training state. The conclusion reports high success across tasks and platforms, while the discussion identifies longer multi-stage bimanual tasks as an open boundary.

V. DISCUSSION

  • ACoB addresses the early temporal asymmetry between a capable fine-tuned policy and online experience that has not yet adequately calibrated the critic.The discussion contrasts slow conservative value-guided updates with aggressive updates that risk policy drift.Source: §V · p. 15
  • As experience accumulates, ACoB-Stream’s joint management of reusable computation, experience, and policy state is presented as sustaining update throughput under finite compute and memory.The discussion identifies redundant computation, transfer, and random access as sources of increasing update latency.Source: §V · p. 15
  • The evaluation includes one bimanual precision task, not longer sequences with multiple precision-critical stages where local errors can accumulate and degrade later operations.The discussion says longer sequences make the reference scheme for delta actions increasingly important.Source: §V · p. 15
  • Increasing the executed horizon from 3 to 6 reduced average autonomous success from 58.3% to 6.7% and increased relative action-prediction RMSE by 29.3%.The discussion motivates chunk-wise deltas for long-horizon bimanual manipulation; step-wise deltas with He = 3 are used for frequent closed-loop correction in the current precision setting.Source: §V · p. 15
  • The discussion identifies joint multi-task optimization of a shared large VLA as a direction for transferring interaction experience across related skills while reducing per-task data collection.It presents task-conditioned value learning with balanced interaction data and retained demonstrations as a feasible path to limiting interference and catastrophic forgetting.Source: §V · p. 16

VI. CONCLUSION

  • 98.3% average success in 45.8 min/task, with 27.6 s episodes running at 1.2× and 1.8× the speeds of VLA and RL baselines.Evaluations covered nine high-precision chemistry manipulation tasks and four robotic platforms.Source: §VI · p. 16

Related papers

Loading 2609.04355v3…