PaperlayerView exact revision on arXiv

Source-linked AI summary

RL Token: Bootstrapping Online RL with Vision-Language-Action Models

Charles Xu, Jost Tobias Springenberg, Michael Equi, Ali Amin, Adnan Esmail, Sergey Levine, Liyiming Ke

arXiv:2604.23073v2cs.LGcs.RO

TL;DR

Pretrained VLAs can perform diverse manipulation skills but may remain slow and unreliable during precise task phases, motivating efficient online RL fine-tuning. RLT exposes a compact RL token for a lightweight actor-critic policy, which improves success and speed across four robot tasks, with up to 3× speedups on critical phases and occasional faster-than-expert teleoperation.

  • Problem

    Pretrained VLAs can struggle with slow motions, pauses, retries, and small errors that compound during precise manipulation, motivating reinforcement-learning fine-tuning.

  • Method

    RLT adapts a pretrained VLA to expose an RL token that compresses task-relevant knowledge into an RL state for a lightweight actor-critic policy.

  • Results

    Across four challenging manipulation tasks, RLT improves success rate and execution speed, achieving up to 3× faster critical phases and sometimes surpassing expert teleoperation speed.

  • Takeaways & Limitations

    RLT enables fast online RL adaptation of pretrained VLA representations for precise and delicate robot manipulation using a lightweight policy and a few hours of real-world practice.

Abstract

from arXiv · show

Vision-language-action (VLA) models can learn to perform diverse manipulation skills "out of the box," but achieving the precision and speed that real-world tasks demand requires further fine-tuning -- for example, via reinforcement learning (RL). We introduce a lightweight method that enables sample-efficient online RL fine-tuning of pretrained VLAs using just a few hours of real-world practice. We (1) adapt the VLA to expose an "RL token," a compact readout representation that preserves task-relevant pretrained knowledge while serving as an efficient interface for online RL, and (2) train a small actor-critic head on this RL token to refine the actions, while anchoring the learned policy to the VLA. Online RL with the RL token (RLT) makes it possible to fine-tune even large VLAs with RL quickly and efficiently. Across four real-robot tasks (screw installation, zip tie fastening, charger insertion, and Ethernet insertion), RLT improves the speed on the hardest part of the task by up to 3x and raises success rates significantly within minutes to a few hours of practice. It can even surpass the speed of human teleoperation on some of the tasks.

I. INTRODUCTION

RLT addresses the tension between preserving pretrained VLA generalization and enabling rapid, sample-efficient online RL. It exposes a compact RL token for a lightweight actor-critic that refines VLA behavior, improving speed and success on precise robot tasks.

  • Motivation: VLAs often struggle with slow motions, pauses, retries, and small errors during precise task execution.These errors can compound into failure, while real-world practice is constrained by time, effort, and robot wear.
  • Problem: The central challenge is achieving the VLA’s generalization with the speed and sample efficiency of lightweight online RL.Conventional foundation-model RL uses large-scale data, whereas data-efficient real-world RL often sacrifices VLA generalization.
  • Method: RLT exposes a compact RL token that makes task-relevant pretrained knowledge accessible to a lightweight online RL policy.The frozen VLA supplies broad perceptual understanding and action recommendations, while actor and critic networks adapt difficult task phases online.
  • Results: Across four precise manipulation tasks, RLT improves success rate and execution speed within a few hours of online training.The tasks are screw installation, zip tie fastening, Ethernet insertion, and charger insertion.
  • Results: Up to 3× speedup occurs in critical task phases, with one screw-insertion task improving from 20% to 65% success.The method can also surpass expert teleoperation speed on a dexterous task part while maintaining reliability.
  • Relation to prior work: RLT differs from related VLA-RL approaches by compressing VLA embeddings, using chunked actions, and locally refining a sampled VLA action policy.These choices avoid the cost of full-model RL and shorten the effective horizon for sparse-reward online learning.

III. PRELIMINARIES

The paper formulates robot control as chunked-action reinforcement learning and trains off-policy actor-critic models from sparse-reward replay data. VLA policies produce longer action chunks, while the RL policy uses shorter chunks for greater reactivity.

  • VLA policy: The VLA produces action chunks of H = 50 actions corresponding to 1 s of control, while executing only a prefix before replanning.This chunked interface supports sequential open-loop execution from new observations.
  • RL formulation: Robot control is modeled as an MDP with state observations, continuous actions, transition dynamics, rewards, and discount factor γ.The RL objective is to maximize expected discounted return.
  • Reward: The reward is sparse and binary: a human labels episode success or failure, producing rT = 1 or rT = 0 at the end.This setting makes long-horizon credit assignment important.
  • Chunked RL: Policies and critics operate over C-step action chunks, with C < H so the RL policy can react more frequently than the VLA horizon.The critic estimates value at the chunk level over the current action sequence and subsequent policy actions.
  • Optimization: Off-policy actor-critic learning reuses replay-buffer transitions generated by the VLA, RL learner, and human interventions.The actor and critic are trained jointly from data regardless of which policy produced each transition.

IV. REINFORCEMENT LEARNING FROM RL TOKEN

RLT uses the frozen VLA to provide state representations and reference actions while training a lightweight online RL policy. This division keeps online adaptation focused on refining promising pretrained behavior.

  • Design rationale: Training the entire VLA with online RL can be too compute- and sample-inefficient for improvement within a few hours.RLT instead maximizes reuse of the pretrained VLA during RL training.
  • Division of labor: The frozen VLA supplies the RL state representation and reference actions, while lightweight RL components learn task-specific improvements.The design uses pretrained behavior as the starting point for online refinement rather than learning from scratch.

A. Adapting the VLA to expose an RL interface

The RL token is a compact learned readout of pretrained VLA embeddings that serves as the state for lightweight online RL. An encoder-decoder bottleneck is trained to retain task-relevant information while the VLA and token extractor are then frozen for RL.

  • Motivation: Direct online RL on the full VLA is computationally expensive and sample-inefficient because its representation is high-dimensional.The paper therefore seeks a compact embedding that preserves task-relevant information for lightweight actor-critic learning.
  • RL token: RLT adds an encoder-decoder transformer whose special-token output is a compressed embedding called the RL token.The token summarizes VLA knowledge into a small vector serving as the RL state.
  • Extraction: The encoder processes pretrained VLA token embeddings augmented with a learned RL-token embedding, producing zrl at the special-token position.This readout is designed to summarize the sequence of VLA representations.
  • Training objective: A decoder is trained to autoregressively reconstruct the original VLA embeddings from zrl, making the representation a bottleneck.The reconstruction objective is trained on a small task-specific demonstration dataset.
  • Online RL interface: After training, the VLA and token-extractor parameters are frozen, and online RL operates on the RL-token representation.The construction can omit language embeddings when each task uses a fixed language instruction.

B. Online RL to refine VLA action chunks

The RL actor-critic refines pretrained VLA action chunks using the RL token, proprioception, and reference actions rather than generating behavior from scratch. Training combines critic value optimization with regularization toward the VLA proposal and reference-action dropout to avoid simple copying.

  • The critic estimates action-chunk values from the RL-token state, auxiliary proprioception, and executed action chunk.
  • The actor produces a Gaussian distribution over action chunks conditioned on the input state and a VLA reference chunk.
  • Conditioning on sampled VLA actions preserves multimodal proposal information while allowing online RL to refine, rather than relearn, behavior.
  • The actor maximizes critic value while regularizing its actions toward the sampled VLA reference, with β controlling regularization strength.
  • Reference-action dropout replaces reference chunks with zeros on random transitions, preventing the actor from simply copying the VLA proposal.

V. THE COMPLETE SYSTEM

The complete system freezes the adapted VLA and RL token before lightweight online actor-critic training. It alternates asynchronous robot rollouts with replay-buffer updates, using VLA demonstrations, online data, and optional human interventions.

  • Training begins with VLA warmup episodes that initialize the replay buffer and provide the critic with competent behavior.
  • During rollouts, the frozen VLA supplies reference chunks and the RL token supplies state features for the actor’s action-chunk policy.
  • Human interventions can overwrite actor actions, and the resulting transitions retain executed actions and corresponding references for off-policy learning.
  • Intermediate observations are stored with stride 2, increasing replay data from action-chunk trajectories.
  • The replay loop samples batches for critic TD backups and performs asynchronous off-policy updates with two critic updates per actor update.
  • RLT targets each task’s high-precision critical phase while the base VLA performs easier stages, with handoff selected during collection.

VI. REAL-WORLD EXPERIMENTS

The experiments test whether RLT improves pretrained VLA manipulation under limited interaction time, sparse supervision, and lightweight online learning. They evaluate gains over the base VLA, alternative RL methods, individual components, and strategy changes.

  • The evaluation uses four real-world manipulation tasks requiring dexterous control and sub-millimeter precision.
  • The experiments examine whether RLT improves performance over the base VLA model.
  • The study compares RLT with alternative RL approaches on the same manipulation tasks.
  • Ablations assess contributions from the RL token, chunked action prediction, policy regularization, and reference-action pass-through.
  • The evaluation also tests whether RLT discovers a better strategy and compares that strategy with the original demonstration data.

A. Tasks and Setup

The study covers four long-horizon robot tasks whose success depends on a shorter, high-precision critical phase. It evaluates both isolated critical phases and full-task execution under realistic state variation.

  • Tasks: The tasks are screw installation, zip tie fastening, Ethernet insertion, and charger insertion.
  • Tasks: Screw installation requires sub-millimeter alignment and is complicated by screw orientation, amplified end-effector rotation, and difficult visual access.
  • Tasks: Zip tie fastening requires bimanual control of a deformable object and millimeter-precision insertion using wrist-camera observations.
  • Tasks: Ethernet insertion combines positional and angular alignment with a decisive insertion motion sensitive to contact dynamics.
  • Tasks: Charger insertion requires alignment despite incomplete observability of prongs and sockets, with small errors causing probing or failed insertion.
  • Evaluation setup: Tasks last 30–120 s, while critical insertion, fastening, or rotation phases last 5–20 s and contain the highest precision demands.
  • Evaluation setup: Full-task evaluation starts from the robot’s home position and includes state variation produced by earlier base-policy execution.
  • Evaluation setup: The RL policy combines RL-token features from three camera images with proprioception, using joint position or end-effector pose depending on the task.

B. Baselines and Ablations

RLT is compared with experience-based RL and imitation baselines, while ablations isolate the contributions of its RL token, action chunks, BC regularizer, and reference-action pass-through. Across the evaluated comparisons, RLT improves throughput and success relative to the base VLA and alternative methods.

  • Baselines: RLT is evaluated against HIL-SERL, PLD, DSRL, and DAgger using the same amount of training data.HIL-SERL, PLD, and DSRL are experience-based alternatives, while DAgger fine-tunes the base VLA on human intervention data.
  • Ablations: Removing action chunks makes the effective horizon much longer and prevents the single-step variant from reliably matching base-policy performance.The single-step variant also requires replacing the RL token because querying the VLA at 50 Hz is infeasible.
  • Overall comparison: RLT increases throughput and success over the base VLA, with especially pronounced improvements on harder tasks.The figures summarize gains in critical-phase speed, consistency, and success across tasks.
  • Baseline comparison: Single-step methods HIL-SERL and PLD perform poorly on the long-horizon Ethernet task, whereas DSRL achieves high success but significantly lower throughput.The comparison highlights the role of action chunks for propagating sparse rewards over a task spanning hundreds of steps.

C. Experimental Results

Across controlled critical phases and full-task evaluations, RLT improves execution speed and success over the base VLA, with larger success gains on harder tasks. Ablations and qualitative analysis indicate that the complete system learns quickly, preserves strong success, and can develop faster contact strategies.

  • Overall results: RLT improves critical-phase success and execution speed across all four tasks, including about 3× faster performance on charger and Ethernet.The full-task setting remains harder because errors from earlier stages compound before the critical phase.
  • Full-task evaluation: 40% and 60% success-rate improvements are achieved on the screwdriver and zip tie tasks, respectively, in full-task evaluation.These harder tasks show the more pronounced success-rate gains, while overall full-task rates are lowered by earlier compounding errors.
  • Baseline comparison: RLT matches the base policy’s high success rate while reducing mean steps to completion by 2× on Ethernet.Compared with baselines, DAgger and DSRL provide significantly less speed improvement, while HIL-SERL and PLD fail to learn effectively.
  • Ablations: All four design choices contribute meaningfully: the RL token, action chunks, BC regularizer, and reference-action pass-through.The full system learns fastest and performs best at the end of Ethernet training; RLT already outperforms the alternative after 5 minutes of critical-phase data.
  • Ablations: RLT quickly matches the VLA’s Ethernet success rate while increasing throughput, whereas removing the RL token or pass-through slows learning.The pass-through ablation can eventually reach the best final performance but incurs significantly more failures during training.
  • Emergent strategies: On Ethernet, RLT’s final policy is faster than expert teleoperation demonstrations and the base VLA, with half of RL episodes faster than all teleoperated demonstrations.The speed distribution compares teleoperated demonstrations, the base policy, and the final RL policy during the critical insertion phase.
  • Emergent strategies: RLT replaces the VLA’s repeated probing with fluid insertion and sometimes uses pressure and slight connector wiggling to exploit compliance.This behavior emerges from online exploration and is not present in the demonstration data.

VII. CONCLUSION

RLT enables fast online RL for precise robot manipulation by combining a compact representation from a pretrained VLA with lightweight policy refinement. Across four difficult tasks, it improves success rate and execution speed, with up to 3× speedup on the hardest phases and occasional gains over expert teleoperation.

  • RLT combines a compact representation extracted from a pretrained VLA with lightweight actor-critic networks for fast online RL.The method targets highly precise and delicate tasks using only a few hours of real-world practice.
  • Across four difficult precision tasks, RLT consistently improves both success rate and execution speed.
  • Up to 3× speedup occurs on the hardest phase of each task, with some strategies surpassing expert human teleoperation speed.
  • RLT requires additional human intervention for reward signals, intervention corrections, and switching between RL and the base policy during training.The authors identify automating these components as future work.

APPENDIX

The appendix describes the data preparation, frozen-VLA online-RL setup, action and reward design, phased training, and baseline implementations used in the experiments. It also records a setting in which HIL-SERL failed to succeed because of control-frequency and exploration-space differences.

  • Training setup: The method collects target-task demonstrations, fine-tunes the base VLA, trains the RL token for 2000 to 10000 gradient steps, and freezes the VLA during online RL.
  • RL networks: RL actor and critic networks use two-layer MLPs with hidden dimension 256, except for screw installation, which uses a three-layer MLP with hidden dimension 512.
  • Action and reward design: The actor outputs action chunks with C=10 and training subsamples chunks two control steps apart, yielding roughly 25 RL samples per second of data.
  • Action and reward design: Operators provide a sparse +1 reward when the RL task is completed.
  • Phased training: Screw installation and zip tie fastening use critical-phase training before switching from the base policy to RL within full-task episodes.This strategy improves training efficiency and exposes RL to initial distributions induced by the base policy.
  • Baselines: All baseline methods use the same environment and action-space setup, with policies executing delta actions at 50 Hz.
  • Baselines: PLD pretrains its critic on 50 base-policy rollouts with Cal-QL before online RL.
  • Baselines: DSRL predicts a (1, 32)-dimensional latent action and repeats it 50 times to match the action-chunk VLA noise space.
Loading 2604.23073v2…