PaperlayerView exact revision on arXiv

Source-linked AI summary

eRLT: Efficient VLA Reinforcement Learning via Action-Relevant Token Routing

Dehao Huang, Jianbang Liu, Jianpan Gao, Chao Tang, Zilang Cen, Zedong Dan, Jiaheng Wang, Tingguang Li, Yue Wang, Hong Zhang

arXiv:2610.00913v1cs.RO

TL;DR

Online RL adaptation of frozen VLAs depends on state representations that retain task-specific action-relevant information, which existing independent encoders and fixed compression do not explicitly extract. eRLT routes information across VLA tokens and layers, initializing the routing module with expert-action prediction and refining it with critic feedback. It improves learning-curve AUC across simulation benchmarks and two real-robot insertion tasks.

  • Problem

    Existing VLA-independent encoders and fixed compression of internal VLA features do not explicitly extract task-specific information needed for action refinement and action-value estimation.

    Source: §1 · p. 2

  • Method

    eRLT forms a fixed-dimensional RL token by routing information across frozen-VLA tokens and layers, initializing with expert-action prediction and refining the router with critic feedback online.

    Source: §1 · p. 2

  • Results

    Across LIBERO and RoboTwin, eRLT outperforms RLT on all seven tasks and DSRL on six of seven; across the seven simulation tasks, mean normalized learning-curve AUC is 23.7% higher than fixed VLA feature-compression baselines.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    Real-robot insertion results show improved sample efficiency, and USB connector insertion reaches a higher final-window success rate under a limited physical-interaction budget.

    Source: §6 · p. 10

    The real-world evaluation covers two insertion tasks on one robot under a human-in-the-loop protocol, with run-to-run variation and intervention-frequency sensitivity insufficiently characterized.

    Source: §6 · p. 10

Abstract

from arXiv · show

Vision-Language-Action (VLA) models provide strong behavioral priors for robotic manipulation, yet efficiently adapting them to downstream tasks remains challenging. Recent work addresses this challenge by adapting frozen VLAs through online reinforcement learning (RL), whose sample efficiency depends on the quality of the state representation used by the actor and critic. Existing methods construct such representations either with VLA-independent visual encoders or through fixed compression of internal VLA representations. Neither design explicitly extracts the task-specific action-relevant VLA features most useful for downstream action refinement and action-value estimation, therefore limiting sample efficiency. To address this limitation, we introduce eRLT, which constructs an effective state representation by routing task-specific action-relevant information across both tokens and layers of the frozen VLA. Specifically, learned routing tokens dynamically aggregate visual-language features at multiple depths, while a lightweight layer router combines these summaries into a fixed-dimensional RL token. The routing module is initialized using expert demonstrations to capture features predictive of expert actions and then refined using critic feedback from online interactions for action-value estimation. Across seven LIBERO and RoboTwin tasks, eRLT improves mean normalized learning-curve AUC by up to 23.7% over representative baselines. Real-robot experiments on USB connector insertion and motherboard ribbon-cable insertion further show AUC improvements of 108.9% and 46.7%, respectively, over the strongest baseline.

1 INTRODUCTION

Lightweight online RL adapts frozen VLAs without collecting additional task-specific demonstrations, but its sample efficiency depends on the actor and critic receiving useful task-specific state representations. eRLT routes action-relevant information across VLA tokens and layers, and improves learning-curve AUC across simulation and real-robot tasks.

Motivation

  • Online RL can learn from policy-distribution experience, including successful and failed interactions, while keeping the pretrained VLA frozen and adapting behavior through a small actor-critic.By contrast, supervised fine-tuning relies heavily on additional high-quality task-specific demonstrations with inherently limited coverage of deployment states.Source: §1 · p. 1
  • VLA-independent visual encoders and fixed compression of predetermined VLA layers and tokens do not explicitly extract the task-specific features most useful for action refinement and action-value estimation.Those useful features vary across tasks and are distributed across VLA tokens and layers.Source: §1 · pp. 1–2

Approach

  • eRLT routes task-specific action-relevant information across frozen-VLA tokens and layers, using learned routing tokens and a lightweight layer router to form a fixed-dimensional RL token.Expert-action prediction initializes the routing module, which is then refined using the critic’s Bellman objective during online RL; the VLA remains frozen.Source: §1 · p. 2

Evaluation

  • Across seven simulation tasks, eRLT improves mean normalized learning-curve AUC by 23.7% over fixed VLA feature-compression baselines and 7.1% over VLA-independent state encoders.On USB connector insertion and motherboard ribbon-cable insertion, AUC improves by 108.9% and 46.7%, respectively, over the strongest baseline.Source: §1 · p. 2

2 RELATED WORK

Prior lightweight VLA online-RL methods vary in whether they construct state representations independently of the VLA or compress fixed internal features. eRLT draws on distributed internal representations and combines token- and layer-level aggregation for RL state construction.

2.1 LIGHTWEIGHT ONLINE REINFORCEMENT LEARNING FOR VLA MODELS

  • Lightweight VLA online-RL methods keep the pretrained policy frozen and train a lightweight actor to correct actions or steer generation, sometimes with human guidance or improved initialization.eRLT adopts this frozen-VLA actor-critic framework and focuses on the state representation supplied to the actor and critic.Source: §2.1 · p. 3
  • Existing state representations use VLA-independent visual encoders or compress predetermined VLA layers and tokens, leaving no mechanism to select task-relevant internal information for action refinement and value estimation.DSRL and PLD use independent encoders; Mi-DAS mean-pools internal VLM tokens, while RLT compresses final-layer tokens using reconstruction.Source: §2.1 · p. 3

2.2 INTERNAL VLA REPRESENTATIONS AND FEATURE AGGREGATION

  • Prior findings describe complementary representations across layers and selective token aggregation that can improve action generation; eRLT uses frozen internal VLA representations as its RL-token source.eRLT adapts within-layer feature aggregation and across-depth representation combination to build a fixed-dimensional token for the actor and critic.Source: §2.2 · p. 3

3 PRELIMINARIES AND PROBLEM FORMULATION

The formulation keeps the pretrained VLA as a frozen action prior while lightweight trainable modules learn from online interactions. This work focuses on constructing the RL state from task-relevant information distributed across the VLA’s tokens and layers.

3.1 LIGHTWEIGHT ONLINE RL WITH A FROZEN VLA

  • At each policy step, the robot receives camera observation(s), proprioceptive state, and a language instruction; the frozen VLA and trainable actor jointly produce an executable action chunk.The VLA remains the action prior and generator while lightweight modules such as actor and critic are trained around it.Source: §3.1 · p. 3
  • Controlled RLT comparisons show that the best source layer differs across Drawer Opening and Bowl-to-Plate when all tokens are retained.The figure reports mean normalized success-rate AUC over three seeds.Source: Figure 2 · p. 4
  • At the final layer, selected token subsets outperform all tokens on Drawer Opening and Bowl-to-Plate, with a different subset best for each task.The layer and token comparisons vary one choice at a time using the same actor-critic architecture and online RL procedure.Source: Figure 2 · p. 4
  • Online training alternates environment interaction with actor-critic updates, storing rewards, next observations, and termination indicators in a replay buffer for repeated off-policy learning.The critic estimates long-term return, and the actor favors decisions assigned higher value; the VLA stays unchanged.Source: §3.1 · p. 4

3.2 RL STATE REPRESENTATION

  • The actor and critic use a state representation from the frozen VLA’s multi-layer token representations, with separate adapters combining it with proprioception.The experiments hold the VLA, RL algorithm, and action-generation procedure fixed while varying RL-token construction and training.Source: §3.2 · p. 4
  • The lightweight controller is determined by the chosen online RL algorithm, while this work isolates the shared state representation supplied to the actor and critic.The formulation leaves controller construction to the algorithm.Source: §3.2 · p. 4, §3.1 · p. 4

4 METHOD

eRLT learns task-specific representations by routing information across VLA tokens and layers, using expert-action prediction for initialization and critic feedback for online adaptation.

4.1 MOTIVATING STUDY: LAYER AND TOKEN CHOICES

  • In controlled RLT comparisons, best layers improve Drawer Opening and Bowl-to-Plate by 2.8 and 2.5 points, while task-best token subsets improve them by 3.9 and 5.8 points over all tokens.The study holds the frozen VLA, actor-critic architecture, online RL procedure, and interaction budget fixed while varying the supplied features.Source: §4.1 · p. 4

4.2 OVERVIEW

  • Expert action prediction initializes the routing representation, and critic feedback from online transitions then adapts it for action-value estimation while the VLA remains frozen.The expert-prediction stage uses existing demonstrations; online transitions include successful and failed outcomes.Source: §4.2 · p. 5

4.3 TASK-SPECIFIC ACTION-RELEVANT MULTI-LAYER TOKEN ROUTING

  • Learned routing tokens gather input-dependent multimodal features at selected VLA layers, whose mean-pooled summaries are combined with task-shared layer weights into one RL token.Token routing selects cues for the current input, while layer routing selects which processing depths contribute.Source: §4.3 · pp. 5–6, §4.2 · p. 5

4.4 TASK-SPECIFIC ACTION-RELEVANT INITIALIZATION

  • Before online interaction, an action probe predicts expert action chunks from the RL token, training it to preserve distinctions needed for action refinement.The probe is removed before online RL, and routing parameters are selected by validation action-prediction error.Source: §4.4 · p. 6

4.5 CRITIC-DRIVEN ONLINE ADAPTATION

  • During online RL, critic error updates routing parameters so the RL token exposes cues that distinguish transitions with different progress or outcomes.Actor updates use the current RL token but detach it, so they do not change how the token is constructed.Source: §4.5 · p. 6

5 EXPERIMENTS

The experiments compare eRLT with alternative state representations across simulation and real-world manipulation, measure online sample efficiency, and test routing components. eRLT achieves the strongest aggregate simulation performance and improves learning-curve AUC on both physical insertion tasks.

5.1 EXPERIMENTAL SETUP

  • Across four LIBERO and three RoboTwin tasks plus two real-world insertion tasks, variants shared the VLA and online RL procedure while only the state representation varied.Normalized learning-curve AUC is the primary sample-efficiency measure; real-world evaluation also reports final-window success.Source: §5.1 · p. 7

5.3 ABLATION STUDIES

  • Removing any routing or training component reduces Bowl-to-Plate AUC, with the largest degradation when token-wise routing is removed; more routing tokens do not improve performance monotonically.Offline initialization is the next most influential component, while layer-wise routing and online adaptation provide complementary refinement; a single routing token is the default.Source: §5.3 · p. 8, Table 3 · p. 9

5.2 MAIN SIMULATION RESULTS

  • eRLT achieves the strongest aggregate performance across seven simulation tasks, outperforming RLT* on every task and DSRL* on six of seven despite similar eventual success rates.The gains are in normalized learning-curve AUC; Table 1 summarizes task and aggregate results.Source: §5.2 · pp. 7–8, Table 1 · p. 8

5.4 REAL-WORLD EXPERIMENTS

  • On USB connector insertion, eRLT achieves AUC 0.562, exceeding RLT by 0.293 (108.9% relative), and raises final-window success from 50% to 90%.Table 4 reports normalized AUC and final-window success under a fixed interaction budget.Source: §5.4 · p. 8, Table 4 · p. 9
  • On ribbon-cable insertion, eRLT increases AUC from 0.484 to 0.710, an absolute gain of 0.226 (46.7% relative), while both eRLT and RLT reach 100% final-window success.Table 4 reports normalized AUC and final-window success under a fixed interaction budget.Source: Table 4 · p. 9, §5.4 · p. 9
  • In USB insertion, eRLT more consistently corrects offsets before contact, while its remaining failures still involve fine alignment near the interface.RLT often retains small offsets that prevent insertion; DSRL* produces less stable corrections and larger positional or angular errors.Source: §5.4 · p. 9

6 CONCLUSION AND LIMITATIONS

eRLT improves simulation and real-robot learning-curve AUC over comparison methods, while its frozen-VLA representation and real-world evaluation impose stated limits. Lightweight online RL trains the actor and critic around a frozen VLA, with the RL token supplying task-specific action-relevant information.

6 CONCLUSION AND LIMITATIONS

  • Across LIBERO and RoboTwin, eRLT beats RLT on all seven tasks and DSRL on six of seven in normalized learning-curve AUC.The real-world insertion experiments also show improved sample efficiency; USB insertion has a higher final-window success rate under a limited physical-interaction budget.Source: §6 · p. 9
  • Real-world insertion results show improved sample efficiency and higher final-window success on USB insertion under a limited physical-interaction budget.The evaluation covers USB connector and motherboard ribbon-cable insertion.Source: §6 · p. 10
  • eRLT is limited by the frozen VLA’s encoded information, and multi-layer extraction with periodic token recomputation adds computation and memory versus a fixed final-layer token.Source: §6 · p. 10

A LIGHTWEIGHT ONLINE RL WITH A FROZEN VLA

  • The frozen VLA and lightweight actor jointly generate action chunks; online interactions train the actor and critic, with the RL token supplying task-specific action-relevant information.The pretrained VLA remains fixed during replay-based off-policy actor-critic updates.Source: Figure 6 · p. 13, A LIGHTWEIGHT ONLINE RL WITH A FROZEN… · p. 13

B.1 ROUTING ARCHITECTURE

eRLT appends learned routing tokens to the frozen VLM’s input, pools their hidden states across selected layers, and combines the summaries into an RL token. Demonstrations initialize the routing parameters for action prediction; online critic feedback then adapts them for value estimation.

B.1 ROUTING ARCHITECTURE

  • At selected VLM layers, routing tokens attend to the original visual-language tokens; mean pooling yields layer summaries that a router combines into one RL token.Original prefix tokens cannot attend to the appended routing tokens, and the module adds KD + m effective trainable parameters.Source: §B.1 · p. 13

B.2 OFFLINE ACTION-RELEVANT INITIALIZATION

  • Offline initialization trains routing tokens and layer logits to predict expert action chunks, keeps VLA parameters frozen, and discards the temporary action probe afterward.It retains routing parameters with the lowest validation action MSE.Source: §B.2 · pp. 13–14

B.3 CRITIC-DRIVEN ONLINE ADAPTATION

  • Online critic loss updates the critic and routing parameters, with routing gradients enabled every U learner iterations; actor updates use detached RL tokens.Replay stores raw observations because the RL token changes online.Source: §B.3 · p. 14

C.1 SIMULATION BENCHMARKS

Simulation evaluation covers four LIBERO manipulation tasks and three RoboTwin tasks. The benchmarks use different episode horizons and share a success-based reward definition.

C.1 SIMULATION BENCHMARKS

  • The simulation suite includes Drawer Opening, Bowl-to-Plate, Cabinet-Bowl-to-Plate, and Two-Moka-Pots in LIBERO, plus Adjust Bottle, Move Can–Pot, and Place Container Plate in RoboTwin.Source: §C.1 · p. 14, Figure 7 · p. 15
  • LIBERO evaluation uses 100 deterministic environments over a 50-state pool and caps episodes at 240 low-level steps; RoboTwin horizons are 200 or 150 steps by task.The reward is −1 on non-success steps and 0 on success; success rate records whether the task predicate becomes true before its task-specific horizon.Source: §C.1 · p. 14

C.2 SIMULATION IMPLEMENTATION AND TRAINING PROTOCOL

Simulation comparisons hold the frozen VLA and online RL procedure fixed while varying state representations. The implementation uses demonstration-based routing initialization, a shared actor-critic design, and periodic critic-driven routing updates.

C.2 SIMULATION IMPLEMENTATION AND TRAINING PROTOCOL

  • All simulation variants share the VLA checkpoint, observations, actor and critic architectures, replay data, rewards, update schedule, and interaction budget.LIBERO uses a 10-action VLA chunk, while RoboTwin uses a 32-step, 14-dimensional action sequence; both actor policies act in a 32-dimensional latent space.Source: §C.2 · p. 15
  • The routing configuration uses one token and hidden-state indices {0, 3, 6, 9, 12, 15, 18}, yielding 2,055 active parameters out of 2,112 allocated.Source: §C.2 · p. 16
  • LIBERO initialization used 6,300 training and 332 validation frames; RoboTwin used 62,799 training and 7,008 validation frames, retaining the lowest-validation-MSE routing parameters.Source: §C.2 · p. 16
  • The actor maps the 2,048-dimensional RL token through 1,024-, 512-, and 256-wide blocks to a 32-dimensional latent action; ten critic heads each output a scalar.Bellman targets and actor updates use the mean of the ten heads.Source: §C.2 · p. 16
  • The Bellman target uses the first low-level reward in each chunk and discounts the bootstrap across the full chunk; Q-heads minimize MSE and the actor minimizes negative mean Q.No entropy term is added to either objective.Source: §C.2 · pp. 16–17
  • Training uses 16 parallel environments, begins at online epoch 9, and performs 200 replay updates per subsequent epoch; routing parameters update on the final replay update.The complete target model is Polyak-updated after every replay update.Source: §C.2 · p. 17

C.3 REAL-WORLD TASKS AND HARDWARE

The real-world evaluation tests USB connector and motherboard ribbon-cable insertion with a wrist-camera-equipped robot. Episodes start with the connector grasped, and success receives a sparse terminal reward.

C.3 REAL-WORLD TASKS AND HARDWARE

  • The USB receptacle is upright on a power-strip fixture, while the motherboard is secured flat on the work surface.Both tasks use the robot’s left arm and two wrist-mounted RGB cameras; the head-camera slot is masked.Source: §C.3 · p. 17
  • Each episode begins with the connector already grasped, and a rollout typically lasts about 20 seconds, extending to 30 seconds when human assistance is needed.The operator declares a timeout after approximately 1.5 times the usual task-completion time.Source: §C.3 · p. 17
  • At reset, each of seven left-arm joint targets receives independent zero-mean Gaussian noise with a 0.01 rad standard deviation.The perturbation varies the initial end-effector position and orientation; the torso and right arm remain unchanged.Source: §C.3 · p. 17
  • Success requires insertion into the task’s connector; only successful episodes receive a terminal reward of one, while failures receive zero.The operator labels episodes success, failure, or truncation; truncations are recorded separately.Source: §C.3 · p. 17

C.4 REAL-WORLD IMPLEMENTATION AND TRAINING PROTOCOL

The real-world protocol holds the VLA checkpoint, observations, actor-critic architecture, replay data, rewards, update schedule, and interaction budget constant while varying state representation. It also specifies separate motivating-study comparisons and the evaluation-curve calculation.

C.4 REAL-WORLD IMPLEMENTATION AND TRAINING PROTOCOL

  • All adapted policies share the task-specific VLA checkpoint, observations, actor-critic architectures, replay data, reward, update schedule, and interaction budget; only state representation varies.The comparison includes DSRL∗, RLT, and eRLT, with the frozen VLA reported as a reference; RLT uses the original RL token.Source: §C.4 · p. 18
  • RLT and eRLT are initialized separately from task demonstrations, and eRLT’s routing module is trained with an expert-action probe that is discarded afterward.Both task-specific π0.5 checkpoints remain frozen during online RL; the actor uses reference actions, the RL token, and proprioception, while the critic evaluates candidate action chunks. Training stages behavior cloning before increasing the Q-objective weight.Source: §C.4 · p. 18
  • Real-robot runs use fixed budgets of 80 USB or 90 ribbon-cable trajectories, including about 15 successful human-assisted warm-up trajectories on average.Five consecutive autonomous failures trigger five additional operator-provided successful episodes; the number of assisted trajectories therefore depends on policy performance.Source: §C.4 · p. 19

C.5 MOTIVATING-STUDY PROTOCOL

  • The motivating study varies one RLT factor at a time: source layer with all tokens retained, or token subset with the final layer retained.It compares all 712 positions with 328-token subsets retaining all language positions and 128 image positions, including uniform and action-relevance selections; variants remain frozen online.Source: §C.5 · p. 19

C.6 EVALUATION PROTOCOL AND METRICS

  • Learning-curve AUC uses trapezoidal integration of raw evaluation points, with no extrapolation before the first or after the last point.Multiseed results average per-run AUCs; real-world curves use trailing success over autonomous episodes against the fixed trajectory budget.Source: §C.6 · p. 19

D FURTHER ANALYSIS OF REAL-WORLD ONLINE ADAPTATION

In real-world adaptation, eRLT records more autonomous successes and requires fewer assisted trajectories than the compared methods. Optimization traces show aligned critic behavior and the intended staged actor-objective changes.

D.1 AUTONOMOUS AND ASSISTED TRAJECTORIES

  • On USB insertion, eRLT records 46 successes in 66 autonomous trajectories and 14 assisted trajectories, versus DSRL∗’s 20/57 and 23, and RLT’s 18/44 and 36.Successes are counted over autonomous trajectories only.Source: §D.1 · p. 19, Table 9 · p. 20
  • On ribbon-cable insertion, eRLT records 63 successes in 70 autonomous trajectories and 20 assisted trajectories, versus RLT’s 44/65 and 25, and DSRL∗’s 19/53 and 37.The common intervention rule requests assistance after five consecutive autonomous failures.Source: §D.1 · pp. 19–20, Table 9 · p. 20

D.2 ONLINE OPTIMIZATION DYNAMICS

  • Across six runs, critic heads remain closely aligned and policy/replay Q estimates follow similar trends; the behavior-cloning term decreases early as the Q term activates later.Figures 8 and 9 show losses, Q estimates, and weighted actor-objective terms against collected trajectories, with raw curves and trailing-five-trajectory means.Source: §D.2 · p. 20, Figure 8 · p. 21, Figure 9 · p. 21

Related papers

Loading 2610.00913v1…