PaperlayerView exact revision on arXiv

Source-linked AI summary

VLRLA: Deployment-Time Adaptation of Vision-Language-Action Models through In-Context Reinforcement Learning

Diyuan Shi, Yiqi Tang, Hongyin Zhang, You Wu, Donglin Wang

arXiv:2610.19517v1cs.AIcs.RO

TL;DR

Pretrained VLA policies lack a mechanism for gradual adaptation from accumulated deployment experience. VLRLA equips existing VLA models with in-context reinforcement learning, conditioning actions on interaction history while keeping parameters fixed. Experiments report improved task success from accumulated experience, including on unseen tasks and in real-world settings.

  • Problem

    Conventional pretrained VLA policies lack a mechanism for gradual adaptation from accumulated deployment experience, despite encountering unfamiliar viewpoints, lighting, or instruction phrasing.

    Source: §1 · p. 1

  • Method

    VLRLA equips existing VLA models with in-context reinforcement learning that conditions action generation on accumulated interactions and reward feedback while keeping model parameters fixed.

    Source: §1 · pp. 1–2

  • Results

    Across simulation and real-world settings, VLRLA improves task success using accumulated interaction experience, including on challenging unseen tasks.

    Source: §7 · p. 10

  • Takeaways & Limitations

    The general framework accommodates multiple learning-history construction strategies and policy instantiations for deployment-time adaptation.

    Source: §7 · p. 9

Abstract

from arXiv · show

Vision-language-action (VLA) models connect visual perception and language understanding with robot control, providing a foundation for instruction-conditioned manipulation. However, conventional pretrained VLA policies lack a mechanism for gradual adaptation through accumulated interaction experience during deployment. We therefore propose VLRLA, a general framework that equips existing VLA models with \textit{in-context reinforcement learning} capabilities for deployment-time adaptation. Our approach augments a VLA model with a reinforcement learning module that learns from historical observations, actions, rewards, and termination signals. During deployment, the policy gradually refines its action generation and progressively improves task success while keeping model parameters fixed. Our approach is also designed for general applicability and compatibility with diverse learning history construction strategies and model training pipelines. Finally, through extensive experiments, we systematically investigate the learning properties of VLRLA in simulation and demonstrate its ability to adapt to unseen scenarios during real-world deployment.

1 INTRODUCTION

Pretrained VLA policies can generalize, but lack a mechanism to adapt gradually from accumulated deployment experience. VLRLA adds in-context reinforcement learning to existing VLA models, aiming to improve task success without updating model parameters.

1 INTRODUCTION

  • Conventional demonstration-based VLA training does not provide a mechanism for gradual adaptation after deployment to unfamiliar viewpoints, lighting, or instruction phrasing..Source: §1 · p. 1
  • Online DRL and interactive imitation learning adapt through parameter updates, but their sample collection can take tens of minutes to hours, require task-specific tuning, and hinder multi-task scaling..Source: §1 · p. 1
  • VLRLA is designed to improve task success through continued deployment interaction while keeping model parameters fixed.It conditions action generation on accumulated interactions and reward feedback.Source: §1 · pp. 1–2
  • The framework formulates deployment-time adaptation as in-context reinforcement learning from accumulated interactions and reward feedback..Source: §1 · p. 2
  • VLRLA unifies learning-history construction and adaptation-module integration across diverse data settings, model architectures, and deployment tasks..Source: §1 · p. 2
  • Experiments in simulation and on real-world robots report adaptation across pretrained backbones and improved task success with accumulated experience, including on challenging unseen tasks..Source: §1 · p. 2

2 RELATED WORK

Prior VLA work advances generalist control, fine-tuning, and inference-time use of demonstrations or interaction context. VLRLA focuses on reward-driven in-context adaptation for existing VLA models across varied settings, including continuous robot control.

Vision-Language-Action Models

  • RT, OpenVLA, π0.5, and LingBot-VLA advance generalist control through multi-task learning, broad pretraining, open-source models, or cross-platform data and infrastructure.The cited works are RT1, 2022; Brohan et al., 2023; Collaboration et al., 2023; Kim et al., 2024; Intelligence et al., 2025b; and Wu et al., 2026.Source: §2 · p. 2

Deep Reinforcement Learning

  • PPO-based OpenVLA fine-tuning improves semantic and execution generalization over supervised fine-tuning; RLToken and DSRL perform online real-world RL fine-tuning, and these methods require parameter updates during RL fine-tuning.Source: §2 · p. 2

In-Context Learning

  • Inference-time context can guide VLA action generation through demonstrations, multimodal prompts, or interaction histories, while RoboTTT uses test-time training and AD addresses reward-driven adaptation in discrete simulation.The passage contrasts those approaches with VLRLA’s focus on a general framework for existing VLA models.Source: §2 · pp. 2–3

3 PRELIMINARIES

The preliminaries define chunked VLA action generation and an in-context RL policy that conditions actions on completed interaction history. Algorithm Distillation motivates learning how actions change with ordered experience.

Vision-Language-Action Policy

  • At each timestep, a VLA policy receives a language instruction and visual-proprioceptive observation, then generates an action chunk for the robot controller to execute before replanning.The chunk has length h and contains h actions, excluding endpoint t + h.Source: §3 · p. 3

In-Context Reinforcement Learning

  • The available history contains only completed interactions before timestep t, including observations, action chunks, rewards, and episode-boundary signals.The history is empty at the start, and the in-context policy generates actions conditioned on the current observation, instruction, and history while keeping its parameters fixed.Source: §3 · p. 3

Algorithm Distillation

  • Algorithm Distillation trains a causal sequence model on ordered learning histories from a source RL algorithm, so action generation can condition on experience in unseen tasks without parameter updates.Preserving the order of learning across episodes helps the model learn how action generation changes with experience.Source: §3 · p. 3

4 TRAINING ADAPTIVE VISION-LANGUAGE-ACTION MODELS

VLRLA adapts pretrained VLA policies by combining visual-language representations, recurrent memory, and action-generation modules. Its design supports alternative policy interfaces and history-construction strategies for training adaptation from progressively improving experience.

4.1 MODEL DESIGN

  • VLRLA is intended as a general framework for integrating adaptive VLA models across diverse settings, and its design includes distinct policy variants for simpler versus complex or real-world tasks.VLRLA-simp is intended for simple tasks or simulation; VLRLA-bridge for complex tasks or real-world deployment.Source: §4 · p. 3, Figure 1 · p. 4, §4.1 · p. 4
  • VLRLA uses recurrent memory to retain interaction information across episodes within a trial while keeping model parameters fixed during deployment.The framework combines local causal attention with cached representations from earlier windows to reduce the memory and computation costs of dot-product causal attention.Source: §4.1 · pp. 3–4
  • The framework supports both prefix-embedding and KV-cache interfaces from a pretrained encoder, with recurrent memory supplying context from earlier interactions.The KV cache exposes layer-specific attention keys and values across VLM layers for action generation.Source: §4.1 · p. 4
  • VLRLA-simp pairs a frozen visual-language encoder with a trainable recurrent Transformer and MLP that produces deterministic actions, trained with reconstruction loss.The target is the dataset action chunk.Source: §4.1 · pp. 4–5
  • VLRLA-bridge connects a pretrained visual-language encoder and stochastic action expert through a trainable module that modifies the encoder KV cache with a residual.The frozen expert generates actions conditioned on the adapted cache and proprioceptive state; training uses a flow-matching objective with gradients propagated through the expert to the bridge.Source: §4.1 · p. 5

4.2 LEARNING HISTORY CONSTRUCTION

  • Simulation RL histories store trajectories across successive online-RL rounds with domain-randomized environments, capturing behavior as task performance improves.The resulting histories train VLRLA to transfer a source RL algorithm’s adaptation behavior to unseen tasks.Source: §4.2 · p. 5
  • The human-in-the-loop curriculum orders trajectories from uncorrected policy rollouts through increasing teleoperation correction to nearly all-human demonstrations.Correction increases from the first 1/3 of each episode to 2/3 and then 3/3.Source: §4.2 · p. 5
  • Hindsight relabeling changes a trajectory’s instruction-conditioned reward and language instruction, allowing histories of progressively adaptive behavior to be assembled without additional data collection.The example reuses successful placements in either a green or yellow plate for the task of placing an adapter in one of the plates.Source: §4.2 · p. 6

5 SIMULATION EXPERIMENTS

Simulation experiments compare VLRLA with other algorithms and examine adaptation across VLM backbones and context lengths. VLRLA-bridge improves over the initial policy across seen and unseen RoboTwin tasks, while a history-reuse experiment highlights the value of continued interaction information.

5 SIMULATION EXPERIMENTS

  • The simulation study uses RoboTwin and averages success rates across 128 randomized environment instances; it uses VLRLA-bridge for the algorithm comparison and VLRLA-simp for the other questions.The other questions concern different VLM backbones, context length, and transfer of one rollout’s history.Source: §5 · p. 6

5.1 HOW DOES VLRLA COMPARE WITH OTHER ALGORITHMS?

  • Across all 12 tasks, VLRLA-bridge gains 12.4 percentage points over π0.5 and exceeds GRPO and RoboTTT in overall average success.Table 1 reports success rates on seen and unseen RoboTwin tasks; on the six seen tasks, the gain over π0.5 is 17.6 percentage points.Source: Table 1 · p. 6, §5.1 · p. 7
  • On six unseen tasks, VLRLA-bridge improves over π0.5 on every task by an average of 7.3 percentage points, reaching a 60.6% average success rate.It outperforms RoboTTT on three unseen tasks and matches it on the other three.Source: Table 1 · p. 6, §5.1 · p. 7

5.2 CAN VLRLA LEARN TO ADAPT USING DIFFERENT VLM BACKBONES?

  • Figure 2 examines adaptation across VLM modules, with shaded regions representing one standard deviation across five seeds.Source: Figure 2 · p. 7

5.3 HOW DOES CONTEXT LENGTH AFFECT VLRLA-S I M P’S ABILITY TO ADAPT?

  • Reusing only the first episode’s history yields an initial success-rate improvement, then a plateau; continued interaction-history accumulation is important for adaptation.Source: Figure 3 · p. 7, §5.3 · p. 8

6 REAL WORLD EXPERIMENTS

The real-world experiments examine history transfer and whether VLRLA can improve manipulation through accumulated experience, including fruit placement, an unseen pepper task, and cross-episode reward feedback.

6.1 FRUIT-PLACEMENT

  • The fruit-placement evaluation uses apple, starfruit, and grape with bowl and plate, then tests transfer to red pepper and metal pot, absent from training.The dataset includes about 80 teleoperation episodes for fine-tuning π0.5, followed by three groups of 20 episodes with human-in-the-loop corrections.Source: §6.1 · p. 8
  • Fruit-placement performance steadily improves with accumulated interaction history and surpasses π0.5, including on the unseen pepper-and-pot task.The pepper and pot were absent from training; the task used a medium-sized asymmetric pepper that was more challenging to grasp.Source: §6.1 · pp. 8–9, Figure 6 · p. 8
  • Apple and starfruit success rates increase steadily and exceed π0.5 at VLRLA2, while grape performance fluctuates across adaptation stages.The authors attribute the difference to grape’s small, soft form and the gripper’s relatively high force, versus the greater grasp-placement precision required for apple and starfruit.Source: §6.1 · p. 9, Table 2 · p. 9

6.2 CROSS-EPISODE MEMORY TASK

  • Success rates rise after the first episode, when reward feedback becomes available, suggesting earlier-episode rewards guide later plate choices.The instruction omits the target plate color, and reward is 1 for placing the adapter in the target plate and 0 otherwise.Source: §6.2 · p. 9, Figure 7 · p. 9

7 CONCLUSION

VLRLA adapts VLA behavior at deployment by conditioning action generation on interaction history while keeping model parameters fixed. Experiments in simulation and the real world show improved task success, including on challenging unseen tasks.

7 CONCLUSION

  • VLRLA uses accumulated interactions, rewards, and termination signals to condition action generation while keeping model parameters fixed.The framework accommodates multiple learning-history construction strategies and policy instantiations.Source: §7 · p. 9
  • Experiments in simulation and real-world settings show improved task success from accumulated experience, including on challenging unseen tasks.Simulation studies examine different pretrained backbones and continual interaction-history updates; real-world experiments show improved manipulation and reward feedback informing later choices.Source: §7 · p. 10

A.1.1 VLRLA-BRIDGE

The VLRLA-bridge implementation summarizes the base model’s cache, encodes interaction inputs with positional information, and applies learned low-rank cache modifications.

A.1.1 VLRLA-BRIDGE

  • Table S1 reports the VLRLA-bridge hyperparameters and common architecture and training settings.The implementation passage points to this table for those settings.Source: §A.1.1 · p. 14, Table S1 · p. 14
  • A learned summarization module reduces cross-attention memory costs by attending across valid cache positions and layers to produce compact KV summary tokens.The bridge supplies these tokens to action-query cross-attention, which extracts information to modify the current cache.Source: §A.1.1 · p. 14
  • Separate encoders and projections map proprioceptive states and actions into a common token space, while reward, termination, and truncation use separate binary embeddings.Learned position and index embeddings distinguish summary and action queries, action-chunk order, and token locations within a trial.Source: §A.1.1 · p. 14
  • A low-rank residual adapter combines native cache content with historical context to modify the KV cache.Its output bases are zero-initialized, so the modified cache initially equals the original and preserves pretrained behavior.Source: §A.1.1 · p. 14

A.2 LEARNING HISTORY CONSTRUCTION PROCEDURES

The appendix describes three ways to construct learning histories, spanning simulation reinforcement learning, human-in-the-loop episodes, and hindsight relabeling. It also specifies a real-world dataset and discusses representation constraints in the FastWAM integration.

A.2.1 IN SIMULATION RL DATASET

  • π0.5 is fine-tuned with PPO on Robotwin tasks using sparse first-success rewards and clipped actor and critic objectives.The vision-language backbone remains frozen, while the action expert and associated policy parameters are optimized.Source: §A.2.1 · p. 15
  • Each PPO iteration collects four rollout rounds from 256 parallel environments, then reuses the batch for five optimization epochs.A complete run records up to 204,800 action-chunk slots; post-termination slots are excluded from the objective.Source: §A.2.1 · p. 15
  • The simulation dataset records task instructions, three RGB observations, proprioception, 50×14 action chunks, rewards, and termination and truncation arrays.Successful and unsuccessful rollouts are retained; evaluation occurs before training and every five PPO iterations.Source: §A.2.1 · p. 15

A.2.2 HUMAN-IN-THE-LOOP CORRECTION DATASSET

  • The HITL dataset contains 391 episodes across four ordered stages and six fruit-and-target subtasks, retaining both successful and unsuccessful episodes.Training trails concatenate one independently sampled episode per stage; same-subtask sampling keeps the selected subtask constant across the trail.Source: §A.2.2 · p. 17
  • For HITL training, checkpoint preprocessing is reused, with seven-dimensional normalized states and action histories and action targets padded to 32 dimensions.Source: §A.2.2 · p. 17

A.3.1 GENERAL PROCEDURE

  • Hindsight trials replace color-specific instructions with “put the power adapter into the plate” and use binary initial feedback to determine the latent target color.The reward sequence is (b, 1, . . . , 1), and counterfactual pairs share the initial attempt but continue toward different colors depending on feedback.Source: §A.3.1 · p. 19
  • Training trials are assembled from 169 demonstrations after holding out 16 for validation, with a curriculum alternating K = 2 and K = 6 episodes.The curriculum is intended to train immediate feedback adaptation and retention of the inferred color across later episodes.Source: §A.3.1 · p. 19

A.3.2 DETAILED SETTINGS

  • The real-world dataset contains 185 successful Franka demonstrations, partitioned into six 30-action chunks per episode with observations at each chunk’s start.Each chunk spans two seconds, and observations include front, side, and wrist cameras plus robot state.Source: §A.3.2 · p. 19

Related papers

Loading 2610.19517v1…