PaperlayerView exact revision on arXiv

Source-linked AI summary

On-the-Fly VLA Adaptation via Test-Time Reinforcement Learning

Changyu Liu, Yiyang Liu, Taowen Wang, Qiao Zhuang, James Chenhao Liang, Wenhao Yang, Renjing Xu, Qifan Wang, Dongfang Liu, Cheng Han

arXiv:2601.06748v3cs.RO

TL;DR

VLAs remain difficult to deploy flexibly because fixed training and existing test-time objectives do not adequately handle evolving, multimodal environments. TT-VLA adapts policies online with dense task-progress rewards while preserving SFT/RL priors, and experiments report improved performance across unseen simulated and real-world tasks and VLA backbones.

  • Problem

    VLAs trained through fixed SFT or training-time RL need adaptation during deployment, while existing TTT cannot be directly applied to their evolving multimodal distribution shifts.

  • Method

    TT-VLA performs online inference-time policy fine-tuning with dense reward signals while preserving SFT/training-time RL priors and avoiding retraining cycles.

  • Results

    TT-VLA consistently enhances performance on unseen tasks across diverse simulated and real-world scenarios and various VLA backbones.

  • Takeaways & Limitations

    TT-VLA supplements existing SFT- and RL-based VLAs with continuous policy refinement under dynamic, previously unseen test-time conditions.

Abstract

from arXiv · show

Vision-Language-Action models have recently emerged as a powerful paradigm for general-purpose robot learning, enabling agents to map visual observations and natural-language instructions into executable robotic actions. Though popular, they are primarily trained via supervised fine-tuning or training-time reinforcement learning, requiring explicit fine-tuning phases, human interventions, or controlled data collection. Consequently, existing methods remain unsuitable for challenging simulated- or physical-world deployments, where robots must respond autonomously and flexibly to evolving environments. To address this limitation, we introduce a Test-Time Reinforcement Learning for VLAs (TT-VLA), a framework that enables on-the-fly policy adaptation during inference. TT-VLA formulates a dense reward mechanism that leverages step-by-step task-progress signals to refine action policies during test time while preserving the SFT/RL-trained priors, making it an effective supplement to current VLA models. Empirical results show that our approach enhances overall adaptability, stability, and task success in dynamic, previously unseen scenarios under simulated and real-world settings. We believe TT-VLA offers a principled step toward self-improving, deployment-ready VLAs.

1 Introduction

VLAs must adapt during deployment because fixed training regimes and policies struggle with dynamic environments and distribution shifts. TT-VLA addresses this gap by refining policies online with dense test-time rewards while preserving prior SFT/RL knowledge.

  • Motivation: Fixed training regimes and policies leave VLAs poorly suited to dynamic deployment environments and evolving distribution shifts.VLAs translate visual observations and natural-language instructions into executable actions, but most remain trained through SFT or training-time RL.
  • Motivation: Test-time adaptation is underexplored for VLAs, and existing TTT methods cannot be directly applied because VLA settings involve substantial multimodal distribution shifts.The paper identifies evolving visual and linguistic conditions as a central obstacle to transferring standard TTT approaches.
  • Approach: TT-VLA performs online inference-time policy fine-tuning without retraining cycles, preserving SFT/training-time RL priors and using dense inference-time reward signals.The framework combines test-time training accessibility with reinforcement learning to address changing environmental conditions and distributions.
  • Approach: Task-agnostic proxies provide dense shaping signals that operate independently across time frames, supporting stable and continuous policy adjustment from limited test-time information.The design is presented as a bridge between offline-RL/VLA pipelines and continuous self-improvement under previously unseen conditions.

2 Related Work

Prior work combines VLAs with supervised learning, reinforcement learning, or self-supervised test-time training, but these approaches leave important adaptation and efficiency gaps. TT-VLA is proposed as an efficient RL-driven alternative for online adaptation in evolving environments.

  • VLA adaptation: Supervised fine-tuning gives VLAs strong performance in static settings but can produce brittle behavior and limited robustness under distribution shifts.The related-work discussion attributes this limitation to the lack of adaptive learning mechanisms.
  • VLA adaptation: RL-augmented VLAs use interaction-driven feedback to refine behavior toward task-specific objectives and environmental variations.These methods extend beyond static supervision through interaction with environments or demonstrations.
  • Test-time training: Test-time training adapts pretrained models using unlabeled test data and self-supervised objectives, without human-curated labels or external feedback.The paper contrasts this inference-time paradigm with SFT and traditional RL.
  • Test-time training: Applying standard TTT to VLAs is difficult because robotic tasks undergo substantial, evolving shifts across visual and linguistic modalities.The passage contrasts this multimodal variability with more limited inter-task discrepancies in some single-domain settings.
  • RL-based test-time adaptation: EVOLVE-VLA uses task progress as a deployment-time reward, but its GRPO-style optimization imposes computational overhead that limits real-time applicability.TT-VLA is introduced as an RL-driven framework for efficient online adaptation.

3 Method

TT-VLA adapts pretrained VLA policies during deployment by using dense progress rewards and value-free PPO updates, enabling continuous within-episode refinement without training data, resets, or human intervention. Its theoretical analysis motivates immediate-reward updates because conventional value-based estimation can eliminate or bias the learning signal under progress-difference rewards.

  • Problem formulation: TT-VLA formulates robotic manipulation as a POMDP in which a VLA maps multimodal observation histories and language instructions to action sequences.The observation space can include RGB and proprioception, while the instruction space contains natural-language tasks.
  • Test-time adaptation: At test time, TT-VLA updates the pretrained policy online during deployment without training data, environment resets, or human intervention.This directly addresses the fixed-policy assumption of standard VLA decision processes.
  • Value-free PPO: The method removes value-function learning and entropy regularization from PPO, retaining the clipped surrogate objective for rapid, stable policy refinement.Value-free adaptation is motivated by limited single-episode samples and strict online latency constraints.
  • Progress-based reward: TT-VLA estimates task progress from observation history and instructions, then defines dense rewards as temporal differences in progress.The progress predictor is instantiated with VLAC, a pretrained multimodal scalar regressor.
  • Progress-based reward: These progress-based rewards provide autonomous, step-wise feedback for continuous mid-episode adaptation while encouraging monotonic progress and discouraging regressions or oscillations.Sparse rewards delay updates until task completion, preventing timely mid-episode correction.
  • Value-free PPO: Setting λ = 0 and γ = 0 reduces the advantage to the immediate reward, so each policy update directly reflects current-step progress.The resulting policy is updated after each dense reward and then used for subsequent actions throughout the episode.
  • Theoretical analysis: With progress-difference rewards and the remaining-progress value function, γ = 1 makes the TD error and GAE identically zero, while 0 < γ < 1 introduces systematic negative TD bias.These results provide the theoretical rationale for the value-free, immediate-reward formulation.

4 Experiment

The experiments evaluate TT-VLA on unseen simulated and real-world manipulation tasks across execution, vision, and semantic generalization. TT-VLA consistently improves baseline VLA performance, with progress-based rewards and instantaneous feedback supporting test-time adaptation and recovery.

  • Experimental Setup: TT-VLA is evaluated on unseen robotic tasks in both simulated and real-world settings.Simulation uses standard pick-and-place tasks across Execution, Vision, and Semantics generalization; real-world evaluation uses nine unseen tasks on a Franka Research 3.
  • Simulation Results: Across baselines and task categories, TT-VLA consistently improves performance during test time.The simulation benchmark covers four open-source VLAs and reports success rates across Execution, Vision, and Semantics dimensions.
  • Simulation Results: On Nora, TT-VLA improves 14 of 15 tasks, with relative gains ranging from 5.26% to 44.4%.The largest reported gains are 44.4% on Task Obj. Rep. and 18.15% on Task Noise-s.
  • Real-World Results: On nine unseen real-world tasks, TT-VLA consistently improves performance over OpenVLA.The tasks cover Execution, Vision, and Semantics generalization, extending the evaluation beyond simulation.
  • Diagnostic Experiments: Instantaneous reward feedback enables rapid recovery from trajectory errors during real-world deployment.In the banana-placement case study, TT-VLA detects regression, adjusts behavior, realigns with the objective, and completes placement successfully.
  • Diagnostic Experiments: Progress-based rewards outperform standard GAE by emphasizing each action’s immediate contribution to task progress.With OpenVLA on the Vision task, the immediate-progress setting reaches 57.08% success versus 55.00% with standard GAE.

5 Conclusion

TT-VLA targets flexible policy refinement during deployment, extending VLA use beyond closed-form benchmarks. The framework improves performance on unseen tasks across simulated and real-world scenarios and multiple VLA backbones.

  • TT-VLA focuses on adapting VLA action policies during test time in evolving environments.
  • TT-VLA consistently enhances performance on unseen tasks across diverse simulated and real-world scenarios.
  • The framework improves flexibility across various VLA backbones and previously unseen test-time cases.

SUMMARY OF THE APPENDIX

The appendix organizes supplementary material covering technical contributions, related work, proofs, tasks, diagnostics, qualitative results, practical considerations, reproducibility, ethics, future directions, licensing, and AI disclosure.

  • Additional material covers task details, diagnostic experiments, Test-Time Training, and qualitative results.
  • The appendix discusses the practicalness of Test-Time GRPO in VLAs and provides reproducibility materials with pseudocode.
  • The appendix includes technical contributions, related work on VLA models, and a lemma proof.
  • It also includes licenses and consent, ethics discussions, future directions, and an AI disclosure describing limited grammar-checking assistance.

S1.1 More Discussions on VLA

The paper focuses on discretization-based VLA approaches because diffusion-head architectures use separate action decoders that make reinforcement-learning optimization impractical.

  • Diffusion-head VLA models typically map visual and instruction embeddings through latent diffusion to action embeddings, followed by an MLP for joint-space regression.
  • The study focuses on and generalizes discretization-based VLA approaches.
  • This separate action-decoder design renders reinforcement-learning optimization impractical for diffusion-head VLA models.

S1.2 RL Methods for VLA

Prior work has applied reinforcement learning mainly during VLA training, while this paper targets the underexplored problem of on-the-fly policy adaptation. Existing approaches use preference optimization, reinforced fine-tuning, dense rewards, iterative SFT/RL, and policy-gradient methods.

  • Recent VLA research has integrated reinforcement learning during training, leaving test-time adjustment underexplored.
  • GRAPE uses Direct Preference Optimization to train VLAs with human preferences, while ConRFT alternates reinforcement learning and supervised fine-tuning in real-world environments.
  • ReinboT introduces dense reward design, and iRe-VLA combines supervised fine-tuning and reinforcement-learning stages to address instability and computational overhead.
  • RIPT-VLA employs REINFORCE Leave-One-Out, while VLA-RL applies PPO and TGRPO and SimpleVLA-RL evaluate performance beyond static supervision.

S2 Lemma Proof

Lemma 1 characterizes how GAE relates to TT-VLA’s reward-only advantage in limiting cases, formally supporting the value-free test-time optimization objective.

  • The proof establishes the relationship between GAE and TT-VLA’s reward-only advantage.
  • When λ = 0, the geometric weighting term vanishes for l > 0, yielding A_t = δ_t.
  • When γ = 0, equations (14) and (15) yield the corresponding limiting forms used in the lemma.

S3 Task Details

The experiments define simulated and real-world generalization settings across execution, vision, and semantic variation, including altered positions, appearances, objects, receptacles, and instructions.

  • Simulation generalization is organized into Execution, Vision, and Semantics dimensions for unseen tasks.
  • Execution: Execution varies object and receptacle positions, robot initialization, mid-episode object placement, and multi-object or multi-receptacle configurations.
  • Training setting: The training setting samples objects and table appearances, randomizes placements, and uses the instruction template “put O on R”.
  • Vision: Vision tests unseen table appearances, dynamic textures, and dynamic noise at weak or strong transparency levels.
  • Semantics: Semantics tests unseen objects, receptacles, instruction phrasings, and distractive receptacles.
  • Real-world evaluation: Real-world evaluation uses nine unseen manipulation tasks spanning execution, vision, and semantic generalization.

S4 Additional Details on Diagnostic Experiments

The supplementary material details diagnostic experiments, test-time training baselines, qualitative real-world rollouts, implementation information, contributions, and deployment limitations.

  • Diagnostic experiments: Diagnostic experiments use Nora and OpenVLA on one controlled task from each of the execution, vision, and semantics dimensions.
  • Diagnostic experiments: The advantage-design comparison contrasts standard GAE, using γ = 0.99 and λ = 0.95, with TT-VLA’s one-step reward-only advantage using γ = 0 and λ = 0.
  • Test-time training comparisons: TLM minimizes instruction-prompt perplexity, while TTRL uses majority voting over eight candidate action tokens and updates at every environment step.
  • Real-world qualitative results: Three real-world banana-placement rollouts show recovery from incorrect placement, directional overshoot, and motion past the plate without retraining or human intervention.
  • Technical contributions: TT-VLA introduces deployment-time policy adaptation, dense progress-based rewards, and improvements in robustness and success rates across simulated and real-world environments.
  • Ethics concerns: Online policy adaptation may produce unintended or unsafe behaviors in physical environments, motivating constrained actions, safety constraints, conservative updates, and monitoring.
  • Future direction: The study primarily focuses on discretization-based VLA architectures rather than diffusion-based formulations.
Loading 2601.06748v3…