Interactive Post-Training for Vision-Language-Action Models

Shuhan Tan; Kairan Dou; Yue Zhao; Philipp Krähenbühl

Source-linked AI summary

Interactive Post-Training for Vision-Language-Action Models

Shuhan Tan, Kairan Dou, Yue Zhao, Philipp Krähenbühl

arXiv:2505.17016v1May 22, 2025cs.LGcs.AIcs.CVcs.RO

TL;DR

Existing VLA training depends on offline demonstrations and supervised imitation, limiting adaptation to new tasks and environments in low-data settings. RIPT-VLA adds interactive reinforcement-learning post-training with sparse binary rewards and stable rollout optimization, producing strong benchmark performance and rapid one-demo adaptation.

Problem
Existing VLA pipelines rely on offline supervision and costly, high-quality task-specific demonstrations, limiting interactive adaptation to new tasks and environments.
Method
RIPT-VLA adds a third training stage that optimizes pretrained VLA policies through environment interaction using sparse binary rewards, dynamic rollout sampling, and leave-one-out advantages.
Results
RIPT-VLA improves VLA performance across LIBERO, LIBERO-90, and MetaWorld45, while raising one-demo success from below 4% to above 97% within 15 iterations.
Takeaways & Limitations
RIPT-VLA provides a stable, data-efficient post-training paradigm that complements pretraining and supervised fine-tuning through direct environment interaction.
Takeaways & Limitations
RIPT-VLA requires the VLA model to output a probability distribution over actions at each step.

Abstract

from arXiv · show

We introduce RIPT-VLA, a simple and scalable reinforcement-learning-based interactive post-training paradigm that fine-tunes pretrained Vision-Language-Action (VLA) models using only sparse binary success rewards. Existing VLA training pipelines rely heavily on offline expert demonstration data and supervised imitation, limiting their ability to adapt to new tasks and environments under low-data regimes. RIPT-VLA addresses this by enabling interactive post-training with a stable policy optimization algorithm based on dynamic rollout sampling and leave-one-out advantage estimation. RIPT-VLA has the following characteristics. First, it applies to various VLA models, resulting in an improvement on the lightweight QueST model by 21.2%, and the 7B OpenVLA-OFT model to an unprecedented 97.5% success rate. Second, it is computationally efficient and data-efficient: with only one demonstration, RIPT-VLA enables an unworkable SFT model (4%) to succeed with a 97% success rate within 15 iterations. Furthermore, we demonstrate that the policy learned by RIPT-VLA generalizes across different tasks and scenarios and is robust to the initial state context. These results highlight RIPT-VLA as a practical and effective paradigm for post-training VLA models through minimal supervision.

1 Introduction

RIPT-VLA adds interactive reinforcement-learning post-training to the standard VLA pipeline, using sparse binary rewards and stable rollout optimization. It improves multiple VLA models across tasks and enables strong adaptation in extreme low-data settings.

RIPT-VLA adds a third interactive post-training stage after pretraining and supervised fine-tuning, optimizing success across multiple tasks with binary success/failure rewards.Its optimization uses dynamic rollout sampling and leave-one-out advantage estimation without shaped or learned rewards or critic models.
10.9% absolute success-rate improvement is achieved on average across QueST’s four LIBERO task suites.
94.3% success on LIBERO-90 and 92.2% on MetaWorld45 demonstrate performance across many-task benchmarks.
A single training demonstration raises success from below 4% to above 97% within 15 reinforcement-learning iterations.

2 Related Works

VLA research builds on supervised pretraining and fine-tuning, while related reinforcement-learning work seeks to overcome offline-data limitations. Existing RL approaches introduce critics, shaped rewards, or hybrid training components that RIPT-VLA aims to avoid.

VLA models interpret visual observations and natural-language instructions to produce actions in the physical world.
The prevailing two-stage paradigm uses web-scale pretraining followed by supervised fine-tuning, but its offline nature limits adaptation and requires extensive task-specific data.
Large-language-model research motivates reinforcement-learning post-training from interactive feedback for complex reasoning, planning, and constraint-satisfaction tasks.
Prior RL methods for VLAs address supervised-fine-tuning limitations but may require learned value critics, shaped rewards, success weighting, or offline Q-learning.

3 Preliminary

The paper formulates VLA control as autoregressive policy optimization in an environment that returns sparse binary success rewards. RIPT-VLA combines rollout sampling, leave-one-out advantages, and clipped PPO updates to improve policies without learned critics.

VLA formulation: A VLA autoregressively predicts each action from observations, the language goal, and previous actions.
VLA formulation: Rollouts begin from an observation-goal context, generate an action sequence, and receive reward 1 for success or 0 for failure in simulated or real environments.
VLA formulation: VLA models use either tokenized discrete action prediction or continuous action-vector regression.
Supervised training: The standard pipeline consists of large-scale demonstration pretraining followed by supervised fine-tuning on smaller multitask data.
Supervised training: Offline supervision can cause distribution-shift failures and cascading errors, while task-specific demonstrations are costly and quality-sensitive.
Reinforcement optimization: RIPT-VLA directly optimizes expected task success through environment interaction using sparse binary rewards.
Reinforcement optimization: Leave-one-out advantage estimation compares each rollout’s reward with the mean reward of the other rollouts from the same context.
Reinforcement optimization: PPO applies a clipped objective to advantage-weighted updates, encouraging positive advantages while limiting policy deviation.

4 RIPT-VLA

RIPT-VLA adds an interactive reinforcement-learning stage after pretraining and SFT, optimizing VLA policies directly from sparse binary rollout rewards. Its procedure alternates sampled environment rollouts with PPO-based optimization, using leave-one-out advantages and dynamic rejection to stabilize learning across multitask contexts.

Motivation: RIPT-VLA bridges offline-training weaknesses by directly optimizing pretrained VLA policies through interaction with the environment for task success.The preceding supervised objective need not improve online execution because of compounding errors, distribution gaps, and limited demonstrations.
Training paradigm: The first two stages pretrain on diverse data for visual-language representations and general visuomotor skills, then use small task-specific data for instruction following.These stages produce a policy with potentially low but non-zero success on target tasks.
Reinforcement Interactive Post-Training: Stage 3 samples initial contexts, rolls out the policy in the environment, collects binary rewards, estimates advantages, and alternates rollout collection with policy optimization.Contexts are drawn from an initial-context dataset, typically extracted from SFT sequences.
Design challenges: RIPT-VLA requires sparse binary rewards and addresses long-horizon credit-assignment challenges without a learned shaped-reward model.The framework is designed for settings where reward hacking from learned shaped rewards is a concern.
Policy optimization: RLOO estimates advantages from repeated trajectories sharing one context, while PPO performs stabilized updates over the collected rollout dataset.With N = 1, the procedure is on-policy RLOO; N > 1 reuses samples for partially off-policy optimization.
Dynamic rollout sampling: Dynamic rejection discards contexts whose K rollouts all succeed or all fail, resampling until groups provide informative reward variation.This filters solved contexts as performance improves and maintains the effective PPO batch size across minibatches.
Model compatibility: The framework applies to VLA models that sample actions from per-step distributions and compute sequence log-probabilities for policy optimization.Discrete tokenized heads use softmax probabilities, while continuous regression heads can be augmented with a scale-prediction head and probabilistic output distribution.

5 Experiments

Experiments evaluate RIPT-VLA across standard multitask, few-shot, cross-scenario, and cross-goal settings using LIBERO and related benchmarks. Results show improvements across lightweight and large VLA models, strong low-data performance, and generalization beyond the supervised fine-tuning setting.

Standard multitask training: 10.9 absolute SR points: RIPT-VLA improves QueST across all four LIBERO suites, including an 18.7-point gain on the LONG suite.The 20M QueST model reaches performance comparable with π0 on the reported comparisons.
Standard multitask training: 5.7 and 1.2 absolute SR points: RIPT-VLA improves QueST on LIBERO-90 and ML45, respectively, setting new state-of-the-art results on both benchmarks.These benchmarks contain 90 and 45 diverse tasks, testing scalability across many skills.
Few-shot multitask training: 21.2 and 12.4 absolute SR points: RIPT-VLA improves QueST in the 5-shot LIBERO-LONG and ML45 settings, respectively.The evaluation uses only five demonstrations per task, where baseline models struggle.
Few-shot multitask training: 20.8 absolute gain: RIPT-VLA improves the standard SFT model with one demonstration per task on LIBERO-LONG, and continues improving as demonstrations increase.The few-shot curve spans 1 to 10 demonstrations per task, indicating performance across multiple data scales.
Generalization: 93.7% absolute SR gain: RIPT-VLA raises cross-scenario performance from 3.5% SFT to 97.2%, while cross-goal performance reaches 59.7% versus 0.7% for SFT with three demonstrations.Cross-scenario results often approach 100% with 3–5 demonstrations; at 10 demonstrations, cross-goal SR is 79.7% for RIPT-VLA versus 29.4% for SFT.

6 Conclusion

RIPT-VLA is a reinforcement-learning paradigm for post-training pretrained VLA models with sparse binary task rewards. It provides stable, data-efficient optimization and improves adaptability in extremely low-data settings.

RIPT-VLA post-trains pretrained VLA models using sparse binary task rewards.It is presented as a third-stage training paradigm after pretraining and supervised fine-tuning.
RIPT-VLA enables stable and data-efficient optimization without shaped rewards, value functions, or reward modeling.
RIPT-VLA significantly improves performance across multiple VLA benchmarks.
RIPT-VLA demonstrates adaptability in extremely low-data settings through direct environment interaction.
RIPT-VLA complements existing pretraining and supervised fine-tuning pipelines as a scalable third-stage training paradigm.The authors identify combining it with reasoning and planning as a future direction for more sophisticated behaviors in complex environments.