PaperlayerView exact revision on arXiv

Source-linked AI summary

What Can RL Bring to VLA Generalization? An Empirical Study

Jijia Liu, Feng Gao, Bingwen Wei, Xinlei Chen, Qingmin Liao, Yi Wu, Chao Yu, Yu Wang

arXiv:2505.19789v4cs.LG

TL;DR

VLA training is dominated by SFT, but its robustness under distribution shifts and the specific generalization benefits of RL remain insufficiently understood. This paper builds a benchmark spanning visual, semantic, and execution generalization, compares RL methods with SFT, and finds PPO-based RL improves semantic and execution robustness while matching SFT’s visual robustness.

  • Problem

    SFT-dominated VLA training is susceptible to compounding errors under distribution shifts, while RL’s specific generalization benefits relative to SFT remain insufficiently understood.

  • Method

    The paper benchmarks RL fine-tuning against SFT across visual, semantic, and execution dimensions and develops an efficient PPO recipe.

  • Results

    RL improves semantic grounding and action execution under distribution shifts over SFT while matching its resilience to visual perturbations, with PPO outperforming GRPO and DPO.

  • Takeaways & Limitations

    A streamlined PPO approach yields strong and efficient gains in VLA generalization within the evaluated setting.

  • Takeaways & Limitations

    The evaluation is confined to simulated pick-and-place tasks and uses motion-planner-generated demonstrations rather than human-collected data.

Abstract

from arXiv · show

Large Vision-Language Action (VLA) models have shown significant potential for embodied AI. However, their predominant training via supervised fine-tuning (SFT) limits generalization due to susceptibility to compounding errors under distribution shifts. Reinforcement learning (RL) offers a path to overcome these limitations by optimizing for task objectives via trial-and-error, yet a systematic understanding of its specific generalization benefits for VLAs compared to SFT is lacking. To address this, our study introduces a comprehensive benchmark for evaluating VLA generalization and systematically investigates the impact of RL fine-tuning across diverse visual, semantic, and execution dimensions. Our extensive experiments reveal that RL fine-tuning, particularly with PPO, significantly enhances generalization in semantic understanding and execution robustness over SFT, while maintaining comparable visual robustness. We identify PPO as a more effective RL algorithm for VLAs than LLM-derived methods like DPO and GRPO. We also develop a simple recipe for efficient PPO training on VLAs, and demonstrate its practical utility for improving VLA generalization. The project page is at https://rlvla.github.io

1 Introduction

The paper studies how reinforcement learning improves VLA generalization relative to SFT, addressing limited systematic evidence across visual, semantic, and execution dimensions. It benchmarks RL methods, identifies PPO as preferred, and develops an efficient PPO recipe.

  • Motivation: SFT-based VLA training is vulnerable to compounding errors under distribution shifts, motivating reward-driven alternatives.Small deviations from demonstrations can accumulate and steer policies into unfamiliar states.
  • Research gap: The study systematically compares RL fine-tuning with SFT to identify RL’s distinct benefits for VLA generalization.The central question concerns how their generalization strengths differ.
  • Benchmark: The benchmark evaluates generalization across vision, semantics, and execution on a representative pick-and-place task.Tests include novel visual appearances, unseen objects and language, varied initial states, positional changes, and disturbances.
  • Methods: PPO is preferred over GRPO and DPO for VLA fine-tuning, and the paper develops an efficient PPO-based recipe.The recipe uses a shared actor-critic backbone, VLA warm-up, and minimal PPO epochs.
  • Findings: RL improves semantic understanding and embodied execution over SFT while maintaining comparable visual robustness.This is the paper’s principal cross-dimensional finding.

2 Related works

Prior work establishes VLAs as foundation-model-based robot policies and explores supervised, preference-based, and reward-driven fine-tuning. However, systematic evidence about RL’s generalization benefits for VLAs remains limited.

  • Vision-Language-Action models: VLAs combine pretrained vision-language knowledge with large robotic datasets to map sensor observations and language instructions to actions.This research direction targets policy generalization in embodied tasks.
  • Foundation-model fine-tuning: Large-model fine-tuning research includes supervised learning, preference alignment, online RL, and offline preference-based updates.These approaches have primarily been developed for language and vision-language models.
  • VLA fine-tuning: VLA fine-tuning largely relies on supervised data, while limited high-quality data can hinder generalization to unseen scenarios and perturbations.Prior RL efforts span small models, human-intervention settings, and offline updates.

3 Preliminaries

The paper formulates language-conditioned robotic tasks as POMDPs and contrasts SFT’s demonstration-based learning with RL’s reward- or preference-driven optimization. OpenVLA emits discretized action tokens for robot control.

  • Problem formulation: Each language-conditioned task is modeled as a POMDP with states, actions, transitions, rewards, observations, language instructions, initial-state distribution, and discounting.The policy uses a history of observations and an instruction to sample actions and form trajectories.
  • Supervised fine-tuning: SFT minimizes a loss over expert demonstration trajectories, with the specific loss determined by the VLA architecture and action representation.Examples include next-token prediction, L1 regression, and diffusion losses.
  • Reinforcement learning: RL fine-tuning maximizes scalar rewards or preference signals through direct environment interaction or trajectory-level preference optimization.Policy-gradient methods optimize reward-based objectives, while DPO uses preferred and rejected trajectories.
  • OpenVLA: OpenVLA discretizes each continuous action scalar into one of 256 bins and trains the language model to predict the resulting action tokens.The action tokens replace the least-used vocabulary tokens, and cross-entropy is computed only on predicted action tokens.
  • Fine-tuning methods: The fine-tuning overview distinguishes offline-demonstration SFT from RL-based DPO, GRPO, and PPO updates.The methods use preference alignment, group-relative advantage estimation, and actor-critic PPO with GAE, respectively.

4 Effective RL fine-tuning of VLA models

The study compares PPO, GRPO, and DPO for VLA fine-tuning and finds PPO consistently more effective, then develops efficient PPO designs for OpenVLA.

  • RL algorithm comparison: PPO consistently improves performance, whereas GRPO and DPO struggle to learn some VLA tasks.The comparison evaluates three RL algorithms on the pick-and-place task, using two random seeds per experiment.
  • RL algorithm comparison: PPO outperforms GRPO and DPO under the study’s robotic training setup.The authors hypothesize that non-stationary robotic dynamics destabilize GRPO advantage estimates, while sparse rewards and offline-to-interactive distribution shifts hinder DPO.
  • Efficient PPO design: A shared actor-critic backbone with an h0-based three-layer MLP critic is the most efficient critic design.A separate Transformer backbone achieves comparable rewards but trains 35 % slower and uses 83 % more VRAM, 81.3 GB versus 44.4 GB.
  • Efficient PPO design: VLA warm-up with 140 demonstration trajectories reaches convergence with roughly 50 % fewer environment steps.Warm-up and non-warm-up initializations attain comparable asymptotic returns given sufficient interaction.
  • Efficient PPO design: Increasing the PPO epoch beyond one adds no return or sample-efficiency gains while increasing wall-clock time almost linearly.The study therefore fixes epoch = 1 for subsequent experiments to achieve the fastest training without sacrificing performance.

5 Evaluating fine-tuning methods on VLA generalization

The benchmark tests VLA generalization across visual, semantic, and execution shifts in pick-and-place tasks. RL matches SFT visually but performs better on semantic and execution generalization, with broader behavioral coverage.

  • 5.1 Environments and datasets: The benchmark evaluates Vision, Semantics, and Execution generalization in a pick-and-place task.Training randomizes tables, objects, and poses, while testing holds out factors including novel objects, receptacles, table surroundings, and textures.
  • 5.2 SFT baseline: SFT performance plateaus at roughly 16k trajectories, establishing SFT-16k as the comparison baseline.The study trains on datasets ranging from a few hundred to 64k expert trajectories and averages scores over three random seeds.
  • 5.3 Performance comparison between RL and SFT: 42.6 % better performance on unseen objects and tables is achieved by RL than by SFT-16k at convergence.RL performs comparably to SFT-16k in the training setting and overtakes it on OOD tasks after roughly 0.4 M environment steps.
  • 5.3 Performance comparison between RL and SFT: RL performs comparably to SFT in Vision, noticeably better in Semantics, and significantly better in Execution.Execution improvements span OOD object and receptacle positions, OOD robot initial positions, and mid-episode object repositioning.
  • 5.4 Dissecting RL’s contributions to generalization: RL succeeds on representative visual and semantic cases where SFT drops or stalls.Under strong dynamic noise, SFT repeatedly drops the object, while on an unseen object it repeatedly grasps the item it already holds; RL completes both tasks.
  • 5.4 Dissecting RL’s contributions to generalization: RL trajectories cover a broader workspace and richer end-effector orientations than SFT trajectories clustered along motion-planner paths.The broader trajectory coverage appears key to RL’s superior generalization on Execution tasks.

6 Conclusion

The study introduces a benchmark for VLA out-of-distribution generalization and finds that streamlined PPO improves semantic grounding and execution robustness relative to SFT while matching visual resilience. Its scope is limited by planner-generated demonstrations, pick-and-place simulation, and the absence of physical-robot validation.

  • A streamlined PPO variant with a shared actor-critic backbone, simple warm-up, and minimal PPO epochs yields the strongest and most efficient gains.
  • Relative to SFT, PPO substantially improves semantic grounding and action execution under distribution shifts while matching resilience to visual perturbations.
  • The study relies solely on motion-planner–generated demonstrations, which may not capture variability in human-collected data.
  • Evaluation is confined to pick-and-place tasks in simulation, leaving broader multi-task settings and sim-to-real validation for future work.

A.1 Implementation of different fine-tuning algorithm

The appendix describes LoRA-based implementations for PPO, GRPO, and trajectory-wise preference optimization, along with their reward and trajectory-handling choices.

  • All RL and SFT methods use LoRA modules with rank r = 32 on OpenVLA linear layers while fully fine-tuning the value head.
  • PPO updates the policy by maximizing its standard objective using token-level action probabilities and GAE advantages.
  • GRPO computes advantages from normalized outcome rewards within trajectory groups, with ri denoting trajectory reward and G denoting group size.
  • Training uses 32 groups of 8 trajectories, truncating successful placement episodes after the first successful step while leaving unsuccessful episodes untruncated.
  • TPO aligns VLA policies with preferences over entire trajectories by comparing preferred and rejected trajectory pairs.
  • Because only sparse rewards are available, the implementation uses the sparse reward rather than TPO’s richer original reward combination.

A.2 Details of motion planner and SFT dataset

The motion-planner pipeline generates locally feasible screw-like paths and time parameterizations, while action filtering removes planner idling that can cause SFT policies to get stuck.

  • The plan_screw function uses task-space guided iterative inverse kinematics to approximate screw motion, then applies TOPP to assign timing.
  • SFT on motion-planner data can cause OpenVLA to get stuck because the demonstrations contain idling actions.
  • Actions below the delta position and Euler-angle thresholds are discarded, removing approximately one-third of planner actions and alleviating stuck execution.
  • The dataset uses 3D objects and receptacles from multiple asset sources, with synthesized table appearances and downloaded distractor textures.

A.4 Details of tasks for generalization evaluation

The evaluation varies training conditions and tests visual, semantic, and execution generalization through unseen tables, textures, objects, receptacles, instructions, and dynamic perturbations.

  • Training: Training samples one of 16 objects and one of 16 table appearances, randomizes their positions, and uses the instruction “put $O$ on $R$”.
  • Vision: Unseen-table evaluation selects from 5 unseen appearances while keeping other settings unchanged.
  • Vision: Dynamic texture and noise conditions overlay changing textures with transparency 0.3 or 0.5, either on task entities and the robot arm or across the whole image.
  • Semantics: Semantic tests use 9 unseen objects, 16 unseen receptacles, and 16 unseen language templates with varied phrasing, punctuation, and capitalization.
  • Training sensitivity: PPO performance declines with very large generation temperatures, whereas moderate or small temperatures train well; the study uses temperature 1.0.
  • Training sensitivity: Lower LoRA ranks train slightly more efficiently, but rank 32 is used to preserve sufficient model capacity.

B.3 Full results of OOD tasks

The study extends OOD evaluation to preliminary sim-to-real transfer and action-chunking settings. RL outperforms SFT in real-world grasping and pick-and-place, and retains its advantage with OpenVLA-OFT.

  • Sim-to-real evaluation: The real-world evaluation changes the robot arm, camera configuration, backgrounds, objects, and receptacles, creating an OOD sim-to-real setting.Simulation uses WidowX, 16 objects, seven backgrounds, and a plate; real-world evaluation uses Franka Panda, six new objects, a table, and a green bowl.
  • Sim-to-real evaluation: RL outperforms SFT in both grasp and pick-and-place success rates across 30 real-world trials.SFT often overshoots, whereas RL iteratively adjusts the end-effector pose despite some jitter.
  • Action chunking: Action-chunking PPO uses action-dimension-wise clipping and adjusted clipping, discount, and GAE hyperparameters for training stability.The clipping ratio is reduced from 0.2 to 0.1, with γ = 0.96 and λ = 0.85.
  • Action chunking: RL maintains its advantage over SFT when OpenVLA-OFT predicts action chunks of four actions.The comparison evaluates both success rate and relative performance drop under action chunking.

B.6 Extended performance comparison on opening articulated objects

The articulated-objects extension tests richer manipulation with a Franka Panda arm and OOD variation across vision, semantics, and execution. RL continues to outperform SFT on semantic and execution splits.

  • Task design: The task requires grasping an object handle and opening the door past 20° with staged rewards for grasping, sustained grasp, and opening.The stated reward structure includes 0.1 for initial grasp, 0.1 for five-step sustained grasp, and a final 1.0 for opening.
  • Task design: Training randomizes 16 backgrounds, eight articulated objects with varied properties, and object-frame translations up to ±8 cm plus rotations up to ±π/48.These perturbations target vision, semantics, and execution generalization.
  • Evaluation: Evaluation uses six OOD splits covering novel backgrounds, dynamic noise, unseen objects, re-phrased instructions, and randomized robot-joint offsets.The supplied splits include five novel backgrounds, transparency-0.5 noise, four unseen objects, 16 re-phrasings, and offsets on joints 0/1/2/4.
  • Results: RL outperforms SFT on semantic and execution OOD splits in the open articulated objects task.The result is consistent with the study’s main pick-and-place findings.
Loading 2505.19789v4…