PaperlayerView exact revision on arXiv

Source-linked AI summary

World-Gymnast: Training Robots with Reinforcement Learning in a World Model

Ansh Kumar Sharma, Yixiang Sun, Ninghao Lu, Yunzhe Zhang, Jiarao Liu, Sherry Yang

arXiv:2602.02454v1cs.ROcs.AI

TL;DR

Robot learning is constrained by expensive physical interaction, while SFT has limited demonstration coverage and simulator RL faces a sim-to-real gap. World-Gymnast fine-tunes VLA policies through VLM-rewarded rollouts in an action-conditioned video world model, achieving stronger real-robot performance while enabling diverse training, test-time adaptation, and iterative improvement.

  • Problem

    Robot learning alternatives are limited because physical interaction is expensive, demonstrations cover narrow situations, and software simulators face costly construction and a sim-to-real gap.

  • Method

    World-Gymnast fine-tunes a VLA policy by rolling it out in an action-conditioned video world model and using a VLM to compute rewards for policy-gradient updates.

  • Results

    On Bridge robot tasks, World-Gymnast substantially outperforms SFT and software-simulator RL in real-robot evaluation.

  • Takeaways & Limitations

    World-Gymnast supports training from novel frames and language instructions, test-time training, and iterative world-model and policy improvement.

  • Takeaways & Limitations

    World-Gymnast may fail on initial frames far outside the world model’s training distribution, and pretrained VLM reward hallucinations can produce suboptimal RL training.

Abstract

from arXiv · show

Robot learning from interacting with the physical world is fundamentally bottlenecked by the cost of physical interaction. The two alternatives, supervised finetuning (SFT) from expert demonstrations and reinforcement learning (RL) in a software-based simulator, are limited by the amount of expert data available and the sim-to-real gap for manipulation. With the recent emergence of world models learned from real-world video-action data, we ask the question of whether training a policy in a world model can be more effective than supervised learning or software simulation in achieving better real-robot performance. We propose World-Gymnast, which performs RL finetuning of a vision-language-action (VLA) policy by rolling out the policy in an action-conditioned video world model and rewarding the rollouts with a vision-language model (VLM). On the Bridge robot setup, World-Gymnast outperforms SFT by as much as 18x and outperforms software simulator by as much as 2x. More importantly, World-Gymnast demonstrates intriguing capabilities of RL with a world model, including training on diverse language instructions and novel scenes from the world model, test-time training in a novel scene, and online iterative world model and policy improvement. Our results suggest learning a world model and training robot policies in the cloud could be the key to bridging the gap between robots that work in demonstrations and robots that can work in anyone's household.

1. Introduction

World-Gymnast addresses the cost and limited coverage of real-robot learning by fine-tuning VLA policies with RL inside a learned video world model. It is evaluated against SFT and software-simulator RL on real Bridge robots, while supporting broader training and adaptation scenarios.

  • Physical interaction bottlenecks robot learning because hardware trials consume operator time, cause wear, and raise safety concerns.
  • SFT is limited by narrow expert-demonstration coverage, while software-simulator RL faces costly scenario creation and a visual sim-to-real gap.
  • World-Gymnast fine-tunes VLA policies by rolling them out in an action-conditioned video world model and using VLM-computed rewards for policy-gradient updates.
  • World-Gymnast enables RL from arbitrary image frames, test-time training on novel frames, and online iterative improvement of the world model and policy.
  • On Bridge tasks evaluated with AutoEval, World-Gymnast substantially outperforms SFT and software-simulator RL while supporting novel instructions, distractor scenes, test-time training, and iterative improvement.

2. Preliminaries

The paper formulates robot learning as model-based RL in a partially observable, multi-task environment and replaces unknown dynamics and rewards with learned foundation models. Policy-gradient updates then optimize behavior using imagined trajectories.

  • The framework models robot learning as a finite-horizon, multi-task POMDP with states, actions, observations, tasks, rewards, transitions, emissions, and a horizon.
  • A policy samples actions from observations and task goals, while the environment transitions between states and emits new observations.
  • A policy’s value is the total expected future reward over trajectories.
  • Model-based RL estimates unknown transition and reward functions from previously collected interaction data, then rolls out policies in the learned model.
  • An action-conditioned video world model can parameterize transitions, while a VLM can parameterize rewards in pixel-based model-based RL.
  • Policy-gradient methods maximize expected return using advantage estimates computed from Monte Carlo rollouts, including rollouts from learned dynamics and reward models.

3. RL with a World Model

World-Gymnast trains VLA policies through imagined world-model rollouts scored by a VLM, then expands training beyond fixed demonstrations using varied frames, language, distractions, and iterative adaptation. These mechanisms target broader coverage, recovery behavior, and adaptation without equivalent real-world interaction costs.

  • World-Gymnast combines world-model RL, diverse training scenarios, test-time training, and iterative world-model and policy improvement.
  • 3.1. Model-Based GRPO with World Model Rollouts: GRPO normalizes rewards within groups of trajectories to compute advantages, assigns each trajectory’s advantage across its time steps, and optimizes a clipped PPO-style objective.
  • 3.1. Model-Based GRPO with World Model Rollouts: For each instruction and initial observation, the policy generates trajectories in the learned world model, and a VLM assigns each trajectory a binary completion reward.
  • Training from Any Frame: Training from any frame increases available RL configurations beyond expert demonstrations and supports learning recovery behaviors for improved robustness.
  • Training on Novel Language Instructions: Language augmentation enables training on novel tasks and interactions with objects not explicitly represented in the original task instructions.
  • Training with Distractions: Image editing adds distractor objects to training frames, encouraging robustness to irrelevant visual clutter during deployment.
  • Test-Time Training from a Novel Frame: At test time, the policy can run RL in the world model from a novel frame, trading computation for adaptation without collecting real-world interaction data.
  • Iterative World Model and Policy Improvement: Iterative refinement incorporates new environment interactions into the world model, producing increasingly accurate imagined rollouts for policy optimization.

4. Experiments

Experiments evaluate World-Gymnast against software-simulator RL and supervised learning, then test robustness, language augmentation, distractors, test-time training, and iterative world-model improvement. Across these settings, World-Gymnast improves real-robot performance and supports adaptation using imagined rollouts.

  • Experimental Setup: World-Gymnast is evaluated on BridgeData V2 tasks with WorldGym and AutoEval, using initial frames and language instructions to generate up to 40-step rollouts.AutoEval evaluates real-robot policies across supported tasks, while WorldGym estimates performance before physical testing.
  • Comparing with Software Simulation: World-Gymnast outperforms SIMPLER on all tasks except close the drawer, despite World-Gymnast’s RL training set excluding the evaluated tasks.SIMPLER was trained on the exact environment-task setups, whereas World-Gymnast transferred from other tasks.
  • Comparing with Supervised Learning: World-Gymnast achieves the best performance against supervised learning and Iter-SFT, with 18-fold and nearly 10-fold improvements over the base policy on two eggplant tasks.Iter-SFT improves slightly on harder tasks but degrades on easier ones.
  • Training with Distractors: World-Gymnast-Distract is most robust to visual distractions, while SFT is easiest to distract; distractor training also improves performance on original tasks.Qualitative rollouts show more reliable grasping, placement, and object grounding despite world-model visual artifacts.
  • Diverse Settings: Novel language instructions and additional tasks improve held-out performance, showing that World-Gymnast can leverage imagined rollouts across diverse configurations.World-Gymnast-Language improves over World-Gymnast, and scaling with five additional tasks also improves held-out performance.
  • Test-Time and Iterative Improvement: Test-time training from a novel frame raises close-the-drawer success from 62 ± 6% to 100 ± 0% without real-world rollouts, but overfits that task and degrades others.Online world-model updates further improve the close-the-drawer success rate to 95%.

5. Related Work

World-Gymnast builds on model-based RL, sim-to-real RL, video-based robot learning, and policy evaluation with learned world models. It differs by using a video world model to improve policies and testing the resulting policies on real robots while exploring broader training capabilities.

  • Model-based RL learns dynamics from prior data and rolls out the learned model for policy evaluation and improvement.
  • Sim-to-real RL commonly uses domain randomization, whereas World-Gymnast uses a world model learned directly from real-world data to avoid manual asset creation and physics tuning.
  • Video-based learning has supported visual representation learning, goal extraction, planning, and imitation from expert actions.
  • Prior work uses real-robot world models to evaluate policies, while this work focuses on improving policy performance and testing that improvement on real robots.
  • Compared with similar RL in video world models, World-Gymnast emphasizes accessible real-world evaluation and emergent capabilities such as novel language, novel initial frames, test-time training, and iterative improvement.

6. Conclusion and Limitations

The paper concludes that World-Gymnast uses learned world models to make VLA policy training cheaper, more scalable, and more flexible than existing paradigms. Its main limitations are dependence on the world model’s training distribution and on a pretrained VLM reward model.

  • World-Gymnast fine-tunes VLA policies by rolling them out in a learned world model and using a VLM for reward.
  • World-model rollouts and VLM rewards are described as cheap, scalable, and more robust to out-of-distribution scenarios than existing training paradigms.
  • World-Gymnast generates diverse training data from an initial scene and task description, supporting image editing, language variation, novel environments, test-time training, planning, and iterative improvement.
  • World-Gymnast may fail to generalize to arbitrary initial frames far from the world model’s training distribution.
  • A pretrained VLM can hallucinate task outcomes, producing suboptimal RL training and motivating improved reward models, dense rewards, and defenses against reward hacking.

Impact Statement

World-Gymnast aims to reduce dependence on costly hardware and manual simulator engineering while improving sim-to-real transfer. The paper cautions that generative video models can introduce hallucinated physics, requiring safety verification before deployment in safety-critical settings.

  • World-Gymnast trains robot policies within a generative world model to reduce dependence on expensive physical hardware and manual simulator engineering.
  • Generative video models can introduce hallucinated physics, so policies trained in these environments should undergo rigorous safety verification before safety-critical deployment.

Init:

The implementation accelerates world-model rollouts with KV caching and uses OpenVLA-OFT with a VLM-based binary success reward. Rollouts are scored from temporally downsampled generated videos, with the prompt explicitly warning about video-model artifacts.

  • KV caching reduces world-model rollout time by 10× when generating 20 trajectories in parallel over 40 frames.
  • OpenVLA-OFT is the base policy, providing parallel decoding, action chunking, and continuous actions for faster inference and learning.
  • GPT-4o serves as the reward model by judging whether a generated rollout completes the instructed task.
  • The VLM outputs a binary final score, where 0 denotes failure and 1 denotes success.
  • The reward prompt warns that video-prediction artifacts may appear because the rollout is generated by a video model conditioned on robot actions.
  • Each policy evaluation uses five independent rollouts, with videos downsampled at stride 3 before VLM scoring.

A.4. Details of Hyperparameters for World Model and RL Training

World-Gymnast uses a pretrained video world model to encode visual frames, represent robot actions, and train VLA policies with RL; baseline policies are additionally fine-tuned or evaluated in alternative settings.

  • World model: WorldGym encodes 256 × 256 RGB frames with a pretrained Stable Diffusion 3 VAE and models dynamics using a 16-layer transformer.The transformer has hidden dimension 1024 and 16 attention heads.
  • Action representation: WorldGym represents actions as 10-dimensional vectors, using seven dimensions for OpenVLA’s 6D end-effector pose and binary gripper state.Long rollouts use a sliding window of the most recent 20 frames.
  • RL training: RL training initializes each task from one observation frame and language instruction, rolls out up to 40 steps, and assigns binary completion rewards using GPT-4o.Actions are generated in chunks of five, and training uses GRPO.
  • Baselines: The SFT baseline fine-tunes OpenVLA-7B on expert trajectories from the Bridge V2 dataset.The recipe follows OpenVLA-OFT.
  • Baselines: Iterative SFT rolls out the base SFT policy in the world model on held-out scenarios, matching RL’s total rollout steps and filtering successful rollouts with a VLM.SIMPLER provides a digital twin of the Bridge setup for simulator-based RL comparison.
  • Baselines: Because base SFT produced low reward variance on SIMPLER, simulator training uses summed step rewards with partial credit for grasping and lifting before awarding success reward 1.Each listed partial condition receives reward 0.1.

C.1. OpenVLA Evaluation Task Set

The OpenVLA evaluation suite tests tabletop manipulation generalization across visual, motion, physical, semantic, and language variations, using altered scenes, instructions, and task diversity.

  • Task suite: The task suite assesses generalization across visual, motion, physical, and semantic variations, including language grounding in multi-object scenes.RL uses an 80/20 split sharing tasks but differing in initial frames.
  • Task suite: The evaluation task set includes tasks categorized by their corresponding generalization types.The supplied table caption identifies the task-to-generalization mapping but does not enumerate its rows.
  • Visual variation: Visual-robustness training adds diverse, task-irrelevant objects to initial frames while preserving the original task, goal, and robot configuration.Evaluation uses held-out distractor frames and original OpenVLA tasks in WorldGym.
  • Visual variation: The distractor-editing prompt requires the robotic arm to remain visible and the instructed task to remain achievable.The prompt adds distractors without moving or changing original scene objects.
  • Language variation: OOD language training changes only natural-language instructions while keeping scenes and objects unchanged.Four new instruction variants are mixed 50/50 with original training frames.
  • Task diversity: Task-diversity scaling augments the OpenVLA training split with five Bridge V2 manipulation tasks, including closing cabinets and flipping pots upright.The added tasks expand the training distribution beyond the original evaluation tasks.

C.6. Real-Robot Evaluation Dataset

Real-robot transfer is evaluated on AutoEval, which runs policies on WidowX arms across two scenes and three Bridge-distribution manipulation tasks using standardized rollouts and VLM-based success judgments.

  • Dataset and tasks: AutoEval evaluates two drawer tasks and one sink pick-and-place task across two scenes with a WidowX robot arm.The tasks are opening or closing the drawer and putting an eggplant in the blue sink.
  • Evaluation protocol: Policies are evaluated in WorldGym, a world-model simulator designed to approximate real-robot execution.This provides the rollout environment used before real-robot transfer evaluation.
  • Evaluation protocol: Each WorldGym rollout starts from an initial frame and language instruction and runs for up to 40 steps, producing predicted future frames for task assessment.The protocol evaluates task completion from the generated frame sequence.
  • Evaluation protocol: A VLM compares each rollout outcome with the task instruction, and five independent judgments determine binary success by majority vote.Repeated judgments reduce variance in VLM-based scoring.

D.2. Autoeval Evaluation

AutoEval evaluation compares World-Gymnast rollouts with base policies, examines varied language and initial scenes, and contrasts world-model training with SIMPLER simulator training.

  • AutoEval setup: AutoEval uses two WidowX setups, each supporting opposite operations such as opening and closing a drawer, with resets between trials.The reset policy restores conditions for the next evaluation trial.
  • World-Gymnast rollouts: Figure 4 compares RL-fine-tuned World-Gymnast rollouts with the pre-RL base policy on lifting a skull and putting an eggplant in a pot.The figure presents qualitative policy-rollout examples.
  • AutoEval rollouts: AutoEval qualitative examples cover closing and opening drawers and placing an eggplant in a blue sink or yellow basket.These examples span the two drawer operations and pick-and-place settings.
  • Qualitative evidence: The qualitative results supplement reported comparisons and illustrate World-Gymnast’s base-policy improvement, AutoEval behavior, diverse training, and novel-scene evaluation.The supplementary visualizations include WorldGym rollouts and slightly out-of-distribution AutoEval frames.
  • Diverse training: World-Gymnast is qualitatively evaluated on out-of-distribution language instructions involving pots, plates, corn, and a drying rack.The examples include moving a pot with grapes and placing yellow corn in a red cup.
  • Diverse training: Qualitative evaluations also test policies trained from novel initial image frames on drawer, fridge, pot, and sushi tasks.The examples include closing a fridge and taking sushi off a plate.
  • SIMPLER comparison: Figure 8 provides qualitative evaluations of a policy trained with RL in SIMPLER, whose training is described as less desirable for real-life deployment.The displayed examples include closing a drawer and putting an eggplant in a sink.
  • Cross-setting evaluation: WorldGym is also evaluated on AutoEval image frames that were not used during WorldGym training.The examples show putting an eggplant in a blue sink and opening a drawer.
Loading 2602.02454v1…