PaperlayerView exact revision on arXiv

Source-linked AI summary

VLAW: Iterative Co-Improvement of Vision-Language-Action Policy and World Model

Yanjiang Guo, Tony Lee, Lucy Xiaoyang Shi, Jianyu Chen, Percy Liang, Chelsea Finn

arXiv:2602.12063v2cs.RO

TL;DR

Real-world VLA post-training is constrained by expensive rollout collection and world models that lack fidelity for diverse, contact-rich physical interactions. VLAW uses online rollout data to improve an action-conditioned world model, then generates synthetic trajectories for policy improvement, consistently improving performance across multiple contact-rich manipulation tasks.

  • Problem

    Real-world policy rollouts require substantial human labor and are expensive and time-consuming, while existing world models lack physical fidelity for diverse interactions and contact-rich manipulation.

  • Method

    VLAW iteratively fine-tunes an action-conditioned world model on online rollout data, then uses the improved model to generate and filter synthetic trajectories for VLA policy optimization.

  • Results

    VLAW consistently improves performance across multiple contact-rich manipulation tasks and achieves substantially larger gains across all tasks than filtered behavioral cloning.

  • Takeaways & Limitations

    Online rollout data can ground a world model sufficiently to support high-fidelity synthetic trajectories for improving real-world VLA policies.

  • Takeaways & Limitations

    Flow-matching VLA objectives do not provide explicit policy likelihoods, making conventional policy-gradient methods difficult to apply.

Abstract

from arXiv · show

The goal of this paper is to improve the performance and reliability of vision-language-action (VLA) models through iterative online interaction. Since collecting policy rollouts in the real world is expensive, we investigate whether a learned simulator-specifically, an action-conditioned video generation model-can be used to generate additional rollout data. Unfortunately, existing world models lack the physical fidelity necessary for policy improvement: they are predominantly trained on demonstration datasets that lack coverage of many different physical interactions (particularly failure cases) and struggle to accurately model small yet critical physical details in contact-rich object manipulation. We propose a simple iterative improvement algorithm that uses real-world roll-out data to improve the fidelity of the world model, which can then, in turn, be used to generate supplemental synthetic data for improving the VLA model. In our experiments on a real robot, we use this approach to improve the performance of a state-of-the-art VLA model on multiple downstream tasks. We achieve a 39.2% absolute success rate improvement over the base policy and 11.6% improvement from training with the generated synthetic rollouts. Videos can be found at this anonymous website: https://sites.google.com/view/vla-w

1. Introduction

VLAW addresses the cost and limited scalability of real-world VLA rollouts by grounding an action-conditioned world model with online data, then using it to generate synthetic trajectories for policy improvement. The framework targets the physical-fidelity gaps of existing world models and improves a pretrained VLA policy on contact-rich manipulation tasks.

  • Real-world policy rollouts require substantial labor for environment resets and execution monitoring, limiting the data available for VLA post-training.These constraints make online interaction expensive and time-consuming.
  • Existing action-conditioned world models often lack physical fidelity because demonstration data underrepresent diverse interactions, especially failures, and omit critical contact details.They can also produce blurry predictions and unreliable synthetic data for collisions or deformable objects.
  • VLAW fine-tunes a pretrained action-conditioned world model on online rollout data, including failures, to model success and failure dynamics more accurately.The grounded model then generates large-scale, high-fidelity synthetic trajectories automatically annotated by a vision-language reward model.
  • VLAW improves a pretrained π0.5 policy across contact-rich deformable-object tasks by using world-model-generated synthetic data with stable supervised learning objectives.The experiments use the DROID real-robot platform and a multi-task setup.

2. Related Works

Prior work uses supervised fine-tuning, online reinforcement learning, and action-conditioned world models for VLA improvement, but real-world rollout requirements and likelihood limitations constrain standard approaches. VLAW grounds a pretrained world model with online policy data so it can support synthetic rollouts for policy learning.

  • VLA policies are commonly trained on large-scale data and then supervised-fine-tuned on target tasks, with online rollout data emerging as another post-training resource.
  • Standard on-policy reinforcement learning requires many rollouts and is mainly validated in simulation, while flow-matching VLAs lack explicit policy likelihoods for conventional policy gradients.
  • VLAW instead performs iterative policy improvement with real-world and world-model-generated batches, updating the policy exclusively through stable supervised fine-tuning objectives.
  • Action-conditioned world models predict future outcomes from current observations and actions and are used for model-based reinforcement learning and visual planning.
  • Online policy rollout data ground pretrained world models in new environments and improve accuracy around the policy’s state-action distribution.

3. Preliminaries

The problem is formulated as multi-task robotic manipulation with language-conditioned policies and action-conditioned world models. Real trajectories provide online training data, while the policy and world model interact in a closed loop to generate imagined trajectories.

  • Each language-specified task is modeled as an MDP with state space, action space, transition dynamics, task reward, and discount factor.
  • A pretrained VLA policy maps the current state and instruction to an action distribution, while the world model predicts the next state from the current state and action.
  • Online real-world trajectories contain states and actions and receive task-level success or failure rewards.
  • Imagined trajectories start from an initial state sampled from a real trajectory and are generated autoregressively through closed-loop policy-world-model interaction.

4. Co-Improvement of VLA and World Model

VLAW alternates real-world rollouts, world-model and reward-model updates, and synthetic-rollout policy training to co-improve the world model and VLA policy. The procedure targets physical-interaction failures by adapting the world model to online data while retaining broad DROID coverage.

  • 4.1. World Model Learning with Real Roll-outs: Real-world policy rollouts capture diverse physical interactions, including both success and failure cases, for world-model finetuning.The method initializes from Ctrl-World and trains on online rollout data using the original diffusion objective.
  • 4.1. World Model Learning with Real Roll-outs: The world model is co-trained on progressively accumulated online data and the original DROID dataset to limit overfitting.The regularization strength is controlled by λ.
  • 4.1. World Model Learning with Real Roll-outs: The pipeline finetunes a vision-language reward model on real-world success labels and uses a yes-token probability threshold to identify successful trajectories.A threshold of 0.8 makes the reward model more conservative when labeling generated success.
  • 4.2. Iterative Improvement for VLA Policy: The updated world model generates synthetic trajectories, which are filtered for success before VLA policy finetuning.The procedure alternates world-model and policy updates across iterations.
  • 4.2. Iterative Improvement for VLA Policy: The VLA policy is updated with a weighted flow-matching objective over real and generated data, assigning weight 1 to successful and 0 to failed transitions.The flow-matching loss is defined over observation–action pairs.
  • 4.3. Relation to Regularized Reinforcement Learning: VLAW frames its policy update as an approximate regularized reinforcement-learning optimization that keeps the learned policy close to a reference policy while optimizing reward.The corresponding update uses a weighted regression objective derived from the regularized optimum.

5. Experiments

Experiments evaluate VLAW on complex real-world manipulation tasks, testing world-model fidelity, synthetic-data benefits, and iterative policy improvement. Online rollout data improves contact-rich prediction, while generated trajectories support stronger policy gains than non-world-model baselines.

  • Experimental Settings: Experiments on DROID evaluate VLAW across five categories of contact-rich manipulation tasks involving collisions and deformable objects.The setup uses a Franka Panda arm with a Robotiq gripper, two third-person cameras, and one wrist-mounted camera.
  • World-Model Evaluation: World-model fidelity is assessed with video distance metrics and an interaction-event confusion matrix comparing predicted success or failure with real-world outcomes.Video metrics are computed on 256 five-second replayed clips, while the event-level matrix uses 50 interaction clips.
  • World-Model Evaluation: Online rollout finetuning substantially improves all video-quality metrics and reduces false-positive interaction predictions relative to pretrained and expert-only-finetuned baselines.Mixed success and failure trajectories largely eliminate the over-optimistic bias associated with expert demonstrations alone.
  • World-Model Evaluation: The post-trained world model maintains visual fidelity and physical plausibility for policy-in-the-loop rollouts lasting up to 20 seconds.This stability supports searching for successful trajectories within the world model.
  • Policy Improvement: Imagined trajectories can provide supervision for failure cases, including real-world failures to grasp a scoop or draw a complete circle.The world model searches successful trajectories from the same real-world initial frame.
  • Policy Improvement: VLAW consistently outperforms filtered BC and DSRL across tasks by generating large-scale synthetic rollouts and selectively filtering successful trajectories.Two rounds of iterative training are performed, with “Ours-1” denoting the method after the first online-rollout round.

6. Conclusions and discussions

VLAW jointly improves the VLA policy and action-conditioned world model, achieving consistent gains on multiple contact-rich manipulation tasks. The evaluation covers five task categories, while broader and more diverse task scaling remains future work.

  • Conclusions: VLAW jointly enhances the VLA policy and action-conditioned world model through an iterative improvement pipeline.The paper reports consistent performance improvements across multiple contact-rich manipulation tasks.
  • Limitations: The current evaluation is limited to five task categories, making broader and more diverse online rollout data a future direction.The authors identify scaling online rollout data across tasks as promising future work.

A. Relation to Regularized Reinforcement Learning.

The policy update is framed as projecting an improved target policy into the finite parametric VLA policy class. Because flow-matching policies lack tractable action likelihoods, the projection uses a flow-matching-compatible surrogate and weighted dataset samples.

  • The regularized RL formulation constrains the learned policy to remain close to a reference policy while optimizing reward.
  • The optimal improved policy generally cannot be represented by a finite parametric class, so policy improvement uses a divergence-minimizing projection onto πθ.
  • Standard AWR uses KL divergence and weighted log-likelihood, but this is not directly applicable because the VLA policy uses flow matching without explicit action likelihoods.
  • The proposed surrogate divergence measures how well πθ matches samples from the improved target policy under the flow-matching loss.
  • The projection replaces target-policy samples with exponentially advantage-weighted samples from a fixed dataset.
  • With γ →1 and large negative rewards for failures, the regularized RL derivation reduces to the objective used for the policy update.

B. Task Details

The evaluation covers five manipulation tasks using outcome-based success criteria judged from final states or short post-action observations. Success rates are reported across repeated evaluations, with most tasks evaluated 50 times per iteration.

  • Success is defined using simple outcome-based criteria that can be judged from the final state or a short post-action observation window.
  • Task Details: Stacking succeeds when block A is stably supported by block B rather than the table and remains upright briefly.
  • Task Details: Open Book succeeds when the front cover exceeds a predefined opening angle and remains open at episode end.
  • Task Details: Erase Marks succeeds when no clearly detectable marker strokes remain on the whiteboard.
  • Task Details: Scooping succeeds when at least a minimum amount enters the bowl, with most transferred items inside rather than spilled.
  • Task Details: Drawing succeeds when the robot produces one visually complete closed circle within the designated whiteboard region.
  • Task Details: All tasks are evaluated 50 times because each iteration collects 50 online rollouts, while DSRL is evaluated 10 times because online updating is time-consuming.

C. Reward Model Details

The reward-model pipeline fine-tunes a vision-language model and uses a confidence threshold instead of direct binary prompting. The threshold reduces false positives and provides more reliable supervision for policy learning.

  • Each trajectory is downsampled to a 16-frame video and evaluated with a fine-tuned Qwen3-VL-4B-Instruct reward model.The model is fine-tuned for 200 steps with batch size 128.
  • A trajectory is labeled successful only when the model-assigned probability of the “yes” token exceeds 0.8.This threshold makes the generated success labels more conservative than direct binary prompting.
  • Higher confidence thresholds substantially reduce false-positive trajectories compared with naive binary querying, improving the reliability of downstream policy-learning supervision.
  • The comparison uses human-annotated ground-truth labels on a manually labeled subset of 40 trajectories.
Loading 2602.12063v2…