PaperlayerView exact revision on arXiv

Source-linked AI summary

Reinforcing Action Policies by Prophesying

Jiahui Zhang, Ze Huang, Chun Gu, Zipei Ma, Li Zhang

arXiv:2511.20633v2cs.RO

TL;DR

VLA imitation objectives are misaligned with long-horizon task rewards, while real-robot RL and conventional simulators are costly or difficult to transfer. The paper introduces Prophet, a few-shot-adaptable action-conditioned world model, and ProphRL with FlowScale for flow-based policy optimization. It reports 5–17% success gains on public benchmarks and 24–30% on real robots across diverse VLA variants.

  • Problem

    Imitation-heavy VLA training does not directly optimize long-horizon task rewards, while online RL is costly and conventional simulators face engineering and transfer challenges.

  • Method

    The paper pretrains Prophet on heterogeneous robot data, adapts it few-shot to new situations, and combines its rollouts with FA-GRPO and FlowScale for VLA post-training.

  • Results

    5–17% success gains on public benchmarks and 24–30% on real-robot evaluations are reported across diverse VLA variants.

  • Takeaways & Limitations

    The combined paradigm provides a data- and compute-efficient route to VLA post-training using a rollout-ready world simulator and stabilized flow-aware RL.

  • Takeaways & Limitations

    FlowScale’s theoretical analysis is approximate, and training dynamics also depend on clipping, KL regularization, and correlations between flow steps.

Abstract

from arXiv · show

Vision-Language-Action (VLA) policies excel in aligning language, perception, and robot control. However, most VLAs are trained purely by imitation, which overfits to demonstrations, and is brittle under distribution shift. Reinforcement learning (RL) directly optimizes task reward and thus addresses this misalignment, but real-robot interaction is expensive and conventional simulators are hard to engineer and transfer. We address both data efficiency and optimization stability in VLA post-training via a learned world model and an RL procedure tailored to flow-based action heads. Specifically, we first introduce Prophet, a unified action-to-video robot world model pretrained on large-scale, heterogeneous robot data to learn reusable action-outcome dynamics and then few-shot adapted to new robots, objects, and environments, yielding a rollout-ready simulator. Upon Prophet, we reinforce action policies with our proposed FlowScale, which couples Flow-GRPO with intrinsic stepwise reweighting to stabilize gradients. Together, our solution provides a practical, data- and compute-efficient path to VLA post-training. Experiments show 5-17% success gains on public benchmarks and 24-30% on real robots across diverse VLA backbones.

1 Introduction

VLA imitation training is misaligned with long-horizon task rewards, while existing robotics RL approaches face costly interaction and limited simulator transfer. The paper proposes Prophet and ProphRL, combining few-shot-adaptable world-model rollouts with FlowScale-based policy optimization.

  • 1 Introduction: Imitation-heavy VLA objectives do not directly optimize long-horizon task reward, making policies brittle under distribution shift and prone to accumulated errors.The motivation is to align post-training with task rewards rather than demonstration likelihood.
  • 1 Introduction: World-model rollouts provide scalable, action-conditioned practice without physical risk, while reusing the visual interface used by VLAs.This offers a middle ground between expensive online RL, engineered simulators, and offline RL without current-policy closed-loop data.
  • 1 Introduction: Existing world-model efforts often remain single-scene or function mainly as data augmentors, leaving general-purpose real-world simulator adaptation unresolved.The paper targets few-shot adaptation across embodiments, tasks, and scenes under realistic data and compute budgets.
  • 1 Introduction: Prophet is a history-aware, action-conditioned world model pretrained on heterogeneous robot data to generate long-horizon manipulation rollouts.It predicts action-to-video dynamics from first-frame observations and multi-step actions, then adapts to new environments, objects, and trajectories with few shots.
  • 1 Introduction: ProphRL combines FA-GRPO with FlowScale to stabilize and efficiently optimize VLA policies with flow-based action heads.FA-GRPO aligns ratios with environment-level actions, while FlowScale reweights per-step gradients according to the noise schedule.

2 Related Works

Prior work spans language- or low-level-signal-conditioned manipulation world models and policy-gradient methods for outcome-aligned post-training. Flow-based VLA action heads motivate specialized RL treatment beyond standard log-likelihood optimization.

  • World models for robot manipulation: Text-conditioned video models provide language and scene priors, but coarse action cues limit controllability for precise robot manipulation.More recent models condition on low-level signals such as end-effector poses or joint trajectories.
  • Related work: World-model research commonly emphasizes video quality or controllability, while VLA RL research emphasizes outcome-aligned policy optimization.These strands motivate combining adaptable action-conditioned simulation with specialized reinforcement learning.
  • RL for VLA policies: Recent VLA RL work typically optimizes log-likelihood objectives on image-conditioned action sequences, creating a need for methods tailored to flow-based action heads.The paper positions FA-GRPO and FlowScale within this flow-based policy setting.

3.1 Overall training paradigm

ProphRL uses Prophet to generate closed-loop visual rollouts for VLA policies and a reward model to score them. FA-GRPO then updates the policy with group-normalized advantages, while FlowScale reweights denoising steps.

  • 3.1 Overall training paradigm: At each outer step, the policy predicts an action chunk from the instruction and initial image, and Prophet generates a conditioned video clip.The generated clip is fed back to the policy and Prophet to enable long-horizon closed-loop rollouts.
  • 3.1 Overall training paradigm: A frozen VLM-based reward model scores each rollout and produces group rewards for policy optimization.The reward model uses a prompt template to evaluate the world-model-generated trajectories.
  • 3.1 Overall training paradigm: FA-GRPO optimizes the policy with group-normalized advantages, while FlowScale reweights denoising-step contributions.The design couples environment-level policy optimization with flow-step-specific gradient handling.

3.2 World model

Prophet is a latent video world model that predicts robot manipulation rollouts from visual observations and action sequences. It combines geometry-aware action conditioning with history memory to support long-horizon, action-faithful simulation and evaluation.

  • World-model foundation: Prophet models robot trajectories with a latent video diffusion pipeline that encodes clips, denoises noisy latents under conditioning, and decodes predicted videos.The conditioning feature includes the first-frame observation and action information.
  • Action representation: Actions are represented as time-indexed 7-dimensional per-end-effector commands containing translation, rotation, and normalized gripper opening.The action semantics are adapted to each environment’s low-level controller during fine-tuning.
  • Action representation: Prophet uses local delta poses to make motion actions more homogeneous across tasks and datasets, while preserving controller-specific interpretations during adaptation.Each delta specifies translation and rotation relative to the previous end-effector frame.
  • Action-frame construction: Action frames project end-effector geometry into camera images, rendering position, orientation, depth-aware size, and gripper opening as visual conditioning.The construction uses camera intrinsics and extrinsics, then renders the projected geometry on a black canvas.
  • Action conditioning: The model conditions on both a global scalar action embedding and an optional latent action-frame embedding injected into the DiT timestep representation.The scalar stream flattens each action chunk, while the action-frame stream is encoded and projected into DiT channels.
  • History-aware mechanism: A history-aware memory supplies long-range temporal context for stable geometry and contact evolution while keeping computation predictable.Past latent frames are compressed into memory key and value vectors used by the DiT attention blocks.
  • Action-faithful evaluation: The optical-flow protocol evaluates action conditioning through motion magnitude and direction, using endpoint error and flow-direction cosine rather than appearance alone.Near-static pixels are filtered before aggregating mean and median metrics across time.

3.3 RL-based VLA post-training

This section organizes flow-based VLA rollouts as environment-level actions and adapts policy-gradient optimization to aggregate internal flow steps. FlowScale then reweights those steps using their noise scales to balance gradient contributions without changing the action-level objective.

  • Trajectory layout: Rollouts use tensors shaped [B, S, K, CH, D], separating episodes, outer environment steps, denoising steps, action chunks, and action dimensions.Each policy call emits sequential low-level commands, with D = 7 in the experiments.
  • Trajectory masking: Variable-length episodes are padded to a fixed horizon, while masks remove policy-loss and advantage contributions after termination or invalid actions.The mask is broadcast across denoising steps and action dimensions.
  • FA-GRPO: FA-GRPO aggregates internal flow-step log-likelihoods into one action-level quantity and computes PPO ratios per action dimension for each environment action.Each pair (s, c) is treated as one environment action, while the internal denoising index k does not advance environment time.
  • FlowScale: FlowScale uses a normalized, mixed, and clipped stepwise weight derived from the flow noise schedule to modulate internal-step gradient contributions.The procedure preserves mean scale, mixes weights toward a uniform baseline, and bounds reweighting with [wmin, wmax].
  • FlowScale: FlowScale upweights earlier noisier steps and downweights later low-noise refinement steps, while stop-gradient treatment preserves the optimization target.The method changes relative gradient contributions rather than the underlying stochastic policy or action-level surrogate.
  • Theoretical rationale: The theoretical rationale approximates each per-step likelihood as Gaussian, under which smaller noise levels produce larger score norms.This analysis motivates correcting the non-uniform gradient magnitudes across flow steps.

3.4 Reward model

The reward-model pipeline converts trajectory outcomes into normalized chunk-level advantages for RL. It uses simulator labels where available and world-model-adapted VLM rewards when real-world or benchmark annotations are scarce.

  • Reward-to-advantage conversion: Trajectory-level reward-model scores are group-normalized and broadcast across each trajectory’s action chunks as the advantages used by FA-GRPO and FlowScale.The approach substitutes chunk-level advantages for the scalar GRPO advantage while retaining padding and termination masking.
  • RMs for LIBERO: LIBERO uses simulator rollouts with binary success labels and completion-step estimates to fine-tune a Qwen2.5-VL-7B binary reward model.The model receives task text and subsampled video frames resized to 224 × 224.
  • RMs for BRIDGE and real robots: For world-model rollouts, a second reward model is trained on world-model videos using simulator-derived success labels to bridge visual and dynamics differences.This semi-synthetic supervision is described as adequate but noisy, with training-signal stability varying across tasks.
  • RMs for BRIDGE and real robots: BRIDGE and real-robot settings use Qwen2.5-VL-72B zero-shot reward models that classify sampled trajectory frames with task-specific prompts.The real-world reward model outputs only binary success or failure, and short rollouts use an all-ones temporal mask.

4.1 Experimental setups

The experiments evaluate Prophet and ProphRL across heterogeneous robot data, simulator and real-robot settings, and four tabletop manipulation tasks. Real-world evaluation varies object configurations on fixed grid layouts while reporting repeated-run success statistics.

  • World-model setup: Prophet is initialized from Cosmos-Predict2-2B-Video2World, adds history-aware and dual-action conditioning, and predicts 20 frames from a first frame and 20-step action chunk.Gripper signals are normalized to [0, 1] across datasets.
  • World-model setup: Prophet is pretrained on over 31M sampled trajectories from heterogeneous robot datasets, including AgiBot, DROID, LIBERO, and curated Open-X subsets.The Open-X data are filtered for resolution, end-effector quality, and reliable pose or gripper-state annotations.
  • Real-world experiment setting: The custom real-robot dataset uses a UR30e arm and contains 800 trajectories across four tabletop tasks with fixed interfaces and varied initial object configurations.The tasks are GraspBottle, PlaceCube, PulloutTissue, and PlaceBowl.
  • Real-world experiment setting: Real-world tasks evaluate varied object placements on predefined tabletop grids, with each policy–task pair measured over three runs and 20 trials per run.Reported success rates are means and standard deviations across 3 × 20 trials per task.
  • Task details: The four tasks span grasping, placement, deformable-tissue extraction, and bowl placement, with task-specific sensitivity to grasp stability, alignment, and placement precision.PulloutTissue requires smooth extraction without tearing or dropping the tissue, while PlaceBowl uses a small plate support area.
  • Policy and training setup: The evaluated policies include VLA-Adapter-0.5B, Pi0.5-3B, and OpenVLA-OFT-7B, each producing 7D delta actions through a lightweight flow action head.Real-world RL uses groups of 8, total batch size 256, and 100 training steps on 8 H200 GPUs.

4.2 World model evaluation

Prophet generalizes action-conditioned manipulation across datasets and transfers effectively to unseen objects, scenes, and trajectories with few-shot fine-tuning. Its action-faithful rollouts and flow-based evaluation better reflect downstream task utility than conventional visual metrics.

  • 4.2.1 Evaluation of the pretrained Prophet: Pretrained Prophet already produces high-fidelity, physically plausible long-horizon manipulation on held-out trajectories, including unseen motion patterns and diverse scenes and object configurations.Qualitative rollouts show end-effector control, contact formation, and object-state changes across viewpoints.
  • 4.2.2 Fine-tuning on BRIDGE: Prophet achieves perfect success on all six challenging BRIDGE few-shot transfer tasks, closely reproducing conditioned actions and physical interactions on unseen scenes.The comparison uses identical ground-truth action sequences for all models, isolating action realization fidelity.
  • 4.2.2 Fine-tuning on BRIDGE: Prophet consistently matches or exceeds baselines in visual fidelity while yielding markedly better action consistency across BRIDGE same-task, cross-object, and full-data evaluations.The strongest gains occur in few-shot settings, while advantages remain across-the-board with abundant data.
  • 4.2.3 Flow-guided evaluation: Optical-flow metrics correlate more strongly with downstream VLA success than PSNR and SSIM, making flow-guided evaluation a more faithful measure of action-conditioned world-model quality.The gap between Prophet and baselines is larger in EPE and flow-direction cosine than in PSNR and SSIM.
  • 4.2.4 Ablation studies on Prophet components: On LIBERO, pretraining and the history-aware mechanism provide consistent gains, while adding action-frame conditioning further improves action consistency despite a small PSNR drop.The trade-off favors faithful action execution for the downstream objective.
  • 4.2.1 Evaluation of the pretrained Prophet: Calibrated failure rollouts enrich downstream RL with informative negative examples and reduce reliance on overly optimistic dynamics.Prophet generates realistic failures such as stopping short of targets, drifting after contact, and missed grasps under perturbed actions.

4.3 RL with world models across simulators and real-robot

ProphRL uses Prophet rollouts with FA-GRPO and FlowScale to improve VLA policies across simulator, few-shot, iterative, multitask, and real-robot settings. Gains persist under severe data scarcity, transfer across world models and simulators, and remain positive despite learned-rollout drift and reward-model bias.

  • 4.3.2 Iterative RL with low-reward scene initialization: Iterative low-reward scene initialization increases PutCarrot success from 29.2 after SFT to 54.2 over three RL rounds, a total gain of 25.0 points.Later rounds focus rollouts on remaining failure cases.
  • 4.3.5 Multi-task RL with Prophet: Multi-task RL with Prophet consistently improves grasp metrics and full-task success across three VLA variants without requiring task-specific RL runs.Policies are jointly optimized on four SimplerEnv-WidowX tasks.
  • 4.3.3 World model choice and fine-tuning data: Prophet improves success over Cosmos-Predict2 under identical RL settings, indicating that more action-faithful rollouts provide better policy-training signals.Checkpoint gains correlate strongly with flow-based action-faithfulness metrics, including Spearman ρ = 0.9370 for 1/EPE.
  • 4.3.1 Single-task RL with Prophet: World-model RL raises overall few-shot success from 23.3 to 41.0 with 100 training images and reaches 34.7 with only 10 images over the SFT-only baseline.Improvements occur in grasp and full-task success across four SimplerEnv-WidowX tasks.
  • 4.3.4 Evaluation on real robots: 24–30% success gains occur on UR30e real-robot tasks across VLA variants using 20 image snapshots and 100 RL updates, versus 50k SFT steps.The policies are refined from Prophet rollouts in a low-data regime.
  • 4.3.6 Behavior analysis: RL amplifies rare successful behaviors and reshapes action distributions through diverse closed-loop rollouts, including right-side grasps and improved soft-object approach strategies.Prophet exposes both successful and failed trajectories for reward-based policy updates.
  • 4.3.3 Evaluation on LIBERO: Prophet gains are smaller than simulator gains because long rollouts accumulate geometric and contact drift alongside learned-reward-model bias, so Prophet complements rather than replaces high-fidelity simulators.Despite this noisier regime, the rollouts still provide training signals that improve policy performance.
  • 4.3.7 FlowScale sensitivity: FlowScale improves LIBERO Spatial SuccGain from 9.7 to 11.9 and reaches peak validation performance 3.4× faster than FA-GRPO alone.A tighter clipping choice reaches the peak in 49 updates, corresponding to an 8.3× speedup.

4.4 Reward model discussion

Reward-model feedback can improve VLA reinforcement learning when recall and precision remain high, but late precision collapse or low recall makes optimization misaligned with true success. These diagnostics motivate monitoring reward-model quality and selecting reward sources carefully.

  • Reward-source and RM comparisons: Reward-source comparisons show that the ground-truth simulator reward performs best, while the 72B RM still improves substantially over SFT as a usable but imperfect signal.Larger reward models perform better, and voting improves reliability at the cost of latency and GPU hours.
  • High-recall RM, long run: Ground-truth success improved from roughly 0.55 to 0.7 during the first 300–350 updates despite FPR rising from ∼0.08 to ∼0.3.Precision stayed ≈0.85–0.9 and recall remained almost perfect at ≈0.98, allowing FA-GRPO to exploit the reward-model ranking.
  • High-recall RM, long run: After about 300 updates, success stopped improving and declined as precision dropped while FPR remained high.The reward model increasingly labeled failures as successes, misaligning its gradients with the true task.
  • High-recall RM, short run: With the same high-recall RM stopped after 200 updates, success improved monotonically while precision stayed above ∼0.85 and recall remained close to 0.98.FPR grew to ∼0.35, indicating that moderate false positives were tolerable while successful trajectories remained well recognized and RM positives stayed reasonably clean.
  • Diagnostics setup: Figure 15 compares held-out true-reward success, RM-based success, precision, recall, and FPR across three reward-model training regimes.The regimes vary by checkpoint mixture, trajectory count, and rollout duration, while RL uses only RM-based rewards.
  • Low-recall RM, short run: The low-recall RM produced no success improvement, with ground-truth success oscillating around 0.5 despite low FPR and precision near ∼0.8.Recall fluctuated around 0.7–0.8, so many successful trajectories received no advantage and the policy obtained weak preference information.

5 Conclusions

The paper presents Prophet as an adaptive action-conditioned world model for long-horizon VLA post-training and combines it with FlowScale for stabilized reinforcement learning. Across diverse VLA variants, the approach improves success on both public benchmarks and real-robot evaluations, while its current rollout cost limits feasible iteration counts.

  • Results: Success gains reach 5–17% on public benchmarks and 24–30% on real-robot evaluations across diverse VLA variants.These results summarize the reported performance of the Prophet-based post-training approach with FlowScale.
  • Method: Prophet generates long-horizon, action-aligned manipulation rollouts from first-frame observations and multi-step actions, transferring across robots, objects, and environments after few-shot adaptation.It is pretrained at scale and used as an adaptive simulator for VLA post-training.
  • Method: FlowScale is a flow-aware GRPO variant that stabilizes gradients for long-horizon reinforcement learning within the world-model loop.The conclusion describes it as the optimization component paired with Prophet.
  • Limitation: The current system is computationally demanding because closed-loop RL rollouts require interaction with a 2B-parameter Prophet.This rollout cost dominates training and limits feasible iterations.
Loading 2511.20633v2…