PaperlayerView exact revision on arXiv

Source-linked AI summary

WCM: A World Critic Model for Vision-Language-Action Reinforcement Learning

Senyu Fei, Xiaopeng Yu, Siyin Wang, Xianzhong Zhao, Jingjing Gong, Xipeng Qiu

arXiv:2607.29613v1cs.ROcs.CLcs.CV

TL;DR

Critic-based VLA reinforcement learning typically relies on single-frame or weakly supervised history representations, leaving temporal state information under partial observability insufficiently captured. WCM jointly predicts future latent dynamics and values, achieving state-of-the-art performance across 149 simulation tasks and stable results on seven real-world manipulation tasks.

  • Problem

    Under partial observability, existing VLA-RL critics mainly use single-frame inputs or weakly supervised history embeddings, limiting temporally informed value estimation.

  • Method

    WCM jointly predicts future latent dynamics and estimates values, training a unified critic representation to encode task-relevant temporal structure.

  • Results

    Across 149 tasks and four simulation benchmarks, WCM consistently achieves state-of-the-art performance and strong OOD generalization, with stable results on seven real-world tasks.

  • Takeaways & Limitations

    WCM provides a unified critic approach that transfers across VLA backbones, training pipelines, simulation benchmarks, and diverse real-world manipulation tasks.

Abstract

from arXiv · show

Reinforcement learning (RL) post-training of Vision-Language-Action (VLA) models has shown strong promise for robotic manipulation. Among RL methods, critic-based approaches rely on a value estimator that predominantly operates on single-frame observations or single-frame VLM backbone latents, which is a fundamental mismatch with the partially observable nature of robot control. A naive approach to incorporate observation history into the critic incurs exponential complexity with high-dimensional visual space, and still fails because pure scalar-return regression provides insufficient supervision for learning cross-temporal dynamics. We identify the root cause as a state approximation problem: without an explicit world modeling objective, the critic's representation cannot capture the temporal structure needed for accurate value estimation. To address this, we propose the World Critic Model (WCM), built on a lightweight LeJEPA architecture; WCM jointly predicts future latent state and estimates values, such that the critic's representation is explicitly trained to capture temporal dynamics rather than merely regress scalar returns. WCM integrates seamlessly into both on-policy and off-policy training pipelines and is compatible with state-of-the-art VLA backbones including Pi0, Pi0.5, and OpenVLA-OFT. Extensive experiments on 149 tasks across four benchmarks demonstrate that WCM consistently achieves state-of-the-art performance in both in-distribution and out-of-distribution settings, with particularly strong generalization gains. We further validate WCM on seven real-world manipulation tasks using OpenVLA-OFT and Pi0.5 with off-policy RL, confirming stable deployment across diverse settings.

1 Introduction

Critic-based VLA-RL is limited by partial observability: single-frame or weakly supervised history representations miss temporally informative state. WCM addresses this bottleneck by jointly predicting future latent dynamics and estimating values, achieving strong simulation and real-world results.

  • Problem: Robotic manipulation is a POMDP, so single-frame critics can miss motion, contact progress, and possible future evolution needed for value estimation.Optimal decisions depend on a sufficient statistic of observation history rather than the instantaneous observation alone.
  • Problem: Frame stacking and temporal aggregation do not reliably solve partial observability because scalar value regression weakly supervises cross-temporal dynamics in high-dimensional observations.Even Vision Transformer temporal critics with positional encoding struggle to improve performance.
  • Method: WCM unifies world modeling and critic learning by jointly predicting future latent dynamics and estimating values from historical observations.Joint optimization trains a critic representation to better approximate a compact, updatable predictive state rather than merely regress returns.
  • Results: 149 tasks across four manipulation benchmarks show that WCM consistently outperforms existing methods with diverse backbones and demonstrates strong OOD generalization.The experiments validate WCM across both in-distribution and out-of-distribution settings.
  • Results: Seven real-world RL tasks show that WCM surpasses standard VLM-critic baselines with better performance and smoother operation.These results validate WCM beyond simulation in real-world manipulation.

2 Related Works

VLA-RL has emerged as a promising paradigm for robot manipulation, but its critic models remain mismatched with the partially observable nature of control because they typically rely on single-frame observations.

  • VLA-RL: VLA models are a promising robot-manipulation paradigm, pretrained on large-scale robot manipulation and image-text data before task-specific post-training.These methods show strong performance and promising generalization.
  • Critic Models in VLA-RL: Critic models provide dense supervision and improve sample efficiency in both on-policy and off-policy VLA-RL methods.Critics play a central role across both training settings.
  • Critic Models in VLA-RL: Existing critics typically regress values from single-frame observations, implicitly assuming one frame suffices to reconstruct the system state.This assumption is problematic because VLA-RL is partially observable.

3 Methodology

WCM models VLA manipulation as a partially observable control problem and combines history-conditioned world prediction with value estimation in a lightweight LeJEPA-based critic. Its end-to-end objective uses next-latent prediction, SIGReg regularization, and return regression, while supporting both on-policy and off-policy training pipelines.

  • RL Integration: WCM serves as the critic in both on-policy PPO or Flow-SDE training and off-policy AWR or flow-matching training.Off-policy iterations combine teleoperation SFT data with erroneous rollouts and typical failure cases to improve value estimation and reduce overly optimistic predictions.
  • Problem Formulation: VLA manipulation is formulated as a POMDP in which observations partially reveal hidden states before the policy selects actions.Observations may include images, proprioception, and language.
  • Model Architecture: WCM encodes each observation in a K-frame history, conditions the sequence on language, and uses a causal Transformer to represent temporal dynamics.The encoder can be a ViT or the underlying VLA model’s VLM backbone.
  • Model Architecture: Two decoder heads estimate returns and predict the action-conditioned next latent state, unifying value estimation with world modeling.The world decoder uses an action encoder and gated FiLM residual blocks.
  • Training Objective: The training objective combines teacher-forced next-latent prediction, SIGReg, and L2 value loss, with all components trained end-to-end.SIGReg matches random one-dimensional projections to a standard Gaussian to prevent dimensional collapse and mode degeneration; returns are min-max normalized to [−1, 1].

4 Experiments

Across 149 simulation tasks and seven real-world robotic tasks, WCM consistently improves manipulation performance, generalization, and practical RL training efficiency. Its gains extend to long-horizon and distribution-shifted settings, while world prediction helps exploit history for value estimation.

  • Simulation: WCM achieves state-of-the-art performance across ManiSkill, MetaWorld, and CALVIN simulation benchmarks.The simulation evaluation covers 149 tasks across four benchmarks, including in-distribution and out-of-distribution settings.
  • Simulation: 252% improvement is reported for OpenVLA-OFT over weak SFT baselines on ManiSkill.The result highlights substantial gains from low-performing initial policies.
  • Simulation: 12,551% improvement is achieved when OpenVLA-OFT starts from no ManiSkill exposure with an initial performance of 0.78%.WCM produces stable improvement even from extremely low initial performance.
  • Generalization: WCM improves IND and OOD performance, outperforming Flow-SDE, PPO, and the strong-OOD 𝜋-StepNFT baseline.The gains are attributed to richer state information and robust value estimation under distribution shifts.
  • Generalization: After about 250 RL training steps from one-shot SFT, WCM surpasses full-shot SFT trained on 20k trajectories in LIBERO-Plus.“One-SFT” uses one demonstration per task, whereas “Full-SFT” uses 50 demonstrations per task.
  • Real-world evaluation: On seven physical-robot tasks, WCM enables rapid refinement with hundreds to a few thousand trajectories and less than one hour of training.The real-world evaluation uses 𝜋0.5 and OpenVLA-OFT with off-policy RL across dynamic, deformable, long-horizon, and pick-and-place tasks.

5 Analysis

The analysis shows that explicit world prediction is necessary for robust history-aware critics: WCM outperforms alternative critic architectures across configurations, with three-frame history performing best on average. The results suggest that modeling temporal dynamics, rather than merely adding frames, is central to critic performance.

  • Critic architecture ablation: Adding more observation history to the original MLP critic may produce suboptimal performance, whereas history-based ViT remains insufficient without world prediction.The comparison evaluates π0, π0.5, and OpenVLA-OFT on ManiSkill and MetaWorld using 2–5 frames.
  • Critic architecture ablation: WCM achieves the highest success rates across all critic-architecture and observation-history configurations in the ablation study.MLP is the baseline critic, while ViT denotes WCM without world prediction (λ = 0) but with temporal modeling capability.
  • Observation history length: Three-frame observation history delivers the best average performance when WCM history length varies from 1 to 5.The authors hypothesize that three consecutive frames may capture second-order dynamics, while two frames capture first-order dynamics.

6 Conclusion

The conclusion identifies a temporal-structure limitation in existing critic-based VLA-RL methods under partial observability and presents WCM as a unified solution. WCM jointly learns latent state prediction and value estimation, with experiments conducted across 149 tasks and four simulation benchmarks.

  • Limitation: Existing critic-based VLA-RL methods fail to capture temporal structure needed for state reconstruction under partial observability.The limitation arises when value estimation uses single-frame observations or weakly supervised history.
  • Proposed method: WCM is a unified architecture that jointly learns latent state prediction and value estimation.The model is proposed to address the temporal-structure limitation in existing methods.
  • Evaluation: 149 tasks across four simulation benchmarks were used for extensive experimental evaluation.

A How does the weight allocation of training objectives affect performance?

The weight λ balances world prediction against value regression, with intermediate settings delivering the strongest IND and OOD performance. OOD performance benefits more from world prediction and is substantially more sensitive to λ than IND performance.

  • Weight allocation: When supervision is weak, OOD performance is comparable to the original πRL baseline even when IND performance remains reasonable.With strong supervision, OOD performance remains competitive.
  • Weight allocation: λ balances world prediction and value learning, with the best IND and OOD performance both occurring within [0.3, 0.5].Neither extremely large nor small λ yields optimal results, indicating a stable intermediate range.
  • Weight allocation: λ = 0.9 achieves better OOD performance than λ = 0.1, indicating that world prediction contributes positively to generalization.λ = 0.9 is dominated by world prediction, whereas λ = 0.1 is dominated by value regression.
  • Weight allocation: 10.6 percentage points of fluctuation occur in OOD performance as λ varies, compared to 2.7 percentage points in IND.OOD performance is therefore more sensitive to λ than IND performance.

B Can the trained policy achieve sim-to-real transfer? … D.1 Algorithms

The experiments show that RL enables meaningful sim-to-real transfer and broader object generalization, while critic choice affects OOD training dynamics. The paper then specifies critic-based on-policy and off-policy pipelines, including PPO, Flow-SDE, and WCM updates.

  • B Can the trained policy achieve sim-to-real transfer?: RL achieves successful real-world carrot pick-and-place where the simulation-trained SFT model fails to grasp any object.The RL checkpoint used 16,800 SFT samples followed by 285 RL steps, with 18,240 additional simulated samples.
  • B Can the trained policy achieve sim-to-real transfer?: After 50 real-world SFT samples, success reaches only 6.9% in simulation, contrasting with simulation SFT’s failure to complete a real-world pick-and-place task.The comparison highlights a mismatch between simulation and real-robot data rather than a simple advantage for either training domain.
  • B Can the trained policy achieve sim-to-real transfer?: After 1,000 simulation RL steps, carrot performance slightly degrades while grasping performance on bananas and peppers significantly improves.The policy is trained exclusively on carrot grasping and placement, yet retains non-zero success probability on the other objects.
  • C Does Critic Model Affect Generalization?: Flow-SDE exhibits a later-stage OOD performance drop, while zero-value ablation performs better at comparable IND levels.These results suggest critic-related overfitting, whereas WCM shows no overfitting during the first 500 steps without being claimed entirely immune.
  • D Details for On-policy and Off-policy Training: The training pipelines use PPO for autoregressive on-policy learning, Flow-SDE for flow-matching on-policy learning, and AWR for off-policy learning.SIGReg is omitted from the on-policy pipeline because constraining the VLM latent would add unnecessary computational overhead.
  • D.1 Algorithms: Algorithm 1 jointly updates the policy and WCM using trajectory data collected from the policy and WCM.Inputs include policy πθ, WCM Wϕ, language instruction ℓ, loss weight λ, learning rate α, optimization steps U, batch size B, and history length K.
  • D.1 Algorithms: PPO with WCM computes the policy-gradient loss, estimates advantages with GAE, forms the total critic loss, and updates both models by stochastic gradient descent.The probability ratio is defined as ρ_t(π) = π(a_t|o_t) / π_old(a_t|o_t).

D.2 Training Details

Training uses on-policy RL in simulation and off-policy RL on real robots, with WCM integrated into established VLA training pipelines. Experiments follow matched baseline settings while covering multiple benchmarks and seven physical manipulation tasks.

  • On-policy training in simulation: Simulation uses on-policy Flow-SDE training improved by WCM for the π[1] [10] series models.Flow-SDE combines DPPO and Flow-GRPO for VLA adaptation, with emphasis on engineering practicality and reproducibility.
  • On-policy training in simulation: ManiSkill experiments exactly match RL4VLA settings across 25 pick-and-place tasks and in-distribution and out-of-distribution evaluations.The OOD setting perturbs tasks across three dimensions.
  • On-policy training in simulation: 64 trajectories per update and 1,000 training steps expose each model to 64,000 trajectories during the RL phase.These settings use RLinf’s officially provided hyperparameters without additional tuning for fair comparison.
  • On-policy training in simulation: MetaWorld and CALVIN evaluations retain RLinf hyperparameters, reporting final success rate and average task completion length, respectively.MetaWorld broadens task variety, while CALVIN evaluates long-horizon capabilities.
  • Off-policy training in real-world: Real-world training uses off-policy AWR for OpenVLA-OFT and RECAP for π0.5 because on-policy rollouts are costly and risk hardware damage.The two pipelines are largely identical, differing primarily in their policy-update loss functions.
  • Off-policy training in real-world: Real-robot experiments train WCM separately on seven tasks using a WidowX-250S, third-person and wrist-camera views, proprioception, and teleoperated data.Each task begins with 100 collected trajectories; control uses delta end-effector pose.

D.3 Inference Throughput · D.4 Traning Curve

Inference throughput is lowest for SFT-only, improves after RL fine-tuning, and is highest among RECAP variants with the WCM-based critic. Training curves compare eight settings from step 0 to optimal reported performance, revealing differing convergence behavior and test-setting discrepancies for Maniskill-OOD and LIBERO-Plus.

  • D.3 Inference Throughput: Throughput measures successful rollouts per hour across towel folding, stovetop cleaning, and rotating sushi picking.Results are reported in Figure 10.
  • D.3 Inference Throughput: The SFT-only model has the lowest throughput across all three tasks.Its low initial success rate and inefficient execution cause frequent pauses and small-magnitude movements that prolong rollouts.
  • D.3 Inference Throughput: Among RECAP variants, the WCM-based critic consistently outperforms the Gemma VLM-based critic across all tasks.This indicates that critic choice substantially affects inference efficiency.
  • D.3 Inference Throughput: RL fine-tuning significantly improves throughput by making the policy more successful and its actions smoother.The resulting rollouts are less prolonged than those of the SFT-only model.
  • D.4 Traning Curve: Training curves cover eight settings from step 0 until each setting reaches its reported optimal performance.Most settings improve stably, but convergence speeds and final performance vary across configurations.
  • D.4 Traning Curve: For Maniskill-OOD and LIBERO-Plus, numerical discrepancies arise because test settings differ from training settings.This qualification accompanies the training-curve comparisons.

E Value Curve Visualization of WCM

WCM value curves are visualized on successful and failed trajectories in simulation and real-world tasks. In simulation, WCM shows strong discrimination, while real-world curves are less strictly monotonic because of environmental imperfections and suboptimal actions.

  • Simulation: 7k simulation trajectories combine 5k successes with 2k failures, including 1k random EEF movements and 1k planner-perturbed near-success trajectories.A single WCM is trained on all trajectories and evaluated on unseen trajectories.
  • Real-world tasks: 500 trajectories per real-world task include successes and failures, but successful trajectories are not strictly monotonic because collected actions can be suboptimal.Value stagnation or decline is primarily associated with factors such as pauses during grasping and table collisions.
  • Simulation: WCM exhibits strong discriminative capability between successful and failed simulation trajectories, with exceptionally smooth estimates for ideal planner-generated successes.The simulation results are shown in Figure 12.

F Additional Generalization Analysis of the Critic Model

This analysis evaluates whether WCM trained on in-distribution data preserves reasonable value distributions out of distribution. Heatmaps show that WCM retains value discriminability and reduces local outliers compared with the original method after scene generalization.

  • Real-world value curves: In real-world trajectories, successful and failed behaviors remain highly discriminable despite visual noise and trajectory fluctuations.Teleoperated trajectories, environments, and cameras introduce confounding factors and visual noise, causing fluctuations even in successful trajectories.
  • Experimental setup: The analysis compares π0.5 trained with Flow-SDE and WCM augmentation against original Flow-SDE before evaluation on LIBERO-Plus and the LIBERO Object Suite.The reported training setup uses 200 steps for the LIBERO experiment.
  • Value-distribution generalization: WCM retains value discriminability after scene generalization, whereas the original method’s discriminability decreases.The comparison uses planar heatmaps generated from critic-inferred values at 1000 sampled points in the robot’s reachable space.
  • Value-distribution generalization: WCM is less susceptible to excessively large or small local values on out-of-distribution samples.The original critic overfits the original distribution and produces outliers when encountering OOD observations.
Loading 2607.29613v1…