PaperlayerView exact revision on arXiv

Source-linked AI summary

World Value Models for Robotic Manipulation

Zhihao Wang, Jianxiong Li, Yu Cui, Yuan Gao, Xianyuan Zhan, Junzhi Yu, Xiao Ma

arXiv:2606.24742v1cs.RO

TL;DR

Existing robotic value models have limited temporal understanding and future-planning capacity because their VLM backbones model sparse visual observations. WVM uses a pretrained video world model for value estimation and achieves state-of-the-art results across standard and suboptimal-data benchmarks, while improving downstream policy learning in simulation and real-world manipulation.

  • Problem

    Existing robotic value models have limited temporal understanding and future-planning capabilities because their VLM backbones rely on sparse visual modeling.

  • Method

    WVM predicts chunk-wise task progress using features from a pretrained video world model to capture temporal dynamics and future outcomes.

  • Results

    WVM achieves higher-quality values than baselines across Suboptimal-Value-Bench and expert VOC, with downstream gains in simulated and real-world manipulation policy learning.

  • Takeaways & Limitations

    WVM provides robust guidance for learning manipulation policies from mixed-quality data across simulation and real-world deployment.

  • Takeaways & Limitations

    Limited training-data scale restricts WVM’s zero-shot capacity on entirely unseen tasks and scenes.

Abstract

from arXiv · show

Generalist value models play a pivotal role in scaling robotic policy learning from large-scale, mixed-quality data. Mathematically, accurate value estimation demands deep temporal understanding, requiring models to both ground the current belief using historical context and plan over future outcomes. However, most existing robotic value models are built on Vision-Language Model (VLM) backbones that are pretrained primarily on static or temporally sparse visual observations, lacking the requisite temporal modeling capabilities for value estimation. Unlike VLMs, world models naturally excel at temporal modeling and future planning, making them ideal foundations for learning generalizable value functions. Driven by this insight, we marry world models with value estimation to construct a new generalist robotic value model, World Value Model (WVM), that offers accurate task progressions to assess data quality. On standard benchmarks, WVM delivers state-of-the-art (SOTA) Value-Order Correlation (VOC) results. Complementing standard evaluation suites that contains only expert data, we further introduce Suboptimal-Value-Bench, a multi-embodiment benchmark consisting of 800 suboptimal trajectories with high-fidelity, human-labeled frame annotations. Our evaluations show that WVM maintains its SOTA performance on Suboptimal-Value-Bench, establishing its robustness in handling both expert and suboptimal data. When deployed for policy learning, WVM improves manipulation performance across various policy extraction approaches in both simulated and real-world deployment, providing robust guidance for learning from mixed-quality data.

1 Introduction

Generalist value models require temporal understanding of past context and forward planning, but existing VLM-based approaches suffer from limited temporal modeling and task-specific generalization. WVM repurposes world-model spatiotemporal priors for distributional value learning, while Suboptimal-Value-Bench evaluates progress estimation on diverse suboptimal trajectories.

  • Motivation: Accurate value estimation requires both comprehension of past temporal contexts and forward-looking planning over long-term future outcomes.
  • Limitations: Existing value models face inefficient scalar supervision, narrow task-specific customization, and impaired temporal understanding caused by sparse visual modeling.
  • WVM: WVM repurposes world-model spatiotemporal priors as a foundation for generalist robotic value learning.It couples video with a lightweight value Diffusion-Transformer through a Mixture-of-Transformers architecture and models value as a distributional chunk trained by flow matching.
  • Evaluation: Suboptimal-Value-Bench comprises 800 multi-embodiment trajectories with human-annotated task progress for evaluating value models beyond optimal trajectories.The benchmark includes retries and hesitations as two prevalent suboptimal behavioral modes.
  • Results: WVM achieves higher-quality values than baselines on Suboptimal-Value-Bench and expert VOC, while improving policy learning with noisy data in simulation and real-world manipulation.Ablations validate the necessity of the world-model prior and core architectural choices.

2 Related Work

Prior robotic value models suffer from sparse supervision and limited task generality, while world models offer temporal reasoning and forward-planning capabilities suited to value estimation. Evaluation remains challenging, motivating systematic measures such as Value-Order Correlation (VOC).

  • Value models for robotic manipulation: Existing robotic value models face sparse supervision from scalar regression and brittle predictions when scaled to heterogeneous video corpora.Scalar value regression on high-dimensional observations provides low-information training signals and is sample-inefficient.
  • Value models for robotic manipulation: Many prior value models are tailored to a single task, limiting their suitability as generalist progress estimators.
  • World models for robotic manipulation: World Action Models jointly model action-conditioned visual dynamics, and prior work shows video priors improve learning efficiency through temporal reasoning and forward prediction.
  • World models for robotic manipulation: The proposed MoT framework preserves intrinsic video-modeling capacity while deriving robust value predictions from latent temporal features.
  • Evaluation of robotic value models: Evaluating robotic value models is difficult because qualitative inspection does not scale, while downstream policy success conflates value fidelity with policy choice and adds computational overhead.

3 Method

WVM formulates value estimation as chunk-wise prediction of normalized task progress, using temporal world-model features to capture future outcomes and local progress dynamics. Its architecture combines video and value diffusion transformers with flow-matching training and augmentations that improve continuity and expose plateaus or regressions.

  • Value formulation: WVM predicts a length-h sequence of per-frame values from h-frame observations and a language instruction, rather than an isolated scalar.The targets are normalized progress values v_t = t/T, enabling modeling of local profiles such as plateaus and regressions.
  • Value formulation: Under sparse rewards, value estimation reduces to predicting negative expected distance-to-goal, directly motivating a video world model as a feature extractor.This formulation makes the value function intrinsically focused on future outcomes.
  • Architecture: WVM couples a video DiT and lightweight value DiT through multi-modal self-attention, with value tokens attending to video features but not vice versa.The video stream uses Wan2.2’s video VAE and video DiT, while the value stream predicts noisy value chunks from intermediate video-DiT features.
  • Training objective: Flow matching trains both future video latents and value chunks by predicting their velocity fields from noise-interpolated samples.A coefficient λ controls the weight of video co-training.
  • Augmentations: Prefix randomization prevents shortcut extrapolation, while video rewinding augments expert data with rising, plateau, and descending local progress patterns.Prefix values are randomly replaced with scalars from [0, 1] with probability p; rewinding relabels chunks after preserving, repeating, or reversing frames.

4 Suboptimal-Value-Bench

Suboptimal-Value-Bench evaluates robotic value models on 800 human-annotated trajectories spanning three embodiments and 15 tasks, with dense frame-level value curves emphasizing hesitation and retry. It introduces evaluation procedures tailored to invariant hesitation segments and monotonically decreasing retry windows.

  • Benchmark: Suboptimal-Value-Bench contains 800 human-annotated trajectories across three embodiments and 15 tasks, with dense value curves for every frame.The benchmark focuses specifically on hesitation and retry segments in suboptimal robot data.
  • Hesitation: During hesitation, robots remain stationary or make task-irrelevant micro-movements, leaving task progress invariant throughout the segment.Hesitation can result from teleoperator cognitive pauses or physical hardware constraints.
  • Hesitation: Hesitation-RMSE penalizes prediction drift against the constant ground-truth value within hesitation segments.A constant, accurate prediction has zero error, while fluctuations produce higher RMSE proportional to tracking deviation.
  • Retry: Retry evaluation isolates windows with monotonically decreasing ground-truth progress and computes VOC only within those windows.Retries involve a failed manipulation attempt followed by release and retraction before another attempt, making the value drop central to evaluation.

5 Experiments

Experiments show that WVM delivers superior value estimation on both suboptimal and expert trajectories, improves downstream policy learning, and benefits from its video co-training, prefix randomization, and flow-matching value head design.

  • 5.1 Value Estimation: WVM achieves the lowest Hesitation-RMSE across all three embodiments, reducing average error to 0.05 versus 0.14 for both GVL and Robometer.This indicates more stable value estimation during task-invariant stagnation.
  • 5.1 Value Estimation: On expert trajectories, WVM attains the highest average VOC of 0.95 versus 0.88 for the strongest baseline and ranks first on five of six datasets.WVM exceeds 0.99 on all three self-collected datasets, while RoboReward surpasses it on EgoDex, 0.95 versus 0.92.
  • 5.2 Policy Improvement: All three WVM-guided variants—AWR and binary or percentile Filtered BC—consistently outperform vanilla BC across simulated and physical environments.Policy finetuning uses only 10 trajectories per simulated task and 50 trajectories per real-world task.
  • 5.3 Ablations: Removing video co-training increases Hesitation-RMSE from 0.05 to 0.08 and reduces Retry-VOC from 0.78 to 0.68; random video initialization further reduces Retry-VOC to 0.62.These ablations show that video co-training materially supports Suboptimal-Value-Bench performance.
  • 5.3 Ablations: Without prefix randomization, Hesitation-RMSE worsens to 0.09 and Retry-VOC drops to 0.67, while Expert-VOC increases to 0.98.The divergence indicates that Expert-VOC alone can be insufficient for evaluating value models.
  • 5.3 Ablations: Replacing the flow-matching value head with an HL-Gaussian alternative increases Hesitation-RMSE from 0.05 to 0.06 and sharply reduces discriminative scores.The passage attributes this degradation to categorical heads discarding fine-grained density variations.

6 Conclusion

WVM is a generalist robotic value flow model rooted in a pretrained world model’s predictive capabilities. It achieves state-of-the-art results on standard benchmarks and Suboptimal-Value-Bench, a multi-embodiment suite of 800 human-annotated suboptimal trajectories.

  • 6 Conclusion: WVM is a generalist robotic value flow model rooted in the predictive capabilities of a pretrained world model.Its design inherits the world model’s strengths in historical grounding and future planning.
  • 6 Conclusion: WVM delivers SOTA results across standard benchmarks and the new Suboptimal-Value-Bench.The benchmark complements expert-only evaluations with suboptimal trajectories across multiple embodiments.
  • 6 Conclusion: 800 high-fidelity suboptimal trajectories with frame-level human annotations constitute Suboptimal-Value-Bench.The suite is multi-embodiment and complements standard evaluations containing only expert data.

7 Limitations · Appendix

WVM’s limitations stem from computational constraints that restrict training scale and zero-shot generalization, while Suboptimal-Value-Bench remains narrow in task scope. The authors identify larger training mixtures and broader evaluations as next steps.

  • 7 Limitations: WVM’s training dataset is currently limited in scale because of computational constraints.This limitation constrains the amount of data used to train the model.
  • 7 Limitations: Restricted training scale limits WVM’s zero-shot capacity on entirely unseen tasks and scenes.The limitation specifically affects generalization beyond the training distribution.
  • 7 Limitations: Suboptimal-Value-Bench broadens evaluation beyond expert-only demonstrations.The benchmark was introduced to assess suboptimal trajectories in addition to expert data.
  • 7 Limitations: The benchmark’s current scope is primarily focused on pick-and-place tasks.This leaves more dexterous and long-horizon manipulations insufficiently covered.
  • Appendix: Expanding evaluation to more dexterous manipulations is identified as a critical next step.The passage cites references [30] [76] for this expansion direction.
  • Appendix: Expanding evaluation to long-horizon manipulations is also identified as a critical next step.The passage cites references [63] [66] for this expansion direction.
  • Appendix: The authors plan to scale up both the training mixture and evaluation.These plans address the current limitations in training scale and benchmark coverage.

A Implementation Details

WVM uses a Wan2.2-TI2V-5B-based video architecture with spatiotemporal compression and patchification before a deep video DiT. Its implementation specifies main-run hyperparameters, a single-step Euler inference procedure, and a mixed-source pretraining dataset summarized by source-specific subset and trajectory counts.

  • Architecture: WVM builds its video stream on the publicly released Wan2.2-TI2V-5B checkpoint, using Wan2.2-VAE compression and (1, 2, 2) latent patchification before the transformer.The VAE compresses videos by 4 × 16 × 16 along temporal and spatial axes into a 48-channel spatiotemporal latent.
  • Architecture: The video DiT has 30 layers, hidden dimension 3072, 24 attention heads, and head dimension 128.
  • Training: Main-run training hyperparameters are summarized in Table A.1.
  • Inference: At test time, WVM applies an explicit Euler solver to the learned flow-matching velocity field using one denoising step for all reported results.More denoising steps produce no measurable gains on Suboptimal-Value-Bench or Expert-VOC.
  • Training dataset mixture: The pretraining mixture is summarized in Table A.2 using source-specific counts of subsets and trajectories.For self-collected sources, each subset corresponds to exactly one task; other sources use source-dependent subset semantics.

B Suboptimal-Value-Bench Details

Suboptimal-Value-Bench spans three embodiments and 15 tasks, with two suboptimal trajectory groups per task. Its frame-level human annotations are produced through VLM-assisted verification and converted into mode-specific piecewise-linear ground-truth values for evaluation.

  • Benchmark composition: The benchmark covers 3 embodiments, 15 manipulation tasks, and two trajectory groups per task: hesitation and retry.The two groups correspond to the benchmark’s suboptimal modes.
  • Annotation pipeline: 800 trajectories are annotated with a two-stage pipeline that uses VLM proposals followed by human frame-level correction.Annotators can adjust boundaries, split or merge intervals, replay sub-clips, and review trajectories with empty VLM proposals.
  • Ground-truth values: Each trajectory receives a suboptimal type, segment endpoints m and n, total length T, and a per-frame ground-truth value vt ∈[0, 1].The values are constructed by linear interpolation through four control points, with values outside the trajectory clamped to 0 or 1.
  • Hesitation labels: For hesitation, the ground-truth curve forms a plateau over [m, n], while progress advances at a constant effective rate before and after the segment.The effective progress frames are T −1 −x, where x = n −m.
  • Retry labels: For retry, the curve models backward motion followed by recovery, using slope 1/(T −2x) and clamping progress to 0 when n > 2m.These piecewise-linear curves provide ground truth for Hesitation-RMSE and Retry-VOC.

C Expert VOC Dataset Composition

Expert-VOC combines three public corpora with four self-collected expert sets spanning AgileX, ARX, and RoboSuite, while holding every evaluation trajectory out of WVM training. Sampling is generally 5% per source, with embodiment- and source-specific adjustments for pool size and demonstration quality.

  • Dataset composition: The evaluation set covers OXE, RoboCOIN, EgoDex, and four self-collected expert sets matching the AgileX, ARX, and RoboSuite embodiments.The four self-collected sources are consolidated as “Self-collected (3 embodiments)” in the main paper.
  • Data splitting: All Expert-VOC trajectories are held out from the WVM training mixture at the trajectory level to prevent leakage.Under the default protocol, 5% are sampled for evaluation and the remaining 95% remain in training, producing disjoint splits.
  • Source-specific sampling: OXE uses the filtered RoboReward release pool, including the train split, reward = 5 successes, and DROID left view only.The Expert-VOC selection protocol operates on this filtered pool rather than the raw OXE distribution.
  • Source-specific sampling: AgileX and ARX use 20% sampling because their uniformly high-quality sources are small, whereas RoboSuite uses the default 5% and yields 99 expert trajectories.No additional quality filtering is applied to these self-collected sources because AgileX, ARX, and RoboSuite trajectories are uniformly expert-quality.
  • Source-specific sampling: RoboCOIN and EgoDex select the single shortest demonstration per language instruction, with EgoDex restricted to basic pick-and-place categories.This replaces default random sampling to favor demonstrations closer to expert behavior and activities relevant to robotic manipulation.

D Value Model Baseline Reproduction

The baseline reproduction evaluates six robotic value models under a unified video-sampling pipeline while preserving each method’s officially recommended inference protocol. The baselines span autoregressive completion prediction, action–critic progress estimation, preference-trained progress modeling, discrete rollout scoring, zero-shot temporal scoring, and incremental process rewards.

  • Evaluation Protocol: All six baselines use a unified sampling pipeline: trajectories are downsampled to 2 fps, or 3 fps for AgileX and ARX, before protocol-specific evaluation.Robometer and RoboReward use multi-anchor prefix evaluation, while the other four baselines use single-pass full-trajectory evaluation.
  • Baseline Methods: GVL predicts autoregressive completion percentages for shuffled video frames using gpt-5.4 with a 32-frame per-call budget.The model receives a task description and frame–index pairs, then outputs task-progress percentages for each frame.
  • Baseline Methods: VLAC fine-tunes InternVL to emit signed progress deltas and task-completion signals from paired observations and language goals.The reproduction uses InternRobotics/VLAC in single-pass mode with 32 frames per trajectory and default decoding settings.
  • Baseline Methods: Robometer combines frame-level progress, per-frame success, and inter-trajectory preference losses to estimate dense progress from short prefix clips.The public Robometer-4B checkpoint is evaluated with five uniformly spaced anchors and eight frames per anchor.
  • Baseline Methods: RoboReward predicts discrete end-of-episode progress scores from full rollout videos and obtains dense rewards by re-querying partial prefixes.The reproduction uses RoboReward-4B with five anchors, use_frame_steps=true, subsample_n_frames=5, and max_frames raised to 32.
  • Baseline Methods: TopReward forms a zero-shot temporal value function from affirmative-token log-probabilities averaged across K = 15 prefix samples.The setup uses Qwen3-VL-8B-Instruct with mean reduction and the official chat template.
  • Baseline Methods: Robo-Dopamine is a 3B-parameter step-aware process reward model that predicts discretized relative progress and emits per-frame signals incrementally.The reproduction uses Robo-Dopamine-GRM-3B in incremental mode with one-frame intervals and up to 32 uniformly sampled frames.

E Downstream Policy Learning Details · F HL-Gaussian Value Head Details

Downstream policy learning uses WVM-derived chunk-level advantage proxies to weight or filter behavioral-cloning losses, with suite-specific settings and normalized AWR weights. The HL-Gaussian ablation changes only the value head, replacing flow matching with 51-bin Gaussian-smoothed classification and expectation-based decoding.

  • E Downstream Policy Learning Details: Each action chunk receives a WVM-derived advantage proxy measuring its approximate value improvement from frame t through min(t + H − 1, T − 1).The proxy uses per-frame WVM values V(·) ∈ [0, 1] and the chunk length H.
  • E Downstream Policy Learning Details: Weighted behavioral cloning multiplies each sample loss by its corresponding proxy-based weight wi.The per-sample loss ℓi can be the flow-matching loss for π0.5-base.
  • E Downstream Policy Learning Details: Filtered-BC uses hard thresholding: the binary variant sets κ = 0.0, while the percentile variant retains the top 70% of chunks by Δi.The percentile threshold κ is reported separately for each task suite.
  • E Downstream Policy Learning Details: AWR applies clipped exponential weighting with δ = 2.0 across both task suites and tunes temperature τ separately for each suite.The clip limits the highest-advantage chunks to 2× their baseline contribution, and batch weights are renormalized to preserve vanilla-BC gradient scale.
  • F HL-Gaussian Value Head Details: The HL-Gaussian ablation changes only WVM’s value head, retaining the video stream, value stream, MoT coupling, prefix randomization, and video rewinding.It replaces flow-matching regression with classification over fixed bins.
  • F HL-Gaussian Value Head Details: HL-Gaussian discretizes [0, 1] into K = 51 equally spaced bin centres and trains against Gaussian-smoothed soft targets with σ set to one bin width by default.Values near 0 or 1 are snapped to hard one-hot endpoint bins.
  • F HL-Gaussian Value Head Details: The head replaces the flow-matching value loss with token-level soft cross-entropy, while inference uses one transformer forward and expectation decoding over bin centres.The video co-training loss and objective weight λ remain unchanged, followed by overlapping-window averaging across adjacent chunks.
Loading 2606.24742v1…