PaperlayerView exact revision on arXiv

Source-linked AI summary

WEAVER, Better, Faster, Longer: An Effective World Model for Robotic Manipulation

Arnav Kumar Jain, Yilin Wu, Jesse Farebrother, Gokul Swamy, Andrea Bajcsy

arXiv:2606.13672v2cs.RO

TL;DR

Robotic world models must balance fidelity, long-horizon consistency, and efficient generation, but existing systems do not achieve all three together. WEAVER combines multi-view prediction with design choices for coherent, fast simulation, achieving strong manipulation-task results across policy evaluation, improvement, and planning.

  • Problem

    Existing robotic world models lack a system that jointly provides high fidelity, long-horizon consistency, and efficient generation.

  • Method

    WEAVER is a multi-view world model that predicts future latents and rewards using flow matching, pretrained encoders, memory, and multi-view observations.

  • Results

    ρ = 0.870 correlation with real-world success, 38% improvement for π0.5 without real-world interaction, and 5−10× faster planning than Ctrl-World.

  • Takeaways & Limitations

    WEAVER supports policy evaluation, policy improvement, and test-time planning across real-hardware manipulation tasks with limited real-world interaction.

  • Takeaways & Limitations

    Because WEAVER relies primarily on visual observations, hidden contacts, forces, grasp stability, and occluded geometry may remain inaccessible from image history alone.

Abstract

from arXiv · show

The potential impacts of world models (WMs, i.e., learned simulators) on robotics are far-reaching -- policy evaluation, policy improvement, and test-time planning -- all with limited real-world interaction. To unlock these downstream capabilities, a WM needs to jointly satisfy three desiderata: $\textit{(i)}$ fidelity (i.e., producing simulated trajectories that correlate with reality), $\textit{(ii)}$ consistency (i.e., producing simulated trajectories that are coherent over long horizons), and $\textit{(iii)}$ efficiency (i.e., producing simulated trajectories quickly). We propose WEAVER (World Estimation Across Views for Embodied Reasoning): a WM architecture that simultaneously achieves all three desiderata, providing state-of-the-art results on robotic manipulation tasks. WEAVER is a multi-view WM trained to predict future latents and reward values via a flow-matching loss. We distill the key design decisions across model architecture, memory, and prediction objectives required to unlock the kinds of long-horizon dynamic manipulation tasks that have confounded prior world modeling approaches. We apply WEAVER in robotic hardware, demonstrating its effectiveness at policy evaluation ($ρ$=0.870 correlation with real-world success rate), policy improvement (real-world success rate improvement of $38\%$ on top of the $π_{0.5}$ robot foundation model), and test-time planning (real-world success rate improvement of $14\%$ with a $5-10\times$ speedup over prior WMs). WEAVER also demonstrates better performance than prior WMs when evaluated on out-of-distribution scenarios. Code, models, and videos at: https://arnavkj1995.github.io/WEAVER/ .

1 Introduction

WEAVER is introduced as a world model for robotic manipulation designed to jointly provide fidelity, long-horizon consistency, and efficient generation. On real-hardware manipulation tasks, it supports policy evaluation, policy improvement, and test-time planning with strong reported results.

  • World models promise policy evaluation and improvement without costly or unsafe real-world interaction, while also enabling test-time scaling through planning.
  • A capable robot world model must jointly satisfy fidelity, consistency over long horizons, and efficient prediction.Fidelity requires physically accurate predictions correlating with real-world outcomes; consistency requires coherent long-horizon predictions; efficiency requires fast generation.
  • Existing robot world models do not satisfy all three desiderata simultaneously, trading off fidelity, decodability, efficiency, or out-of-distribution robustness.Video generation models are high-fidelity but inefficient, JEPA-style latent states may not decode into evaluation images, and learning encoders from scratch can harm out-of-distribution robustness.
  • Robotic manipulation further requires multi-view handling, inference of occluded objects from history, and high-fidelity predicted states, often reducing efficiency.Ctrl-World is identified as operating at far slower speeds than the real world, precluding its use in test-time planning.
  • WEAVER is introduced as a world-model architecture achieving high fidelity, long-horizon consistency, and efficient generation for policy evaluation, improvement, and test-time planning.
  • ρ = 0.870 correlation with real-world success rate, 38% improvement for the π0.5 robot foundation model, and no real-world interaction are reported on five real-hardware manipulation tasks.The task suite ranges from pick and place to deformable object manipulation.

2 Related Work

Prior robotic world models often achieve high fidelity but remain inefficient for test-time planning or inconsistent over long horizons. WEAVER combines video-generation techniques, pretrained components, multi-view prediction, history, memory, and latent reward/value heads to target efficient, coherent, robust manipulation modeling and downstream policy use.

  • Robot World Models: Prior robotic world models often provide high fidelity but are not efficient enough for test-time planning.Flow matching and diffusion forcing from vision-generation research are identified as ingredients for improving efficiency.
  • WEAVER: WEAVER uses pretrained video-generation encoders for robustness to out-of-distribution visual inputs and pretrained decoders to evaluate arbitrary visuomotor policies.Its latent reward and value heads support efficient evaluation and planning without decoded images or an external VLM judge.
  • WEAVER: Multi-view prediction, history, and memory help WEAVER maintain temporal consistency across long horizons, including occluded gripper-object interactions.This design contrasts with WorldGym, DreamerV4, and DreamDojo, which are described as earlier world-model approaches.
  • WEAVER: Compared with Ctrl-World, WEAVER targets higher-fidelity, more temporally coherent generations with more efficient inference, while its pretrained encoder likely improves robustness over Dreamer-v4’s learned-from-scratch encoder.The passage characterizes this combination as Pareto dominating Ctrl-World.
  • Uses of World Models in Robotics: World models in robotics are being explored for policy evaluation, policy improvement, and test-time planning.Prior work supports scalable policy evaluation with sufficiently faithful models, while synthetic trajectories may improve policies but the extent remains an open question.

3 WEAVER: World Estimation Across Views for Embodied Reasoning

WEAVER combines multi-view and proprioceptive prediction, long- and short-term latent context, flow-based training, and inference acceleration to jointly target fidelity, consistency, and efficiency. These components support policy evaluation, improvement, and test-time planning, including long-horizon rollouts requiring 40+ iterative model evaluations.

  • World Model Architecture: WEAVER predicts external and wrist-camera observations alongside proprioceptive states to improve fidelity and consistency during contact-rich, partially observed manipulation.Each view is encoded into patch tokens, while proprioception is projected to the same token dimension and concatenated.
  • World Model Architecture: Sparse memory of every kth observation and a two-frame history capture long-term context and short-term action consequences across occlusions and viewpoint changes.The memory preserves what changes and stays constant, while recent frames capture immediate consequences of actions.
  • World Model Architecture: Flow matching trains future latent prediction, while Diffusion Forcing improves long-horizon consistency and SPRINT blocks reduce patch tokens for efficiency.The dynamics model predicts future latents from Gaussian-noised targets and independently sampled future-timestep noise levels.
  • Inference Acceleration: KV caching reduces repeated forward-pass costs, a cosine progressive noise schedule improves fidelity, and rectified-flow distillation enables high-quality generation within a few forward passes.These changes address both model execution and iterative denoising costs required for test-time planning.
  • Latent-Space Value Estimation: A latent-space reward head enables efficient action-chunk scoring, while a critic estimates value beyond truncated imagined horizons using bootstrapped λ-returns.The reward head avoids decoding images and querying an external VLM judge model.
  • Downstream Applications: WEAVER’s simultaneous fidelity, consistency, and efficiency support evaluation, improvement, and planning, including long-horizon policy evaluation requiring 40+ iterative latent-dynamics evaluations.Recorded real-world action trajectories are executed open-loop inside WEAVER while predicted rewards are recorded.

4 Experimental Setup

The experiments use a single Panda manipulator with three cameras, while π0.5 and WEAVER consume only the right and wrist views. WEAVER is pretrained on DROID and evaluated after real-world fine-tuning across five manipulation tasks spanning rigid, deformable, and dynamic behaviors.

  • Base Policy & Hardware: The setup uses one Franka Emika Panda, two external Zed 2i cameras, and one wrist-mounted Zed Mini camera.The cameras are mounted on the workspace’s left and right sides, with the wrist camera attached to the robot.
  • Base Policy & Hardware: π0.5 and WEAVER use only the right external camera and wrist camera, while all three cameras are installed because the main baseline uses all views.
  • Datasets & Tasks: WEAVER is pretrained on DROID and then fine-tuned on data collected from π0.5 operating five real-world manipulation tasks, with 50 rollouts per task plus 20 evaluation rollouts.
  • Datasets & Tasks: The five tasks span rigid pick-and-place, deformable-object manipulation, and dynamic manipulation: Stack Bowls, PnP Bag, PnP Marker, PnP Towel, and Pour Beans.
  • World Model Training: WEAVER has 928M parameters and is pretrained for 1M steps with batch size 32, learning rate 1e−4, and 4 × H100 GPUs for 10 days.Progress rewards for the reward model and critic come from Robometer annotations, reduced by 1 to obtain negative rewards.

5 Results

WEAVER balances perceptual fidelity, long-horizon consistency, and inference efficiency, outperforming Ctrl-World across world-model evaluations. It also supports policy evaluation, synthetic-data policy improvement, and test-time planning with measurable gains over real-world or base-policy performance.

  • World-model performance: Up to 16× less inference time enables WEAVER to Pareto-dominate Ctrl-World across views and out-of-distribution task datasets.At NFE values of 8, 16, 32, and 50, Ctrl-World takes 30–50s versus 10–30s for WEAVER to generate a 10s chunk on one H100 GPU.
  • World-model performance: WEAVER outperforms Ctrl-World in perceptual quality while using lower inference time, and its quality degrades less as function evaluations decrease.Both models incur their highest error on wrist-camera viewpoints.
  • World-model performance: WEAVER maintains consistently lower FID than Ctrl-World for long-horizon 150-step or 10s rollouts on DROID.Generation quality is measured for each predicted 15-step interval across the rollout horizon.
  • Policy evaluation: WEAVER achieves better agreement with real policy performance than Ctrl-World, with higher Pearson correlation and lower MMRV despite rollouts lasting up to 40 seconds.Pretrained models tend to underestimate policy performance, while pouring is particularly difficult because granular dynamics are underrepresented and hard to model.
  • Policy improvement: 4% average performance gap separates synthetic-data finetuning from real-data finetuning, while combining both increases average success rate by 11% over real-data finetuning alone.All finetuned policies substantially improve over the base π0.5 VLA, indicating that imagined rollouts can reduce costly real-world collection and manual filtering.
  • Test-time planning: 15% average success-rate improvement over the base policy, with a maximum gain of 20%, is achieved by test-time planning using advantage-based selection.The gain is larger for weaker base policies but remains below direct finetuning because planning is limited to one action chunk and constrained by latency.

6 Conclusion

WEAVER achieves fidelity, temporal coherence, and efficient generation, supporting policy evaluation, improvement, and faster test-time planning. The conclusion also identifies sensing, physics, latency, and responsible-deployment limitations for broader robotic use.

  • Conclusion: ρ = 0.870 correlation with real-world success rate, 38% success-rate improvement for π0.5 without real-world interaction, and 5−10× faster test-time planning than Ctrl-World.These results summarize WEAVER’s evaluation, policy-improvement, and planning capabilities.
  • Limitations: WEAVER remains limited by partial visual state, missing tactile sensing, absent physics priors for deformable objects, and generation latency restricting planning to short horizons.The paper identifies tactile sensing, physics priors, and lower latency as directions for addressing these limitations.
  • Broader Impact: Imagined rollouts can reduce costly real-world interaction for evaluation, improvement, and planning, but inaccurate or biased predictions may cause risky decisions in safety-critical domains.Responsible deployment requires careful validation, especially for assistive robots.

A1 Robot Setup & Tasks

The study uses five diverse real-world manipulation tasks on the DROID setup, collecting π0.5 finetuning data from tasks where the base policy achieves at least 20% success. The world model operates in joint space, with an action adapter converting joint-velocity commands into predicted joint-position and gripper-state changes.

  • Robot Setup & Tasks: Five manipulation tasks are selected on the DROID setup, with 50 π0.5 rollouts per task and at least 20% base-policy success.The task set includes rigid-object pick-and-place, deformable-object manipulation, and dynamic manipulation, while retaining both successful and failed executions.
  • Robot Setup & Tasks: Stack Bowls requires placing one randomly selected bowl on top of another among blue, green, and pink bowls.The placed bowl A is blue or green, while bowl B is randomly selected from the remaining colors.
  • Robot Setup & Tasks: PnP Bag, PnP Towel, and PnP Marker test deformable manipulation or precise grasping and reorientation under randomized object and target variants.The bag and towel tasks involve grasp-dependent deformability, while the marker task requires vertical insertion into a randomly selected paper cup or blue mug.
  • Robot Setup & Tasks: Pour Beans tests dynamic manipulation by requiring controlled cup pose, pouring angle, and motion to transfer coffee beans into a blue bowl without spilling.The granular motion of the beans is difficult to predict, making accurate control essential.
  • Robot Setup & Tasks: The world model uses joint-space actions, while a lightweight adapter converts joint-velocity commands and gripper signals into future absolute joint positions and gripper states.This avoids action-conversion compounding errors, although direct conditioning on joint velocities produces lower generation quality.

A2 Implementation Details

WEAVER uses a multi-view latent transformer with causal temporal attention, DROID-based training and task finetuning, and reward supervision derived from interpolated Robometer progress labels. Its implementation also compares deterministic noise schedules and applies ReFlow post-training to reduce inference cost.

  • Architecture: WEAVER encodes 190×32 image frames with the Stable Diffusion 3 VAE and processes them using a 32-layer, 1536-dimensional transformer with 16 attention heads.Each layer combines spatial attention over current patches with causal temporal attention over prior observations; actions and proprioceptive states use DROID training-set statistics.
  • Pretraining: 928M parameters are pretrained on DROID for 1M gradient steps on 4×H100 GPUs over 10 days, using EMA β = 0.9999 and a post-warmup learning rate of 1e−4.The learning-rate warmup lasts 10000 steps.
  • Finetuning: 16K gradient steps of task finetuning at learning rate 2e−5 use 250 π0.5-collected trajectories to address inaccurate predictions on underrepresented complex tasks such as pouring.The dataset contains 50 trajectories for each task.
  • Reward labeling: Reward labels are generated by Robometer as per-frame progress predictions from 1-fps right-camera frames and language instructions, then linearly interpolated to the original video length.The interpolated reward_progress signal is shifted to the range [−1, 0] for training.
  • Inference noise schedules: Deterministic inference schedules map denoising indices to noise levels using linear, sigmoid, power, or cosine functions, with cosine and power emphasizing low-noise regions.The sigmoid schedule concentrates updates around the trajectory midpoint, while the linear schedule distributes them uniformly.
  • Rectified-Flow: ReFlow post-training produces WEAVER-ReFlow from WEAVER-FT to reduce inference time and NFE, using 2K gradient steps on 4 × H100 GPUs for 6 hours at learning rate 2e−5.The teacher is frozen, and the student learns teacher-predicted future latents with mean squared error loss.

A3 Additional World Model Evaluation Results

Additional evaluations show that WEAVER maintains fidelity under low inference budgets, accelerates generation through KV caching, benefits from noise schedules emphasizing low-noise regions, and improves further with finetuning and rectified-flow post-training.

  • Coherent generations: At similar NFE values, WEAVER generates rollouts 3× faster than Ctrl-World, while NFE=8 WEAVER outperforms Ctrl-World with NFE=50.Ctrl-World deteriorates more sharply as NFE decreases, whereas WEAVER shows only a slight performance drop.
  • KV Cache: KV Cache reduces inference time by up to 30% by caching memory and history key-value vectors after the first denoising step.Memory and history latents use a constant noise level k = 1 throughout iterative denoising.
  • Noise schedules: Power and cosine noise schedules outperform sigmoid and linear schedules because they allocate more bandwidth to low-noise regions for fine-grained details.The schedule comparison reports results with NFE=16.
  • Finetuning: With NFE=16, WEAVER-FT outperforms WEAVER with NFE=50 and finetuned Ctrl-World across metrics on the OOD Task dataset.The Ctrl-World baseline was finetuned for 20K gradient steps on 4 × H100 GPUs.
  • Rectified-flow post-training: WEAVER-ReFlow reduces the performance gap with WEAVER-FT at NFE=16, making it suitable for test-time steering with a small inference budget.This post-training step further reduces the required inference budget.

A4 Additional Downstream Application Results

Additional results show that WEAVER-FT provides stronger and more accurate policy evaluation than alternatives, while WEAVER enables substantially faster test-time planning. Qualitative policy-improvement examples show smoother control, though noisy reward labels remain a limitation.

  • Policy Evaluation: Both Ctrl-World and WEAVER struggle to predict performance on challenging dynamic and deformable manipulation tasks, including pouring beans, bag manipulation, and towel manipulation.PnP Bag is especially difficult because the model must infer gripper depth across two views and model contact dynamics.
  • Policy Evaluation: WEAVER-FT achieves the strongest correlations and lowest prediction error across RMSE, Spearman correlation, Pearson correlation, and MMRV.WEAVER also outperforms Ctrl-World in zero-shot policy evaluation on out-of-distribution task data.
  • Policy Improvement: Finetuning reduces imprecise grasping, placement errors, and large per-step motions, producing smoother and more stable robot control during dynamic manipulation.The base policy often makes insufficient adjustments and generates an unstable action distribution.
  • Reward Supervision: RoboMeter reward labels can miss fine-grained placement accuracy, introducing noise into predicted rewards, so the advantage threshold is set to 0.1.More diverse failure data could improve reward supervision.
  • Inference-Time Latency: 20.2×: At horizon 15 and batch size 4, WEAVER requires 1.4547s for dynamics prediction versus 29.4244s for Ctrl-World.At batch size 1, WEAVER achieves a 16.6× speedup, taking 0.4476s versus 7.4236s.

A5 Limitations

WEAVER’s limitations stem from incomplete visual state access, difficult deformable and dynamic-object prediction, restricted planning horizons, narrow pretraining coverage, and noisy reward supervision. Future progress therefore requires richer sensing, broader data, stronger physical priors, and more accurate reward labels.

  • Partial observability: Visual observations may hide contacts, grasp stability, forces, and occluded geometry, especially with wrist cameras and cluttered scenes.These partial observations limit access to task-relevant physical state.
  • Difficult dynamics: Deformable and dynamic manipulation remains challenging because history-dependent dynamics and small contact or configuration errors can compound into incorrect rollouts.Pouring is especially sensitive to cup pose, velocity, and container contact; physics priors and hybrid neural-physics models are proposed directions.
  • Planning horizon: Single-action-chunk online planning improves near-term action selection but cannot yet perform long-horizon lookahead for delayed consequences or multi-stage recovery.Further improvements in sampling efficiency, distillation, value estimation, or hierarchical planning could address this restriction.
  • Data coverage: Pretraining primarily on DROID may limit generalization across robot embodiments, camera configurations, and end-effectors, while some dynamics such as granular pouring are underrepresented.More diverse cross-embodiment robot datasets are identified as a future direction.
  • Reward supervision: Off-the-shelf reward-model labels can be noisy or incomplete, potentially missing subtle contact, placement, semantic, or task-completion failures.The paper summarizes the needed remedies as richer sensing, broader data coverage, stronger physical inductive biases, and more accurate reward supervision.
Loading 2606.13672v2…