PaperlayerView exact revision on arXiv

Source-linked AI summary

Video Prediction Policy 2: Predict Better, Act Better

Yanjiang Guo, Haodong Yan, Zhide Zhong, Zhongru Zhang, Qingyuan Yang, Qingzhou Lu, Xiaoyu Chen, Yen-Jen Wang, Shuying Deng, Chenghan Yang, Puzhen Yuan, Chenxin Liu, Tun Ban, Xiang Zhu, Yichen Liu, Kun Feng, Haoang Li, Jianyu Chen

arXiv:2610.10270v1cs.CVcs.RO

TL;DR

Existing world action models may predict unreliable motion on open-ended tasks, while action-specific additions can degrade video-model generalization. VPP2 combines manipulation-focused video training and action learning, and reports gains in open-ended instruction following alongside strong zero-shot and benchmark results.

  • Problem

    Existing WAMs can make unreliable future-motion predictions on unseen scenarios, while base video models are not optimized for manipulation and action-specific additions can degrade generalization.

    Source: §1 · p. 2

  • Method

    VPP2 continues video pretraining with detailed captions and event-level prediction, distills a fixed-horizon visual planner, and adds a MoT action expert conditioned on predicted futures.

    Source: §1 · p. 2

  • Results

    VPP2-14B exceeds Cosmos3-64B by 11% in open-ended instruction-following success, and VPP2 leads evaluated methods on LIBERO-Pro, LIBERO-OOD, and RoboDojo after benchmark-specific post-training.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results support emphasizing generalizable video prediction alongside action modeling in future WAM research.

    Source: §6 · p. 9

Abstract

from arXiv · show

World action models (WAMs) have emerged as an important class of generalist robot policies, aiming to transfer video prediction priors to action learning. However, we find that existing WAMs frequently produce incorrect motion predictions in open-ended environment, leading to erroneous actions. We attribute this limitation to two factors: (1) base video models are not optimized for manipulation, and (2) naively incorporating action components into video models can substantially degrade their generalization capabilities. We introduce Video Prediction Policy 2 (VPP2), a WAM that enables strong zero-shot generalization in both video prediction and action generation. First, we curate a large-scale, diverse dataset of manipulation videos to continue pretraining the base video foundation model. We annotate video clips with detailed captions and perform \textit{event-level} video pretraining to promote generalization across open-ended manipulation tasks. Second, we post-train and distill the video model into a single-step visual planner with fixed prediction horizon. Finally, we introduce action module via a mixture-of-transformers (MoT) architecture to learn implicit inverse dynamics model. Experiments demonstrate three key results: (1) VPP2-14B outperforms Cosmos3-64B by 11.0\% points in video prediction instruction-following success rate on open-ended tasks; (2) VPP2 surpasses the strongest baseline by 18.5\% points in success rate on real-world zero-shot ALOHA manipulation tasks; and (3) following benchmark-specific post-training, VPP2 achieves the highest success rates among evaluated methods on the challenging LIBERO-Pro, LIBERO-OOD, and RoboDojo benchmarks.

1 INTRODUCTION

Existing world action models can make unreliable future-motion predictions on unseen tasks, and adding action-specific components can weaken video-model generalization. VPP2 combines manipulation-focused video training with action learning and reports stronger results across open-ended prediction and manipulation benchmarks.

1 INTRODUCTION

  • Prior WAMs can produce unreliable future-motion predictions on unseen, out-of-distribution scenarios, leading to erroneous actions.The paper identifies two contributing factors: base video models are not optimized for precise manipulation dynamics, and action-specific additions can substantially degrade video-model generalization.Source: §1 · p. 2
  • VPP2 continues video pretraining with detailed manipulation captions and event-level prediction, then post-trains and distills a fixed-horizon planner and adds a MoT action expert.The action expert learns implicit inverse dynamics conditioned on predicted futures; a VLM planner translates open-ended instructions into explicit subtasks during execution.Source: §1 · p. 2
  • VPP2-14B exceeds Cosmos3-64B by 11% in open-ended instruction-following success, while VPP2 reports strong ALOHA zero-shot results and top results on three post-trained benchmarks.On ALOHA, VPP2 averages 58.5% success versus 40.0% for π0.5 and 20.5% for Fast-WAM; it also leads evaluated methods on LIBERO-Pro, LIBERO-OOD, and RoboDojo.Source: §1 · p. 2

2 DATA PROCESS PIPELINE

The data pipeline builds a diverse manipulation-video collection, filters and segments trajectories, and adds detailed captions. It also standardizes multi-view inputs and aligns robot coordinate systems across datasets.

2 DATA PROCESS PIPELINE

  • The training collection spans robot manipulation, human activity, and filtered general-purpose videos, with Table 1 summarizing the datasets used.General-purpose videos are filtered to retain manipulation-related activities.Source: §2 · p. 2, Table 1 · p. 3
  • Trajectories with camera failures or abrupt discontinuities are removed, then segmented into semantically coherent clips lasting 1–12 seconds.Heuristic transition cues propose boundaries, and a VLM refines clip boundaries by merging or adjusting clips.Source: §2 · p. 3
  • Captions specify the task, active end effectors, target objects, and camera views to connect conditioning information with sub-task trajectories.The stated objective is to reduce uncertainty in future prediction and encourage consistent mappings from conditioning information to trajectories.Source: §2 · p. 3, Figure 2 · p. 3
  • Robotics observations are arranged as a T-shaped composite, with the primary view on the left and up to two auxiliary views on the right.Missing auxiliary views are filled with black placeholders.Source: §2 · p. 3
  • Because similar motions can have conflicting action representations across datasets, the pipeline aligns workspace frames and end-effector origins and axes to canonical coordinates.Figure 3 depicts aligned workspaces and end-effector coordinate frames.Source: §2 · pp. 3–4, Figure 3 · p. 3

3 VPP2: A GENERALIST POLICY WITH ZERO-SHOT CAPABILITY

VPP2 adapts a pretrained video foundation for manipulation, then trains an action expert on video-model representations. Its pipeline combines event-level learning, fixed-horizon single-pass prediction, and hierarchical VLM planning.

3.1 VIDEO PREDICTION MODEL TRAINING PIPELINE

  • Event-level pretraining aligns each manipulation subtask description with visual evolution from the initial observation to completion.Training samples future frames across the event and optimizes a flow-matching objective conditioned on the instruction and initial observation.Source: §3.1 · p. 4
  • Fixed-duration chunk post-training gives action learning a consistent temporal scale; predictions span 2 seconds for human hands and 8 seconds for robot manipulation.Consistency distillation enables single-forward-pass video-latent generation at inference.Source: §3.1 · pp. 4–5, Figure 4 · p. 5

3.2 ACTION MODELING

  • The action expert is a 0.9B-parameter MoT diffusion transformer trained on unified-coordinate bimanual data, with the video backbone initially frozen and later adapted using LoRA.The video and action modules take approximately 0.12 and 0.1 seconds, respectively, for approximately 0.22 seconds total per action chunk.Source: §3 · p. 4, §3.2 · p. 5

3.3 VLM FOR HIGH-LEVEL PLANNING.

  • A hierarchical VLM generates detailed subtasks for long-horizon or ambiguous instructions, handling high-level reasoning, semantic understanding, and memory.Detailed prompts resemble the training captions used for video generation.Source: §3.3 · pp. 5–6
  • For robot manipulation, VPP2 jointly predicts one to three camera views arranged in a T-shaped composite.Single-step predictions after distillation are shown in Appendix Figure 10.Source: Figure 5 · p. 6

4 EXPERIMENTS

The experiments evaluate VPP2’s video predictions, real-world zero-shot manipulation, and benchmark-specific post-training. Results report state-of-the-art RoboDojo performance, strong LIBERO generalization, and ALOHA success without task-specific fine-tuning.

4.1 VIDEO PREDICTION QUALITY ANALYSES

  • Human evaluators compared video predictions, and VPP2’s advantage was most pronounced on complex tasks requiring spatial understanding.The section also reports GPT-based instruction-following success rates and pairwise win rates against baselines; Figures 6–7 and Table 2 identify these comparisons.Source: §4.1 · p. 7, Figure 7 · p. 7, Figure 6 · p. 7

4.2 POLICY PERFORMANCE ANALYSIS

  • On real-world ALOHA tasks without task-specific fine-tuning, VPP2 averages 58.5% success across 10 categories, versus 40.0% for π0.5 and 20.5% for Fast-WAM.VPP2 performs best in 9 of 10 categories; Figure 8 presents the zero-shot success-rate comparison.Source: §4.2 · p. 7, Figure 8 · p. 8
  • On LIBERO-Pro, VPP2 reaches 45.0% overall versus at most 11.0% for VLA baselines, and on LIBERO-OOD it achieves 63.9% versus at most 10.5% for video-based policies.The LIBERO-OOD result also surpasses Temporal Ratio; Table 3 covers post-training on LIBERO-ID, LIBERO-Pro, and LIBERO-OOD.Source: §4.2 · p. 8, Table 3 · p. 8
  • VPP2 achieves state-of-the-art RoboDojo performance, with an average score of 35.51 and a success rate of 29.47%.It outperforms representative VLAs, world action models, and GPT-6-Astra; Table 4 reports simulation-benchmark results.Source: §4.2 · p. 8, Table 4 · p. 9

5 RELATED WORKS

The related work frames VPP2 as a prediction-first approach to WAMs: it adapts video priors for manipulation and then supports efficient action learning. It emphasizes preserving video generalization while learning actions.

Video Foundation Model

  • Video foundation models provide general-purpose priors, motivating a strategy that first learns a manipulation-focused video model and then adapts it for efficient prediction.VPP2 also uses VLM-generated manipulation-specific descriptions to reduce ambiguity, then post-trains for fixed-horizon prediction and distills for single-step generation.Source: §5 · p. 9

World Action Model

  • WAMs have progressed from future-frame prediction followed by inverse dynamics to conditioning actions on video features or jointly learning visual prediction and action generation.Iterative video generation can incur substantial latency for closed-loop control.Source: §5 · p. 9
  • VPP2 combines event-level pretraining and detailed captions with fixed-horizon post-training and single-step distillation, emphasizing instruction-aligned prediction as an efficient backbone for action learning.The related work notes that action learning can compromise pretrained video-model generalization and VLM semantic understanding.Source: §5 · p. 9

6 CONCLUSION

The conclusion presents VPP2 as a WAM with zero-shot generalization in video prediction and action generation. It reports strong results across video instruction following, real-world ALOHA, and post-trained challenge benchmarks, and calls for greater emphasis on generalizable video prediction and preserving it during action learning.

6 CONCLUSION

  • VPP2 continues pretraining the base video model for generalizable, instruction-following predictions, then learns a policy while preserving those predictive capabilities.The conclusion characterizes the resulting WAM as having strong zero-shot generalization in video prediction and action generation.Source: §6 · p. 9
  • The experiments report more faithful manipulation instruction following than substantially larger video models, strong zero-shot performance on real-world ALOHA, and the highest success rates after post-training on challenge benchmarks.Source: §6 · p. 9
  • The authors call for future WAM research to emphasize generalizable video predictions and preserve that generalization during action learning, beyond architectural choices for action modeling.Source: §6 · p. 9

AI USE STATEMENT

The authors report using generative AI tools for manuscript work, research assistance, and dataset annotation, with authors retaining responsibility for the final paper.

AI USE STATEMENT

  • Generative AI tools assisted with language polishing, manuscript drafting, and clarity and readability improvements.Source: AI USE STATEMENT · p. 10
  • The tools also suggested experimental parameter settings and assisted with literature and technical-information searches.Source: AI USE STATEMENT · p. 10
  • AI tools were used to annotate human and robotic manipulation datasets at scale, while authors evaluated suggestions and made final experimental decisions.Authors state that AI-assisted text was reviewed and retrieved information checked against original sources.Source: AI USE STATEMENT · p. 10

A DATASET PROCESS DETAILS

The dataset process uses detailed captions to specify manipulation events and align end effectors, targets, and camera views. It also maps robot poses into shared workspace and end-effector coordinate systems.

A.1 VIDEO CAPTIONING.

  • Captions specify how the end effector completes the task, including the sequence and manner of manipulation.Source: §A.1 · p. 15
  • Captions identify visible active end effectors and trim clips to begin when an end effector first appears; motions are described separately when multiple end effectors act.Source: §A.1 · p. 15
  • When similar objects appear, captions distinguish the target by attributes or spatial relationships, and require it to be visible in the first frame.Source: §A.1 · p. 15
  • Optical flow filters egocentric clips with excessive viewpoint changes, while captions describe camera views and moderate motion.Source: §A.1 · p. 15

A.2 UNIFIED ACTION SPACE

  • For each arm dataset, workspace and end-effector transformations align poses to canonical frames, making workspaces, origins, and axis orientations comparable.The pose is transformed as eT_d = A_d T_d B_d.Source: §A.2 · p. 15

B MORE VIDEO PREDICTION RESULTS

Additional open-ended video-prediction results compare single-step predictions before and after distillation. Distillation enables high-quality prediction in one sampling step.

B MORE VIDEO PREDICTION RESULTS

  • Additional video-prediction results cover open-ended tasks.Source: Figure 10 · p. 16
  • The comparison examines single-step video predictions before and after distillation.Source: Figure 10 · p. 16
  • Distillation enables high-quality video prediction with just one sampling step.Source: Figure 10 · p. 16

C.1 DETAILED LIBERO RESULTS

This section reports per-suite success rates on the standard LIBERO benchmark and relates them to LIBERO-ID results presented in Table 3.

C.1 DETAILED LIBERO RESULTS

C.2 DETAILED ROBODOJO RESULTS

The RoboDojo results report per-dimension average scores and success rates across five capability dimensions, including standard and randomized generalization settings.

C.2 DETAILED ROBODOJO RESULTS

  • RoboDojo evaluates generalist manipulation on 42 bimanual simulation tasks spanning five capability dimensions.Source: §C.2 · p. 17
  • The dimensions test generalization, precision, long-horizon completion, memory, and open task specifications.Generalization covers unseen backgrounds, lighting, clutter, and target objects under standard and randomized settings.Source: §C.2 · p. 17
  • Average score captures partial task progress, while success rate measures binary task completion.Source: §C.2 · p. 17
  • Tables 6 and 7 report per-dimension average scores and success rates, respectively.Table 6 averages over five dimensions and combines standard and randomized generalization scores; VPP2's pooled result uses 25 episodes per setting.Source: §C.2 · p. 17, Table 6 · p. 17, Table 7 · p. 18
  • Baseline results come from the official RoboDojo simulation leaderboard as of September 2026.Source: §C.2 · p. 17
Loading 2610.10270v1…