PaperlayerView exact revision on arXiv

Source-linked AI summary

Turning Video Models into Generalist Robot Policies

Sizhe Lester Li, Evan Kim, Xingjian Bai, Tong Zhao, Tao Pang, Max Simchowitz, Vincent Sitzmann

arXiv:2605.27817v1cs.ROcs.AIcs.CVcs.LG

TL;DR

Robot policies must generalize across embodiments and environments, but action data is limited and translating a video model’s visual plan into robot commands remains a challenge. This paper pairs an action-free video planner with a Jacobian-based inverse dynamics model in closed-loop VERA, reporting successful simulated and real-world control across embodiments. Its results support decoupled video planning and faithful action translation as a modular route to robot control, while leaving practical limits around robot-specific planner post-training, optical-flow supervision, and RGB-only force reasoning.

  • Problem

    Robot control seeks to generalize across embodiments and environments, yet video plans must still be translated into robot actions and paired video-action-text training data is rare.

    Source: §1 · p. 2

  • Method

    VERA pairs an action-free video world model with a Jacobian-IDM that inverts a learned tangent map from action perturbations to pixel motion, executing action chunks and replanning.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    VERA succeeds in simulated and real settings, including zero-shot Panda manipulation and Allegro-hand cube reorientation, and the video planner can serve multiple embodiments with different J-IDMs.

    Source: §1 · p. 2, §4.1 · pp. 5, 7

  • Takeaways & Limitations

    Decoupled video planning with a faithful IDM is a modular alternative for controlling diverse embodiments and tasks while preserving the planner’s visual reasoning.

    Source: §5 · p. 8

    VERA still needs robot-specific video for planner post-training, off-the-shelf optical-flow trackers for J-IDM supervision, and cannot infer force-based control from RGB alone.

    Source: §5 · p. 8

Abstract

from arXiv · show

Video generative models have emerged as a promising robotics backbone, capable of generating videos that depict the completion of complex tasks across embodiments and environments. Recent work proposes robot foundation models that jointly predict future observations and actions by finetuning video models with action-labeled data. In this paper, we test the limits of an alternative approach: leave the video planner as-is while training an embodiment-specific inverse dynamics model (IDM). This decoupling offers several natural benefits: the video planner remains embodiment-agnostic, different video models can be interchanged easily without re-training the IDM, and the IDM can be independently trained with readily available self-play data. We present a closed-loop, video-to-action policy that combines an action-free video world model with a carefully-designed IDM based on the robot embodiment Jacobian. We demonstrate that our IDM design is both data-efficient and scalable to high-dimensional action spaces. Our policy, which we coin the Video-to-Embodied Robot Action Model (VERA), achieves strong performance across simulated and real-world benchmarks, including zero-shot Panda arm manipulation and 16-DoF Allegro-hand dexterous cube re-orientation. The same video planner can be used across multiple embodiments by pairing it with different embodiment-specific IDMs. Our results show that decoupled video planning plus faithful video-to-action translation is a viable alternative route towards zero-shot, cross-embodiment, and generalizable robot control. More results are available on our project website: https://vera.csail.mit.edu.

1 Introduction

The paper addresses the challenge of generalizing robot control across embodiments, tasks, and environments when robot action data is limited. It proposes decoupling video planning from action translation, pairing an action-free planner with an embodiment-specific Jacobian IDM in the closed-loop VERA policy.

1 Introduction

  • Robot control must generalize across embodiments, tasks, and environments, but strong generalization with vision-language-action models remains challenging with comparatively small action datasets.Unlike web-scale paired text and image data, web-scale robot action data does not exist.Source: §1 · p. 2
  • Video models offer an alternative because they generate visual plans, are not tied to one robot’s action space, and can draw on abundant video data.Integrating actions into the video backbone instead requires rare paired video, action, and text data.Source: §1 · p. 2
  • A separate IDM can use task-agnostic video-action data, be swapped across embodiments, and remain compatible with stronger video backbones, but inaccurate translation can fail despite a correct plan.Prior direct-regression IDMs often generalize poorly with limited action data, with accuracy degrading as action spaces grow more complex.Source: §1 · p. 2
  • VERA combines a 14B video model with a Jacobian-IDM that translates visual lookahead into action chunks, then replans from new observations.The Jacobian-IDM predicts actions by inverting a learned tangent map between action perturbations and pixel motion.Source: §1 · p. 2
  • VERA succeeds in simulation and real settings, including zero-shot Panda manipulation and 16-DoF Allegro-hand cube reorientation, and supports planner reuse with embodiment-specific IDMs.The paper also compares VERA with recent VLA and WAM baselines on real-world manipulation tasks.Source: §1 · p. 2

2 Related Work and Preliminaries

Related work spans action-supervised behavior cloning and VLA policies, generative video planners, and models that combine video prediction with action generation. VERA instead keeps planning in video space and uses separate embodiment-specific bridges to translate visual motion into controls.

Behavior Cloning and Vision-Language-Action Models

  • VLA and generalist policies extend action-supervised behavior cloning across broader data mixtures and action spaces, while still coupling perception and control within an embodiment-specific model.Examples include RT-1 and RT-2 [2], PaLM-E, RoboCat, Open X-Embodiment / RT-X, Octo, OpenVLA, and the π-family [25] [26] [27] [28].Source: §2 · p. 3

Video Generative Models

  • Video-model research builds on large-scale pretraining and open-weight WAN diffusion transformers [31], with Diffusion Forcing [32] [33] enabling variable-context conditioning.Large-scale video pretraining has produced models with emergent visual priors and rudimentary physical reasoning [30].Source: §2 · p. 3

Joint Video-and-Action Models

  • Joint video-and-action models co-predict visual futures and robot actions, whereas VERA separates action-free visual planning from embodiment-specific action translation.Examples include GR-2 [7], Video Prediction Policy [8], LingBot-VA and DreamZero / World Action Models [9] [10], and action-conditioned models or simulators [11] [12] [34].Source: §2 · p. 3

Video Models with Inverse Dynamics Models

  • Related pixel-space planner-and-translator methods include Direct Video-Action, whose real-world long-horizon bimanual manipulation provides evidence that decoupled architectures can scale to deployment.Source: §2 · p. 3

3 The Video-to-Embodied Robot Action Model

The method separates visual planning from embodiment-specific action recovery: a video world model proposes a visual future, a Jacobian IDM translates motion into controls, and the robot executes a prefix before replanning. The IDM is structured around pixel motion to support faithful translation under limited data and complex action spaces.

3.1 The Video-to-Action Problem

  • The video-to-action problem is recovering embodiment-specific commands from predicted frames, with a good IDM expected to be faithful, data-efficient, and scalable to complex action spaces.Direct IDM regresses actions from image pairs, but may lose faithfulness with limited data and increasing action dimension.Source: §3.1 · pp. 3–4

3.2 The Jacobian Inverse Dynamics Model

  • The embodiment Jacobian maps action perturbations to 3D point motion, but those points are not directly observed, motivating an image-space Jacobian conditioned on the current image.The image-space field assigns each pixel a matrix that linearizes its motion under an action increment; optical flow supplies the pixel-motion measurement used for inversion.Source: §3.2 · p. 4
  • J-IDM trains on observation, action-perturbation, and next-observation tuples, combining a pixel-flow prediction objective with an action-reconstruction term.The flow supervision is extracted using off-the-shelf motion estimators, and action recovery uses a regularized pseudoinverse.Source: §3.2 · p. 4

3.3 Video World Model as a Closed-Loop Robot Planner

  • The planner is post-trained for video prediction on robot video without an action head, with video-to-video finetuning and tiled multi-view inputs supporting policy use.Source: §3.3 · p. 4
  • The controller translates and executes only the first K frames of the M-frame visual plan, yielding a K-action chunk while preserving a longer visual lookahead.The paper reports that chunking improves performance [23] [43].Source: §3.3 · p. 5
  • After each chunk, the robot’s new observations return to the planner for replanning; larger K favors coherent chunks and amortized generation time, while smaller K reduces execution drift.Source: §3.3 · p. 5, Figure 2 · p. 5

4 Experiments

The experiments evaluate VERA’s closed-loop manipulation, J-IDM’s translation fidelity and scaling, and comparisons with robot foundation-model baselines. Results span zero-shot Panda manipulation, multi-embodiment control, simulated tasks, and real-world reasoning-heavy prompts.

4 Experiments

  • The evaluation covers closed-loop simulation and hardware tasks, using a Wan-family pretrained planner and embodiment-specific J-IDMs initialized from VGGT.Benchmarks include PushT, MimicGen Panda, Allegro-Sim cube reorientation, Panda-Real, and Allegro-Real.Source: §4 · p. 5

4.1 Main Results

  • VERA performs zero-shot, prompt-following Panda manipulation across varied lighting, camera configurations, and object orientations.The evaluation includes complex prompts and vision-language reasoning tasks.Source: §4.1 · p. 5, Figure 3 · p. 6
  • One video model controls both the Panda arm and Allegro hand when paired with their embodiment-specific J-IDMs, including contact-rich block reorientation.The video model is fine-tuned across DROID and simulated and real Allegro-hand data.Source: Figure 4 · p. 6, §4.1 · p. 7

4.2 Designing a Faithful IDM for Video-to-action Translation

  • At fixed data, J-IDM alone preserves faithful reconstruction as DoFs grow; at five DoFs, it is approximately 2× more data-efficient.The comparison uses a controlled 2D toy-finger study that sweeps DoFs and self-play data quantity.Source: §4.2 · p. 7, Figure 5 · p. 7
  • With the planner held fixed, J-IDM outperforms UniPi* on every closed-loop task, and yields higher success across planar pushing, 7-DoF arm, and 16-DoF dexterous manipulation.J-IDM also achieves the lowest action-reconstruction MSE on all but one tested environment.Source: §4.2 · p. 7, Table 1 · p. 7, Table 2 · p. 8

4.3 VERA and other Robotic Foundation Models

  • 90% for DreamZero, 60% for VERA, and 30% for π0.5 on basic “push A”/“pick up B” tasks.VERA’s failures generally stem from low-fidelity video-to-action translation despite successful dreamed futures.Source: §4.3 · p. 8
  • On challenging real-world tasks, VERA is substantially better than foundation-model baselines, while remaining competitive on basic tasks.DreamZero and π0.5 frequently act on incorrect objects in challenging prompts, whereas VERA succeeds.Source: Figure 6 · p. 8, §4.3 · p. 8
  • VERA produces executable plans for finding and pressing a button hidden behind a wall and visible from only one of three cameras.DreamZero and π0.5 struggle to locate and navigate around the wall; DreamZero’s dreamed future also fails to plan the task.Source: Figure 3 · p. 6, §4.3 · p. 8

5 Discussion and Conclusion

The paper concludes that strong video models paired with faithful inverse dynamics can control diverse embodiments and tasks, while identifying several remaining constraints and research directions.

Discussion and Conclusion

  • Strong video models paired with a faithful IDM can control diverse embodiments and tasks, with Jacobian IDM offering data efficiency and scaling with action dimensionality.The paper also argues this design preserves planner visual reasoning that an end-to-end action head would otherwise dilute.Source: §5 · p. 8

Limitations

  • VERA still requires robot-specific video for planner post-training, optical-flow trackers for J-IDM supervision, and RGB-insufficient force reasoning; faithful recovery also degrades with little visible motion.These constraints bound the data, supervision, and control settings addressed by the method.Source: §5 · p. 8

Future Work

  • Future directions include sim-to-real J-IDM transfer, planner co-training across humans and robots, and an embodiment-conditioned Jacobian for multiple morphologies.The proposed directions target reduced reliance on real-robot action data and broader embodiment coverage.Source: §5 · p. 8

A Implementation Details

The appendix collects implementation details for the video model, inference, datasets, and conditioning, while fixing notation for context, prediction, and committed-prefix lengths.

A Implementation Details

  • Appendix A covers model architecture and training, ablations, inference constants, datasets, and tokenization and conditioning encoders.Its notation uses N context frames, M predicted frames, and a committed prefix of length K.Source: A Implementation Details · p. 14

A.1 Architecture and Training Recipe of Video Models

The planner uses Wan-family video diffusion transformers to predict observations without an action head, then adapts to robot video and supplies visual futures for closed-loop control.

A.1 Architecture and Training Recipe of Video Models

  • The planner uses Wan-family open-weight video diffusion transformers and predicts future observations from observation history and, when available, a task specification, without robot actions or an action head.The action-free design separates visual planning from embodiment-specific control.Source: §A.1 · p. 14
  • Robot post-training uses generative prediction on robot video, with future RGB observations—not actions—as targets, adapting to robot scenes, viewpoints, objects, and contact dynamics.The robot-video objective preserves the planner’s action-free interface.Source: §A.1 · p. 14
  • Training conditions on Nctx context latent frames and predicts M future latent frames, applying loss only to the future segment to match closed-loop visual look-ahead.The context frames condition prediction, while the target is a short visual future translated into actions at deployment.Source: §A.1 · p. 14
  • Diffusion forcing assigns different noise levels across frames, allowing clean or partially noised history to condition denoising of future frames.The objective minimizes mean-squared error against the diffusion target for future latents; context frames are excluded from the loss.Source: §A.1 · p. 14
  • Longer context can aid temporal consistency and contact-state disambiguation, while longer look-ahead gives the IDM a richer path but raises compute and prediction difficulty.The controller therefore uses short local look-aheads and replans frequently from new observations.Source: §A.1 · pp. 14–15
  • Multiple camera views are concatenated into one frame, and missing views are blank-padded to support mixed-embodiment planner training without changing the backbone.The Wan-14B planners use one node of 8 H200 GPUs, while Wan-1.4B planners use a single H100; the smaller model serves development and ablations, and 14B the main experiments.Source: §A.1 · pp. 14–15, Table 3 · p. 14

A.2 Video Model Ablations

The ablations favor pretrained video initialization and global camera context for prediction, while the deployment controller commits short action chunks and replans from fresh observations.

A.2 Video Model Ablations

  • The LVP-initialized 14B model has the lowest reported validation loss, while Wan initialization substantially lowers loss versus random initialization under matched data and views.These are held-out future-frame prediction diagnostics, not closed-loop success metrics; some scaling runs terminated early.Source: §A.2 · p. 15, Table 4 · p. 15
  • Longer context can clarify contact state and longer look-ahead gives the IDM a richer path, but longer sequences cost more memory and computation and make prediction harder.The controller therefore uses short local look-aheads and frequent replanning.Source: §A.2 · p. 15
  • Two external views yield the lowest normalized validation loss; wrist-only prediction performs substantially worse, and the main real-robot planner uses multi-view inputs when available.The triview configuration is slightly worse by this metric but combines global context with wrist information near contact.Source: §A.2 · pp. 15–16, Table 5 · p. 16

A.3 Inference Constants and Closed-Loop Rollout

  • VERA predicts M future frames, translates the first K into actions, executes that chunk, then replans from new observations and discards the uncommitted frames.K is kept small to replan often during contact-rich manipulation; low-level command counts per latent transition vary by task control rate and frame stride.Source: §A.3 · p. 16, Table 6 · p. 16

A.4 Datasets and Embodiments

The evaluation spans simulated and real-world pushing, arm manipulation, and dexterous hand tasks, with RGB-only video planning paired with embodiment-specific action translation. The section also specifies how video is tokenized, conditioned, and used in the closed-loop policy.

A.4 Datasets and Embodiments

  • Evaluation spans planar pushing, 7-DoF arm manipulation, and 16-DoF dexterous in-hand tasks; the video planner is trained on RGB observations only.The Jacobian IDM additionally uses paired robot actions and visual-motion estimates for embodiment-specific video-to-action translation.Source: §A.4 · p. 16
  • PushT-Sim uses 206 expert episodes, while MimicGen supplies four Panda tasks with 2,000–3,000 episodes per task.PushT-Sim is planar pushing; the MimicGen tasks are square, coffee, stack, and stack_three.Source: §A.4 · pp. 16–17
  • Allegro-Sim uses 2,126 planner-generated reorientation episodes plus 401 auxiliary random-motion episodes for local Jacobian-IDM coverage.The auxiliary trajectories are not goal-directed reorientation demonstrations.Source: §A.4 · p. 17
  • Real Allegro data includes 60 Phase 1 episodes and a Phase 2 pretraining mix of up to 217, while Panda planner post-training uses 39,816 successful DROID trajectories.Physical-robot evaluation uses held-out trial conditions for Allegro and held-out prompts, object configurations, and camera placements for Panda.Source: §A.4 · p. 17
  • Closed-loop task success is the primary metric, with fresh simulation rollouts or task instances and physical-robot execution used for evaluation.Validation losses are reserved for model selection and diagnostics.Source: §A.4 · pp. 17–18

A.5 Tokenization and Conditioning Encoders

  • The fixed Wan-family tokenizer and decoder map RGB clips to latent video tokens and back, while causal temporal compression maps N latent context frames to 4N−3 RGB frames and M predicted frames to 4M RGB frames.The closed-loop controller decodes only the committed prefix before translating it into actions.Source: §A.5 · p. 18
  • The planner uses language prompts when available, concatenates camera views into wide RGB frames, and pads missing view slots for shared cross-embodiment inputs.Without task-conditioning signals, it is trained as an observation-history-to-video predictor.Source: §A.5 · p. 18
  • Robot actions are excluded from video-model tokenization and prediction; at deployment, the Jacobian IDM translates decoded visual prefixes into low-level actions.This preserves action-free visual planning with embodiment-specific control.Source: §A.5 · p. 18

B Jacobian Training Details

The Jacobian IDM uses regularized inversion to recover actions from predicted visual motion. Its model predicts a dense per-pixel mapping from robot commands to optical flow, with architecture choices varying by scale.

B.2 Jacobian Architecture Details

  • The Jacobian field maps a low-level command δu to a 2D optical-flow vector at each pixel and is supervised using flow from adjacent video frames.At deployment, the model is inverted to obtain commands from generated visual flow.Source: §B.2 · p. 19

B.1 Jacobian Inversion

  • Ridge-regularized inversion provides the closed-form inverse used to recover command perturbations from generated visual flow.At deployment, the predicted Jacobian is inverted to obtain δu from visual flow.Source: §B.2 · p. 19

B.2 Jacobian Architecture Details

  • Simulation-scale Jacobian models use a DPT neck and head over frozen DINOv2, while larger real-world models use VGGT.Source: §B.2 · p. 19

B.3 Jacobian Ablation Details

The ablation setup matches D-IDM baseline capacity, and the hidden-button qualitative results show the model finding and pressing the concealed button.

B.3 Jacobian Ablation Details

  • D-IDM baselines match the DINOv2 encoder and DPT neck parameter sizes, with the decoding head changed to predict actions instead of a spatial Jacobian.The ablation Jacobian model uses the DINOv2 architecture, making the baseline parameter counts matched.Source: §B.3 · p. 19
  • The hidden-button visualization pairs a dream rollout with robot execution, where the model finds and pushes the concealed button.Source: Figure 7 · p. 19

C Additional Results and Qualitative Analyses

The additional analyses identify pretrained video initialization and closed-loop chunking choices as important to rollout quality. They also describe simulation data as useful but complementary to real-world adaptation.

C Additional Results and Qualitative Analyses

  • Pretrained Wan-family initialization is substantially more data-efficient than training from scratch on the same robot data, which did not match its temporal coherence or rollout stability within the compute budget.Main-result planners therefore use pretrained video checkpoints.Source: C Additional Results and Qualitative… · pp. 19–20
  • Longer action chunks initially improve smoothness and reduce replanning, but overly long chunks can hurt performance as contact or visual-tracking errors accumulate.The main experiments use an intermediate chunk length.Source: Figure 8 · p. 20
  • The default planner setting is N=6 context frames and M=4 future frames; short context hurts long-horizon coherence, while aggressive look-ahead adds limited benefit under frequent replanning.Fixed-length future generation was more reliable than variable-length generation because it avoids mismatch with the deployment call pattern.Source: C Additional Results and Qualitative… · p. 20
  • Simulation pretraining can improve real-world performance when it covers relevant local motion and contact, but real-image adaptation remains important for appearance, lighting, and camera-geometry mismatch.The paper treats simulation pretraining as complementary to, not a replacement for, real-world post-training.Source: C Additional Results and Qualitative… · p. 20

Related papers

Loading 2605.27817v1…