PaperlayerView exact revision on arXiv

Source-linked AI summary

Masked Visual Actions for Unified World Modeling

Hadi Alzayer, Wenlong Huang, Haonan Chen, Christopher Luey, Lvmin Zhang, Maneesh Agrawala, Gordon Wetzstein, Li Fei-Fei, Yilun Du, Jiajun Wu, Jia-Bin Huang

arXiv:2607.19343v1cs.CVcs.RO

TL;DR

Video models offer broad interaction priors but lack action representations aligned with their visual space and grounded in physical manipulation. Masked Visual Actions addresses this gap with pixel-space trajectory conditioning, supporting forward and inverse modeling across manipulation applications using one checkpoint finetuned on 15 hours of data.

  • Problem

    Existing video models lack action representations expressed directly in their visual space while remaining grounded in physical manipulation.

  • Method

    Masked Visual Actions conditions a pretrained video model on partially revealed pixel-space trajectories, enabling forward or inverse modeling by revealing robot or object motion.

  • Results

    A single checkpoint finetuned on 15 hours of data achieves strong visual fidelity and controllability across embodiments and supports policy evaluation, planning, and inverse modeling.

  • Takeaways & Limitations

    The framework unifies forward and inverse robot world modeling as conditional prediction problems in one video model across simulation and real-world manipulation.

  • Takeaways & Limitations

    The model learns interaction correlations rather than causal relationships and remains limited by the base video model’s inference speed and expressivity.

Abstract

from arXiv · show

Video models absorb rich priors over how the visual world moves, interacts, and responds to contact, making them promising substrates for robotic world modeling. The central challenge is how to communicate action to such models in a form aligned with the visual space in which they learned these interaction priors, yet still grounded in physical manipulation. We introduce Masked Visual Actions, a pixel-space control interface that expresses action as a partially revealed trajectory of an arbitrary entity in a video. Revealing robot motion makes the model act as a forward dynamics model that predicts the scene's response to low-level robot actions, while revealing desired object motion makes the same model recover robot behavior consistent with that outcome. Finetuned with only 15 hours of masked examples from real videos and simulation, a single checkpoint achieves strong visual fidelity and controllability across diverse scenes and multiple embodiments. In downstream manipulation settings, the model produces imagined rollouts whose outcomes correlate with real-world execution for policy evaluation, improves decision making by ranking candidate futures in model-based planning, and supports inverse modeling by synthesizing robot motion from desired object motion.

1 Introduction

Masked Visual Actions turns action into partially revealed pixel-space trajectories, enabling one pretrained video model to perform both forward prediction of scene responses and inverse recovery of robot motion. A single checkpoint, finetuned with limited real and simulated interaction data, is evaluated across robot-manipulation applications in simulation and the real world.

  • Motivation: Robotic world models should couple forward prediction of sensory consequences with inverse recovery of movements that realize desired states.The introduction frames purposeful interaction as reasoning from actions to effects and from desired states back to actions.
  • Motivation: Video models offer broad priors over motion, contact, persistence, deformation, and change, but most remain passive observers rather than intervention tools.Their priors come from large-scale observation and typically exceed what can be distilled from robot data alone.
  • Method: Masked Visual Actions represents action as a partially revealed spatiotemporal trajectory in pixel space, directly within a pretrained video model’s native representation.The method finetunes a pretrained video model to ingest masked trajectories of entities in the scene.
  • Method: Revealing the robot trajectory enables forward dynamics prediction, while revealing a desired object trajectory enables inverse modeling of robot behavior consistent with that outcome.The two directions are cast as complementary conditional prediction problems using the same video model and different revealed entities.
  • Evaluation: 15 hours of robot interaction data suffice to finetune a single checkpoint evaluated for policy evaluation, model-based planning, and inverse modeling in simulation and the real world.The framework is reported to improve visual fidelity relative to prior work and is validated across three robot-manipulation applications.

2 Related Work

Prior robotic video interfaces condition models with physical, geometric, trajectory, or goal signals, but these are not dense, pixel-aligned, or embodiment-shareable. Masked visual actions extend visual prompting and modality masking into a dense, image-aligned interface for unified robotic world modeling.

  • Controllable video generation as a robotic interface: Existing controllable video interfaces use physical forces, warped flow, hand poses, trajectory tracks, or goal images as robotic control signals.These approaches turn video models into simulators by adding control, but their signals are not dense, pixel-aligned, or shareable across embodiments.
  • Pixel-grounded action conditioning for robot world models: Masked visual actions differ by making robot motion and action directly visible through dense, image-aligned conditioning.Compared with compact low-dimensional actions or sparse visual representations, this exposes action in the image space used by video models.
  • Controllable video generation as a robotic interface: Masked visual actions extend inpainting-based visual prompting by using masked frames as a generic control interface for robotic world modeling.The approach varies which visual regions are revealed, adapting conditional inference in visual domains to robotics.
  • Pixel-grounded action conditioning for robot world models: Most robotic video world models encode actions through embodiment-specific end-effector poses, joint vectors, or skeletons, while newer methods use rendered masks, pointmaps, or Gaussian heatmaps.These pixel-grounded alternatives include URDF-rendered robot representations in BridgeV2W and Kinema4D and multi-view 7-DoF action heatmaps in Action Images.
  • Unified video-action models and downstream uses: Unified video-action models combine forward dynamics, inverse dynamics, policy, and video generation by masking modality channels or manipulating diffusion timesteps.Examples include UVA, UWM, AIM, X-WAM, and MotuBrain, alongside foundation-scale platforms such as Cosmos, Genie Envisioner, and DreamGen.

3 Masked Visual Actions

Masked Visual Actions condition a video model by revealing selected entities’ spatiotemporal pixels while predicting the rest. Partitioning entities into active and passive roles yields forward action-conditioned prediction and inverse behavior recovery, with passive-conditioned queries emerging zero-shot despite training only on active robotic masks.

  • Masked conditioning: Masked Visual Actions reveal selected entities’ spatiotemporal pixels and train the model to complete the remaining video conditioned on the initial scene.The mask is the union of regions occupied by conditioned entities, and training samples masks from a mask distribution.
  • Active and passive entities: Active entities act on the scene, whereas passive entities move through interaction with active entities, defining two inference directions.The partition describes model usage rather than an intrinsic property learned by the model.
  • Forward model: Conditioning on active entities produces a forward model that predicts passive scene responses from embodiment-agnostic masked videos of robot motion.This corresponds to action-conditioned dynamics modeling while replacing low-dimensional action commands with masked visual trajectories.
  • Inverse model: Conditioning on passive entities produces an inverse model that recovers agent behavior consistent with a user-specified desired world outcome.The passage identifies this direction as having no analog in conventional action-conditioned world models.
  • Generalization: The model generalizes zero-shot to passive-entity conditioning despite being trained only on masks depicting active robotic entities.At inference, any subset of entities can be selected, although the model has no explicit notion of agency.

4 Method

The method trains masked visual actions from real DROID videos and Robocasa simulations using segmentation- or rendering-based robot conditioning. It uses spatially aligned video conditioning with LoRA finetuning, while rendering enables arbitrary action trajectories at inference.

  • Dataset construction: The dataset combines success and failure trajectories from real-world DROID videos and Robocasa simulations, constructing masked conditioning through segmentation or robot-state rendering.Both approaches are designed to provide masked action inputs for video modeling.
  • Segmentation-based dataset: Segmentation isolates entities such as the robot arm without camera calibration or explicit robot identification, helping the model learn inpainting and joint scene distributions.DROID videos use the prompt “A robotic arm” with SegmentAnything.
  • Segmentation-based dataset: Segmentation conditioning requires accurate test-time masks and can leak scene-dynamics information through occluded robot regions.Rendering robot geometry is introduced to mitigate both limitations.
  • Rendering-based dataset: Rendering-based conditioning aligns a robot mesh to the input video, allowing arbitrary action trajectories at inference but requiring robot state and camera calibration.The rendering-based dataset follows protocols from the DROID setup.
  • Model and training: The model encodes masked conditioning with the video model’s autoencoder, concatenates spatially aligned inputs, fills missing regions with gray, and applies rank-256 LoRA finetuning.Wan-Fun-Control 2.2 14B is used as the base model with batch size 4.

5 Experiments

Experiments evaluate Masked Visual Actions for visual world modeling, unseen-embodiment generalization, planning, policy evaluation, and inverse modeling. The model follows robot actions, supports imagined trajectory selection and policy assessment, and recovers competitive robot behavior from desired object motion.

  • World modeling and generalization: Masked Visual Actions accurately follow robot actions on held-out DROID scenes and generalize to the unseen bimanual R1-Pro embodiment, unlike raw robot-state conditioning.Ctrl-world also follows actions on held-out DROID scenes, but its training included those scenes; raw-state conditioning cannot generalize to unseen embodiments.
  • Planning: Video-model rollouts let Best-of-N planning rank stochastic-policy trajectories in imagination, improving task success consistently as more candidate samples are evaluated.The planning experiments use Diffusion Policy as the stochastic policy and simulate candidate action trajectories before execution.
  • Policy evaluation: The model’s simulated success rates track ground-truth policy performance across RoboCasa tasks and real-world demonstrations, but imagination consistently overestimates task progress.For four real-world tasks, the study collected 20 demonstrations per task and scored real and simulated executions with partial-progress rubrics.
  • Action extraction: For inverse modeling, desired object motion is rendered as a masked visual action, after which synthesized robot videos are converted into executable low-level actions with learned inverse dynamics.Although the video model was not trained specifically for inverse modeling, it recovers competitive robot behavior by exploiting interaction priors.

6 Discussion and Conclusions · Appendix Overview · A Project webpage

The discussion presents Masked Visual Actions as an efficient way to reuse pretrained video priors for counterfactual simulation, forward modeling, and inverse modeling, while noting limitations from learned correlations and the base model. The appendix and project webpage provide supplementary materials, reproducibility resources, quantitative results, and video demonstrations.

  • 6 Discussion and Conclusions: Finetuning on a small amount of Masked Visual Actions data enables efficient counterfactual synthesis conditioned on selected scene entities.The approach leverages the prior of a pretrained video model.
  • 6 Discussion and Conclusions: Conditioning on robotic embodiment visualization lets the model simulate robot actions as an action-conditioned forward model.This uses the model to predict scene responses to robot motion.
  • 6 Discussion and Conclusions: As an inverse model, the system synthesizes suitable robot motion for realistically manipulating a target object.The inverse formulation conditions behavior on the desired object outcome.
  • 6 Discussion and Conclusions: The model learns correlations between object interactions rather than causal relationships, and its capabilities remain limited by the base video model.These limitations include the base model’s inference speed and expressive capacity.
  • Appendix Overview: The appendix supplements the main paper with eight sections covering implementation details, evaluation protocols, and data collection procedures.It includes project webpage references, reproducibility materials, full quantitative results with standard error bars, and additional training details.
  • A Project webpage: The project webpage provides video results and qualitative demonstrations of model performance on manipulation tasks.The referenced webpage is https://masked-visual-actions.github.io

C Full quantitative results

Table C1 reports quantitative results for all reconstruction experiments, including standard errors for every metric. The method consistently outperforms all baselines, while real-world performance with an unseen custom gripper is within the standard error of skeleton conditioning.

  • Quantitative reconstruction results: Table C1 presents quantitative results across all reconstruction experiments with a standard error reported for each metric.The table covers the complete set of reconstruction experiments.
  • Quantitative reconstruction results: The method consistently outperforms all baselines across the reported reconstruction metrics.This comparison is stated for the quantitative results in Table C1.
  • Real-world reconstruction: For real-world data captured with an unseen custom gripper, performance is within the standard error of conditioning on the skeleton.The custom gripper was used on the robot during data capture.

D Additional training details

Training data were constructed to prevent DROID scene overlap with validation and testing while combining approximately 1,000 DROID demonstrations and 4,000 RoboCasa examples, including failure cases for counterfactual generation.

  • Dataset construction: DROID scenes from labs CLVR and RAD were excluded from training and reserved for testing to prevent scene overlap.This separation ensures validation and testing contain no scenes from the training data.
  • Dataset construction: Approximately 1,000 DROID demonstrations using two external cameras were processed with both segmentation-based and rendering-based pipelines.DROID failure cases were retained to support accurate counterfactual generation.
  • Dataset construction: 4,000 RoboCasa examples across non-navigation tasks were added, with generated failure cases incorporated into training data.The included RoboCasa tasks exclude robot navigation.

E VLM Evaluation Protocol · E.1 Gemini system prompt

The evaluation uses Gemini 3.1 Pro Preview to judge generated manipulation rollouts for physically caused outcomes, realism, and task success under a strict, structured protocol. Rollouts are ranked lexicographically, and the selected rollout determines planner success against ground truth.

  • E VLM Evaluation Protocol: Gemini 3.1 Pro Preview evaluates every rollout video at temperature 0 and 15 fps, exposing all 81 clip frames to a task-agnostic system instruction.Per-rollout prompts add only the task description and success criterion; evaluation policy remains in the system instruction.
  • E VLM Evaluation Protocol: Rollouts are selected by descending κ = (1[success], realism, confidence, 1[contact], −r), prioritizing success, realism, confidence, contact, then lower rollout index.The selected rollout is arg maxr κ(vs,r).
  • E VLM Evaluation Protocol: Planner success is the fraction of scenes whose selected rollout is ground-truth correct, while success-versus-N averages Gemini’s top-ranked rollout over every size-N candidate subset.At N=1, the random baseline equals the per-scene mean ground-truth rate.
  • E.1 Gemini system prompt: The evaluator credits success only when the robot visibly causes goal-relevant motion through plausible physical contact and reaches the goal during that contact window.Final-frame correctness alone is insufficient.
  • E.1 Gemini system prompt: The prompt rejects ghost contact unless gripper-object coupling and co-located motion are visible, treating near, glancing, implied, or magic contact as autonomous motion.The gripper must touch or compress against the object while the object moves.
  • E.1 Gemini system prompt: Because the simulated environment has no momentum, motion after gripper disengagement is autonomous, and only progress during active contact and gripper movement counts.Post-disengagement coasting cannot complete the credited goal state.
  • E.1 Gemini system prompt: The evaluator labels causal mechanisms as robot_pushed, robot_grasped, teleported, vanished, autonomous, passthrough, no_attempt, or other, while penalizing common visual artifacts.Penalized artifacts include teleporting, morphing, gripper penetration, vanishing or duplicated objects, frame jumps, ghost contact, and coasting.
  • E.1 Gemini system prompt: The structured JSON response contains five ordered fields covering causal outcome, task success, gripper contact, physics realism, and a free-form summary.Physics realism is scored as an integer from 1 to 5, and causal mechanism precedes the binary success flag.

F Policy and Inverse-Dynamics Training Details

The section distinguishes three training settings: simulation policies for RoboCasa planning and evaluation, an inverse-dynamics model for action extraction, and policies for real-world rollouts.

  • Training settings: Training details cover three settings: RoboCasa simulation policies, a learned inverse-dynamics model, and real-world rollout policies.The settings correspond to planning and policy evaluation, action extraction, and real-world rollout studies, respectively.
  • Simulation policies: Simulation policies are used for planning and policy evaluation in RoboCasa.
  • Inverse dynamics and real-world policies: The learned inverse-dynamics model is used for action extraction, while separate policies support real-world rollout studies.

F.1 Simulation Policy Training for Planning and Policy Evaluation · F.2 Learned Inverse Dynamics for Action Extraction · F.3 Real-World Policy Training

The paper trains task-specific simulation policies for planning and evaluation, uses learned inverse dynamics to convert generated interactions into executable actions, and trains real-world policies on captured manipulation data. These components share explicit observation-action contracts tailored to simulation or real-world control.

  • F.1 Simulation Policy Training for Planning and Policy Evaluation: Simulation policies are Diffusion Policies trained on task-specific mixtures of human demonstrations and MimicGen trajectories from official RoboCasa sources.Each policy conditions on RGB observations from two fixed cameras and robot state, predicting a full 12-D action sequence over the episode horizon.
  • F.1 Simulation Policy Training for Planning and Policy Evaluation: In planning, sampled policy actions are simulated with the video model, and a VLM-based evaluator ranks the resulting predicted futures.In policy evaluation, the same policy class supplies trajectories whose success rates are compared between the simulator and video model.
  • F.2 Learned Inverse Dynamics for Action Extraction: For action extraction, the video model synthesizes a robot interaction from desired object motion, after which learned inverse dynamics converts the video into low-level actions.The inverse-dynamics model is trained on COFFEESERVEMUG data using the native RoboCasa action space.
  • F.2 Learned Inverse Dynamics for Action Extraction: The inverse-dynamics model predicts full action sequences in the native 12-D RoboCasa action space with a transformer architecture optimized by AdamW for 300 epochs.This establishes a direct mapping from synthesized visual interactions to executable control sequences.
  • F.2 Learned Inverse Dynamics for Action Extraction: Diffusion Policy, ACT, and SmolVLA serve as narrow action-extraction controls using only the generated video’s first frame rather than the full interaction.Because these baselines do not process the complete synthesized interaction, they require no separate discussion beyond this subsection.
  • F.3 Real-World Policy Training: Real-world rollout studies train image-conditioned Diffusion Policy and ACT models on mug-on-rack and close-cabinet tasks, each with 40 trajectories sampled at 10 Hz.Examples contain four synchronized RGB views and an 8-D joint-and-end-effector action sequence under a shared observation-action contract.

G Robot Data Collection … I Adapting Baselines to Unseen Embodiments

The paper collects synchronized real-robot demonstrations with calibrated wrist and external cameras, stores trajectories with rendering metadata, evaluates task progress by sub-stage completion, and adapts to unseen robots through URDF renderings. Calibration uses a shared AprilTag reference, while baseline evaluation targets bimanual humanoid manipulation in BEHAVIOR-1K.

  • G Robot Data Collection: Demonstrations use a Franka Panda with a compliant end-effector, teleoperated through GELLO while recording synchronized robot states and RGB observations.Observations come from wrist-mounted and external cameras during rollouts from initial scene state s0.
  • G.1 Hardware Setup: Two ZED Mini 2 cameras provide egocentric wrist and fixed third-person workspace views, recording RGB frames at 720p and 30 Hz.The wrist camera is mounted near the end-effector, while the external camera observes the workspace.
  • G.2 Camera and Robot Frame Calibration: A table-mounted AprilTag calibrates camera and robot frames by combining Franka kinematics with wrist-camera tag observations across diverse robot poses.The procedure estimates the wrist-camera hand-eye transform and the AprilTag pose in the robot base frame.
  • G.2 Camera and Robot Frame Calibration: The known AprilTag pose enables computation of the external-camera pose relative to the robot base from detections of the same workspace tag.The tag supplies a shared reference frame between the robot, wrist camera, and external camera.
  • G.3 Trajectory Representation: Each rollout is stored in HDF5 with RGB frames, joint states, timestamps, and calibration metadata for trajectory-based experiments.Stored trajectories include external and wrist observations and measured robot joint state.
  • G.3 Trajectory Representation: Visual robot masks are rendered by combining joint states with the Franka kinematic model, custom end-effector geometry, and calibrated camera parameters.This renders robot geometry into the image plane.
  • H Real World Tasks Rubrics: Demonstration scores award one point per completed sub-stage and report progress as the fraction earned, enabling comparison across tasks of differing length.Rubrics cover bagging an orange, stacking blocks, placing a towel, and closing a drawer.
  • I Adapting Baselines to Unseen Embodiments: Unseen-embodiment evaluation uses BEHAVIOR-1K household mobile manipulation with a bimanual humanoid robot, adapting the model by rendering the new robot’s URDF.The model accepts the new embodiment through its masked visual action representation.
Loading 2607.19343v1…