PaperlayerView exact revision on arXiv

Source-linked AI summary

BridgeV2W: Bridging Video Generation Models to Embodied World Models via Embodiment Masks

Yixiang Chen, Peiyan Li, Jiabing Yang, Keji He, Xiangnan Wu, Yuan Xu, Kai Wang, Jing Liu, Nianfeng Liu, Yan Huang, Liang Wang

arXiv:2602.03793v1cs.ROcs.CV

TL;DR

Embodied world models struggle to align coordinate-space actions with pixel-space videos, handle changing viewpoints, and share one architecture across embodiments. BridgeV2W renders actions as camera-specific embodiment masks and injects them into a pretrained video model with ControlNet-style conditioning and flow-based motion supervision, improving generation quality across single- and dual-arm settings and supporting downstream evaluation and planning.

  • Problem

    Embodied world models face coordinate-space action and pixel-space video mismatch, viewpoint sensitivity, and non-unified architectures across embodiments.

  • Method

    BridgeV2W renders actions into pixel-aligned embodiment masks using URDF and camera parameters, injects them into a pretrained video model via ControlNet-style conditioning, and adds flow-based motion supervision.

  • Results

    BridgeV2W improves video generation quality over prior state-of-the-art methods on single-arm DROID and dual-arm AgiBot-G1 datasets covering unseen viewpoints and scenes.

  • Takeaways & Limitations

    BridgeV2W shows potential for downstream policy evaluation and goal-image-conditioned planning, including strong correlation between simulated evaluation and real-world success.

  • Takeaways & Limitations

    The operational definition does not prescribe a particular architecture or training objective and assumes an initial frame, action sequence, and corresponding RGB video output.

Abstract

from arXiv · show

Embodied world models have emerged as a promising paradigm in robotics, most of which leverage large-scale Internet videos or pretrained video generation models to enrich visual and motion priors. However, they still face key challenges: a misalignment between coordinate-space actions and pixel-space videos, sensitivity to camera viewpoint, and non-unified architectures across embodiments. To this end, we present BridgeV2W, which converts coordinate-space actions into pixel-aligned embodiment masks rendered from the URDF and camera parameters. These masks are then injected into a pretrained video generation model via a ControlNet-style pathway, which aligns the action control signals with predicted videos, adds view-specific conditioning to accommodate camera viewpoints, and yields a unified world model architecture across embodiments. To mitigate overfitting to static backgrounds, BridgeV2W further introduces a flow-based motion loss that focuses on learning dynamic and task-relevant regions. Experiments on single-arm (DROID) and dual-arm (AgiBot-G1) datasets, covering diverse and challenging conditions with unseen viewpoints and scenes, show that BridgeV2W improves video generation quality compared to prior state-of-the-art methods. We further demonstrate the potential of BridgeV2W on downstream real-world tasks, including policy evaluation and goal-conditioned planning. More results can be found on our project website at https://BridgeV2W.github.io .

1. Introduction

BridgeV2W addresses mismatches in embodied world models by converting actions into pixel-aligned embodiment masks and injecting them into pretrained video generation models. It further emphasizes task-relevant motion and demonstrates improved generation quality and downstream utility across embodiments, viewpoints, and scenes.

  • Motivation: Existing embodied world models support policy evaluation and goal-conditioned planning but remain constrained by limited task-relevant data and coordinate-to-pixel action mismatch.Coordinate-space actions are misaligned with pixel-space videos, weakening conditioning of video generation models.
  • Motivation: Pixel-aligned masks align action conditioning with video pixels, support viewpoint robustness, and unify conditioning across embodiments.The masks are rendered from robot structure and camera parameters, making supervision image-plane based and robot-action-space agnostic.
  • Method: BridgeV2W injects embodiment masks into pretrained video generation models through a ControlNet-style mechanism while preserving pretrained visual and motion priors.This design bridges coordinate-space actions and pixel-space video generation without requiring separate architectures for different embodiments.
  • Method: A flow-based motion loss focuses learning on the embodiment and manipulated objects rather than static background regions.The objective addresses the limitation of uniform frame-level reconstruction losses for robotic manipulation.
  • Results: BridgeV2W improves generation quality over prior state-of-the-art methods on single-arm and dual-arm datasets spanning unseen viewpoints and scenes.The framework is evaluated on DROID and AgiBot-G1 and also shows strong correlation with real-world success for policy evaluation and promising goal-conditioned planning performance.

2. Related Works

Prior embodied world models either train from scratch on robot data or adapt pretrained video models with action conditioning. These approaches benefit from domain data or broad visual priors but retain limitations that motivate BridgeV2W’s mask-based conditioning.

  • Embodied world-model paradigms: One paradigm trains video generation models from scratch on domain-specific robot data to learn environment dynamics from task demonstrations.Such models can capture domain physics and visual patterns but may generalize poorly beyond their training domain.
  • Embodied world-model paradigms: A second paradigm adapts large-scale Internet-video or pretrained video-generation models with action conditioning to obtain richer visual and motion priors.The scale and diversity of pretraining data can improve adaptability to varied visual scenes.
  • ControlNet conditioning: ControlNet adds condition-encoding branches to diffusion models, initially preserving pretrained behavior while learning guidance from signals such as sketches, depth maps, or poses.BridgeV2W adapts this strategy to inject embodiment masks into a pretrained video generation model.
  • ControlNet conditioning: BridgeV2W combines ControlNet-style embodiment-mask conditioning with flow-based loss that emphasizes embodiments and manipulated objects.The flow-based component focuses training on task-relevant regions rather than treating all image regions uniformly.

3. BridgeV2W Framework

BridgeV2W defines an action-conditioned embodied world model that maps an initial image and action sequence to an RGB video, using view-specific embodiment masks as pixel-space control. Its training combines pretrained video generation with diffusion, temporal dynamics, and flow-based motion objectives.

  • Definition: An embodied world model takes an initial RGB frame and action sequence as inputs and produces an RGB video reflecting the actions’ physical consequences for a specific embodiment.The definition permits multiple manipulators and Cartesian- or joint-space actions without prescribing a particular architecture or training objective.
  • Embodiment mask extraction: URDF-based forward dynamics and camera intrinsics and extrinsics convert each action into a viewpoint-specific, pixel-aligned embodiment mask.The resulting mask sequence is paired frame-by-frame with the initial image, actions, and target video.
  • Embodiment mask extraction: The framework can also extract embodiment masks from raw videos with segmentation tools when explicit action annotations or camera calibration are unavailable.This supports datasets containing an initial image, mask sequence, and video without requiring precise geometric calibration.
  • ControlNet-based video generation: BridgeV2W encodes the initial image, video, and mask sequence with a 3D VAE and injects mask features into corresponding DiT blocks through a ControlNet branch.Zero-initialized convolutions and additive fusion guide generation toward the action-to-mask correspondence while retaining the pretrained backbone pathway.
  • Training objectives: Training combines frame-wise diffusion, latent dynamics-consistency, and optical-flow supervision to model temporal motion and emphasize task-relevant dynamic regions.The dynamics objective compares latent changes across temporal offsets, while the flow objective compares predicted and ground-truth motion fields using frozen RAFT features.

4. Experiments

BridgeV2W is evaluated across single-arm and dual-arm datasets, distribution shifts, ablations, and downstream robotic applications. Results show improved video quality, cross-view and cross-embodiment robustness, flexible mask supervision, and usefulness for policy evaluation and planning.

  • Experimental Setup: Experiments evaluate five questions spanning mask compatibility, pretrained-prior transfer, viewpoint robustness, cross-embodiment scaling, and downstream robotic applications.The study uses DROID and AgiBot-G1 datasets, unseen viewpoints and scenes, and real-world tasks.
  • Video Generation Evaluation: BridgeV2W consistently improves temporal realism, perceptual quality, and action–video alignment across in-domain, unseen-viewpoint, and unseen-scene DROID settings.It achieves lower FVD/LPIPS and higher Mask-IoU while keeping PSNR/SSIM competitive.
  • Video Generation Evaluation: Pretrained visual and motion priors combined with pixel-space mask actions improve generalization to novel scenes and camera viewpoints without trading off per-frame fidelity.The pixel-aligned control signal matches the space used by pretrained video models and accommodates diverse viewpoints.
  • Video Generation Evaluation: On AgiBot-G1, BridgeV2W achieves the best overall performance, scaling from single-arm to dual-arm embodiments without redesigning action encoders.It maintains temporal realism and action–video consistency despite increased kinematic complexity.
  • Video Generation Evaluation: Ablations show that removing pretraining or replacing pixel-aligned masks with coordinate-space poses worsens quality, stability, and Mask-IoU, especially under distribution shift.The results support pretrained priors and pixel-aligned mask actions as complementary design choices.
  • Video Generation Evaluation: Segmentation-derived masks provide meaningful supervision, while mixing uncalibrated human videos with limited calibrated robot data nearly matches full calc-mask training.These findings indicate that human videos supply motion priors and limited calibrated data can align the model to a target embodiment.
  • Downstream Applications: BridgeV2W-based policy evaluation correlates strongly with real-world success, with Pearson r = 0.84, and supports goal-image-conditioned manipulation planning.The policy-evaluation analysis uses autoregressive re-querying, while planning performance is strongest for moderate-rotation tasks.

5. Conclusion

BridgeV2W represents actions as pixel-aligned embodiment masks and injects them into pretrained video models through a ControlNet-style pathway with motion-centric flow loss. Across robotic platforms and real-world settings, it demonstrates strong action-conditioned video generation and downstream manipulation-task performance.

  • Conclusion: BridgeV2W combines pixel-aligned embodiment masks, ControlNet-style conditioning, pretrained visual and motion priors, and a motion-centric flow loss.The design targets viewpoint robustness and a unified world model across robotic embodiments.
  • Conclusion: Evaluations on two robotic platforms and real-world settings demonstrate strong action-conditioned video generation and effectiveness for policy evaluation and goal-image-conditioned planning.The conclusion covers both model quality and downstream manipulation tasks.

A.1. Datasets

The study uses DROID single-arm Franka data and AgiBot-G1 dual-arm data, with actions represented in synchronized pixel-aligned mask space. URDF-based kinematics and calibrated camera parameters render each action into an image-plane embodiment mask.

  • Datasets: Training uses 19k DROID trajectories and 15k AgiBot-G1 trajectories, with 200 trajectories from each dataset reserved for evaluation.DROID trajectories include two calibrated third-person cameras, while AgiBot-G1 uses all available calibrated external views per scene.
  • Action Representation: DROID actions are 7-D end-effector commands containing Cartesian position, roll-pitch-yaw orientation, and normalized gripper control.AgiBot-G1 concatenates left- and right-arm commands for synchronized bimanual control.
  • Mask Rendering: Each action is converted to joint space by inverse kinematics, propagated through URDF forward kinematics, and projected with calibrated camera parameters to render a pixel-aligned embodiment mask.The process projects robot meshes onto the image plane using camera intrinsics and extrinsics.

A.2. Implementation Details

BridgeV2W uses CogVideoX-5B-I2V as its pretrained video-generation backbone and injects embodiment masks through ControlNet-style conditioning. The section also situates this implementation against IRASim and Cosmos, including Cosmos’s limitation under unseen viewpoints and diverse embodiments.

  • Implementation: CogVideoX-5B-I2V serves as the pretrained video-generation backbone, with a T5 text encoder and a neutral placeholder prompt keeping the text encoder active.The model does not use dataset language instructions because conditioning is provided through action masks.
  • Implementation: Embodiment masks are injected into CogVideoX through a ControlNet-style conditioning branch.The model is fine-tuned on 2 × 8 H20 GPUs for approximately two weeks.
  • Related baselines: IRASim predicts future video frames from historical observations and action sequences using latent-space diffusion with a VAE and DiT backbone.Its design targets efficient long, high-resolution rollouts.
  • Related baselines: Cosmos is a two-stage world foundation model that pretrains on large-scale videos and post-trains on prompt–video pairs for target environments.Its end-effector pose representation constrains transfer to unseen camera viewpoints and diverse embodiments.
  • Related baselines: EVAC uses projected action maps, delta-action encodings, camera-aware attention, and ray-direction embeddings for view-consistent synthesis.Because EVAC trains only on in-domain data without pretrained visual or motion priors, its performance degrades in unseen scenes.

A.4. Ablation Settings

The ablation study isolates four BridgeV2W design choices: pretrained visual and motion priors, pixel-aligned mask actions, ControlNet-style conditioning, and flow-based motion loss.

  • Ablation design: The ablation study probes four BridgeV2W design choices while keeping all other factors fixed.The tested components are pretrained weights, mask actions, ControlNet conditioning, and flow loss.
  • Pretrained model: Removing pretrained weights trains the CogVideoX transformer from scratch to isolate the contribution of visual and motion priors.The T5 text encoder and VAE remain pretrained in this ablation.
  • Mask action: Removing mask actions replaces pixel-aligned masks with MLP-encoded end-effector pose actions to test viewpoint robustness and cross-embodiment conditioning.The pose representation includes xyz, rpy, and gripper values.
  • ControlNet: Removing ControlNet replaces the zero-initialized residual pathway with a separate non-zero-initialized conditioning branch.This tests whether the residual design helps preserve pretrained priors during fine-tuning.
  • Flow loss: Removing flow loss tests whether emphasizing task-relevant motion regions improves contact timing, motion sharpness, and background stability.The remaining objectives are used for training.

A.5. Training with Segmentation-Derived Masks and Reduced Calibration Requirements

BridgeV2W can train with segmentation-derived masks from uncalibrated videos, while requiring approximate camera parameters and a URDF only at inference. Additional experiments show that large-scale segmentation data plus limited calibrated robot data can approach full calibrated-mask training.

  • Calibration requirements: BridgeV2W uses camera parameters and URDF models only at inference, allowing segmentation-derived masks to provide action supervision during training.This enables training on large uncalibrated datasets without robot models or camera parameters.
  • Segmentation-derived training: Large-scale Ego4D segmentation masks combined with a small fraction of AgiBot-G1 calc-mask data nearly match full calc-mask training performance.Segmentation-only robot training still provides meaningful supervision despite the mask mismatch.
  • Deployment boundary: At inference, BridgeV2W requires approximate camera parameters and a URDF model to render embodiment masks.The experiments use estimated geometry in some cases, yet the framework maintains strong performance across benchmarks.
  • Evaluation: BridgeV2W’s action–video consistency is evaluated with Mask-IoU against ground-truth masks containing object interactions and occlusions.The metric does not compare predictions with the conditioning inputs, and ablations with the same masks obtain lower Mask-IoU.
  • Baseline comparison: Baseline world models supplied with ground-truth camera intrinsics and extrinsics do not improve and degrade unseen-camera results because of distribution shift.The comparison appears in Table 9.
  • Real-world evaluation: The real-world platform uses a statically mounted Franka Research 3 arm and a fixed third-person Zed2i RGB-D camera.The task setup evaluates four manipulation tasks using teleoperated trajectories.
  • VLA baselines: The real-world evaluation considers OpenVLA-OFT, π0, and SpatialVLA as representative VLA policy baselines.Their descriptions cover action chunking, flow-matching control, and spatially aware action representations.

B.2. Policy Evaluation

BridgeV2W evaluates VLA policies by autoregressively rolling out generated video and re-querying each policy on predicted frames. Its evaluation correlates strongly with real-world success, but it can overestimate success for small action perturbations.

  • Evaluation procedure: Each VLA policy outputs an action chunk that is converted into pixel-aligned masks before BridgeV2W predicts the corresponding video clip.The rollout starts from an initial observation and may include an instruction.
  • Evaluation procedure: The last generated frame becomes the next policy input, enabling autoregressive closed-loop rollout until the evaluation horizon is reached.The resulting synthetic trajectory approximates scene evolution under each policy.
  • Implementation: Task success in generated videos is assessed by human raters, while automated multimodal evaluation is identified as future work.VLA action chunks shorter than 25 are padded by repeating the final action.
  • Metrics: BridgeV2W uses Mean Maximum Rank Violation and Pearson correlation to assess agreement between proxy and real-world policy success rates.MMRV measures ranking inconsistency, while Pearson r measures linear association.
  • Failure cases: BridgeV2W tends to overestimate success when small action deviations still produce apparently successful generated grasps.The authors attribute this behavior to training predominantly on expert demonstrations and suggest adding imperfect or failure trajectories.

B.3. Goal-Image Conditioned Manipulation.

BridgeV2W uses CEM within an MPC loop to plan action sequences toward goal images, achieving strong pick-and-place performance but facing challenges in rotation-heavy tasks. Search refinements improve these tasks without modifying the world model, while VLA integration enables more search-efficient closed-loop planning at the cost of increased inference latency.

  • Planning Algorithm: CEM–MPC samples 7-DoF action sequences, rolls them out with BridgeV2W, scores terminal predictions in latent space, and refits elite distributions.Actions include translation, rotation, and gripper state; planning proceeds in a receding-horizon loop.
  • Planning Results: BridgeV2W performs strongly on pick-and-place tasks but lags on tasks requiring substantial rotations because rotational subspaces are difficult to search.The reported planning results identify rotation-heavy manipulation as the main weakness.
  • Rotation-Heavy Manipulation: BridgeV2W remains stable under large wrist rotations, with LPIPS, FVD, and Mask-IoU not degrading on rotation-heavy tasks.The analysis attributes dominant errors to action search rather than collapse of the world model under self-occlusion or viewpoint changes.
  • Rotation-Heavy Manipulation: Naive CEM under-explores rotational dimensions, causing failures where the robot reaches the correct position but does not complete the required rotation.The evaluated remedies include rotation-first search, reallocating samples to rotation, and axis-wise decomposition.
  • Rotation-Heavy Manipulation: Success rates increase from 3/10 to 5/10 for close drawer and from 0/10 to 3/10 for flip cup after search refinements.These gains are achieved without modifying BridgeV2W, indicating that the model already encodes sufficient geometry and dynamics for rotation reasoning.
  • Integration with VLA Frameworks: OpenVLA-OFT plus BridgeV2W consistently improves over OpenVLA alone across multiple manipulation tasks, but evaluating multiple rollouts increases inference latency.The VLA proposes structured action hypotheses, making this closed-loop setup more search-efficient than blind offline CEM sampling.

C. Additional Visualization Results

Additional visualizations examine BridgeV2W across camera viewpoints, unseen scenes, and robot embodiments. The results include DROID rollouts for viewpoint and scene generalization and AgiBot G1 examples for cross-embodiment applicability.

  • DROID Viewpoint Results: Figure 10 compares in-domain DROID rollouts with the same action sequences rendered from unseen camera viewpoints.The visualization assesses viewpoint robustness.
  • DROID Scene Results: Figure 11 illustrates DROID scenes with unseen backgrounds to assess generalization beyond the training distribution.The figure focuses on unseen-scene visualizations.
  • AgiBot G1 Results: Figure 12 presents additional video-generation examples on the AgiBot G1 dataset to demonstrate cross-embodiment applicability.The visualization extends the examples beyond the single-arm DROID setting.
  • Goal-Conditioned Planning: Figure 9 shows goal-conditioned planning rollouts with an initial observation, two goal images, and model-predicted execution frames.The figure provides qualitative examples of goal-image-conditioned planning.
Loading 2602.03793v1…