WorldBridge: Bridging Video and Action for Task Generalization in World Action Models

1 Introduction

• The paper frames task generalization as predicting visual evolution beyond training tasks and translating those predictions into executable actions.This addresses the gap between understanding an instruction and having the physical competence to execute it.Source: §1 · pp. 1–2
• OOD success rises from 25.07% with ID video-action training to 55.90% with OOD videos, then to 70.70% after one action-labeled demonstration per OOD task.The progression supports complementary roles for broader video coverage and action supervision.Source: §1 · p. 3, Figure 2 · p. 3
• WorldBridge connects video prediction to action generation through separate experts, structured attention, joint flow matching, two-scale visual memory, and action forcing.Its unified pipeline standardizes robot and egocentric demonstrations into a shared action representation and filters trajectories.Source: §1 · p. 3
• 96.23% on RoboTwin 2.0, 80.9% on RoboTwin 2.0 C2R, 82.8% on RoboCasa GR1 TableTop, and 57.6% on DOMINO Level-1; WorldBridge leads compared methods on multiple benchmarks.It also matches the best comparison result on RMBench at 83.0% success.Source: §1 · p. 3
• Evaluations span six manipulation benchmarks and real-world deployments on AgiBot G2 and ARX AC-1, covering long-horizon execution, memory, spatial alignment, and bimanual coordination.The same jointly pretrained checkpoint is fine-tuned for each platform.Source: §1 · p. 4

2.1.1 Architecture

• Separate video and action experts exchange information through joint self-attention, while the video expert also incorporates language features through cross-attention.The experts produce video and action velocity predictions and are optimized with joint flow matching.Source: §2.1.1 · pp. 4–5

2.1.2 Memory

• Short-term memory captures local motion, while long-term memory represents broader task progression alongside the current observation.The long-term sampling stride spans one predicted action chunk, and the resulting latent frames are prepended to future-video tokens.Source: §2.1.2 · pp. 5–6

2.1.3 Action Forcing

• Action forcing varies flow timesteps across action groups during training, mixing noise levels within a chunk and encouraging predictions to rely on visual context.At inference, all action tokens are jointly denoised with a shared action flow timestep.Source: §2.1.3 · p. 6

2.1.4 Attention Mask

• Pretraining keeps video prediction independent of actions; post-training routes future-video features to action predictions, with direct memory access retained for memory-demanding tasks.During pretraining, action tokens can attend to video and action tokens, while video tokens do not attend to the action branch.Source: §2.1.4 · pp. 6–7, Figure 4 · p. 6

2.2.1 Robot Data Processing

• Robot source fields are audited and mapped separately for state and action into a shared 46-dimensional layout, with masks marking unavailable entries.Source-specific mappings also standardize rotations, gripper values, and supported hand and torso channels.Source: §2.2.1 · p. 7
• Robot trajectories are trimmed to active boundaries, episodes shorter than 48 frames are removed, and source-configured kinematic and temporal checks filter inconsistent data.Kinematic checks compare forward-kinematics end-effector poses with recorded poses; other checks include sudden-change, state–action consistency, and extreme-value tests.Source: §2.2.1 · pp. 7–8

2.2.2 Ego Data Processing

• Egocentric demonstrations pair RGB observations and camera poses with hand keypoints, then standardize hand geometry into the shared action representation.Future hand geometry is expressed in the current observation's camera frame, which remains fixed across the prediction chunk.Source: §2.2.2 · pp. 8–9
• Hand-derived targets include canonical wrist poses, gripper openness, and dexterous-hand joint targets; geometric and temporal checks filter unreliable supervision.Rejected-frame targets are masked, and windows with more than 20% rejected action timesteps are discarded.Source: §2.2.2 · p. 9

2.3.1 Robot and Ego Data

• Robot sources span real-world and simulated demonstrations, while egocentric sources provide first-person observations with hand-derived supervision.Validity masks retain available hand targets while excluding unreliable or missing ones.Source: §2.3.1 · pp. 9–10
• After curation, the corpus contains 3,381,561 episodes and 22,321.21 hours across robot and egocentric data.Robot data contributes 1,275,715 episodes (9,729.58 hours), while ego data contributes 2,105,846 episodes (12,591.63 hours); training contributions also depend on the sampling mixture.Source: §2.3.1 · p. 10

2.3.2 Unified Action Space

• Actions use a shared 46-dimensional layout with validity masks, end-effector-relative pose targets, masked dataset-specific normalization, and an absolute current-state token.The state token provides the reference configuration and is excluded from action prediction loss.Source: §2.3.2 · pp. 10–11

2.3.3 Pretraining Setup

• Pretraining jointly optimizes video and action flow-matching losses, samples robot and ego data at 0.7 and 0.3, and enables four observations at each memory scale.Video tokens cannot attend to actions, while action tokens use the full video context; all actions in a pretraining chunk share one flow timestep.Source: §2.3.3 · pp. 11–12

2.4 Efficiency Implementation

• Video-action denoising shares key–value features: after synchronized updates, remaining action steps reuse the final video features, approximating joint denoising.Visual features refresh for each observation, and future-video RGB decoding is omitted at deployment.Source: §2.4 · p. 12
• Compilation, operator fusion, and CUDA Graph replay reduce action-inference overhead by fusing GPU operations and replaying captured execution sequences.The policy also reuses language context, scheduler metadata, and timestep-dependent AdaLN tensors.Source: §2.4 · p. 12
• Asynchronous execution lets the server compute a new action chunk while the robot executes the current one, discarding expired actions and resampling remaining actions onto the 100 Hz servo grid.At most one request remains outstanding to avoid accumulating stale observations.Source: §2.4 · p. 12

3.1 Task Generalization

• With ID-only video-action supervision, OOD success is 25.07%; adding OOD videos without OOD action labels raises it to 55.90%.ID success remains nearly unchanged at 91.96% with added OOD videos.Source: §3.1 · pp. 12–13
• One action-labeled demonstration per OOD task raises success from 55.90% to 70.70%, while 20 demonstrations reach 90.03% OOD success.With 100 action demonstrations per task, increasing OOD videos from 100 to 550 raises success from 88.83% to 93.17%; increasing actions from 100 to 550 with 550 videos fixed yields 92.83% rather than 93.17%.Source: §3.1 · p. 13

3.2 Simulation Benchmarks

• WorldBridge reports 96.23% success on RoboTwin 2.0, 82.75% on RoboCasa GR1 TableTop, 84.4% on SimplerEnv-WidowX, 83.0% on RMBench, and 57.6% on DOMINO Level-1.RMBench performance matches MemoryWAM and exceeds LingBot-VA; DOMINO evaluation reuses the RoboTwin checkpoint without dynamic-task fine-tuning.Source: §3.2 · pp. 14, 16
• Adding action forcing, short-term memory, then long-term memory raises success from 91.28% to 95.20% on RoboTwin, 67.50% to 78.58% on RoboCasa, and 14.50% to 21.07% on RoboDojo.The cumulative ablation supports complementary gains from action forcing and temporal context, with memory additions particularly beneficial on RoboCasa and RoboDojo.Source: §3.2 · p. 17, Table 6 · p. 17
• Robot pretraining raises RoboTwin Clean-to-Random average success from 62.7% to 77.4%, and adding egocentric pretraining further raises it to 80.9%.The combined sources improve average success by 18.2 points over training without pretraining.Source: §3.2 · p. 17, Table 7 · p. 17

3.3 Real-Robot Experiments

• WorldBridge averages 72.5% success across real-robot tasks, versus 57.5% for π0.5, 39.2% for LingBot-VA, 29.2% for FastWAM, and 10.8% for Motus.It leads on five tasks and ties π0.5 at 80% on separating LEGO bricks; sorting screws remains its least successful task.Source: §3.3 · p. 19, Figure 7 · p. 19
• Joint video–action pretraining raises average real-robot success by 25.8 percentage points; sorting screws improves from 10% to 55%.Pretraining improves all six tasks, with packing books and separating LEGO bricks each increasing from 50% to 80%.Source: §3.3 · p. 19
• CUDA Graph replay reduces median model-call latency from 957.2 to 103.9 ms, a 9.21× speedup, and the deployed system sustains nearly 7 Hz.The final P95 latency is 105.6 ms; reported timing excludes JPEG processing, text encoding, communication, and robot execution.Source: §3.3 · p. 20, Table 8 · p. 20

4 Related Work

• VLA models use pretrained vision-language models for instruction-following robot policies, but still struggle with new tasks when action-labeled demonstrations are limited or unavailable.Examples discussed include RT-2, OpenVLA, π0, and π0.5.Source: §4 · p. 20
• World Action Models use video priors for robot control beyond action-labeled data, with prior work exploring unseen-task execution and human-video guidance.The passage cites DreamZero and Zero-WAM as examples.Source: §4 · p. 20
• Prior work also combines video and action learning or extends temporal context through longer observation histories and persistent memory.WorldBridge studies how video coverage and action supervision contribute to task generalization through a video-to-action bridge.Source: §4 · p. 20

• WorldBridge connects visual dynamics to executable control through structured attention, joint flow matching, temporal memory, action forcing, and heterogeneous-data pretraining.Source: §5 · p. 21
• Experiments across simulation benchmarks and two real-robot platforms demonstrate strong manipulation performance and benefits from joint video–action pretraining.Source: §5 · p. 21
• Broader task videos enable transfer beyond action-supervised tasks, while a single action-labeled demonstration per new task substantially improves execution.Source: §5 · p. 21
• Extending the video-to-action bridge to more diverse tasks with less action supervision remains an important direction for future work.Source: §5 · p. 21