PaperlayerView exact revision on arXiv

Source-linked AI summary

OpenWAM: An Open Framework for Composable World-Action Models

Heng Yu, David D. Yuan, Juze Zhang, Changan Chen, Yao Feng, Michelle Baldonado, Steve Cousins, Li Fei-Fei, Jiajun Wu, Ehsan Adeli

arXiv:2610.07922v1cs.ROcs.CV

TL;DR

OPENWAM addresses the difficulty of comparing world-action designs when systems vary in their backbones, supervision, and inference. It combines a common causal video foundation with configurable video–action interaction and independently trained dynamics models. The reported results show strong control performance and benefits from counterfactual supervision for dynamics-component reuse and prediction.

  • Problem

    Existing WAMs vary in backbones, training data, objectives, and inference procedures, making the effects of video–action interaction difficult to isolate.

    Source: §1 · p. 1

  • Method

    OPENWAM combines a causal video backbone with configurable video–action interaction and independently trained local-context inverse and forward dynamics models.

    Source: §1 · p. 1

  • Results

    OPENWAM shows strong performance across four LIBERO suites and real-world bimanual tasks, while counterfactual supervision improves reuse of local-context inverse models and forward prediction.

    Source: §5 · pp. 8–9

  • Takeaways & Limitations

    OPENWAM provides a common framework for comparing video–action configurations and studying dynamics learning beyond successful demonstrations.

    Source: §1 · p. 2, §5 · p. 8

    Composability is established primarily in simulation; target-task experiments adapt the video predictor and do not establish zero-shot transfer of the complete video–action model.

    Source: §5 · p. 9

Abstract

from arXiv · show

World-action models (WAMs) couple future prediction with robot control, yet existing systems often vary the video backbone, interaction structure, supervision, and inference procedure simultaneously, making their design choices difficult to compare. We introduce OPENWAM, an open world-action modeling framework built around a common causal robot-video foundation and configurable video-action interaction. Starting from Wan2.2-5B, we perform causal robot-video pretraining on over 10,000 hours of video, then integrate an action expert through a shared Mixture-of-Transformers architecture that supports joint, video-then-action, action-then-video, and decoupled generation. OPENWAM achieves high success rates on four LIBERO suites and real-world bimanual tasks; robot-video training with causal adaptation improves VTA success on LIBERO-Long from 68.4% to 97.8%. The same configurable architecture naturally extends to inverse and forward dynamics, allowing us to study how counterfactual transitions improve independently trained dynamics models beyond demonstrations alone. When only the video predictor is adapted to a new task, a frozen local-context inverse dynamics model trained on counterfactual data and demonstrations achieves 84.0% mean success across four held-out LIBERO-90 tasks, compared with 47.0% for a full-context inverse model and 21.5% for a local-context model trained only on demonstrations. For forward dynamics, counterfactual supervision reduces RGB prediction error by 34.5% and raises outcome identification from 21.1% to 71.3% among 16 same-state outcomes. OPENWAM provides a common testbed for comparing WAM interaction designs and for studying dynamics learning from video data beyond successful demonstrations.

1. Introduction

OPENWAM provides a shared framework for comparing video–action interaction designs while studying reuse of independently trained dynamics models. It combines causal robot-video pretraining, configurable video–action generation, and local-context dynamics learning.

1. Introduction

  • Existing WAMs vary in backbones, training data, objectives, and inference, while limited demonstration coverage leaves the value of counterfactual transitions for dynamics-model reuse unclear.The paper frames controlled comparison and transfer after visual task adaptation as two open challenges.Source: §1 · p. 1
  • OPENWAM uses a common causal video foundation and configurable video–action models to compare interaction patterns under shared training and evaluation protocols.Its Mixture-of-Transformers supports joint, sequential, and restricted interaction.Source: §1 · p. 1, Figure 1 · p. 2
  • OPENWAM independently trains local-context inverse and forward models, using alternative action sequences from the same initial state to broaden dynamics supervision beyond demonstrations.The inverse model maps observations and future visual trajectories to actions; the forward model predicts trajectories from actions.Source: §1 · p. 2
  • The evaluation measures robot-video pretraining, closed-loop success across interaction modes, and dynamics-component reuse, including comparisons of supervision coverage and conditioning context.Forward prediction is also tested under alternative actions from the same initial state.Source: §1 · p. 2

2. Related Work

Related work spans predictive control, video–action architectures, and dynamics learning beyond successful demonstrations. OPENWAM positions itself as a controlled comparison of interaction designs and a study of independently trained dynamics-component reuse.

2.1. Visuomotor Policies and Predictive Control

  • Predictive visual models support action generation, simulation, planning, and evaluation, but action-conditioned prediction and video–action policy generation are often separate, leaving component reuse across tasks open.Earlier visual-foresight work combined action-conditioned video prediction with model-predictive control [16] [17]; later work uses latent or visual dynamics for policy learning, planning, and imagined rollouts [11] [24] [25] [80] [82] [107].Source: §2.1 · p. 3

2.2. Video Foundations and World–Action Architectures

  • Prior WAMs vary in coupling and predictive interfaces; OPENWAM fixes a causal robot-video foundation while comparing generation order and future cross-modal visibility under shared evaluation.It also studies reuse of independently trained inverse- and forward-dynamics components through trajectory interfaces.Source: §2.2 · p. 3

2.3. Dynamics Learning Beyond Successful Demonstrations

  • Successful demonstrations cover only action–outcome relationships selected by the demonstrating policy, while OPENWAM branches alternative action continuations from task-relevant simulator states for local dynamics supervision.Prior approaches broaden supervision through learner-visited states, trajectory perturbations, corrective interventions, or non-successful interactions [27] [32] [40] [57] [65] [2] [4] [15] [42] [60] [88] [91] [98].Source: §2.3 · p. 3

3. Methodology

OPENWAM combines a causal robot-video backbone, configurable video–action interaction, and independently trained local-context dynamics components. Its methodology defines common generation routes and trajectory interfaces, then varies interaction order and transition supervision within that framework.

3. Methodology

  • OPENWAM combines a causal video backbone, configurable video–action architecture, and separately trained inverse and forward dynamics components through local trajectory interfaces.The design varies interaction between observation and action streams while enabling composition of independently trained predictors.Source: §3 · pp. 3–4

3.1. World–Action Modeling Formulation

  • Joint generation, video-then-action, and action-then-video specify concurrent or sequential routes for predicting future observations and actions.These factorizations define modeling interfaces and do not require all factors to share one trained parameterization or final checkpoint.Source: §3.1 · p. 4

3.2. An Open Causal Robot-Video Backbone

  • Chunk-causal pretraining lets each future video chunk attend to current and preceding observations and generated chunks, but not later chunks.The backbone starts from a general-purpose video model and is pretrained on robot videos without action supervision, providing a shared video-prediction initialization.Source: §3.2 · p. 4

3.3. Configurable Video–Action Interaction Programs

  • Four interaction programs vary generation order and future cross-modal attention while sharing the same backbone, tokenization, and objective family.Joint generates video and actions concurrently; VTA generates video before conditioned actions; ATV reverses that order; Decoupled removes future cross-modal attention.Source: §3.3 · pp. 4–5, Figure 2 · p. 5, Figure 3 · p. 5

3.4. Counterfactual Learning of Local-Context Dynamics Interfaces

  • Local-context inverse and forward models respectively map a supplied visual trajectory to actions and supplied actions to their visual consequences.The models restrict conditioning to local inputs, rather than language or observations and actions preceding the current time.Source: §3.4 · pp. 5–6
  • Alternative action continuations from a restored simulator state broaden transition supervision beyond demonstrated continuations and need not complete the task.The simulator state controls collection but is not provided to the model; demonstration-only, counterfactual-only, and mixed supervision are supported without changing the conditioning interface.Source: §3.4 · p. 6
  • Independently trained components communicate through generated future-observation or action trajectories, making them explicitly replaceable without requiring one shared learned joint distribution.For video-to-action composition, the visual predictor transfers future video latents to the receiving inverse dynamics model.Source: §3.4 · pp. 6–7

4. Experiments

Experiments evaluate OPENWAM’s interaction programs, native policies, component reuse after visual adaptation, and action-conditioned forward dynamics. They compare policy success across LIBERO and real-world tasks and test how transition supervision and conditioning scope affect dynamics performance.

4.1. Experimental Setup

  • The evaluation covers four interaction programs across four LIBERO suites, three real-world bimanual tasks, and four held-out LIBERO-90 transfer targets.Closed-loop success is the primary policy and composition metric; forward dynamics is evaluated by visual prediction error and outcome identification.Source: §4 · p. 7, §4.1 · p. 7

4.2. Native Policy Performance

  • OPENWAM’s principal configurations achieve strong performance across all four LIBERO suites, supporting multiple world–action generation orders.Table 3 reports average success over three random seeds.Source: §4.2 · p. 7, Table 3 · p. 8
  • 97.8% VTA success on LIBERO-Long with the robot-video-pretrained causal backbone, versus 68.4% with Wan2.2-5B initialization (+29.4 points).The comparison measures robot-video pretraining and causal adaptation together, not either factor individually.Source: §4.2 · pp. 7–8

4.3. Component Composability

  • 84.0% mean target success for mixed-supervision local-context IDMs, versus 47.0% for full-context IDMs and 21.5% for demonstration-only local-context IDMs.Under matched mixed supervision, local conditioning improves target success by 37.0 percentage points while retaining 94.4% source-task success; counterfactual-only local-context IDMs reach 84.5%.Source: §4.3 · p. 8

4.4. Action-Conditioned Forward Dynamics

  • Counterfactual-only supervision reduces RGB MSE by 34.5% and raises outcome accuracy from 21.1% to 71.3% for K = 16.For K = 2, CF-only supervision raises outcome accuracy from 68.3% to 93.6%; mixed supervision yields similar improvements.Source: §4.4 · p. 8

5. Discussion and Conclusion

OPENWAM combines capable policies and configurable video–action generation with independently reusable dynamics models. The conclusion reports strong control results while identifying limits in real-robot composability, transfer scope, and forward-model horizon.

5. Discussion and Conclusion

  • Strong performance across four LIBERO suites and real-world bimanual tasks supports multiple effective modeling configurations, while an initialization ablation establishes the benefit of robot-video pretraining.The shared causal backbone supports joint, sequential, and decoupled generation within a common training and evaluation stack.Source: §5 · p. 8
  • Counterfactual-only and mixed supervision improve reuse of local-context inverse dynamics models after visual task adaptation while retaining strong source-task performance.The framework also supports independently trained inverse and forward dynamics components.Source: §5 · p. 8
  • Counterfactual forward-dynamics supervision improves visual prediction accuracy and identification of action-dependent outcomes.Table 7 evaluates local-context forward prediction on counterfactual transitions; its caption defines outcome accuracy as RGB-MSE matching among same-state outcomes.Source: §5 · p. 9, Table 7 · p. 9
  • Composability is established primarily in simulation; real-robot reuse needs more data-efficient physical dynamics supervision, and target-task adaptation does not establish zero-shot transfer.The forward-dynamics study is local and short-horizon; long-horizon planning and physical deployment remain extensions.Source: §5 · p. 9

A.1. Model Architecture and Representation

  • The 30-layer Mixture-of-Transformers uses modality-specific video and action experts, with future cross-modal attention controlling information exchange.Each observation combines agent-view and wrist-camera images into an 8 × 16 × 48 latent canvas; policies predict 16-step action chunks with seven dimensions per action and an eight-dimensional proprioceptive state.Source: §A.1 · p. 14

A.2. Downstream Training

Downstream variants share a robot-video-pretrained backbone, training data, and objective family, while varying generation order and future cross-modal attention. The appendix specifies their initialization, training objective, and four interaction programs.

A.2. Downstream Training

  • All four programs start from the same robot-video-pretrained causal backbone, and differ only in generation order and future cross-modal attention under shared data, optimization settings, and flow-matching objective.The action expert is initialized from width-adapted video-expert layers, with action-specific input/output and conditioning layers initialized separately, then jointly optimized with the video expert.Source: §A.2 · p. 14
  • Downstream models use AdamW with a 10^-5 learning rate, effective batch size 10, gradient clipping at 2.0, bf16 precision, and 10,000 optimization steps.The schedule uses 10 warmup steps followed by a constant learning rate.Source: §A.2 · p. 14
  • Both streams use rectified-flow training with modality-specific noise shifts and timestep-dependent weighting that emphasizes intermediate noise levels and excludes the pure-noise endpoint.The video and action shift values are 5.0 and 1.0, respectively.Source: §A.2 · p. 14
  • Video and action losses receive equal weight, with video losses normalized over supervised latent elements and action losses averaged over valid action entries.Invalid action dimensions are masked.Source: §A.2 · p. 14

A.3. Video–Action Interaction Programs

  • Joint generation denoises both streams together with bidirectional interaction; VTA predicts video then conditioned actions, ATV predicts actions then conditioned video, and Decoupled removes future cross-modal interaction.VTA and ATV train the second stage on the first modality’s ground-truth trajectory but use its generated trajectory at inference; the sequential programs use two generation stages.Source: §A.3 · pp. 14–15

A.4. Inference and Closed-Loop Evaluation

The appendix describes closed-loop inference and evaluation, real-robot data collection, and causal robot-video pretraining. Pretraining adapts a video backbone using a large heterogeneous corpus without action labels.

A.4. Inference and Closed-Loop Evaluation

  • Inference uses 20 denoising steps per modality, video guidance scale 5.0, and a 15-chunk history; policies execute each 16-step action chunk before replanning.Evaluation uses three independently trained seeds, 500 closed-loop episodes per seed, and shared environment seeds across interaction programs.Source: §A.4 · p. 15

A.5. Real Robot Setup

  • Real-robot demonstrations use two Franka FR3 arms with bimanual GELLO teleoperation; the Toast Bread and Rubik’s Cube tasks each have 200 demonstrations, and Sort Cups has 180.Policies are evaluated over 50 episodes for Toast Bread and Rubik’s Cube, and 36 for Sort Cups.Source: §A.5 · p. 15

B. Robot-Video Pretraining

  • Robot-video pretraining adapts Wan2.2-5B to robot-centric motion and physical interaction using robot demonstrations, human manipulation video, and synthetic robot trajectories without action labels.The corpus spans different embodiments, viewpoints, objects, and interaction patterns.Source: B. Robot-Video Pretraining · p. 15

B.1. Pretraining Corpus

  • The pretraining corpus contains approximately 3.34 million trajectory or take units and 14.64k nominal hours across heterogeneous robot, human-interaction, and synthetic data.Table 8 summarizes the corpus sources.Source: §B.1 · pp. 15–16, Table 8 · p. 16

B.2. Causal Adaptation

  • Causal adaptation starts from Wan2.2-5B and trains a chunk-causal video predictor that cannot attend to later chunks; the resulting checkpoint initializes downstream OPENWAM programs.This stage supplies video alone, without actions, proprioception, or task-specific policy targets.Source: §B.2 · p. 16

B.3. Training Compute

Causal robot-video pretraining used 32 NVIDIA B200 GPUs over 14 days. Downstream OPENWAM variants in the main experiments start from the resulting checkpoint unless otherwise specified.

B.3. Training Compute

  • Causal robot-video pretraining used 32 NVIDIA B200 GPUs.The compute description specifies the hardware count.Source: §B.3 · p. 16
  • Causal robot-video pretraining ran for 14 days.The duration applies to the causal robot-video pretraining stage.Source: §B.3 · p. 16
  • Main-experiment downstream OPENWAM variants start from the resulting pretraining checkpoint unless otherwise specified.This initialization qualification applies to the downstream variants in the main experiments.Source: §B.3 · p. 16

C. LIBERO-Long-CF: Counterfactual Dynamics Data

LIBERO-Long-CF broadens dynamics supervision beyond the single successful continuation supplied by demonstrations. It adds counterfactual action branches and substantially increases local transition coverage.

C. LIBERO-Long-CF: Counterfactual Dynamics Data

  • Successful demonstrations provide essentially one action continuation per visited state, whereas alternative action sequences supply supervision for different local dynamics outcomes.Counterfactual rollouts need not complete the task or carry a success label.Source: C. LIBERO-Long-CF: Counterfactual… · p. 16
  • LIBERO-Long-CF branches from task-relevant states in ten LIBERO-Long tasks, producing 32,000 counterfactual segments balanced across tasks.Each segment contains 128 future controls and 129 synchronized two-view observations; the original tasks, assets, and simulator physics are preserved.Source: C. LIBERO-Long-CF: Counterfactual… · p. 16

C.1. Scale and Intervention Coverage

  • 4.10 million counterfactual control records versus 138,090 controls in 500 source demonstrations provide approximately 29.7× more local transition supervision.At 20 Hz, these correspond to 56.9 and 1.92 control-equivalent hours, respectively.Source: §C.1 · p. 16, Table 9 · p. 16
  • Branch interventions vary motion magnitude, direction, timing, action axes, arm motion, and gripper behavior, including reversed or redirected motion and structured control noise.All commands stay within LIBERO’s normalized controller range.Source: §C.1 · p. 16

C.2. State and Interaction Coverage

LIBERO-Long-CF extends the starting-state and interaction coverage of demonstrations, and its local-dynamics representation pairs visual trajectories with actions. Robot-video pretraining also improves performance across VTA and Joint interaction programs.

C.2. State and Interaction Coverage

  • 75% of counterfactual segments start from demonstration-prefix states, while 25% start after an executed perturbation and extend the starting-state distribution beyond demonstrated trajectories.Among characterized perturbed starts, the end-effector is on average 3.34 cm from the nearest demonstrated-path point (median 1.90 cm), and 61.6% also contain object configurations outside the corresponding demonstrations under the stated thresholds.Source: §C.2 · pp. 16–17
  • 90.4% of characterized segments contain gripper contact, 50.0% contain a detected grasp, and 72.3% alter object configuration relative to the same-start reference continuation.The physical-interaction categories overlap.Source: §C.2 · p. 17, Table 11 · p. 17

C.3. Representation for Local Dynamics Learning

  • Local-context IDM predicts actions from the boundary observation, proprioception, and future visual trajectory, while local-context FDM predicts that trajectory from the observation, proprioception, and actions.Both objectives exclude task language and pre-start history; the representation encodes one boundary observation and 32 future latent frames, with four controls aligned to each future frame.Source: §C.3 · p. 17

D.1. Effect of Video-Backbone Initialization

  • Robot-video pretraining improves success over Wan2.2 initialization by 29.4 percentage points for VTA and 34.4 points for Joint, reaching 97.8% and 96.6%, respectively.Random initialization performs substantially worse for both programs.Source: §D.1 · p. 17, Table 12 · p. 17

D.2. Effect of the Mixture-of-Transformers Architecture

With the pretrained visual foundation fixed, Mixture-of-Transformers improves closed-loop success over a shared-DiT variant. Independently trained dynamics specialists also outperform UVA on the reported counterfactual dynamics metrics, while supervision mixture affects demonstration fidelity.

D.2. Effect of the Mixture-of-Transformers Architecture

  • MoT raises success from 92.8% to 97.8% for VTA and from 93.6% to 96.6% for Joint, gains of 5.0 and 3.0 percentage points.Both variants use the same robot-video-pretrained visual checkpoint, downstream data, and training procedure.Source: §D.2 · p. 17

E.1. Independent Dynamics Components

  • Counterfactual-only LC-IDM and LC-FDM specialists are strongest on held-out counterfactual data, while a 60/40 counterfactual-and-demonstration mixture substantially improves fidelity on real demonstrations.Both specialist settings remain well ahead of UVA on the counterfactual dynamics metrics.Source: §E.1 · p. 18, E. External Dynamics Comparison · p. 18

E.2. Unified Policy and Dynamics Modeling

A unified OPENWAM checkpoint jointly trains policy, inverse dynamics, and forward dynamics. It outperforms UVA on the reported policy and dynamics metrics, with tradeoffs relative to specialized models and limits on interpreting demonstration-fit errors.

E.2. Unified Policy and Dynamics Modeling

  • The unified OPENWAM checkpoint outperforms UVA on policy success and the evaluated IDM and FDM metrics, but its counterfactual dynamics accuracy is weaker than dedicated specialists.It gives the lowest IDM and FDM errors on real demonstrations, which measure demonstration-distribution fit rather than held-out generalization; its policy score is below policy-specialized OPENWAM models.Source: §E.2 · p. 18
Loading 2610.07922v1…