PaperlayerView exact revision on arXiv

Source-linked AI summary

FlowWAM: Optical Flow as a Unified Action Representation for World Action Models

Yixiang Chen, Peiyan Li, Yuan Xu, Qisen Ma, Jiabing Yang, Kai Wang, Jianhua Yang, Dong An, He Guan, Gaoteng Liu, Jianlou Si, Jun Huang, Jing Liu, Nianfeng Liu, Yan Huang, Liang Wang

arXiv:2607.13017v1cs.ROcs.CV

TL;DR

WAMs need action representations that fit pretrained video generators while preserving the motion cues needed for control. FlowWAM uses optical flow as a shared video-native action representation for policy prediction and world modeling, and reports gains on RoboTwin and WorldArena.

  • Problem

    WAM action representations must align with pretrained video generators while preserving dense cross-frame motion cues; existing numerical and visual representations leave this gap unresolved.

    Source: §1 · p. 2

  • Method

    FlowWAM jointly models RGB and optical-flow videos with a shared pretrained video generator, generating flow for policy action decoding and using target flow to guide world-model video generation.

    Source: §1 · pp. 2–3, §3.2 · p. 5

  • Results

    FlowWAM reports 92.94% success on RoboTwin Clean and 92.14% on Random, outperforming VLA and WAM baselines, and a state-of-the-art WorldArena EWMScore of 63.71.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The authors conclude that dense cross-frame flow improves control and world modeling while remaining decodable into robot actions and usable with action-unlabeled egocentric videos.

    Source: §5 · p. 10

    Official WorldArena evaluation requires remote submission and has no ground-truth videos for local scoring, so ablations use a custom validation split and differ from Table 2 results.

    Source: §4.4 · p. 8

Abstract

from arXiv · show

World Action Models (WAMs) are able to leverage pretrained video generators for both world modeling and action prediction. However, directly leveraging such video generators for control raises a new challenge: how to represent actions in a suitable form that aligns with pretrained video generators while carrying enough motion cues for accurate control. Existing numerical actions fail to satisfy the former, and prior visual action representations overlook the temporal motion structure across frames. We address this issue with FlowWAM, a dual-stream diffusion framework that adopts optical flow as a unified, video-native action representation. Flow videos share the same format as RGB videos and encode rich per-pixel displacement. By jointly modeling them within a shared pretrained video generator, FlowWAM can naturally implement two modes of WAMs. In policy mode, FlowWAM generates flow for action prediction, while in world-model mode, it uses target flow sequences to guide future video generation. Moreover, since flow can be easily extracted from raw videos without action labels, FlowWAM can leverage large-scale action-unlabeled video datasets for pretraining. We empirically find that our flow-based action representation delivers gains across both modes. On RoboTwin manipulation, FlowWAM raises the success rate to 92.94% on the Clean setting and 92.14% on Random, outperforming both VLA and WAM baselines. On WorldArena world modeling, it achieves the best overall EWMScore (63.71) with an 18.4% relative improvement in trajectory accuracy. More results can be found on our project website: https://flow-wam.github.io .

1 Introduction

FlowWAM addresses how WAM actions can fit pretrained video generators while retaining dense motion cues for control. It uses optical flow as a unified representation for policy inference and world modeling, with reported gains on RoboTwin and WorldArena.

1 Introduction

  • WAMs leverage pretrained video generators’ motion and temporal-dynamics priors, but control requires actions that fit their visual inputs while retaining cross-frame motion cues.This is especially relevant to manipulation, where success depends on continuous motion of the robot and environment.Source: §1 · pp. 1–2
  • Numerical actions vary across robots, latent actions may lack dense spatial cues, and prior visual actions largely provide static spatial rather than evolving temporal signals.The open question is how to represent actions in video-native form while retaining dense motion cues and remaining decodable into executable control.Source: §1 · p. 2
  • Optical flow encodes per-pixel displacement as RGB-formatted video and can be extracted from raw egocentric footage, supporting action-unlabeled pretraining and downstream control.FlowWAM treats flow as a primary action modality integrated into the WAM generative latent space, rather than as an auxiliary signal or modular intermediate.Source: §1 · pp. 2–3
  • FlowWAM jointly models RGB and flow with a shared pretrained video generator, generating future flow for policy action decoding and using target flow to guide world-model video generation.The framework uses lightweight stream-specific adapters and is presented as a unified diffusion approach for action prediction and world modeling.Source: §1 · pp. 2–3
  • 92.94% success on RoboTwin Clean and 92.14% on Random outperform both VLA and WAM baselines; WorldArena EWMScore reaches 63.71 with significant gains in action-conditioned trajectory modeling.The results cover FlowWAM’s policy and world-model settings.Source: §1 · p. 3

2 Related Work

Prior WAMs connect video generation and action prediction through separate action streams, latent codes, or image-space cues. FlowWAM instead models dense optical flow as a visual stream that serves both policy prediction and world-model conditioning.

World Action Models

  • DreamZero [1] and Cosmos Policy [5] use action tokens, UWM [2] and CoVAR [3] share video-action latents, while other methods infer actions after generating future videos.These approaches connect heterogeneous action streams to video latents through tokens, heads, or decoders; FlowWAM models action-relevant motion as a visual stream inside the pretrained video generator.Source: §2 · p. 3

Action Representations in WAMs

  • Unified WAMs use numerical tokens [1] [5] [2], learned latent codes [12] [4] [24], or image-space signals such as ray maps [13], masks [14], and multi-view images [15].FlowWAM replaces sparse spatial cues with dense flow videos used as both the policy target and world-model conditioning signal.Source: §2 · p. 3

Flow for Manipulation

  • Prior manipulation systems use flow as auxiliary supervision [16] [17] [4], an intermediate in control pipelines [18] [25] [26] [19], or structured pixel motion [27] [28]; ATM [29] models sparse tracks.FlowWAM instead makes dense flow a native stream for policy inference and action-sensitive world modeling within one video generator.Source: §2 · p. 3

3 Method

FlowWAM represents motion as optical-flow video and jointly models it with RGB in a shared diffusion framework, supporting both action prediction and motion-conditioned future-video generation.

3.1 Overview

  • FlowWAM jointly generates future RGB and flow latents in policy mode, while an action expert decodes video-generator features into executable robot actions.In world-model mode, desired flow trajectories condition generation of future RGB frames that follow the specified motion.Source: Figure 2 · p. 4

3.3 Action Prediction from Flow Latents

  • The action expert decodes the video generator’s intermediate features into executable actions.The overview identifies this decoding role as the route from generated motion representation to robot control.Source: Figure 2 · p. 4

3.2 Unified Video-Flow Generation

  • FlowWAM encodes per-pixel displacement as an RGB-format flow video, allowing RGB and flow streams to use shared VAE and transformer components.The streams have separate patch embeddings and output heads, while concatenated tokens interact through joint attention in shared transformer blocks.Source: §3.2 · p. 4
  • Policy inference jointly denoises future RGB and flow latents, whereas world-model inference holds desired-flow latents fixed and denoises RGB latents.The first latent frame in each stream is fixed to the clean encoding of the reference observation.Source: §3.2 · pp. 4–5

3.4 Training

  • Motion-aware reweighting of the flow loss counters static-background dominance by assigning per-location weights from flow-latent deviation from the reference frame.The weighting directs learning toward motion-rich regions.Source: §3.4 · p. 5
  • The video-only objective trains the dual-stream generator on RGB and extracted flow, including action-unlabeled video, before a joint labeled-robot-data stage adds action decoding.The video loss combines RGB- and flow-stream flow-matching losses, with the first conditioning frame masked from the loss.Source: §3.4 · pp. 5–6

4 Experiments

Experiments assess FlowWAM’s control, world-modeling, and transfer performance, then probe the design choices and flow decodability behind those results. Across simulated and real-robot tasks, dense flow serves as the shared motion representation.

4.2 Action-Conditioned World Modeling

  • 63.71 EWMScore: FlowWAM leads WorldArena overall and has the strongest Trajectory Accuracy, nearly ten points above the second-best method.It remains close to the best baselines on Subject and Background Consistency and also tops Depth Accuracy.Source: Table 2 · p. 7, §4.2 · p. 8

4.1 Manipulation Policy

  • 92.94% on Clean and 92.14% on Random: FlowWAM leads VLA and WAM baselines on RoboTwin under both settings.The evaluation reports success over 100 rollouts per task; Random includes randomized object poses, distractors, lighting, and backgrounds.Source: §4.1 · p. 7
  • Action-unlabeled egocentric-video pretraining lifts FlowWAM further, with a larger gain on Random than Clean; without it, FlowWAM already surpasses VLA baselines.The passage attributes the larger Random-setting gain to visual robustness accumulated by the video generator.Source: §4.1 · p. 7

4.3 Real-World Manipulation

  • 75.7% average real-world success: FlowWAM leads on every task, with a larger gap over baselines on dual-arm than single-arm tasks.Across the suite, π0.5 averages 61.4% and Motus 57.1%.Source: Figure 3 · p. 8, §4.3 · p. 8

4.4 Analysis

  • 69.8% success after replacing flow with numerical action vectors; raw (u, v) tensors also break the pretrained VAE and DiT input format.Removing motion-aware reweighting lets static background dominate the flow loss, while omitting stochastic latent conditioning leaves the action expert brittle to inference-time residual denoising error.Source: §4.4 · p. 8, Figure 4 · p. 9
  • 49.31 to 65.23 EWMScore: full RGB flow outperforms alternative conditioning inputs when the video generator is held fixed.Text, numerical actions, raw (u, v) tensors, and image-space masks each omit either dense motion, video-native format, or cross-frame motion direction and magnitude; the analysis concludes flow is needed as both an action target and generation condition.Source: Figure 4 · p. 9, §4.4 · p. 9
  • r = −0.81: across 50 RoboTwin tasks, lower predicted-flow error consistently accompanies higher policy success.Flow error is measured against RAFT-extracted pseudo-ground-truth flow from 50 newly generated demonstrations per task.Source: Figure 4 · p. 9, §4.4 · p. 9
  • In qualitative examples, policy-mode flow traces goal-directed robot motion while world-model rollouts follow conditioning flow and preserve object identity and static background.The RGB stream preserves surrounding object context in policy mode.Source: Figure 5 · p. 9, §4.4 · pp. 9–10

5 Conclusion

FlowWAM establishes optical flow as a unified action representation for WAMs, supporting policy decoding, flow-conditioned world modeling, and learning from action-unlabeled egocentric video. Experiments report improvements in both control and world modeling; scaling action-free pretraining and extending planning horizons remain future work.

5 Conclusion

  • FlowWAM establishes optical flow as a unified action representation for World Action Models.Robot motion is encoded as dense flow videos that bridge executable control and pretrained video priors.Source: §5 · p. 10
  • The same representation supports generating flow for policy decoding, conditioning world modeling on flow, and learning from action-unlabeled egocentric videos.The paper attributes its gains in control and world modeling to preserving dense cross-frame motion while remaining decodable into robot actions.Source: §5 · p. 10
  • Scaling action-free pretraining to internet-scale datasets and extending flow-based planning to longer temporal horizons are future work.Source: §5 · p. 10

A Implementation Details and Training Pipeline

FlowWAM integrates motion-specific modules into a pretrained video generator, retaining its core architecture while adding a flow stream and an action expert.

A.1 Architectural Components

  • The base is Wan2.2-TI2V-5B, with its pretrained DiT, VAE, and text encoder architectures preserved; the VAE and text encoder stay frozen during training.The DiT is updated according to stage-specific protocols.Source: §A.1 · p. 14
  • A dedicated HSV-flow pathway uses separate patch-embedding and output layers initialized from RGB layers, while flow and RGB tokens share transformer blocks and latent geometry.A learnable flow-token embedding supplies stream identity; using the same VAE and RGB-initialized weights provides a compatible image-latent prior.Source: §A.1 · p. 14
  • A roughly 780M-parameter action expert attends to RGB-flow features and instruction tokens, with initial joint positions included in its context, and predicts action-chunk flow-matching velocity.The flow-matching sampler integrates the prediction at inference.Source: §A.1 · p. 14

A.2 Two-Stage Training Strategy

FlowWAM separates action-free motion learning from embodied action decoding, using RGB-flow training data and a shared video-generation framework. The section also details the input preparation and comparison baselines.

A.2 Two-Stage Training Strategy

  • Stage 1 learns manipulation motion priors from unlabeled EgoDex RGB and extracted flow; Stage 2 adds an action expert and jointly trains on RoboTwin 2.0.The stages use different supervision, data sources, and trainable parameters.Source: §A.2 · p. 14

A.3 Dataset Construction and Preprocessing

  • RoboTwin training uses 50 bimanual tasks with Clean and randomized demonstrations, while EgoDex clips are subsampled and resized for action-free video training.RoboTwin uses 50 Clean and 500 Random demonstrations per task; EgoDex clips are sampled at several temporal lengths.Source: §A.3 · p. 15
  • RAFT flow targets isolate robot-induced motion in RoboTwin through robot-only replay, while EgoDex flow is computed between adjacent video frames.The RoboTwin replay uses a static background to reduce background, lighting, and object-motion artifacts.Source: §A.3 · p. 15

A.4 Input Representation and Flow Encoding

  • Multi-view RGB cameras are tiled into one T-shaped frame; the head region receives flow supervision and wrist regions use a constant placeholder.Flow direction and magnitude are encoded as HSV hue and saturation, with a 25 px magnitude cap.Source: §A.3 · p. 15, §A.4 · p. 15

A.5 Hyperparameter Summary

  • Table 3 summarizes architecture, data, optimization, and numerical settings for both training stages; training uses 32 NVIDIA H100 GPUs across four nodes.The caption identifies the table as a two-stage hyperparameter summary.Source: §A.5 · p. 16, Table 3 · p. 16

B Baseline Details

  • The baseline section identifies π0.5 and Motus as real-world comparison models and describes the simulated manipulation and world-modeling baseline groups.The real-world comparison follows the protocol in Section 4.3 and Appendix E.Source: B Baseline Details · p. 16

B.1 Manipulation Policy Baselines

  • Manipulation policy baselines include direct action prediction by π0.5 and embodiment-aware prompting by X-VLA, alongside WAMs that use latent, pixel-action, or numerical-token representations.Motus uses learned latent actions; GigaWorld-Policy, X-WAM, and Fast-WAM differ in how video and action prediction are coupled.Source: §B.1 · pp. 16–17

B.2 World-Modeling Baselines

  • World-model baselines span generic video priors and action-conditioned models using numerical actions, pose and memory, inverse dynamics, or other visual-control designs.The described examples include CogVideoX and Veo 3.1, as well as IRASim, Ctrl-World, Vidar, Genie Envisioner, and GigaWorld-1.Source: §B.2 · pp. 17–18

C Additional RoboTwin Results

This material points to the full per-task RoboTwin policy results and explains how WorldArena aggregates evaluation across dimensions of video and action quality.

C Additional RoboTwin Results

  • Table 4 provides full per-task RoboTwin 2.0 policy success rates corresponding to the compact main-text comparison.The supplied passage identifies the table’s scope but does not report its individual values.Source: C Additional RoboTwin Results · p. 18

D WorldArena Evaluation Details

  • The WorldArena benchmark comprises 16 metrics, each normalized to [0, 100] with higher values indicating superior performance, and defines EWMScore as the arithmetic mean of the 16 metrics.Source: §D.1 · p. 18

D.1 Metric Grouping and Aggregate Scoring

  • WorldArena groups its metrics into six functions: visual quality, motion quality, content consistency, physics adherence, 3D accuracy, and controllability.The groups cover frame realism, motion, consistency, interactions and trajectory accuracy, geometry, and instruction/action response.Source: §D.1 · p. 18

D.2 Metric Definitions and Implementation

WorldArena’s metric definitions cover visual, motion, consistency, physical, geometric, and controllability dimensions. FlowWAM’s reported strengths center on action-grounded prediction.

D.2 Metric Definitions and Implementation

  • WorldArena follows official metric definitions and normalizes raw measures to a common scoring range before evaluation.The listed definitions describe the evaluators and what each metric measures.Source: §D.2 · p. 18
  • Visual-quality measures assess frame quality, aesthetics, and generated-to-reference video similarity, while motion measures capture motion strength, intensity, and smoothness.The measures include IQ, AQ, JEPA, DD, FS, and MS.Source: §D.2 · pp. 19–20
  • Consistency measures track subject, background, and photometric stability, while physics and 3D measures assess interaction plausibility, trajectory alignment, depth, and perspective.The definitions include SC, BC, PC, IntQ, TA, DA, and Per.Source: §D.2 · p. 20
  • Controllability measures assess instruction following, semantic agreement with reference executions, and whether different instructions produce distinct outputs.These measures are InsF, SA, and AcF.Source: §D.2 · p. 21

D.3 Performance Analysis on WorldArena

  • FlowWAM achieves state-of-the-art results in Trajectory Accuracy, Depth Accuracy, and Semantic Alignment, with gains primarily on action-grounded prediction metrics.The passage reports competitive subject and background consistency and characterizes reliable motion rollouts as a core strength.Source: §D.3 · p. 21

E Real-World Experimental Setup

The real-world evaluation uses a single-arm Franka Research 3 and a bimanual ARX 5 platform to assess transfer across embodiments.

E Real-World Experimental Setup

  • The experimental setup evaluates FlowWAM on a single-arm Franka Research 3 and a bimanual ARX 5 robot.The platforms represent single-arm and two-arm manipulation settings.Source: Figure 6 · p. 21, §E.1 · p. 21

E.1 Robot Platforms and Sensors

  • Franka Research 3 uses a parallel-jaw gripper and one ZED workspace camera; ARX 5 uses an OCS2 control stack and three RealSense cameras for global and wrist views.The Franka is used for precision pick-and-place and contact-rich manipulation; ARX 5 is used for coordinated two-arm tasks.Source: §E.1 · p. 21

E.2 Task Suite and Data Collection

The real-world suite tests spatial precision, object interaction, and bimanual coordination across seven manipulation tasks. All methods use the same demonstrations and randomized-pose evaluation protocol.

E.2 Task Suite and Data Collection

  • Seven tasks test spatial precision, object interaction, and bimanual coordination across single-arm Franka and dual-arm ARX platforms.Franka tasks cover bowl stacking, drawer placement, plate placement, and cup placement; ARX tasks cover towel folding, bowl stacking, and plate cleaning.Source: §E.2 · p. 22
  • All methods train on 100 teleoperated demonstrations and are evaluated over 10 trials with randomized object poses.Source: §E.2 · p. 22

F Visualization Results

The visualizations make FlowWAM’s generated rollouts inspectable across simulated policy tasks, physical robot executions, and action-conditioned world modeling. They show how optical-flow plans relate to predicted frames and realized motion.

F Visualization Results

  • The visualizations complement quantitative evaluations by making intended robot motion, object interaction, and scene geometry inspectable over time.Source: F Visualization Results · p. 22

F.1 RoboTwin Policy Rollouts

  • In Clean and Random RoboTwin rollouts, predicted flow traces coherent goal-directed robot motion while RGB frames preserve object context, and decoded actions complete tasks despite scene randomization.The examples pair predicted RGB and flow with simulation rollouts decoded by the action expert.Source: §F.1 · p. 22, Figure 7 · p. 24, Figure 8 · p. 25

F.2 Real-World Policy Rollouts

  • Real-world examples pair predicted RGB and flow with physical executions across four single-arm Franka tasks and three bimanual ARX tasks.The comparison makes the dense image-space motion plan and its translation into end-effector motion and object interaction inspectable.Source: §F.2 · p. 22, Figure 9 · p. 26

F.3 World Model Visualization

  • Action-conditioned predictions pair future RGB frames with motion structure, illustrating how dense optical flow guides coherent robot-object motion and reduces action drift.Source: §F.3 · pp. 22–23, Figure 10 · p. 27

G Broader Impacts

  • The work studies flow-based action representations in simulated and lab-scale manipulation and anticipates no direct societal impacts beyond foundational robot-learning research.Source: G Broader Impacts · p. 23
Loading 2607.13017v1…