PaperlayerView exact revision on arXiv

Source-linked AI summary

UniWAM: Unified World-Action Model

Jiayi Chen, Wenxuan Song, Jingbo Wang, Shuai Zhou, Xicheng Gong, Zehua Fan, Ziyang Zhou, Junwu E, Haodong Yan, Fuhao Li, Qize Yu, Xu Huang, Pengwei Wang, Wen Chen, Shunbo Zhou, Haoang Li

arXiv:2610.02054v1cs.RO

TL;DR

Action-only supervision in VLAs provides limited grounding in world dynamics, while WAMs retain limitations in semantic understanding and reasoning under distribution shifts. UniWAM combines a physical reasoner, world generator, and action predictor, with complementary human, robot, and VQA supervision. The authors report SOTA results across multiple evaluation dimensions and a log-linear scaling law for unified human-robot co-training.

  • Problem

    VLAs have limited grounding in world dynamics under action-only supervision, while WAMs remain limited in semantic understanding and reasoning under distribution shifts.

    Source: §1 · p. 3

  • Method

    UniWAM unifies a VLM-based physical reasoner, a VGM-based world generator, and an action predictor, trained with complementary VQA, human, and robot supervision.

    Source: §1 · p. 3

  • Results

    UniWAM achieves SOTA performance across in-distribution performance, robustness, generalization, instruction following, and long-horizon task execution, and exhibits a log-linear human-robot co-training scaling law.

    Source: §1 · p. 4

  • Takeaways & Limitations

    The authors conclude that pretraining a unified world-action architecture on heterogeneous data offers a promising path toward generalist robot models.

    Source: §6 · p. 17

Abstract

from arXiv · show

Vision-language-action models benefit from the understanding and reasoning capabilities of pretrained vision-language models, but action-only supervision provides limited grounding in world dynamics. Conversely, world-action models inherit spatiotemporal priors from video generation models, yet remain limited in semantic understanding and reasoning under distribution shifts. We introduce UniWAM, a unified architecture that integrates a physical reasoner, a world generator, and an action predictor to jointly learn semantic understanding of the physical world, visual generation, and action prediction. To ensure the quality of the training data, we developed a rigorous data cleaning and annotation pipeline for both human egocentric data and robot data. To adapt the vision-language component to embodied tasks while preserving its inherited language capabilities, we represent low-level actions in natural language and introduce a pre-training recipe that assigns complementary supervision from visual question answering (VQA) data, human egocentric data, and robot demonstrations to the appropriate model components. During post-training, future visual noise augmentation reduces reliance on precise future predictions, while history-conditioned flow matching uses encoded action history to initialize action generation. Together, these designs significantly reduce denoising steps while maintaining performance. UniWAM achieves state-of-the-art (SOTA) performance across multiple evaluations, including in-distribution performance, robustness, generalization, instruction following, and long-horizon task execution. Furthermore, we uncover a log-linear scaling law of unified human-robot co-training, demonstrating the effectiveness of large-scale pre-training on a mixture of human and robot data.

1 Introduction

VLAs leverage pretrained language understanding but action-only supervision limits grounding in world dynamics, while WAMs model dynamics yet remain limited in semantic understanding and reasoning under distribution shifts. UniWAM unifies reasoning, world generation, and action prediction, with complementary data supervision and post-training designs; the authors report broad SOTA performance and a log-linear co-training scaling law.

Architecture

  • UniWAM combines a VLM-based physical reasoner, a VGM-based world generator, and an action predictor to support semantic understanding, visual prediction, and action generation.The components are jointly trained in a unified architecture.Source: §1 · p. 3

Pre-training

  • Natural-language representations of low-level actions adapt the VLM to embodied tasks while preserving pretrained language capabilities, and the pre-training scheme assigns VQA, human, and robot supervision to different components.VQA supervises the VLM; human data supervises the VLM and VGM; robot data supervises all three components.Source: §1 · p. 3

Data quality

  • A rigorous data-cleaning pipeline and automated human-trajectory segmentation and annotation support training on human egocentric and robot data.The segmentation pipeline addresses human trajectories that often contain multiple actions.Source: §1 · p. 3

Post-training

  • Future-visual noise augmentation and history-conditioned flow matching reduce denoising steps while maintaining performance.Perturbed future latents encourage control-relevant semantic use rather than reliance on precise future predictions, while encoded action history initializes action generation.Source: §1 · pp. 3–4

Results

  • UniWAM achieves SOTA performance across in-distribution performance, robustness, generalization, instruction following, and long-horizon task execution.The authors also report a log-linear scaling law for unified human-robot co-training.Source: §1 · p. 4
  • A log-linear scaling law in unified human-robot co-training demonstrates the effectiveness of large-scale pre-training on mixed human and robot data.The authors observe the pattern by plotting validation loss at convergence against data scale.Source: §1 · p. 4

2 Related Work

Prior work studies language-action or video-action modeling, WAM dynamics, and the use of egocentric data for embodied learning. UniWAM is distinguished in this account by unifying a VLM, a video-generation model, and an action expert with complementary data and supervision.

Large-scale Pretraining for Robotics

  • Earlier robotics pretraining approaches emphasize language-action or video-action joint modeling, whereas UniWAM unifies semantic understanding, dynamics prediction, and action generation through diverse data and complementary supervision.The cited approaches include web-plus-robot training, VLM adaptation, flow matching, semantic subtask supervision, and shared motion learning.Source: §2 · p. 4

World-action Models

  • WAMs use video-generation backbones and future prediction for dynamics supervision, while UniWAM combines a VLM, a video-generation model, and an action expert in one framework.Prior WAMs also use video backbones as representation encoders or extend representations to four-dimensional spatiotemporal geometry.Source: §2 · p. 4

Learning from Egocentric Data

  • Egocentric video supplies diverse human-object interactions and long-horizon activities, with research progressing from representation learning toward structured supervision and policy learning.Recent embodied models also combine human and robot experience through shared action representations and cross-embodiment pretraining.Source: §2 · pp. 4–5

3 Data

Section 3 combines robot and human egocentric demonstrations with VQA supervision for pretraining, then applies quality control and aligns observations, language, and motion. Its EgoANT pipeline segments long videos into manipulation events and assigns each a subtask description.

3 Data

  • Robot and human egocentric demonstrations are combined with VQA data to provide complementary pretraining supervision.Robot and human data receive language–action supervision, while VQA responses train the understanding expert.Source: §3.1.1 · p. 5, §3.1.2 · p. 5, §3.1.3 · p. 6

3.1.1 Robot Data

  • Approximately 4,363 hours of robot data from six sources cover single-arm and dual-arm embodiments and receive local language–action descriptions.The sources are AgiBotWorld2026, Bridge, Droid, Fractal, Robocoin, and Interdata-a1.Source: §3.1.1 · p. 5

3.1.2 Human Egocentric Data

  • Three egocentric sources provide approximately 5,000 hours of video with language and dense motion annotations across varied manipulation tasks, objects, and environments.EgoDex, EgoVerse, and processed VITRA subsets supply the data; EgoDex receives automatically generated subtask annotations.Source: §3.1.2 · p. 5

3.1.3 VQA Data

  • VQA datasets supervise spatial, compositional, temporal, and manipulation reasoning, with responses training the understanding expert through an autoregressive language objective.The sources include object-grounding and spatial-reasoning datasets, video reasoning, and robot task-state and interaction datasets.Source: §3.1.3 · p. 6

3.2 Data

  • Robot data undergo quality control across temporal and statistical reliability, geometric consistency, and visual validity.The procedure targets heterogeneous failures in trajectory signals and visual observations across datasets and acquisition pipelines.Source: §3.2 · p. 6

3.2.1 Robot Data Processing

  • Robot trajectory screening checks abrupt transitions, state–action timing and directional agreement, and robust per-dimension outliers; geometry checks are conditional on reliable metadata.Visually invalid frames are removed while interaction-critical frames such as gripper closures are preserved.Source: §3.2.1 · pp. 6–7

3.2.2 Human Egocentric Data Processing and Annotation

  • Human annotations align subtask descriptions with local behavior, frames, head-camera poses, and bilateral wrist trajectories in a common representation.Wrist poses are expressed in the head-camera frame at conditioning time; samples with missing annotations, invalid alignment, insufficient duration, or implausible geometry are removed.Source: §3.2.2 · pp. 7–8
  • EgoANT first proposes manipulation-event boundaries for a full episode, refines them locally, then generates candidate descriptions and selects one using text alone.The final description is paired with the refined interval; the selector does not re-examine video frames.Source: §3.2.2 · pp. 7–8

4 Method

UniWAM combines physical reasoning, visual generation, and action prediction in a jointly trained architecture. Its method pairs natural-language supervision for physical actions with joint objectives and post-training strategies for robust, history-conditioned action generation.

4.1 Model Architecture

  • UniWAM combines understanding, video, and action experts through joint attention, conditioning on observations, proprioceptive state, and task instructions.The model jointly learns language outputs, future observations, and actions.Source: §4.1 · pp. 9–10

4.2.1 Physical Language Supervision

  • End-effector motions become natural-language targets, giving human and robot data shared motion semantics alongside task instructions.Structured templates and a fixed coordinate convention describe continuous actions at arbitrary precision.Source: §4.2.1 · p. 10

4.2.2 Training Objectives

  • Visual and action velocity fields are trained with flow-matching losses, while an autoregressive language loss uses physical-language descriptions or VQA responses.The velocity targets are differences between source and clean target samples; during pretraining, both sources are Gaussian noise.Source: §4.2.2 · p. 10

4.3 Posttraining

  • Post-training jointly supervises all three experts to adapt understanding, generation, and prediction to embodied scenarios while maintaining unified capabilities.Source: §4.3 · p. 10

4.3.1 Future-frame Noise Augmentation

  • Future-frame noise augmentation perturbs future visual latents in half of post-training samples while keeping the current observation clean.This encourages action prediction to tolerate imperfect future representations while retaining the observed task context.Source: §4.3.1 · p. 11

4.3.2 History-conditioned Flow Matching

  • History-conditioned flow matching replaces Gaussian action noise with a perturbed chunk of previously executed actions to encourage temporal consistency and refine generation with fewer inference steps.The history source adds small stochastic perturbations with η = 0.02; the visual source remains Gaussian noise.Source: §4.3.2 · p. 11, Figure 4 · p. 11

5 Experiments

The experiments assess UniWAM in simulation and on real robots, then examine human–robot data scaling, learned representations, and training-design ablations. The reported results span instruction following, long-horizon execution, robustness, and generalization.

5.1 Simulated Experiments

  • UniWAM reaches 99.2% average LIBERO success, 0.5% above Xiaomi-Robotics-0, and leads LIBERO-Plus at 92.6% and RoboTwin C2R at 68.32%.Its C2C-to-C2R drop is 6.82 percentage points, versus 24.70 for π0.5 and 50.46 for Spatial Forcing.Source: Table 4 · p. 12, §5.1 · p. 12, Table 5 · p. 13

5.3 In-depth Analysis of Human-robot Co-training

  • OOD task completion rises from 0.35 at 250 robot-data hours to 0.67 at 5k; human data begins yielding gains at 1k hours, with larger gains as data grows.Validation loss follows a log-linear scaling trend with R2 ≈0.983, and tracks downstream task-success improvements.Source: §5.3 · p. 14
  • With physical-language supervision, robot and human representations overlap more; without it, their samples largely occupy separate regions.The authors attribute the overlap to shared physical-language targets for common manipulation semantics across embodiments.Source: §5.3 · pp. 14–15

5.2 Real-world Experiments

  • UniWAM averages Task Progress 5.0 versus 4.8 for π0.5 and 3.2 for Motus on the long-horizon task, given only “Tidy up the desk.”Training predicts fine-grained subtasks alongside physical-reasoner action reasoning; the score tracks progress through six sequential milestones.Source: §5.2 · pp. 13–14, Figure 5 · p. 15

5.4 Ablation Studies

  • The three-source pretraining variant leads evaluated variants at 75.14% C2C, 68.32% C2R, and 71.73% average success.Robot-data pretraining raises average success from 46.23% to 67.53%; adding human data improves both C2C and C2R, while VQA primarily improves C2R.Source: §5.4 · pp. 15–16

5.2 Real-world Experiments

  • UniWAM averages 67.5% success and 82.5% instruction following across four real-world tasks, exceeding π0.5 by 13.1 and 21.9 percentage points, respectively.On Diverse-Interaction, success rises from 40.0% to 72.5%, and instruction following from 50.0% to 80.0%.Source: §5.2 · p. 13, Figure 6 · p. 16

5.4 Ablation Studies

  • Unfreezing the reasoner raises average success from 42.61% to 46.49%; physical-language supervision raises it to 51.74%, and pretraining to 71.34%.Source: §5.4 · p. 16
  • A2A raises average success from 64.80% to 69.73% over N2A, and future visual augmentation raises it further to 71.34%.Source: §5.4 · p. 16

6 Conclusion

UniWAM jointly learns physical-semantic reasoning, world modeling, and action prediction, using cleaned human and robot data with component-specific supervision. The conclusion reports strong results across task performance and robustness, alongside a log-linear human–robot co-training scaling law.

6 Conclusion

  • UniWAM combines physical-semantic reasoning, world modeling, and action prediction in one framework.The authors also developed data cleaning and annotation pipelines for both human egocentric and robot data.Source: §6 · p. 16
  • Physical-language supervision and component-specific co-training on VQA, human egocentric, and robot data build embodied knowledge while preserving pretrained semantic capabilities.Source: §6 · p. 17
  • The authors report strong performance across task performance, robustness, generalization, instruction following, and long-horizon execution, plus a log-linear human–robot co-training scaling law.They describe heterogeneous-data pretraining of a unified world-action architecture as a promising path toward generalist robot models.Source: §6 · p. 17

B Real-world Tasks

The real-world task set covers household objects and dual-arm manipulation; object-centric tasks use the arm nearer the target, with both arms used when coordination is required. Each task is evaluated over 40 trials.

Pick-Anything

  • Pick-Anything tests identifying and grasping an instructed target among eight objects, then placing it on a plate.Source: B Real-world Tasks · p. 26

Diverse-Interaction

  • Diverse-Interaction tests grounding both object and action, including different interactions with the same object across instructions.Source: B Real-world Tasks · p. 27

Place-Relative

  • Place-Relative tests identifying source and reference objects and placing the source to the instructed left or right of the reference.Source: B Real-world Tasks · p. 27

Drawer Storage

  • Drawer Storage tests target-object and drawer-level grounding, sequential manipulation, and bimanual coordination to store an object on the specified level.Source: B Real-world Tasks · p. 27

C Action Generation with Different Inference Steps

The appendix compares action-generation strategies across inference-step counts and describes the shared physical-language action template used for human and robot data.

C Action Generation with Different Inference Steps

D Physical Action Templates

  • The shared physical-language template describes end-effector changes over a sliding action window, using rounded translations and rotations and gripper state when available.It converts absolute poses to per-step changes, omits unchanged components, and prefixes bimanual summaries with the relevant arm.Source: D Physical Action Templates · pp. 27–28

E Details of Evaluation on RoboTwin 2.0

RoboTwin 2.0 evaluation uses 50 bimanual manipulation tasks, with clean and randomized settings to assess performance and generalization.

E Details of Evaluation on RoboTwin 2.0

F RoboTwin 2.0 Task-Level Results

Table 7 is the authoritative location for task-level RoboTwin 2.0 percentages; the main text reports selected values for aggregate performance and failure boundaries.

F RoboTwin 2.0 Task-Level Results

  • Table 7 reproduces all task-level values supplied in the experiment record.The main text instead reports values needed to show aggregate performance and important failure boundaries.Source: F RoboTwin 2.0 Task-Level Results · p. 29

G Implementation Details

UniWAM combines three specialist components in a Mixture-of-Transformers architecture, with distinct model dimensions and defined state, frame, and action-chunk formats.

G Implementation Details

  • UniWAM combines Qwen3-VL-2B-Instruct, Wan2.2-TI2V-5B, and an action predictor in a Mixture-of-Transformers architecture.Each of the three experts has 30 layers.Source: Model architecture · p. 29
  • The world generator has hidden dimension 3,072 and 24 attention heads; the physical reasoner and action predictor have hidden dimensions 512 and 1,024.Source: Model architecture · p. 29
  • The model uses 14-dimensional robot states and actions, predicts eight future frames at 384 × 320 resolution, and generates action chunks of length 16.Source: Model architecture · p. 29

Model architecture.

The implementation specifies the model’s loss weighting and compute, post-training setup, and low-step inference initialization.

Model architecture.

  • Pretraining runs for seven days on 32 NVIDIA H800 GPUs, with video, action, and language losses weighted 1:1:0.1.Source: Pretraining · p. 29

Pretraining.

  • RoboTwin post-training uses 200,000 optimization steps; future-frame noise augmentation is applied with probability 0.5 and latent retention sampled from [0.5, 1.0].The RoboTwin and LIBERO runs use eight NVIDIA H800 GPUs for 12 and 20 hours, respectively.Source: Post-training · p. 29

Post-training.

  • Inference initializes action flow from a recent 16-action history chunk with Gaussian perturbation and future video latents from Gaussian noise, using 2–4 flow integration steps.The perturbation standard deviation is 0.02.Source: Post-training · p. 29
Loading 2610.02054v1…