Source-linked AI summary
RepWAM: World Action Modeling with Representation Visual-Action Tokenizers
Junke Wang, Qihang Zhang, Shuai Yang, Yiming Luo, Yujun Shen, Zuxuan Wu, Yu-Gang Jiang, Yinghao Xu
TL;DR
Existing world action models often rely on reconstruction-oriented visual representations that provide limited guidance for instruction-following dynamics linking future prediction and robot control. RepWAM uses semantic visual-action tokenization to jointly model future visual states and latent actions, achieving competitive closed-loop behavior across real-world and simulated manipulation tasks.
Problem
World action models lack a representation that jointly supports semantic future-visual prediction and robot-action inference under language instructions.
Method
RepWAM aligns visual latents with a frozen visual foundation model, learns latent actions between visual states, and jointly models instruction-conditioned futures and actions.
Results
RepWAM reaches 89.3 on Easy tasks and 88.4 on Hard tasks in RoboTwin 2.0 while delivering competitive closed-loop behavior across real-world and simulation evaluations.
Takeaways & Limitations
Semantic visual-action alignment is a promising foundation for world action modeling across diverse manipulation settings.
Takeaways & Limitations
The authors identify scaling pretraining beyond robotics-domain videos, particularly to large-scale egocentric human videos, as future work.
Abstract
from arXiv · showhide
This work presents RepWAM, a representation-centric world action model (WAM) built on representation visual-action tokenizers. Existing WAMs typically inherit reconstruction-oriented video tokenizers from pretrained video generation models. Although these tokenizers preserve visual fidelity, pixel reconstruction alone provides limited guidance for learning instruction-following dynamics that connect future prediction with robot control. To address this, we explore a semantic visual-action latent space for representation-centric world action modeling. Specifically, we train a representation visual-action tokenizer that maps visual inputs into aligned visual and latent action tokens. We then pretrain our WAM to jointly model future visual states and the latent actions that connect them under language instructions, followed by adaptation to real robot trajectories for closed-loop manipulation. Experiments on real-world manipulation tasks and simulation benchmarks show that RepWAM delivers strong performance across diverse manipulation settings, while ablations highlight the value of semantic visual-action tokenization over reconstruction-oriented alternatives. These results establish representation visual-action tokenization as a promising foundation for world action models and a step toward generalist robot policies. Code and weights will be available at https://github.com/wdrink/RepWAM.
1 Introduction
RepWAM addresses representation shortcomings in world action models by aligning visual and latent action tokens in a shared semantic space for instruction-conditioned dynamics and control. It achieves competitive closed-loop behavior, reaching 89.3 on RoboTwin 2.0 Easy tasks and 88.4 on Hard tasks.
- Motivation: World action models require a shared latent space that forecasts visual dynamics and infers robot actions for embodied control.Their capabilities depend fundamentally on the representation through which they perceive the world.
- Motivation: Reconstruction-driven video tokenizers overemphasize low-level appearance, underrepresenting object identity, spatial relations, and interaction cues needed for manipulation.This weakens instruction-conditioned world modeling because language goals must be inferred from appearance-dominated latents.
- Method: RepWAM introduces representation visual-action tokenizers by aligning a video autoencoder’s latent space with a frozen visual foundation model and learning manipulation-centric latent actions.The resulting semantic visual and latent action tokens are designed to support representation-centric world action modeling.
- Evaluation: RepWAM is evaluated on real-world manipulation tasks and simulation benchmarks against vision-language-action and WAN-pretrained WAM baselines.The evaluation covers both physical manipulation and simulated settings.
- Results: 89.3 on Easy tasks and 88.4 on Hard tasks in RoboTwin 2.0 demonstrate RepWAM’s competitive closed-loop behavior.The results support shared semantic grounding of visual and action latents as a foundation for world action modeling.
2 Related Work
Related work develops world action models and latent action models that connect visual dynamics to robot control, including approaches that learn from unlabeled videos. RepWAM instead learns latent actions as transformations between semantically aligned visual states.
- World Action Models: World action models use predicted scene evolution as an intermediate representation for generating robot actions.Motus uses specialized experts for understanding, video generation, inverse dynamics, and action prediction, while Lingbot-VA jointly learns frame prediction and policy execution.
- Latent Action Models: Latent action models recover action-like variables from observation sequences, enabling unlabeled videos to support downstream control.Genie makes video dynamics controllable without ground-truth action labels; LAPO recovers latent-action policies and dynamics before limited-supervision or reward adaptation.
- Latent Action Models: Prior work progressively treats visual change as a substitute for missing motor labels through latent actions and autoregressive video pretraining.These methods transfer motion regularities into robot policies, extending controllable generation toward policy learning.
- Semantic Visual-Action Tokenization: RepWAM learns latent actions in a semantic visual latent space, representing each action as a transformation between semantic visual states.Its tokenizer aligns visual latents with a frozen visual foundation model and induces latent actions through coupled inverse and forward dynamics models.
3 Method
RepWAM addresses latent-space limitations in world action models by aligning visual tokens with semantic representations and learning latent action tokens as transitions between visual states. It then jointly generates visual and action-token streams under language conditioning using a causal diffusion transformer.
- Motivation: WAMs forecast future observations before inferring actions, but reconstruction-oriented visual latents and decoupled action spaces hinder instruction-following dynamics.The visual latent space emphasizes appearance rather than semantics, while the action expert must bridge a modality gap at every step.
- Representation visual-action tokenizer: RepWAM introduces RepViTok, which aligns visual latents with a visual foundation model and induces latent action tokens within the same semantic space.The tokenizer represents visual content with visual tokens and temporal dynamics with action tokens, jointly representing states and action-induced transitions.
- Visual tokenization: The visual tokenizer uses a temporally causal, spatially attentive ViT autoencoder and combines reconstruction supervision with frozen-foundation-model feature alignment.Its total objective is ℒvis = ℒrec + 𝜆alignℒalign, balancing reconstruction fidelity and semantic alignment.
- Latent action tokenization: The latent action tokenizer freezes visual tokens and trains an IDM-FDM pair to compress each transition into a low-dimensional action token and reconstruct the next latent.The FDM represents the transition with a soft transport operator over spatial tokens plus a residual, while backward consistency is also trained.
- Latent action tokenization: Because latent actions transform semantic visual states rather than embodiment-specific motor coordinates, they describe transferable task-level transitions.This construction is intended to connect state changes in a shared representation space without relying on robot-specific action coordinates.
- World action model: RepWAM jointly generates visual and latent-action chunks under language conditioning with a block-causal diffusion transformer trained by conditional flow matching.Chunks contain visual latents and actions over short temporal windows, and the loss jointly weights their visual and latent-action components.
4 Experiments
RepWAM performs strongly in real-robot manipulation and randomized bimanual simulation, with larger models excelling on long-horizon tasks. Ablations show that semantic visual-action tokenization and staged training improve both visual dynamics and closed-loop control.
- Real-world manipulation: Both RepWAM sizes reach 60% on pick-the-fruit, exceeding 𝜋0.5 by 50 points and Lingbot-VA by 10 points.The two model sizes are tied on this short-horizon task, whose bottleneck is perception and grasping rather than model capacity.
- Simulation benchmark: RepWAM-5B outperforms 𝜋0.5 and Motus on the RoboTwin 2.0 50-task average in both Easy and Hard settings.The evaluation uses the benchmark’s official randomization settings, covering scene composition, lighting, viewpoints, and object physics.
- Tokenizer ablations: RepViTok reduces gFVD by 9.5% / 13.2% versus WAN2.2 VAE and improves OLS from 13.68 / 11.21 to 18.82 / 14.15.On PickFruit, RepViTok reaches 30% success, compared with 20% for WAN2.2 VAE and 10% for ViTok.
- Tokenizer ablations: Replacing WAN2.2 VAE with RepViTok raises average success from 78.0 to 86.6 on Easy and from 76.0 to 83.1 on Hard with the 1.3B WAM fixed.This isolates the visual tokenizer and supports semantics-aware tokenization as a contributor to world action modeling performance.
- Training ablations: Two-stage training achieves the best result on every reported metric, reducing gFVD to 48.23 / 58.83, raising PSNR to 22.86 / 19.93 and OLS to 19.87 / 16.98.Joint Pred instead increases gFVD to 94.25 / 98.77; two-stage training also raises PickFruit closed-loop success to 50%.
5 Conclusion
RepWAM is a representation-centric world action model that uses semantic visual-action tokenization to jointly model instruction-conditioned future observations and latent actions. Future work will scale pretraining beyond robotics videos using large-scale internet and egocentric human videos.
- Conclusion: RepWAM aligns visual latents with a frozen visual foundation model and learns latent actions as transitions between visual states.It uses semantic visual-action tokenization instead of reconstruction-oriented video latents inherited from pretrained generation backbones.
- Conclusion: Future work will scale WAM pretraining beyond robotics-domain videos by leveraging large-scale internet videos, especially egocentric human videos.The authors expect this could broaden the range of behaviors and interaction patterns available during pretraining.