Source-linked AI summary
Token-World: World Modeling in Vision-Language Model Token Space for Robot Manipulation
Chuyao Fu, Xiaowei Chi, Yuhan Rui, Yu-kai Wang, Zezhong Qian, Xiaojie Zhang, Yunfan Lou, Kevin Zhang, Kuangzhi Ge, Chak Wing Mak, Zhiyang Chen, Athena Zhuoming Zhong, Hongyang Chen, Haoran Li, Yike Guo, Sirui Han, Shanghang Zhang
TL;DR
VLA world-model pipelines often predict RGB futures and re-encode them, while direct prediction in policy-facing VLM tokens is challenging because those tokens are high-dimensional. Token-World compresses them into a compact state for action-conditioned dynamics and maps predictions back for policy use. Across manipulation benchmarks it improves open-loop fidelity and action consistency, with stronger closed-loop correlation to reference policy performance than Ctrl-World.
Problem
VLA world-model pipelines often predict future RGB observations and re-encode them as policy inputs, while direct dynamics modeling in high-dimensional policy-facing VLM tokens is computationally and methodologically burdensome.
Source: §I · p. 2Method
Token-World compresses policy-facing VLM features into a compact token state, predicts action-conditioned futures there, and maps predicted states back to the original representation for downstream policies.
Source: §I · p. 2Results
Across manipulation benchmarks, Token-World improves open-loop feature fidelity and policy-action consistency with slower long-rollout degradation; closed-loop correlation with reference policy performance is r = 0.794 vs. 0.583 for Ctrl-World.
Source: §I · p. 2Takeaways & Limitations
Representation design and dimensionality affect future-state prediction, and Token-World reports lower simulation latency than recent world-model baselines.
Source: §I · p. 2, §V · p. 7The current evaluation focuses on manipulation tasks.
Source: §V · p. 7
Abstract
from arXiv · showhide
A common approach to world-model simulation for vision-language-action (VLA) systems is to predict future RGB observations and then re-encode them into policy inputs, introducing an indirect interface between simulation and downstream policy execution. We instead investigate whether world dynamics can be modeled in a compact, policy-oriented state derived from VLM visual tokens. A key challenge is that raw VLM visual tokens are high-dimensional, making efficient and accurate autoregressive dynamics modeling challenging. To address this, we introduce Token-World, an action-conditioned world model that compresses VLM features into a compact token state, learns future dynamics in this reduced space, and maps predicted states back to the original policy-facing representation for downstream use. Across manipulation benchmarks, Token-World improves open-loop feature fidelity and policy-action consistency over recent world-model simulators, with slower degradation over long rollout horizons. In closed-loop evaluation, its simulated policy performance correlates more strongly with reference policy performance than Ctrl-World ($r=0.794$ vs.\ $0.583$), while requiring lower simulation latency. Ablations further show that compact-representation design and dimensionality substantially affect future-state prediction. Code will be available at https://chuyaofu.github.io/Token-World/.
I. INTRODUCTION
World models can simulate action-conditioned futures, but VLA pipelines often predict RGB and re-encode it even though policies consume visual tokens. Token-World models dynamics in compact policy-aligned VLM tokens and improves reported manipulation-simulation measures.
I. INTRODUCTION
- VLA world-model pipelines often predict future RGB observations and re-encode them as policy inputs, creating an indirect interface between simulation and policy execution.RGB reconstruction can devote capacity to visual details irrelevant to the policy, while policy-facing tokens are the representation used for action generation.Source: §I · p. 2
- Directly predicting high-dimensional policy-facing VLM tokens imposes substantial modeling burden and computational cost.The tokens are optimized primarily for perception and action rather than generative dynamics modeling.Source: §I · p. 2
- Token-World compresses VLM features into a compact semantic token state, predicts action-conditioned futures there, and maps states back to policy-facing tokens without intermediate RGB generation.Its dynamics model is a flow-matching DiT and does not rely on pretrained videogeneration models.Source: §I · p. 2
- Token-World improves open-loop feature fidelity and policy-action consistency across manipulation benchmarks, degrades more slowly over long rollouts, and correlates more strongly with reference policy performance than Ctrl-World (r = 0.794 vs. 0.583).It also provides higher simulation efficiency; representation ablations show that compact-state design and dimensionality affect future-state prediction.Source: §I · p. 2
- The paper evaluates direct token-space simulation through open-loop prediction, closed-loop policy evaluation, and representation ablations.The reported evaluation covers long-horizon feature and action fidelity, policy-success estimation, and simulation cost.Source: §I · p. 2
II. RELATED WORK
The related work spans world models used as learned simulators and predictive latent representations, while Token-World focuses on constructing a compact dynamics state from policy-facing VLM tokens. Its overview describes compression, action-conditioned modeling, and autoregressive rollout in that compact space.
A. World Models as Learned Simulators
- World models support planning and policy learning through imagined rollouts, with robotic applications including policy evaluation, synthetic interaction generation, and policy optimization.Token-World asks what representation should serve as the state of a world model coupled with a VLA policy.Source: A. World Models as Learned Simulators · p. 2
B. Representation Design for Latent World Models
- Prior work predicts futures in visual features, latent embeddings, semantic masks, VLM features, or compact view-level states; Token-World instead compresses policy-facing VLM tokens along the feature dimension while preserving spatial layout.Related compression methods include compact image-generation latents and world-model tokens that summarize feature changes or visual views.Source: B. Representation Design for Latent… · p. 2
A. Overview
- Token-World compresses Qwen3-VL visual tokens, models action-conditioned dynamics with a spatiotemporal Transformer, and autoregressively generates future compact tokens using a sliding window and recurrent state.Predicted states can be decoded to policy-facing features when needed; training uses an x0-parameterized flow objective with shortcut forcing.Source: Figure 2 · p. 3, A. Overview · p. 3
B. Compact VLM World State
Token-World uses a separately trained semantic VAE to form a compact state from frozen VLM features. Compression preserves the spatial token layout while reducing each token's feature dimension.
B. Compact VLM World State
- A separately trained semantic VAE constructs the compact state from features produced by a frozen VLM encoder.The compression encoder parameterizes a diagonal Gaussian posterior; the posterior mean is used when exporting representations for world-model training.Source: B. Compact VLM World State · p. 3
- The codec is optimized independently of the dynamics model, and its encoder and decoder are frozen during world-dynamics learning.The decoder reconstructs the original policy-facing representation; the codec objective includes latent regularization controlled by λKL.Source: B. Compact VLM World State · p. 3
- Compression acts only along the feature dimension, retaining the original spatial token layout; the main configuration compresses 108 tokens from 2560 channels to 16 channels per token.No spatial pooling or token fusion is performed.Source: B. Compact VLM World State · p. 3
C. Action-Conditioned Dynamics Model
Token-World forms its world state from compact visual tokens and robot proprioception, then models action-conditioned transitions with a spatiotemporal Transformer. A recurrent state carries interaction context beyond the finite temporal attention window.
C. Action-Conditioned Dynamics Model
- The compact world state combines compact visual tokens with the robot proprioceptive state.The transition model is action-conditioned.Source: C. Action-Conditioned Dynamics Model · p. 3
- The transition model projects compact visual tokens, proprioception, and actions into a shared space and uses spatial attention within timesteps and causal temporal attention across time.Factorizing the attention axes avoids dense attention over the full space-time sequence.Source: C. Action-Conditioned Dynamics Model · p. 3
- A fixed-size GRU state carries earlier interaction context that has fallen outside the active temporal-attention window across successive rollout windows.The recurrent state and Transformer are components of the transition model, not separate representation spaces.Source: C. Action-Conditioned Dynamics Model · p. 4
D. Training and Autoregressive Rollout
Token-World trains an action-conditioned transition model with a flow-matching objective, then rolls out predicted compact states autoregressively. A frozen semantic decoder returns policy-facing features, while a separate RGB decoder serves visualization and manual success assessment.
D. Training and Autoregressive Rollout
- The flow-matching objective trains the transition model to predict clean future compact states conditioned on interaction history.The history contains previous compact states, proprioception, actions, and recurrent context; the endpoint prediction induces a flow trained against a target flow.Source: D. Training and Autoregressive Rollout · p. 4
- At inference, each denoised future state is appended to the rollout history and conditions subsequent transitions.Temporal attention uses a bounded context window, while recurrent state carries interaction context across windows.Source: D. Training and Autoregressive Rollout · p. 4
- Predicted compact visual states are decoded into policy-facing features; a separate frozen RGB decoder is used only for rendering and manual success assessment.RGB reconstruction does not participate in world-model rollout or policy inference.Source: D. Training and Autoregressive Rollout · p. 4
IV. EXPERIMENTS
The experiments assess Token-World through open-loop prediction fidelity, closed-loop policy simulation, and controlled studies of compact-state representation design.
IV. EXPERIMENTS
- Open-loop comparisons with recent robotic world models assess prediction quality and its change over long rollout horizons.The evaluation also examines whether prediction gains translate into more reliable and efficient closed-loop policy simulation.Source: §IV · p. 4
- Closed-loop evaluation compares simulated and reference success rates to assess policy-simulation reliability and efficiency.The evaluation compares simulated and reference success rates.Source: §IV · p. 4
- Controlled ablations examine how compact S-VAE state design and dimensionality affect dynamics prediction.Source: §IV · p. 4
A. Experimental Setup
The evaluation uses simulated and real-world manipulation data, compares Token-World with three learned simulators, and measures open-loop fidelity, closed-loop policy agreement, and simulation latency.
A. Experimental Setup
- The simulated dataset covers 50 RoboTwin tasks with 5,000 trajectories, while the real-world dataset contains 600 successful trajectories across six tasks.Both datasets use a 9:1 train–validation split, and all compared methods share the same training and evaluation data; the real-world benchmark uses a Franka Research 3 arm, Robotiq gripper, and two RealSense 435 cameras, with 100 trajectories per task.Source: A. Experimental Setup · p. 4, Figure 3 · p. 5
- Token-World is compared with IRASim, Ctrl-World, and WorldGym, described as competitive learned simulators based on generative visual dynamics.Source: A. Experimental Setup · p. 4
- Open-loop evaluation measures VLM-feature fidelity with NMSE and cosine similarity, and policy-action consistency by comparing actions from predicted versus ground-truth representations.Metrics are averaged across trajectories within each task and then macro-averaged across tasks.Source: A. Experimental Setup · p. 4, Table I · p. 5
- Closed-loop evaluation feeds predicted states back to three StarVLA variants and compares simulated with reference success rates using Pearson correlation.Rollouts start from the same recorded state as their reference counterparts, and success is manually assessed from decoded RGB rollouts.Source: A. Experimental Setup · p. 4
- Batch-1 latency is measured on PPU-ZW810E accelerators, with RGB baselines using 50 denoising steps and Token-World using 16 steps with shortcut forcing.Timing includes RGB decoding and VLM re-encoding for RGB baselines, and compact-state prediction plus direct decoding to policy-facing VLM features for Token-World.Source: A. Experimental Setup · pp. 4–5
B. Open-Loop Prediction Fidelity
Token-World reports higher open-loop feature and policy-action fidelity than the compared baselines on RoboTwin, and better preservation of task-relevant configurations in qualitative long-horizon rollouts.
B. Open-Loop Prediction Fidelity
- 0.7714 feature cosine similarity and 0.4892 NMSE for Token-World on RoboTwin, versus 0.7372/0.5685 for IRASim and 0.7097/0.6437 for Ctrl-World.After policy inference, action cosine similarity/NMSE are 0.9439/0.1145 for Token-World, 0.9260/0.1533 for IRASim, and 0.9265/0.1463 for Ctrl-World.Source: B. Open-Loop Prediction Fidelity · p. 5
- Long-horizon similarity is compared at replay chunks 0, 5, 10, and 15, each containing 16 actions.Figure 5 reports chunk-wise Qwen-feature and action similarity.Source: B. Open-Loop Prediction Fidelity · p. 5
- In qualitative long open-loop rollouts, Token-World better preserves task-relevant object configurations and robot–object interactions, while Ctrl-World deviates from reference evolution at later steps.Token-World predictions are decoded to RGB for visualization; RGB reconstruction is not used during rollout.Source: B. Open-Loop Prediction Fidelity · p. 5
C. Closed-Loop Policy Evaluation
Token-World’s simulated success rates track reference performance more closely than Ctrl-World’s, while its open-loop predictions degrade more slowly over long rollouts. It also simulates each step faster than the compared models.
C. Closed-Loop Policy Evaluation
- 0.794 vs. 0.583: Token-World raises the Pearson correlation between simulated and reference success rates over Ctrl-World.The regression is closer to the oracle y = x.Source: C. Closed-Loop Policy Evaluation · p. 5
- In qualitative long-horizon rollouts, Token-World better preserves task-relevant object configurations and interaction progression than Ctrl-World.Predicted states are rendered to RGB only for visualization.Source: Figure 4 · p. 6
- Across replay chunks 0, 5, 10, and 15, Token-World degrades more slowly than Ctrl-World in chunk-wise VLM-feature and action similarity.Each chunk contains 16 steps.Source: Figure 5 · p. 6
- 0.359 s per step makes Token-World the fastest simulator, with 2.0×, 5.3×, and 6.1× speedups over IRASim, Ctrl-World, and WorldGym, respectively.Source: C. Closed-Loop Policy Evaluation · p. 6
D. Representation Design Ablations
The ablations show that compact-state design and dimensionality affect dynamics prediction: greater reconstruction fidelity does not necessarily mean better modelability, and d = 16 performs best among tested dimensions.
D. Representation Design Ablations
- From 8 to 128, NMSE falls 0.1026→0.0409 and Cos. rises 0.9389→0.9762, while SEC rises 0.3511→0.4740 and LDS falls 0.3100→0.1182.The caption summarizes the opposing reconstruction and modelability trends as compact-state dimension increases.Source: Table II · p. 6, D. Representation Design Ablations · p. 6
- d = 16 achieves the lowest feature NMSE (0.1517), highest feature cosine similarity (0.8306), and highest action cosine similarity (0.9641) among evaluated dimensions.Dimension 8 slightly degrades feature and action consistency; dimensions 32 and 48 yield worse future-feature prediction.Source: Table III · p. 7, D. Representation Design Ablations · p. 7
- S-VAE lowers feature/action NMSE from 0.7851/0.2180 to 0.4892/0.1145 and raises cosine similarity from 0.6375/0.8948 to 0.7714/0.9439 versus raw VLM features.The controlled comparison holds the dynamics backbone and training protocol fixed while changing the state representation and its input/output projections.Source: Table IV · p. 7, D. Representation Design Ablations · p. 7
V. CONCLUSION
Token-World models action-conditioned dynamics in a compact, policy-facing VLM token space. The paper reports improved fidelity, policy-evaluation agreement, and simulation efficiency in manipulation, while identifying its evaluation scope and representation analysis as open boundaries.
V. CONCLUSION
- Token-World learns action-conditioned dynamics in a compact policy-facing VLM token space, avoiding intermediate RGB generation while retaining compatibility with downstream VLA policies.Source: §V · p. 7
- Across simulated and real-world manipulation, Token-World improves open-loop feature and policy-action fidelity, stabilizes long-horizon rollouts, strengthens agreement with reference policy performance, and lowers latency than recent baselines.Source: §V · p. 7
- Evaluation focuses on manipulation tasks and policies sharing a VLM backbone; what makes a semantic representation suitable for world modeling remains uncharacterized systematically.The conclusion identifies extension across policy backbones, interaction data, and representation designs as future work.Source: §V · p. 7