Source-linked AI summary
GigaWorld-Policy-0.5: A Faster and Stronger WAM Empowered by AutoResearch
GigaWorld Team, Angen Ye, Angyuan Ma, Boyuan Wang, Chaojun Ni, Fangzheng Ye, Guan Huang, Guo Li, Guosheng Zhao, Haodong Yan, Hengtao Li, Jiwen Lu, Kai Wang, Mingming Yu, Qitang Hu, Qiuping Deng, Songling Liu, Xiaoyu Tian, Xiaofeng Wang, Xinyu Zhou, Xiuwei Xu, Xinze Chen, Yang Wang, Yejun Zeng, Yifan Chang, Yun Ye, Zhenyu Wu, Zhanqian Wu, Zheng Zhu
TL;DR
Existing World Action Models often require costly future-video generation during deployment, limiting real-time robot control. GigaWorld-Policy-0.5 uses action-centered training with specialized Transformer experts and automated configuration search, achieving stronger long-horizon performance and 85 ms inference latency with C++ deployment.
Problem
Existing World Action Models often generate future videos during inference, creating computational overhead that limits real-time closed-loop robot control.
Method
GigaWorld-Policy-0.5 combines mixed action-conditioned world modeling and WAM pretraining with specialized Mixture-of-Transformers experts and agent-based AutoResearch.
Results
GigaWorld-Policy-0.5 achieves the best performance on all three long-horizon tasks, with average success rate 0.80 and 0.20 absolute improvement over the strongest baseline.
Takeaways & Limitations
The action-centered formulation preserves future-visual-dynamics training benefits while enabling efficient action-only inference for robot control.
Abstract
from arXiv · showhide
World Action Models (WAMs) improve robot policy learning by jointly modeling actions and future visual observations, using future scene evolution as dense supervision for physically grounded action generation. However, a common design in existing WAMs is to explicitly generate future videos at inference time, incurring substantial computational overhead and hindering real-time closed-loop deployment. GigaWorld-Policy addresses this issue with an action-centered formulation, where future visual dynamics are used during training while action-only decoding is used at inference time. Building upon this framework, we present GigaWorld-Policy-0.5, an enhanced action-centered WAM designed for more efficient robot control. During pretraining, GigaWorld-Policy-0.5 adopts a mixed Action-Conditioned World Modeling (AC-WM) and WAM training strategy. This strengthens the coupling between visual dynamics and robot actions and improves the transferability of action representations for downstream policy learning. For efficient inference, GigaWorld-Policy-0.5 introduces a Mixture-of-Transformers architecture that separates visual dynamics modeling and action generation into specialized experts, reducing active computation during action-only inference and achieving 85 ms inference latency on a local RTX 4090 setup. In addition, we employ an agent-based AutoResearch pipeline to systematically search training configurations, enabling more efficient identification of optimal experimental setups while reducing the time and manual intervention required for hyperparameter tuning. Experiments and ablations show that GigaWorld-Policy-0.5 preserves the training benefits of future visual dynamics while improving inference efficiency for robot control.
1. Introduction
GigaWorld-Policy-0.5 is an action-centered World Action Model that uses future visual dynamics for dense training supervision while decoding actions without explicit future-video generation at inference. It further improves this framework with specialized Transformer experts, mixed action-conditioned and WAM training, and an agent-based AutoResearch pipeline.
- Motivation: World models provide predictive information about scene evolution that enriches robot policy learning beyond sparse action supervision.They capture priors about object motion, interaction dynamics, and long-horizon state transitions, providing temporal context for action prediction.
- World Action Models: WAMs jointly model robot actions and future observations, coupling action generation with future scene evolution within one framework.This unified formulation can support zero-shot generalization, but existing approaches often require future-video generation, iterative denoising, or predictive rollout during inference.
- Core formulation: GigaWorld-Policy-0.5 retains action-centered training with future visual dynamics and performs action-only inference without explicit future-video generation.Action tokens are predicted from the current observation, robot state, and language instruction within an action-centered causal token structure.
- Architecture and training: Its Mixture-of-Transformers architecture assigns visual dynamics modeling and action generation to specialized experts while preserving the action-centered WAM design.Expert specialization enables action-only inference with reduced active computation, while the model continues learning both general scene evolution and action-induced future changes.
- Architecture and training: The training strategy mixes Action-Conditioned World Modeling with WAM training to strengthen visual-dynamics and action coupling and improve action-representation transfer.This mixed strategy is presented as an enhancement over the prior GigaWorld-Policy framework.
- Experimental search: An agent-based AutoResearch pipeline automates pilot runs, validation monitoring, candidate comparison, and promotion of promising configurations to longer training.The pipeline makes experimental search more systematic and reduces manual intervention in identifying effective training configurations.
2. Related Work
Related work develops world models for predicting embodied-environment dynamics and increasingly couples future visual prediction with robot action generation. Recent action-centered approaches avoid explicit future-video generation at inference, but low-latency closed-loop control remains challenging.
- World Models: World models learn temporal embodied-environment dynamics to synthesize or imagine visual experiences from limited real-world or simulated trajectories.They have also become scalable data engines and learned simulators for robot learning.
- Action-Conditioned Prediction: Action-conditioned models extend world-model controllability from free-form text actions to continuous robot actions with motion-intensity guidance.Pandora enables interactive video prediction, while FreeAction introduces action-scaled classifier-free guidance.
- World Models for Robot Control: Most prior world models serve as external data engines or simulators, motivating policies that jointly model future prediction and action generation.This shift seeks to embed predictive priors into deployable robot policies rather than using world models only for data generation or simulation.
- World Action Models: World Action Models couple robot actions with future visual dynamics, providing dense temporal supervision and predictive priors beyond sparse demonstration actions.Representative systems include VideoVLA, Motus, UWM, and related video-action diffusion or multiview approaches.
- Efficient Inference: GigaWorld-Policy and Fast-WAM use video-based predictive supervision during training while avoiding explicit future-video generation at test time.These methods show that world-modeling benefits can be retained without computationally expensive pixel-level future prediction during execution.
- Efficient Inference: Low latency for closed-loop control on edge devices remains challenging, motivating GigaWorld-Policy-0.5’s action-centered formulation and lightweight action-only execution pathway.Prior designs span latent video plans, autoregressive video-action modeling, and distilled multi-step video foresight, but all reduce reliance on explicit pixel-level rollout to varying degrees.
3. Method
GigaWorld-Policy-0.5 is an action-centered World Action Model that jointly models action chunks and future visual observations during training. Its Mixture-of-Transformers architecture separates visual-dynamics modeling from action generation while preserving cross-modal, action-centered dependencies.
- Mixture-of-Transformers architecture: The MoT architecture assigns visual dynamics modeling to a visual expert and action generation to an action expert, connecting them through multi-modal self-attention.The visual expert processes current and future visual tokens, while the action expert models action tokens within a unified token sequence.
- Action-centered formulation: The model takes multi-view observations, proprioceptive state, language instructions, action chunks, and future visual observations as inputs or prediction targets.Composite observations concatenate left, front, and right camera views, with the front view positioned above the side views.
- Cross-modal causal structure: The action-centered causal mask lets future visual tokens attend to action tokens but prevents action tokens from attending to future visual tokens.This enables action-conditioned future visual prediction while preserving action-centered information flow for policy learning.
- Training pipeline: Training proceeds through robot-data pretraining and target-robot post-training, combining action prediction with future visual dynamics modeling.Pretraining uses 2K hours of filtered open-source and internally collected real-robot data, while post-training uses aligned target-robot trajectories.
- Efficient inference: Inference is optimized with KV caching, graph compilation, and a lightweight C++ runtime to avoid redundant attention computation during autoregressive action generation.Key-value states are cached after encoding the unchanged visual and language context and reused for subsequent action-token predictions.
4. Experiments
Experiments show that GigaWorld-Policy-0.5 improves text-following and long-horizon manipulation performance, while mixed AC-WM/WAM pretraining and MoT architecture improve policy learning and inference efficiency. AutoResearch further automates systematic hyperparameter exploration under a unified evaluation protocol.
- Text Following: 0.85 average success rate on fruit picking, outperforming 𝜋0.5, Motus, FastWAM, and GigaWorld-Policy by 0.09, 0.05, 0.07, and 0.05, respectively.The model achieves the highest success rate across all six fruit categories, with particularly clear gains on lemon and avocado.
- Long-Horizon Execution: 0.80 average success rate across three long-horizon tasks, with a 0.20 absolute and 33% relative improvement over the strongest baseline.GigaWorld-Policy-0.5 achieves the best performance on every task.
- Ablations: AutoResearch automates candidate generation, pilot training, validation collection, candidate selection, and extended training for learning-rate and warmup exploration.The study uses approximately 3.9 hours of demonstrations, with 930 episodes, 300 for training, 30 for validation, and 1K-step pilot runs.
- Ablations: 0.85 success rate after mixed AC-WM and WAM pretraining, with faster convergence and stronger performance throughout post-training than standard WAM pretraining.Conditioning future visual observations on robot actions yields more transferable action representations and earlier strong performance.
- Ablations: 189 ms inference latency on an A100 with KV cache and torch compilation, down from 229 ms for FastWAM and 225 ms for 𝜋0.5.The reduction from 229 ms to 189 ms corresponds to a 17.5% speedup over FastWAM under the same torch-compiled setting.
5. Conclusion
GigaWorld-Policy-0.5 is an enhanced action-centered World Action Model that retains future visual dynamics as training supervision while enabling action-only inference. Its Mixture-of-Transformers architecture separates visual dynamics modeling from action generation to reduce active computation for low-latency deployment.
- 5. Conclusion: GigaWorld-Policy-0.5 extends GigaWorld-Policy with an enhanced action-centered World Action Model formulation.The report presents it as being built upon GigaWorld-Policy.
- 5. Conclusion: Future visual dynamics remain training-time supervision, while inference uses action-only decoding.This preserves the model’s core training principle while avoiding future-video decoding at inference.
- 5. Conclusion: A Mixture-of-Transformers architecture separates visual dynamics modeling from action generation to reduce active computation for low-latency deployment.The architecture is introduced on top of the action-centered formulation.