Source-linked AI summary
World Pilot: Steering Vision-Language-Action Models with World-Action Priors
Zefu Lin, Rongxu Cui, Junjia Xu, Xiaojuan Jin, Wenling Li, Lue Fan, Zhaoxiang Zhang
TL;DR
VLA policies lack an internal model of how scenes evolve under action, limiting manipulation robustness beyond training conditions. World Pilot routes World-Action Model priors into perception and action generation, achieving 84.7% Total success on LIBERO-Plus and the highest success rate in every real-robot setting.
Problem
VLA pretraining lacks scene-dynamics modeling, and effective integration of world-model signals into policy layers remains unresolved.
Method
World Pilot routes WAM scene-evolution latents through Latent Steering and anticipated trajectories through Action Steering.
Results
84.7% Total success on LIBERO-Plus and the highest success rate in every real-robot setting, with largest margins under viewpoint, geometry, deformable-state, and pose shifts.
Takeaways & Limitations
The scene-evolution prior remains effective even when supplied by a video-pretrained WAM without action post-training.
Takeaways & Limitations
World Pilot inherits WAM coverage limits, and its gains shrink when test scenes fall outside the WAM’s video-pretraining distribution.
Abstract
from arXiv · showhide
Vision-Language-Action (VLA) models inherit semantic grounding from large-scale pretraining and perform competently across in-distribution manipulation tasks. This grounding, however, is built on static image-text pairs, whereas manipulation is a continuous, contact-rich process whose dynamics this pretraining cannot capture. We present World Pilot, a VLA framework that augments the policy with priors from a World-Action Model (WAM), routed into the decision chain through two complementary pathways. Latent Steering conditions the perception layer on a scene-evolution latent, and Action Steering supplies an anticipated trajectory as a motion prior to the action generator. Together the two priors equip the VLA with an anticipated view of the scene and a trajectory-level motion hint alongside its semantic conditioning, and the scene-evolution prior remains effective even when supplied by a video-pretrained world model that has not been action-post-trained. World Pilot attains a state-of-the-art Total success rate of 84.7% on the LIBERO-Plus zero-shot OOD benchmark and the highest success rate on every real-robot setting across four manipulation tasks, with the largest margins under shifts in viewpoint, geometry, deformable state, and pose. Project Website: https://world-pilot.github.io/
1 Introduction
World Pilot augments VLA policies with World-Action Model priors through Latent Steering and Action Steering, addressing the missing scene-evolution and trajectory-level dynamics absent from static image-text pretraining. It achieves 84.7% Total success on LIBERO-Plus, leads every real-robot setting, and remains effective with a video-pretrained world model without action post-training.
- Motivation: Static image-text pretraining provides semantic grounding but no internal account of how manipulation scenes evolve under action.The downstream action generator consumes semantic hidden states without modeling the dynamics it must produce.
- Motivation: Video-pretrained World-Action Models provide scene-dynamics representations, including scene-evolution latents and anticipated trajectories, that complement what VLAs lack.These representations transfer broadly across embodiments and visual conditions.
- Method: World Pilot routes WAM outputs through two pathways: Latent Steering updates VLM hidden states at perception, while Action Steering supplies a trajectory prior to the action generator.Latent Steering uses residual cross-attention; Action Steering compresses the anticipated trajectory into a prior token for flow-matching generation.
- Analysis: Ablations show that each pathway contributes independently and that the chosen prior forms and entry points outperform alternatives tied to noisy intermediate outputs.The scene-evolution prior remains effective even when supplied by a video-pretrained world model that has not been action-post-trained.
- Results: 84.7% Total success is achieved on LIBERO-Plus, while World Pilot records the highest success rate in every real-robot setting and remains competitive on RoboCasa.The largest margins occur under shifts in viewpoint, geometry, deformable state, and pose.
2 Related Work
VLA policies combine vision-language backbones with action generators to produce continuous robot actions, but image-text pretraining alone does not capture manipulation dynamics. WAMs learn action-conditioned scene evolution and contact dynamics from video, motivating their combination with VLAs.
- Vision-Language-Action Models: VLAs attach action generators to Vision-Language Model backbones, producing continuous robot actions from visual observations and language instructions.Recent systems including π0.5, ABot-M0, and CogACT achieve competent in-distribution performance on standard manipulation benchmarks.
- Vision-Language-Action Models: Image-text pretraining alone provides no representation of manipulation dynamics.
- World-Action Models: WAMs such as Cosmos Policy, mimic-video, and DreamZero learn action-conditioned scene evolution and contact dynamics from large-scale video sequences.Their video-pretrained representations transfer broadly across embodiments and visual conditions.
- World-Action Models: Combining WAM-derived priors with a VLA is presented as a natural design question.
3 Method
World Pilot augments a VLA policy with a frozen video-pretrained World-Action Model whose scene-evolution latent and coarse action trajectory are routed through complementary steering pathways. Latent Steering enhances perception through residual cross-attention, while Action Steering supplies a single trajectory prior token to the flow-matching action generator.
- 3 Method: World Pilot jointly obtains a scene-evolution latent and coarse action-trajectory hypothesis from a shared WAM encoder, preserving structural alignment between predicted effects and actions.The WAM receives the same visual, language, and optional proprioceptive inputs as the policy.
- 3.2 Latent Steering: Latent Steering cross-attends VLM hidden states to a dynamics-enhanced future-scene representation and adds the result residually, preserving token order and the standard action-generation interface.The latent captures predicted object motion, contact outcomes, and local state changes while avoiding action-irrelevant pixel details.
- 3.3 Action Steering: Action Steering resamples the WAM trajectory to horizon K, encodes it as one prefix token, and conditions denoising through self-attention without replacing the generated action chunk.A single token summarizes the trajectory shape and leaves the generator free to produce a specific continuous chunk.
- 3.3 Action Steering: The encoded single-token Action Steering design attains the highest success rate among four trajectory-conditioning alternatives in Table 6.The single-token form is more robust than per-step conditioning when the WAM trajectory is approximate.
- Training: During fine-tuning, the WAM remains frozen and only VLA-side parameters are updated, so the priors enter through conditioning without a separate prior loss.The clean-action flow-matching objective retains the expert action chunk as its supervision target.
4 Experimental Results
World Pilot achieves leading performance on zero-shot LIBERO-Plus and every real-robot setting, with especially strong robustness to viewpoint, geometry, deformable-state, and pose shifts. Ablations show complementary benefits from Latent Steering and Action Steering, including useful priors from video-pretrained world modeling alone.
- Simulation benchmarks: LIBERO-Plus evaluates 10,030 zero-shot perturbed tasks across seven perturbation axes, while RoboCasa targets long-horizon manipulation in everyday kitchen scenes.Total averages success over all LIBERO-Plus perturbed tasks.
- Simulation results: 84.7% Total success on LIBERO-Plus is highest, exceeding the strongest reported baseline by 2.6 points.The result is averaged over three random seeds.
- Real-robot results: World Pilot leads every real-robot setting, with ID-to-OOD drops within 20 points versus 25–50 for other baselines.Under OOD pose and object changes in container-lid alignment, it succeeds in 13–14 of 20 trials while no baseline exceeds 6.
- Pathway ablations: 84.7% is achieved by combining Latent Steering and Action Steering, versus 83.7% and 83.1% for the individual pathways and 80.5% for ABot-M0.The two pathways provide complementary anticipated-scene and trajectory-level signals beyond semantic representation.
- World-model prior transfer: 82.6% on LIBERO-Plus, 62.7% on RoboCasa, and 85.3% on clean RoboTwin2.0 show that Cosmos-Predict’s world-model-only prior improves over ABot-M0 without action post-training.The Cosmos-Policy counterpart reaches 83.7% on LIBERO-Plus with Latent Steering alone, 1.1 points higher under matched settings.
- Action-prior ablations: 84.7% from a single trajectory-level token surpasses per-step tokens at 83.6%, flow-matching initialization at 84.1%, and raw trajectories at 83.0%.The alternatives propagate trajectory noise, compound errors, or constrain correction by the VLA generator.
5 Conclusion and Limitations
World Pilot augments VLA policy learning with World-Action Model priors through Latent Steering and Action Steering, achieving state-of-the-art LIBERO-Plus performance and the highest success rate across real-robot settings. Its gains shrink outside the WAM’s video-pretraining coverage, remain uneven across LIBERO-Plus axes, and reduce but do not eliminate real-robot OOD degradation.
- Conclusion: World Pilot routes World-Action Model priors through Latent Steering on perception and Action Steering on the action generator.This training recipe augments VLA policy learning with WAM priors.
- Conclusion: World Pilot attains state-of-the-art performance on LIBERO-Plus and the highest success rate in every real-robot setting.These results instantiate the proposed training recipe as World Pilot.
- Limitations: World Pilot’s priors degrade and gains shrink when test scenes fall outside the WAM’s video-pretraining distribution.World Pilot inherits the WAM’s coverage limitations.
- Limitations: World Pilot trails on LIBERO-Plus’s Language, Robot, and Layout axes, while real-robot OOD success drops by 10 to 20 points relative to ID.The priors reduce but do not eliminate the effects of OOD shifts.