Source-linked AI summary
Action Images: End-to-End Policy Learning via Multiview Video Generation
Haoyu Zhen, Zixian Gao, Qiao Sun, Yilin Zhao, Yuncong Yang, Yilun Du, Pengsheng Guo, Tsun-Hsuan Wang, Yi-Ling Qiao, Chuang Gan
TL;DR
World action models still struggle to turn strong video prediction into policy generalization, especially when actions use separate modules or non-pixel-grounded representations. Action Images formulates policy learning as multiview video generation by encoding 7-DoF actions as interpretable, pixel-grounded action images, allowing the video backbone to act as a zero-shot policy. The model also supports multiple video-action tasks under a shared representation, while the authors identify closed-loop deployment as an unresolved limitation.
Problem
World action models have not fully connected video prediction to policy generalization because existing action modules and non-pixel-grounded representations limit transfer across viewpoints and environments.
Method
Action Images translates 7-DoF robot actions into interpretable multiview action videos grounded in 2D pixels and models them jointly with observations in video space.
Results
The video backbone serves as a zero-shot policy without a separate policy head or action module, while the unified model supports video-action joint generation, action-conditioned video generation, and action labeling.
Takeaways & Limitations
Pixel-grounded action images provide a promising route toward more generalizable policy learning and robotics world modeling in a common video space.
Takeaways & Limitations
The current system demonstrates strong open-loop results but has not yet been fully developed into a closed-loop policy.
Abstract
from arXiv · showhide
World action models (WAMs) have emerged as a promising direction for robot policy learning, as they can leverage powerful video backbones to model the future states. However, existing approaches often rely on separate action modules, or use action representations that are not pixel-grounded, making it difficult to fully exploit the pretrained knowledge of video models and limiting transfer across viewpoints and environments. In this work, we present Action Images, a unified world action model that formulates policy learning as multiview video generation. Instead of encoding control as low-dimensional tokens, we translate 7-DoF robot actions into interpretable action images: multi-view action videos that are grounded in 2D pixels and explicitly track robot-arm motion. This pixel-grounded action representation allows the video backbone itself to act as a zero-shot policy, without a separate policy head or action module. Beyond control, the same unified model supports video-action joint generation, action-conditioned video generation, and action labeling under a shared representation. On RLBench and real-world evaluations, our model achieves the strongest zero-shot success rates and improves video-action joint generation quality over prior video-space world models, suggesting that interpretable action images are a promising route to policy learning.
1 Introduction
Action Images addresses the gap between video generalization and policy generalization by representing robot control as pixel-grounded, multiview action videos. This unified representation lets one video backbone model and generate both observations and actions without a separate policy module.
- Motivation: Existing world action models can generate plausible future frames yet still fail to generalize policies to unseen environments.The paper identifies the disconnect between predictive video quality and reliable action selection as a central bottleneck.
- Action representation: Action Images converts 7-DoF robot control into interpretable action images that explicitly track robot-arm motion across multiple views.The representation is grounded in image pixels rather than low-dimensional signals or latent action codes.
- Action representation: Multi-view action videos bridge 2D image observations and 3D robot motion while reducing ambiguity from single-view projections and partial occlusions.Multiple views make the action representation more reconstructable and robust when motion is partially hidden.
- Unified policy learning: The unified representation supports video-action joint generation, action-conditioned video generation, and action labeling within a single model.These capabilities share one video-space representation of observations and actions and one training objective.
- Unified policy learning: The video backbone itself serves as a zero-shot policy, eliminating the need for a separate policy head or action module.This connects acting directly to the model’s visual representation rather than delegating control decoding to a specialized module.
2 Related Work
Prior robotics world models use video generation for future-state prediction and increasingly unify video and action generation, but often retain separate action modules or lack broad spatial grounding. Related 4D-generation work uses multiview generation for dynamic scenes, while robotic applications commonly remain limited to fixed single views or training scenes.
- Robotics World Models: Video-based robotics world models commonly predict future observations and then generate actions, while newer approaches jointly model video and action.DreamZero is cited as demonstrating strong zero-shot generalization and cross-embodiment transfer, but uses additional action modules.
- Robotics World Models: Existing methods often leave pretrained video knowledge underused by encoding actions through additional action modules rather than pixel-grounded multiview images.Action Images is positioned as using the video backbone itself as a zero-shot policy.
- 4D Generation Models: 4D-generation research spans optimization-based, native, and reconstruction-based approaches, with recent work using multiview generation for replayable dynamic scenes.The passage defines 4D as 3D plus time and distinguishes major generation strategies.
- 4D Generation Models: For robotic tasks, 4D generation is typically limited to a fixed single view, and some multiview methods still fail to generalize beyond their training scenes.These limitations distinguish robotic 4D generation from broader multiview dynamic-scene generation.
3 Method
The method converts 7-DoF robot actions into multi-view, pixel-grounded action videos, decodes them back into continuous control, and trains a unified video model for zero-shot policy learning.
- Action as Images: 7-DoF actions become three semantic 3D points—position, normal, and up—which are projected into each camera view and rendered as RGB Gaussian heatmaps.The red, green, and blue channels encode the three points, while gripper openness is added to low-response blue-channel regions.
- Action as Images: The three projected points capture end-effector pose, with orientation-derived normal and up directions defined using a rotation matrix and canonical axes.The position point is the end-effector position; the other points are offset by a small length ℓ along rotated canonical directions.
- Action as Images: Stacking rendered action images over time produces multi-view action videos with the same spatial and temporal structure as robot RGB observations.This shared structure forms a unified video-space representation of observation and action compatible with the same video backbone.
- Action Images Decoding: The decoder reads gripper openness from the blue channel, anchors each heatmap in the main view, and uses ray casting plus side-view matching to reconstruct 3D semantic points.The best-matching candidate along each main-view ray resolves depth ambiguity before recovering the original action vector.
- Training Unified World Action Model: A fine-tuned pretrained video generator jointly models multi-view robot and action videos under diverse masking, text, and camera conditions.The masked-token objective learns coupled visual observation and action dynamics across multiple views.
4 Experiments
Experiments evaluate Action Images in zero-shot, in-domain, joint-generation, action-conditioned generation, and video-to-action settings. The model achieves strong zero-shot generalization, remains competitive in-domain, and performs well across unified-model capabilities.
- Zero-shot policy results: The model delivers the best overall zero-shot task success across RLBench and real-world settings, with the largest improvement under strong distribution shift.RLBench tasks use seen robots and environments but remove evaluated tasks from training; real-world objects, environments, and the xArm are unseen.
- RLBench in-domain results: The model remains competitive on in-domain RLBench tasks, while an optional action head provides substantial gains on precision-sensitive tasks.The optional head directly regresses continuous 7-DoF action sequences and is not required for the main zero-shot policy claim.
- Joint generation quality: The method outperforms prior world-model baselines on all video metrics while maintaining action accuracy in video-and-action joint generation.Video quality is evaluated with PSNR, SSIM, FVD, and LPIPS; action quality uses 2D and 3D trajectory error.
- Action-conditioned video generation: The method achieves better action-conditioned video-generation results than Tora on PSNR, SSIM, FVD, and LPIPS.This evaluates future robot-video generation when the action sequence is given.
- Video-to-action labeling: The model outperforms TAPIR and CoTracker3 on trajectory error, Jaccard@4, and average Jaccard for video-to-action labeling.The result indicates that the pixel-grounded action representation supports action inference from input videos in addition to control and generation.
- Additional generalization: In unseen FR3M rooms without BridgeV2 action supervision during training, the model generates coherent action images and more accurately localizes targets than LTX-2-Fast.This evaluates transfer across datasets and domains.
5 Conclusion
Action Images unifies observation and action in video space by translating 7-DoF control into pixel-grounded multiview action images. The model supports zero-shot policy use and multiple generation tasks, but remains an open-loop system rather than a fully developed closed-loop policy.
- Action Images translates 7-DoF robot control into interpretable, pixel-grounded multiview action videos.The action images explicitly track robot-arm motion in image space.
- The video backbone serves as a zero-shot policy without a separate policy head or action module.
- Real-world rollouts generalize to unseen objects and environments while producing executable action predictions consistent with generated visual outcomes.
- The shared generative framework supports video-action joint generation, action-conditioned video generation, and action labeling.
- The current system demonstrates strong open-loop results but has not yet been fully developed into a closed-loop policy.The authors propose faster inference and integration into a closed-loop control pipeline as future work.
- The model demonstrates zero-shot results in an unseen object, task, and environment, with predicted trajectories aligned to scene geometry.
1 Implementation Details
The unified world-action model is fine-tuned from a pretrained video backbone on mixed robotics datasets and uses multiview action-image construction. Inference preserves the training configuration and uses classifier-free guidance, denoising steps, and multi-GPU sequence parallelism.
- The model is fine-tuned from a pretrained Wan2.1-I2V-14B-480P backbone using Bridge, RLBench, and DROID data.The datasets are sampled with mixture ratios of 0.2, 0.5, and 0.3, respectively.
- Full two-view training samples contain 164 frames combining robot videos and action videos.Each single-view, single-modality sample contains 41 frames.
- Camera conditioning uses Plücker embeddings, with encoder pooling and projection into the model hidden dimension.Zero-initialized encoder projection and identity final projection stabilize optimization at the beginning of training.
- Inference uses classifier-free guidance at scale 10.0 with 50 denoising steps and 4-GPU Unified Sequence Parallelism.
- Each action is converted into position, normal, and up 3D points, projected into image space, and rasterized as Gaussian heatmaps.The normal and up points are placed 0.1 from the position point; the heatmap standard deviation is σ = 0.05 relative to image resolution.
- Table 7 reports inference efficiency.
2 More Zero-shot Qualitative Results
Additional qualitative evaluations cover action labeling, robot manipulation, camera control, and action-conditioned video generation. The reported examples span robot and human-hand videos, diverse grasping scenes, complex viewpoint changes, and future-video prediction conditioned on actions.
- Action labeling: Action labeling is demonstrated on both a π0 robot video and a Genie 3 human-hand video.
- Robot manipulation: Additional robot manipulation results mainly evaluate grasping across diverse objects and scenes.
- Camera control: Camera control results generate videos with controlled viewpoint changes in complex scenes from the Pi0 website.
- Action-conditioned generation: Action-conditioned generation uses the first frame from π0 demonstration videos to generate future videos conditioned on actions.