Source-linked AI summary
AIM: Intent-Aware Unified world action Modeling with Spatial Value Maps
Liaoyuan Fan, Zetian Xu, Chen Cao, Wenyao Zhang, Mingqi Yuan, Jiayu Chen
TL;DR
Unified world action models need substantial robot-domain adaptation because future visual representations do not explicitly encode where and why to interact. AIM introduces an intent-aware spatial value-map interface, jointly models future observations and value maps, and further optimizes the action head with self-distillation reinforcement learning. On RoboTwin 2.0, it achieves strong performance, including 94.0% average success in the Easy setting, while the study is conducted in simulation.
Problem
Unified world action models still require substantial robot-domain adaptation because future visual representations do not explicitly encode the spatial interaction and manipulation intent needed for control.
Method
AIM jointly predicts future frames and aligned spatial value maps in a shared architecture, routes future information to actions through the value representation, and self-distills reinforcement learning into the action head.
Results
94.0% average success rate under RoboTwin 2.0 Easy and 92.1% under Hard, with AIM outperforming external baselines.
Takeaways & Limitations
Explicit spatial value modeling provides a practical bridge between RGB world modeling and robot action generation, with gains appearing especially on contact-sensitive and stage-dependent tasks.
Takeaways & Limitations
The study uses a simulation-only setting to obtain scalable training, automatic labels, precise geometry, and controllable scene diversity.
Abstract
from arXiv · showhide
Pretrained video generation models provide strong priors for robot control, but existing unified world action models still struggle to decode reliable actions without substantial robot-specific training. We attribute this limitation to a structural mismatch: while video models capture how scenes evolve, action generation requires explicit reasoning about where to interact and the underlying manipulation intent. We introduce AIM, an intent-aware unified world action model that bridges this gap via an explicit spatial interface. Instead of decoding actions directly from future visual representations, AIM predicts an aligned spatial value map that encodes task-relevant interaction structure, enabling a control-oriented abstraction of future dynamics. Built on a pretrained video generation model, AIM jointly models future observations and value maps within a shared mixture-of-transformers architecture. It employs intent-causal attention to route future information to the action branch exclusively through the value representation. We further propose a self-distillation reinforcement learning stage that freezes the video and value branches and optimizes only the action head using dense rewards derived from projected value-map responses together with sparse task-level signals. To support training and evaluation, we construct a simulation dataset of 30K manipulation trajectories with synchronized multi-view observations, actions, and value-map annotations. Experiments on RoboTwin 2.0 benchmark show that AIM achieves a 94.0% average success rate, significantly outperforming prior unified world action baselines. Notably, the improvement is more pronounced in long-horizon and contact-sensitive manipulation tasks, demonstrating the effectiveness of explicit spatial-intent modeling as a bridge between visual world modeling and robot control.
1 Introduction
AIM addresses a structural gap in unified world action models: future visual representations describe scene evolution but do not explicitly encode where and why the robot should interact. It introduces a spatial value-map interface, intent-causal routing, self-distillation reinforcement learning, and a 30K-trajectory dataset, achieving strong RoboTwin 2.0 performance.
- Unified world action models still require substantial robot-domain adaptation to decode reliable actions.
- Future RGB representations describe scene appearance but leave spatial interaction and manipulation intent implicit for control.Dense future visual representations can be dominated by appearance details irrelevant to the next motor command.
- AIM predicts aligned spatial value maps between future prediction and action generation, encoding task-relevant interaction structure in a control-oriented form.Intent-causal attention restricts the action branch's access to future information through the predicted value map.
- AIM adds self-distillation reinforcement learning that updates only the action head using projected value-map rewards and sparse task-level rewards.
- 30K manipulation trajectories with synchronized multi-view videos, actions, and value-map annotations support training and evaluation.On RoboTwin 2.0, AIM reports 94.0% and 92.1% average success rates under Easy and Hard settings.
2 Related Work
AIM builds on pretrained video-generation priors and unified world-action modeling while adding spatially grounded value prediction directly into the generative architecture. This distinguishes it from approaches that use spatial representations only as standalone policy or perception components.
- Pretrained video-generation models provide transferable priors over motion and long-horizon scene evolution for robot learning.AIM adopts Wan2.2-TI2V-5B as its video-generation backbone.
- Unified world-action models jointly represent future observations and actions within a shared architecture.AIM differs by introducing an explicit spatial intermediate interface between future prediction and action generation.
- Spatially grounded representations localize where interaction should occur in robotic manipulation.Prior work studies actionable regions and voxel-aligned action spaces for object-centric and contact-rich tasks.
- AIM integrates spatial value prediction into a unified generative world-action model rather than using it as a standalone policy head or auxiliary perception module.
3 Overview
AIM predicts future RGB frames, aligned spatial value maps, and actions from synchronized observations, action history, and task instructions. Its factorization makes the value map the future-facing interface for control, within a mixture-of-transformers architecture and a two-stage training framework.
- AIM models manipulation trajectories as synchronized multi-view observations and robot actions, predicting future RGB frames, value maps, and actions.Each value map is spatially aligned with its corresponding RGB frame and highlights task-relevant interaction regions.
- The predictive distribution factorizes future visual states and value maps separately from actions conditioned on the predicted value maps.This design decodes control from the spatial interface rather than raw future RGB representations.
- AIM uses a mixture-of-transformers architecture with a video-generation branch and a narrower action-denoising head sharing self-attention but retaining separate feed-forward transformations.
- The video branch jointly denoises RGB and value-map tokens, while the action head denoises continuous control vectors under masked interactions among token streams.
- Training first jointly learns future frames, value maps, and actions, then applies reinforcement learning with sparse and dense rewards.The dense reward is derived from projected spatial value responses in the second stage.
4 Method
AIM inserts an explicit spatial value-map interface between future prediction and action generation, using intent-causal attention to route future information to actions through value representations. Its training combines joint RGB/value/action modeling with action-head-only self-distillation reinforcement learning.
- Spatial interface: AIM adds an explicit value-map pathway so action decoding uses task-relevant interaction structure instead of inferring inverse-dynamics cues directly from dense RGB futures.The value stream is aligned with visual tokens and provides a control-oriented intermediate representation.
- Unified architecture: AIM jointly models future RGB frames, spatial value maps, and continuous dual-arm actions within a shared mixture-of-transformers architecture.The architecture preserves stream-specific projections while coupling streams through shared masked attention.
- Training objectives: The training objective combines RGB flow-matching, value-map flow-matching, and inverse-dynamics losses for the action head.The three losses supervise future RGB denoising, future value-map denoising, and action prediction, respectively.
- Intent-causal attention: Intent-causal attention lets video tokens use instructions and history, value tokens use future video tokens, and action tokens use future value tokens but not future RGB tokens.This routing makes task semantics enter action generation through the shared world and value representations rather than direct language conditioning.
- Post-training: Self-distillation RL freezes the video and value branches and optimizes only the action head using projected value-map responses alongside task-success rewards.The dense reward encourages projected action targets to land on high-value interaction regions predicted by the frozen value head.
5 Dataset and Value-Map Annotation
AIM’s simulation dataset contains synchronized manipulation trajectories and supports spatial value-map annotations generated from task-relevant contact geometry. The simulation setting provides scale, controllable diversity, and precise geometric information for consistent annotation.
- 30K manipulation trajectories contain synchronized multi-view videos, action sequences, task identifiers, and per-step value-map annotations.
- Simulation provides training scale while enabling automatic label generation from contact events and physics states.
- Value-Map Annotation: For pick tasks, projected gripper-contact points are smoothed into heat maps marking grasp affordance regions.
- Value-Map Annotation: For place tasks, projected object-support contact regions become feasibility heat maps marking where placement contact should occur.
6 Experiments
AIM is evaluated on 50 RoboTwin simulation tasks under Easy and Hard settings using common training data, task definitions, and success criteria. It achieves high average success rates, with especially strong gains on contact-sensitive tasks where interaction localization matters.
- Experimental Setup: The benchmark covers 50 RoboTwin simulation tasks under Easy and Hard settings with identical task definitions and success criteria.
- Main Results: 94.0% and 92.1% average SR under Easy and Hard settings outperform all external baselines.
- Main Results: +5.3%/+5.0% average SR over Motus and +11.3%/+15.3% over π0.5 under Easy/Hard settings, respectively.
- Task Analysis: Place Mouse Pad, Scan Object, and Turn Switch show strong performance on contact-sensitive and stage-dependent manipulation tasks.Their reported Easy/Hard SRs are 97%/95%, 100%/98%, and 100%/98%, respectively.
- Experimental Setup: Figure 3 contrasts representative task execution processes in Easy and Hard settings across five manipulation tasks.
- Qualitative Analysis: Value maps concentrate on meaningful interaction regions, and projected action targets fall within corresponding high-value areas.
7 Conclusion
AIM bridges future video generation and robot action decoding through explicit spatial value maps. The framework combines joint future prediction, intent-causal routing, and action-head-only self-distillation RL, supported by a 30K dataset and RoboTwin evaluation.
- AIM jointly predicts future frames, value maps, and actions while routing future information to the action head through the value representation.
- Self-distillation RL freezes the video and value branches and updates only the action head using projected value-map rewards.
- A 30K simulation dataset and evaluation on 50 RoboTwin 2.0 tasks support spatially grounded world-action modeling.
- AIM consistently outperforms prior external baselines, supporting explicit spatial value modeling as a bridge between RGB world modeling and robot action generation.