Source-linked AI summary
iMaC: Translating Actions into Motion and Contact Images for Embodied World Models
Zhenyu Wu, Xiuwei Xu, Yukun Zhou, Yifan Li, Qiuping Deng, Xiaofeng Wang, Zheng Zhu, Bingyao Yu, Ziwei Wang, Jiwen Lu, Haibin Yan
TL;DR
Robot world models need scalable, visually grounded action conditioning and reliable policy evaluation without relying solely on costly hardware trials. iMaC translates future actions into motion and contact images, achieving the best task-averaged video-prediction metrics and strong policy-ranking alignment on most tasks.
Problem
Long-horizon embodied world models face exposure bias when closed-loop rollouts use imperfect generated observations instead of clean training references.
Method
iMaC conditions future-video prediction on URDF/FK-rendered motion images and pointcloud-based contact images, with training-time rollout across consecutive chunks.
Results
iMaC achieves the best task-averaged FID, PSNR, SSIM, and FVD, while policy-ranking correlations reach 0.833–0.956 on six of eight tasks.
Takeaways & Limitations
iMaC provides a learned real-world simulator for ranking policy checkpoints and reducing hardware trials, while complementing rather than eliminating real-world evaluation.
Takeaways & Limitations
iMaC relies on estimated depth from multi-view RGB observations, which can introduce centimeter-level errors affecting contact timing, collision localization, and long-horizon rollout reliability.
Abstract
from arXiv · showhide
Embodied world models have emerged as a pivotal paradigm for visual robotic decision-making and interactive environment simulation. However, conventional embodied frameworks rely on low-dimensional structured action vectors (e.g., joint angles and end-effector poses), which suffer from limited expressive capacity, poor generalization across diverse embodiments, and unnatural dynamic modeling for complex physical interactions. To address these limitations, this paper proposesiMac (Image as Action Control), a novel unified control paradigm that treats raw visual images as native action representations for embodied world models. Departing from traditional explicit kinematic action encoding, iMac formulates continuous visual manipulation as image-based action tokens, which inherently encapsulate spatial motion intentions, interactive geometric constraints and subtle physical dynamics. We construct a dual-branch embodied architecture consisting of an image-action encoder and a dynamic world predictor: the encoder compresses target-driven visual images into compact action embeddings, while the predictor learns environment transition rules conditioned on image actions to achieve high-fidelity future state prediction and closed-loop embodied control. Extensive experiments are conducted on public embodied manipulation benchmarks and real-world robotic scenarios. The results demonstrate that iMac outperforms vector-based action control baselines in prediction accuracy, task success rate and cross-scene generalization ability. Moreover, our image-action design eliminates the reliance on manually defined action spaces, realizing flexible and universal control for heterogeneous embodied agents. This work provides an innovative visual-action perspective for embodied world models, offering a simple yet effective paradigm for scalable robotic perception and manipulation.
1 Introduction
The introduction frames embodied world models as visual simulators for planning, control, and robot policy evaluation, but emphasizes that manipulation models must respond accurately to action-dependent contact and object motion. iMaC addresses this by translating future actions into dense image-like controls for future-video prediction, encoding both motion and its geometric relation to the observed scene.
- Motivation: Embodied world models support planning and control by predicting action consequences before real-world execution, with future states represented as visual observations.These models are also attractive for evaluating robot policies.
- Problem: A useful learned simulator must predict how different actions alter contact, object motion, and task outcomes, rather than merely generate realistic videos.Manipulation is especially demanding because small positional differences can determine whether a gripper touches, misses, or changes an object’s pose.
- Method: iMaC translates future actions into dense image-like controls that condition future-video prediction on an initial multi-view RGB observation and future action sequence.The action sequence specifies both where the robot moves and how that motion relates geometrically to the observed scene.
2 Related Work
Prior work uses video generation models for robotic data generation and embodied world-model forecasting, while policy evaluation spans costly real-world trials, simulators, benchmarks, and digital twins. Video world models provide a complementary in-silico evaluation approach through action-conditioned rollouts, which iMaC follows while targeting long-horizon action sensitivity.
- Video Generation Models for Robotics: Video generation models support cross-embodiment transfer, visual composition, missing-view synthesis, demonstration alignment, and synthetic data generation for robotics.These applications include 3D reconstruction and scene editing, as well as using video models as embodied world models for forecasting observations in planning and manipulation.
- Evaluation for Robotic Policies: Real-world trials provide authoritative policy evaluation but are expensive and difficult to reproduce across checkpoints or rare failure cases.This makes reliable policy evaluation a central bottleneck for robot learning.
- Evaluation for Robotic Policies: Physics simulators, manipulation benchmarks, and digital-twin systems improve evaluation repeatability and alignment but require assets, tuned dynamics, and careful scene construction.These requirements limit the simplicity of conventional in-silico evaluation environments.
- Evaluation for Robotic Policies: Video world models complement existing evaluators through action-conditioned rollouts for policy comparison, OOD and safety testing, and scores correlated with real-world performance.iMaC follows this direction while targeting the action sensitivity of long-horizon evaluation.
3 Approach
iMaC uses an action-conditioned IT2V world model that injects kinematics- and geometry-derived image controls to predict future RGB-D observations for closed-loop policy evaluation. Its training-time multi-chunk rollouts address the mismatch between clean training references and imperfect generated observations during inference.
- World model for policy evaluation: The world model predicts future observation chunks conditioned on policy-proposed actions, forming a closed loop that evaluates policy checkpoints without modifying the policy.The policy acts on generated observations while the world model predicts their visual consequences.
- IT2V world-model backbone: iMaC injects three action-derived control videos—motion, scene-to-gripper contact, and robot-to-scene contact—into the noised future tokens of a WAN2.2 image-to-video DiT.Reference tokens remain clean, while control-specific patchification and VAE encoding produce the injected controls.
- Two-stream Geometry Controls: Depth prediction supplies geometric state for contact controls, which encode bidirectional nearest distances between future robot or gripper pointclouds and the current scene pointcloud.The distances produce robot-centric Cr→s and scene-centric Cs→g contact images.
- Motion images: Motion images render future robot configurations from the action sequence using the robot URDF, forward kinematics, and the same three camera views as the predicted video.The resulting mosaic directly specifies future robot and gripper appearance in image space.
- Long-horizon policy evaluation: Training-time multi-chunk rollouts expose iMaC to generated references, matching inference where each predicted RGB-D-style frame becomes the next reference for newly constructed controls.This reduces the mismatch caused by training only on ground-truth reference frames.
4 Experiment
Experiments on eight real-world manipulation tasks evaluate iMaC for future-video prediction and closed-loop policy assessment. iMaC achieves the strongest aggregate prediction quality and generally aligns world-model policy scores with real-world performance, while ablations show complementary roles for motion and contact images.
- Tasks and data: iMaC is evaluated on eight real-world manipulation tasks using paired multi-view RGB videos and robot action trajectories from teleoperation and policy rollouts.Each observation combines one fixed head-camera view with two wrist-camera views, and the data include both successful and failed executions.
- World-model evaluation protocol: Closed-loop evaluation deploys policies inside the learned world model, where predicted observations become references for subsequent action and video-chunk prediction.The protocol evaluates π0.5 and GigaBrain-0.5 using three checkpoints from each policy family.
- Future-video prediction: iMaC obtains the best task-averaged FID, PSNR, SSIM, and FVD while matching the best MSE within rounding.These results indicate that motion and contact images improve action-conditioned future-video prediction by specifying future robot state and dense robot-scene distance cues.
- Closed-loop policy evaluation: Across six of eight tasks, world-model scores correlate strongly with real-world performance, with per-task correlations between 0.833 and 0.956.Task 3 and Task 5 have lower correlations of r = 0.678 and r = 0.428, respectively, due to the missing-observation failure mode.
- Ablation studies: Removing contact images prevents contact-aware action following, while removing motion images prevents precise future-configuration guidance; DA3 depth improves action following partially but yields less consistent contact geometry than iMaC’s RGB-D state.In the qualitative example, the gripper fails to grasp the cloth without contact images and misses the cloth corner when contact geometry comes from DA3 depth.
5 Limitation
iMaC depends on accurate 3D information for depth prediction and pointcloud-based contact images, while its current Depth Anything 3 supervision can introduce centimeter-level errors. Nevertheless, heatmap-based contact images remain useful by capturing coarse approaching and separating trends rather than exact metric contact.
- 3D information dependence: iMaC relies on accurate 3D information to train depth prediction and construct pointcloud-based contact images.This dependence is a limitation of the current system.
- Depth supervision: Depth supervision estimated by Depth Anything 3 from multi-view RGB observations and camera poses can introduce centimeter-level errors in manipulation scenes.The error arises from the current depth-supervision pipeline.
- Contact-image robustness: Two-stream contact images remain useful because their heatmaps over distance fields capture coarse approaching and separating trends without requiring exact metric contact.This allows the model to exploit approximate interaction dynamics despite depth inaccuracies.
6 Concluding Remark
iMaC translates future robot actions into motion and contact images for spatially explicit conditioning in embodied world models. Its design combines rendered robot controls, depth prediction, point-cloud contact images, and training-time rollout to guide future robot states and robot–scene geometry.
- 6 Concluding Remark: iMaC translates future robot actions into motion images and contact images for spatially explicit action conditioning.The paper presents this as its embodied world-model formulation.
- 6 Concluding Remark: The model combines URDF/FK-rendered robot controls, auxiliary depth prediction, two-stream pointcloud-based contact images, and training-time rollout.These components provide direct guidance about future robot state and robot-scene geometry.
- 6 Concluding Remark: iMaC gives an image-to-video model direct guidance about future robot state and robot-scene geometry.The conclusion reports evaluation across eight real-world manipulation tasks.
Appendix · A Task Suite
The appendix details an eight-task real-world manipulation suite for world-model-based policy evaluation, spanning varied initial configurations and complete closed-loop rollouts. The tasks test contact-sensitive prediction across rigid, articulated, granular, contact-rich, deformable, transparent, and precise-placement interactions.
- A Task Suite: The suite contains eight real-world manipulation tasks, visualized by representative initial and successful final observations.Initial conditions vary in object placement, object pose, and robot approach state; rollouts span the full manipulation sequence.
- A Task Suite: Tasks 1 and 2 require transporting a banana into a basket and placing a green bowl onto a pink plate despite distractors.They test container geometry, object identity, relative placement, and contact preservation.
- A Task Suite: Task 3 requires folding box ears into a slot, making small height and alignment differences critical to success.The world model must predict whether an ear clears the side wall and enters the slot, rather than merely appearing closed.
- A Task Suite: Task 4 combines opening, lifting, and tilting a chip box with predicting downstream granular motion onto a plate.Success requires transferring the chips from the box to the plate while forecasting both box pose and chip motion.
- A Task Suite: Task 5 tests contact-rich broom sweeping, where errors in broom pose, dustpan placement, or trash trajectory can alter the outcome.The target outcome is trash inside the dustpan.
- A Task Suite: Task 6 requires folding a spread-out shirt, testing prediction of large nonrigid cloth-shape changes that are only partly constrained by robot motion.Success is defined by reaching the desired compact folded configuration.
- A Task Suite: Tasks 7 and 8 test transparent-tape attachment to a metal box and precise panda placement inside a shallow pink plate.These tasks require subtle contact inference, small-object identity preservation, and release within the target boundary.
B Generated Rollout and Control Visualizations
Qualitative rollouts show that iMaC supports long-horizon closed-loop video generation aligned with motion and contact-image controls. Failure analysis identifies both general video-model limitations and errors caused by missing task-relevant physical relations.
- Generated Rollouts: Qualitative rollouts visualize long future-video generation under closed-loop chunk-wise rollout and alignment between generated videos and corresponding video controls.The visualizations cover normal generation behavior across tasks.
- Generated Rollouts: Each rollout presents reference observations, future RGB and depth predictions, URDF/FK-rendered motion images, and two contact-image streams.Figures 5 and 6 use this paired generated-video and video-control layout.
- Generated Rollouts: Additional rollouts show how differences in generated scene motion are supported by corresponding motion and contact images.Figure 6 retains the same generated-video and video-control layout.
- Failure Analysis: The world model can mispredict scene evolution when all available observations omit a task-relevant physical relation that determines task success.This failure is more directly tied to world-model-based policy evaluation than the general video-model limitations.
- Failure Analysis: Failures include low visual fidelity, accumulated temporal error, and insufficiently accurate action following during long closed-loop video generation.These are described as common limitations of current video models.