Source-linked AI summary
EnerVerse-AC: Envisioning Embodied Environments with Action Condition
Yuxin Jiang, Shengcong Chen, Siyuan Huang, Liliang Chen, Pengfei Zhou, Yue Liao, Xindong He, Chiming Liu, Hongsheng Li, Maoqing Yao, Guanghui Ren
TL;DR
Robotic policy evaluation is costly because it requires interaction with dynamic environments, and existing video world models do not simulate responses to agent actions. EVAC generates action-conditioned observations, expands training data with failure trajectories, and supports policy-data augmentation and evaluation. The reported relative performance trends across tasks were consistent between EVAC and real-world evaluations.
Problem
Robotic policy testing is costly and challenging because it requires real-time interaction with dynamic environments, while existing video world models do not simulate environment responses to agent actions.
Source: §1 · p. 1Method
EVAC is an action-conditional world model that generates future visual observations from predicted actions, using multi-level action conditioning and camera ray maps for multi-view generation.
Source: §1 · p. 2Results
Relative performance trends across tasks were consistent between EVAC and real-world evaluations.
Source: §4.3 · p. 7Takeaways & Limitations
EVAC can augment human-collected trajectories and provide action-conditioned policy evaluations with reduced reliance on real robot hardware during development.
Source: §1 · p. 2Potential applications, including integration with actor-critic methods for reinforcement learning, remain unexplored.
Source: §6 · p. 9
Abstract
from arXiv · showhide
Robotic imitation learning has advanced from solving static tasks to addressing dynamic interaction scenarios, but testing and evaluation remain costly and challenging due to the need for real-time interaction with dynamic environments. We propose EnerVerse-AC (EVAC), an action-conditional world model that generates future visual observations based on an agent's predicted actions, enabling realistic and controllable robotic inference. Building on prior architectures, EVAC introduces a multi-level action-conditioning mechanism and ray map encoding for dynamic multi-view image generation while expanding training data with diverse failure trajectories to improve generalization. As both a data engine and evaluator, EVAC augments human-collected trajectories into diverse datasets and generates realistic, action-conditioned video observations for policy testing, eliminating the need for physical robots or complex simulations. This approach significantly reduces costs while maintaining high fidelity in robotic manipulation evaluation. Extensive experiments validate the effectiveness of our method. Code, checkpoints, and datasets can be found at <https://annaj2178.github.io/EnerverseAC.github.io>.
1 Introduction
Robotic policy evaluation is costly because it requires interaction with physical robots or large-scale simulations, while existing video world models do not simulate dynamics in response to agent actions. EVAC generates action-conditioned, multi-view observations and serves as both a data engine and policy evaluator.
1 Introduction
- EVAC generates future visual observations directly conditioned on an agent’s predicted actions, addressing the gap in world models that do not simulate responses to actions.The framework supports static head-camera and dynamic wrist-camera views, with memory for long-term video generation.Source: §1 · pp. 1–2, Figure 1 · p. 2
- EVAC’s training data includes diverse failure trajectories alongside Agibot-World data, expanding coverage and improving generalization across scenarios.Source: §1 · p. 2
- EVAC augments human-collected trajectories into diverse datasets and generates action-conditioned observations for policy testing, reducing reliance on robot hardware and complex simulations.Human evaluators or automated systems such as Video-MLLMs can review the generated videos.Source: §1 · p. 2
2 Related Work
Prior work explores generative video world models, robotic video generation, physical simulation, and imitation learning. The cited approaches leave open challenges around continuous action-conditioned video, simulation scalability, and reliance on large human-collected datasets.
Video Generation Model as World Model
- Video generation has been proposed for world models [4], but one cited universal model generates only next-step frames rather than continuous video sequences [5].Source: §2 · pp. 2–3
- Robotic video-generation work predicts future frames from textual and visual inputs [10] [1], while action-conditioned generation has received limited attention in robotics [11].The passage notes that gesture-conditioned approaches [11] had not been tested in robotics, where environments and object interactions are more complex.Source: §2 · p. 3
Physical Simulators for Robotics
3 Method
EVAC uses a UNet-based video-generation model to predict future multi-view frames from observed video and robot actions. It injects action information at multiple levels, encodes moving-camera geometry with ray maps, and supports trajectory augmentation and policy evaluation.
3 Method
- EVAC predicts future frames from observed video and robotic actions using a UNet-based latent-diffusion video-generation model.The action trajectory represents end-effector pose, with seven dimensions in the single-arm case and fourteen in the bi-arm case.Source: §3 · p. 3
3.1 Mutli-Level Action Condition Injection
- Spatial-aware pose injection projects end-effector position and orientation into image-space action maps, which are encoded and combined with RGB features.The action maps also encode gripper openness and distinguish left- and right-hand poses.Source: §3.1 · p. 4, Figure 2 · p. 4
- Delta action attention encodes changes between consecutive poses into latent tokens and fuses them with reference-image features through cross-attention.The module captures temporal changes such as speed and acceleration.Source: §3.1 · p. 4, Figure 2 · p. 4
3.2 Multi-View Condition Injection
- For multi-view generation, spatial cross-attention combines views, while camera ray maps encode motion and add spatial context to trajectory maps.Ray maps help distinguish wrist-camera movement when end-effector projections appear nearly identical across views.Source: §3.2 · pp. 4–5, Figure 3 · p. 5
3.3 Applications
- For data augmentation, EVAC segments collected trajectories into fetching, grasping, and homing phases, then generates new sequences from spatially augmented actions.The new action trajectories are interpolated, passed with observations to EVAC, and reordered into a dataset.Source: §3.3 · p. 5
- For policy evaluation, EVAC iteratively generates observations from policy action chunks until a threshold is reached, after which evaluators assess task success.Experiments report high correlation between EVAC-based and real-world evaluation results.Source: §3.3 · pp. 5–6
4 Experiments
Across the experiments, EVAC’s evaluations tracked real-world performance trends, generated stable action-conditioned videos within a bounded horizon, and supported policy-data augmentation. Failure trajectories helped the model distinguish unsuccessful grasps from successful ones.
4.1 Experiment Details
- EVAC training used AgiBot World data plus mined and collected failure cases; multi-view training took about 32 A100 GPUs for 8 days.The source dataset contains over 210 tasks and 1 million trajectories; single-view training took about 32 A100 GPUs for 2 days.Source: §4.1 · p. 6
4.2 Controllable Manipulation Video Generation
- EVAC synthesized realistic complex robot-object interactions while maintaining visual fidelity and following input action trajectories.Figure 5 presents qualitative multi-view video-generation results.Source: §4.2 · p. 6, Figure 5 · p. 7
- Generated videos remained sharp for up to 30 consecutive single-view chunks and 10 multi-view chunks, while longer sequences developed artifacts and blur.Figure 6 illustrates scene consistency across inference chunks.Source: §4.2 · p. 6, Figure 6 · p. 7
4.3 EVAC as Policy Evaluator
- EVAC matched relative real-world success-rate trends across four manipulation tasks, despite minor differences in absolute rates.Three independent evaluators judged success or failure from real executions or EVAC-generated sequences.Source: §4.3 · pp. 6–7
- EVAC and real-world tests both showed improving success rates as training steps increased for the Take a Bottle policy.The policy was evaluated at three training steps.Source: §4.3 · p. 7, Figure 7 · p. 7
4.4 EVAC as Data Engine
- Success rate rose from 0.28 to 0.36 when 30% EVAC-generated trajectories were added to 20 expert episodes.The comparison used the same 20 expert episodes in both training setups.Source: §4.4 · p. 8, Table 1 · p. 8
4.5 Further Analysis
- Without failure trajectories, EVAC tended to hallucinate a successful bottle grasp when no bottle was present; with them, it recognized the failed attempt.The test scenario involved the robot arm pretending to grasp an absent bottle.Source: §4.5 · p. 8, Figure 8 · p. 8
5 Conclusion
EVAC is an action-conditioned embodied world model that uses multi-level action conditioning and camera ray maps to model dynamic camera motion. Experiments demonstrate its dual role as a policy-learning data engine and an evaluator of trained policies.
5 Conclusion
- EVAC is an embodied world model with action-conditioned capabilities.The conclusion identifies multi-level action-condition injection and camera ray maps for modeling dynamic camera motion.Source: §5 · p. 8
- EVAC uses multi-level action-condition injection strategies and camera ray maps to model dynamic camera motion.Source: §5 · p. 8
- Experiments demonstrate EVAC’s dual functionality as a data engine for policy learning and an evaluator for trained policies.Source: §5 · p. 8
6 Limitations and Discussions
The discussion identifies limits in end-effector generality, multi-view inference efficiency, and explored applications of ENERVERSE-AC.
6 Limitations and Discussions
- The unit-circle, color-intensity representation of gripper openness may not generalize effectively to dexterous hands.Adapting the framework to different hardware configurations requires additional preprocessing and refinement.Source: §6 · p. 9
- Background noise in wrist-camera footage restricts multi-view inference to 10 chunks, versus 30 chunks for single-view generation, reducing multi-view efficiency.The cited noise includes people moving around the robot workspace.Source: §6 · p. 9
- Potential applications remain unexplored, including integration of the action-conditioned world model with actor-critic methods for reinforcement learning.The authors identify these applications as directions for future research.Source: §6 · p. 9
A Appendix
The appendix outlines additional training and model details, ablations, quantitative experiments, and EVAC application setups.
A Appendix
- Section A.1 provides additional training details and model parameters.The appendix directs readers to the project website for further details and videos with clearer indications.Source: A Appendix · p. 12
- Section A.2 presents an ablation study of the proposed delta action attention module.Source: A Appendix · p. 12
- The appendix includes more quantitative results from real-world and simulator experiments and demonstrates EVAC as a policy evaluator and data engine.Source: A Appendix · p. 12
A.1 Training Details
EVAC uses a UNet-based denoising model with concatenated visual and action-related conditions; Table 2 covers architecture and training hyperparameters.
A.1 Training Details
- The UNet-based denoising model receives latent images and concatenated conditions, including repeated condition-frame features, action maps, ray maps, and a dropout mask.The dropout strategy is intended to improve model robustness.Source: §A.1 · p. 12
- Table 2 lists the model architecture hyperparameters and training setup.Its caption notes that F indicates the fisheye camera.Source: §A.1 · p. 12, Table 2 · p. 12
A.2 More Ablations
The ablation examines whether the Delta Action Module improves action-following consistency in pan manipulation, especially for motions with different acceleration profiles.
A.2 More Ablations
- Without the module, spatial-aware action recognition models often confuse upward tossing and upward shaking, causing flicker or sudden object disappearance.The ablation task also includes rapidly and slowly shaking the pan.Source: §A.2 · p. 13
- Acceleration-aware action decomposition lets the module distinguish high-acceleration tossing from smoother, low-acceleration shaking.It explicitly models the time-derivative of actions to capture second-order dynamics, described as velocity changes.Source: §A.2 · p. 13
- The Delta Action Module yields stronger motion consistency and reduces temporal errors compared with configurations without it.Figure 9 compares generated videos with and without the module; dashed red boxes mark inconsistent or hallucinated regions.Source: Figure 9 · p. 13, §A.2 · p. 13
A.3.1 More Real-World Generated Multi-View Results
The appendix presents qualitative multi-view and trajectory-conditioned outputs, then describes policy-evaluation examples and synthetic-data augmentation. Examples cover synchronized packaging views, Libero cases, differing trajectories, real-versus-EVAC task rollouts, and generated augmentation data.
A.3.1 More Real-World Generated Multi-View Results
- For desk-to-bag packaging of a Coke can, generated views show synchronized perspectives across four action-sequence timesteps.Source: Figure 10 · p. 14, §A.3.1 · p. 14
A.3.2 Libero Results
- After fine-tuning on 417 trajectories, Libero examples include two successful item-picking cases and one failed empty catch.Source: §A.3.2 · p. 15, Figure 11 · p. 15
A.3.3 Same Initial Conditions but Different Trajectories
- With identical initial conditions, the top and bottom rows show generated results for two different trajectories, illustrating trajectory-following.Source: §A.3.3 · p. 16, Figure 12 · p. 16
A.4.1 EVAC as a Policy Evaluator
- Each retrieval task was evaluated 40 times with randomized initial conditions; success meant retrieving the target item.Source: §A.4.1 · p. 17, Figure 13 · p. 17
- Across four retrieval tasks, EVAC-generated videos and real-world rollouts show consistent outcomes, including two successes and two failures.Source: Figure 14 · p. 18, Figure 15 · p. 18, Figure 16 · p. 19
A.4.2 EVAC as a Data Engine
- For augmentation, EVAC generates images from interpolated actions conditioned on a real frame, then reorders them into training sequences for Go-1.Source: §A.4.2 · p. 20, Figure 18 · p. 20