Source-linked AI summary
Genie Envisioner: A Unified World Foundation Platform for Robotic Manipulation
Yue Liao, Pengfei Zhou, Siyuan Huang, Donglin Yang, Shengcong Chen, Yuxin Jiang, Yue Hu, Jingbin Cai, Si Liu, Jianlan Luo, Liliang Chen, Shuicheng Yan, Maoqing Yao, Guanghui Ren
TL;DR
Robotic manipulation needs scalable systems that connect embodied perception, policy learning, simulation, and evaluation. Genie Envisioner addresses this gap with a unified video-generative platform whose components model robotic interactions, infer executable policies, simulate action-conditioned rollouts, and benchmark embodied world models. The framework demonstrates precise task execution and cross-embodiment generalization with minimal adaptation, while its authors identify limited training-source diversity as an open boundary.
Problem
Robotic manipulation remains difficult to scale because existing systems use separate components, while embodied world models need evaluation of visual, physical, and instruction-aligned behavior.
Method
Genie Envisioner unifies GE-Base video world modeling, GE-Act policy inference, GE-Sim closed-loop simulation, and EWMBench evaluation within one video-centric platform.
Results
The framework achieves precise execution across diverse tasks and strong generalization to novel robot embodiments after minimal adaptation, with alignment to human assessments.
Takeaways & Limitations
Genie Envisioner provides a practical, scalable foundation for downstream research on real-world robotic manipulation and embodied intelligence.
Takeaways & Limitations
Training uses only the single-platform AgiBot-World-Beta corpus, leaving robustness across heterogeneous sources and low-resource domains underexplored.
Abstract
from arXiv · showhide
We introduce Genie Envisioner (GE), a unified world foundation platform for robotic manipulation that integrates policy learning, evaluation, and simulation within a single video-generative framework. At its core, GE-Base is a large-scale, instruction-conditioned video diffusion model that captures the spatial, temporal, and semantic dynamics of real-world robotic interactions in a structured latent space. Built upon this foundation, GE-Act maps latent representations to executable action trajectories through a lightweight, flow-matching decoder, enabling precise and generalizable policy inference across diverse embodiments with minimal supervision. To support scalable evaluation and training, GE-Sim serves as an action-conditioned neural simulator, producing high-fidelity rollouts for closed-loop policy development. The platform is further equipped with EWMBench, a standardized benchmark suite measuring visual fidelity, physical consistency, and instruction-action alignment. Together, these components establish Genie Envisioner as a scalable and practical foundation for instruction-driven, general-purpose embodied intelligence. All code, models, and benchmarks will be released publicly.
1 Introduction
Genie Envisioner unifies robotic manipulation policy learning, evaluation, and simulation in a single video-generative platform. Its components model embodied dynamics, infer executable policies, support closed-loop evaluation, and demonstrate generalization across tasks and robot embodiments.
- Existing robotic manipulation systems often rely on separate components for data collection, policy learning, evaluation, and simulation.
- Genie Envisioner integrates sensing, policy learning, and evaluation within a single closed-loop video-generative world model.
- GE-Base models spatial, temporal, and semantic regularities of robotic interactions, while GE-Act translates language-conditioned visual latents into executable motor commands.
- EWMBench evaluates video-based world models using visual fidelity, physical consistency, and instruction–action alignment rather than generic perceptual metrics alone.
- GE-Act generates 54-step torque trajectories within 200 ms and transfers to novel robot embodiments with only 1 hour of teleoperated demonstrations.
- The platform reports precise execution across diverse manipulation scenarios, cross-embodiment generalization, and alignment with human assessments, with code, models, and benchmarks planned for release.
2 GE-Base: World Foundation Model
GE-Base is an instruction- and image-conditioned autoregressive video world model that predicts temporally coherent multi-view robotic manipulation segments using sparse long-term memory. Its architecture combines compact diffusion backbones, cross-view attention, and instruction cross-attention, while staged pretraining adapts general video generation to robotic dynamics.
- World-model formulation: GE-Base formulates robotic world modeling as text-and-image-to-video generation, forecasting future robot-environment interactions from task instructions and an initial visual observation.The model is intended to provide an embodied predictive representation of manipulation behavior.
- Autoregressive generation: At each autoregressive step, GE-Base generates an N-frame video chunk conditioned on the initial observation, language instruction, and sparsely sampled historical frames.Sparse memory supplies long-term context from previous generated chunks.
- Model architecture: GE-Base uses compact DiT-compatible video generators, with LTX-Video 2B emphasizing speed and COSMOS2 2B emphasizing higher-quality synthesis.The architecture is designed to integrate with different diffusion-transformer video-generation backbones.
- Model architecture: Multi-view observations are encoded with view-specific positional and learnable embeddings, then processed with sparsely inserted cross-view attention to balance consistency and efficiency.The hybrid scheme enables joint reasoning across camera perspectives while allowing other blocks to process views independently.
- Model architecture: Instruction semantics enter the visual token stream through cross-attention using embeddings from a frozen T5-XXL encoder.This aligns generated video with the language task description.
- Autoregressive generation: The unified formulation jointly captures spatial layouts, temporal dynamics, and semantic intent, producing coherent and controllable predictions of embodied robotic behavior.Long-term sparse memory supports extended temporal dependencies while maintaining semantic alignment and visual consistency.
- Pretraining: GE-Base is pretrained on approximately one million real-world dual-arm manipulation episodes, using multi-resolution temporal adaptation followed by low-frequency policy alignment.The dataset includes synchronized multi-view videos paired with natural-language instructions; the two stages expose the model to varied motion rates and downstream policy temporal abstraction.
- Qualitative evaluation: Generated sequences on AgiBot G1 cover varied tasks, objects, and environments across head-mounted, left-arm, and right-arm camera views.The figure presents three synchronized viewpoints for each example.
3 GE-Act: World Action Model
GE-Act extends GE-Base with a latent-space action module that converts instruction-conditioned visual representations into executable policies for robotic manipulation. Its staged training and asynchronous inference support efficient closed-loop control, while experiments show strong performance across tasks and embodiments.
- 3 GE-Act: World Action Model: GE-Act is a plug-and-play action module that transforms multi-view visual and language-conditioned latent representations into temporally structured action policies.It operates in parallel with GE-Base’s visual backbone and supports policy generation without explicit video generation during deployment.
- 3.1 Basic Architecture: The model preserves GE-Base’s DiT depth while using a reduced hidden dimension and separate action-specific transformer blocks with cross-attention to visual features.The visual pathway processes initial and sparse historical observations, while the action pathway processes noise-initialized action tokens.
- 3.2 Training: GE-Act training uses action-space pretraining followed by video adaptation and action specialization on text–video–policy data.The training pipeline first projects video into a latent action-policy space, then adapts visual representations and fine-tunes the action head for downstream tasks.
- 3.3 Asynchronous Inference: Slow-Fast Asynchronous Inference separates visual and action computation by combining cached single-step video denoising with five-step action denoising.The video model runs at 5 Hz while the action model runs at 30 Hz, enabling sparse visual prediction and dense control generation.
- 3.4 Action Planning via GE-Act on AgiBot G1: GE-Act consistently outperforms UniVLA and GR00T N1 across SR and E2E metrics on real-world AgiBot G1 manipulation tasks.An ablation reports 64 SR and 81 E2E with in-domain pretraining, improving to 76% and 89% when combined with general video pretraining.
- 3.4 Action Planning via GE-Act on AgiBot G1: GE-Act achieves comparable or superior performance in fast mode, especially on latency-sensitive tasks such as dynamic object tracking, reactive grasping, and detergent packing.Qualitative results also show precise and reliable execution of complex real-world manipulation tasks conditioned on natural-language instructions.
4 Cross-Embodiment Generalization with Genie Envisioner
GE generalizes across robotic embodiments through few-shot adaptation of GE-Act, achieving precise manipulation on Agilex Cobot Magic and Dual Franka, while also outperforming baselines across most RoboTwin tasks.
- Evaluation Setup: GE evaluates cross-embodiment transfer on Franka, Agilex Cobot Magic, and the dual-arm RoboTwin simulator.These platforms extend evaluation beyond the in-domain AgiBot G1 setting.
- Adaptation Strategy: A two-stage task-specific fine-tuning strategy adapts GE-Act to new embodiments using limited instruction-conditioned demonstrations.The first stage adapts visual generation while freezing CLIP and video encoders; the stated pipeline transfers perceptual and motor capabilities under minimal supervision.
- Agilex Cobot Magic: With approximately 1 hour of demonstrations per task, GE-Act outperforms GR00T N1, π0, and UniVLA on complex Agilex Cobot Magic manipulation tasks.UniVLA and GR00T N1 achieve 0% success on the cited complex fine-grained tasks, while GE-Act significantly outperforms π0 on deformable-object manipulation.
- Agilex Cobot Magic: GE-Act generates coherent multi-view predictions and executes cloth-folding and box-folding tasks with high precision and reliability on Agilex Cobot Magic.The results cover both rigid and non-rigid object dynamics and consistency across camera views.
- Dual Franka: On Dual Franka, GE-Act consistently outperforms task-specific baselines after adaptation with 250 teleoperated episodes, approximately one hour of data.The evaluation uses cloth folding and compares against GR00T N1, π0, and UniVLA.
- RoboTwin: On RoboTwin, jointly trained GE-Act outperforms π0 and GO-1 on three of four tasks but trails VLA methods slightly on lift pot.The authors attribute the minor gap to possible task interference from joint training.
5 GE-Sim: World Simulator
GE-Sim extends GE-Base into an action-conditioned video world simulator for closed-loop policy evaluation and controllable data generation. It combines spatial pose and temporal motion conditioning to produce visually aligned, temporally coherent rollouts.
- Simulator Role: GE-Sim generates temporally coherent visual predictions conditioned on robotic actions for policy interaction and evaluation.It provides a unified testbed for policy learning and generalization without relying on physics-based constraints.
- Simulator Architecture: GE-Sim transfers GE-Base into an action-conditioned video generator using LTX-Video for speed and COSMOS2 2B for high-fidelity simulation.Action trajectories drive video synthesis over time.
- Hierarchical Action Conditioning: The hierarchical conditioning mechanism projects spatial poses into image space and injects temporal motion deltas through cross-attention.This preserves pretrained spatiotemporal semantics while interfacing with diverse policy models.
- Trajectory Representation: A single-arm control step uses a 7-dimensional position, orientation, and gripper vector, while dual-arm trajectories use 14 dimensions over a K-step horizon.The resulting trajectory is represented as A ∈ R^K×14 in the dual-arm setup.
- Pose Conditioning: Pose and history-frame features are encoded with a shared video encoder, added element-wise, and inserted into the visual token stream.This fused token combines contextual visual semantics with explicit pose information.
- Action-Conditioned Generation: Ground-truth action tests show that GE-Sim’s predicted end-effector motion aligns consistently with the spatial intent of the input across tasks and viewpoints.The evaluation visualizes projected action targets alongside synthesized next frames.
- Closed-Loop Use: GE-Sim generates diverse manipulation sequences by replaying action trajectories under different initial visual environments.This makes the simulator a data engine as well as a policy-evaluation environment.
6 EWMBench: Embodied World Model Benchmark
EWMBench evaluates embodied world models across scene, motion, and semantic dimensions, emphasizing the structural constraints of robotic manipulation. GE-Base outperforms baselines across multiple dimensions, especially temporal alignment and dynamic consistency, while GE-Sim shows strong action-conditioned fidelity.
- Benchmark Scope: EWMBench measures visual fidelity, physical consistency, and instruction–action alignment for robotic video world models.Its design targets spatial, temporal, and semantic dynamics rather than visual realism alone.
- Benchmark Construction: The benchmark uses 10 held-out household and industrial tasks, each decomposed into 4–10 atomic sub-actions.The tasks are disjoint from the 1M-scale pretraining phase and require reasoning over affordances and action ordering.
- Benchmark Construction: EWMBench promotes trajectory diversity by selecting minimally overlapping dual-arm motions using voxelized 3D IoU similarity.A greedy selection procedure broadens motion-pattern coverage and reduces redundancy.
- Metrics: Scene consistency assesses stability of visual appearance, environment layout, and viewpoint alignment across generated video frames.The metric uses patch-level feature similarity from a DINOv2 encoder adapted to robotic manipulation data.
- Metrics: Action Trajectories Quality evaluates spatial alignment, temporal alignment, and dynamic consistency against manually annotated reference trajectories.Generated gripper trajectories are reconstructed using an end-effector detector and evaluated with symH distance, NDTW, and velocity-acceleration comparisons.
- Results: GE-Base consistently outperforms baselines across multiple EWMBench dimensions, with particular strengths in temporal alignment and dynamic consistency.The reported advantages target action-plausible and temporally stable robotic behaviors.
- Results: GE-Sim demonstrates high spatial accuracy, precise trajectory execution, and strong semantic coherence under ground-truth action conditioning.The results also identify limited visual diversity as a remaining issue in the action-conditioned evaluation.
- Human Alignment: EWMBench rankings show strong concordance with human judgments, whereas VBench misaligns particularly on embodied consistency and goal-conditioned reasoning.Rankings aggregate annotations across annotators and samples with multiple review rounds.
7 Related Works
Related work spans analytical, neural, video-generative, vision-language-action, simulation, and embodied-world-model evaluation approaches. The paper positions GE against limitations in action conditioning, real-world transfer, scalable evaluation, and task-relevant metrics.
- World Models for Robotic Manipulation: Analytical and model-based world models require task-specific engineering, while neural world models aim to learn compact predictive representations with broader generalizability.The related work traces a progression from analytical modeling and system identification to learned world models.
- Video Generative Models for Robot Learning: General video generators provide high visual quality but remain limited for robotics by action conditioning, temporal coherence, and multi-view reasoning.These limitations are especially relevant to robotic manipulation videos.
- Vision-Language-Action Models: VLA models predict action sequences from vision-language pretraining and robot demonstrations but retain limitations associated with behavior cloning.The paper contrasts this paradigm with its video-generative world-model formulation.
- Policy Evaluation in Robotics: Physics simulators provide fast evaluation but require extensive manual tuning and still face a sim-to-real gap, while real-world evaluation is slow and resource-intensive.These constraints motivate generative simulation approaches.
- Evaluation of Embodied World Models: Traditional video metrics often correlate poorly with real-world task success because they emphasize visual realism over task relevance.Embodied-world-model benchmarks therefore introduce structured metrics for manipulation scenarios.
8 Limitations
The authors identify remaining limitations in data diversity, embodiment scope, and completeness toward general-purpose robotic manipulation. Training uses a single-platform corpus, while evaluation excludes dexterous hands and whole-body behaviors.
- Data Coverage and Source Diversity: Training relies exclusively on the AgiBot-World-Beta dataset, limiting exposure to diverse embodiments, sensor modalities, and scene configurations.Robustness across heterogeneous sources and low-resource domains remains underexplored.
- Embodiment Scope and Dexterity: The study is confined to upper-body tabletop manipulation with parallel-jaw grippers, excluding dexterous hand coordination and full-body locomotion.The authors identify fine-grained multi-contact interactions and whole-body behaviors as future integration targets.
- Overall Scope: Genie Envisioner is not yet a complete solution for embodied intelligence.The authors characterize it as a meaningful step toward broader manipulation capabilities rather than a finished system.
9 Conclusion
Genie Envisioner unifies high-fidelity video generation, policy learning, simulation, and evaluation for instruction-driven dual-arm manipulation. Its components support precise execution, cross-embodiment transfer, closed-loop refinement, and multi-axis assessment.
- Conclusion: GE-Base captures spatiotemporal and semantic robotic-interaction dynamics for instruction-aligned video synthesis.It serves as the platform’s foundation for representing embodied interactions.
- Conclusion: GE-Act enables high-precision execution across diverse in-domain tasks and transfers to novel robotic platforms with minimal adaptation.The conclusion highlights cloth folding and box packing among its complex-task demonstrations.
- Conclusion: GE-Sim supports closed-loop simulation for continuous policy refinement.The simulator extends the platform beyond video generation and direct policy inference.
- Conclusion: EWMBench assesses visual realism, semantic alignment, and policy consistency across the unified framework.It provides the platform’s evaluation component for embodied manipulation.