Source-linked AI summary
GE-Sim 2.0: A Roadmap Towards Comprehensive Closed-loop Video World Simulators for Robotic Manipulation
Boxiang Qiu, Liliang Chen, Yue Liao, Nan Wang, Lintao Wang, Jiayi Luo, Wenzhi Zhao, Shengcong Chen, Di Chen, Ye Li, Chen Gao, Shuicheng Yan, Si Liu, Maoqing Yao, Guanghui Ren
TL;DR
Evaluating scalable manipulation policies remains difficult because real-robot benchmarking is slow and existing simulators have limited coverage. GE-Sim 2.0 is a closed-loop video simulator with state prediction, rollout judging, and accelerated inference; it leads WorldArena at 2B parameters and supports measurable real-world policy gains.
Problem
Real-robot benchmarking is slow and hard to reproduce, while existing robotic benchmarks and simulators provide limited coverage for long-horizon, contact-rich manipulation.
Method
GE-Sim 2.0 retrains action-conditioned video generation on thousands of hours of robot data and adds state-expert, world-judge, and acceleration modules for closed-loop policy learning.
Results
GE-Sim 2.0 tops WorldArena at 2B parameters, outperforming dedicated robotic world models and closed-source video generators across six manipulation tasks.
Takeaways & Limitations
GE-Sim 2.0 provides an efficient platform for scalable evaluation and closed-loop learning of robotic manipulation policies.
Takeaways & Limitations
The closed-loop and ablation studies use a single VLA policy family, so validation across broader policy families remains future work.
Abstract
from arXiv · showhide
We introduce GE-Sim 2.0 (Genie Envisioner World Simulator 2.0), a closed-loop video world simulator for robotic manipulation. Building on the action-conditioned video generation framework of Genie Envisioner, GE-Sim 2.0 is re-trained on thousands of hours of real-world robot data spanning teleoperation, contact-rich interaction, and on-robot policy deployment, substantially improving action-following fidelity and trajectory coverage. On top of this foundation, three new modules close the loop from video simulation to policy learning: a state expert that decodes proprioceptive state from video latents to support next-chunk prediction by downstream VLA policies; a world judge that scores generated rollouts against task instructions, yielding machine-verifiable success signals and rewards in place of manual inspection; and an acceleration framework that delivers a 25-frame rollout in 2.3 seconds on a single H100, with up to 4* frame skipping at inference for long-horizon evaluation. GE-Sim 2.0 tops the public WorldArena leaderboard at only 2B parameters, outperforming both dedicated robotic world models and closed-source general video generators, and policies trained against its rollouts and rewards translate into measurable real-world gains, establishing GE-Sim 2.0 as a practical platform for scalable evaluation and closed-loop learning of manipulation policies.
1 Introduction
GE-Sim 2.0 is introduced as a neural, closed-loop video world simulator for robotic manipulation, motivated by the need to evaluate and train increasingly capable policies on long-horizon, contact-rich tasks. It strengthens action-conditioned video generation with diverse real-world robot data and adds modules for state decoding, rollout judging, and scalable closed-loop learning.
- Motivation: Scaling robot learning is expanding manipulation toward long-horizon, contact-rich, and deformable-object tasks, while making policy evaluation increasingly important.The introduction frames larger models, internet-scale demonstrations, and capable vision-language-action policies as drivers of this shift.
- Motivation: Neural world simulators use generative video models to synthesize diverse, photorealistic manipulation scenes that handcrafted simulators cannot easily reproduce.The proposed paradigm conditions simulation on an initial observation and an action trajectory.
- GE-Sim 2.0: Thousands of hours of real-world robot data retrain GE-Sim 2.0 across teleoperation, contact-rich arm-object interaction, and policy-deployment rollouts.This data scale and diversity improve action-following accuracy, contact and deformation fidelity, and trajectory coverage across successes, failures, tasks, and scenes.
- GE-Sim 2.0: Three modules close the loop: a state expert decodes dual-arm joint angles and gripper states from video latents, while world judging and acceleration support scalable policy learning.The state expert provides downstream policies with proprioceptive state alongside visual observations.
- Validation: 2B parameters: GE-Sim 2.0 tops the public WorldArena leaderboard and outperforms dedicated robotic world models and closed-source general video generators.Named robotic baselines include Ctrl-World, DreamDojo, GigaWorld, and ABot; general video baselines include Sora and Veo.
2 Preliminaries
GE-Base models robotic worlds through autoregressive, multi-view video generation conditioned on language and observations. GE-Sim repurposes this backbone for action-conditioned simulation by replacing language conditioning with spatially aligned robot actions and camera geometry.
- GE-Base: GE-Base predicts future multi-view video chunks from a language instruction and an initial observation as the scene evolves.The model uses onboard head, left-wrist, and right-wrist cameras.
- GE-Base: Sparse keyframe memory extends temporal context beyond the current chunk while keeping the model input tractable.Memory is constructed from keyframes sampled from previously generated chunks.
- GE-Sim: GE-Sim replaces language conditioning with aligned action sub-trajectories while retaining GE-Base’s multi-view, autoregressive, sparse-memory backbone.The resulting action-conditioned simulator provides the foundation for GE-Sim 2.0.
- GE-Sim: GE-Sim encodes dual-arm control as a 14-dimensional trajectory of end-effector positions, orientations, and gripper openness.The trajectory over a K-step horizon is A = [a1, . . . , aK] ∈R^K×14.
- GE-Sim: Spatial action conditioning combines pose images and camera raymaps to align robot actions and viewpoints with the target image grid.Pose images project end-effector positions, orientation axes, and gripper openness; raymaps expose camera centers and view directions.
3 Genie Envisioner World Simulator 2.0
GE-Sim 2.0 upgrades GE-Sim from a view-only video simulator into a closed-loop system that interacts with policy models, predicts visual and proprioceptive outcomes, and provides machine-verifiable feedback. Its design supports scalable policy evaluation and reward-driven learning through a world judge and accelerated rollout generation.
- Closed-loop simulator: GE-Sim 2.0 transforms GE-Sim into a closed-loop simulator that precisely interacts with policy models and feeds back the resulting interaction.The simulator accepts actions from evaluated policies as well as teleoperation logs, motion planners, and hand-authored trajectories.
- Dual-expert architecture: The vision expert generates future video chunks from action conditions, while the parallel proprioceptive state expert predicts corresponding joint-space states for downstream policy prediction.The state expert decodes proprioceptive state from the vision expert’s visual context using lightweight transformer blocks with self-attention, cross-attention, and feed-forward layers.
- World judge: The world judge scores generated rollouts frame by frame and outputs machine-verifiable success signals for automated policy evaluation and reward-driven learning.Its success curve replaces reliance on manual inspection and supplies feedback for methods including filtered BC and RL.
- Training and conditioning: The vision expert is trained on thousands of hours of real-world robot data spanning teleoperation, on-robot policy deployment, and object-interaction trajectories.The training set includes both successful and failed trajectories, while the vision expert uses action-conditioned multi-view diffusion with ray-map and EE-pose-map conditioning.
- Acceleration: 100-frame rollout generation takes about 2.3 seconds on a single H100 with only four inference steps, enabling throughput suitable for large-scale parallel evaluation.This acceleration comes from distillation-based post-training and supports long-horizon rollout generation.
4 Experiments
Experiments show that GE-Sim 2.0 provides strong long-horizon, closed-loop simulation, especially when proprioceptive state feedback and task-aware reward judging are included. Its rollouts align better with real-robot outcomes and improve downstream policy learning.
- WorldArena leaderboard: Using only a 2B-parameter backbone, GE-Sim 2.0 attains the top overall score on the public WorldArena leaderboard, outperforming dedicated robotic world models and larger closed-source video generators.Compared systems include Ctrl-World, DreamDojo, GigaWorld, ABot, Sora, and Veo.
- Replay quality: +3.96 dB PSNR over Ctrl-World and +5.67 dB over DreamDojo, GE-Sim 2.0 dominates all five replay metrics in both head-camera and multi-view settings.On the head view, FID is 32.3 versus 62.7 and FVD is 481 versus 1084; the multi-view setting is considerably harder.
- Closed-loop calibration: GE-Sim 2.0 with state conditioning shows the closest task-level alignment between simulated and real-robot success rates across six tasks.Its fitted trend has a slope close to one with a small negative offset, while contact-sensitive tasks retain gaps caused by grasp state, contact timing, and object interaction errors.
- Proprioceptive state expert: 0.81 average episode-level accuracy and 0.82 recall are achieved with state conditioning, versus 0.74 and 0.67 without it.The state expert improves tracking across action chunks, particularly for tasks involving fine-grained motion, contact maintenance, or long-horizon accumulation.
- World judge: 79% WM-rollout accuracy versus 60% for Qwen yields +19 pp and reduces event distance from 57.8 to 28.2 frames; on GT videos, accuracy is 87% versus 58% (+29 pp).The accuracy ranking holds on five of six tasks, with Command grasp & release as the exception.
- Policy learning: 0.150 absolute average success gain results when WM-filtered behavioral-cloning data is added, increasing real-robot success from 0.417 to 0.567.Success improves from 0.40 to 0.55 on Pour water, 0.40 to 0.50 on Fold towels, and 0.45 to 0.65 on Pull out plug.
5 Conclusion
GE-Sim 2.0 advances Genie Envisioner from a view-only video world simulator toward an omni world simulator for robotic manipulation. Its closed-loop design combines real-robot-data retraining with state decoding and rollout-based success evaluation.
- Conclusion: GE-Sim 2.0 extends Genie Envisioner toward an omni world simulator for robotic manipulation.The paper characterizes this as a step beyond a “view-only” video world simulator.
- Conclusion: The action-conditioned video generation framework was retrained on thousands of hours of real-robot data.
- Conclusion: Three coordinated components close the loop: a proprioceptive state expert, a world judge, and a DMD-based acceleration framework.The state expert decodes joint-space state from video latents, while the world judge converts rollouts into machine-verifiable success signals.
6 Looking Forward
GE-Sim 2.0 presents one viable omni world simulator for robotic manipulation, while identifying open directions toward general embodied AI. These directions center on scaling data and models, unifying world-model outputs, and moving from offline filtered behavior cloning to online closed-loop learning.
- 6 Looking Forward: GE-Sim 2.0 demonstrates competitive robotic-manipulation simulation as one viable instantiation of an omni world simulator, while leaving general embodied AI as an open goal.The paper defines components such a system should include and outlines several directions for extending it.
- Data and model scaling: A general embodied world simulator requires large-scale cross-embodiment, ego-view, and UMI-style data, broader manipulation video, and systematic data- and model-scaling studies.The current foundation model uses dual-arm teleoperation and on-robot replay from a fixed embodiment; scaling the foundation backbone remains a core bottleneck.
- From an external judge to a unified model: The longer-term goal is a unified world model whose forward pass jointly produces future frames, proprioceptive state, and task-completion signals, replacing the external reward network.The current world judge is an external module sufficient for task-success assessment; unification would let rewards inherit priors learned by the world model at scale.
- From offline filtered BC to online closed-loop learning: GE-Sim 2.0 rollouts and rewards already improve a real-robot policy through filtered behavior cloning, motivating online policy learning and reinforcement learning inside the simulator.This would advance the system from an evaluator and data filter into an environment for policy training.
8 Appendix
The appendix specifies training-time augmentations, student–critic implementation details, and qualitative rollout comparisons across manipulation tasks. These details cover robustness to history mismatch, few-step inference support, and visual comparisons with Ctrl-World and DreamDojo.
- Augmentation settings: Memory-frame latents receive three perturbation types during training, gated by an outer activation probability of 0.8 to address rollout drift at inference.Progressive noise mixing uses a per-frame activation probability of 0.5.
- Student–critic implementation: The student targets 4 inference steps, while training randomizes denoising steps between 1 and 4 to support one- to few-step inference.The vision teacher remains frozen, and both the student and fake-score critic are initialized from it.
- Qualitative comparisons: Additional qualitative comparisons evaluate GE-Sim 2.0 against Ctrl-World across six manipulation tasks using multi-view rollouts, and against DreamDojo on four tasks using head-view comparisons.The comparisons include Command grasp & release, Clean mirror stains, Borrow flame, Fold towels, Pull out plug, and Pour water.