Source-linked AI summary
GE-Act 2.0: Pretraining and Scaling a World-Action Model for Robotic Manipulation
AgiBot Research Team, Renhang Liu, Wenzhi Zhao, Zhuo Yang, Liliang Chen, Pengfei Zhou, Shengcong Chen, Guanghui Ren, Youlun Peng, Rongjun Jin, Nan Wang, Sukai Wang, Xindong He, Jinyuan Feng, Ziyu Xiong, Linqing Zhong, Yifei Wei, Feng Han, Long Zhang, Da Huang, Nanshu Zhao, Chenghao Yin, Mo Wu, Zhaodong Yan, Kongtao Hu, Yuxiang Yan, Aogelijiang Niyazi, Yu Fang, Jia Zeng, Lizhu Meng, Daizhen Lv, Haoyu Cao, Zhiwen Hou, Lianjin Ye, Yuehan Niu, Zhikai Cai, Xuan Hu, Hui Min, Xiongfeng Cai, Yue Liao, Jing Wu, Soujanya Poria, Ye Li, Sanping Zhou, Maoqing Yao
TL;DR
Existing WAMs leave whole-system pretraining and scaling underexplored, despite the need to learn from heterogeneous manipulation data. GE-Act 2.0 trains a control-oriented autoencoder, single-step visual planner, and inverse dynamics model from scratch, connecting them with action-compatible co-training. Across zero-shot OOD evaluations, more manipulation data improves performance broadly across tasks and embodiments, while instruction grounding remains strong.
Problem
Existing WAMs typically build on pretrained video generators, leaving WAM-wide pretraining and scaling unresolved.
Method
GE-Act 2.0 combines a control-oriented autoencoder, single-step visual planner, inverse dynamics model, and KASO-based co-training, with trainable components initialized from scratch.
Results
Across zero-shot OOD evaluations on 100 real-robot tasks and two embodiments, increasing manipulation data improves nearly all skill groups and supports transfer to a sparsely represented embodiment.
Takeaways & Limitations
Skill-specific data coverage strongly predicts success, while complementary experiments show fine-grained instruction grounding and adherence under behavioral and semantic conflicts.
Takeaways & Limitations
The study has not yet explored egocentric video pretraining at the scale its availability permits.
Abstract
from arXiv · showhide
World-action models (WAM) predict future states to guide robot actions, enabling learning from both action-free video and action-labeled interaction. Most inherit pretrained video generators, leaving WAM pretraining and scaling underexplored. We introduce Genie Envisioner Act 2.0 (GE-Act 2.0), a world-action model whose trainable generative and action components are all initialized from scratch on manipulation data. It combines a control-oriented autoencoder (CoAE), a single-step visual planner (SVP), and an inverse dynamics model (IDM). CoAE retains action- and instruction-relevant information under aggressive compression, while SVP produces a complete future state in one differentiable pass, so visual planning and inverse dynamics can be pretrained separately on complementary data. The components are then jointly trained with knowledge-aligned selective optimization (KASO), which reduces mismatched supervision by selecting only predicted futures judged behaviorally compatible with the recorded action. We evaluate pretrained checkpoints directly, without per-task fine-tuning, on 100 tasks across 20 manipulation skill groups with held-out scenes, backgrounds, lighting, and object instances. Scaling co-training data from 300 to 30,000 hours raises success from 17.1% to 44.1% on G1-OP and from 13.4% to 31.1% on G2-90D; despite comprising less than 2% of the co-training data, G2-90D improves by 17.7 points, suggesting cross-embodiment transfer. Gains span 19/20 and 18/20 skill groups, and skill-specific coverage strongly correlates with zero-shot out-of-distribution (OOD) success (Pearson r=0.80; Spearman rho=0.85). Under the same protocol, the model grounds object, color, shape, and position references in at least 90% of trials and follows explicit instructions even when they conflict with an already-committed behavior or a conventional scene association.
1 Introduction
GE-Act 2.0 addresses underexplored WAM pretraining and scaling by learning visual generation and action prediction from heterogeneous manipulation data. Its design combines compact control-oriented representations, single-step future prediction, and action-compatible co-training to support zero-shot OOD evaluation.
- Existing WAMs typically couple action models to pretrained video generators, leaving whole-system pretraining and scaling unresolved.
- GE-Act 2.0 learns from action-free, instruction-free, and fully labeled data, scaling with all three sources.
- Representation: Its control-oriented autoencoder preserves information for visual prediction and action recovery in an aggressively compressed latent space.
- Generation: The single-step visual planner enables visual-generation pretraining on action-free video while the inverse dynamics model uses instruction-free robot trajectories.
- Alignment: The validity gap occurs when a generated future depicts a different valid behavior from the recorded action, producing mismatched visual-action supervision.
- Alignment: KASO selects action-compatible predicted futures during co-training, preserving broader action-space coverage for later exploration.
- Zero-shot OOD evaluation measures pretrained capability directly across a 100-task real-robot suite without relying on downstream task-specific fine-tuning.
2 Related Work
Related work spans generalist language-conditioned policies, latent representations, efficient visual generation, world-action interfaces, and evaluation protocols. GE-Act 2.0 is positioned within decoupled world-action designs while emphasizing control-oriented latents and zero-shot assessment.
- Generalist robot policies scale language-conditioned control across broad task collections, while world-action models connect visual prediction to control through multiple interfaces.
- Representations: Most latent video generators use autoencoder spaces optimized for content reconstruction rather than action recovery.
- Representations: CoAE provides a compact generative latent space aligned to multiple visual teachers and evaluated by fixed-capacity action recovery.
- Generation: Robot world models commonly adapt natural-video generators and retain or distill multi-step diffusion, whereas one- and few-step generation offer an efficiency lineage.
- World-action interfaces: World-action training couples generated future visual states to action models, exposing a generated-versus-recorded input gap under teacher forcing.
- Evaluation: Most robot models are evaluated after task-specific fine-tuning, which conflates pretrained quality with adaptation-data amount and coverage.
3 GE-Act 2.0
GE-Act 2.0 is built from scratch around a control-oriented latent space, a single-step visual planner, and an inverse dynamics model. Its separate pretraining and KASO-based co-training address data diversity, extreme compression, and mismatched visual-action supervision.
- System architecture: GE-Act 2.0 combines CoAE, SVP, and IDM, with the flow generator and IDM randomly initialized and pretrained from scratch on manipulation data.The SVP generates future visual states, while the IDM translates visual transitions into dense and sparse actions.
- Control-oriented representation: CoAE aggressively compresses visual observations while preserving information needed for action recovery and instruction grounding.Its favorable compactness–informativeness trade-off uses one-sixteenth as many tokens per frame as DINOv3 and V-JEPA 2.1 while remaining within 13–31% of their action-recovery errors.
- Single-step visual planning: The SVP produces complete future visual latents in one differentiable flow-generator pass, allowing visual planning and inverse dynamics to use complementary data during separate pretraining.The IDM receives generated future states during co-training, while language influences action through those states rather than as a direct IDM input.
- Validity gap: The validity gap occurs when a generated future depicts a different valid outcome mode from the recorded action paired with it.Manipulation is multimodal, so visually plausible futures can still be action-incompatible with the recorded behavior.
- Knowledge-Aligned Selective Optimization: KASO selects action-compatible candidate futures in action space before replaying them with gradients, coupling the SVP and IDM during co-training.In the toy system, KASO recovers four video modes and two action modes, whereas competing strategies produce diffuse or collapsed action distributions.
4 Training Data and Mixture Composition
GE-Act 2.0 uses distinct data mixtures to pretrain visual planning and inverse dynamics before connecting them through KASO co-training. These mixtures combine instruction–video data, action-labeled trajectories, and heterogeneous robot and human manipulation sources.
- Stage-specific mixtures: SVP pretraining uses instruction–video data, including manipulation recordings without robot action labels and egocentric video.
- Stage-specific mixtures: IDM pretraining uses action-labeled robot trajectories, including demonstrations, failures, and deployment data.
- Stage-specific mixtures: KASO co-training connects the pretrained components through instruction–video–action data while retaining their separate objectives.
- Data sources and composition: The data span two robot embodiments, simulation, open-source datasets, egocentric and human video, and rollout and failure data.G1-OP contributes more than 50% of co-training data, while G2-90D contributes less than 2%.
- Data sources and composition: The three training mixtures contain 39,000 hours for SVP pretraining, 32,000 hours for IDM pretraining, and 30,000 hours for co-training.
5 Evaluating Zero-Shot OOD Manipulation at Scale
Zero-shot evaluation across 100 tasks shows that scaling manipulation data improves performance broadly across embodiments, skill groups, and instruction dimensions. Skill-specific coverage is strongly associated with OOD success, while grounding is strongest for object, color, position, and shape references and weakest for order.
- Scaling with training data volume: 44.1% G1-OP suite-level SR is reached with 30,000 hours, up from 17.1% with 300 hours; G2-90D rises from 13.4% to 31.1%.The total gains are 27.0 and 17.7 percentage points, respectively, with no evidence of saturation through 30,000 hours.
- Scaling with training data volume: 19 of 20 G1-OP and 18 of 20 G2-90D skill groups perform better at 30,000 than at 300 hours.Nonzero-SR tasks increase from 39 to 76 for G1-OP and from 24 to 72 for G2-90D.
- Scaling with training data volume: The data-scarce G2-90D gains 17.7 points despite contributing less than 2% of co-training data.Five of six skill groups with fewer than two hours improve from zero SR to nonzero SR at the largest scale.
- Scaling with skill-specific data volume: Skill-specific training coverage strongly correlates with zero-shot OOD success, with Pearson r = 0.80 and Spearman ρ = 0.85.The fitted relationship corresponds to a 1.94-logit SR increase per tenfold increase in skill-specific hours.
- Scaling with skill-specific data volume: Demanding skills improve with coverage, while similar motions diverge: Pass reaches 70.0% with 590.8 hours, whereas Wipe reaches 76.7% and Sweep 3.3%.
- Instruction grounding: Follow Score is at least 90.0% for object, color, direct position, and shape, but falls to 82.5%/65.7% for size and 13.3%/26.7% for order.Across 295 rollouts, overall Follow Score is 83.1% versus 72.9% full-task SR.
- Instruction grounding: The grounding profile tracks a long-tailed instruction distribution, but the evidence establishes association rather than causation.Size, shape, and order occur in only 0.96%, 0.67%, and 0.13% of training instructions, respectively; shape remains a low-frequency exception.
- Instruction conflict: Explicit commands can redirect behavior after target switches and override conventional scene associations in physically feasible placements.Target changes near contact require recovery from the previous trajectory, while counter-conventional placement succeeds qualitatively.
6 Conclusion
GE-Act 2.0 unifies visual generation and action prediction through a control-oriented autoencoder, single-step visual planner, and KASO. Across zero-shot OOD evaluation, more manipulation data improves nearly all skill groups, including a sparsely represented embodiment, while instruction grounding and conflict handling are also demonstrated.
- Contributions: GE-Act 2.0 combines a control-oriented autoencoder, single-step visual planner, and KASO for unified world–action pretraining.
- Empirical conclusions: Across 100 real-robot tasks and two embodiments, increasing manipulation data broadly improves nearly all evaluated skill groups.
- Empirical conclusions: Performance improves on the sparsely represented embodiment, while skill-specific data coverage strongly predicts downstream success.
- Empirical conclusions: Complementary experiments show fine-grained instruction grounding and adherence to explicit commands under behavioral and semantic conflict.
7 Limitations and Outlook
The study leaves important scaling and reasoning capabilities unexplored. Future work should test egocentric-video pretraining at larger scale and complement direct instruction-conditioned control with deliberative systems.
- Data scaling: The study has not explored egocentric manipulation video at the scale its availability permits.Future work should establish how its volume, diversity, and composition affect visual-generation pretraining and robot capability.
- Deliberative capability: GE-Act 2.0 focuses on System-1-style instruction-conditioned control without explicit deliberation.Open-environment execution additionally requires reasoning, long-horizon planning, decomposition, memory, and self-correction.
8 Contributions and Acknowledgments
GE-Act 2.0 is evaluated under standardized simulation shifts spanning environment randomization and instruction following. It leads most clean-to-random conditions and achieves the highest average GenieSim-Instruction score, though its advantages are not uniform.
- RoboTwin Clean-to-Random Generalization: 60.52% success under combined Hard randomization leads the strongest baseline by 12.62 points.GE-Act 2.0 leads five of six reported clean-to-random conditions.
- RoboTwin Clean-to-Random Generalization: Lighting is the only clean-to-random condition not led by GE-Act 2.0, where it reaches 65.92% versus π0.5 at 69.20%.
- GenieSim-Instruction: 0.770 is GE-Act 2.0’s highest average GenieSim-Instruction score, above ACoT-VLA at 0.757 and π0.5 at 0.746.It leads four tasks, with largest margins on logical composition and common-sense reference.
- GenieSim-Instruction: The GenieSim-Instruction advantage is non-uniform, with the strongest baseline ahead on number, shape, size, and several direct attribute-matching tasks.
A.3 LIBERO-Plus Robustness across Perturbation Axes
LIBERO-Plus evaluates GE-Act 2.0 across seven observation, instruction, and execution perturbation axes. Performance is strong overall but remains uneven, especially under background and robot-state shifts.
- Overall robustness: 80.4% overall LIBERO-Plus success exceeds StarVLA by 6.3 points but trails π0.5 by 4.0 points.The benchmark covers camera viewpoint, robot initial state, language, lighting, background, sensor noise, and object layout.
- Per-axis results: 95.5% success under sensor-noise perturbations is GE-Act 2.0’s strongest reported LIBERO-Plus axis.Camera perturbations are also strong at 94.1%.
- Remaining weaknesses: Background and robot-state perturbations remain the principal LIBERO-Plus weaknesses, so robustness is not uniform across axes.
- Interpretation: The LIBERO-Plus results complement real-robot atomic-task evaluation by exposing broad OOD capability and benchmark-specific limitations.
B Training Recipes and Hyperparameters
The training recipes use a shared mixed-precision optimization setup with fixed optimizer, regularization, and learning-rate conventions. Batch sizes are specified per GPU.
- Optimization defaults: All stages use bf16 mixed precision with DeepSpeed ZeRO stage 2 and AdamW using β1=0.9, β2=0.95, and weight decay 10^-5.
- Optimization defaults: Training applies gradient-norm clipping at 1.0 and a constant learning rate after linear warmup.
- Regularization and batching: The recipes use neither classifier-free guidance nor caption dropout, and batch sizes are reported per GPU.
B.1 Architectures
The architecture and training pipeline combine compact visual representations, flow-based future prediction, inverse dynamics, and selective joint optimization. A toy stress test illustrates that recorded-pair fit can coexist with failure on generated off-manifold video.
- Data and temporal setup: The default input uses four dense-displacement frames, two sparse frames, and a 52-frame horizon covering 1.73 seconds at 30 Hz.Each sample carries 52 dense actions and two sparse actions.
- Training pipeline: The SVP and IDM are pretrained separately before fused updates combine recorded-video, recorded-future, and selected-generated-future losses.The fused coefficients are λSVP=1, λIDM=0.1, and λE2E=1, with the VLM and CoAE frozen during this update.
- Selective optimization: KASO selects the single lowest-energy candidate from four generated futures and discards the remaining candidates for generated-future action supervision.Selection uses the CoAE-encoded recorded future as reference and averages energy over 52 dense action tokens.
- Toy objectives: The toy pipeline trains flow matching, recorded-pair IDM, and generated-video action losses, with connected arms reducing pretraining-loss weights to 0.3 and 0.1.Decoupled Pretraining retains unit weights on both pretraining losses.
- Toy stress test: 0.0112 recorded-pair IDM RMSE occurs for both sz=0.2 and sz=1.0, yet Decoupled fails when its IDM receives generated off-manifold video.The sz=0.2 setting amplifies sensitivity to the third coordinate fivefold.
- Toy stress test: The toy Figure 6 is a single-run illustration rather than a seed-averaged benchmark because action-density maxima vary substantially across seeds 0–4.
C.1 Atomic-Skill Taxonomy and Evaluation Details
The evaluation covers 100 atomic real-robot tasks across 20 skill groups on two embodiments, using fixed instruction-specific terminal success criteria. Trials use standardized resets and exclude test scenes, backgrounds, lighting, and object instances from pretraining and co-training.
- Evaluation scope: 100 atomic tasks span 20 skill groups and are evaluated on both G1-OP and G2-90D.The taxonomy’s parenthetical group counts sum to 100.
- Success criteria: Each task uses the same instruction and a fixed, observable, instruction-specific terminal success criterion.Examples include grasping and lifting for Pick, reaching the specified receptacle for Place, and fully closing the target for Close.
- Trial protocol: 10 trials are run for every task, embodiment, and training-data scale, with matched resets across checkpoints.Success rate is reported as the percentage of successful trials.
- Held-out conditions: Test scenes, backgrounds, lighting, and object instances are excluded from pretraining and co-training.This defines the suite’s held-out evaluation conditions.
C.2 Task-Level Data-Scaling Results
Task-level results are organized as empirical success rates across all 100 tasks, multiple co-training scales, and both embodiments. Table 8 separates G1-OP and G2-90D results and computes every value from 10 trials.
- Results organization: Table 8 reports empirical success rate for every task, data scale, and embodiment.All 100 tasks are included in the zero-shot OOD suite.
- Results organization: G1-OP and G2-90D results appear in separate four-column blocks.The columns correspond to the nested co-training pools used for scaling analysis.
- Measurement: Every task-level success rate is computed over 10 trials.The same trial count applies across both embodiments and all data scales.
C.3 Instruction-Following Prompts and Results
The grounding study tests instruction-specific object and scene references across controlled Pick and Place prompts. It reports both instruction-following and complete-task success, while descriptor frequencies characterize lexical coverage in two corpora.
- Grounding results: 83.1% Follow Score and 72.9% SR were averaged over 295 grounding-study rollouts.The suite contains 59 instructions, each executed for five trials; Follow Score tests target contact, while SR also requires complete Pick or Place behavior.
- Prompt dimensions: The grounding suite covers object identity, color, size, shape, position, and order references.Instructions include direct and relational references such as left/right, near/far, middle, and ordinal positions.
- Measurement: Instruction-level Follow and SR values are percentages of five trials, so each reported value is a multiple of 20%.Table 9 presents these measures for individual instructions.
- Descriptor analysis: Descriptor frequencies are lexical matches measured separately in RefCOCOg and the GE-Act 2.0 training corpus.Non-object descriptors are counted independently, so the reported shares are not a partition.