PaperlayerView exact revision on arXiv

Source-linked AI summary

AgiBot World Colosseo: A Large-scale Manipulation Platform for Scalable and Intelligent Embodied Systems

AgiBot-World-Contributors, Qingwen Bu, Jisong Cai, Li Chen, Xiuqi Cui, Yan Ding, Siyuan Feng, Shenyuan Gao, Xindong He, Xuan Hu, Xu Huang, Shu Jiang, Yuxin Jiang, Cheng Jing, Hongyang Li, Jialu Li, Chiming Liu, Yi Liu, Yuxiang Lu, Jianlan Luo, Ping Luo, Yao Mu, Yuehan Niu, Yixuan Pan, Jiangmiao Pang, Yu Qiao, Guanghui Ren, Cheng Ruan, Jiaqi Shan, Yongjian Shen, Chengshi Shi, Mingkang Shi, Modi Shi, Chonghao Sima, Jianheng Song, Huijie Wang, Wenhao Wang, Dafeng Wei, Chengen Xie, Guo Xu, Junchi Yan, Cunbiao Yang, Lei Yang, Shukai Yang, Maoqing Yao, Jia Zeng, Chi Zhang, Qinglin Zhang, Bin Zhao, Chengyue Zhao, Jiaqi Zhao, Jianchao Zhu

arXiv:2503.06669v4cs.ROcs.CVcs.LG

TL;DR

Real-world robotic manipulation remains challenging across diverse environments, while collecting high-quality robot data is difficult. This paper introduces AgiBot World and the latent-action GO-1 policy; policies pretrained on its dataset improve average task-completion scores by 0.30 in-distribution and 0.29 out-of-distribution.

  • Problem

    Real-world robotic manipulation remains challenging across fine-grained interaction, mobile manipulation, collaboration, and generalization across diverse environments, while high-quality data collection is difficult.

    Source: §I · p. 1

  • Method

    The paper builds AgiBot World, a large-scale dataset collected through a human-verified pipeline, and introduces GO-1, a generalist policy using latent action representations.

    Source: §I · p. 2

  • Results

    Across in- and out-of-distribution setups, AgiBot World pretraining raises average task-completion scores by 0.30 and 0.29, respectively.

    Source: B. Does AgiBot World boost policy… · p. 7

  • Takeaways & Limitations

    The open-source ecosystem offers researchers large-scale robot data, tools, and pretrained models for community collaboration on embodied intelligence.

    Source: §I · p. 2

Abstract

from arXiv · show

We explore how scalable robot data can address real-world challenges for generalized robotic manipulation. Introducing AgiBot World, a large-scale platform comprising over 1 million trajectories across 217 tasks in five deployment scenarios, we achieve an order-of-magnitude increase in data scale compared to existing datasets. Accelerated by a standardized collection pipeline with human-in-the-loop verification, AgiBot World guarantees high-quality and diverse data distribution. It is extensible from grippers to dexterous hands and visuo-tactile sensors for fine-grained skill acquisition. Building on top of data, we introduce Genie Operator-1 (GO-1), a novel generalist policy that leverages latent action representations to maximize data utilization, demonstrating predictable performance scaling with increased data volume. Policies pre-trained on our dataset achieve an average performance improvement of 30% over those trained on Open X-Embodiment, both in in-domain and out-of-distribution scenarios. GO-1 exhibits exceptional capability in real-world dexterous and long-horizon tasks, achieving over 60% success rate on complex tasks and outperforming prior RDT approach by 32%. By open-sourcing the dataset, tools, and models, we aim to democratize access to large-scale, high-quality robot data, advancing the pursuit of scalable and general-purpose intelligence.

A Universe of Robot Data Human-in-the-loop Versatile Scenarios · I. INTRODUCTION

AgiBot World combines a large, standardized real-world robot dataset with GO-1, a latent-action generalist policy. Policies pretrained on the dataset improve average success rate over OXE, and the project open-sources its data, tools, and models.

A Universe of Robot Data Human-in-the-loop Versatile Scenarios

  • AgiBot World is an open-source manipulation platform comprising data, models, benchmarks, and an ecosystem.Source: Figure 1 · p. 1

I. INTRODUCTION

  • Existing robot datasets remain limited by fragmented, heterogeneous data and short-horizon or controlled tasks, leaving real-world manipulation underrepresented.The introduction identifies collection quality and standardization as central challenges for generalization across environments and scenarios.Source: §I · pp. 1–2
  • AgiBot World contains over 1 million trajectories from 100 robots across five domains and over 100 real-world scenarios, collected with a standardized pipeline and human-in-the-loop verification.Its 4000-square-meter facility supports mobile humanoid robots with whole-body control, dexterous hands, visuo-tactile sensors, and multimodal episode annotations.Source: §I · p. 2
  • 30% average success-rate improvement over OXE; with only 1/10 of OXE’s data volume in hours, pretrained-policy generalizability rises by 18%.The paper also reports that GO-1 outperforms prior generalist policies, including RDT, and a variant without the latent action planner.Source: §I · p. 2
  • GO-1 uses latent action representations to learn from heterogeneous data, bridging general-purpose vision-language models with robotic sequential decision-making.It is pretrained on web-scale data including human videos and the robot dataset, and the paper reports performance scalability as dataset size increases.Source: §I · p. 2
  • The authors open-source the dataset, toolchain, and pretrained models to support research across household and industrial applications.Source: §I · p. 2
  • Evaluations are currently limited to real-world scenarios; a simulation environment aligned with the real-world setup is under development.The planned simulation environment is intended to enable faster, reproducible evaluation.Source: §I · p. 2

II. RELATED WORK

AgiBot World is positioned as a full-stack platform that addresses limitations in dataset scale, diversity, practicality, and quality assurance. Its policy-learning approach builds on prior large-scale robotics work by combining web-scale knowledge, vision-language models, and latent actions for robotic control.

Data scaling in robotics

  • RoboTurk and BridgeData provide 2.1k and 7.2k trajectories, respectively; RT-1 has 130k, while Open X-Embodiment exceeds 1.4 million.Open X-Embodiment's embodiment and viewpoint variability and inconsistent data quality, along with DROID 's limited scale and quality control, constrain effectiveness; Pumacay et al. is a simulation benchmark, unlike AgiBot World's full-stack platform.Source: §II · pp. 2–3
  • AgiBot World offers over 1 million trajectories across five scenarios, with humanoid robots, visuo-tactile sensors, and dexterous hands, distinguishing it from prior datasets.The platform also includes industrial and retail environments, extensive item-, scene-, skill-, and task-level annotations, and failure-recovery data with annotated error states; policy deployment results inform collection-protocol improvements.Source: §II · p. 3, Table I · p. 3

Policy learning at scale

  • Prior policy-learning approaches use web-scale video,, or robot trajectories,,,; RDT, π0 [26], and LAPA explore distinct pretraining strategies.RDT pre-trains on heterogeneous multirobot datasets and fine-tunes on over 6k dual-arm trajectories; π0 [26] pairs a pre-trained VLM backbone with a flow-based action expert, while LAPA 's latent planning does not persist downstream.Source: §II · p. 3
  • AgiBot World transfers web-scale knowledge to robotic control by adapting vision-language models with latent actions, using human videos and robot data for scalable training.The work presents a latent action planner as a way to improve long-horizon task execution and policy-learning efficiency relative to existing generalist policies.Source: §II · p. 3

III. AGIBOT WORLD: PLATFORM AND DATA · A. Hardware: A Versatile Humanoid Robot · B. Data Collection: Protocol and Quality

AgiBot World combines a large, diverse real-world manipulation dataset with standardized humanoid hardware and a human-in-the-loop collection process designed to improve data quality. Its platform supports varied end effectors, sensing, and teleoperation, while retaining annotated failure-recovery trajectories and releasing resources for community use.

III. AGIBOT WORLD: PLATFORM AND DATA

  • The latest release contains 1,001,552 trajectories across 217 tasks, 87 skills, and 106 scenes, collected by more than 100 homogeneous robots over 2,976.4 hours.The dataset emphasizes real-world scenarios including dual-arm manipulation, dexterous hands, and collaborative tasks rather than only basic tabletop tasks.Source: §III · p. 3
  • The dataset is released under CC BY-NC-SA 4.0 alongside model checkpoints and code.The authors state that they plan to release all resources to support community development on AgiBot World.Source: §III · p. 3

A. Hardware: A Versatile Humanoid Robot

  • The standardized robot has dual 7-DoF arms, a mobile chassis, an adjustable waist, durable 6-DoF dexterous hands, and visuo-tactile sensing.Modular end effectors support either a standard gripper or dexterous hand, with visuo-tactile grippers available when tasks require tactile feedback.Source: A. Hardware: A Versatile Humanoid Robot · pp. 3–4
  • Eight cameras capture image observations, while joint and end-effector positions are recorded as proprioceptive states at 30 Hz.The camera setup includes front RGB-D and fisheye views, end-effector cameras, and two rear fisheye cameras.Source: A. Hardware: A Versatile Humanoid Robot · p. 4
  • VR teleoperation maps hand gestures to end-effector motion but limits dexterous hands to predefined gestures; motion capture maps human finger and body joints for more nuanced control.Motion capture enables individual finger movements, torso pose, and head orientation for complex manipulation.Source: A. Hardware: A Versatile Humanoid Robot · p. 4

B. Data Collection: Protocol and Quality

  • The collection process validates task feasibility and standards, records demonstrations with local completeness checks, then verifies episodes and adds language annotations during post-processing.Data is uploaded to the cloud after initial checks such as confirming that frames are not missing.Source: B. Data Collection: Protocol and Quality · p. 4
  • The dataset retains recovered failure trajectories—about one percent of the total—with manually annotated reasons and timestamps for policy alignment [28] and failure reflection.These are episodes in which teleoperators recover from errors and complete tasks without fully reconfiguring the setup.Source: B. Data Collection: Protocol and Quality · p. 4
  • Human feedback and policy deployment iteratively refine collection; after pauses exposed inconsistent transitions and idle time, protocols were revised and idle frames removed.The cycle collects demonstrations, trains and deploys a policy, then uses performance and annotator feedback to address data gaps or inefficiencies.Source: B. Data Collection: Protocol and Quality · p. 4, Figure 2 · p. 4

C. Dataset Statistics and Analysis: Beyond Scale

AgiBot World combines broad real-world scene and object coverage with longer, multi-skill trajectories and intentionally balanced coverage of valuable atomic skills.

Dataset breadth

  • The collection facility spans over 4,000 square meters, with over 3,000 objects across five domains: domestic, retail, industrial, restaurant, and office.Scenes are designed to reflect real-world settings, with categories such as bedrooms, kitchens, shelving units, and fresh produce sections.Source: C. Dataset Statistics and Analysis:… · p. 4, Figure 3 · p. 5

Long-horizon manipulation

  • Most trajectories span 30s to 60s, compared with DROID’s 5s to 20s and OXE v1.0’s predominantly 5s trajectories.The dataset emphasizes complete tasks composed of multiple atomic skills; key-frame and instruction annotations mark each sub-step.Source: Figure 3 · p. 5, C. Dataset Statistics and Analysis:… · p. 5

Comprehensive skill coverage

  • Each valuable atomic skill is supported by at least 100 trajectories, including less common skills such as “chop” and “plug.”Although generic skills such as “pick-and-place” dominate, the task design deliberately includes less frequently used skills to support robust policy learning.Source: Figure 3 · p. 5, C. Dataset Statistics and Analysis:… · p. 5

A. Latent Action Model · B. Latent Planner

The latent action model (LAM) learns transferable action representations from unlabeled human videos and cross-embodiment robot data, while the latent planner uses a vision-language model to predict these representations from observations and task instructions. This shared latent space connects general reasoning and long-horizon planning to robot actions and supports efficient policy adaptation.

A. Latent Action Model

  • LAM incorporates unlabeled internet-scale human videos and cross-embodiment robot data, using inverse dynamics to transfer real-world dynamics into universal manipulation knowledge.This expands the action-data pool beyond action-labeled robot demonstrations.Source: A. Latent Action Model · p. 5
  • LAM encodes consecutive frames with an inverse-dynamics model and decodes the next frame from the initial frame and discretized latent action tokens.The encoder uses a spatial-temporal transformer with causal temporal masks [31], following Bruce et al.; the decoder uses a spatial transformer.Source: A. Latent Action Model · p. 5
  • The latent action tokens use k = 4 and are quantized with a VQ-VAE objective and a codebook of size |C|.The supplied passage does not give a numerical codebook size.Source: A. Latent Action Model · p. 5

B. Latent Planner

  • ViLLA combines a web-scale-pretrained VLM with an embodiment-agnostic latent planner; its InternVL2.5-2B [33] backbone encodes multiview observations through InternViT and conditions 24 transformer layers with full bidirectional attention.The paper motivates the two-billion-parameter backbone by preliminary robotic experiments and prior studies, [26].Source: B. Latent Planner · pp. 5–6
  • Given multiview images and a task instruction, the latent planner predicts latent action tokens supervised by LAM encoding of the head view, bridging image-text inputs to robot actions.The planner performs temporal reasoning through latent action prediction.Source: Figure 4 · p. 6, B. Latent Planner · p. 6
  • The latent action space is orders of magnitude smaller than OpenVLA’s discretized low-level action space, facilitating efficient adaptation of general-purpose VLMs into robot policies.Source: B. Latent Planner · p. 6

C. Action Expert

Stage 3 adds a diffusion-based action expert to GO-1 for high-frequency, dexterous manipulation, complementing the latent planner by generating continuous low-level actions through iterative denoising.

C. Action Expert

  • The action expert models continuous low-level actions with diffusion, unlike the latent planner’s discretized action tokens generated through masked language modeling.Both modules use the same architectural framework and are hierarchically conditioned on preceding modules, including the action expert itself.Source: C. Action Expert · p. 6
  • The action expert decodes low-level action chunks over H = 30 timesteps using proprioceptive state.The chunk is denoted A_t = [a_t, a_{t+1}, ..., a_{t+H}].Source: C. Action Expert · p. 6
  • At inference, GO-1 combines the VLM, latent planner, and action expert: it predicts k latent action tokens, then conditions denoising to produce final control signals.Source: C. Action Expert · p. 6

V. EXPERIMENT AND ANALYSIS · A. Experiment Setup · 1) Evaluation Tasks

The evaluation tests real-world policies across varied tasks and generalization scenarios; policies pretrained on AgiBot World outperform OXE-pretrained policies in seen and out-of-distribution settings.

V. EXPERIMENT AND ANALYSIS

  • The real-world evaluation compares policies pretrained on different data sources, including AgiBot World, to assess GO-1 policy learning.Source: §V · p. 6

1) Evaluation Tasks

  • Evaluation tasks span tool use, deformable-object manipulation, human-robot interaction, and language following, including Wipe Table, Fold Shorts, Handover Bottle, and Restock Beverage.Source: 1) Evaluation Tasks · p. 6
  • Two unseen scenarios per task test position generalization, visual distractors, and language generalization.Source: 1) Evaluation Tasks · p. 6
  • The six evaluated tasks are Restock Bag, Table Bussing, Pour Water, Restock Beverage, Fold Shorts, and Wipe Table.Source: 1) Evaluation Tasks · p. 6

A. Experiment Setup

  • Normalized scores average 10 rollouts per task, scenario, and method; episodes receive 1.0 for full success and fractional scores for partial success.Source: 1) Evaluation Tasks · p. 6

1) Evaluation Tasks

  • GO-1 outperforms RDT-1B and the no-latent-planner baseline across tasks; its planner helps on Fold Shorts and improves Restock Beverage generalization.All policies were pretrained on AgiBot World beta.Source: Figure 5 · p. 7
  • 0.77 vs. 0.47 in seen scenarios and 0.67 vs. 0.38 out of distribution: AgiBot World-pretrained policies outperform OXE-pretrained policies.These comparisons are reported in Fig. 6.Source: Figure 6 · p. 7

2) Implementation Details · B. Does AgiBot World boost policy learning at scale?

AgiBot World pretraining improves policy completion scores in both in-distribution and out-of-distribution setups, while its smaller alpha dataset achieves higher success than OXE. The section also specifies GO-1’s pretraining and fine-tuning setup.

2) Implementation Details

  • The alpha dataset is an early-stage subset with partial tasks and roughly 14% of the trajectories in the full beta version.Source: 2) Implementation Details · p. 7
  • After third-stage pretraining, GO-1 exhibits basic competency in task completion.Source: 2) Implementation Details · p. 7
  • Unless otherwise specified, GO-1 is fine-tuned with task-specific demonstrations using a 2e-5 learning rate, batch size 768, and 30,000 optimization steps.The fine-tuning adapts the model to new evaluation tasks.Source: 2) Implementation Details · p. 7

B. Does AgiBot World boost policy learning at scale?

C. Is GO-1 a more capable generalist policy? … VI. CONCLUSION

GO-1 outperforms other generalist policies in the evaluated tasks and scales predictably with training-data volume, while human verification improves learning from a smaller dataset. The conclusion presents AgiBot World and GO-1 as an open-source ecosystem for advancing embodied general intelligence.

C. Is GO-1 a more capable generalist policy?

  • GO-1 significantly outperforms RDT and π0 across five tasks, and adding the latent planner improves average task completion score by 0.12.The evaluation averages 30 trials per task: 10 in a seen setup and 20 under variations or distractions.Source: C. Is GO-1 a more capable generalist… · p. 7
  • GO-1 is evaluated on five tasks spanning varying visual richness and task horizons, with trials split between a seen setup and variations or distractions.Each task is evaluated over 30 trials: 10 in the seen setup and 20 under variations or distractions.Source: C. Is GO-1 a more capable generalist… · p. 7

D. Does GO-1’s ability scale with data size?

  • GO-1’s performance shows predictable power-law scaling with pre-training trajectories, supported by a Pearson correlation coefficient of r = 0.97.The analysis uses 9.2k to 1M training trajectories and evaluates out-of-the-box performance on four seen pre-training tasks.Source: D. Does GO-1’s ability scale with data… · p. 7

VI. CONCLUSION

  • AgiBot World is presented as an open-source ecosystem of large-scale, diverse, high-quality robot-learning data, toolchains, and foundation models.The conclusion frames community collaboration as a way to advance embodied general intelligence.Source: §VI · p. 8
  • GO-1 uses latent actions for web-scale pre-training and is described as excelling on real-world complex tasks while scaling with increased data volume.The conclusion also says GO-1 outperforms existing generalist policies.Source: §VI · p. 8

APPENDIX … Data Curation

The appendix assigns project-wide, algorithmic, product, manuscript, and data-curation contributions to named team members.

Core Contributors

  • Qingwen Bu and colleagues contributed across data collection, algorithms, experiments, and writing throughout the project.The listed core contributors are Qingwen Bu, Guanghui Ren, Chiming Liu, Modi Shi, Chengen Xie, Xindong He, Jianheng Song, Yuxiang Lu, and Siyuan Feng.Source: Core Contributors · p. 9

Algorithm

  • Yao Mu, Li Chen, and Yan Ding led the roadmap and methodology contributions.These contributions are listed alongside technical roadmap, model training, and evaluation.Source: Algorithm · p. 9
  • Algorithm contributions also covered pre-training, post-training, and deployment and evaluation, assigned to Yi Liu, Yuxin Jiang, Xiuqi Cui; Ziyu Xiong, Xu Huang, Dafeng Wei; and Guo Xu, respectively.The passage also lists technical roadmap, model training, and evaluation as areas of work.Source: Algorithm · p. 9

Product & Ecosystem

  • Chengyue Zhao and colleagues contributed to system architecture design, project management, and community engagement.The listed contributors are Chengyue Zhao, Shukai Yang, Huijie Wang, Yongjian Shen, Jialu Li, Jiaqi Zhao, Jianchao Zhu, and Jiaqi Shan.Source: Product & Ecosystem · p. 9

Manuscript Preparation

Data Curation

  • Cheng Ruan, Jia Zeng, and Lei Yang contributed to data collection and quality checks.Source: Data Curation · p. 9

Hardware & Software Development … July 2025: IROS camera-ready update

The supplied timeline records hardware and embedded-software development, staged dataset and manuscript releases, and a July 2025 comparison with π0.

Hardware & Software Development

  • Hardware design and embedded software development are credited to Yuehan Niu, Cheng Jing, Mingkang Shi, Chi Zhang, Qinglin Zhang, Cunbiao Yang, Wenhao Wang, and Xuan Hu.Source: Hardware & Software Development · p. 9

Jan 2025: agibot-world alpha version release

March 2025: full data release and technical report

July 2025: IROS camera-ready update

  • The cited passage reports that the authors conduct additional comparisons with π0, but it does not identify the comparison conditions, outcomes, or when those comparisons were added.Source: July 2025: IROS camera-ready update · p. 9

Related papers

Loading 2503.06669v4…