PaperlayerView exact revision on arXiv

Source-linked AI summary

AgiBot World Colosseo: A Large-scale Manipulation Platform for Scalable and Intelligent Embodied Systems

AgiBot-World-Contributors, Qingwen Bu, Jisong Cai, Li Chen, Xiuqi Cui, Yan Ding, Siyuan Feng, Shenyuan Gao, Xindong He, Xuan Hu, Xu Huang, Shu Jiang, Yuxin Jiang, Cheng Jing, Hongyang Li, Jialu Li, Chiming Liu, Yi Liu, Yuxiang Lu, Jianlan Luo, Ping Luo, Yao Mu, Yuehan Niu, Yixuan Pan, Jiangmiao Pang, Yu Qiao, Guanghui Ren, Cheng Ruan, Jiaqi Shan, Yongjian Shen, Chengshi Shi, Mingkang Shi, Modi Shi, Chonghao Sima, Jianheng Song, Huijie Wang, Wenhao Wang, Dafeng Wei, Chengen Xie, Guo Xu, Junchi Yan, Cunbiao Yang, Lei Yang, Shukai Yang, Maoqing Yao, Jia Zeng, Chi Zhang, Qinglin Zhang, Bin Zhao, Chengyue Zhao, Jiaqi Zhao, Jianchao Zhu

arXiv:2503.06669v4cs.ROcs.CVcs.LG

TL;DR

Real-world robotic manipulation still lacks high-quality, diverse data for open-set tasks requiring dexterity and generalization. AgiBot World provides a large-scale robot learning platform and GO-1 generalist policy, with policies pretrained on its data improving completion scores by 0.30 in-distribution and 0.29 out-of-distribution.

  • Problem

    Robotics lacks high-quality, diverse data for open-set real-world manipulation tasks requiring physical dexterity and generalization.

  • Method

    The paper constructs AgiBot World with standardized human-verified collection and develops GO-1, a generalist policy using latent action representations.

  • Results

    Policies pretrained on AgiBot World increase completion scores by 0.30 in-distribution and 0.29 out-of-distribution, while GO-1 outperforms prior generalist policies across varied tasks.

  • Takeaways & Limitations

    AgiBot World’s open-source dataset, toolchain, and models provide a foundation for research on scalable, general-purpose robotic intelligence.

Abstract

from arXiv · show

We explore how scalable robot data can address real-world challenges for generalized robotic manipulation. Introducing AgiBot World, a large-scale platform comprising over 1 million trajectories across 217 tasks in five deployment scenarios, we achieve an order-of-magnitude increase in data scale compared to existing datasets. Accelerated by a standardized collection pipeline with human-in-the-loop verification, AgiBot World guarantees high-quality and diverse data distribution. It is extensible from grippers to dexterous hands and visuo-tactile sensors for fine-grained skill acquisition. Building on top of data, we introduce Genie Operator-1 (GO-1), a novel generalist policy that leverages latent action representations to maximize data utilization, demonstrating predictable performance scaling with increased data volume. Policies pre-trained on our dataset achieve an average performance improvement of 30% over those trained on Open X-Embodiment, both in in-domain and out-of-distribution scenarios. GO-1 exhibits exceptional capability in real-world dexterous and long-horizon tasks, achieving over 60% success rate on complex tasks and outperforming prior RDT approach by 32%. By open-sourcing the dataset, tools, and models, we aim to democratize access to large-scale, high-quality robot data, advancing the pursuit of scalable and general-purpose intelligence.

A Universe of Robot Data Human-in-the-loop Versatile Scenarios · I. INTRODUCTION

AgiBot World Colosseo is presented as a full-stack, open-sourced platform addressing the scarcity and limitations of high-quality robot data through large-scale, diverse, human-verified real-world collection. Its dataset and GO-1 policy improve policy performance and generalization while supporting scalable embodied intelligence, though evaluation remains limited to real-world scenarios.

  • I. INTRODUCTION: AgiBot World targets robotics’ data bottleneck, where open-set real-world manipulation spans diverse tasks beyond controlled-lab pick-and-place settings.The introduction contrasts robotics’ data-collection difficulty with progress in natural language processing and computer vision.
  • I. INTRODUCTION: Prior large-scale datasets remain limited by naive short-horizon tasks, weak out-of-domain generalization, insufficient human-feedback quality assurance, or constrained hardware.The passage specifically discusses Open X-Embodiment and DROID as examples of these limitations.
  • A Universe of Robot Data Human-in-the-loop Versatile Scenarios: AgiBot World spans five authentic deployment domains in a 4000square-meter facility and contains over 1 million trajectories collected from 100 real robots.The platform is designed for bimanual manipulation across domestic, retail, industrial, restaurant, and office environments.
  • I. INTRODUCTION: 30% average success rate improvement is achieved by policies pretrained on AgiBot World compared with policies trained on OXE.This result is reported as an average improvement over the prior large-scale robot dataset.
  • I. INTRODUCTION: 18% higher generalizability is obtained even when using only 1/10 of OXE’s data volume in hours.The finding supports AgiBot World’s effectiveness in bridging controlled laboratory environments and real-world robotic applications.
  • A Universe of Robot Data Human-in-the-loop Versatile Scenarios: Its inclusive pipeline covers scene configuration, task design, collection, and human-in-the-loop verification to ensure high data quality.The dataset is accompanied by open-source tools for policy learning at scale.
  • A Universe of Robot Data Human-in-the-loop Versatile Scenarios: GO-1 uses latent action representations to unlock web-scale pre-training, while the platform open-sources its dataset, toolchain, and pretrained models.The stated aim is to foster community-wide innovation toward scalable, general-purpose robotic intelligence.
  • I. INTRODUCTION: AgiBot World evaluations are conducted only in real-world scenarios, while a simulation environment aligned with real-world deployment is still under development.The planned simulation environment is intended to enable faster and more reproducible evaluation.

II. RELATED WORK

Prior robotic datasets have grown from small teleoperation collections toward million-scale aggregation, but remain constrained by variability, limited quality control, and restricted practical coverage. Recent policy-learning work transfers web-scale knowledge to robotic control, while this work uses VLM adaptation with latent actions for scalable training and long-horizon execution.

  • Data scaling in robotics: Existing datasets remain limited by variability in embodiments and observation perspectives, inconsistent quality, insufficient scale and task practicality, and inadequate quality assurance.DROID increases scene diversity through crowdsourced demonstrations but falls short in data scale and quality control.
  • Policy learning at scale: Robotic foundation models increasingly rely on diverse, large-scale training, including web-scale video and large end-to-end models, to improve general-purpose capabilities.Web-scale video is used because action-labeled robot datasets remain limited in scale.
  • Policy learning at scale: This work adapts vision-language models with latent actions, combining human videos and robot data for scalable training and improving long-horizon task execution and policy-learning efficiency.The latent action planner is presented as enabling more efficient policy learning and improving existing generalist policies.

III. AGIBOT WORLD: PLATFORM AND DATA · A. Hardware: A Versatile Humanoid Robot · B. Data Collection: Protocol and Quality

AgiBot World combines a standardized, versatile humanoid platform with a staged, human-verified collection protocol. Its open-source dataset scales to 1,001,552 trajectories across diverse tasks, skills, and scenes while retaining recoverable failures and iteratively refining data quality.

  • III. AGIBOT WORLD: PLATFORM AND DATA: 1,001,552 trajectories spanning 217 specific tasks, 87 skills, and 106 scenes form AgiBot World’s large-scale manipulation dataset.The latest version contains 2976.4 hours of data collected by more than 100 homogeneous robots.
  • A. Hardware: A Versatile Humanoid Robot: The platform standardizes distributed collection with dual 7-DoF arms, a mobile chassis, visuo-tactile sensors, and durable 6-DoF dexterous hands.These hardware choices support reproducible results and challenging manipulation tasks.
  • B. Data Collection: Protocol and Quality: Human feedback iteratively refines the pipeline by cycling through demonstration collection, policy training, deployment, and targeted correction of data gaps or inefficiencies.The framework uses human input during collection and post-collection review.
  • A. Hardware: A Versatile Humanoid Robot: Modular end effectors switch between a standard gripper, a 6-DoF dexterous hand, and visuo-tactile grippers according to task requirements.Eight cameras provide RGB-D and fisheye observations across front, end-effector, and rear viewpoints.
  • A. Hardware: A Versatile Humanoid Robot: VR headset and whole-body motion-capture systems teleoperate the robot, mapping operator gestures and controls to end-effector, joint, base, and body movements.The VR controller’s dexterous-hand control is limited to a few predefined gestures.
  • B. Data Collection: Protocol and Quality: Collection proceeds through task-feasibility validation, standardized teleoperation, and local initial validity verification before cloud upload.Skilled teleoperators arrange the initial scene after reviewing collection standards.
  • B. Data Collection: Protocol and Quality: Post-processing verifies each episode against collection standards, checks completeness such as missing frames, and adds language annotations.Data annotators perform the episode-level review after cloud upload.
  • B. Data Collection: Protocol and Quality: Failure-recovery trajectories are retained rather than discarded, with failure reasons and timestamps manually annotated after teleoperators recover from errors.This preserves demonstrations that successfully complete tasks despite intermediate mistakes.

C. Dataset Statistics and Analysis: Beyond Scale

AgiBot World extends dataset diversity beyond scale through broad real-world coverage, long-horizon trajectories, and deliberately comprehensive manipulation-skill representation. Its collection facility spans over 4,000 square meters and contains over 3,000 unique objects across varied scenes.

  • Dataset diversity: Over 4,000 square meters and over 3,000 unique objects support diverse scenes designed to reflect real-world settings.The facility is intended to ensure both scale and diversity for generalizable robot policies.
  • Dataset diversity: Five domains—domestic, retail, industrial, restaurant, and office—cover diverse robotic application environments and scene categories.Examples include bedrooms, kitchens, living rooms, balconies, shelving units, and fresh-produce areas.
  • Long-horizon manipulation: 30s to 60s trajectories make long-horizon manipulation central to AgiBot World, compared with DROID’s 5s to 20s and OXE v1.0’s predominantly 5s trajectories.The dataset emphasizes continuous, complete tasks composed of multiple atomic skills, such as “make a coffee”.
  • Comprehensive skill coverage: Beyond dominant “pick-and-place” tasks, the dataset includes less frequently used skills such as “chop” and “plug”.This design provides broader skill coverage and sufficient data for robust policy learning.

A. Latent Action Model · B. Latent Planner

The latent action model learns universal, discretized action representations from heterogeneous videos, while the latent planner uses a VLM to predict these actions for embodiment-agnostic, long-horizon robot planning. This latent action space bridges visual-language inputs and robot actions while enabling efficient adaptation of general-purpose VLMs.

  • A. Latent Action Model: The latent action model expands limited action-labeled robot data by modeling inverse dynamics across consecutive frames from human videos and cross-embodiment robot data.It transfers real-world dynamics from heterogeneous sources into universal manipulation representations.
  • A. Latent Action Model: The model encodes latent actions with an inverse-dynamics encoder and reconstructs future frames using a forward-dynamics decoder.The encoder uses a spatial-temporal transformer with causal temporal masks, while the decoder is a spatial transformer.
  • A. Latent Action Model: k = 4 latent action tokens are quantized with a VQ-VAE objective and a codebook of size |C|.The tokens are discretized representations used by the latent action model.
  • B. Latent Planner: The ViLLA model combines a web-scale-pretrained VLM with a latent planner to support scene understanding, general reasoning, and embodiment-agnostic planning.InternVL2.5-2B is used as the VLM backbone, with multiview images encoded by InternViT before projection into language space.
  • B. Latent Planner: GO-1’s latent action model learns universal action representations from web-scale human videos, including Ego4D, and quantizes them into discrete latent action tokens.The latent planner then predicts these tokens to connect image-text inputs with robot actions generated by the action expert.
  • B. Latent Planner: Given multiview images and a language instruction, the latent planner predicts latent action tokens for the ongoing task.The observations typically come from the head, left-wrist, and right-wrist views, with supervision produced by the LAM encoder from the head view.
  • B. Latent Planner: The latent action space is orders of magnitude smaller than OpenVLA’s discretized low-level action space, facilitating efficient adaptation of general-purpose VLMs into robot policies.This compact representation supports temporal reasoning and embodiment-agnostic planning before low-level action generation.

C. Action Expert

Stage 3 adds a diffusion-based action expert for high-frequency, dexterous manipulation, modeling continuous low-level actions through iterative denoising. Within GO-1, it follows latent-token prediction and converts the resulting representations into final control signals.

  • Action Expert: The action expert uses a diffusion objective to model the continuous distribution of low-level actions for high-frequency and dexterous manipulation.It is integrated in Stage 3 of the system.
  • Action Expert: Unlike the latent planner’s masked language modeling, the action expert regresses low-level actions through iterative denoising.Both components share the same architectural framework but have different objectives.
  • Action Expert: The expert decodes low-level action chunks A_t = [a_t,a_t+1,...,a_t+H] over H = 30 timesteps using proprioceptive state.The chunk covers an interval of 30 timesteps.
  • Action Expert: GO-1 first predicts k latent action tokens, then conditions the action expert’s denoising process to produce final control signals.The VLM, latent planner, and action expert operate synergistically during inference.

V. EXPERIMENT AND ANALYSIS · A. Experiment Setup · 1) Evaluation Tasks

The evaluation measures real-world policy performance across diverse manipulation capabilities and tests generalization to unseen positions, visual distractors, and language variations. It also compares GO-1 and AgiBot World pretraining against established baselines.

  • V. EXPERIMENT AND ANALYSIS: Policies pretrained on different data sources are evaluated for real-world performance, including the contribution of GO-1 to policy learning.
  • 1) Evaluation Tasks: The evaluation covers tool usage, deformable-object manipulation, human-robot interaction, and language following through tasks including Wipe Table, Fold Shorts, Handover Bottle, and Restock Beverage.
  • 1) Evaluation Tasks: 2 unseen scenarios per task test position generalization, visual distractors, and language generalization.These scenarios provide thorough generalization evaluations for the policies.
  • 1) Evaluation Tasks: GO-1 outperforms RDT-1B and the baseline without the latent planner by a large margin across all evaluated tasks.All policies are pretrained on AgiBot World beta.
  • 1) Evaluation Tasks: The latent planner boosts performance on complex Fold Shorts manipulation and substantially improves Restock Beverage generalizability.
  • 1) Evaluation Tasks: 0.77 v.s. 0.47 in seen scenarios and 0.67 v.s. 0.38 in out-of-distribution scenarios are achieved by AgiBot World- versus OXE-pretrained policies.

2) Implementation Details · B. Does AgiBot World boost policy learning at scale?

AgiBot World improves policy learning at scale, with models pretrained on the dataset achieving higher completion scores in both in-distribution and out-of-distribution settings. Even the smaller alpha dataset outperforms OXE, highlighting the importance of data quality alongside volume.

  • 2) Implementation Details: The alpha dataset contains roughly 14% of the trajectories in the full beta version and includes only partial tasks.It is described as an early-stage subset.
  • 2) Implementation Details: Following third-stage pre-training, GO-1 demonstrates basic competency in task completion.The model is further fine-tuned with high-quality, task-specific demonstrations unless otherwise specified.
  • B. Does AgiBot World boost policy learning at scale?: RDT is used as the open-source policy model for evaluating how AgiBot World supports policy learning.Task completion scores are reported for three tasks in Fig. 6.
  • B. Does AgiBot World boost policy learning at scale?: Nearly tripling performance, AgiBot World pre-training substantially improves completion on the “Table Bussing” task.This is the most notable task-level improvement reported in the comparison.
  • B. Does AgiBot World boost policy learning at scale?: 0.30 and 0.29 are the average completion-score increases for in-distribution and out-of-distribution setups, respectively.The gains are reported across the evaluated tasks.
  • B. Does AgiBot World boost policy learning at scale?: 236h compared to ∼2000h, the AgiBot World alpha dataset uses much less data than OXE while achieving a higher success rate.The result underscores the exceptional data quality of AgiBot World.

C. Is GO-1 a more capable generalist policy? … VI. CONCLUSION

GO-1 is evaluated as a capable generalist policy that outperforms prior policies across varied tasks, while its performance scales predictably with pre-training data and benefits from human-verified data. AgiBot World concludes as an open-source ecosystem designed to expand access to large-scale, diverse, high-quality robot learning resources.

  • C. Is GO-1 a more capable generalist policy?: GO-1 significantly outperforms RDT and π0 across five tasks varying in visual richness and task horizon.Results average 30 trials per task, including 10 seen-setup trials and 20 trials with variations or distractions.
  • C. Is GO-1 a more capable generalist policy?: GO-1 is especially strong on “Pour Water” robustness to object positions and “Restock Beverage” instruction following.These tasks highlight robustness and instruction-following capabilities, respectively.
  • D. Does GO-1’s ability scale with data size?: 9.2k to 1M training trajectories are used to test whether policy capability follows a power-law relationship with pre-training data size.The analysis uses 10% alpha, 100% alpha, and beta dataset subsets and evaluates out-of-the-box performance on four seen pre-training tasks.
  • D. Does GO-1’s ability scale with data size?: Policy performance exhibits a predictable power-law scaling relationship with the number of training trajectories.This scaling analysis varies the pre-training dataset size across the specified alpha and beta subsets.
  • E. How does data quality impact policy learning?: 528 verified trajectories and 482 unverified trajectories are compared in a Wipe Table ablation using an RDT model.Verification denotes the human-in-the-loop quality-assurance method.
  • E. How does data quality impact policy learning?: Larger quantity does not necessarily translate to improved performance, underscoring the role of human-in-the-loop verification in data quality.The comparison fine-tunes RDT on verified and unverified Wipe Table data.
  • VI. CONCLUSION: AgiBot World is introduced as an open-source ecosystem democratizing access to large-scale, high-quality robot learning datasets, toolchains, and foundation models.Its stated goal is to advance embodied general intelligence through community collaboration.
  • VI. CONCLUSION: The dataset distinguishes itself through scale, diversity, quality, and carefully crafted tasks, while evaluations confirm improved policy performance and generalizability.The conclusion presents these evaluations as evidence of AgiBot World’s value.

APPENDIX … Data Curation

The appendix assigns project-wide, algorithmic, product, manuscript, and data-curation responsibilities to named contributors. These roles span the project’s full lifecycle, from data collection and quality checks through methodology, deployment, evaluation, writing, and ecosystem development.

  • Core Contributors: Core Contributors covered the project’s full span, including data collection, algorithms, experiments, and writing.
  • APPENDIX: The listed contributions collectively cover research execution, technical development, productization, manuscript preparation, and dataset curation.
  • Algorithm: Algorithm contributors developed the technical roadmap, methodology, pre-training, post-training, deployment, and evaluation.
  • Product & Ecosystem: Product & Ecosystem contributors handled system architecture design, project management, and community engagement.
  • Manuscript Preparation: Manuscript Preparation contributors were responsible for the manuscript outline, writing, and revising.
  • Data Curation: Data Curation contributors performed data collection and quality checks.

Hardware & Software Development … July 2025: IROS camera-ready update

The supplied update records the platform’s hardware and embedded-software development, staged dataset and manuscript releases, and an additional comparison with π0 in the July 2025 camera-ready version.

  • Hardware & Software Development: Hardware design and embedded software development are attributed to Yuehan Niu, Cheng Jing, Mingkang Shi, Chi Zhang, Qinglin Zhang, Cunbiao Yang, Wenhao Wang, and Xuan Hu.
  • Project Co-lead and Advising: The supplied passages identify hardware and embedded-software development but provide no separate details for Project Co-lead and Advising.
  • Jan 2025: agibot-world alpha version release: 92,214 trajectories comprise a sub-split representing around 10% of the dataset.
  • March 2025: full data release and technical report: The March 2025 release records an initial submission to arXiv and a research blog.
  • March 2025: full data release and technical report: The March 2025 version includes the complete manuscript, covering introduction, methodology, results, discussion, and appendix.
Loading 2503.06669v4…