PaperlayerView exact revision on arXiv

Source-linked AI summary

DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset

Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ashwin Balakrishna, Sudeep Dasari, Siddharth Karamcheti, Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yunliang Chen, Kirsty Ellis, Peter David Fagan, Joey Hejna, Masha Itkina, Marion Lepert, Yecheng Jason Ma, Patrick Tree Miller, Jimmy Wu, Suneel Belkhale, Shivin Dass, Huy Ha, Arhan Jain, Abraham Lee, Youngwoon Lee, Marius Memmel, Sungjae Park, Ilija Radosavovic, Kaiyuan Wang, Albert Zhan, Kevin Black, Cheng Chi, Kyle Beltran Hatch, Shan Lin, Jingpei Lu, Jean Mercat, Abdul Rehman, Pannag R Sanketi, Archit Sharma, Cody Simpson, Quan Vuong, Homer Rich Walke, Blake Wulfe, Ted Xiao, Jonathan Heewon Yang, Arefeh Yavary, Tony Z. Zhao, Christopher Agia, Rohan Baijal, Mateo Guaman Castro, Daphne Chen, Qiuyu Chen, Trinity Chung, Jaimyn Drake, Ethan Paul Foster, Jensen Gao, Vitor Guizilini, David Antonio Herrera, Minho Heo, Kyle Hsu, Jiaheng Hu, Muhammad Zubair Irshad, Donovon Jackson, Charlotte Le, Yunshuang Li, Kevin Lin, Roy Lin, Zehan Ma, Abhiram Maddukuri, Suvir Mirchandani, Daniel Morton, Tony Nguyen, Abigail O'Neill, Rosario Scalise, Derick Seale, Victor Son, Stephen Tian, Emi Tran, Andrew E. Wang, Yilin Wu, Annie Xie, Jingyun Yang, Patrick Yin, Yunchu Zhang, Osbert Bastani, Glen Berseth, Jeannette Bohg, Ken Goldberg, Abhinav Gupta, Abhishek Gupta, Dinesh Jayaraman, Joseph J Lim, Jitendra Malik, Roberto Martín-Martín, Subramanian Ramamoorthy, Dorsa Sadigh, Shuran Song, Jiajun Wu, Michael C. Yip, Yuke Zhu, Thomas Kollar, Sergey Levine, Chelsea Finn

arXiv:2403.12945v2cs.RO

TL;DR

Robot manipulation datasets are difficult to collect with sufficient scene and task diversity, limiting the environments represented in training data. DROID introduces a distributed, large-scale dataset and reports improved policy performance, robustness, and generalizability, while releasing resources for reuse and reproduction.

  • Problem

    Robot manipulation data is difficult to collect across diverse environments because it requires recorded observations and actions, logistical and safety planning, and substantial hardware and human labour.

  • Method

    DROID constructs a distributed robot manipulation dataset spanning diverse scenes, tasks, and objects, using data collected across institutions and a shared hardware stack.

  • Results

    20% average improvement in policy performance, robustness, and generalizability was reported across 6 tasks and 4 locations versus approaches using existing large-scale robot datasets.

  • Takeaways & Limitations

    DROID is presented as a valuable data resource for improving policy performance and robustness compared with existing large robot data sources.

  • Takeaways & Limitations

    Automatic calibration still produces false positives, and its zero-shot generalizability to robots other than Franka Panda remains uncertain.

Abstract

from arXiv · show

The creation of large, diverse, high-quality robot manipulation datasets is an important stepping stone on the path toward more capable and robust robotic manipulation policies. However, creating such datasets is challenging: collecting robot manipulation data in diverse environments poses logistical and safety challenges and requires substantial investments in hardware and human labour. As a result, even the most general robot manipulation policies today are mostly trained on data collected in a small number of environments with limited scene and task diversity. In this work, we introduce DROID (Distributed Robot Interaction Dataset), a diverse robot manipulation dataset with 76k demonstration trajectories or 350 hours of interaction data, collected across 564 scenes and 84 tasks by 50 data collectors in North America, Asia, and Europe over the course of 12 months. We demonstrate that training with DROID leads to policies with higher performance and improved generalization ability. We open source the full dataset, policy learning code, and a detailed guide for reproducing our robot hardware setup.

I. INTRODUCTION

DROID addresses the limited scene and task diversity of existing robot manipulation datasets by introducing a large, diverse, distributed dataset. Its evaluations report improved policy performance, robustness, and generalizability.

  • Robot manipulation datasets are difficult to collect at scale because data requires recorded observations and actions, diverse environments create logistical and safety challenges, and supervision demands hardware and human labour.
  • DROID contains 76k demonstration trajectories across 564 scenes, 52 buildings, and 86 tasks, collected through a distributed effort.
  • The dataset uses a shared Franka Panda-based hardware stack across institutions, with three camera views, depth, calibration, and language annotations in each episode.
  • 20% average improvement in policy performance, robustness, and generalizability was reported across 6 tasks and 4 locations versus approaches using existing large-scale robot datasets.
  • DROID releases the dataset, policy-training code, and a guide for reproducing the robot hardware setup.

II. RELATED WORK

Prior robot-learning datasets have increased scale or diversity, but often remain concentrated in few scenes or institutions. DROID targets broader scene diversity while using explicit definitions for tasks and scenes.

  • Large machine-learning datasets show that scale and diversity can support model capability and robustness across computer vision, language, and 3D modeling.
  • Robot-learning datasets include autonomously collected and human-teleoperated data, with prior diversity efforts such as RH20T and BridgeV2 spanning limited tasks or scenes.
  • Most prior datasets collect data in a small number of scenes within a single research lab or building.
  • Open X-Embodiment aggregates more data but still contained around 300 scenes, motivating DROID’s emphasis on substantially greater scene diversity.
  • DROID counts tasks using unique verbs from language instructions and counts scenes only after substantial changes to the robot workspace.
  • Table I compares datasets by task and scene diversity, with DROID offering high diversity in both verbs and scenes.

III. DROID DATA COLLECTION SETUP

DROID is an open-source dataset designed to provide broad variation in scenes, tasks, and objects through a shared robot platform and distributed collection effort.

  • DROID provides 76k robot demonstration trajectories spanning 86 tasks and 564 scenes.
  • The dataset was collected over 12 months using 18 robots and 50 data collectors across 13 institutions.
  • DROID releases the full dataset, visualizer, policy-training code, pretrained checkpoints, and hardware and control-stack reproduction guide.

A. DROID Robot Platform

The DROID robot platform standardizes data collection across institutions while remaining portable and adaptable to changing scenes and tasks.

  • The shared platform was designed to make distributed robot control consistent and reproducible across institutions, locations, and time zones.
  • The platform combines a Franka Panda 7DoF arm, two adjustable Zed 2 stereo cameras, a wrist-mounted Zed Mini, and Oculus Quest 2 teleoperation.
  • The robot and cameras are mounted on a height-adjustable wheeled desk, supporting transportation between scenes and buildings.
  • Quest 2 controllers provide 6D arm-pose and continuous gripper control, and the setup was replicated 18 times across North America, Asia, and Europe.

B. Data Collection Protocol

DROID uses a distributed protocol designed to encourage diverse scene and task collection while standardizing data quality and calibration. Post-processing adds multiple natural-language annotations, and calibration refinement addresses inaccuracies in initial estimates.

  • Data collection protocol: 50 data collectors followed a shared protocol designed to prevent common recording mistakes, encourage diverse data, and allow creative scene and task choices.The protocol specifically addresses camera visibility, teleoperator visibility, diversity, and collector autonomy.
  • Scene and task collection: Each session began by moving the robot to a new scene, selecting adaptable camera views, calibrating extrinsics, and entering potential tasks.Collectors were encouraged to choose cluttered scenes with multiple objects and interesting tasks.
  • Annotation: Each episode received up to three independently labeled natural-language instructions from different crowd workers.Crowdsourcing was performed through the tasq.ai data-labeling platform.
  • Calibration: Initial extrinsic calibration could be inaccurate because of checkerboard misalignment, inconsistent lighting, or errors in the calibration method.The dataset provides automatic post-hoc calibration and quality-assessed calibration matrices to address these inaccuracies.
  • Data analysis: DROID’s verb distribution is analyzed after semantic parsing and GPT-4 de-duplication to represent behavioral diversity.The resulting distribution is visualized as a long-tailed range of verbs.

IV. DROID DATASET ANALYSIS

DROID evaluates diversity across task, object, scene, viewpoint, and interaction-location axes using language annotations, sampled camera frames, and workspace geometry. Compared with existing large-scale datasets, it reports broader coverage across these dimensions.

  • Diversity dimensions: DROID analyzes task, object, scene, viewpoint, and interaction-location diversity as distinct axes relevant to policy generalization.Interaction-location diversity concerns 3D object-interaction positions relative to the robot base.
  • Analysis protocol: The analysis uses one randomly sampled third-person frame and the provided language instruction for each episode, with results consistent across sampled frames.DROID is compared with existing large-scale robot manipulation datasets.
  • Overall findings: DROID significantly increases diversity in tasks, objects, scenes, viewpoints, and interaction locations over existing large-scale robot manipulation datasets.The authors attribute this broad coverage in part to collecting across many people and buildings, switching scenes frequently, and allowing collectors to move freely.
  • Task and object diversity: DROID represents task diversity with GPT-4-deduplicated verb distributions and includes a wide range of everyday objects across categories.Its verb distribution is long-tailed, emphasizing coverage across many behaviors rather than concentration in a few frequent tasks.
  • Scene diversity: DROID contains 564 unique scenes spanning office environments to households, an order of magnitude more than existing large robot manipulation datasets.Scene types were assigned using GPT-4V for DROID and manually determined for existing datasets.
  • Viewpoint and interaction-location diversity: DROID covers 1417 unique camera viewpoints and interactions across a wider portion of the robot workspace than datasets focused on tabletop surfaces.The viewpoint analysis includes intrinsic and extrinsic stereo-camera calibration, while interaction locations use the first gripper-closing point as a proxy.

V. EXPERIMENTS

The experiments test whether DROID’s diverse data improves policy performance and robustness across varied tasks and environments. Across six tasks and four locations, the policies improve success rates and robustness to distractors and novel objects.

  • Experimental question: DROID’s diverse data resource is evaluated for its ability to boost policy performance and robustness across real-world robotic research settings.The evaluation spans simple pick-place behavior through multistage cooking tasks and multiple environmental contexts.
  • Evaluation scope: Policies are evaluated across six tasks and four locations spanning labs, offices, and households.The evaluation is intended to cover a wide spectrum of robot manipulation tasks and environments.
  • Findings: DROID improves policy success rate while increasing robustness to scene changes such as distractors and novel object instances.The experiments use representative state-of-the-art robot policy-learning approaches.

A. Experimental Setup

The experimental setup uses six tasks with distinct horizons, locations, demonstrations, and out-of-distribution modifications. Evaluations use the standardized DROID hardware stack and diffusion-policy training pipeline.

  • Tasks: The six evaluation tasks range from short-horizon waffle-maker closing to long-horizon desk-cleanup and lentil-cooking sequences.Tasks span lab, office, and kitchen settings, with randomized or fixed object positions depending on the task.
  • Out-of-distribution evaluation: Each task includes an out-of-distribution variant involving distractors, novel objects, camera shifts, or other scene changes.Examples include novel objects for toasting and added distractors for desk cleanup and cooking lentils.
  • Evaluation procedure: All evaluation data uses the DROID teleoperation setup and the same standardized policy-learning backbone.This keeps the hardware and training procedure consistent across task evaluations.
  • Policy training: The experiments use diffusion policies because the paper introduces a dataset rather than a new policy-learning method.Diffusion policies are described as denoising diffusion models for action prediction with strong performance across applications.
  • Demonstrations: The tasks use between 50 and 150 demonstrations, depending on the evaluation task.The setup covers simple pick-place behavior through multistage cooking tasks.

B. Does DROID Improve Policy Performance and Robustness?

DROID co-training improves policy performance relative to in-domain-only training and co-training with OXE, especially in out-of-distribution settings. Qualitatively, DROID-trained policies are smoother and more precise and complete challenging multi-step tasks more reliably.

  • Quantitative evaluation: DROID co-training substantially improves policy performance across tasks compared with both in-domain-only training and OXE co-training.The evaluation compares diffusion policies using in-domain demonstrations alone, DROID mixed 50/50 with in-domain data, or OXE mixed 50/50 with in-domain data.
  • Quantitative evaluation: Out-of-distribution evaluations particularly expose the weakness of the No Co-training baseline, while DROID co-training has the strongest overall performance.The comparison uses separate policies for each task and evaluates both in-distribution and out-of-distribution settings.
  • Qualitative evaluation: DROID-trained policies are notably smoother and more precise than comparison policies, especially in challenging out-of-distribution evaluations.This qualitative pattern is reported across the evaluated task rollouts.
  • Qualitative evaluation: In OOD Waffle Closing, DROID is the only method that consistently reaches for the waffle maker, while baselines become confused.The example illustrates a task-specific difference in behavior under distribution shift.
  • Qualitative evaluation: In the multi-step Cook Lentils task, DROID co-training consistently completes all three steps, whereas baselines often fail after one or two steps.The comparison concerns full task completion in the OOD setting.

C. How important is the scene diversity in DROID?

Scene diversity contributes to DROID’s out-of-distribution policy performance beyond dataset size alone. A diverse-scene subset performs better than an equally sized subset concentrated in 20 scenes, while the full dataset matches or exceeds the subsampled results.

  • Controlled scene-diversity comparison: The controlled experiment compares DROID subsets with equal dataset size but different scene diversity on challenging OOD evaluation tasks.One subset selects the 20 scenes with the most demonstrations; the other uniformly samples successful demonstrations while retaining high scene diversity.
  • Overall policy comparison: Across all evaluation tasks, DROID co-training outperforms the next-best method by 22% absolute success rate in-distribution and 17% out of distribution.The figure reports success rate averaged across tasks with standard error.
  • Controlled scene-diversity comparison: 7362 demonstrations sampled from diverse scenes outperform 7362 demonstrations concentrated in DROID’s 20 highest-volume scenes in OOD evaluation.The comparison controls dataset size and uses the same 50/50 co-training setup with individual task data.
  • Overall policy comparison: Co-training with the full DROID dataset matches or outperforms the subsampled dataset on all three tasks, suggesting strength from dataset size and especially diversity.The comparison relates Figure 10’s task results to the corresponding full-DROID results in Figure 8.

VI. DISCUSSION

The discussion presents DROID as a diverse, open robot-manipulation resource for improving policy performance and robustness, while documenting its data format, collection interface, scene definition, and scene-labeling limitation. Its broader use remains an open research direction.

  • Discussion: DROID provides diverse scenes, tasks, objects, and viewpoints, and policy evaluations show value for improving performance and robustness against existing large-scale robot datasets.The paper reports an order-of-magnitude larger scene diversity than existing large robot manipulation datasets.
  • Future directions: The authors identify open questions about combining DROID with existing datasets, learning without in-domain data, and choosing between the full dataset and data slices.They also position the open hardware platform as a way to support reproducibility and future dataset additions.
  • Scene definition: A unique scene is defined as a substantial change to the robot’s workspace, not merely a change in interacted objects or external-camera pose.A home kitchen can therefore contain multiple scenes when the robot moves among workspace regions.
  • Data collection: The collection GUI randomly samples entered feasible tasks for demonstrations, broadening task coverage within each scene and reducing bias toward easy or familiar tasks.Collectors can select suggested tasks or enter free-form instructions.
  • Limitations: Scene labeling may double-count the same physical scene when different robots are placed there, although the authors consider their estimate conservative.Duplicate detection is performed within groups sharing robot serial number, lab, and building.

APPENDIX E EVALUATION PROCEDURE

The evaluation procedure compares policy-training data mixtures across six tasks and out-of-distribution variants, while documenting the architecture, calibration resources, and experimental controls used. It also reports automated calibration outcomes and limitations under challenging visual conditions.

  • Evaluation tasks: Policies are evaluated across six tasks, each paired with out-of-distribution variants that alter object layouts, distractors, camera viewpoints, or task conditions.The procedures include short-, medium-, and long-horizon manipulation tasks in lab and kitchen settings.
  • Policy setup: The diffusion-policy pipeline combines two external camera observations, a frozen DistilBERT language embedding, gripper position, and gripper state.Images are resized to 128 × 128, augmented, encoded with ImageNet-pretrained ResNet-50, and processed through an observation MLP.
  • Training comparisons: Three training-data settings compare in-domain demonstrations alone with 50/50 mixtures of in-domain data and either DROID or OXE trajectories.The compared policies use identical architectures and training parameters; only the data used to construct batches differs.
  • Calibration resources: The dataset provides camera-to-base calibration for 36k unique scenes and a curated 24k-scene superset with both cameras calibrated relative to the robot base.The release includes multiple calibration sets with corresponding quality-assessment metrics.
  • Calibration assessment: Camera-to-camera calibration uses reciprocal nearest-neighbor matches between projected pointmaps, with quality varying across laboratories and visual conditions.Clutter and poor lighting reduce geometric consistency, while temporal ensembling and improved 3D models are suggested for future stabilization.

D. Limitations and Future Work

The calibration release broadens DROID’s usefulness for 3D vision and robotics, but its automatic procedures retain false positives and model- and scene-dependent weaknesses. Future improvements target more general robot detection and more robust camera-to-camera estimation.

  • Limitations: The fully automatic calibration process still produces false positives, and the robot-detection component’s zero-shot transfer to other robots remains untested.The approach relies partly on an out-of-the-box model trained on the Franka Panda robot.
  • Limitations: Camera-to-camera calibration can fail in cluttered tabletop scenes with little or no image overlap because it relies on original DUSt3R checkpoints.Improved point-map models are proposed for more robust estimation in these settings.
  • Calibration release: The final release offers three calibration sets: 36k camera-to-base scenes, camera-to-camera matrices for all scenes, and a 24k-scene superset with both cameras calibrated to the base.The superset applies IOU, reprojection-error, and match-quantile thresholds across the three calibration stages.
  • Future use: The calibration resources are intended to support 3D vision and robotics manipulation research, including geometric understanding and offline automatic calibration of in-the-wild datasets.The release is positioned as both a research resource and an inspiration for future calibration pipelines.
  • Dataset diversity: DROID’s skill and object distributions show broad interaction diversity, with most objects supporting multiple interaction types.The verb distribution has a long tail, while the joint verb-object distribution indicates varied interactions across everyday objects.
Loading 2403.12945v2…