PaperlayerView exact revision on arXiv

Source-linked AI summary

RoboCOIN: An Open-Sourced Bimanual Robotic Data Collection for Integrated Manipulation

Shihan Wu, Xuecheng Liu, Shaoxuan Xie, Pengwei Wang, Xinghang Li, Bowen Yang, Zhe Li, Kai Zhu, Hongyu Wu, Yiheng Liu, Zhaoye Long, Runtian Xu, Yue Wang, Chong Liu, Dihan Wang, Ziqiang Ni, Xiang Yang, You Liu, Ruoxuan Feng, Lei Zhang, Denghang Huang, Chenghao Jin, Anlan Yin, Xinlong Wang, Zhenguo Sun, Junkai Zhao, Mengfei Du, Mingyu Cao, Xiansheng Chen, Hongyang Cheng, Xiaojie Zhang, Yankai Fu, Ning Chen, Cheng Chi, Sixiang Chen, Huaihai Lyu, Xiaoshuai Hao, Yequan Wang, Bo Lei, Dong Liu, Xi Yang, Yance Jiao, Tengfei Pan, Yunyan Zhang, Songjing Wang, Ziqian Zhang, Xu Liu, Ji Zhang, Caowei Meng, Zhizheng Zhang, Jiyang Gao, Song Wang, Xiaokun Leng, Zhiqiang Xie, Zhenzhen Zhou, Peng Huang, Wu Yang, Yandong Guo, Yichao Zhu, Suibing Zheng, Hao Cheng, Xinmin Ding, Yang Yue, Huanqian Wang, Chi Chen, Jingrui Pang, YuXi Qian, Haoran Geng, Lianli Gao, Haiyuan Li, Bin Fang, Gao Huang, Yaodong Yang, Hao Dong, He Wang, Hang Zhao, Yadong Mu, Di Hu, Hao Zhao, Tiejun Huang, Shanghang Zhang, Yonghua Lin, Zhongyuan Wang, Guocai Yao

arXiv:2511.17441v3cs.RO

TL;DR

Bimanual manipulation lacks large, diverse, structurally annotated datasets because robotic platforms differ substantially in hardware. RoboCOIN addresses this gap with a multi-embodiment dataset, hierarchical annotations, and the CoRobot processing pipeline, with experiments reporting improved performance across platforms. The paper also identifies teleoperation bias, annotation cost, and expert-heuristic dependence as limitations.

  • Problem

    Existing datasets lack sufficient diversity and structural annotations for bimanual, multi-embodiment learning, partly because heterogeneous hardware complicates large-scale data collection.

  • Method

    RoboCOIN combines a multi-embodiment bimanual dataset with hierarchical capability annotations and CoRobot, which validates trajectories, generates annotations, and manages heterogeneous robot data.

  • Results

    RoboCOIN improves bimanual manipulation performance across robotic embodiments, with relative gains of 75.7% on ARX-X5 and 212.9% on Franka Emika Panda.

  • Takeaways & Limitations

    The open dataset and codebase provide a resource for research on bimanual and multi-embodiment manipulation.

  • Takeaways & Limitations

    Teleoperation may introduce operator bias, manual annotation limits scalability, and expert-defined RTML heuristics may constrain flexibility on niche tasks or novel platforms.

Abstract

from arXiv · show

Despite the critical role of bimanual manipulation in endowing robots with human-like dexterity, large-scale and diverse datasets remain scarce due to the significant hardware heterogeneity across bimanual robotic platforms. To bridge this gap, we introduce RoboCOIN, a large-scale multi-embodiment bimanual manipulation dataset comprising over 180,000 demonstrations collected from 15 distinct robotic platforms. Spanning 16 diverse environments-including residential, commercial, and industrial settings-the dataset features 421 bimanual tasks systematically categorized by 39 bimanual collaboration actions and 432 objects. A key innovation of our work is the hierarchical capability pyramid, which provides granular annotations ranging from trajectory-level concepts to segment-level subtasks and frame-level kinematics. Furthermore, we present CoRobot, an efficient data processing pipeline powered by the Robot Trajectory Markup Language (RTML), designed to facilitate quality assessment, automated annotation, and unified multi-embodiment and data management. Extensive experiments demonstrate the effectiveness of RoboCOIN in enhancing the performance of various bimanual manipulation models across a wide spectrum of robotic embodiments. The entire dataset and codebase are fully open-sourced, providing a valuable resource for advancing research in bimanual and multi-embodiment manipulation.

I. INTRODUCTION

RoboCOIN addresses limited diversity and structure in bimanual, multi-embodiment datasets with a large-scale dataset and an integrated processing pipeline. It combines heterogeneous platforms, structured annotations, and evaluations showing improved bimanual manipulation performance.

  • Motivation: Existing bimanual datasets are constrained by hardware heterogeneity, limited diversity, and insufficient structural annotations for manipulation processes.These limitations affect multi-embodiment learning and generalization to novel environments.
  • RoboCOIN Dataset: Over 180,000 demonstrations across 421 bimanual tasks and 16 scenarios are collected from 15 robotic platforms.The platforms include bimanual, half-humanoid, and humanoid configurations with parallel grippers and dexterous hands.
  • RoboCOIN Dataset: The dataset organizes demonstrations using action collaboration and object flexibility, supporting systematic task design and progressive skill acquisition.Its action taxonomy includes 39 collaboration-based action patterns, while objects span rigid, articulated, and deformable categories.
  • Structured Learning: The Hierarchical Capability Pyramid provides trajectory-level concepts, segment-level subtasks, and frame-level kinematic states for structured learning across embodiments.These levels connect holistic planning, executable subtasks, and precise control.
  • Processing Pipeline: CoRobot integrates RTML-based trajectory assessment, automated hierarchical annotation, and unified multi-embodiment control and data management.The pipeline is designed for dataset development and robot deployment.
  • Evaluation: 75.7% and 212.9% relative gains are reported on ARX-X5 and Franka Emika Panda, respectively, while RTML filters 35.3% of low-quality trajectories and yields a 22.2% relative performance boost.Real-world experiments report 53.6% in-distribution and 155.6% out-of-distribution relative improvements.

C. Hierarchical Capability Pyramid

RoboCOIN’s hierarchical capability pyramid organizes synchronized annotations at trajectory, segment, and frame levels. The representation links global task understanding with executable subtasks and dense control information.

  • Trajectory-Level Annotations: Trajectory-level annotations describe global concepts, scene configuration, and task objectives for holistic spatial and physical reasoning.They characterize the environment and object placement.
  • Segment-Level Annotations: Segment-level annotations decompose tasks into temporally aligned executable subtasks for structured reasoning.Temporal synchronization keeps the task hierarchy cohesive.
  • Frame-Level Annotations: Frame-level annotations provide dense low-level details, including motion trajectories, kinematic states, gripper states, and action labels for precise control.The three levels are temporally synchronized into one data structure.

D. Comparison with Existing Datasets

RoboCOIN is positioned as an open, generalizable dataset for complex bimanual manipulation, addressing gaps in existing datasets through multi-embodiment coverage and structured processing.

  • Existing Datasets: Existing datasets commonly provide single-arm data, insufficient bimanual coverage, or data from a single dual-arm platform.RoboCOIN is contrasted with datasets including BridgeData V2, DROID, Open X-Embodiment, RoboMIND, AgiBot World Beta, and Open Galaxea.
  • RoboCOIN: RoboCOIN includes 15 diverse robotic platforms and hierarchical annotations for complex bimanual manipulation.Its design targets broader multi-embodiment applicability than the compared datasets.
  • CoRobot: CoRobot combines RTML trajectory validation, hierarchical annotation generation, and a unified interface for robot control and multi-embodiment data management.These components support standardized processing across heterogeneous platforms.
  • RTML: RTML converts expert rules into configurable machine-readable constraints covering global trajectory properties and local sequential phases.Constraints include workspace, velocity, acceleration, duration, motion stability, pose consistency, and execution efficiency.

B. Annotation Pipeline

RoboCOIN’s annotation and evaluation workflow combines automated extraction, rule-based segmentation, human refinement, and cross-embodiment experiments. The pipeline produces multi-level descriptions while testing transfer on unseen robotic platforms.

  • Annotation Pipeline: The annotation pipeline generates trajectory-, segment-, and frame-level descriptions using detectors, rule-based programs, and human refinement.Open-vocabulary detectors and language models provide scene descriptions; rule-based tools propose segmentation points for annotator refinement.
  • Annotation Pipeline: Frame-level processing extracts gripper kinematics with sliding windows and converts numerical motion features into consistent text labels using predefined thresholds.The procedure covers displacement, velocity, acceleration, and higher-order motion features.
  • Unified Robotic Interface: The Unified Robotic Interface integrates vendor SDKs with ROS and ROS 2 to provide homogeneous control across diverse hardware.It also normalizes camera configuration, coordinate alignment, and joint units.
  • Unified Robotic Interface: The interface partitions datasets into reusable subsets tagged by embodiment, task type, and environment for flexible training-data composition.The storage strategy supports querying and assembling data for specific research needs.
  • Evaluation Setup: Simulation evaluates unseen ARX-X5 and Franka Emika Panda embodiments on dual-arm tasks, while real-world experiments use Realman RMC-AIDA-L and Unitree G1edu-u3.The simulation suite contains 16 tasks, evaluated on 14 ARX-X5 tasks and 11 Franka tasks after trajectory-generation failures.

B. Impact of Multi-Embodiment Adaptability

RoboCOIN improves bimanual manipulation across robotic embodiments through cross-embodiment transfer and hierarchical capability annotations. Its pyramid also strengthens robustness and addresses failures at scene, subtask, and frame levels.

  • Cross-Embodiment Transfer: 212.9% relative improvement raises Franka Emika Panda average success from 8.5% to 26.6% with π0-RoboCOIN.The gains are reported across selected RoboTwin 2.0 tasks.
  • Cross-Embodiment Transfer: 75.7% relative improvement raises ARX-X5 average success from 17.3% to 30.4% with π0-RoboCOIN.The comparison uses π0 as the baseline and π0-RoboCOIN trained with the RoboCOIN-10K subset.
  • Hierarchical Capability Pyramid: 155.6% relative improvement raises average OOD success from 22.5% to 57.5% with the pyramid-augmented model.Under distribution shift, the pyramid model’s relative performance drop is 17.9%, compared with 43.8% for π0.
  • Hierarchical Capability Pyramid: The capability pyramid improves scene-level perception, segment-level handover, and frame-level fine manipulation in qualitative cases.Reported examples include color-changed Pass Bowl, Pass Bowl handover, and Peach Drawer grasping.

D. Impact of RTML

RTML improves trajectory quality by filtering low-quality demonstrations and applying global or phase-wise constraints during policy training. The experiments identify dominant failure modes and show progressively higher success rates with finer filtering.

  • Trajectory Quality: 35.3% of trajectories are filtered on average across two tasks by RTML, indicating substantial inconsistency in the original demonstrations.The filtering result motivates data-quality optimization before policy learning.
  • Failure Analysis: 52.7% of failures occur during grasping, followed by 17.8% during moving.The phase-wise analysis identifies grasping as the principal failure phase.
  • Failure Analysis: 46.2% of trajectory invalidity is attributed to velocity violations, followed by 24.5% from duration violations.The metric-wise analysis identifies velocity as the dominant invalidity cause.
  • Implications: The identified failure modes provide actionable guidance for future demonstration collection and policy optimization.The conclusion links phase-wise and metric-wise analyses to subsequent data and policy improvements.
  • Policy Learning: 22.2% relative improvement raises GR00T-Fine success by 16% over GR00T-Raw’s 72% success rate.GR00T-Coarse, using global constraints only, achieves a 3% gain or 4.2% relative improvement, while GR00T-Fine also uses phase-wise constraints.

APPENDIX A ACTION AND OBJECT TAXONOMY

The proposed taxonomy organizes robotic manipulation actions into four functional categories and uses semantically consistent synonyms to improve language grounding and reduce annotation ambiguity.

  • Taxonomy Structure: Four categories organize manipulation actions by functional purpose, physical interaction type, and environmental effect.The categories are General Manipulation, Object State Change, Object Relation Change, and Task-Specific Actions.
  • General Manipulation: General Manipulation covers universal primitives such as grasping, placement, pushing, pulling, and alignment.These actions describe basic low-level robot motions.
  • Object State Change: Object State Change covers actions that alter an object’s physical state, shape, or configuration.Examples include pressing, folding, twisting, shaking, and tearing.
  • Object Relation Change: Object Relation Change covers operations that modify spatial or structural relationships between multiple objects.Examples include stacking, insertion, assembly, attachment, and exchange.
  • Task-Specific Actions: Task-Specific Actions cover specialized operations, while action synonyms support textual augmentation and reduce annotation ambiguity.Examples include cleaning, cooking, pouring, cutting, device control, and writing.

APPENDIX B RTML SPECIFICATION

RTML represents manipulation tasks with hierarchical global and local constraints. Global limits govern the whole execution, while local subtask constraints specify spatial, kinematic, idle-arm, orientation, and temporal requirements.

  • Architecture: RTML decouples global and local constraints to make manipulation requirements structured and machine-interpretable.Global constraints apply task-wide, while local constraints apply to semantically meaningful stages.
  • Global Constraints: Global constraints limit task-wide linear velocity and acceleration to support safety and motion smoothness.The specified limits include 0.5 m/s maximum velocity, 0.3 m/s mean maximum velocity, and 12.0 m/s² maximum acceleration.
  • Local Constraints: Local constraints partition tasks into sequential subtasks with human-readable descriptions aligned to natural-language definitions.Examples include moving a bowl, grasping bread, and placing bread in a bowl.
  • Subtask Constraints: Workspace and kinematic constraints bound active-arm regions, velocity statistics, and motion variability for each subtask.Examples include a 3D Cartesian workspace, a 0.10 m/s mean velocity maximum, and a 0.08 m/s standard-deviation maximum.
  • Subtask Constraints: Idle-arm and orientation constraints reduce interference and regulate end-effector pose stability during precise manipulation.The examples constrain the inactive arm’s velocity and angular deviation or variance during grasping.
  • Temporal Constraints: Temporal constraints require each stage to remain within specified duration bounds, such as 2.0–8.0 seconds for bread grasping.The hierarchical structure supports trajectory quality control and downstream learning algorithms’ adherence to task constraints.

APPENDIX C ANNOTATION PIPELINE

RoboCOIN’s annotation pipeline converts raw manipulation demonstrations into hierarchical semantic and kinematic representations. It combines trajectory segmentation, frame-level motion extraction, threshold-based labeling, and human verification for quality control.

  • Pipeline structure: Three hierarchical pipelines annotate trajectories at trajectory, segment, and frame levels, as illustrated in Figure 10.The levels respectively support semantic scene understanding, atomic action decomposition, and fine-grained gripper kinematics.
  • Trajectory-level annotation: Object detection and scene-description generation lift visual observations into structured representations of objects, quantities, positions, and spatial relationships.These representations provide contextual information for understanding robot actions before segment-level analysis.
  • Segment-level annotation: Stationary states, gripper opening or closing transitions, and task-end moments identify temporal boundaries for semantically labeled atomic action segments.Additional boundaries may be based on sudden kinematic changes or significant object-position shifts.
  • Frame-level annotation: A cascaded window scheme computes displacement, velocity, and acceleration from gripper motion to quantify fine-grained frame-level dynamics.The implementation is adjustable and supports trajectory quality assessment.
  • Frame-level annotation: The summary operator maps continuous kinematic values to discrete labels such as stationary, slow, and fast using predefined thresholds.Both grippers are processed in parallel, and users can extend the pipeline with features such as jerk or angular velocity.
  • Quality control: Human verification reports 84.2% overall accuracy for trajectory-level annotations, with errors including missed objects, localization errors, and redundant objects.The passage also states that segment-level labels undergo online verification of boundaries and action labels.

APPENDIX D EXPERIMENTAL TASK SETUPS

The experimental setup combines simulation and real-world tasks to test cross-embodiment transfer, hierarchical capability, and robustness under visual variation. It includes distinct task suites, robot platforms, and out-of-distribution lighting conditions.

  • Experimental design: The dual-environment design uses simulation for cross-embodiment transfer testing and real-world tasks for hierarchical capability evaluation.The setup comprises 16 simulation tasks and 8 real-world tasks.
  • Simulation task setups: Simulation data are collected in RoboTwin 2.0 with ARX-X5 and Franka Emika Panda dual-arm robots, using 50 trajectories per task.Both robots have an inter-arm distance of 0.8 meters, with state information and natural-language instructions recorded.
  • Real-world task setups: Real-world experiments use the Realman RMC-AIDA-L half-humanoid and Unitree G1edu-u3 humanoid platforms.Six hierarchical-pyramid tasks each contain 50 trajectories, while two RTML tasks contain 298 and 997 trajectories.
  • Robustness evaluation: OOD real-world evaluation varies lighting across original, strong, and weak conditions, introducing shadows, reduced texture visibility, and brightness disparities.The setup also considers object-color and background variations as additional OOD conditions.
  • Experimental illustrations: Figure 11 presents example task setups in simulation and real-world environments, while Figure 13 presents hierarchical integration alongside a VLA baseline.The baseline directly maps multimodal inputs to primitive actions, whereas hierarchical integration adds multilevel task knowledge and motion constraints.

APPENDIX E HIERARCHICAL CAPABILITY PYRAMID INTEGRATION

The hierarchical capability pyramid integrates human-demonstration structure, segment-level subtask switching, and frame-level kinematic constraints into VLA control. RTML examples encode global and local constraints for staged bimanual tasks.

  • Hierarchical integration: The baseline VLA model maps state, observations, and instructions directly to control signals without explicit hierarchical knowledge integration.The proposed implementation instead incorporates hierarchical annotations into structured action generation.
  • Trajectory-level information: Trajectory-level information supplies the complete execution flow and high-level manipulation goal from human demonstrations.This establishes the global structure used by the integrated controller.
  • Segment-level information: Segment-level information uses manually predefined atomic subtasks and rule-based switching frames based on gripper states and task progress.The procedure is intended to support stable and smooth transitions between consecutive subtasks.
  • Frame-level information: Frame-level information extracts displacement, velocity, and acceleration, then converts them into natural-language constraints on direction, speed, and acceleration trend.The resulting descriptions provide explicit frame-wise motion constraints.
  • RTML task representation: RTML examples represent tasks such as pulling bowl storage bread with global constraints and sequential local stages for grasping and placing.The listings include stage-specific subtask matches, arm assignments, motion limits, and duration bounds.
Loading 2511.17441v3…