PaperlayerView exact revision on arXiv

Source-linked AI summary

RoboTwin: Dual-Arm Robot Benchmark with Generative Digital Twins

Yao Mu, Tianxing Chen, Zanxin Chen, Shijia Peng, Zhiqian Lan, Zeyu Gao, Zhixuan Liang, Qiaojun Yu, Yude Zou, Mingkun Xu, Lunkai Lin, Zhiqiang Xie, Mingyu Ding, Ping Luo

arXiv:2504.13059v1cs.ROcs.AIcs.CL

TL;DR

Dual-arm manipulation lacks diverse, high-quality demonstrations and real-world-aligned benchmarks. RoboTwin addresses this gap with generative digital twins, spatially aware LLM code generation, and coordinated simulated and real evaluation data. Policies using 300 generated samples and 20 real samples improved success rates by 70% on a single-arm task and over 40% on a dual-arm task versus 20 real samples alone.

  • Problem

    Dual-arm robotics lacks diverse, high-quality training demonstrations and comprehensive evaluation benchmarks aligned with real-world tasks.

  • Method

    RoboTwin generates varied 3D digital twins from single RGB images and uses spatial annotations with LLMs to decompose tasks and generate executable robot behavior.

  • Results

    70% higher success on single-arm hammer beat and over 40% higher success on dual-arm ball sweep were achieved versus policies trained exclusively on 20 real-world samples.

  • Takeaways & Limitations

    RoboTwin provides a benchmark integrating synthetic and real-world data for dual-arm manipulation and supports higher success rates with less real data.

  • Takeaways & Limitations

    Dual-arm bottle rearrangement remained suboptimal across diverse initial bottle states, indicating a need for more effective dual-arm imitation-learning algorithms.

Abstract

from arXiv · show

In the rapidly advancing field of robotics, dual-arm coordination and complex object manipulation are essential capabilities for developing advanced autonomous systems. However, the scarcity of diverse, high-quality demonstration data and real-world-aligned evaluation benchmarks severely limits such development. To address this, we introduce RoboTwin, a generative digital twin framework that uses 3D generative foundation models and large language models to produce diverse expert datasets and provide a real-world-aligned evaluation platform for dual-arm robotic tasks. Specifically, RoboTwin creates varied digital twins of objects from single 2D images, generating realistic and interactive scenarios. It also introduces a spatial relation-aware code generation framework that combines object annotations with large language models to break down tasks, determine spatial constraints, and generate precise robotic movement code. Our framework offers a comprehensive benchmark with both simulated and real-world data, enabling standardized evaluation and better alignment between simulated training and real-world performance. We validated our approach using the open-source COBOT Magic Robot platform. Policies pre-trained on RoboTwin-generated data and fine-tuned with limited real-world samples demonstrate significant potential for enhancing dual-arm robotic manipulation systems by improving success rates by over 70% for single-arm tasks and over 40% for dual-arm tasks compared to models trained solely on real-world data.

1. Introduction

RoboTwin addresses limited, costly demonstration data and inadequate real-world-aligned benchmarks for complex dual-arm manipulation by generating diverse digital twins, expert data, and standardized evaluation scenarios.

  • Robotic systems need precise dual-arm coordination, but diverse high-quality training data and real-world-aligned evaluation benchmarks remain scarce.
  • Existing trajectory generators often require task-specific designs and struggle with variants beyond predefined configurations.
  • Integrated dual-arm benchmarks remain limited because many existing platforms focus on single-arm tasks, separated arms, fixed environments, or VR teleoperation.
  • RoboTwin generates varied 3D object instances from single RGB images and transfers spatial annotations across instances within an object class.
  • Its spatial-aware code generation framework decomposes tasks, infers terminal-state constraints, and produces executable robot trajectories through planning modules.
  • 70% higher success in single-arm hammer beat and over 40% higher success in dual-arm ball sweep were achieved using 300 synthetic samples plus 20 real samples versus 20 real samples alone.

2. Related Work

Prior robotics benchmarks and demonstration methods are limited by fixed scenarios, task-specific design, constrained environments, and incomplete support for integrated dual-arm manipulation. RoboTwin combines generative 3D assets, spatial reasoning, and LLM-based code generation to automate diverse task variations and expert demonstrations.

  • 2. Related Work: Existing simulation data generators commonly require manual task-specific design and remain limited by fixed scenarios, predefined configurations, or fixed 3D objects.
  • 2. Related Work: RoboTwin uses 3D generative foundation models and LLMs to autonomously create task variations and corresponding expert demonstrations.
  • 2. Related Work: Many benchmarks emphasize single-arm manipulation or separated bimanual setups, while other platforms use fixed environments or VR teleoperation.
  • 2. Related Work: Its pipeline generates visually diverse yet functionally consistent assets, spatial annotations, and executable collision-free robot behavior from 2D images.
  • 2. Related Work: Robot manipulation learning includes behavioral cloning, offline reinforcement learning, action chunking with transformers, and diffusion-based policies.
  • 2. Related Work: LLMs have been used to translate high-level task descriptions into robot control programs, while prior spatial-reasoning methods have limitations with functional-axis constraints.

3. Bridging Physical and Digital Worlds for Diverse Robot Behavior Generation

RoboTwin turns single 2D images into diverse 3D digital assets, annotates their functional geometry, and uses spatially constrained code generation to produce validated dual-arm behaviors.

  • 3.1. Generation of Diverse Digital Assets: RoboTwin uses Rodin to generate varied 3D models from simple 2D RGB images, with geometry, surface normals, wireframes, and textures.GPT-4V analyzes object photographs, while language-model modifications produce visually distinct instances of one object type; assets are checked quantitatively and qualitatively.
  • 3.2. Spatial Annotation: Spatial annotations are extracted semi-automatically with principal axes and approach directions for geometry-aware manipulation and code generation.
  • 3.2. Spatial Annotation: Key-point annotations identify functional and contact locations, while function, approach, and lateral axes encode tool directionality and orientation.These annotations make tool functionality more accessible to LLMs for complex task code generation and spatially aware manipulation.
  • 3.3. Expert Data Generation: The behavior-generation pipeline infers spatial constraints, translates them into executable code with an LLM and MPlib, validates execution, and self-corrects from errors.It supports synchronized arm movements, coordinated gripper actions, and dynamic collision avoidance through safe intermediate positions.
  • 3.3. Expert Data Generation: For complex subtasks, the system aligns an object’s functional point with a target point and computes directional constraints before optimizing collision-free joint trajectories.The objective may include energy efficiency, execution time, and motion smoothness, while forward-kinematics constraints enforce the desired end-effector pose.

4. Benchmark

RoboTwin benchmarks dual-arm manipulation with simulated and real-world data on a hardware-matched platform, including tasks requiring handovers and interference avoidance.

  • 4. Benchmark: RoboTwin includes 15 dual-arm manipulation tasks built on the ManiSkill3 physics engine.
  • 4. Benchmark: The benchmark uses the open-source Cobot Magic platform with four robot arms and four Intel RealSense D-435 RGBD cameras.Cameras are mounted on the stand and robot wrists, with front, left, and right views captured simultaneously at 30Hz.
  • 4. Benchmark: Tasks require selecting collaboration strategies based on target distance, including arm handovers and coordination that avoids interference in confined spaces.Examples include handover, placing a cup on a coaster, and placing shoes in a shoe box.
  • 4. Benchmark: Each task provides 100 simulation data sets and 20 real-world data sets, using a real-world hardware setup that strictly matches simulation.
  • 4. Benchmark: The datasets include synchronized RGB and depth images, transformed point clouds, and joint and end-effector poses for both arms.

5. Experiment on RoboTwin Benchmark

RoboTwin evaluates imitation-learning policies across dual-arm tasks using varying demonstration counts and real-world transfer settings. Results show strong few-shot performance for DP3, better scaling for DP, and substantial gains from simulation pretraining, while complex coordination remains difficult.

  • Experimental setup: Across 14 tasks, the benchmark evaluates DP3 and DP policies with 20, 50, and 100 expert demonstrations using success rate under pose and collision constraints.The evaluation includes DP3 with and without color and 2D Diffusion Policy.
  • Imitation-learning results: DP3 achieves strong performance with merely 20 demonstrations but shows minimal improvement or decreases as training data expands to 100 samples.This indicates strong few-shot behavior but limited scalability in the reported experiments.
  • Imitation-learning results: With 100 demonstrations, DP improves from 1.7% to 85.7% in Dual Bottles Pick (Easy) and outperforms DP3 in several tasks.The reported pattern associates DP with stronger performance as the dataset grows.
  • Representation effects: RGB fusion with point clouds produces inconsistent benefits, improving DP3 in cluttered Pick Apple Messy while degrading performance in Container Place.The results motivate better fusion of RGB semantics and point-cloud geometry.
  • Coordination complexity: 85.7% success is achieved on Dual Bottles Pick at 100 demonstrations, whereas Dual Shoes Place remains below 15% across methods because complex coordination is harder.Tasks allowing more independent arm operation outperform those requiring tightly coordinated dual-arm behavior.
  • Real-world transfer: 72% higher single-arm success and over 40% higher dual-arm success result when policies use simulation data before fine-tuning on 20 real-world samples.The comparison is against policies trained exclusively on real-world data; the setup uses 300 simulation datasets plus 20 real-world datasets.
  • Real-world transfer: Simulation data enables non-zero success in bottle rearrangement, but dual-arm performance remains suboptimal across diverse upright and lying bottle states.The authors identify a need for imitation-learning algorithms tailored to dual-arm coordination.

6. Conclusion

RoboTwin integrates real-world and synthetic data into a dual-arm manipulation benchmark, using generative digital twins and spatial-aware LLM code generation. Experiments report higher success with less real data while identifying persistent limitations in dual-arm coordination.

  • Conclusion: RoboTwin integrates real-world and synthetic data into a comprehensive benchmark for dual-arm robotic manipulation.The framework is built on the COBOT Magic Robot platform.
  • Conclusion: 3D generative models create diverse training data from single RGB images, while spatial-aware LLM code generation produces expert demonstrations and precise movements.The code-generation framework decomposes complex tasks using object annotations.
  • Conclusion: Policies trained with RoboTwin-simulated data achieve higher success rates with less real data than policies trained solely on real-world data.The experiments also report limitations in dual-arm coordination tasks.
  • Conclusion: Future work targets advanced dual-arm coordination algorithms and more complex manipulation tasks.

A. Task Description for RoboTwin

RoboTwin contains 15 benchmark and real-world tasks with randomized target-object positions. Tasks differ in whether both arms are required or an appropriate arm is selected based on object position.

  • Task coverage: The benchmarks and real-world experiments comprise 15 tasks with randomized initial positions for target objects.
  • Task requirements: Some tasks, including Shoes Place, must be completed using both arms.
  • Task requirements: Container Place and Empty Cup Place include dual-arm and single-arm versions, with the appropriate arm selected from the object’s initial position.

B. Implementation Details for Simulation Experiments

The simulation experiments use Diffusion Policy baselines and the 3D Diffusion Policy, with 2D images and downsampled point clouds as observations.

  • Diffusion Policy models visuomotor control as conditional denoising diffusion, producing temporally consistent action sequences from visual observations.Its inference iteratively optimizes the action-distribution score through stochastic Langevin dynamics.
  • DP3 extends Diffusion Policy with compact 3D representations extracted from sparse point clouds and predicts 3D end-effector pose sequences.The predicted poses include translations and rotations over a fixed duration.
  • The experiments adopt original DP and DP3 hyper-parameters to maintain consistent performance and fair comparison with published results.
  • The camera setup uses 320 × 240 2D observations and FPS-downsamples image-derived point clouds to 1024 points for 3D observations.

C. Sim2Real Experiment Setup

The sim-to-real experiments test whether generated simulation data supports policy learning under limited real-world exposure, using visual comparisons to assess fidelity.

  • The real-world experiments evaluate whether generated simulation data enables high real-world performance with limited real-world data exposure.
  • RoboTwin-generated scenes achieve near photorealistic visual quality relative to real scenarios, accurately reproducing lighting, shadows, and object textures.The authors describe this fidelity as promising for bridging the sim-to-real gap.

C.2. Details of Sim2Real Fine-Tuning

Sim-to-real fine-tuning aligns real and simulated observations, pretrains on 300 simulated samples, and adapts with 20 real-world samples.

  • Brightness adjustment enhances typically darker real-world observations to better align them with simulation images.The alpha parameter can be tuned for specific lighting conditions.
  • 300 RoboTwin-generated simulation samples pretrain a Diffusion Policy network to learn basic manipulation skills.
  • 20 real-world samples fine-tune the pretrained policy, reducing real-world data collection while adapting the network across the sim-to-real gap.
  • The two-stage strategy combines abundant simulation data with minimal real-world data for robot skill acquisition and transfer.
  • Table 4 reports success rates and standard deviations for DP3 and DP across 14 tasks with 20, 50, and 100 expert demonstrations.
  • Figure 9 compares real-world data in the top row with RoboTwin-generated data in the bottom row for each task.
  • Table 7 documents hyper-parameters for pretraining with RoboTwin-generated data and fine-tuning with limited real-world data.

D. Prompts

The prompt framework supplies task descriptions, spatial annotations, APIs, and examples so an LLM can generate executable robot-task code.

  • D. Prompts: Prompts combine task information, available API descriptions, and function examples to guide expert demonstration generation.
  • D.1. Task Information and General Prompt: Task prompts specify scene objects, poses, coordinate conventions, and the required sequence of manipulation actions.
  • D.1. Task Information and General Prompt: The block-stacking task assigns arms by block x-coordinate and requires collision-avoidance poses when alternating between arms.
  • D.1. Task Information and General Prompt: World_direction_dic provides named approach directions and corresponding orientation quaternions, while actor lists identify scene objects and their data.
  • D.2. Introduction of Available APIs: The APIs provide gripper controls, single-arm and dual-arm screw-motion execution, actor poses, grasp poses, and collision-avoidance poses.
  • D.3. Function Example: Function examples show how to retrieve actors and actor data, compute grasp poses, and execute pre-grasp, grasp, and placement motions.
  • D.2. Introduction of Available APIs: Grasp and placement functions map actor annotations, target points, approach directions, and optional orientations to end-effector poses.
  • D.2. Introduction of Available APIs: The API documentation defines actor orientation after grasping and supports target-point, approach-direction, and pre-distance parameters.
Loading 2504.13059v1…