PaperlayerView exact revision on arXiv

Source-linked AI summary

DexMimicGen: Automated Data Generation for Bimanual Dexterous Manipulation via Imitation Learning

Zhenyu Jiang, Yuqi Xie, Kevin Lin, Zhenjia Xu, Weikang Wan, Ajay Mandlekar, Linxi Fan, Yuke Zhu

arXiv:2410.24185v2cs.ROcs.AIcs.CVcs.LG

TL;DR

Data collection for bimanual dexterous robots is costly and difficult because operators must control multiple arms and multi-fingered hands. DexMimicGen automatically transforms and replays a handful of human demonstrations in simulation to generate training data, producing 21K demonstrations from 60 source demonstrations and achieving 90% success in a real-world can-sorting evaluation.

  • Problem

    Data acquisition for bimanual dexterous robots is difficult and costly because simultaneous multi-arm and multi-finger control increases operator burden and data requirements.

  • Method

    DexMimicGen automatically synthesizes bimanual dexterous manipulation trajectories from a small number of human demonstrations using simulation, with parallel, coordination, and sequential subtask handling.

  • Results

    21K demos generated from 60 source human demos, while a policy trained on 40 DexMimicGen demos achieved 90% success in real-world can sorting versus 0% from 4 source demos.

  • Takeaways & Limitations

    DexMimicGen provides simulation datasets and environments for studying bimanual dexterous manipulation and supports real-world humanoid deployment through a real2sim2real pipeline.

Abstract

from arXiv · show

Imitation learning from human demonstrations is an effective means to teach robots manipulation skills. But data acquisition is a major bottleneck in applying this paradigm more broadly, due to the amount of cost and human effort involved. There has been significant interest in imitation learning for bimanual dexterous robots, like humanoids. Unfortunately, data collection is even more challenging here due to the challenges of simultaneously controlling multiple arms and multi-fingered hands. Automated data generation in simulation is a compelling, scalable alternative to fuel this need for data. To this end, we introduce DexMimicGen, a large-scale automated data generation system that synthesizes trajectories from a handful of human demonstrations for humanoid robots with dexterous hands. We present a collection of simulation environments in the setting of bimanual dexterous manipulation, spanning a range of manipulation behaviors and different requirements for coordination among the two arms. We generate 21K demos across these tasks from just 60 source human demos and study the effect of several data generation and policy learning decisions on agent performance. Finally, we present a real-to-sim-to-real pipeline and deploy it on a real-world humanoid can sorting task. Generated datasets, simulation environments and additional results are at https://dexmimicgen.github.io/

I. INTRODUCTION

DexMimicGen addresses the high cost and difficulty of collecting demonstrations for bimanual dexterous robots by automatically expanding a small set of human demonstrations in simulation. It combines flexible multi-arm data generation with imitation learning and real-world deployment.

  • Motivation: Data acquisition is a major bottleneck because humanoid robots require simultaneous control of two arms and multi-fingered hands.Their higher degrees of freedom and task complexity also increase data requirements.
  • Approach: DexMimicGen transforms a small set of human demonstrations into large amounts of bimanual dexterous manipulation data through simulation.The system builds on automated data generation previously shown effective for single-arm manipulation.
  • Approach: DexMimicGen uses per-arm segmentation, asynchronous execution, synchronization, and sequential constraints to handle independent, coordinated, and ordered actions.These mechanisms extend beyond MimicGen’s single subtask segmentation.
  • Contributions: DexMimicGen introduces nine simulation environments and generates 21K demos from 60 source human demos across different bimanual coordination behaviors.The environments span three embodiments and support investigation of data-generation and policy-learning choices.
  • Motivation: Teleoperation and other large-scale collection methods require substantial human time and resources, whereas DexMimicGen automates dataset generation from a handful of demonstrations.The related-work discussion contrasts this with costly teleoperation and pre-programmed simulation experts.

III. PREREQUISITES

The paper frames manipulation as a partially observable control problem and describes Behavioral Cloning, MimicGen’s object-centric trajectory transformation, and the subtask taxonomy used for bimanual generation. These prerequisites establish how source demonstrations become physically executable training data.

  • Imitation Learning: Each manipulation task is modeled as a Partially Observable Markov Decision Process with states, observations, and actions.The learned policy maps observations to a distribution over actions.
  • Imitation Learning: Behavioral Cloning learns a policy by maximizing the likelihood of demonstrated actions conditioned on observations.Policies are trained on datasets generated by DexMimicGen.
  • Assumptions: The framework assumes object-centric task decomposition and observes or estimates object poses before arm contact.Its action space includes end-effector pose commands and hand actuation commands for each arm.
  • MimicGen: MimicGen decomposes demonstrations into contiguous object-centric manipulation segments and transforms them into novel scenes using relative object poses.The transformed trajectories preserve relative poses between the end effector and object frame.
  • Subtask Types: DexMimicGen distinguishes parallel, coordination, and sequential subtasks according to whether the arms act independently, jointly, or in a required order.This taxonomy captures the distinct coordination requirements of bimanual tasks.

IV. DEXMIMICGEN METHOD

DexMimicGen extends MimicGen to bimanual dexterous manipulation by explicitly handling independent, coordinated, and sequential actions. Its method combines a subtask taxonomy with generation procedures adapted to each coordination pattern.

  • Method: DexMimicGen addresses three bimanual challenges: independent arm goals, coordinated shared goals, and ordered subtasks.It introduces parallel, coordination, and sequential categories and modifies data generation for each.

A. Parallel Subtasks

DexMimicGen represents bimanual tasks as independent per-arm subtask sequences and executes them asynchronously when the arms need not remain synchronized. For coordination subtasks, it instead synchronizes both trajectories and applies a common transformation, with replay retained for handover cases that must remain executable.

  • Independent arm execution: Bimanual tasks use separate subtask sequences for each arm because the arms may grasp objects and complete subtasks at different times.This flexible ordering replaces MimicGen’s single fixed subtask sequence for parallel two-arm behavior.
  • Independent arm execution: Asynchronous execution maintains one action queue per arm, dequeuing actions in parallel and loading each arm’s next transformed segment whenever its queue is empty.Each source demonstration is split into object-centric manipulation segments independently for the two arms.
  • Coordinated execution: Coordination subtasks require synchronized arm trajectories and matching transformations so the relative end-effector poses follow the source demonstration.This is needed for precise interactions such as placing the lid in Box Cleanup.
  • Coordinated execution: Transform and Replay provide alternative schemes for obtaining trajectories in coordination subtasks.Transform derives a common transformation from object poses, whereas Replay uses source trajectories without transformation.
  • Coordinated execution: Replay can benefit handover phases in Can Sorting and Transport because it keeps trajectories within kinematic limits and fully executable.The scheme directly reuses the source trajectories rather than applying a transformation.

C. Sequential Subtasks

DexMimicGen generates bimanual trajectories by segmenting demonstrations per arm and transforming source motions in randomized simulation scenes. Sequential tasks use ordering constraints so one arm waits for the other's prerequisite subtask.

  • Sequential coordination: The ordering mechanism makes a post-subtask arm wait until the other arm completes its required pre-subtask.Pouring requires the ball to be poured before the bowl is moved to the pad.
  • Workflow: DexMimicGen segments source demonstrations into per-arm subtasks and generates trajectories by transforming and executing those segments in new simulation environments.The workflow records reference-object poses and applies relative transformations to source trajectories.
  • Simulation scope: The system spans nine tasks across parallel-jaw arms, dexterous-hand arms, and a humanoid embodiment.Tasks include precision manipulation, articulated-object manipulation, long-horizon behavior, coordination, and sequential execution.
  • Research platform: The environments and generated datasets provide a platform for analyzing factors that influence imitation-learning performance in bimanual dexterous manipulation.Source demonstrations are collected through embodiment-specific teleoperation systems.

VI. EXPERIMENTS

The experiments evaluate DexMimicGen across dataset sizes and broader initial-state distributions, while also presenting a real-world application. The study uses simulation experiments to examine data-generation and policy-learning effects.

  • Scope: The experiments analyze how data-generation and policy-learning choices affect policy performance.They cover setup, DexMimicGen features, performance analysis, and real-world application.
  • Dataset size: Dataset-size experiments compare policy success rates for datasets containing different numbers of generated demonstrations.Figure 5 specifically compares policies trained on varying dataset sizes.
  • Initial-state distributions: Broader initial-state distributions are evaluated by training and testing policies on data generated with those same broader distributions.Table II reports success rates under matched broader initial-state distributions.

A. Experimental Setup

The experimental setup uses source human demonstrations, generates 1000 trajectories per task with DexMimicGen, and trains several visuomotor behavioral-cloning policies. Evaluation compares source-only and generated-data training across tasks and distributions.

  • Data collection: DexMimicGen generates 1000 demonstrations per task from ten gripper-based or five dexterous-hand source demonstrations.Dexterous-hand collection uses fewer source demonstrations because of additional operator burden and time cost.
  • Policy training: Each dataset trains visuomotor policies with BC-RNN, BC-RNN-GMM, and Diffusion Policy, evaluated across three random seeds.The reported evaluation takes the maximum performance across the three seeds.
  • Evaluation: Policies trained on DexMimicGen datasets outperform source-only policies across all tasks.Drawer Cleanup rises from 0.7% to 76.0%, Threading from 1.3% to 69.3%, and Piece Assembly from 3.3% to 80.7% success.
  • Distribution robustness: DexMimicGen generates datasets that support performant policies under broader initial-state distributions.Policies are trained and evaluated using the same broader distributions.
  • Benchmark coverage: The system is evaluated on BiGym tasks FlipCup, DishwasherLoadPlates, and CupBoardsCloseAll.Data-generation success rates are 29.1%, 43.6%, and 76.4%, respectively.

C. DexMimicGen Analysis

DexMimicGen’s analysis compares generation strategies, dataset sizes, ordering constraints, and policy architectures. Results favor DexMimicGen over DemoNoise, show dataset-size gains with possible diminishing returns, and support ordering constraints and diffusion policies in relevant settings.

  • Generation baselines: Policies trained with DexMimicGen outperform DemoNoise policies by more than 58% across all tasks.DemoNoise cannot generate broader initial-state distributions D1 and D2 because it replays source configurations.
  • Dataset size: Performance increases substantially from 100 to 500 and 1000 generated demonstrations, but does not always improve from 1000 to 5000.The result indicates task-dependent diminishing returns at larger dataset sizes.
  • Generation strategies: Replay outperforms Transform on Transport, 63.3% versus 46.0%, while achieving comparable Can Sorting success, 97.3% versus 98.6%.Replay is therefore used as the default for handover tasks.
  • Sequential subtasks: Ordering constraints outperform unconstrained generation on Drawer Cleanup, 50.7% versus 48.0%, and Pouring, 88.7% versus 76.7%.Using different source demonstrations per arm increases diversity but requires ordering constraints for sequential subtasks.
  • Policy architectures: Diffusion Policy generally outperforms BC-RNN and BC-RNN-GMM on DexMimicGen datasets.BC-RNN-GMM especially underperforms on dexterous-hand tasks, differing from the RoboMimic study’s finding about GMM heads.

D. Real-World Evaluation

DexMimicGen supports real-world humanoid deployment through a digital-twin pipeline that reduces human data-collection effort. On can sorting, policies trained on generated data substantially outperformed those trained on the small source set.

  • Hardware and digital twin: DexMimicGen uses a Fourier GR1 humanoid with two dexterous hands and head-mounted and third-person cameras for real-world can sorting.The setup uses two 6-DoF Inspire hands and two Intel RealSense D435i cameras.
  • Data-generation pipeline: Four source demonstrations are replayed in a digital twin, which synchronizes initial object states before generating new real-world demonstrations.The digital twin aligns simulated and real object configurations and supports autonomous generation apart from environment resets.
  • Data-generation pipeline: 40 successful demonstrations were generated through the real-to-sim-to-real pipeline.The digital twin is used to improve safety during real-world data generation while reducing human effort.
  • Policy evaluation: 90% success versus 0% was achieved by policies trained on 40 generated demonstrations versus 4 source demonstrations, respectively.Both policies were evaluated over 10 trials for red and blue cups.
  • Scope: The paper deploys DexMimicGen on a real humanoid through a real2sim2real pipeline and releases datasets and environments for further research.The broader system includes nine simulation environments across three embodiments with different coordination behaviors.

VIII. APPENDIX OVERVIEW

The appendix covers implementation, result analysis, author contributions, human-versus-automated process steps, and limitations affecting task success and trajectory generation. It also documents action-distribution analysis using PCA.

  • Appendix contents: The appendix includes implementation details, result analysis, and a list of author contributions.These materials are organized as Appendix IX, X, and XI.
  • Human input and automation: Human input includes collecting source demonstrations, while segmentation and subtask-structure specification can require manual choices.Once demonstrations are collected and segmented and the subtask structure is specified, generation is fully automated.
  • Implementation details: Task-specific success checks filter failed demonstrations during data generation, typically by evaluating the final simulation state.An example is checking whether an object is placed in its target container.
  • Limitations: DexMimicGen does not explicitly handle robot-object collisions, which can cause generated-trajectory failures.The authors propose adding motion-planning modules in future work.
  • Result analysis: Threading success can fall below 70%, possibly because the threading object and hole are occluded from the third-person camera.The authors suggest visual reinforcement learning for active perception under high occlusion.
  • Result analysis: PCA projections of TwoArmCoffee actions show that DexMimicGen significantly expands end-effector action-distribution coverage.The analysis compares generated and source demonstrations using end-effector poses and finger-joint actions.

XI. AUTHOR CONTRIBUTIONS

The author-contribution section assigns project leadership, implementation, experimentation, infrastructure, resource acquisition, and paper-writing responsibilities across the team.

  • Project leadership and development: Zhenyu Jiang co-led project ideation and development, implemented data-generation code and environments, oversaw infrastructure, ran most experiments, and wrote the paper.His work spans core development, teleoperation, control infrastructure, and experimental execution.
  • Simulation and experiments: Yuqi Xie developed simulation environments, simulation teleoperation infrastructure, and rendering, and ran humanoid and real-robot experiments.The contribution combines environment development with experimental work.
  • Control infrastructure: Kevin Lin developed simulation control infrastructure, including whole-body IK controllers, and ran humanoid experiments.His contribution focuses on control for simulation experiments.
  • Real-robot infrastructure: Zhenjia Xu implemented real-robot teleoperation and policy-deployment infrastructure and helped oversee real-robot experiments.His role centers on transferring policies and teleoperation to the physical robot.
  • Prototype and benchmarks: Weikang Wan implemented the initial data-generation prototype and ran the BiGym experiments.His contribution covered early system development and benchmark experimentation.
  • Project leadership and support: Ajay Mandlekar, Linxi Fan, and Yuke Zhu co-led project development while contributing to oversight, resources, experiments, feedback, and paper writing.Their roles included algorithm and environment oversight, hardware and compute acquisition, and presentation support.
Loading 2410.24185v2…