PaperlayerView exact revision on arXiv

Source-linked AI summary

MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations

Ajay Mandlekar, Soroush Nasiriany, Bowen Wen, Iretiayo Akinola, Yashraj Narang, Linxi Fan, Yuke Zhu, Dieter Fox

arXiv:2310.17596v1cs.ROcs.AIcs.CVcs.LG

TL;DR

Human demonstrations are effective for robot imitation learning but costly to collect at scale. MimicGen adapts a small source set into demonstrations for diverse contexts, generating over 50K examples across 18 tasks from roughly 200 human demonstrations. The resulting data trains capable agents and can perform comparably to equal-sized human datasets.

  • Problem

    Large and diverse imitation-learning datasets require costly, time-consuming human demonstration collection.

  • Method

    MimicGen segments human demonstrations by object-centric subtask, transforms segments for new scenes, stitches them, and executes the resulting trajectories.

  • Results

    MimicGen generated over 50K demonstrations across 18 tasks from approximately 200 human demonstrations, with strong performance across diverse scenes, objects, and robot arms.

  • Takeaways & Limitations

    MimicGen data can provide agent performance comparable to an equal number of human demonstrations, supporting economical scaling of robot learning.

  • Takeaways & Limitations

    MimicGen assumes known object-centric subtasks, observes object poses during generation, and was demonstrated on quasi-static rigid-object tasks with same-category novel objects.

Abstract

from arXiv · show

Imitation learning from a large set of human demonstrations has proved to be an effective paradigm for building capable robot agents. However, the demonstrations can be extremely costly and time-consuming to collect. We introduce MimicGen, a system for automatically synthesizing large-scale, rich datasets from only a small number of human demonstrations by adapting them to new contexts. We use MimicGen to generate over 50K demonstrations across 18 tasks with diverse scene configurations, object instances, and robot arms from just ~200 human demonstrations. We show that robot agents can be effectively trained on this generated dataset by imitation learning to achieve strong performance in long-horizon and high-precision tasks, such as multi-part assembly and coffee preparation, across broad initial state distributions. We further demonstrate that the effectiveness and utility of MimicGen data compare favorably to collecting additional human demonstrations, making it a powerful and economical approach towards scaling up robot learning. Datasets, simulation environments, videos, and more at https://mimicgen.github.io .

1 Introduction

MimicGen addresses the cost and time required to collect large human-demonstration datasets by adapting a small set of demonstrations to diverse new contexts. It generates broad training data and compares favorably with collecting additional human demonstrations.

  • Motivation: Large robot-learning datasets can require substantial human labor, including over 20,000 trajectories or a 1.5-year collection effort in prior examples.A 200-demonstration dataset for a simple coke-can task achieved 73.3% success, while RT-1 used extensive collection across operators, kitchens, and robot arms.
  • Motivation: MimicGen targets a general-purpose system that integrates with imitation-learning pipelines across a wide spectrum of tasks.The motivation is to avoid assumptions about specific tasks and algorithms made by earlier replay-based methods.
  • Approach: MimicGen divides human demonstrations into object-centric segments, transforms them for new scenes, stitches them together, and executes the resulting trajectories.The system adapts demonstrations to new object poses and scenes before collecting new demonstrations.
  • Results: 50K+ demonstrations across 18 tasks were generated from approximately 200 source human demonstrations across diverse scenes, object instances, and robot arms.The tasks include long-horizon and high-precision manipulation such as pick-and-place, insertion, and articulated-object interaction.
  • Results: Equal-sized synthetic and human datasets produced comparable agent performance, raising questions about when additional human data is necessary.The comparison is framed as using 200 demonstrations generated from 10 human demonstrations versus 200 human demonstrations.

2 Related Work

Robot-learning data commonly comes from human teleoperation, but large-scale collection is costly; MimicGen instead seeks to reuse a small set of human samples to generate larger datasets.

  • Related Work: Human demonstrators commonly teleoperate robot arms, but collecting large datasets requires extensive time, effort, and cost.The paper contrasts this with trial-and-error and pre-programmed simulation demonstrators, which can be difficult to scale to complex tasks.
  • Related Work: MimicGen trains policies from generated data using imitation learning rather than relying only on additional human data collection.

3 Problem Setup

The problem setup uses a small source dataset to generate demonstrations for changed initial states, objects, or robot arms. MimicGen represents demonstrations as object-relative controller trajectories under explicit task assumptions.

  • Imitation Learning: The task is modeled as an MDP with a policy mapping states S to actions A, and demonstrations provide the imitation-learning data.Behavioral Cloning is used to train the policy with a negative log-likelihood objective.
  • Problem Statement: MimicGen uses a small source dataset to generate a large dataset for the same task or variants with changed initial states, objects, or robot arms.A new demonstration samples a start state, chooses and adapts a source demonstration, and produces a new trajectory.
  • Assumptions: MimicGen assumes delta-pose end-effector actions with gripper open/close commands, allowing actions to be treated as controller target poses.This establishes the correspondence between demonstration actions and target poses used for adaptation.
  • Assumptions: Tasks are assumed to contain a known sequence of object-centric subtasks, each defined relative to one object’s coordinate frame.The sequence is typically specified by a human.
  • Assumptions: Object poses must be observable at the start of each subtask during data collection, though not during policy deployment.

4 Method

MimicGen parses demonstrations into object-centric segments, adapts selected segments to object poses in a new scene, and executes them as end-effector target sequences. It retains only successful generated demonstrations and can span varied task contexts and robot arms.

  • Segmenting Demonstrations: Source trajectories are split into contiguous segments, one for each object-centric subtask, using metrics that detect subtask completion.
  • Generating Demonstrations: For each subtask, MimicGen chooses a source segment, transforms it for the new context, and executes it in the scene.The three-step process is repeated across the task’s subtask sequence.
  • Selecting Segments: Reference segments can be selected randomly or using the relevant object poses at the start of each subtask.
  • Transforming Segments: MimicGen transforms controller target poses from world-relative source trajectories according to the corresponding object pose in the new scene, preserving object-relative motion.The source segment is represented as a sequence of controller target poses over K timesteps.
  • Transforming Segments: MimicGen adds an interpolation segment when the new segment’s first target pose is far from the robot’s current end-effector pose.The interpolation linearly connects the current pose to the transformed segment’s start.
  • Executing Segments: The robot executes each transformed target pose as a delta-pose action paired with the source segment’s gripper command.
  • Filtering Data: Successful demonstrations are retained after execution, and the data generation rate is the ratio of successful trajectories to total attempts.Control and arm-kinematics deviations can otherwise cause task failure.
  • Generality: Because the pipeline depends on object and controller frames, it can generate data across changed initial distributions, objects with canonical frames, and compatible robot arms.

5 Experiment Setup

MimicGen is evaluated across diverse manipulation tasks and task variants, including broader object resets, different objects or robot arms, and mobile manipulation. Each task uses a small teleoperated source dataset to generate larger imitation-learning datasets.

  • Task coverage: The experiments cover pick-and-place, contact-rich interactions, articulation, and mobile manipulation, including long-horizon behavior chains.
  • Task variants: Tasks use default D0, broader D1, and sometimes more challenging D2 reset distributions with increasingly varied object locations.In Threading, the tripod is fixed in D0, both objects move in D1, and both are randomized in novel regions in D2.
  • Task coverage: The task suite combines robosuite and Factory environments, spanning basic stacking, contact-rich manipulation, and other task categories.
  • Data generation: For most tasks, one operator collected 10 D0 demonstrations, then MimicGen generated 1000 demonstrations for each task variant.Mobile Kitchen used 25 source demonstrations, while Square used 10 demonstrations from the robomimic Square PH dataset.

6 Experiments

Experiments show that MimicGen-generated data improves policy performance across source tasks, broader configurations, object instances, robot hardware, mobile manipulation, and high-precision assembly. Additional analyses compare generated data with human demonstrations and examine source-demo dependence, scaling, and real-world transfer.

  • Applications of MimicGen: 11.3% to 90.7% on Square, 19.3% to 98.0% on Threading, and 1.3% to 82.0% on Three Piece Assembly after generating D0 data.These examples illustrate substantial improvement over policies trained on the small source datasets.
  • Applications of MimicGen: 42% to 99% success on D1 shows that generated datasets support policies across broader initial-state distributions.Some source demonstrations contained objects that never moved, yet generated data covered substantial workspace regions.
  • Applications of MimicGen: 90.7% and 75.3% success rates were achieved for an unseen mug and a set of 12 mugs, respectively.
  • Applications of MimicGen: 80%-91% policy performance across four robot arms was obtained despite Square D0 generation rates ranging from 38%-74%.The comparison includes Panda, Sawyer, IIWA, and UR5e arms across D0 and D1 variants.
  • Applications of MimicGen: 2.0% to 46.7% image-based and 2.7% to 76.7% low-dimensional success on Mobile Kitchen demonstrates application beyond static tabletop manipulation.
  • Applications of MimicGen: 82%-99% nominal-task success in Factory assembly improved over 9%-15% source-data policies, while D1 and D2 achieved 37%-81%.The tasks were Nut-and-Bolt Assembly, Gear Assembly, and Frame Assembly.
  • MimicGen analysis: Using 10, 50, or 200 source demonstrations yielded modest 2% to 21% performance differences, while one demonstration sometimes performed much worse.The effect depended on the task: Square degraded with one demonstration, whereas Three Piece Assembly showed no significant change.

7 Limitations

MimicGen depends on task and perception assumptions, can produce biased datasets, and does not guarantee collision-free motions. Its demonstrations were evaluated mainly on quasi-static rigid-object tasks with same-category novel objects.

  • Assumptions: MimicGen assumes known object-centric subtasks and object pose estimates at each subtask start.
  • Dataset limitations: Filtering only by task success can leave generated datasets biased.
  • Motion limitations: Linear interpolation between human segments does not guarantee collision-free motion and may hurt agent performance.
  • Scope: The demonstrated scope is quasi-static tasks with rigid objects, with novel objects assumed to come from the same category.

8 Conclusion

MimicGen uses a small set of human demonstrations to generate large datasets across diverse tasks, scenes, objects, and robots. The paper reports strong policy performance, while identifying practical limitations and boundaries for data generation.

  • MimicGen generated over 50K demonstrations across 18 tasks from less than 200 human demonstrations, including long-horizon and high-precision manipulation.
  • Agent performance on MimicGen data can be comparable to performance on an equal number of human demonstrations.
  • Pose-estimation errors can reduce data generation rates, although policies trained on the generated data maintained the same performance level in simulation.
  • MimicGen data generation can fail when naive interpolation causes collisions or transformed segment endpoints are difficult for the robot arm to reach.
  • MimicGen has been demonstrated on geometrically similar rigid-body objects with similar scales, while soft or more geometrically diverse objects remain future work.

D Limitations

MimicGen is positioned as a general-purpose alternative to costly human data collection, with demonstrations transferred across robot hardware and object variants. Its limitations include assumptions about task structure, object geometry, and single-arm operation.

  • Limitations: MimicGen assumes a known, fixed sequence of object-centric subtasks, observed reference-object poses, and one reference object per subtask.These assumptions constrain how tasks and subtask motions are represented during data generation.
  • Limitations: MimicGen supports single-arm tasks, while multi-arm manipulation remains outside its current scope.Extending the system to multi-arm tasks is left for future work.
  • Related work: MimicGen uses small human datasets to generate data across diverse tasks, contrasting with approaches that collect thousands of demonstrations or rely on task-specific replay assumptions.The related-work discussion contrasts MimicGen’s use of a small number of demonstrations with large-scale human collection, simulation demonstrators, and replay-based methods.
  • Robot transfer: 80%-91% success on Square D0 and 89%-98% on Threading D0 were achieved across transferred robot arms despite 20%-74% data generation rates.The source demonstrations were collected on Panda and transferred to Sawyer, IIWA, and UR5e arms.
  • Limitations: Object transfer was demonstrated only for geometrically similar rigid-body objects with aligned canonical frames and similar scales.Soft objects and more geometrically diverse objects remain future-work directions.
  • Object transfer: MimicGen was evaluated on an unseen mug and a set of 12 mugs, extending object transfer beyond the source Mug Cleanup dataset.The O1 variant uses an unseen mug, while O2 selects a new mug per episode.

H Real Robot Results

Real-world MimicGen data collection achieved substantial generation success, but trained policies performed worse than in simulation. Longer interpolation segments and limited real-world data contributed to this gap, while more flexible policies improved results.

  • Real-world performance: 82.3% data collection success for Stack and 52.1% for Coffee contrasted with policy success of 36% and 14%, versus 100% and ∼90% in simulation.The real-world dataset contained 100 demonstrations instead of 1000, alongside other factors affecting the sim-to-real gap.
  • Interpolation: 99.3% to 68.7% on Stack D1 and 50.7% to 11.3% on Pick Place show that larger interpolation segments can sharply reduce image-based agent success.The real-world safety setting used n_interp = 25 and n_fixed = 25 instead of the simulation default n_interp = 5 and n_fixed = 0.
  • Policy models: Diffusion Policy reached 76% success across 50 Stack evaluations, compared with 36% for BC-RNN.The result is reported as an improvement from training a more flexible imitation-learning model on the real-world Stack dataset.
  • Data generation: MimicGen generates physically consistent new data through online environment interaction, unlike offline augmentation that can struggle with plausible interactions in new settings.Offline augmentation remains complementary; pixel-shift randomization was used when training image-based agents.

K.2 Parsing the Source Dataset into Object-Centric Subtask Segments

MimicGen parses each source trajectory into contiguous segments aligned with object-centric subtasks, using subtask-end metrics or human annotations. Each segment is associated with a reference object for spatial adaptation.

  • Parsing procedure: Each trajectory τ is split into contiguous segments τ = (τ1, τ2, ..., τM), with one segment per object-centric subtask.The parsing procedure produces segments corresponding to subtasks S_i(o_Si).
  • Boundary detection: Subtask boundaries are detected with metrics based on robot and object states, such as contact, lifting, or task-success checks.These metrics identify events including mug grasp, mug placement, pod grasp, and machine-lid closure.
  • Object-centric structure: The task structure assumes each subtask has one reference object, such as the nut and peg in Square or the needle and tripod in Threading.The figure summarizes the object-centric subtask endpoints and reference objects for selected tasks.
  • Boundary detection: Humans can instead annotate subtask ends manually, which is practical because the source demonstration set is usually small.The work uses automated metrics when they are readily available or easy to craft.
  • Task examples: Square, Threading, and Gear Assembly each use two subtasks, while Stack Three and Three Piece Assembly use four.Grasping subtasks use contact or lifting checks, whereas insertion or placement subtasks use task-specific success checks.

L Tasks and Task Variants

The appendix covers a broad task suite spanning basic stacking, long-horizon chaining, mobile manipulation, factory assembly, contact-rich interaction, and real-world variants. Tasks vary initial object poses, orientations, object instances, and subtasks, while MimicGen transforms object-centric end-effector segments under a delta-pose control representation.

  • Task coverage: The suite spans pick-and-place, insertion, articulation, mobile manipulation, and long-horizon tasks that chain several behaviors.The task categories include basic, long-horizon, mobile manipulation, factory, and contact-rich behaviors.
  • Task variants: Stack, Stack Three, Pick Place, Kitchen, Coffee Preparation, and Mobile Kitchen vary the number of subtasks and reset distributions.Examples range from two-subtask stacking to eight-subtask pick-and-place and multi-step kitchen and coffee tasks.
  • Task variants: Contact-rich variants progressively broaden object positions and rotations for square insertion, coffee preparation, multi-piece assembly, cleanup, and threading.Variants include fixed objects in D0 and increasingly varied positions or orientations in D1 and D2.
  • Segment transformation: MimicGen’s segment transformation is derived for end-effector controller target poses represented by homogeneous 4×4 transforms.The derivation treats each source subtask segment as a sequence of controller target poses in the world frame.
  • Control representation: The assumed action space uses delta-pose commands with three translation components, three axis-angle rotation components, and one gripper command.The controller converts each delta-pose action into an absolute end-effector pose target.

N.4 Action Noise

Action noise is injected while executing transformed segments, trading lower generation success for better downstream policy performance. Comparisons with replay, selection strategies, and training procedures assess how MimicGen data generation choices affect agents.

  • Action noise: No-noise generation raises data-generation success but reduces learned-agent performance, with drops up to 30% for low-dim and 40% for image observations.The largest reported generation increase is 33% on Threading D0.
  • Selection strategy: Nearest-neighbor and per-subtask selection can substantially improve data-generation rates, while agent performance usually changes little when these strategies are removed.Per-subtask selection was found helpful for pick-and-place tasks.
  • Replay comparison: MimicGen transformation outperforms replaying the 10 source demonstrations with equivalent noise on new configurations.On D0, replay success rises from 11.3 to 42.0 for Square and from 19.3 to 74.0 for Threading, while MimicGen-trained policies reach 90.7 and 98.
  • Hyperparameters: Most simulation tasks use σ = 0.05, ninterp = 5, and nfixed = 0, whereas real tasks reduce noise to σ = 0.02 and increase interpolation and fixed steps.Selection strategies vary by task, including nearest-neighbor selection for several pick-and-place and stacking tasks.
  • Policy evaluation: Policies use low-dim or image observations and are trained with BC-RNN, with evaluation based on 50 rollouts per simulation checkpoint or 50 real-world episodes.Simulation reports the maximum success rate across three seeds; real-world evaluation uses the final checkpoint.

P Data Generation Success Rates

Generated datasets produce strong policy performance across tasks and reset distributions, often exceeding the success rate of the data-generation process itself. Low-dim agents show the same broad effectiveness, and 200 MimicGen demonstrations perform similarly to 200 human demonstrations despite using only 10 source human demonstrations.

  • Generation versus policy success: Policy success can substantially exceed data-generation success; Gear Assembly reaches 92.7% versus 46.9% on D0, 76.0% versus 8.2% on D1, and 64.0% versus 7.1% on D2.These figures compare policy success rates with data-generation rates for the same Gear Assembly variants.
  • Low-dim agents: Low-dim agents show large improvement on D0 and remain performant on broader D1 and D2 reset distributions across the main generated datasets.Table Q.1 compares agents trained on 10 source demonstrations with agents trained on 1000 MimicGen demonstrations per reset distribution.
  • Dataset comparison: 200 MimicGen demonstrations achieve performance similar to 200 human demonstrations, although MimicGen uses only 10 source human demonstrations.Using larger source datasets did not produce significant low-dim agent improvement.

R Bias and Artifacts in Generated Data

The generated data can contain scene-configuration bias and motion artifacts, although policy results are robust across multiple generation seeds and moderate pose-estimation noise. Coverage varies substantially by task, leaving deeper analysis of generated-data properties open.

  • Scene bias: Support coverage ranges from 98.8% for Coffee D1 to 43.5% for Three Piece Assembly D1, confirming substantial bias in some generated datasets.Other reported coverage values include 89.3% for Coffee D2, 92.6% for Square D1, and 61.2% for Threading D2.
  • Artifacts: MimicGen may produce long interpolation paths, unnatural motions, and other artifacts when transformed segments are bridged between source behaviors.The paper reports evidence of such artifacts and notes that generated data retains only successful traces.
  • Seed variation: Across three data-generation seeds, reported generation rates deviate by less than 0.6% and policy success rates by less than 2%.The multi-seed evaluation trains three policy seeds per generated dataset.
  • Pose-estimation noise: Adding 5 mm/5° or 10 mm/10° pose noise lowers data-generation rates while leaving policy success comparatively robust.For Square D0, policy success changes from 90.7% to 89.3% and 84.7% under the two noise levels.
Loading 2310.17596v1…