PaperlayerView exact revision on arXiv

Source-linked AI summary

MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations

Ajay Mandlekar, Soroush Nasiriany, Bowen Wen, Iretiayo Akinola, Yashraj Narang, Linxi Fan, Yuke Zhu, Dieter Fox

arXiv:2310.17596v1cs.ROcs.AIcs.CVcs.LG

TL;DR

Human demonstrations are costly and time-consuming to collect, even as diverse robot-learning settings require large datasets. MimicGen adapts a small set of demonstrations to new contexts, generating over 50K demonstrations across 18 tasks from about 200 source demonstrations, with agents trained on the generated data achieving strong performance.

  • Problem

    Human demonstrations are costly and time-consuming to collect, while diverse robot-learning settings require datasets spanning tens of thousands of demonstrations.

    Source: §1 · p. 1

  • Method

    MimicGen segments human demonstrations around objects, spatially transforms and stitches the segments for new scenes, then has the robot execute the resulting trajectories.

    Source: §1 · p. 2

  • Results

    50K+ new demonstrations were generated for 18 tasks across two simulators and a physical robot arm using about 200 source human demonstrations.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Agent performance on MimicGen data can be comparable to performance on an equal number of human demonstrations.

    Source: §8 · p. 8

    MimicGen assumes known object-centric subtasks and requires object pose estimates at each subtask’s start during data generation.

    Source: §7 · p. 8

Abstract

from arXiv · show

Imitation learning from a large set of human demonstrations has proved to be an effective paradigm for building capable robot agents. However, the demonstrations can be extremely costly and time-consuming to collect. We introduce MimicGen, a system for automatically synthesizing large-scale, rich datasets from only a small number of human demonstrations by adapting them to new contexts. We use MimicGen to generate over 50K demonstrations across 18 tasks with diverse scene configurations, object instances, and robot arms from just ~200 human demonstrations. We show that robot agents can be effectively trained on this generated dataset by imitation learning to achieve strong performance in long-horizon and high-precision tasks, such as multi-part assembly and coffee preparation, across broad initial state distributions. We further demonstrate that the effectiveness and utility of MimicGen data compare favorably to collecting additional human demonstrations, making it a powerful and economical approach towards scaling up robot learning. Datasets, simulation environments, videos, and more at https://mimicgen.github.io .

1 Introduction

MimicGen addresses the cost of scaling imitation learning by adapting a small set of human demonstrations to new contexts. It segments demonstrations into object-centric chunks, transforms and stitches them for new scenes, and uses the resulting trajectories as training data.

1 Introduction

  • A 200-demonstration dataset achieved 73.3% success on a simple bin-transfer task, while RT-1’s nearly 1.5-year collection effort yielded 97% success across a handful of kitchens.Other efforts have used tens of thousands of trajectories to generalize across modest changes in objects and goals.Source: §1 · p. 1
  • Prior replay-based methods exploit similar skills across contexts, but task- and algorithm-specific assumptions motivate a general-purpose system for existing imitation-learning pipelines.MimicGen is motivated by the possibility of re-purposing demonstrations such as similar mug-grasping trajectories across different countertop locations.Source: §1 · pp. 1–2
  • MimicGen segments human demonstrations around objects, spatially transforms segments for new object poses, stitches them together, and has the robot follow the resulting trajectory.The process selects a source demonstration for each new scene and collects the executed trajectory as a new demonstration.Source: §1 · p. 2
  • MimicGen generated 50K+ demonstrations for 18 tasks across two simulators and a physical robot arm from ∼200 source human demonstrations.The tasks span diverse scene configurations, object instances, robot arms, and manipulation skills.Source: §1 · p. 2
  • Comparable agent performance resulted from equal amounts of generated and human data, such as 200 demonstrations generated from 10 human demonstrations versus 200 human demonstrations.This comparison raises the question of when additional human data is necessary.Source: §1 · p. 2

2 Related Work

Related work highlights the cost and scaling limits of existing demonstration collection, while MimicGen uses a small set of human samples to generate large datasets online. Its approach connects to imitation learning and replay-based methods but differs from offline data augmentation by generating new datasets online.

2 Related Work

  • MimicGen uses a small set of human samples to generate large datasets and trains policies on the generated data with imitation learning.Imitation learning has been used extensively in prior work [1] [19] [25] [33] [34].Source: §2 · p. 2
  • Trial-and-error [14] [15] [17] and pre-programmed simulation demonstrators [19] [20] [21] [22] can be difficult to scale to complex tasks, while collecting human teleoperation data [4] [5] [6] [23] [24] [25] [27] demands substantial time, effort, and cost.Source: §2 · p. 2
  • Unlike offline data augmentation [35] [36] [37] [39] [42] [43] [44] [45], MimicGen generates new datasets online using a mechanism similar to replay-based imitation approaches [46] [47].Replay-based imitation approaches solve tasks by replaying prior demonstrations.Source: §2 · p. 2

3 Problem Setup

MimicGen adapts a small set of human demonstrations into successful trajectories for the same task or task variants. Its setup assumes delta-pose actions, a known sequence of object-centric subtasks, and access to relevant object poses at subtask starts during data collection.

Imitation Learning

  • Tasks are modeled as MDPs with policies mapping states to actions, trained by behavioral cloning to minimize expected negative log-likelihood of demonstrated actions.The objective is arg minθ E(s,a)∼D[−log πθ(a|s)].Source: §3 · p. 2

Problem Statement and Assumptions

  • From a small human source set, MimicGen adapts demonstrations to the same task or variants and adds executed trajectories to the generated dataset only after success.The target task may differ in its initial-state distribution, objects, or robot arm.Source: §3 · p. 3

Assumption 1: delta end effector pose action space

  • The action space uses end-effector delta-pose commands and gripper open/close commands, making demonstration actions equivalent to controller target poses [4] [5] [6] [33].This lets the system treat actions as a sequence of end-effector target poses.Source: §3 · p. 3

Assumption 2: tasks consist of a known sequence of object-centric subtasks

  • Tasks are assumed to follow a known sequence of object-centric subtasks, each defined relative to one object’s coordinate frame, as in Di Palo et al..A human can typically specify the sequence.Source: §3 · p. 3

Assumption 3: object poses can be observed at the start of each subtask during data collection

  • During data collection, the relevant object’s pose is assumed observable at each subtask’s start, but not during policy deployment.Source: §3 · p. 3

4 Method

MimicGen parses a small source dataset into object-centric subtask segments, then adapts and executes those segments to generate demonstrations for new scenes.

4.1 Parsing the Source Dataset into Object-Centric Segments

  • Source trajectories are divided into contiguous segments, one per object-centric subtask, using metrics that automatically detect each subtask’s end.This parsing step assumes such end-detection metrics are available.Source: §4.1 · p. 3

4.2 Transforming Source Data Segments for a New Scene

  • For each subtask, MimicGen selects a source segment randomly or using object poses, retargets its end-effector pose sequence to the new object pose, and interpolates from the current end-effector pose when needed.The interpolation bridges the current end-effector pose and the first pose of the transformed segment.Source: §4.2 · p. 4
  • MimicGen executes each adapted segment by converting target poses into delta-pose actions, pairing them with source gripper actions, and retaining only demonstrations that pass a task-success check.Control and arm-kinematics deviations can cause failures; the data generation rate is the ratio of successful trajectories to attempts.Source: §4.2 · p. 4
  • Because the pipeline relies on object and robot-controller frames, it supports different initial-state distributions, objects with canonical frames, and robot arms sharing an end-effector control-frame convention.Experiments varied the initial-state distribution, task object, or robot arm across task variants.Source: §4.2 · p. 4

5 Experiment Setup

MimicGen was evaluated across diverse manipulation tasks and reset variants, using small human-collected source datasets to generate successful demonstrations for behavioral-cloning policy training.

Tasks and Task Variants

  • Task variants used default D0, broader D1, and sometimes harder D2 reset distributions; some also changed the robot arm or object instances.For Threading, D0 fixes the tripod location, D1 allows the tripod and needle to move, and D2 randomizes them in novel workspace regions.Source: §5 · p. 4
  • MimicGen was applied to tasks spanning pick-and-place, insertion, articulation, mobile manipulation, and long-horizon behavior chaining.The tasks showcase data generation across a diverse range of manipulation behaviors.Source: §5 · p. 4, Figure 3 · p. 5
  • Tasks covered basic stacking, contact-rich, long-horizon, mobile, and high-precision Factory assembly categories, using robosuite/MuJoCo [49] and Factory/Isaac Gym [51] [52].The listed categories include tasks such as Stack, Threading, Coffee Preparation, Mobile Kitchen, and Gear Assembly.Source: §5 · p. 5

Data Generation and Imitation Learning Methodology

  • Each task used 10 operator-collected D0 source demonstrations, except Mobile Kitchen used 25 and Square used 10 from the robomimic Square PH dataset.The demonstrations were collected using a teleoperation system [23].Source: §5 · p. 5
  • MimicGen generated 1,000 successful demonstrations per task variant, discarding failed attempts, for RGB-observation RNN behavioral-cloning policies.Policy performance was reported as the maximum success rate across evaluations over three seeds.Source: §5 · p. 5

6 Experiments

Experiments show that MimicGen supports diverse tasks and conditions, often yielding strong policies from few human demonstrations. Its generated data can perform comparably to equal-sized human datasets and can be used on real robot arms.

6.1 Applications of MimicGen

  • On the source tasks, MimicGen raised success from 11.3% to 90.7% on Square, 19.3% to 98.0% on Threading, and 1.3% to 82.0% on Three Piece Assembly.The results compare agents trained on small source datasets with those trained on generated D0 datasets.Source: §6.1 · p. 5, Figure 4 · p. 6
  • On broader initial-state distributions, MimicGen-trained agents achieved 42% to 99% on D1, including regimes where objects moved across substantial portions of the workspace despite not moving in source demos.The paper reports generated datasets and trained policies on broader D1 and D2 distributions; Figure 4 describes agents as performant on both.Source: §6.1 · pp. 5–6, Figure 4 · p. 6

6.2 Comparing MimicGen to using more human data

  • In most cases, agents performed similarly on 200 MimicGen demos and 200 human demos, although the MimicGen demos came from just 10 human demos.MimicGen can also generate larger datasets, unlike the time-consuming process of collecting more human data.Source: Figure 4 · p. 6, §6.2 · p. 7

6.1 Applications of MimicGen

  • MimicGen produced policies with 90.7% success for an unseen mug and 75.3% for 12 mugs, and 80%-91% performance across four robot arms despite differing generation rates.The arm results span Panda, Sawyer, IIWA, and UR5e; policies from inexperienced operators and different teleoperation devices had performance comparable to those in Figure 4.Source: §6.1 · p. 6
  • MimicGen extended to mobile manipulation and simulator-agnostic Factory tasks, with Mobile Kitchen image success rising from 2.0% to 46.7% and Factory D0 policies reaching 82%-99%.In Factory, source-dataset policies achieved 9%-15%, while policies on wider D1 and D2 resets achieved 37%-81%.Source: §6.1 · p. 6

6.2 Comparing MimicGen to using more human data

  • On Hammer Cleanup and Kitchen, MimicGen-trained BC-RNN reached 100.0% on generated D0 data, versus BUDS's 68.6% and 72.0% on D0, respectively.With 10 human demos, MimicGen also enabled proficient Nut Assembly and Pick Place agents, whereas Mandlekar et al. reported proficiency only on easier, single-stage versions using over 1000 demos.Source: §6.2 · pp. 6–7

6.3 MimicGen Analysis

  • Using 10, 50, or 200 source demos changed performance modestly (2%-21%), while generated-data gains were large from 200 to 1000 demos but limited from 1000 to 5000.Source-demo choice and segment selection also mattered: some generated datasets drew most episodes from only a few source demonstrations.Source: §6.3 · p. 7

6.4 Real Robot Evaluation

  • Generation success rate did not necessarily predict policy success: Factory Gear Assembly D1 had an 8.2% generation rate but a 76.0% agent rate.Real-robot generation succeeded at 82.3% for Stack and 52.1% for Coffee; over 50 evaluations, agents reached 36% and 14% success, respectively, versus 0% for source-data agents on both. The authors suggest that 50 interpolation steps for hardware safety may make motions harder to imitate than the 5 used in simulation.Source: §6.3 · p. 7, §6.4 · p. 8, Figure 5 · p. 8

7 Limitations

MimicGen relies on known object-centric subtasks and pose estimates, and its data-generation and motion assumptions can limit dataset quality and scope.

7 Limitations

  • MimicGen assumes object-centric subtasks are known and requires object pose estimates at each subtask’s start during data generation.This is Assumption 3 in Section 3.Source: §7 · p. 8
  • Filtering generation attempts only by task success can bias the resulting datasets.The discussion points to Appendix R.Source: §7 · p. 8
  • Linear interpolation between human segments does not guarantee collision-free motion and can potentially hurt agent performance.See Appendix N.2 and Appendix H.Source: §7 · p. 8
  • MimicGen was demonstrated on quasi-static tasks with rigid objects, with novel objects assumed to come from the same category.Source: §7 · p. 8

8 Conclusion … C FAQ

MimicGen uses a small set of human demonstrations to generate large datasets across tasks, scenes, objects, and robots, and policies trained on its data achieve strong performance. The conclusion highlights comparable performance to an equal number of human demonstrations, while the FAQ discusses the system’s scope, limitations, and supporting resources.

8 Conclusion

  • Over 50K demonstrations across 18 tasks were generated from less than 200 human demonstrations, and MimicGen-trained agents achieved strong performance, comparable to agents trained on an equal number of human demonstrations.The authors say this comparison motivates investigating when to collect additional demonstrations versus making more effective use of a small set, and where operator time is best spent.Source: §8 · p. 8

A Overview

  • Appendices cover limitations, transfer, real-robot results, task and training details, and further analyses; released datasets and reproduction instructions accompany a breakdown of 175 source demos into 36K, 12K, and 2K generated demos.The reported breakdown excludes 200 real-robot demos generated from 20 source demos and extra datasets for appendix experiments and analysis.Source: A Overview · p. 15, C FAQ · pp. 17–18

B Author Contributions

  • Author contributions spanned project leadership and experiments, mobile-manipulation implementation, perception, Factory task design and infrastructure, and project advising.The listed roles also included developing assets, supporting teleoperation and imitation-learning compatibility, and writing parts of the paper.Source: B Author Contributions · p. 16

C FAQ

  • MimicGen handles tasks beyond standard pick-and-place, including 1mm-tolerance gear insertion, 4mm alignment of four picture-frame holes, and long-horizon tasks with up to 8 subtasks.The demonstrated behaviors also include insertion, pushing, and articulation, with pose variation beyond typical prior BC-from-human-demonstration works [1] [4] [5] [6] [33].Source: C FAQ · p. 17
  • Policies trained on generated data maintained the same performance under simulated object-pose estimation errors, although data generation rates could decrease.In real-world use, MimicGen estimates object poses from an initial RGBD image and is compatible with pose-estimation methods.Source: C FAQ · p. 17
  • Online generation produces physically consistent data through environment interaction, while MimicGen-trained agents can be closed-loop, use varied offline learning algorithms, and outperform data-generation success rates in many cases.Offline augmentation may struggle to produce plausible interactions without physics or causal-dependency knowledge, while replay-based agents are generally architecture-specific and open-loop.Source: C FAQ · p. 17
  • Generation failures can result from geometry-unaware linear interpolation, transformed segments that ignore arm kinematics, or source motions that controllers track with difficulty.The first two issues can cause unintended collisions or starting poses that are difficult to reach.Source: C FAQ · pp. 17–18
  • Object transfer was demonstrated for geometrically similar rigid objects of similar scale, assuming aligned canonical frames and pose-based descriptions; soft or more diverse objects remain future work.Examples include mugs, carrots, and pans.Source: C FAQ · p. 18

D Limitations … G Object Transfer

The sections outline MimicGen’s assumptions and limits, situate it among data-collection and imitation-learning approaches, and describe its robot- and object-transfer experiments.

D Limitations

  • MimicGen requires a known, fixed object-centric subtask sequence, object-pose estimates at each subtask start, and exactly one reference object per subtask.The authors report real-world use with pose estimation and some tolerance to pose errors; simulation generation and real-world deployment are also suggested as future work.Source: D Limitations · p. 19
  • Its success-rate-only filtering may leave bias or artifacts, while linear interpolation ignores scene geometry and can cause collisions; longer segments can also harm policy learning.The authors chose simple interpolation to avoid planner complexity and leave better, policy-compatible and safer segments for future work.Source: D Limitations · p. 19
  • MimicGen was demonstrated on quasi-static tasks, not dynamic tasks, and different object dynamics remain untested; tracking target end-effector paths may enable some such transfers.The paper notes that transfer across robot-arm dynamics was demonstrated and gives pushing across table frictions as a possible object-dynamics case.Source: D Limitations · p. 19
  • Mobile manipulation assumes the base and arm do not move simultaneously and copies reference base actions; MimicGen currently supports only single-arm tasks.More sophisticated base-motion generation and multi-arm support are left for future work.Source: D Limitations · pp. 19–20

E Full Related Work

  • MimicGen contrasts with large-scale human, self-supervised, and scripted-simulation data collection by using a few minutes of initial human demonstrations to generate datasets.The related-work discussion also distinguishes it from systems that selectively query humans for additional demonstrations.Source: E Full Related Work · p. 21
  • MimicGen uses BC for single-task policies and can serve diverse offline learners; unlike replay-based agents such as Di Palo et al., it uses replay to generate datasets.The paper describes replay-based approaches as complementary and reports benefits from training on large datasets rather than directly deploying a replay-based agent.Source: E Full Related Work · p. 21

F Robot Transfer

  • Across robots, D0 data-generation rates ranged from 20%-74%, while policies achieved similar success rates: 80%-91% on Square D0 and 89%-98% on Threading D0.Source: F Robot Transfer · p. 22

G Object Transfer

  • Object-transfer experiments used Mug Cleanup with an unseen mug and a 12-mug set, but demonstrated transfer only among similarly scaled, geometrically similar rigid objects with aligned canonical frames.The authors leave transfer to soft or more geometrically diverse objects for future work.Source: G Object Transfer · p. 23, Table G.1 · p. 23

H Real Robot Results … K.1 How Tasks can be broken up into Object-Centric Subtasks

MimicGen generated real-world robot data with object-pose estimates, but policy success lagged simulation; the appendices examine possible causes, alternative data-generation approaches, and object-centric task decomposition.

H Real Robot Results

  • Real-world data generation relied on estimated object poses from calibrated RGB-D imagery, point-cloud segmentation, and pose refinement.The pipeline removed the table plane with RANSAC, clustered objects with DBSCAN, then initialized poses globally and refined them with ICP; the authors note that other perception methods could be used.Source: H Real Robot Results · p. 24
  • 76% success across 50 evaluations with Diffusion Policy on real-world Stack exceeded BC-RNN’s 36%.The authors attribute the real/simulation gap partly to only 100 real demonstrations instead of 1,000 and to longer interpolation segments; matching those segments in simulation reduced image-based Stack D1 success from 99.3% to 68.7% and Pick Place from 50.7% to 11.3%.Source: H Real Robot Results · pp. 24–25, Figure H.1 · p. 24

I Different Demonstrators

  • Policies trained on generated data showed similar performance across demonstrators: deviations were 0%–17% for a different teleoperation device and 0%–14% for lower-quality source data.The authors generated 1,000 demonstrations per task variant from 10 source demonstrations in each tested dataset; the results suggest that large-scale generation may mitigate harmful effects of low-quality demonstrations.Source: I Different Demonstrators · p. 26, Table I.1 · p. 26, Table I.2 · p. 26

J Motivation for MimicGen over Alternative Methods

  • MimicGen’s dataset-generation approach supports offline policy learning and evaluation, while its generated interactions are physically consistent and can adapt to new manipulation contexts.The paper contrasts this with replay-based methods and offline augmentation, while noting that pixel-shift augmentation can complement MimicGen.Source: §J.1 · p. 27, §J.2 · p. 27

J.1 Replay-Based Imitation Learning

  • Unlike replay-based methods [46] [47], MimicGen can train policies with closed-loop reactive behavior rather than requiring blind, open-loop replay.Data-generation success can also be lower than trained-agent performance, potentially because policies train only on successful attempts or generalize beyond them.Source: §J.1 · p. 27

J.2 Offline Data Augmentation

  • Unlike offline augmentation, MimicGen generates physically consistent interactions through environment interaction, including new objects critical to manipulation.Offline augmentation can have difficulty producing plausible interactions without prior knowledge of physics or causal dependencies, especially in new scenes, objects, or robots.Source: §J.2 · p. 27

K Additional Details on Object-Centric Subtasks

K.1 How Tasks can be broken up into Object-Centric Subtasks

  • For coffee preparation, MimicGen segments human demonstrations into object-relative motions and applies those segments in new scenes; the authors state that many robot-learning tasks admit this decomposition.The sequence includes mug grasping, mug placement, pod grasping, and pod insertion with lid closing; specifying the sequence is assumed to be known and is described as intuitive for a human.Source: §K.1 · p. 28

K.2 Parsing the Source Dataset into Object-Centric Subtask Segments · K.3 Specific Examples

MimicGen parses each source trajectory into contiguous, object-centric subtask segments using detected subtask endings. The examples show how subtask boundaries are checked across manipulation tasks, with those checks used for parsing demonstrations rather than policy execution.

K.3 Specific Examples

  • The examples specify reference objects and subtask-end checks for parsing demonstrations; the assumed subtask structure is generally easy and intuitive for a human to specify.These checks are used only for parsing source demonstrations, not during policy execution.Source: Figure K.2 · p. 29, §K.3 · p. 29

K.2 Parsing the Source Dataset into Object-Centric Subtask Segments

  • Each source trajectory is split into a contiguous sequence of segments, one per subtask, using sequentially detected subtask endings.The subtask ending metrics detect events such as grasping or placement.Source: §K.2 · pp. 28–29
  • Subtask endings can be detected with available or crafted metrics, while manual annotation is a no-privileged-information alternative; this work used metrics.Because source demonstrations are few, a human can annotate endings during collection or afterward; the metrics were readily available or easy to craft.Source: §K.2 · pp. 28–29

K.3 Specific Examples

  • Square has two subtasks: grasp the nut relative to the nut, then insert it relative to the peg; grasp ends at finger-nut contact and insertion at task success.Source: §K.3 · p. 29
  • Threading has two subtasks: grasp the needle relative to the needle, then thread it relative to the tripod; grasp ends at finger-needle contact and threading at task success.Source: §K.3 · p. 29
  • Gear Assembly has two subtasks: grasp the gear relative to the gear, then insert it and turn the crank relative to the base; grasp ends at a lift threshold and insertion at task success.Source: §K.3 · p. 30
  • Stack Three has four subtasks—grasp and place the red block, then grasp and place the blue block—with grasp contact checks and place checks for lifting and contact with the supporting block.Source: §K.3 · p. 30
  • Three Piece Assembly has four subtasks—grasp and insert the first piece, then grasp and insert the second—with finger-contact grasp checks and task-success insertion checks.The motions are relative to the grasped piece or, for insertions, the base or first piece.Source: §K.3 · p. 30

L Tasks and Task Variants … N.3 Reference Segment Selection

The appendices describe a diverse suite of simulation tasks and variants, then detail how MimicGen represents, transforms, interpolates, and selects source subtask segments. The tasks span contact-rich, long-horizon, and mobile-manipulation behaviors, while the data-generation details specify the relevant controller and segment-selection procedures.

L Tasks and Task Variants

  • The simulation suite spans pick-and-place, precise insertion and articulation, mobile manipulation, and long-horizon tasks that chain behaviors; control uses delta-pose actions at 20 Hz.The task variants also broaden scene configurations, object instances, and robot-arm settings.Source: Figure L.1 · p. 31
  • Contact-rich tasks include square-nut insertion, needle threading, coffee-machine operation, multi-piece assembly, and drawer cleanup, with at least one object fixed in each D0 variant.Variants progressively change object positions and, in some tasks, rotations; Coffee and Three Piece Assembly each comprise two objects and four subtasks, while Hammer Cleanup has three subtasks.Source: L Tasks and Task Variants · pp. 31–32
  • Long-horizon tasks include Kitchen’s seven subtasks and Coffee Preparation’s five, while Mobile Kitchen combines pan and carrot setup with mobile-base operation and Factory tasks target high-precision assembly.Kitchen varies the placements of bread, pot, stove, button, and serving region; Factory includes nut-and-bolt, gear, and frame assembly tasks with variants that expand object positions and rotations.Source: L Tasks and Task Variants · pp. 32–33

M Derivation of Subtask Segment Transform

  • The source-segment transform changes controller target poses with the corresponding object pose so that target-to-object relative poses are preserved at each timestep.The derivation begins with a source segment of controller target poses and gives the transformation equation used for that segment.Source: M Derivation of Subtask Segment… · p. 34

N Data Generation Details

  • Appendix N provides further data-generation details, including delta-pose conversion, interpolation, reference-segment selection, execution noise, mobile-manipulation procedures, and task hyperparameters.The following points describe the conversion, interpolation, and reference-selection procedures.Source: N Data Generation Details · p. 35, §N.1 · p. 35

N.1 Equivalence between delta-pose actions and controller target poses

  • The assumed action is 7-dimensional: 3 translation deltas, 3 axis-angle rotation deltas, and a gripper command; the controller maps it to an absolute pose target.MimicGen represents source segments as controller poses and converts transformed poses back to delta-pose actions during execution.Source: §N.1 · p. 35

N.2 Details on Interpolation Segments

  • Each transformed segment begins with an interpolation segment using linear position interpolation and spherical linear rotation interpolation, followed by a fixed-pose hold.The interpolation uses ninterp intermediate poses and holds the transformed start pose for nfixed steps.Source: §N.2 · p. 35

N.3 Reference Segment Selection

  • Reference selection can occur per subtask or use one demonstration throughout an episode; MimicGen usually samples uniformly, but nearest-neighbor selection ranks object-pose distance and samples among the first nnk.Pose distance sums L2 position distance and axis-angle rotation difference; per-subtask selection can be disabled to avoid mixing incompatible strategies across segments.Source: §N.3 · pp. 35–36

N.4 Action Noise … O Policy Training Details

These sections describe how action noise and source-segment transformation affect data generation and policy performance, then specify mobile-task generation, task-specific hyperparameters, and imitation-policy training and evaluation.

N.4 Action Noise

  • Without action noise, data generation success rises (33% on Threading D0), but agent performance drops by up to 30% for low-dim and 40% for image observations.The ablation compares datasets generated without action noise against the default σ = 0.05; noise can make end-effector motion deviate from the expected subtask segment.Source: §N.4 · p. 36

N.6 MimicGen Hyperparameters

  • Removing nearest-neighbor or per-subtask selection significantly lowers data generation rates but does not significantly reduce agent performance for most tasks.The selection-strategy ablation compares these strategies with default random selection.Source: Table N.2 · p. 37, §N.6 · p. 38

N.4 Action Noise

  • On D0, replay with noise raises Square success from 11.3 to 42.0 and Threading from 19.3 to 74.0, while MimicGen reaches 90.7 and 98, respectively.Replay used the 10 source demonstrations with σ = 0.05 until 1000 successful demonstrations were collected, resetting the simulator state before each trial; MimicGen transforms and interpolates source segments for new configurations.Source: §N.4 · pp. 36–37

N.5 Data Generation for Mobile Manipulation Tasks

  • For mobile segments containing base actions, MimicGen copies the base-motion segment between two manipulator segments; future work could adapt base motion to different layouts.Manipulator subsegments are generated as in manipulator-only tasks; the authors suggest using a reference frame for base motion or integrating a base motion planner.Source: §N.5 · p. 37

N.6 MimicGen Hyperparameters

  • Most simulation tasks use σ = 0.05, ninterp = 5, nfixed = 0, and random selection; Square and Nut Assembly use nearest-neighbor selection, while Stack, Stack Three, and Pick Place also select per subtask.The default applies to Threading, Coffee, Three Piece Assembly, Hammer Cleanup, Mug Cleanup, Kitchen, Coffee Preparation, Mobile Kitchen, Nut-and-Bolt Assembly, Gear Assembly, and Frame Assembly; nearest-neighbor uses nnk = 3.Source: §N.6 · pp. 37–38
  • Real-robot tasks use σ = 0.02, ninterp = 25, and nfixed = 25; Stack uses nearest-neighbor per-subtask selection, while Coffee uses random selection.These settings were chosen for safety and timely data collection; Stack uses nnk = 3.Source: §N.6 · p. 38

O Policy Training Details

  • Policies use low-dim observations with ground-truth object poses or image observations from front and wrist cameras, with pixel-shift randomization for image agents.Both observation spaces include end-effector poses and gripper finger positions; images are 84x84 except for real-world Stack and Coffee, which use 120x160 images, and pixel shifts reach 10% of each dimension.Source: O Policy Training Details · p. 39
  • Policies use robomimic BC-RNN defaults, except low-dim training uses a 1e-3 learning rate; simulation evaluation reports the best success rate across 3 seeds, while real-world evaluation uses the last checkpoint.Simulation agents are evaluated for 50 rollouts per checkpoint; real-world agents are evaluated over 50 episodes.Source: O Policy Training Details · p. 39

P Data Generation Success Rates · Q Low-Dim Policy Training Results

Low-dimensional agents trained on MimicGen data perform strongly across reset distributions, and policy success can substantially exceed the success rate of data generation. The results also compare the effects of source and generated dataset sizes and of training on generated versus human demonstrations.

P Data Generation Success Rates

  • Gear Assembly policy success rates exceed data-generation rates: D0 is 92.7% versus 46.9%, D1 is 76.0% versus 8.2%, and D2 is 64.0% versus 7.1%.The paper also reports higher agent performance than data-generation rates in its robot transfer experiment.Source: P Data Generation Success Rates · p. 40, Table P.1 · p. 40
  • Data-generation rate is the success rate of the generation process, which retains only attempts that result in task success.Table P.1 reports this rate for each task with generated data.Source: Table P.1 · p. 40

Q Low-Dim Policy Training Results

  • Low-dimensional agents show large success-rate improvements over 10 source demonstrations across tasks on D0 and remain performant on broader D1 and D2 distributions.The reported results use three seeds and 1,000 demonstrations for each reset distribution.Source: Table Q.1 · p. 41, Q Low-Dim Policy Training Results · p. 41
  • Using larger source human datasets to generate MimicGen data did not yield significant improvement in low-dimensional agent performance.This is the source-dataset-size comparison shown in the left panel of Figure Q.1.Source: Figure Q.1 · p. 41
  • Low-dimensional agents trained on 200 MimicGen demonstrations perform similarly to agents trained on 200 human demonstrations, despite MimicGen using only 10 source human demonstrations.The right panel of Figure Q.1 presents this policy-training dataset comparison.Source: Figure Q.1 · p. 41
  • Generating 200, 1,000, or 5,000 MimicGen demonstrations can improve low-dimensional agents, but the gains can diminish as dataset size increases.Figure Q.1 compares these generated dataset sizes.Source: Figure Q.1 · p. 41

R Bias and Artifacts in Generated Data … U Tolerance to Pose Estimation Error

The sections examine bias and artifacts in generated data, then test how source-demo diversity, generation seeds, and pose-estimation noise affect data generation and learned policies.

R Bias and Artifacts in Generated Data

  • The support-coverage analysis discretized object placements into bins, counted occupied bins, and divided by all bins; Threading D1 used n = 3 bins per dimension across n^6 combinations.The authors note that too few bins obscure configuration-space variation, while too many cannot be meaningfully covered by 1,000 demonstrations.Source: R Bias and Artifacts in Generated Data · p. 42
  • Support coverage ranged from 98.8% (Coffee D1) to 43.5% (Three Piece Assembly D1), and several datasets showed substantial configuration bias.Other examples with substantial bias were Square D2 (66.4%), Threading D1 (71%), Threading D2 (61.2%), Three Piece Assembly D0 (67.9%), and Mug Cleanup D1 (64%). The analysis may miss nonuniform bin counts and other biases, including repetitive motions.Source: R Bias and Artifacts in Generated Data · p. 42
  • Interpolation between transformed source segments can produce long, unnatural paths, and accepting trajectories based only on task completion can retain undesirable collisions.The authors leave data curation for future work.Source: R Bias and Artifacts in Generated Data · p. 42

S Using More Varied Source Demonstrations

  • Using 10 D2 source demonstrations instead of D0 changed data-generation success rates but yielded comparable policy success rates across D0, D1, and D2 reset distributions.This suggests downstream performance can be invariant to how much the source demonstrations’ task initializations vary.Source: Table S.1 · p. 43, S Using More Varied Source… · p. 43

T Data Generation with Multiple Seeds

  • Across three generation seeds, DGR varied by less than 1% and by at most 0.6% from reported rates, while mean low-dim policy SR stayed within 2% of reported results.The study generated three datasets for Stack Three (D0, D1) and Square (D0, D1, D2), with three policy-training seeds per dataset.Source: Table T.1 · p. 44, T Data Generation with Multiple Seeds · p. 44

U Tolerance to Pose Estimation Error

  • Square D0 DGR fell from 73.7% to 60.9% (L1) and 30.5% (L2), while policy SR shifted from 90.7% to 89.3% and 84.7%.For Square D2, DGR fell from 31.8% to 25.1% (L1) and 14.5% (L2), while policy SR shifted from 49.3% to 47.3% and 39.3%. L1 adds 5 mm position and 5 deg rotation noise; L2 adds 10 mm and 10 deg.Source: Table U.1 · p. 45, U Tolerance to Pose Estimation Error · p. 45

Related papers

Loading 2310.17596v1…