Multiple Interactions Made Easy (MIME): Large Scale Demonstrations Data for Imitation

1 Introduction

• Existing robotics datasets largely isolate tasks such as grasping, pushing, poking, or knot-tying, motivating broader task and object diversity.The authors argue diverse data could support visual representations transferable across tasks.Source: §1 · p. 1
• Complex manipulation such as stacking is presented as requiring supervised demonstrations, since random-exploration self-supervision is unlikely to succeed.The passage ends mid-sentence after contrasting demonstrations with self-supervision.Source: §1 · p. 1
• MIME contains 8,260 human-robot demonstrations across more than 20 tasks, spanning simple pushing to complex household-object stacking.The paper frames this as a first step toward scaling demonstration data for complex manipulation.Source: §1 · p. 2
• For each kinesthetic robot demonstration, MIME also records a video of the same human demonstrator performing the task by hand.Kinesthetic demonstrations let people express desired motion while staying within the robot's constraints.Source: §1 · p. 2
• Multiple demonstrators reduce dependence on one person's peculiarities and permit parallel collection, while training and cross-checking address the difficulty newcomers face in kinesthetic operation.The paper also proposes mapping human-video features to robot joint angles, evaluating predictions against the nearest of multiple test trajectories.Source: §1 · p. 2

2.1 Data Driven Robotics

• Prior real-world robotics datasets span grasping, pushing, rope manipulation, surgery, and door opening, but typically focus on single, often simple tasks.The authors contrast this with MIME's roughly 20 tasks and its intended use for imitation learning across tasks.Source: §2.1 · p. 3
• Simulators can supply data faster than real-world collection, but the reality gap makes transfer challenging; MIME instead provides real-world data.The passage cites simulator examples across grasping, driving, and indoor navigation.Source: §2.1 · p. 3

2.2 Learning From Demonstrations (LfD)

• Learning from Demonstrations can support complex policies, but prior methods often use only a handful of expert demonstrations for one task.The passage connects demonstrations with reduced exploration and guidance during policy learning.Source: §2.2 · p. 3
• A prior effort collected hundreds of demonstrations across 15 tasks through virtual-reality teleoperation, but its data was not public and was used task-specifically.The authors note that kinesthetic robot movement keeps demonstrations in suitable configuration spaces for whole-arm manipulation.Source: §2.2 · p. 3
• Large visual datasets lack physical actions, limiting transfer to new tasks; MIME pairs rich visual information with demonstrations across 20 tasks.The authors present this combination as relevant to multi-task learning from demonstrations.Source: §2.2 · p. 3

3 The MIME Dataset

• The dataset section frames task vocabulary, scaling kinesthetic collection, and correcting human errors as its three collection challenges.These challenges organize the account of dataset construction.Source: §3 · p. 3

3.1 Vocabulary of Tasks

• Tasks were selected to be feasible for Baxter, not require haptic feedback, and provide enough diversity for learning task embeddings.Demonstrations also vary objects; the pouring examples vary object shape and opacity, arrangement, and the number of pour targets.Source: §3.1 · p. 4, Figure 3 · p. 5

3.2 Robot Setup

• A Baxter robot in gravity-compensated kinesthetic mode records synchronized RGBD images, joint angles, and gripper positions from a head camera and two wrist cameras.The head camera observes the table externally, while wrist cameras move with the arms.Source: §3.2 · p. 4

3.3 Data Collection Procedure

• Collection uses multiple demonstrators, with training, step-by-step instructions, and participant review; rejected demonstrations are checked and redone.Participants give repeated demonstrations using varied objects, and each task is collected by multiple demonstrators.Source: §3.3 · p. 4
• The collection produced 8,260 paired human and robot demonstrations across 20 discrete tasks, with incorrect demonstrations removed after participant review.The paper reports that newer participants made most errors, with fewer errors and faster trajectories after practice.Source: §3.3 · p. 5

4 Experiments

• The dataset is split 80%/10%/10% across training, validation, and test sets, with multiple test trajectories collected for each test demonstration.The same task-category distribution is maintained across splits, and multiple ground truths support trajectory-distance evaluation.Source: §4 · p. 5

4.1 Task Recognition

• DTW task-recognition accuracy rises from 58.9% with 10% of the data to 79.7% with the full dataset.DTW classifies each test trajectory using the nearest training trajectory in time-series distance.Source: §4.1 · p. 5
• The task-recognition LSTM processes joint angles sequentially and predicts a 20-class task distribution at each timestep.A cross-entropy loss is accumulated across timesteps and minimized with Adam.Source: §4.1 · p. 6
• The joint-angle LSTM achieves 61.2% held-out accuracy, with errors often between tasks having similar or overlapping trajectories.Examples include single-hand picking confused with stacking, and placing in a box confused with picking.Source: §4.1 · p. 6, Figure 4 · p. 6

4.2 Behavioural Cloning

• The output space uses 600 joint-angle clusters, with denser clustering during the manipulation portion of each trajectory.The first 25 timesteps have 150 clusters and the second half has 450, to capture finer motion near objects.Source: §4.2 · p. 7
• Behavioural cloning maps human-video features and the robot’s current object-state image to a predicted robot joint-angle trajectory.The LSTM is initialized from VGG features of the robot image and processes human-video features sequentially, predicting joint clusters.Source: §4.2 · p. 7, Figure 5 · p. 7
• The method addresses trajectory multimodality with classification-based outputs and evaluation against multiple ground-truth trajectories.Using two test ground truths reduced MSE from 0.1296 to 0.1076; the reported metric uses the lower error against either trajectory.Source: §4.2 · p. 8

4.3 Comparing baselines

• Behaviour cloning improves as training data increases, outperforming the average-trajectory and k-NN baselines with a steeper improvement.Figure 6 reports MSE variation with data amount; its error bars show ±1 standard deviation across random splits.Source: Figure 6 · p. 8, §4.3 · p. 8

• MIME contains 8,260 human-object interactions and 8,260 corresponding robot trajectories across 20 diverse tasks, and supports mapping third-person video features to robot trajectories.Source: §5 · p. 8