PaperlayerView exact revision on arXiv

Source-linked AI summary

MimicDroid: In-Context Learning for Humanoid Robot Manipulation from Human Play Videos

Rutav Shah, Shuijing Liu, Qi Wang, Zhenyu Jiang, Sateesh Kumar, Mingyo Seo, Roberto Martín-Martín, Yuke Zhu

arXiv:2509.09769v1cs.RO

TL;DR

MimicDroid targets few-shot humanoid manipulation without relying on costly teleoperated training data by learning in-context adaptation from human play videos. It constructs similar trajectory pairs for Meta-ICL, bridges human-robot embodiment differences through pose retargeting and visual masking, and outperforms comparison methods in simulation and the real world.

  • Problem

    Few-shot humanoid manipulation needs scalable, diverse training data because existing in-context learning methods rely on costly teleoperated demonstrations.

  • Method

    MimicDroid meta-trains on human play videos by conditioning target-action prediction on similar trajectory segments, retargeting human wrist poses, and applying random patch masking.

  • Results

    MimicDroid achieves nearly twofold higher real-world success rates than Vid2Robot and exceeds PEFT by 26% at test time.

  • Takeaways & Limitations

    Human play videos can support few-shot humanoid manipulation with instantaneous in-context adaptation to novel objects and environments.

  • Takeaways & Limitations

    The method fails when hands vanish from view and cannot generalize across semantically equivalent tasks because it learns low-level state-action sequences rather than task meaning.

Abstract

from arXiv · show

We aim to enable humanoid robots to efficiently solve new manipulation tasks from a few video examples. In-context learning (ICL) is a promising framework for achieving this goal due to its test-time data efficiency and rapid adaptability. However, current ICL methods rely on labor-intensive teleoperated data for training, which restricts scalability. We propose using human play videos -- continuous, unlabeled videos of people interacting freely with their environment -- as a scalable and diverse training data source. We introduce MimicDroid, which enables humanoids to perform ICL using human play videos as the only training data. MimicDroid extracts trajectory pairs with similar manipulation behaviors and trains the policy to predict the actions of one trajectory conditioned on the other. Through this process, the model acquired ICL capabilities for adapting to novel objects and environments at test time. To bridge the embodiment gap, MimicDroid first retargets human wrist poses estimated from RGB videos to the humanoid, leveraging kinematic similarity. It also applies random patch masking during training to reduce overfitting to human-specific cues and improve robustness to visual differences. To evaluate few-shot learning for humanoids, we introduce an open-source simulation benchmark with increasing levels of generalization difficulty. MimicDroid outperformed state-of-the-art methods and achieved nearly twofold higher success rates in the real world. Additional materials can be found on: ut-austin-rpl.github.io/MimicDroid

I. INTRODUCTION

MimicDroid addresses few-shot humanoid manipulation by replacing costly teleoperated training data with scalable human play videos for in-context learning. It enables adaptation to novel objects and environments and reports stronger simulation and real-world performance than prior methods.

  • Motivation and approach: Human play videos are approximately 18× faster to collect than teleoperated demonstrations and cover diverse tasks, objects, configurations, and manipulation behaviors.
  • Motivation and approach: MimicDroid constructs context-target trajectory pairs from recurring manipulation behaviors so a policy can learn observation-action relationships for test-time adaptation.The approach addresses the need for scalable self-supervised Meta-ICL samples from unlabeled videos.
  • Motivation and approach: MimicDroid uses human play videos as the only training data source for few-shot humanoid manipulation via in-context learning.The method leverages continuous, unlabeled human interactions rather than teleoperated robot demonstrations.
  • Contributions and results: MimicDroid outperforms task-conditioned baselines and achieves a twofold real-world success-rate improvement, while exceeding parameter-efficient finetuning by 26% at test time.It also improves by 20% when training data increases from 64k to 320k human-play frames.
  • Contributions and results: The paper introduces an open-source humanoid few-shot-learning simulation benchmark with three increasing generalization levels and 8 hours of play data.
  • Motivation and approach: The method bridges the embodiment gap through hand-pose-based human-to-humanoid transfer and visual masking that reduces reliance on human-specific appearance cues.

III. METHODOLOGY

MimicDroid formulates humanoid few-shot manipulation as test-time in-context learning from a small set of human demonstrations. It meta-trains a visuomotor policy on an implicit task distribution induced by continuous human play, using context trajectories to supervise target-action prediction.

  • Problem setup and evaluation: At test time, the policy receives a small set of human demonstration trajectories for a novel task and adapts without parameter updates.Evaluation spans three task distributions with increasing difficulty from novel objects and environments.
  • Training data: MimicDroid trains on continuous human-play trajectories, whose diverse interactions provide a richer and more scalable task variation source than manually defined tasks.
  • Policy learning: Meta-ICL training pairs context trajectories with a target trajectory, then uses behavior cloning to predict the target actions conditioned on the context.The policy learns recurring observation-action patterns from the context trajectories.
  • Problem setup and evaluation: Unlike traditional meta-learning, MimicDroid uses an implicit task distribution induced by natural human interactions rather than an explicitly defined discrete task set.
  • Challenges: Raw human videos provide RGB frames but lack actions, proprioception, labels, and ready-made context-target pairs, requiring these training signals to be estimated or constructed.

B. Constructing Training Samples

MimicDroid creates Meta-ICL supervision from unlabeled human play by pairing a target segment with behaviorally similar context segments. Similarity combines pooled visual features with action sequences, allowing recurring observation-action patterns to guide retrieval.

  • Training signal: Fig. 3 illustrates target segments alongside their retrieved context examples, making the quality of behavior-based retrieval the central visual comparison.
  • Constructing context-target pairs: The method randomly samples a target trajectory segment and retrieves the top-k most similar segments from the dataset as context.
  • Training signal: The retrieved context segments and target segment provide self-supervised Meta-ICL training samples from repetitive manipulation behaviors in human play data.
  • Similarity computation: Segment similarity is computed as cosine similarity between feature embeddings that combine temporally mean-pooled visual features with action sequences.Pretrained visual features are extracted frame by frame before temporal pooling and action concatenation.

C. Overcoming the Embodiment Gap

MimicDroid bridges human-to-humanoid transfer by retargeting human wrist poses in task space and using inverse kinematics to obtain robot joint angles. Visual masking addresses appearance and occlusion differences that can disrupt in-context learning.

  • Kinematic gap: Human wrist poses are retargeted to humanoid wrist poses and converted into joint angles through inverse kinematics, preserving task intent through kinematic similarity.Task-space transfer avoids requiring demonstrators to mimic the robot’s morphology.
  • Visual gap: Random visual masking reduces overfitting to human appearance and improves robustness to visual differences between human videos and humanoid observations.
  • Visual gap: The method addresses both kinematic and visual embodiment gaps because either mismatch can hinder transfer of learned in-context capabilities at test time.

D. Meta-training for In-Context Learning

MIMICDROID meta-trains on trajectory segments sampled from continuous human play, pairing similar context segments with a target segment whose action predictions are supervised by behavior cloning.

  • D. Meta-training for In-Context Learning: Human play induces an implicit task distribution, allowing meta-training across diverse manipulation behaviors without a manually defined discrete task set.Training instances are formed by uniformly sampling from continuous human-play trajectories.
  • D. Meta-training for In-Context Learning: For each target trajectory segment, the method retrieves the top k most similar trajectory segments as context examples.The context segments provide examples of task execution, while the target segment supplies the supervised action predictions.
  • D. Meta-training for In-Context Learning: The benchmark data contains 8 hours of play collected with free-floating hands across 30 objects and 8 kitchen environments.These data support evaluation across L1, L2, and L3 generalization levels for humanoid manipulation.
  • D. Meta-training for In-Context Learning: The policy predicts l-step action chunks conditioned on the past trajectory and k context examples, using an L1 behavior-cloning loss against target actions.Action chunking models the multimodal nature of human play by predicting a sequence of future actions rather than only the next action.

IV. EXPERIMENT SETUP

The experiments evaluate few-shot humanoid manipulation using a benchmark with progressively harder generalization levels, two simulated embodiments, human-play data collection, and task-conditioned and fine-tuning baselines.

  • IV. EXPERIMENT SETUP: The benchmark progresses from seen objects and environments to unseen objects and entirely new kitchen scenarios, testing increasingly difficult few-shot generalization.L1 tests new object positions, L2 tests novel objects in seen environments, and L3 tests novel objects, layouts, backgrounds, and motion sequences.
  • IV. EXPERIMENT SETUP: Experiments compare a free-floating 6-DoF hand with the GR1 humanoid, enabling training-data collection on the former and embodiment-gap evaluation on the latter.Real-world evaluation uses human play videos and the GR1 humanoid.
  • IV. EXPERIMENT SETUP: Training data are collected through free-form operator interactions driven by curiosity, capturing diverse tasks and object configurations without specific goals.This setup is intended to provide richer interaction diversity than typical teleoperated demonstrations.
  • IV. EXPERIMENT SETUP: The implementation retrieves k = 10 similar trajectories, masks image patches with p = 0.8, and predicts l = 32 actions per timestep.Trajectory features use WiLoR for hand-pose estimation and DinoV2 for visual features.
  • IV. EXPERIMENT SETUP: Table I reports success rates for the Abstract and GR1 embodiments in the simulation benchmark.The supplied table caption identifies the compared embodiments and metric but does not provide cell values.
  • IV. EXPERIMENT SETUP: Baselines include video-conditioned Vid2Robot, goal-image-conditioned H2R, and parameter-efficient fine-tuning, separating ICL from task conditioning and test-time adaptation.The task-conditioned baselines lack the observation-action context used for in-context learning.

V. RESULTS

MIMICDROID improves few-shot humanoid manipulation across simulation and real-world evaluations, while its performance depends on embodiment robustness, context selection, and training-data scale. It achieves higher success than task-conditioned and finetuning baselines, but remains challenged by the hardest generalization level and novel robot motions.

  • MIMICDROID outperforms task-conditioned baselines by +14% and +18% and exceeds PEFT by +29% and +26% across abstract and humanoid embodiments, respectively.It adapts instantly through ICL, while PEFT fails at L3 and shows greater transfer degradation.
  • Real-world results: 0.53, 0.23, and 0.08 success rates in real-world L1-L3 nearly double Vid2Robot’s 0.28, 0.08, and 0.00, respectively.The evaluation uses a GR1 humanoid and a policy trained solely on human play videos.
  • Visual-gap ablation: Removing visual masking causes a −17% humanoid transfer drop versus −3% for MIMICDROID, while random masking reaches 0.53 compared with EgoMimic’s 0.58.Random patch masking improves robustness to the embodiment visual gap without requiring external segmentation modules.
  • Context scaling: Performance improves from 1 to 3 context examples but gains drop from 4 to 6 because the training-time transformer context length is limited.Extending context length increases memory and compute costs linearly with sequence length.
  • Meta-ICL data quality: Retrieving too few similar segments limits diversity, whereas large k introduces dissimilar examples; k = 50 balances diversity and noise.This result emphasizes the importance of meaningful context-target similarity in Meta-ICL data construction.
  • Data scaling: 35% to 59% in L1 and 21% to 45% in L2 produce +24% gains as training data grows from 128k to 320k frames; L3 rises 16% to 27% (+11%).The improvements across L1-L3 demonstrate scalability with RGB play-video data, though gains are smaller at L3.
  • Failure analysis: Task misidentification accounts for 26% of failures and missed grasps for 16%, while MIMICDROID reduces these errors versus Vid2Robot by −15% and −5%.Other errors, including incomplete cabinet closure and missed placement, account for 8%.
  • Limitations: MIMICDROID still struggles at L3, where tasks may require novel robot motions, and can overfit to hand sizes, causing collisions in cluttered real-world environments.The authors hypothesize that data from varied operators could mitigate the hand-size issue.

VI. LIMITATIONS AND FUTURE WORK

The paper identifies limitations in data coverage, hand-pose extraction, and semantic task understanding, while proposing extensions to address each boundary.

  • Human play videos provide high-quality in-context learning data, but web-scale human videos could expand object and environment diversity.
  • The off-the-shelf hand pose predictor fails when hands disappear behind furniture or inside cupboards, motivating combined hand and full-body motion estimation.
  • MIMICDROID learns low-level state-action sequences rather than why motions matter, limiting generalization across semantically equivalent tasks.
  • Language-trajectory pairs are proposed as a future direction for enabling semantic task understanding, while full-body motion estimation could improve pose extraction.

APPENDIX

The appendix describes how human play data becomes structured trajectories and how a long-context multimodal transformer predicts action chunks during in-context learning.

  • Data Collection: Real-world play sessions use a single static RGB camera for approximately 10 minutes, while simulated sessions use two teleoperated hands, 20-minute sessions, and three cameras.
  • Data Preprocessing: Actions are represented as the next-step 6-DoF wrist pose plus a binary gripper signal indicating open or closed.
  • Policy Architecture: The policy uses three context trajectories and one target trajectory processed by a long-context transformer with modality-specific encoders.
  • Policy Architecture: Visual and proprioceptive features are fused through self-attention so the policy grounds proprioceptive state in visual context.
  • Model Components: The model is built on a 4-layer LLaMA-style transformer and uses CrossMAE visual features with a 768-dimensional embedding space.
  • Policy Architecture: The action decoder predicts 32 future actions, while rollouts execute only the first 16 before replanning.

D. Training Details

Training and evaluation use human-play-derived data on the GR1 humanoid, with three increasingly difficult generalization levels spanning seen and unseen objects and environments.

  • Evaluation Setup: Models are trained on human play videos and evaluated on GR1 using 10 robot rollouts per task.
  • Generalization Levels: Evaluation progresses from seen objects and environments at L1 to unseen objects in seen environments at L2 and unseen objects and environments at L3.
  • L1: Seen Objects, Seen Environment: L1 includes four pick-and-place or oven-tray tasks using training objects and environments with changed object placements.
  • L2: Unseen Objects, Seen Environment: L2 introduces novel object categories or instances while retaining the training environment across four manipulation tasks.
  • L3: Unseen Objects, Unseen Environment: L3 combines novel objects with a new environment and evaluates four tasks, including drawer-to-plate and oven-tray manipulation.

F. Simulation Benchmarking

The simulation benchmark is designed to measure few-shot generalization across three levels, from changed placements to novel objects and environments, with four tasks per level.

  • Benchmark Design: The benchmark uses RoboCasa with 8 training environments and 30 training objects, and evaluates policies on both free-floating-hand and GR1 embodiments.
  • Benchmark Design: Few-shot prompts are drawn from held-out validation play data, separating prompt examples from the training data.
  • Evaluation Structure: Each of the three generalization levels contains four distinct tasks, as summarized in Table V.
  • Generalization Levels: L1 changes object positions, L2 introduces novel objects, and L3 combines novel objects with a new environment and potentially novel robot motions.
Loading 2509.09769v1…