PaperlayerView exact revision on arXiv

Source-linked AI summary

Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots

Cheng Chi, Zhenjia Xu, Chuer Pan, Eric Cousineau, Benjamin Burchfiel, Siyuan Feng, Russ Tedrake, Shuran Song

arXiv:2402.10329v3cs.RO

TL;DR

Robot manipulation learning must bridge costly teleoperation, embodiment gaps, and limited action information in human demonstrations. UMI addresses these issues with portable hand-held grippers and a hardware-agnostic policy interface. Across challenging tasks, it reports zero-shot transfer and 70% success in out-of-distribution tests.

  • Problem

    Existing approaches face costly teleoperation, human–robot embodiment gaps, and insufficiently precise action recovery for transferable manipulation learning.

    Source: §I · p. 1, §VI · p. 10

  • Method

    UMI combines portable hand-held grippers with Fisheye and mirror-based sensing, inertial tracking, latency matching, relative trajectories, and Diffusion Policy.

    Source: §I · p. 2, A. Demonstration Interface Design · p. 4

  • Results

    70% success was achieved in out-of-distribution tests, with zero-shot generalization to novel environments and objects from diverse human demonstrations.

    Source: §I · p. 2, §VI · p. 10

  • Takeaways & Limitations

    UMI supports direct transfer of dynamic, bimanual, precise, and long-horizon skills from in-the-wild demonstrations to multiple robot platforms.

    Source: §I · p. 2, §IX · p. 11

    UMI data collection remains less efficient than human-hand demonstration because the gripper is heavier, bulkier, and less dexterous.

    Source: §VIII · p. 11

Abstract

from arXiv · show

We present Universal Manipulation Interface (UMI) -- a data collection and policy learning framework that allows direct skill transfer from in-the-wild human demonstrations to deployable robot policies. UMI employs hand-held grippers coupled with careful interface design to enable portable, low-cost, and information-rich data collection for challenging bimanual and dynamic manipulation demonstrations. To facilitate deployable policy learning, UMI incorporates a carefully designed policy interface with inference-time latency matching and a relative-trajectory action representation. The resulting learned policies are hardware-agnostic and deployable across multiple robot platforms. Equipped with these features, UMI framework unlocks new robot manipulation capabilities, allowing zero-shot generalizable dynamic, bimanual, precise, and long-horizon behaviors, by only changing the training data for each task. We demonstrate UMI's versatility and efficacy with comprehensive real-world experiments, where policies learned via UMI zero-shot generalize to novel environments and objects when trained on diverse human demonstrations. UMI's hardware and software system is open-sourced at https://umi-gripper.github.io.

I. INTRODUCTION

UMI addresses the limitations of teleoperated robot datasets and unstructured human videos through a carefully designed demonstration and policy interface. Its portable hardware and policy design support transfer of complex human manipulation skills to robots.

  • Teleoperation has high hardware and operator costs, while human videos exhibit a large human–robot embodiment gap.Source: §I · p. 1
  • Hand-held grippers reduce the embodiment gap but still struggle to balance action diversity with transferability.Source: §I · p. 1
  • Wrist-mounted cameras can provide insufficient visual context because proximity to manipulated objects causes heavy occlusions.Source: §I · p. 1
  • Monocular structure-from-motion often yields imprecise global actions because of scale ambiguity, motion blur, or insufficient texture.Source: §I · pp. 1–2
  • Inference-time latency discrepancies can produce out-of-distribution inputs and out-of-sync actions, especially during fast dynamic manipulation.Source: §I · p. 2
  • UMI combines a wide-field Fisheye camera, side mirrors, latency matching, relative trajectories, and Diffusion Policy to support transferable manipulation learning.Source: §I · p. 2
  • 70% success in out-of-distribution tests is reported for policies trained on diverse human demonstrations.Source: §I · p. 2

II. RELATED WORKS

Prior manipulation-learning approaches rely on costly teleoperation or passive videos with missing action information and embodiment gaps. UMI instead collects action-rich demonstrations without physical robots and transfers policies across environments and robot embodiments.

III. METHOD

UMI is a portable hand-held data-collection and policy-learning framework designed to capture sufficient visual, proprioceptive, and action information for transferable robot skills. Its hardware combines wide-field sensing, implicit stereo, inertial tracking, and continuous gripper control.

Framework and hardware

  • The interface targets portability, capability, sufficient information, and reproducibility across different robot arms.Source: §III · p. 3
  • UMI uses a hand-held, trigger-activated parallel-jaw gripper with soft fingers and a GoPro as its only sensor and recording device.Source: A. Demonstration Interface Design · p. 4

Demonstration interface

B. Policy Interface Design

UMI’s policy interface aligns heterogeneous hardware timing and represents proprioception and actions relative to the current end effector. These choices support hardware-agnostic deployment while addressing latency mismatch, coordinate dependence, and bimanual coordination.

Policy interface

  • UMI policies consume synchronized RGB, relative end-effector pose, and gripper-width sequences and output corresponding action sequences.Source: B. Policy Interface Design · p. 5

Policy learning

Latency matching

  • Observation latency matching measures stream delays, aligns streams to the slowest source, interpolates proprioception, and soft-synchronizes bimanual cameras.Source: B. Policy Interface Design · pp. 5–6
  • Action latency matching sends commands ahead of time so robots reach desired poses at policy-specified times.Source: B. Policy Interface Design · p. 6
  • Outdated predictions caused by observation, inference, and execution latency are discarded before actions with the desired timestamps are executed.Source: B. Policy Interface Design · p. 6

Relative representations

IV. EVALUATIONS

UMI is evaluated for capability, generalization, and data collection efficiency across real-world robotic tasks and environments.

  • The evaluation measures capability in complex, dynamic, bimanual, and long-horizon manipulation skills.Source: §IV · p. 6
  • It tests whether in-the-wild data supports generalization to unseen environments and objects.Source: §IV · p. 6
  • Four real-world robotic tasks are evaluated in narrow-domain and in-the-wild environments.Source: §IV · p. 6
  • Data collection efficiency is assessed by comparing the UMI gripper with human-hand demonstrations and a typical teleoperation interface.Source: §IV · p. 6

V. CAPABILITY EXPERIMENTS

Capability experiments test UMI across single-hand, bimanual, dynamic, and long-horizon skills using controlled initial states and multiple robot embodiments.

Scope

  • UMI evaluates single-hand, bimanual, dynamic, and long-horizon manipulation skills across four tasks.Source: §V · p. 6

Cup Arrangement

  • Cup arrangement combines prehensile and non-prehensile actions with clockwise and counter-clockwise solutions, testing multimodal behavior and relative-depth sensing.Source: A. Cup Arrangement · p. 7, Figure 7 · p. 7
  • 20/20 cup-arrangement trials succeed with UMI, while the same policy reaches 18/20 = 90% on a Franka Emika FR2 robot.Source: A. Cup Arrangement · p. 7

Ablations

  • 11/20 = 55% success follows removal of the Fisheye lens, while the absolute-action baseline achieves only 5/20 = 25%.Source: A. Cup Arrangement · pp. 7–8
  • The delta-action baseline achieves 16/20 = 80%, whereas digitally reflected side-mirror images achieve 20/20 = 100%.Source: A. Cup Arrangement · p. 8

B. Dynamic Tossing

Dynamic tossing evaluates whether UMI transfers rapid, precise human motions, with latency matching supporting robot timing during object release.

Task

  • The task sorts six YCB objects by dynamically tossing them into bins beyond the robot’s kinematic reach.Source: B. Dynamic Tossing · p. 8

Capability

  • Dynamic tossing tests fluid rapid motion transfer, RGB-proprioception hand-eye coordination, and timing alignment between robot and gripper.Source: B. Dynamic Tossing · p. 8

Performance

  • 105/120 = 87.5% success is achieved across mixed multi-object and single-object picking and tossing demonstrations.Source: B. Dynamic Tossing · p. 8

Latency Ablation

  • Disabling inference-time latency matching reduces success to 69/120 = 57.5% and produces jitterier, out-of-sync tossing motions.Source: B. Dynamic Tossing · p. 9

Bimanual Cloth Folding

D. Dish Washing

Dish washing tests ultralong-horizon, multimaterial, deformable, articulated, and semantically responsive manipulation, while in-the-wild data supports broader generalization.

Task and Capability

  • Dish washing requires seven sequentially dependent actions involving water, ketchup, a sponge, two arms, and a faucet.Source: D. Dish Washing · p. 9

Performance

Vision Ablation

  • A ResNet-34 trained from scratch achieves 0/10 = 0% because its policy ignores variation in plate and sponge positions.Source: D. Dish Washing · pp. 9–10

In-the-Wild Data

  • Within 12 person-hours, three demonstrators collected 1,400 cup-arrangement demonstrations across 30 diverse physical locations.Source: §VI · p. 10

In-the-Wild Generalization

  • The UMI policy achieves 28/40 = 70% on training cups, 15/20 = 75% on testing cups, and 43/60 = 71.7% combined.Source: §VI · p. 10
  • Using only narrow-domain data yields 0% success in unseen environments, whereas diverse in-the-wild data is identified as critical for generalization.Source: §VI · p. 10

VII. DATA COLLECTION THROUGHPUT AND ACCURACY

UMI improves real-world demonstration throughput over space-mouse teleoperation while providing accurate SLAM tracking across single-gripper and bimanual tasks. Its remaining limitations concern robot kinematic feasibility, visual texture, and efficiency relative to human hands.

Throughput

  • UMI measures throughput over 15-minute trials while counting resets, object randomization, and robot-fault handling.The comparison includes human-hand demonstration, UMI grippers, and space-mouse teleoperation on cup arrangement and dynamic tossing.Source: §VII · pp. 10–11, Figure 11 · p. 11

Accuracy

  • 6.1 mm position ATE and 3.5° rotation ATE were measured against MoCap ground truth across seven single-gripper and seven bimanual tasks.The benchmark included movable objects, natural motion, and rapid human motion.Source: §VII · p. 11, Figure 12 · p. 11

Limitations

  • Unknown deployment-robot kinematic limits require filtering demonstrations to ensure the learned policy remains kinematically feasible.Future work is proposed to transfer valid but hardware-infeasible actions with embodiment-aware policy learning.Source: §VIII · p. 11
  • SLAM-based action recovery requires sufficient environmental texture, limiting recovery in texture-deficient settings such as rooms with pure white walls.The paper suggests static third-person cameras and fiducial markers as a future remedy.Source: §VIII · p. 11
  • UMI data collection remains less efficient than human-hand demonstration because the gripper is heavier, bulkier, and less dexterous.The paper proposes lighter materials, improved ergonomics, or dexterous robotic hands as future directions.Source: §VIII · p. 11

APPENDIX

The appendix documents latency measurement, low-setup data collection, policy-training hyperparameters, and the practical procedures used to collect UMI demonstrations.

Latency Measurement

  • Camera latency is measured by recording a rolling QR code containing display timestamps and subtracting display-refresh latency from the receive-time difference.The procedure uses a wrist-mounted GoPro and external capture hardware.Source: 1) Camera Latency Measurement · p. 15
  • Proprioception latency is measured from robot and policy timestamps when available, or approximated as half the ICMP round-trip time otherwise.The timestamp method applies to Franka FR2, while UR5 and WSG-50 use the ping-based approximation.Source: 2) Proprioception Latency Measurement · p. 15
  • Gripper execution latency is computed as end-to-end latency minus observation latency, with end-to-end delay estimated by aligning commanded and measured sinusoidal width signals.The alignment uses crossconvolution.Source: 3) Gripper Execution Latency Measurement · p. 15

Data Collection

  • UMI data collection starts with optional time synchronization and gripper calibration, followed by scene mapping and repeated video-recorded demonstrations.The setup is described as close to zero time in a new environment.Source: B. Data Collection Protocol · pp. 15–16

Policy Implementation

  • The appendix provides Diffusion Policy hyperparameters and implementation details for image observations, proprioception, action horizons, frequency, speed, and vision architecture.These settings are organized in Table A1.Source: Table A1 · p. 16

C. Evaluation Protocol

The evaluation protocol defines task-specific success criteria, diverse initial-state selection, supervised termination rules, and manual success judgments for complex manipulation tasks.

Evaluation Procedure

  • Evaluation initial states are manually selected for diverse pose coverage, shared across methods, matched with a third-person camera, and checked for kinematic feasibility.Narrow-domain object arrangements are also randomized with approximately the same distribution.Source: 1) Initial State Selection · p. 16

Termination Criteria

  • Episodes terminate for safety concerns, robot faults, conservative timeouts, or operator-judged task success.Faults include collisions, excessive force, and joint or speed-limit violations.Source: 2) Termination Criteria · p. 16

Success Criteria

  • Success and failure are manually judged with task-specific rubrics because compact automatic metrics are difficult for these complex manipulation tasks.The rubric retains subjective elements, so the authors recommend viewing the accompanying video.Source: 3) Success Criteria · p. 16

Task Criteria

  • Cup arrangement succeeds when the upright cup’s handle lies within ±15° to the robot’s left.The in-the-wild variant defines left relative to the gripper’s initial pose.Source: 3) Success Criteria · pp. 16–17
  • Dynamic tossing succeeds when objects reach their corresponding bins, including cases that bounce off a sidewall but later hit the bin bottom.A bounce that never reaches the bin bottom is not counted as success.Source: 3) Success Criteria · p. 16
  • Bimanual cloth folding succeeds when both sleeves are inside and the folded sweater’s projected silhouette is approximately square and one-quarter of the torso area.Non-flat sleeves and cloth still count if the stated silhouette condition is met.Source: 3) Success Criteria · p. 16

D. SLAM System Details

UMI adapts SLAM and policy implementation to improve robustness under fast motion, dynamic scene changes, difficult visual features, and task-specific execution demands.

SLAM Initialization

  • The original ORB-SLAM3 initialization was fragile and lengthy during fast movements, often producing no camera pose and wasting demonstrations.UMI modifies the system to address this initialization problem.Source: D. SLAM System Details · p. 17

Map Initialization

  • UMI continues normal SLAM operation after relocalizing to a disk-loaded map because manipulation dynamically changes the scene.This differs from ORB-SLAM3’s localization mode, which does not update the existing map.Source: D. SLAM System Details · p. 17

Marker-Enhanced Initialization

  • Optional fiducial markers disambiguate monocular-SLAM feature matches when features are distant or repeated, increasing mapping robustness in-the-wild.The stated difficult cases include outdoor environments, repeated patterns, and trees.Source: D. SLAM System Details · p. 17

Policy Implementation

  • Diffusion Policy uses a pretrained CLIP ViT vision encoder with reduced fine-tuning learning rate for tasks requiring intricate perception.The appendix contrasts ViT’s capacity with ResNet and notes that per-task datasets are too small to train ViT from scratch.Source: E. Policy Implementation Details · p. 17, 1) Vision encoder · p. 17
  • Most tasks run at 10 Hz, while dynamic tossing uses 20 Hz because it requires highly reactive behavior.Execution speed can differ from demonstration speed, but tossing requires preserving the original speed for sufficient release velocity.Source: 2) Frequency · p. 17, 3) Speed · p. 17

Hardware Implementation

  • Soft TPU fingers conform to object geometry and provide passive compliance for contact-rich tasks on robots without force-torque control.The same soft fingers are used on UMI collection grippers and deployed robotic grippers.Source: 1) Soft Compliant Fingers · p. 17, Figure A3 · p. 18

Related papers

Loading 2402.10329v3…