PaperlayerView exact revision on arXiv

Source-linked AI summary

UMI-Live for In-The-Wild Bimanual Robotics Data Collection

Govind Velamoor, Adrian Lin, Sam Jung, Spencer Mateega, Brandon Y. Yang, Amish Sethi

arXiv:2610.39385v1cs.ROcs.AI

TL;DR

Robot learning depends on expert demonstrations, but collecting them across environments can require robot infrastructure, and handheld demonstrations still need consistent hardware and data processing. UMI-Live combines reproducible bimanual handheld devices with automated data preparation and robot-learning software. On an orange-to-bowl task, fine-tuned policies succeeded in physical trials, while the system also releases demonstrations across three tasks.

  • Problem

    Robot learning depends on expert demonstrations covering relevant objects and interactions, while robot-based collection requires robot, control, perception, and safety infrastructure at collection sites.

    Source: §1 · p. 2

  • Method

    UMI-Live combines reproducible bimanual handheld grippers with integrated sensing and compute, asynchronous data preparation, and dual-YAM policy training and inference.

    Source: §1 · p. 3, §7 · p. 9

  • Results

    π0.5 succeeds in 10/10 trials and MolmoAct2 in 8/10 after fine-tuning on mostly UMI-Live demonstrations; the evaluation covers one task and one training seed per condition.

    Source: §4.3 · pp. 8–9

  • Takeaways & Limitations

    The release combines approximately 40 hours of demonstrations across three tasks with hardware, software, and physical robot evaluation spanning handheld demonstration to robot learning.

    Source: §7 · p. 9

    The robot evaluation covers one task and one training seed per condition, uses mixed UMI and teleoperation data, and does not isolate either source’s contribution.

    Source: §6 · p. 9

Abstract

from arXiv · show

Universal Manipulation Interface (UMI) devices allow human experts to demonstrate tasks for robot learning without teleoperating physical robots. Using these demonstrations as training data still requires a consistent hardware setup, calibration, and data processing. We present UMI-Live, a full-stack open-source bimanual UMI system for in-the-wild data collection. We design our custom UMI-Live hardware to integrate camera and LiDAR--inertial sensing, onboard compute, and power, without external tracking equipment or calibration within the collection environment. We implement a simple user interface for recording data while an asynchronous pipeline performs parallel SLAM reconstruction and publishes training-ready datasets. We also release software that supports dual-YAM teleoperation and VLA policy training and inference. We validate UMI-Live data quality by fine-tuning $\pi_{0.5}$ and MolmoAct2, achieving success rates of 10/10 and 8/10, respectively on a bimanual pick-and-place task. Furthermore, we release 40 hours of UMI-Live data across three tasks, and complete hardware designs, software, and reproduction guides at our $\href{https://umi-live.github.io/}{\text{project website}}$.

1 Introduction

UMI-Live addresses the equipment and processing burdens of robot-based and handheld demonstration collection with a reproducible bimanual system. It combines integrated handheld hardware and asynchronous data preparation with robot-learning software, a public dataset, and physical policy evaluation.

1 Introduction

  • Robot learning still depends on expert demonstrations that cover relevant objects and interactions across diverse environments.Robot-based collection requires transporting or maintaining robot, control, perception, and safety infrastructure; handheld UMI demonstrations can avoid robot operation during capture.Source: §1 · p. 2
  • Existing UMI systems can require difficult-to-reproduce hardware calibration, processing software, stationary tracking rigs, or backpack computers.Bulky grippers may also hinder dexterous manipulation of small objects.Source: §1 · p. 2
  • UMI-Live combines self-contained handheld bimanual grippers, automated data preparation, dual-YAM teleoperation, and policy training and inference.Operators can continue recording while completed demonstrations are transferred, reconstructed in parallel, validated, and prepared as datasets.Source: §1 · pp. 2–3
  • The system includes hardware designs and construction and calibration guides for reproducing handheld grippers with integrated sensing, compute, storage, power, and operator controls.The grippers are designed for in-the-wild collection.Source: §1 · p. 3
  • The software integrates coordinated recording and asynchronous processing with LeRobot dataset preparation, dual-YAM teleoperation, and policy training and inference.Operators can collect demonstrations while completed ones are processed.Source: §1 · p. 3
  • π0.5 succeeds in up to 10/10 trials and MolmoAct2 in 8/10 after mixed-data fine-tuning; MolmoAct2’s released checkpoint succeeds in 0/10.The released dataset contains approximately 40 hours of demonstrations across three tasks.Source: §1 · p. 3

2 Related Work

UMI-Live is positioned as a bimanual collection system that links handheld demonstration capture to processing and robot learning. Related work spans robot-based collection, handheld UMI hardware and sensing designs, and collection-to-training software.

2 Related Work

  • Robot-collection datasets cover varied tasks, environments, and embodiments but require a robot for capture; UMI-Live supports handheld collection plus separate target-embodiment teleoperation.The teleoperation configuration can provide complementary demonstrations on the target embodiment.Source: §2 · p. 3
  • Handheld UMI systems vary in grippers, sensing, tracking, and pipelines; UMI-Live uses two independently instrumented LiDAR–inertial devices for coordinated bimanual collection.The cited systems include the original UMI [12], FastUMI, and UMI-3D [19].Source: §2 · p. 3
  • Other systems emphasize finger-aligned dexterity, active egocentric perception, or bimanual trajectory fidelity, while UMI-Live integrates sensing, compute, storage, power, and controls on handheld devices.The passage describes YUBI, ActiveUMI [14], and HiFi-UMI.Source: §2 · p. 3
  • LeRobot [21], RoboPocket, and ABC [11] provide dataset, policy, or collection-to-evaluation infrastructure; UMI-Live combines bimanual recording, asynchronous reconstruction, dataset publication, and dual-YAM learning tools.Its workflow includes policy training and inference.Source: §2 · p. 3

3 System Design

UMI-Live connects self-contained handheld bimanual collection to automated dataset preparation and physical robot deployment. Its hardware and software support recording, processing, and policy use without requiring operators to collect in the robot evaluation environment.

3.1 Hardware Design

  • Each gripper combines onboard sensing, compute, storage, and removable power, with a single-button interface for recording in the wild.The system can be used without a robot, tripod-mounted cameras, external tracking equipment, or environment-specific calibration.Source: §3.1 · pp. 4–5
  • Bearing-supported trigger and gear pivots drive opposing parallel-jaw motion, while redesigned fingers pair a rigid PLA+ base with a compliant TPU contact layer.The rigid base supports force application and precision contact with small objects; the TPU layer conforms to varied object shapes.Source: §3.1 · pp. 4–5

3.2 Operator Interface and Recording

  • Completed recordings transfer from the grippers to a server asynchronously, allowing operators to continue collecting demonstrations.The LCD displays recording status and transfer progress.Source: §3.2 · p. 5

3.3 Automated Data Processing

  • The server checks synchronization, estimates TCP trajectories and gripper openings, aligns the streams, and assembles wrist images with measurements into training episodes.Timestamp correction maps camera and LiDAR timing to a shared timeline; pose and gripper values are interpolated for dataset assembly.Source: §3.3 · pp. 5–6
  • Processing runs concurrently across hands and episodes, while dataset assembly uses a serialized writer and finalized episodes are validated and stored in LeRobot format.Completed stages are tracked so interrupted processing can resume from available outputs.Source: §3.3 · p. 7

3.4 Policy Training and Deployment

  • The policies use two wrist images, gripper openings, and a task instruction to predict action chunks; a dual-YAM client converts returned chunks into coordinated arm commands.The shared representation includes query-relative TCP poses and absolute gripper openings.Source: §3.4 · p. 7

4 Experimental Results

The study evaluates bimanual orange-to-bowl execution on dual YAM arms using policies fine-tuned on mixtures of handheld UMI and robot teleoperation data. Both policies achieved successful physical trials after fine-tuning, with the evaluation limited to one task and one training seed per condition.

4 Experimental Results

  • The evaluation tests whether UMI-Live demonstrations support physical execution, comparing fine-tuned MolmoAct2 with its released checkpoint on the orange-to-bowl task.The study covers one task and one training seed per condition.Source: §4 · p. 7, §4.3 · p. 9

4.1 Experimental Setup

  • The task required placing six oranges in a bowl; trials used randomized object positions, allowed grasp retries and dropped-orange recovery, and ended after ten trials per condition.UMI demonstrations came from a different location and lighting, while robot teleoperation demonstrations came from the evaluation scene.Source: §4.1 · p. 7

4.2 Training Conditions

  • Training used 296 UMI and 75 robot teleoperation demonstrations for this task, with UMI/teleoperation shares of 95%/5% and 90%/10% for π0.5 and 75%/25% for MolmoAct2.Each condition used one training seed; π0.5 parameters were updated, while MolmoAct2 used LoRA adapters and a full action expert with remaining backbone parameters frozen.Source: §4.2 · p. 8

4.3 Results

  • 10/10 trials for π0.5 with the 90/10 mixture, 9/10 with the 95/5 mixture, and 8/10 for fine-tuned MolmoAct2 versus 0/10 for its released checkpoint.Success required placing all six oranges in the bowl.Source: §4.3 · p. 8, Table 2 · p. 9

5 Released Dataset

The released dataset contains bimanual demonstrations across three manipulation tasks, with synchronized visual, state, and instruction data. The release also specifies episode-start referencing and source provenance.

5 Released Dataset

  • 3,255 bimanual episodes comprise approximately 39.7 hours at 30 Hz across orange placement, water pouring, and towel folding.Episodes include synchronized wrist videos, bimanual pose and gripper states, and task instructions.Source: §5 · p. 9
  • Each hand uses a fixed episode-start reference, and the release provides source provenance; training loaders derive query-relative targets from the same-frame state.The standardized action field stores the same-frame state, while loaders select future frames.Source: §5 · p. 9

6 Discussion and Limitations

The discussion identifies constraints in reconstruction, device mass, and evaluation scope, and calls for broader measurement of system and policy performance.

6 Discussion and Limitations

  • LiDAR–inertial reconstruction depends on scene geometry and motion, each unit weighs 1.42 kg, and policy evaluation covers one task and one seed per condition.Because the study lacks UMI-only and teleoperation-only comparisons, it does not isolate either data source’s contribution.Source: §6 · p. 9

7 Conclusion

UMI-Live combines handheld bimanual data collection with processing and dual-YAM software to connect demonstrations to physical robot learning. The authors release a three-task dataset and report successful physical experiments after policy fine-tuning.

7 Conclusion

  • UMI-Live combines self-contained handheld hardware, automated data preparation, and dual-YAM software integration for bimanual collection and robot learning.Operators control recording through one interface while completed episodes undergo parallel processing.Source: §7 · p. 9
  • The resulting demonstrations connect policy training and physical deployment through a common task-space representation.Source: §7 · p. 9
  • Approximately 40 hours of demonstrations across three tasks were released, and physical experiments showed successful orange-to-bowl execution after fine-tuning π0.5 and MolmoAct2 on mostly UMI-Live mixtures.The authors describe the hardware, software, dataset, and robot evaluation as a reproducible system spanning handheld demonstration to physical robot learning.Source: §7 · p. 9

Use of Generative AI

This material documents generative-AI use, hardware components, human-operated manipulation examples, an autonomous rollout, and the teleoperation configuration.

Use of Generative AI

  • The authors used LLMs for coding implementation and as a writing and editing aid, while taking responsibility for the manuscript.Source: Use of Generative AI · p. 10

A Hardware Details

  • The gripper assembly combines LiDAR, camera, Raspberry Pi, LCD, printed parts, a linear guide, gears, fasteners, and battery and power electronics.The mechanism includes sector gears, bearing-supported pivots, a trigger and return spring, and a removable power module.Source: Figure 8 · p. 13, Figure 9 · p. 13

B UMI-Live Manipulation Demonstrations

B.1 Small-Object Grasping and Insertion

  • In the small-object sequences, the operator picks up a tool, hands it between grippers, inserts it into a case, and releases it.The examples use screwdriver bits and show six chronological frames per row.Source: Figure 10 · p. 14, §B.1 · p. 14

B.2 Match Lighting

  • The match-lighting sequence shows coordinated grippers opening a matchbox, retrieving a match, and striking it.The final frame shows the lit match.Source: Figure 11 · p. 14, §B.2 · p. 14

B.3 Shirt Folding

  • The shirt-folding sequence shows both grippers spreading and aligning the shirt before folding it lengthwise and completing the folds.Source: Figure 12 · p. 15, §B.3 · p. 15

C Autonomous Rollout

D Teleoperation Setup

  • Teleoperation replaces each gripper’s LiDAR with a VIVE tracker and uses Lighthouse base stations for external tracking.Tracker motion is mapped to Cartesian arm targets, while wrist-camera ArUco tags measure jaw opening.Source: D Teleoperation Setup · p. 16, Figure 14 · p. 17

E Dataset Representation

UMI demonstrations are represented as episodes with task instructions, wrist images, and bimanual TCP states relative to each gripper’s initial pose. Policy training uses current observations to supervise future bimanual action chunks, with teleoperation data converted into the same representation.

E Dataset Representation

  • Each demonstration is an episode with a task instruction, wrist images, and gripper states referenced to each gripper’s initial TCP pose.Figure 15 displays wrist views and reconstructed TCP position traces during folding.Source: E Dataset Representation · p. 17, Figure 15 · p. 18

F Policy Training

  • The authors fine-tune MolmoAct2 and π0.5 to predict bimanual TCP trajectories, extending the UMI representation and losses to mixtures with robot teleoperation data.Source: F Policy Training · p. 18

F.1 Training on UMI Demonstrations

  • Training samples a current trajectory entry as the observation and uses the next H states to supervise a bimanual action chunk with flow matching.TCP pose and joint information are omitted from policy inputs; ultra-wide wrist cameras provide broad workspace visual context for coordination.Source: §F.1 · pp. 18–19

F.2 Training with Teleoperation Data

  • Robot teleoperation episodes are converted to the same observation and action representation as UMI demonstrations, then combined for fine-tuning.The mixture shares assigned to UMI and robot teleoperation data sum to 100%.Source: §F.2 · p. 19

F.3 Training Setup

The paper specifies fresh-base fine-tuning settings for π0.5 and MolmoAct2, then describes chunked policy inference and dual-arm execution on YAM hardware.

F.3 Training Setup

  • Both models are fine-tuned for 12,000 updates with AdamW, 600 warmup updates, and cosine learning-rate decay; π0.5 updates all parameters while MolmoAct2 uses LoRA adapters and a full action expert.The action horizon is H = 24 at 30 Hz; MolmoAct2 combines flow matching with discrete action-token supervision.Source: §F.3 · p. 19

G Policy Inference

  • At inference, the policy integrates a learned flow field from Gaussian noise into a bimanual action chunk, then maps relative TCP predictions to robot targets.The controller converts targets to joint commands through inverse kinematics and executes actions while the server processes subsequent observations.Source: G Policy Inference · pp. 20–21

G.1 Dual-YAM Hardware Integration

  • Dual-YAM integration runs remote policy inference with a local robot client and separate arm workers that schedule left and right commands together.Inverse-kinematics solutions are checked against pose-error tolerances, joint limits, and a bound on joint displacement.Source: §G.1 · p. 21

G.2 Kinematic Model

UMI-Live describes the YAM arm with a classical Denavit–Hartenberg model and adapts its TCP for the replacement gripper. The TCP transform is incorporated into the I2RT model for forward and inverse kinematics during execution.

G.2 Kinematic Model

  • The replacement gripper is accommodated by updating the TCP position and orientation through a fixed flange-to-TCP transform while retaining the YAM arm’s six-joint structure and geometry.Source: §G.2 · p. 21
  • The arm uses classical Denavit–Hartenberg parameters from Table 5, with distances in meters, angles in radians, and q_j denoting joint j’s angle.Source: §G.2 · p. 21, Table 5 · p. 21
  • The TCP frame origin is at the gripper-tip center, 220 mm along the flange frame’s negative z-axis, with axes oriented to match the UMI convention.Source: §G.2 · p. 21
  • The fixed TCP transform is incorporated into the I2RT MJCF model for forward and inverse kinematics during execution.Source: §G.2 · pp. 21–22

Related papers

Loading 2610.39385v1…