PaperlayerView exact revision on arXiv

Source-linked AI summary

OpenJaws: An Open Hardware UMI Framework with Stereo Localization for Generalist Manipulation

Adithyavairavan Murali, Yash Oza, Karanbir Chahal, Neel Jawale, Silas De Vries, Hesam Rabeti, Erwin Coumans, Balakumar Sundaralingam, Stan Birchfield

arXiv:2610.36598v1cs.RO

TL;DR

Handheld UMI demonstrations rely on visual localization for action labels, but existing approaches use monocular-inertial tracking or external tracking systems. OpenJaws combines an open gripper framework with stereo localization and a manipulation-specific benchmark; stereo approaches track essentially full trajectories, while some monocular systems have limited coverage. The paper also reports real-robot policy learning and identifies bimanual validation as future work.

  • Problem

    UMI action labels are affected by SLAM localization errors, while deployment also involves gripper matching and coordinate-transform consistency challenges.

    Source: §I · p. 2

  • Method

    OpenJaws combines an open modular gripper framework with stereo localization, UMI-Loc-Bench-v1, simulation assets, and real-robot policy deployment.

    Source: §I · p. 2

  • Results

    All stereo methods track essentially the full trajectory, while ORB-SLAM3 mono+IMU and UMI-SLAM average 9% and 37% coverage, respectively; ORB-SLAM3 stereo-inertial coverage is 18%.

    Source: A. Localization Evaluations on… · p. 5

  • Takeaways & Limitations

    OpenJaws supports real-robot policy learning, while bimanual validation remains future work and would require jointly localizing two independently moving stereo rigs.

    Source: §VI · p. 7

    Bimanual tasks remain to be validated, requiring joint localization of two independently moving stereo rigs.

    Source: §VI · p. 7

Abstract

from arXiv · show

Handheld data-collection interfaces such as the Universal Manipulation Interface (UMI) allow manipulation demonstrations to be gathered in the wild without a robot in the loop, but the quality of the resulting action labels is bounded by the accuracy of vision-based localization. Existing systems recover end-effector pose from monocular visual-inertial odometry, which is subject to scale ambiguity and drift, or delegate tracking to closed-source modules and external infrastructure that reintroduce cost and setup burden. We present OpenJaws, a fully open hardware and software framework for sensorized UMI that improves localization and simplifies deployment across robot platforms. OpenJaws pairs a modular, 3D-printed gripper exposing a standardized ISO adapter with a low-cost stereo camera to recover metric camera trajectories. We further introduce UMI-Loc-Bench-v1, a benchmark quantifying real-world localization accuracy for handheld data-collection devices, and show that stereo localization substantially outperforms monocular alternatives. Finally, we demonstrate that policies trained on data collected with OpenJaws can be deployed on a real robot across multiple tasks. Hardware, software, and data are released under a commercially permissive license. Code, data, and videos are available at https://open-jaws.github.io/

tasks.

OpenJaws addresses UMI action-label and deployment challenges with an open, stereo-based gripper framework and a benchmark for localization in manipulation data. It combines metric stereo localization with standardized mounting and reports real-robot policy deployment as a contribution.

I. INTRODUCTION

  • Teleoperation can be difficult to scale because it requires a kinematically identical robot copy, while joint-space actions do not readily generalize across embodiments.UMI instead collects data without a robot in the loop and uses relative end-effector task-space actions for robot-arm-agnostic deployment.Source: §I · p. 1
  • UMI action labels are affected by localization errors, while deployment also requires matching gripper kinematics and maintaining a consistent wrist-camera-to-TCP transform.The original UMI setup required a separate actuated parallel-jaw gripper at deployment.Source: §I · p. 2
  • A calibrated stereo pair recovers metric information by triangulating corresponding pixels across views with a known physical baseline.OpenJaws uses a low-cost stereo camera and open-source SLAM and SfM frameworks as an alternative to closed-source or costly tracking systems.Source: §I · p. 2
  • UMI-Loc-Bench-v1 provides a dataset for systematically evaluating visual localization in data representative of UMI-based manipulation policies.The paper presents it as a benchmark for comparing stereo localization with existing monocular approaches.Source: §I · p. 2
  • OpenJaws combines an open hardware and software sensorized UMI framework, a localization benchmark, simulation assets, and real-robot action-chunking policy deployment.The framework uses a standardized ISO 9409-1-50-4-M6 mount for direct attachment to common collaborative robots.Source: §I · p. 2

II. RELATED WORK

Prior UMI systems span robot-free handheld data collection, co-designed grippers, and several approaches to localization, but monocular tracking and external tracking infrastructure have limitations. OpenJaws combines an explicit stereo camera, a standardized robot-side interface, and an open stack, alongside a benchmark for localization in manipulation.

II. RELATED WORK

  • Earlier UMI designs include robot-free trigger-based handheld collection, a large-scale dataset from FastUMI [21], head-mounted stereo-inertial SLAM in HiFi-UMI [22], and purpose-built robot-side grippers in LEGATO [13], Grabette, and Koala [12].LEGATO uses the same actuated gripper for demonstrations and deployment, with the target robot gripping it as a tool.Source: §II · p. 2
  • OpenJaws differs from prior UMI frameworks by pairing a modular stereo wrist camera with an ISO adapter for cross-embodiment deployment and an open-source hardware and software stack.It also presents UMI-Loc-Bench-v1 for evaluating localization with its gripper.Source: §II · p. 3
  • Prior UMI systems use monocular-inertial SLAM or commercial tracking modules, headsets, and external base stations; OpenJaws instead uses an explicit calibrated stereo pair for metric action labels.The cited limitations of monocular-inertial SLAM include scale ambiguity, specialized sensors or infrastructure, and sensitivity to motion blur or preprocessing.Source: §II · p. 3
  • Existing SLAM benchmarks target driving, indoor reconstruction, object pose, or tabletop scenes; UMI-Loc-Bench-v1 is presented as the first benchmark with policy action labels for quantitatively studying UMI localization.The passage identifies KITTI, TUM RGBD, ICL-NUIM, Replica [18], HOPE, and SLAM&Render [35] as related benchmarks.Source: §II · p. 3

III. OPENJAWS

  • OpenJaws is introduced as a modular stereo-based UMI framework, with its relationship to other open UMI frameworks situated in Table I.The supplied passage does not report the table’s comparison values.Source: §III · p. 3

A. Open-Source Modular UMI Gripper

OpenJaws is designed around open, replaceable modules, standardized robot mounting, and components users can assemble from a bill of materials. Its two-finger gripper uses a lightweight, compliant design for pick-and-place tasks.

A. Open-Source Modular UMI Gripper

  • The design uses open, off-the-shelf electronics and 3D-printed structural parts, avoiding a custom PCB so builders can assemble it from purchased bill-of-materials components.The listed components include a Dynamixel actuator, Arduino controller, IMU shield, and commodity stereo camera.Source: A. Open-Source Modular UMI Gripper · p. 3
  • The camera and fingers are interchangeable when they follow the mounting specifications, and the gripper uses the finger-mounting pattern from UMI..Source: A. Open-Source Modular UMI Gripper · p. 3
  • The gripper standardizes on an ISO 9409-1-50-4-M6 mount for collaborative robots, with a custom adapter available for robots such as YAM that lack the mount.Source: A. Open-Source Modular UMI Gripper · p. 3
  • A low-cost global-shutter fisheye stereo camera is the default, and either the camera or Dynamixel motor can be replaced; global shutter avoids image skew during fast handheld motion.Preserving feature correspondences supports localization.Source: A. Open-Source Modular UMI Gripper · p. 3

Gripper Mechanism and Design:

  • OpenJaws uses a two-finger revolute gripper with only one actuated side to reduce complexity and weight; the passive side can be replaced by a second actuated side.Its TPU-printed deformable fingers provide compliance, and the design adapts the Grasp EveryThing finger design [36] without tactile sensing.Source: Gripper Mechanism and Design · p. 3

B. Stereo-Localization

OpenJaws estimates scene geometry from calibrated stereo images, then evaluates localization on varied manipulation tasks using marker-derived ground truth. The evaluation masks the markers from the localization algorithms to avoid giving them tracking features.

B. Stereo-Localization

  • Stereo localization uses calibrated camera centers to triangulate scene points, after fisheye images are undistorted and rectified; gripper and fiducial regions are masked from feature matching.SAM [37] supplies the masks, since moving gripper fingers may affect feature matching; the stereo baseline is later examined as a hyperparameter.Source: B. Stereo-Localization · p. 3

IV. EVALUATING UMI LOCALIZATION: UMI-LOC-BENCH-V1

  • The section evaluates SLAM algorithms for localization in in-the-wild settings.Source: §IV · p. 3

Data Capture Protocol:

  • Ground truth comes from an AprilTag cube fitted with ArUco markers, which are masked so localization methods cannot use the markers as tracking features.Source: Data Capture Protocol · p. 4

IV. EVALUATING UMI LOCALIZATION: UMI-LOC-BENCH-V1

  • The task set spans pick-and-place, kitting, and long-horizon non-prehensile manipulation across different environments.Example rollouts include wrench rotation, picking from a bin, and USB insertion; the figure also shows camera trajectories from separate demonstrations, with gaps marking invalid poses.Source: Data Capture Protocol · p. 4, Figure 3 · p. 4

Quantitative Metrics:

The section defines three evaluation quantities: translational ATE, rotational ATE, and runtime. Translational ATE uses scale-fixed trajectory alignment, while rotational ATE uses the aligned rotation residuals with a frame-convention offset removed.

Quantitative Metrics:

  • Translational ATE aligns the estimated trajectory to ground truth with scale-fixed Umeyama SE(3), then reports per-frame translation residual RMSE in meters, shown as millimeters in tables.Mean, median, standard deviation, minimum, and maximum are also reported; the metric follows standard TUM RGB-D ATE.Source: Quantitative Metrics · p. 4
  • Rotational ATE uses the geodesic rotation angle from the aligned per-frame SE(3) residuals and reports its RMSE in degrees.A constant sensor-to-ground-truth frame-convention offset, computed as the chordal mean of residual rotations, is subtracted before scoring.Source: Quantitative Metrics · pp. 4–5
  • Runtime is wall-clock time around each localization run, including UMI-SLAM's one-time scene-mapping cost.Source: Quantitative Metrics · p. 5

V. RESULTS AND DISCUSSION

On UMI-Loc-Bench-v1, stereo methods track essentially the full trajectories, and COLMAP is the most accurate pure-stereo method. Monocular ViPE recovers strong trajectory geometry but not metric scale; stereo-inertial tracking and mapped UMI-SLAM have substantial coverage limitations.

A. Localization Evaluations on UMI-Loc-Bench

  • ViPE completes 89 of 90 demonstrations, but native-scale common-segment translational ATE is 48.2 mm; ground-truth scale fitting lowers it to 3.5 mm while rotational ATE remains 0.7°.The authors attribute inaccurate metric reconstruction to poor depth and camera-intrinsics prediction, likely because wrist-mounted UMI is out-of-distribution for ViPE's training data.Source: A. Localization Evaluations on… · p. 5
  • On shared UMI-SLAM-tracked segments, COLMAP reaches 2.9 mm/0.4° versus UMI-SLAM's 18.4 mm/4.0°; scale-fitted cuVSLAM mono reaches 12.1 mm/4.8° versus 9.9 mm/2.5° for stereo.All methods are evaluated on identical ground-truth segments, with 27 demonstrations.Source: A. Localization Evaluations on… · p. 5, Figure 4 · p. 6
  • 5.2 mm/0.8° full-trajectory ATE makes COLMAP the most accurate pure-stereo method, versus ORB-SLAM3 (7.4 mm/1.4°) and cuVSLAM (12.4 mm/1.9°).COLMAP's runtime is 836 s, compared with 249 s for ViPE, 43.2 s for ORB-SLAM3, and 34.8 s for cuVSLAM; its offline global bundle adjustment accounts for higher runtime cost.Source: A. Localization Evaluations on… · p. 5, Figure 4 · p. 6
  • ORB-SLAM3 stereo-inertial lowers mean translational/rotational ATE from 5.1 mm/1.3° to 2.0 mm/0.5°, but coverage falls from 100% to 18%.The authors report additional integration and tuning costs for IMU use with handheld dynamics and a small workspace.Source: A. Localization Evaluations on… · p. 5, Figure 4 · p. 6
  • UMI-SLAM's prerecorded map and fiducial initialization raise mean coverage from 9% to 37%, yet require scene-dependent mapping and remain less stable and accurate than pure stereo.UMI-SLAM averages 37% coverage and 18.4 mm/4.0° translational/rotational ATE, while stereo tracks essentially the full trajectory without mapping or fiducial initialization.Source: A. Localization Evaluations on… · pp. 5–6

B. Real Policy Deployment

OpenJaws policies deployed on a UR10 achieved 90% success on both pick-and-place and block-stacking tasks. The trials also expose distinct failure modes: grasp slippage and a dropped block.

B. Real Policy Deployment

  • OpenJaws was used for real-world UMI policy learning with a UR10 on both tasks.Source: B. Real Policy Deployment · p. 6
  • 90% success on both pick-and-place and block stacking over 10 trials under varied initial conditions.Pick-and-place success required placing the target in the bin without collision; its one failure involved grasp slippage, while block stacking included one dropped block.Source: B. Real Policy Deployment · pp. 6–7
  • The UR10 rollouts show pick-and-place at the top and block stacking at the bottom.Source: Figure 5 · p. 7

C. Ablation Experiments

The ablations examine stereo baseline calibration and the effect of localization noise on policy performance. Synthetic drift noise increases action-label error and coincides with lower policy success.

C. Ablation Experiments

  • 3 mm translational ATE at the calibrated baseline; the minimum was 2.7 mm at a −0.25 mm perturbation.Across the −5 mm to 5 mm sweep, error reached 6 mm and 8.0 mm at the two extremes, while rotational ATE varied by only 0.1°.Source: C. Ablation Experiments · p. 7, Figure 6 · p. 7
  • UMI policies use relative trajectories referenced to the initial pose, a representation described as more robust to tracking errors during data collection.Source: C. Ablation Experiments · p. 7
  • 90% to 30%: policy success fell as synthetic drift noise raised training-set translational ATE from approximately 10 mm to more than 30 mm.The noisy-label policy showed frequent grasp slippage and inaccurate placement causing scene collisions.Source: C. Ablation Experiments · p. 7, Figure 7 · p. 7

D. Real-to-Sim Deployment

The real-to-sim pipeline reconstructs simulation environments from UMI demonstrations without ground-truth object poses. The conclusion presents OpenJaws as a framework for real-robot learning and real-to-sim reconstruction, while identifying bimanual localization as future work.

D. Real-to-Sim Deployment

  • The agent receives a UMI demonstration and sim-ready gripper and object assets, then recovers the scene graph without ground-truth object poses.Source: D. Real-to-Sim Deployment · p. 7
  • 7 of 10 gear-insertion demonstrations produced digital twins in which the recorded task played back successfully in simulation.For this contact-rich task, the agent automatically recovered a signed-distance-field collision representation.Source: D. Real-to-Sim Deployment · p. 7

VI. CONCLUSION

  • OpenJaws combines a modular UMI-style gripper with stereo localization and supports real-robot policy learning and agentic real-to-sim reconstruction.Future work could validate bimanual use, requiring joint localization of two independently moving stereo rigs.Source: §VI · p. 7

APPENDIX

The appendix describes trajectory-evaluation metrics adopted from TUM RGB-D and details translational and rotational absolute trajectory error. Both metrics use aligned trajectory residuals, but score different components.

APPENDIX

  • The appendix covers evaluation metrics, benchmark task specifications, the localization-error ablation, and the bill of materials.Source: APPENDIX · p. 8

A. Evaluation Metrics

1) Absolute Trajectory Error:

2) Rotational Absolute Trajectory Error:

B. Benchmark Tasks and Protocol

UMI-Loc-Bench-v1 covers nine manipulation tasks with ten handheld demonstrations each and uses ArUco markers to establish reference poses while preventing localization methods from tracking the markers. The section also specifies synthetic translational drift perturbations and itemizes gripper components and prices.

B. Benchmark Tasks and Protocol

  • UMI-Loc-Bench-v1 contains 90 handheld demonstrations across nine tasks, with marker-based reference poses and marker masking in both camera streams during localization.The tasks span pick-and-place, retrieval, insertion, and non-prehensile manipulation across multiple scenes.Source: B. Benchmark Tasks and Protocol · p. 8

C. Synthetic Localization Noise

  • Synthetic localization drift is injected at intervals of 10 to 30 frames, with each 0.08-probability injection adding an offset of up to 10 cm to the current and subsequent poses.Accumulating injections create a piecewise-constant drift bias in translational action-label trajectories.Source: C. Synthetic Localization Noise · p. 8

D. Bill of Materials

  • Table II itemizes the components for one deployment gripper and its data-collection parts, with prices in USD.The accompanying text specifies that the listed prices use US retail prices as of October 2026.Source: Table II · p. 8, D. Bill of Materials · p. 8
Loading 2610.36598v1…