PaperlayerView exact revision on arXiv

Source-linked AI summary

FOMI: A Force-Feedback Manipulation Interface for Teaching Contact-Rich Manipulation

Yishu Shen, Yiming Luo, Ruilin Wang, Bowen Yang, Shuai Yang, Fangyi Xu, Yujun Shen, Fu Zhang, Boyu Zhou, Yinghao Xu

arXiv:2610.19839v1cs.RO

TL;DR

Limited intuitive force feedback makes contact-rich demonstration collection challenging with handheld interfaces. FOMI uses a scissor-shaped mechanism and an integrated collection system for robot-free collection and teleoperation. It improves collection efficiency and completion rates over evaluated baselines, and demonstrations from both modes support high-success policies on contact-rich tasks.

  • Problem

    Handheld interfaces support scalable robot-free demonstration collection, but limited intuitive force feedback challenges operators collecting contact-rich demonstrations.

    Source: §1 · pp. 1–2

  • Method

    FOMI combines a scissor-shaped interface that transmits interaction forces directly to the hands with a collection system for robot-free demonstrations and mapped robot teleoperation.

    Source: §1 · p. 2

  • Results

    FOMI achieves higher collection efficiency and completion rates than evaluated baselines, and its demonstrations support high-success policies on four contact-rich tasks.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    Demonstrations collected in both modes support policy learning for the evaluated contact-rich tasks without tactile or force inputs.

    Source: §5 · p. 12

    The limited user-study sample size limits generalization across operators.

    Source: §5 · p. 12

Abstract

from arXiv · show

Robot-free collection with handheld interfaces enables scalable acquisition of human demonstrations for policy learning. However, existing handheld interfaces often provide limited intuitive force feedback, making contact-rich demonstration collection challenging for human operators. To address this, we present FOMI, a force-feedback manipulation interface with a scissor-shaped mechanism, which transmits object interaction forces directly to the operator's hands through actuation links without force sensors. This intuitive force feedback supports real-time action adjustment during robot-free collection of contact-rich demonstrations. Around FOMI, we build a demonstration collection system that integrates sensor calibration, data acquisition, and synchronization for robot-free collection. By further mapping pose changes and gripper opening of FOMI to robot end-effector poses and gripper commands, the system also supports teleoperation. Across four contact-rich tasks, FOMI improves collection efficiency by approximately 29% and 18% over the best baselines in robot-free collection and teleoperation, respectively. Demonstrations collected with FOMI in both modes enable policies to perform challenging contact-rich manipulation tasks with high success rates.

1. Introduction

Handheld robot-free collection can scale demonstrations, but limited intuitive force feedback makes contact-rich collection difficult. FOMI combines a scissor-shaped force-feedback interface with a collection system that supports robot-free collection and teleoperation.

1. Introduction

  • Handheld interfaces support scalable robot-free demonstration collection, but limited intuitive force feedback can impede real-time action adjustment and lower collection efficiency.The cited discussion links this challenge to more failed attempts.Source: §1 · pp. 1–2
  • The collection system integrates sensor calibration, data acquisition, and synchronization, and maps FOMI pose changes and gripper opening to robot commands for teleoperation.Source: §1 · p. 2
  • FOMI uses a scissor-shaped mechanism whose rigid handle-to-actuation-link assembly provides a direct force path, while meshing sector gears produce symmetric opening and closing.The design avoids additional transmission stages between handle and actuation link.Source: §1 · p. 2
  • FOMI achieves the highest collection efficiency and improves completion rates over baseline interfaces in both collection modes; its demonstrations support high-success policies on four contact-rich tasks.The introduction also states that hardware designs, code, and datasets will be released.Source: §1 · pp. 2–3

2. Related Work

Related work spans robot-free collection and teleoperation interfaces, approaches to conveying interaction forces, and additional sensing for contact-rich policy learning.

2.1. Demonstration Collection Interfaces

  • Robot-free interfaces collect demonstrations across diverse real-world environments, while leader-follower, VR, and smartphone interfaces support teleoperation through joint-space or end-effector pose control.Some recent interfaces support both robot-free collection and teleoperation.Source: §2.1 · p. 3

2.2. Force Feedback for Demonstration Collection

  • Existing interfaces convey force mechanically or actively; transmission stages and backlash may obscure feedback, while active force feedback requires actuators and control.FOMI instead integrates each handle with an actuation link for direct mechanical feedback and handle support; Table 1 compares designs, force feedback, and support handles.Source: §2.2 · p. 3, Table 1 · p. 3

2.3. Contact Sensing for Policy Learning

  • Contact-rich policy learning interfaces may record tactile observations or interaction forces, whereas FOMI demonstrations support policy learning without tactile or force inputs.Source: §2.3 · pp. 3–4

3. Methodology

The methodology presents FOMI’s scissor mechanism and a calibrated collection system for robot-free demonstrations and teleoperation. It also describes synchronized data processing and policy learning and deployment.

3.1.1. Scissor Mechanism

  • The scissor mechanism directly transmits gripper interaction forces through actuation links to the handles, while meshed gears drive symmetric opening and closing.Without return springs, operators actively drive both directions, so perceived forces arise from object interactions rather than spring preload.Source: §3.1.1 · p. 4

3.1.2. Kinematic and Force Transmission Analysis

  • The parallelogram linkage keeps the gripper carriers at a constant angle, and width increases monotonically with opening angle; maximum width is 84.9 mm with a 14.9 mm downward displacement.The opening range closely matches the target parallel gripper’s range.Source: §3.1.2 · p. 5
  • Across the opening range, the handle-force to gripper-contact-force ratio varies from 0.49 to 0.65, and measured ratios slightly exceed predictions consistent with neglected joint friction.At closure and full opening, predicted ratios are 0.646 and 0.494; measured ratios are 0.667 and 0.511, respectively.Source: §3.1.2 · p. 5

3.2.1. Sensor Configuration and Calibration

  • FOMI’s sensors comprise a fisheye camera, a 6-DoF pose tracker, and two magnetic encoders, while calibration maps encoder signals to gripper commands and tracker poses to end-effector poses.The two encoder readings are combined to reduce calibration errors associated with gear backlash.Source: §3.2.1 · p. 6

3.2.2. Robot-Free Collection

  • In robot-free collection, operators use one handheld FOMI in each hand and record camera images, calibrated end-effector poses, and calibrated gripper commands.Operators rely on force feedback to perceive contact and adjust actions in real time.Source: §3.2.2 · p. 6

3.2.3. Teleoperation

  • In teleoperation, calibrated FOMI pose and gripper signals control the robot online, with each arm’s target motion computed relative to its initial pose.For bimanual operation, each arm uses its corresponding initial interface and robot poses.Source: §3.2.3 · p. 7

3.2.4. Data Synchronization

  • Camera, tracker, and encoder streams are aligned to an 80 Hz timeline using nearest-in-time samples, and episodes with flagged frame loss or tracking failures are excluded.Images are flagged for frame loss when consecutive-image timestamp gaps exceed 20 ms.Source: §3.2.4 · p. 7

3.3. Policy Learning and Deployment

  • The policy is fine-tuned separately on robot-free and teleoperated demonstrations to predict bimanual action chunks, with deployment transforming predicted poses into robot-base coordinates.Training downsamples synchronized data from 80 to 20 Hz; each predicted action chunk contains 50 steps.Source: §3.3 · p. 7

4. Experiments

Across four contact-rich tasks, the experiments evaluate collection efficiency and completion, policy performance from robot-free and teleoperated demonstrations, cross-platform deployment, and gripper-command calibration.

4.1. Tasks and Robot Platforms

  • The four tasks cover fragile chip handling, tube reorientation and insertion, sustained dish wiping, and zipper opening on a deformable pouch.The tasks use dual-arm platforms with identical grippers and wrist cameras.Source: §4.1 · p. 8

4.2.1. Robot-Free Collection Efficiency; 4.2.2. Teleoperation Efficiency

  • Collection efficiency improves by 29.0% over UMI (17.8 vs. 13.8 successful demonstrations per 10 min) and by 18.3% over GELLO (8.4 vs. 7.1).FOMI achieves the highest mean collection efficiency on all four tasks in both modes.Source: §4.2.1 · p. 8, §4.2.2 · p. 8, Table 2 · p. 9

4.2.3. Completion Rate

  • Completion rates are 86.6% for FOMI versus 64.6% for UMI and 64.5% for Ring in robot-free collection, and 75.6% versus 54.0% for GELLO and 34.1% for VR in teleoperation.The authors suggest fewer failed attempts partly explain FOMI’s efficiency gains.Source: §4.2.3 · p. 9, Figure 5 · p. 9

4.2.4. Exploratory User Study in Robot-Free Collection

  • FOMI’s mean control-accuracy and force-feedback ratings were 4.33 and 4.67, versus baseline maxima of 2.67 and 3.33; its comfort rating was 4.33, similar to UMI’s 4.00.Ratings came from three operators on a 0–5 scale and provide preliminary evidence of more favorable perceptions.Source: §4.2.4 · p. 9, Figure 5 · p. 9

4.3.1. Robot-Free Demonstrations

  • With 25, 50, and 100 robot-free demonstrations per task, average success rates were 42.0%, 59.1%, and 78.4%, and progress rates were 50.8%, 67.1%, and 85.8%.At the average collection efficiency, 100 demonstrations require approximately one hour of active collection.Source: §4.3.1 · p. 10, Figure 7 · p. 10

4.3.2. Teleoperated Demonstrations

  • With 25 teleoperated demonstrations per task, policies achieve average success and progress rates of 70.5% and 83.0%, versus 42.0% success with 25 robot-free demonstrations.Robot-free collection is more than twice as efficient (17.8 vs. 8.4 demonstrations per 10 minutes) and requires no robot hardware.Source: §4.3.2 · p. 11, Figure 8 · p. 11

4.4. Deployment Across Robot Embodiments

  • Deploying the policy trained on 25 robot-free demonstrations for Open Pouch to Flexiv achieves 68.2% success and 80.3% progress.The evaluated platforms share the same gripper and wrist camera; demonstrations encode relative end-effector poses and gripper commands rather than arm-specific joint configurations.Source: §4.4 · p. 11, Figure 9 · p. 11

4.5. Gripper-Command Calibration Accuracy

  • The direction-dependent piecewise quadratic mapping has the lowest RMSE: 0.80 command units (≈0.3 mm) on the left interface and 0.88 (≈0.3 mm) on the right.Removing the direction-dependent offset raises errors to 2.48 and 3.49, respectively; the mapping is more interpretable and uses fewer parameters than the comparable-accuracy MLP.Source: §4.5 · pp. 11–12, Table 3 · p. 11

5. Conclusion

The conclusion presents FOMI as a force-feedback manipulation interface and reports higher collection efficiency and completion rates than evaluated baselines. Demonstrations from both collection modes support policy learning for the evaluated contact-rich tasks.

5. Conclusion

  • FOMI provides mechanical force feedback during robot-free collection, while mechanical stops indicate gripper width command limits in teleoperation.The authors describe higher collection efficiency and completion rates than evaluated baselines under the tested conditions.Source: §5 · p. 12
  • FOMI achieves higher collection efficiency and completion rates than the evaluated baselines under the tested conditions.Source: §5 · p. 12
  • Demonstrations collected in both modes support policy learning for the evaluated contact-rich tasks without tactile or force inputs.Source: §5 · p. 12
  • Interaction forces are not recorded for policy learning, and the limited user-study sample constrains generalization across operators.The authors plan to expand the user study and refine the mechanical design.Source: §5 · p. 12

Appendix

The force-transmission measurement setup records handle force, gripper-contact force, and opening angle while the mechanism's links and handles remain free to move.

A. Force Transmission Measurement Setup

B. Image Views During Demonstration Collection

The image views document grippers and object interactions during four contact-rich demonstration tasks and serve as policy-learning visual observations.

B. Image Views During Demonstration Collection

C. Demonstration Collection Protocol and Baseline Settings

The protocol defines the active collection period and unsuccessful attempts, specifies interface settings for robot-free collection and teleoperation, and reports pooled attempt counts underlying completion rates.

C. Demonstration Collection Protocol and Baseline Settings

D.1. Training Details

Policies are fine-tuned from the π0.5 base checkpoint for 10,000 steps, with image preprocessing and 22 evaluation rollouts per task under each condition.

D.1. Training Details

  • Policies start from the π0.5 base checkpoint and are fine-tuned for 10,000 steps with batch size 64, AdamW, warmup, and cosine learning-rate decay.The peak learning rate is 1.5×10−5, with a 1,000-step warmup.Source: §D.1 · p. 16
  • Each 720 × 540 image is center-cropped to 300 × 300 pixels and resized to 224 × 224 pixels for policy input.Source: §D.1 · p. 16
  • Evaluation uses the final checkpoint at step 10,000, with 22 rollouts per task under each experimental condition.Source: §D.1 · p. 16

D.2. Real-Time Chunking

Real-time chunking retains and executes part of a predicted trajectory while the next inference runs. Subsequent chunks are re-expressed against a retained target reference and aligned to execution progress when they arrive.

D.2. Real-Time Chunking

  • RTC executes a committed three-action prefix during inference, triggers a new call after every 20 executed actions, and runs actions at 20 Hz.Inference latency is approximately 120 ms.Source: §D.2 · p. 16
  • After 20 executed actions, the client retains the remaining 30 of 50 predicted actions as overlap for the next inference call.Absolute TCP poses are stored in both robot-base representations, preserving spatial targets when the reference frame changes.Source: §D.2 · p. 16
  • Before inference, overlap and history are re-expressed relative to the current left TCP pose, while the first left-arm overlap target becomes the model reference for both arms.Gripper commands remain unchanged; subsequent right-arm predictions are decoded relative to the first decoded left-arm target.Source: §D.2 · p. 16
  • While inference runs, the controller executes the retained overlap; the replacement chunk is installed from action d through s −1 to avoid replaying elapsed actions.If inference exceeds the switching deadline, the system falls back to synchronous replanning.Source: §D.2 · p. 16
Loading 2610.19839v1…