PaperlayerView exact revision on arXiv

Source-linked AI summary

In-the-Wild Compliant Manipulation with UMI-FT

Hojung Choi, Yifan Hou, Chuer Pan, Seongheon Hong, Austin Patel, Xiaomeng Xu, Mark R. Cutkosky, Shuran Song

arXiv:2601.09988v1cs.RO

TL;DR

Force-aware manipulation needs careful modulation because insufficient force can fail tasks while excessive force can cause damage, yet conventional F/T sensing is costly, bulky, and fragile. UMI-FT adds compact six-axis sensors to each finger and trains an adaptive compliance policy that predicts position, grasp force, and stiffness for standard controllers. Across contact-rich tasks, the resulting policies regulate external and internal forces and outperform baselines lacking compliance or force sensing.

  • Problem

    Existing handheld data-collection systems mainly capture position and visual information, while commercial F/T sensors are costly, bulky, and fragile for scalable force-aware policy learning.

  • Method

    UMI-FT mounts compact six-axis force/torque sensors on each finger and uses the resulting multimodal demonstrations to train an adaptive compliance policy predicting position targets, grasp force, and stiffness.

  • Results

    Across force-sensitive manipulation tasks, UMI-FT policies reliably regulate external contact forces and internal grasp forces, outperforming baselines lacking compliance or force sensing.

  • Takeaways & Limitations

    UMI-FT provides a scalable platform for learning compliant manipulation from in-the-wild demonstrations using standard compliance controllers.

Abstract

from arXiv · show

Many manipulation tasks require careful force modulation. With insufficient force the task may fail, while excessive force could cause damage. The high cost, bulky size and fragility of commercial force/torque (F/T) sensors have limited large-scale, force-aware policy learning. We introduce UMI-FT, a handheld data-collection platform that mounts compact, six-axis force/torque sensors on each finger, enabling finger-level wrench measurements alongside RGB, depth, and pose. Using the multimodal data collected from this device, we train an adaptive compliance policy that predicts position targets, grasp force, and stiffness for execution on standard compliance controllers. In evaluations on three contact-rich, force-sensitive tasks (whiteboard wiping, skewering zucchini, and lightbulb insertion), UMI-FT enables policies that reliably regulate external contact forces and internal grasp forces, outperforming baselines that lack compliance or force sensing. UMI-FT offers a scalable path to learning compliant manipulation from in-the-wild demonstrations. We open-source the hardware and software to facilitate broader adoption at:https://umi-ft.github.io/.

I. INTRODUCTION

UMI-FT adds compact per-finger force/torque sensing to handheld manipulation data collection, capturing external contact and internal grasp forces. Its compliance architecture and experiments target safe, forceful manipulation while remaining compatible with scalable hardware and standard controllers.

  • Device: UMI-FT mounts compact CoinFT force–torque sensors on each finger to capture external forces and internal grasp forces during demonstrations.The design aims to represent the forces experienced by human demonstrators.
  • Device: CoinFT sensors cost $10 BOM per sensor and are resilient to impact, improving scalability relative to expensive, fragile commercial F/T sensors.
  • Control: UMI-FT wrench measurements can feed standard admittance or impedance controllers to implement stable, accurate, user-specified compliance profiles.
  • Device: Finger-level sensing measures contact forces along the finger body, supports internal grasp-force regulation, and avoids covering every potential contact region.The load path must pass through the sensor.
  • Validation: The Adaptive Compliance Policy regulates external contact and internal grasp forces, with experiments showing improvements from finger-level force sensing and robot compliance across safe, forceful tasks.The demonstrated tasks include whiteboard wiping, zucchini skewering, and lightbulb insertion.
  • Impact: The authors open-source the hardware and software to facilitate large-scale adoption and broader research on multimodal robot data collection.

II. RELATED WORK

Prior manipulation data-collection systems provide visual, motor-current, wrist-mounted F/T, tactile, or wearable sensing, but each has limitations for scalable finger-level force-aware learning. UMI-FT focuses on compact fingertip force/torque sensing to approximate the human operator’s experienced forces and grasp force.

  • Prior force sensing: Prior imitation-learning systems obtain force information from motor currents or wrist-mounted commercial F/T sensors, but lack finger-level sensing and are difficult to scale.The stated scaling barriers are size, fragility, and cost.
  • Tactile sensing: Tactile sensors provide compact, rich contact information, including pressure, location, multi-axial force, slippage, temperature, and proximity.
  • Data collection interfaces: Tactile and wearable sensing have been integrated into teleoperation, handheld, wearable, and kinesthetic data-collection interfaces.
  • Policy-learning signals: Contact microphones, tactile systems, and wearable interfaces can contribute dynamic events, contact properties, object pose, contact intensity, and demonstrator haptic strategies to policy learning.
  • UMI-FT rationale: UMI-FT selects miniature fingertip force/torque sensors to capture dynamic structural forces and applied grasp force while keeping the handheld device small, light, robust, and low-cost.CoinFT is described as measuring 20 mm ×3 mm thick and having low weight.

B. Compliance and Compliance Control

The UMI-FT platform combines multimodal sensing with compliance-oriented control for contact-rich manipulation. Its hardware placement and sensor redesign balance measurement capability with robustness constraints.

  • Compliance and Compliance Control: Compliance describes elastic behavior under external force through stiffness, inertia, and damping; at low speed, stiffness is the dominant component.
  • Compliance and Compliance Control: When force feedback is available, standard impedance control suits backdrivable robots, while admittance control suits stiff, high-accuracy robots.
  • UMI-FT Design: UMI-FT collects synchronized vision, pose, and six-axis force/torque signals at each finger during human demonstrations.The system integrates CoinFT sensors with an iPhone 15 Pro.
  • UMI-FT Design: The platform records RGB, ultrawide RGB, depth, pose, and finger-wrench streams at different rates, with CoinFT measurements sampled at 360 Hz.
  • UMI-FT Design: CoinFT data is timestamped and synchronized with the iPhone streams during postprocessing for multimodal data logging.
  • UMI-FT Design: Each sensor is mounted closer to the fingertip to reduce moment arms and keep grasping primarily compressive, mitigating delamination from excessive moments.
  • UMI-FT Design: The redesigned CoinFT increases robustness and sensing range at the cost of sensitivity.

B. UMI-FT Finger Calibration

UMI-FT calibrates compact per-finger CoinFT sensors against a commercial F/T reference, then uses their measurements in learned and model-based compliance control loops. The architecture supports external wrench response, grasp-force regulation, and compliant alignment during manipulation.

  • Calibration: CoinFT sensors exhibit nonlinear capacitance signals, so each finger is calibrated in situ against a Gamma (ATI) reference sensor.Calibration covers randomized forces and torques across the mounted fingertip structure.
  • Calibration: An MLP maps raw CoinFT capacitance measurements to force/torque estimates, with calibration evaluated on unseen inputs.Figure 2 describes reference collection, learned mapping, and unseen-input evaluation.
  • Controller architecture: The learned visuomotor policy consumes the latest 32 force-sensor frames and outputs reference position, grasp force, and stiffness targets.These targets are passed to two model-based compliance controllers.
  • Controller architecture: Finger wrench measurements feed parallel wrist and grasp-force loops, enabling responses to external forces and regulation of the average finger grasp force.Wrist control combines transformed six-axis measurements, while grasp control regulates a target force through velocity-resolved admittance.
  • Controller architecture: Setting the tool center point at the two-fingertip center supports robust compliant behaviors, including better surface alignment during wiping.The wrist and gripper controllers operate at hardware-limited rates of 500 Hz and 30 Hz, respectively.

B. Imitation Learning with Compliance

The imitation-learning system extends Adaptive Compliance Policy with multimodal RGB, depth, force/torque, and proprioceptive observations. It predicts position, stiffness, grasp-force, and gripper targets for execution by low-level compliance controllers.

  • Policy design: Adaptive Compliance Policy predicts target positions, target forces, and compliance parameters rather than positions alone.UMI-FT augments both ACP inputs and outputs to accommodate its force-sensing hardware.
  • Observation encoding: RGB, depth, and finger force/torque streams are encoded separately, fused with self-attention, and combined with proprioception before conditioning a diffusion policy.The force encoder processes the previous 32 wrench timesteps for each CoinFT sensor.
  • Output decoding: The policy outputs a 21-dimensional vector containing robot position targets, a stiffness matrix, reference grasp force, and gripper action.The output includes reference and virtual target poses alongside compliance and gripper variables.
  • Training and execution: Training post-processes force-motion data into stiffness and virtual-target labels and extracts grasp forces from CoinFT sensors.At inference, reconstructed stiffness and virtual targets go to the arm controller, while gripper width and grasp force go to the gripper force controller.

V. EVALUATIONS

The evaluation tests compliant manipulation on three tasks and compares UMI-FT against diffusion-policy baselines with force, visual, or contact-microphone inputs. The experiments focus on whether compliance and force sensing support contact-rich manipulation.

  • Evaluation setup: Three tasks evaluate compliance control with grasp-force modulation: whiteboard wiping, zucchini skewering, and lightbulb insertion.The tasks are selected to highlight force-sensitive manipulation requirements.
  • Evaluation setup: Experiments use a UR5e robot with a WSG50 gripper and UMI-FT fingers, comparing the method with three baseline policies.The baselines differ in compliance control, force observation, and contact-event sensing.
  • Baselines: DP w/ F uses CoinFT wrench inputs and grasp-force modulation but does not perform compliance control.This isolates force observation and grasp-force modulation from the compliance-policy component.
  • Baselines: DP omits force observations and commands the gripper purely through position control.It is the original diffusion policy baseline.
  • Baselines: DP w/ CM combines diffusion policy with dynamic contact events detected by a contact microphone, while all baselines use RGB and depth inputs.The microphone captures audio at 44.1 kHz and is embedded on the right UMI-FT finger.

A. Whiteboard Wiping

Whiteboard wiping requires enough force to secure and cleanly wipe the surface while avoiding excessive contact force. The evaluation tests force-aware policies across varied scenes and unseen conditions.

  • The robot must firmly grasp the eraser, apply sufficient wiping force, and avoid unnecessarily high forces that could cause damage.
  • 275 demonstrations varied initial pose, table height, and drawing patterns, with safety thresholds of 25 N in the grasp direction and 20 N elsewhere.
  • Each test case was executed five times across normal, altered table-height, unseen-drawing, elevated-whiteboard, and narrower-eraser scenarios.
  • High-frequency contact modulation through compliance control and grasp-force regulation supports reliable forceful wiping under varied conditions.
  • Grasp-force modulation improves robustness to unseen eraser shapes: force-aware policies succeeded in the narrower scenario, while force-free baselines never succeeded there.
  • Contact-microphone information alone was insufficient for continuous force regulation, with excessive force the dominant failure mode.

1) In-Lab Experiments:

In-lab experiments evaluate zucchini skewering under varied object and tool conditions, showing that grasp-force regulation is central while manipulator compliance adds less benefit for the compliant zucchini.

  • The zucchini task requires puncturing a pre-grasped slice while maintaining a firm hold and limiting rotation below 45°.
  • 200 demonstrations varied spike pose, grasp location, and natural zucchini diameter, while safety thresholds were 25 N in the grasp direction and 20 N elsewhere.
  • Evaluation covered normal, yellow-zucchini, thicker-zucchini, and fork conditions across 20 rollouts.
  • Without force information, success dropped significantly because width-only control left the gripper unaware of contact forces, causing frequent slip.
  • Adding grasp-force regulation largely eliminated linear and rotational slip during skewering.
  • The difference between policies with and without compliance control was small, plausibly because the zucchini itself is compliant.

2) In-the-Wild Experiments:

In-the-wild and lightbulb experiments test generalization and force-sensitive insertion under unseen scenes, clutter, heights, and socket stiffness.

  • 630 demonstrations across 15 scenes were collected to investigate in-the-wild generalization.
  • Policies trained on diverse in-the-wild data achieved 20/20 success in an unseen cluttered scene, compared with 4/20 for limited scene-diversity training.
  • The bulb task requires visual alignment, insertion against a spring-loaded electrode with at least 15 N, and rotation to illuminate the bulb.
  • 200 demonstrations varied socket pose, table height, and pre-grasp location, with safety thresholds of 50 N in the grasp direction and 20 N elsewhere.
  • Lightbulb policies were evaluated over 20 rollouts spanning normal, unseen table heights, and a stiffer 20 N socket condition.
  • Compliance control is critical for haptic search when occlusion and a small socket slot make vision-only alignment difficult.

VI. FUTURE WORK AND CONCLUSION

UMI-FT provides scalable finger-level force sensing for compliant manipulation, while experiments demonstrate reliable regulation of external contact and internal grasp forces. The current system remains constrained by tethered data collection, limited camera coverage, and CoinFT reliability under large tensile loads.

  • The current data-collection setup is tethered because a USB–serial bridge connects the CoinFT sensors to the laptop.A Bluetooth-capable microcontroller is proposed as a future wireless extension.
  • The experiments used the iPhone’s main RGB camera at approximately 80° diagonal field of view rather than its 120° ultrawide stream.The selected tasks did not require wider coverage.
  • CoinFT can delaminate under sufficiently large tensile forces, motivating improvements to pillar geometry and bonding methods.
  • Real-world experiments show that UMI-FT policies reliably modulate both external contact forces and internal grasp forces during manipulation.
Loading 2601.09988v1…