PaperlayerView exact revision on arXiv

Source-linked AI summary

TacUMI: A Multi-Modal Universal Manipulation Interface for Contact-Rich Tasks

Tailai Cheng, Kejia Chen, Lingyun Chen, Liding Zhang, Yue Zhang, Yao Ling, Mahdi Hamad, Zhenshan Bing, Fan Wu, Karan Sharma, Alois Knoll

arXiv:2601.14550v1cs.RO

TL;DR

Long-horizon, contact-rich manipulation is difficult to decompose when visual and proprioceptive signals miss event transitions. TacUMI combines synchronized tactile, force-torque, and pose sensing in a robot-compatible handheld gripper with temporal segmentation models, achieving more than 90 percent segmentation accuracy on cable mounting and improving with additional modalities.

  • Problem

    Visual and proprioceptive observations can miss contact-rich event transitions, creating a need for efficient multimodal data collection and robust demonstration segmentation.

  • Method

    TacUMI integrates ViTac sensors, a force-torque sensor, and a pose tracker into a compact handheld gripper, then uses multimodal temporal models to detect skill boundaries.

  • Results

    More than 90 percent segmentation accuracy was achieved on a challenging cable mounting task, with performance improving as more modalities were incorporated.

  • Takeaways & Limitations

    TacUMI provides a practical foundation for synchronized multimodal demonstration collection and segmentation of contact-rich, long-horizon manipulation tasks.

Abstract

from arXiv · show

Task decomposition is critical for understanding and learning complex long-horizon manipulation tasks. Especially for tasks involving rich physical interactions, relying solely on visual observations and robot proprioceptive information often fails to reveal the underlying event transitions. This raises the requirement for efficient collection of high-quality multi-modal data as well as robust segmentation method to decompose demonstrations into meaningful modules. Building on the idea of the handheld demonstration device Universal Manipulation Interface (UMI), we introduce TacUMI, a multi-modal data collection system that integrates additionally ViTac sensors, force-torque sensor, and pose tracker into a compact, robot-compatible gripper design, which enables synchronized acquisition of all these modalities during human demonstrations. We then propose a multi-modal segmentation framework that leverages temporal models to detect semantically meaningful event boundaries in sequential manipulations. Evaluation on a challenging cable mounting task shows more than 90 percent segmentation accuracy and highlights a remarkable improvement with more modalities, which validates that TacUMI establishes a practical foundation for both scalable collection and segmentation of multi-modal demonstrations in contact-rich tasks.

I. INTRODUCTION

TacUMI addresses the difficulty of segmenting contact-rich, long-horizon demonstrations by combining synchronized multimodal data collection with temporal skill-boundary detection. Evaluations on cable mounting show that multimodal sensing supports meaningful transitions and strong segmentation performance.

  • Motivation: Long-horizon manipulation remains difficult because demonstrations often lack explicit segmentation into modular skills.Existing imitation learning frameworks can perform well on short-horizon tasks but treat whole demonstrations as monolithic sequences.
  • Motivation: Tactile sensing captures contact dynamics and tension changes that visual observations may miss during cable mounting.Cable tensioning can be difficult to discern visually but is clearly reflected in tactile signals.
  • Motivation: Teleoperation offers precise demonstrations but is costly and unintuitive, while handheld UMI-family devices omit critical tactile information.These limitations motivate a compact system that combines efficient human demonstration with richer sensing.
  • Approach: TacUMI integrates fingertip ViTac sensors, a wrist-mounted 6D force-torque sensor, and a 6D pose tracker with a continuously lockable jaw mechanism.The design synchronizes tactile, force/torque, and pose measurements while reducing user interference during grasping.
  • Approach: The segmentation framework uses multimodal temporal structure to identify transition boundaries and partition demonstrations into modular skills.The resulting segmentation is intended to support scalable and interpretable learning from long-horizon tasks.
  • Evaluation: TacUMI produces clean force/torque data efficiently and achieves comparable segmentation accuracy when applied to a teleoperated-robot dataset.On cable mounting, multimodal inputs reveal clearer semantic transitions across manipulation phases.

II. RELATED WORK

Prior systems provide either precise but cumbersome teleoperation or efficient handheld collection limited to visual and pose data. TacUMI extends this landscape with synchronized tactile, force/torque, and drift-free pose sensing for contact-aware segmentation.

  • Data collection: Teleoperation provides precise control and direct robot correspondence but is costly, cumbersome, and often lacks intuitive force feedback.These constraints limit practical demonstration collection despite its precision.
  • Data collection: UMI and FastUMI lower the barrier to handheld demonstration but cannot capture tactile feedback or interaction forces.Their SLAM-based pose estimation also suffers cumulative drift during long-horizon tasks.
  • TacUMI: TacUMI integrates synchronized tactile, force-torque, and precise 6D pose sensing while eliminating cumulative drift in long tasks.Its continuous locking mechanism also removes internal actuation forces from force/torque recordings.
  • Event segmentation: Camera and proprioceptive modalities alone can miss skills involving minimal observable motion, such as cable tensioning.Such contact-state changes may therefore be difficult to detect reliably from conventional signals.
  • Event segmentation: The proposed segmentation method combines ViTac, force-torque, and third-person visual data with a BiLSTM to detect subtle contact-state changes.This multimodal integration supports segmentation of complex, long-horizon cable mounting demonstrations.

III. METHODOLOGY

TacUMI is a compact, one-handed demonstration gripper that preserves robot-compatible geometry while synchronizing tactile, force/torque, and pose sensing. Its mechanical design targets reliable collection of executable data during contact-rich manipulation.

  • Device overview: TacUMI combines tactile sensing, 6D force-torque sensing, and 6D pose tracking in a compact, robot-compatible handheld device.The preserved wrist–end-effector layout supports demonstrations intended for complex manipulation tasks.
  • Device overview: The exploded design includes a ViTac sensor, Vive tracker, rack-and-pinion trigger, continuous lock, 6-axis force/torque sensor, and single-handed handle.The locking mechanism supports arbitrary jaw opening widths in locked and released states.
  • Force/torque sensing: A force/torque sensor mounted between the gripper and robot-flange-shaped handle preserves the robot physical stack-up.This placement makes measured wrenches directly transferable without frame re-ordering or ad-hoc calibration.
  • Force/torque sensing: The continuous self-locking mechanism fixes the jaw aperture without sustained trigger pressure, allowing force/torque sensing to measure external contact wrenches.It prevents trigger actuation from leaking into the force/torque measurements during grasping.

3) ViTac Sensing for Contact-Rich Transitions:

TacUMI uses tactile sensing alongside pose, force/torque, and third-person vision to represent contact-rich manipulation and detect skill transitions. Its segmentation pipeline extracts multimodal features, models temporal dependencies, and aggregates overlapping predictions.

  • ViTac Sensing for Contact-Rich Transitions:: ViTac modules capture fine surface deformations that disambiguate visually similar states and improve phase understanding.These tactile cues are especially relevant when contact-state changes are subtle.
  • ViTac Sensing for Contact-Rich Transitions:: Drift-free 6-DoF tracking provides stable trajectories for retargeting during extended demonstrations.A rigidly mounted Vive Tracker enables a fixed transform to the tool center point and avoids cumulative drift associated with visual–inertial SLAM.
  • ViTac Sensing for Contact-Rich Transitions:: The method fuses synchronized tactile images, force-torque data, third-person RGB images, and pose data to detect skill transitions with high granularity.The multimodal representation is designed for temporal segmentation of long-horizon demonstrations.
  • Network Architecture:: Tactile frames and third-person images are encoded into 256-dimensional embeddings, while force/torque and pose streams are preprocessed before fusion.The tactile encoder uses ResNet-50, and the visual encoder uses ResNet-18 with GroupNorm and spatial softmax.
  • Network Architecture:: The fused sequence is processed by temporal models including BiLSTM, TCN, and Transformer architectures to produce frame-wise skill predictions.The BiLSTM maps fused timestep representations to hidden states and then class logits for the skills.
  • Network Architecture:: Overlapping windows support training on fixed-length subsequences, while soft voting averages overlapping class probabilities to restore the original sequence length.The pipeline uses windows of length 50 and stride 10 before aggregating per-frame predictions.

2) Sliding Window and Training Strategy:

The framework uses overlapping fixed-length windows to address idle-frame imbalance and focus training on active manipulation transitions. Windows dominated by idle frames are discarded before temporal-model supervision.

  • Sliding-window sampling: Sliding windows extract fixed-length overlapping subsequences, augmenting the dataset and emphasizing meaningful skill transitions.The strategy also reduces the impact of over-represented idle frames.
  • Training data selection: Windows with more than 80% idle frames are discarded, enforcing a minimum 20% action ratio.Retained windows are paired with frame-wise label sequences for temporal-model supervision.
  • Training procedure: The model is trained with Adam, learning-rate scheduling, dropout after BiLSTM layers, and validation-based early stopping.These choices are used during training on the long-horizon task to limit overfitting.

3) Soft Voting for Inference:

During inference, overlapping window predictions are reconciled by averaging class probabilities for each original frame. The highest-mean-probability class produces a smooth, frame-aligned label sequence.

  • Overlapping predictions: Each original frame receives multiple independent classification outputs because it can appear in several overlapping inference windows.The overlap arises from applying the sliding-window strategy during inference.
  • Probability aggregation: Soft voting aggregates the predicted class-probability vectors from all windows containing each frame.For frame t, the method collects the probability vectors from its K_t covering windows.
  • Final labeling: The final frame label is the class with the highest mean probability across the overlapping windows.This restores temporal alignment with the original demonstration sequence.
  • Output sequence: The aggregated output preserves sequence length and provides one semantic label per frame for evaluation and visualization.The method is intended to improve temporal consistency and reduce prediction noise.

1) Trigger Action Filtering:

Trigger actions introduce temporary operator-induced disturbances into force-torque signals, so the framework identifies and replaces those intervals before robotic use. The resulting stream retains interaction-related measurements while removing internal actuation effects.

  • Interference detection: A second segmentation model classifies trigger-interference intervals for explicit filtering.The same segmentation architecture is reused for this classification task.
  • Trigger interference: Pulling, locking, and releasing the trigger create significant disturbances in recorded force and torque signals.These operator-induced events appear as distinct intervals in the original measurements.
  • Signal replacement: Detected interference intervals are replaced with Gaussian noise estimated from the first 20 frames of each sequence.Those initial frames are assumed to contain no trigger actions, and the replacement preserves natural signal statistics.
  • Robot-compatible output: The processed force-torque signal is clean and directly usable by the robot without undesired human-induced artifacts.After locking, the signals drop near zero, resembling wrist-mounted robot force-torque measurements.
  • Frame transformation: The gripper-to-robot mapping expresses measured wrench data in the robot end-effector frame using rotation and sensor-center displacement.This standard wrench transformation supports transfer across robot platforms with different wrist-sensor and TCP configurations.

IV. EXPERIMENTS

Experiments evaluate TacUMI’s multimodal collection and segmentation pipeline on a cable-mounting task using synchronized tactile, force-torque, pose, and visual sensing. Comparisons across gripper designs and teleoperation assess sensing correctness, transferability, and collection efficiency.

  • Task and setup: The cable-mounting task sequentially inserts a cable into three U-type clips with different orientations.The task requires coordinated manipulation across the clip sequence.
  • Multimodal sensing: The handheld system combines a 1000 Hz force-torque sensor, 16.67 Hz ViTac sensor, 60 Hz pose tracker, and 60 Hz RGB-D camera.These sensors support synchronized multimodal data collection after temporal alignment.
  • Dual-arm collection: TacUMI assigns force-torque and pose sensing to the guiding hand and tactile and pose sensing to the manipulating hand in the dual-arm task.The two grippers coordinate grasp stability, tensioning, positioning, and clip insertion.
  • Gripper designs: Four gripper variants are compared, spanning no locking, ratchet locking, tension-spring actuation, and continuous locking mechanisms.The comparison examines how grasping mechanisms affect collected force-torque data and task execution.
  • Force-torque comparison: The ratchet-based design loses the cable during tightening, whereas the tension-spring and TacUMI designs produce force-torque data consistent with teleoperation.The gripper-collected measurements are mapped into the robot end-effector frame for comparison.
  • Data correctness: TacUMI’s processed force-torque data closely match teleoperated wrist-sensor data while supporting a wider range of grasping widths.Minor deviations are attributed to execution differences between handheld and teleoperated demonstrations.
  • Collection efficiency: 4 minutes versus 1 minute 10 seconds: teleoperation requires more average task-completion time than the handheld gripper.Both methods perform the same three-clip cable-mounting task across multiple trials.

C. Ablation on Model Architectures and Input Modalities

The ablation evaluates segmentation across model architectures and input-modality combinations. BiLSTM performs best overall, while adding tactile and force-torque signals improves segmentation, with limited additional benefit from TCP pose.

  • Model architectures: BiLSTM consistently outperforms TCN and Transformer for frame-wise segmentation accuracy.The paper attributes this to bidirectional temporal context and greater robustness to limited data and sensor noise.
  • Input modalities: Third-person visual input alone performs worst, while adding tactile or F/T signals substantially improves accuracy.TCP pose alone provides only marginal gains.
  • Input modalities: The full four-modality configuration achieves the best overall accuracy, with BiLSTM obtaining the highest result.The evaluated modalities include third-person vision, tactile sensing, force-torque measurements, and TCP pose.
  • Class-wise performance: Most models achieve high F1 scores for grasped and idle phases, but performance drops for released and under torque classes.These latter transitions are subtle and less distinguishable from neighboring phases.
  • Class-wise performance: Adding TCP pose to third-person view, tactile, and F/T inputs yields negligible improvement for this cable mounting task.TCP pose remains nearly unchanged during several critical manipulation phases.
  • Class-wise performance: BiLSTM with full multimodal input provides balanced performance across classes, while released has the lowest F1 score.Released lasts typically only 2–5 frames, so one or two errors sharply reduce its F1 score.

D. Cross-Platform Validation on Robot Data

The segmentation models are trained on TacUMI demonstrations and tested on teleoperation-collected robot data. Multimodal sensing substantially improves cross-platform performance, although the robot data remains more difficult than gripper-collected data.

  • Evaluation setup: Models trained exclusively on TacUMI data are evaluated on robot-side demonstrations recorded via teleoperation.This tests whether multimodal handheld demonstrations transfer to robot platforms.
  • Cross-platform results: Third-person camera input alone produces very poor segmentation performance on robot data.The paper attributes this partly to the domain gap between handheld TacUMI demonstrations and robot end-effector demonstrations.
  • Cross-platform results: Adding tactile or F/T signals significantly improves accuracy, but results remain 10–20% lower than on gripper-collected test sets.The paper links the gap to noisier and less stable demonstrations produced through teleoperation.
  • Cross-platform results: Combining tactile and F/T modalities brings accuracy close to that of the gripper test set.This supports multimodal fusion for event segmentation across platforms.
  • Conclusion: TacUMI-collected data supports accurate event segmentation in handheld demonstrations and transfers effectively to robot-executed tasks.The conclusion presents this transfer as evidence for the data collection design’s suitability for downstream robot learning.

V. CONCLUSION

TacUMI combines multimodal sensing hardware with temporal task segmentation for contact-rich, long-horizon manipulation. Cable-mounting experiments show accurate segmentation and reliable transfer from handheld demonstrations to robot-executed tasks, while broader task generalization remains open.

  • System contribution: TacUMI integrates fingertip tactile sensing, wrist-mounted force-torque sensing, and drift-free 6-DoF pose tracking into a robot-compatible handheld interface.A continuous locking mechanism supports clean force measurements and stable grasps.
  • Algorithmic contribution: The segmentation framework uses multimodal data to divide long-horizon tasks into short, reusable skills.This supports learning modular skills from collected demonstrations.
  • Experimental conclusion: Cable-mounting experiments demonstrate accurate segmentation and reliable transfer from handheld demonstrations to robot-executed tasks.The result covers both the segmentation framework and its cross-platform transfer.
  • Limitation and outlook: TacUMI’s potential for tasks beyond the evaluated setting has not yet been established.The authors identify broader evaluation across contact-rich, long-horizon manipulation tasks as future work.
  • Limitation and outlook: Future extensions could use TacUMI as a platform for multimodal imitation learning and end-to-end robot learning.These extensions would move beyond segmentation toward executable skills from segmented demonstrations.
Loading 2601.14550v1…