PaperlayerView exact revision on arXiv

Source-linked AI summary

ViTaMIn: Learning Contact-Rich Tasks Through Robot-Free Visuo-Tactile Manipulation Interface

Fangchen Liu, Chuanyu Li, Yihua Qin, Jing Xu, Pieter Abbeel, Rui Chen

arXiv:2504.06156v2cs.RO

TL;DR

Portable demonstration systems often capture vision without tactile feedback, despite touch's role in contact-rich manipulation. ViTaMIn pairs a handheld tactile gripper with multimodal pre-training, and its visuo-tactile approach outperforms vision-only policies across five real-world tasks.

  • Problem

    Portable data-collection devices primarily capture vision, limiting their use for contact-rich and dexterous manipulation tasks where tactile feedback matters.

    Source: §I · p. 2

  • Method

    ViTaMIn collects visuo-tactile demonstrations with a compliant Fin Ray gripper and pre-trains tactile representations through multimodal learning.

    Source: §I · p. 2

  • Results

    ViTaMIn significantly outperforms vision-only policies across five real-world contact-rich manipulation tasks, with pre-trained representations improving data efficiency, robustness, and generalizability.

    Source: §VI · p. 7

  • Takeaways & Limitations

    The interface provides portable visuo-tactile data collection, while its pre-training strategy uses collected action-free data to learn tactile representations.

    Source: §I · p. 2, A. Visuo-Tactile Representation Learning · p. 4

Abstract

from arXiv · show

Tactile information plays a crucial role for humans and robots to interact effectively with their environment, particularly for tasks requiring the understanding of contact properties. Solving such dexterous manipulation tasks often relies on imitation learning from demonstration datasets, which are typically collected via teleoperation systems and often demand substantial time and effort. To address these challenges, we present ViTaMIn, an embodiment-free manipulation interface that seamlessly integrates visual and tactile sensing into a hand-held gripper, enabling data collection without the need for teleoperation. Our design employs a compliant Fin Ray gripper with tactile sensing, allowing operators to perceive force feedback during manipulation for more intuitive operation. Additionally, we propose a multimodal representation learning strategy to obtain pre-trained tactile representations, improving data efficiency and policy robustness. Experiments on seven contact-rich manipulation tasks demonstrate that ViTaMIn significantly outperforms baseline methods, demonstrating its effectiveness for complex manipulation tasks.

I. INTRODUCTION

ViTaMIn addresses the lack of tactile information in portable demonstration systems while retaining robot-free data collection. It combines a compliant tactile gripper with multimodal pre-training; across five tasks, its visuo-tactile demonstrations outperform vision-only baselines.

  • Portable devices avoid physical-robot teleoperation but mainly capture vision, limiting their use for contact-rich and dexterous tasks that need tactile feedback.Source: §I · pp. 1–2
  • ViTaMIn integrates an omnidirectional compliant Fin Ray gripper with customized tactile sensing into UMI, while letting operators perceive force feedback during manipulation.Source: §I · p. 2
  • Its multimodal pre-training combines masked autoencoding and contrastive alignment to learn tactile representations, with reported gains in data and training efficiency and generalization.Source: §I · p. 2
  • The paper presents ViTaMIn as a portable, scalable visuo-tactile data collection system.Source: §I · p. 2
  • ViTaMIn achieves superior performance over vision-only baselines across five manipulation tasks using visuo-tactile demonstrations.Source: §I · p. 2

II. RELATED WORK

Prior work explores tactile sensing for manipulation and portable or robot-linked systems for collecting demonstrations. ViTaMIn builds on these directions by adding tactile sensing to a portable collection system.

A. Visuo-Tactile Manipulation

  • Force/torque sensors provide contact status, but their low-dimensional information is insufficient for more dexterous manipulation tasks.Source: A. Visuo-Tactile Manipulation · p. 2
  • Vision-based tactile sensors capture high-resolution contact information, while rigid designs can restrict end-effector compliance; Fin-Ray-shaped compliant sensors offer all-directional contact sensing.Source: A. Visuo-Tactile Manipulation · p. 2

B. Data Collection System for Robot Manipulation

C. Multimodal Pre-training for Robotics

The paper situates multimodal pre-training as a way to align heterogeneous sensory inputs and introduces masked contrastive pre-training. Its system overview describes the handheld gripper and its visual, tactile, synchronization, and recording components.

C. Multimodal Pre-training for Robotics

A. System Overview

  • The handheld system combines a GoPro, two tactile sensors, a synchronization camera, and a Raspberry Pi for recording; the gripper weighs approximately 1960g.Source: A. System Overview · p. 3, Figure 2 · p. 3
  • AllTact provides the compliant Fin Ray gripper's omnidirectional tactile sensing, capturing global finger and local contact-surface deformation at 30 FPS.Source: A. System Overview · p. 3
  • The GoPro provides synchronized IMU data for SLAM, while ArUco markers on the fingers support visual computation of gripper width.Source: A. System Overview · p. 3

B. Data Processing

Data processing synchronizes visual and tactile recordings and filters trajectories using SLAM. The representation-learning pipeline pre-trains a tactile encoder to combine contact information with masked images to predict future images.

B. Data Processing

  • ArUco markers captured in both video streams align GoPro and synchronization-camera timestamps, with temporal alignment error below 0.05 seconds.The synchronized videos are cropped using control-button start and end signals.Source: B. Data Processing · p. 3
  • SLAM captures end-effector trajectories for imitation learning, but can fail in low-texture environments; it succeeds on approximately 80% of task trajectories.The representation-learning pipeline also pre-trains a tactile encoder using action-free data that does not depend on SLAM success.Source: B. Data Processing · p. 3, A. Visuo-Tactile Representation Learning · p. 4

A. Visuo-Tactile Representation Learning

  • Because tactile images differ from CLIP’s training distribution, the method pre-trains a tactile encoder to improve sensor-based policy robustness and generalizability.Pre-training uses collected action-free datasets without relying on SLAM success.Source: A. Visuo-Tactile Representation Learning · pp. 3–4
  • The tactile encoder combines a masked current image and tactile observation to align with the future image, focusing on contact information for prediction.Masking helps avoid trivial alignment; the design also addresses unavailable proprioception before SLAM succeeds and variation in images across tasks with similar tactile observations [39].Source: A. Visuo-Tactile Representation Learning · p. 4
  • The image CLIP encoder is frozen while the tactile encoder is fine-tuned, and the fused feature is trained against the next image using standard CLIP loss.The fusion projection maps features back to the 512-dimensional CLIP embedding space, and the loss uses a learnable temperature parameter.Source: A. Visuo-Tactile Representation Learning · p. 4
  • After pre-training, a Diffusion Policy is trained on SLAM-filtered data, using a U-Net noise-prediction network and DDIM for faster action inference.Source: A. Visuo-Tactile Representation Learning · p. 4

V. EXPERIMENTS

Figure 4 presents the hardware setup used to deploy the policy.

  • The figure depicts the hardware setup for policy deployment.Source: Figure 4 · p. 4

A. Experimental Setup

The experiments evaluate ViTaMIn on five manipulation tasks using demonstrations collected for single- and dual-arm manipulation. Comparisons include vision-only and non-pretrained baselines, with deployment on a robotic arm and parallel gripper.

A. Experimental Setup

  • Policy deployment uses a 7-DOF Rokae xMate ER3PRO arm with a PGI-140-80-W-S parallel gripper, operating with an 8 cm stroke range and a 10 Hz control loop.The system runs ROS Noetic on Ubuntu 20.04.Source: A. Experimental Setup · p. 4
  • The five tasks span fragile-object placement, moving-hole peg insertion, tactile tube reorientation, scissor hanging, and coordinated dual-arm knife pulling.Source: A. Experimental Setup · p. 5
  • Single-arm tasks use 87–172 raw demonstrations per task, with successful SLAM tracking in approximately 80% of trajectories; the collection also includes dual-arm demonstrations.Source: A. Experimental Setup · p. 5, Table I · p. 5
  • The baselines include a vision-only policy using GoPro images and pre-trained CLIP, and a non-pretrained approach that concatenates visual and tactile observations.The latter uses separate CLIP ViT-B/16 encoders and behavior cloning.Source: A. Experimental Setup · p. 5
  • Across five tasks, vision-only performs worst, while pre-training improves performance; each task is evaluated over 20 trials with randomized initial conditions.The comparison is presented in Table II.Source: Table II · p. 5, A. Experimental Setup · p. 5

B. Failure Analysis

The failure analysis identifies distinct challenges in orange placement and dynamic peg insertion, alongside contact-rich failures across tube reorientation, scissor hanging, and knife pulling.

B. Failure Analysis

  • Orange placement fails through collisions, unstable placement, or motion-planning errors despite correct object detection, while vision-only peg insertion often misses localization and alignment.Source: B. Failure Analysis · p. 5
  • Tube reorientation, scissor hanging, and knife pulling fail through collisions or incorrect orientation, misdetection or release errors, and poor coordination or weak grasps, respectively.Overall, vision-only policies struggle with contact-rich tasks.Source: B. Failure Analysis · p. 5

C. Compliant Articulated Object Manipulation

ViTaMIn reduces average forces relative to vision-only input in a switch-opening task that requires a 90-degree rotation while minimizing axial force.

C. Compliant Articulated Object Manipulation

  • ViTaMIn achieves significantly lower average forces than pure-vision input during compliant switch opening.The comparison averages 10 experiments per condition.Source: C. Compliant Articulated Object… · p. 5
  • The evaluation suite includes five contact-rich tasks, spanning insertion, object hanging, and transparent in-hand manipulation.Source: Figure 5 · p. 6
  • The robot rotates a handle-connected switch 90 degrees while minimizing axial forces for smooth operation.The switch is fixed to a force gauge.Source: Figure 6 · p. 6

D. Ablation Studies

Pre-trained tactile representations improve success across demonstration quantities and support faster policy learning, including performance with limited data and early task completion.

D. Ablation Studies

  • Data-efficiency comparisons use 20 real-world trials per model across three demonstration quantities: 25%, 50%, and 100%.Trials use different initializations, and success rates are calculated separately for each task stage.Source: D. Ablation Studies · p. 6
  • With pre-trained tactile representations, success rates are consistently higher across tasks and data quantities, including test tube reorientation with 25% of demonstrations.The evaluation uses 25%, 50%, and 100% of demonstrations; the figure caption also reports high success with 25% of data after pre-training.Source: D. Ablation Studies · p. 6, Figure 8 · p. 7
  • Pre-trained policies complete the first task stage within 10 epochs and show consistent task-performance improvements.The figure caption reports that overall success rates increase more rapidly with policy pre-training.Source: D. Ablation Studies · p. 7, Figure 9 · p. 7

E. Generalization Capability

The generalization evaluation varies objects and lighting, and the pre-trained multimodal policy performs consistently better across the reported settings.

E. Generalization Capability

  • The pre-trained multimodal policy achieves consistently better performance across generalization settings involving objects and lighting.The table caption characterizes the policy as more robust to novel objects and different lighting conditions.Source: E. Generalization Capability · p. 7
  • Object generalization tests add six unseen small objects and three unseen scissors, while lighting tests increase brightness and add colored disco-ball lighting.Results are reported for orange placement and scissor hanging.Source: E. Generalization Capability · p. 7

VI. CONCLUSION

ViTaMIn collects visual and tactile demonstrations with a portable interface and uses pre-trained visuo-tactile representations; across five real-world contact-rich tasks, it outperforms vision-only policies. Its stated scope is fixed-base single- and dual-arm tasks with parallel-jaw grippers.

VI. CONCLUSION

  • ViTaMIn significantly outperforms vision-only policies across five real-world contact-rich manipulation tasks.The conclusion also reports improved data efficiency, robustness, and generalizability with pre-trained visuo-tactile representations.Source: §VI · p. 7
  • The portable interface collects demonstrations using both visual and tactile signals, and the pre-training strategy uses collected action-free data for tactile representation learning through multimodal contrastive learning.Source: §VI · p. 7
  • The method primarily targets fixed-base single-arm and dual-arm tasks with parallel-jaw grippers; extending it to dexterous hands is future work.Source: §VI · pp. 7–8
Loading 2504.06156v2…