PaperlayerView exact revision on arXiv

Source-linked AI summary

Simultaneous Tactile-Visual Perception for Learning Multimodal Robot Manipulation

Yuyang Li, Yinghan Chen, Zihang Zhao, Puhao Li, Tengyu Liu, Siyuan Huang, Yixin Zhu

arXiv:2512.09851v2cs.ROcs.CV

TL;DR

Vision can lose useful information under occlusion, tactile sensing is local and lacks pre-contact perception, and STS integration into learning-based manipulation remains underexplored. The paper introduces TacThru and TacThru-UMI, combining simultaneous tactile-visual sensing with an imitation-learning policy. Across five real-world tasks, TacThru-UMI achieved an average success rate of 85.5%, compared with 55.4% for vision-only and 66.3% for tactile policies.

  • Problem

    Existing sensing modalities have complementary gaps across approach and contact, while STS designs often lack simultaneous perception and have seen limited integration into learning-based manipulation.

    Source: §I · pp. 1–2

  • Method

    TacThru combines transparent elastomer, persistent illumination, and keyline tracking, while TacThru-UMI uses a Transformer-based Diffusion Policy to learn from visual, tactile, and proprioceptive signals.

    Source: §I · p. 2, C. Policy Learning and Inference · p. 5

  • Results

    Across five diverse real-world tasks, TacThru-UMI achieved an average success rate of 85.5%, versus 55.4% for vision-only and 66.3% for tactile policies.

    Source: §I · p. 2

  • Takeaways & Limitations

    The policy can use vision for alignment when the object is visible and tactile cues when vision is occluded, while thin tissue remains visually observable despite insufficient tactile forces.

    Source: D. Discussions · p. 7

Abstract

from arXiv · show

Robotic manipulation requires both rich multimodal perception and effective learning frameworks to handle complex real-world tasks. See-through-skin (STS) sensors, which combine tactile and visual perception, offer promising sensing capabilities, while modern imitation learning provides powerful tools for policy acquisition. However, existing STS designs lack simultaneous multimodal perception and suffer from unreliable tactile tracking. Furthermore, integrating these rich multimodal signals into learning-based manipulation pipelines remains an open challenge. We introduce TacThru, an STS sensor enabling simultaneous visual perception and robust tactile signal extraction, and TacThru-UMI, an imitation learning framework that leverages these multimodal signals for manipulation. Our sensor features a fully transparent elastomer, persistent illumination, novel keyline markers, and efficient tracking, while our learning system integrates these signals through a Transformer-based Diffusion Policy. Experiments on five challenging real-world tasks show that TacThru-UMI achieves an average success rate of 85.5%, significantly outperforming the baselines of tactile policy(66.3%) and vision-only policy (55.4%). The system excels in critical scenarios, including contact detection with thin and soft objects and precision manipulation requiring multimodal coordination. This work demonstrates that combining simultaneous multimodal perception with modern learning frameworks enables more precise, adaptable robotic manipulation.

I. INTRODUCTION

Robotic manipulation needs perception from approach through contact, but vision, tactile sensing, and existing see-through-skin designs each have limitations. The paper introduces TacThru and TacThru-UMI to combine simultaneous tactile-visual sensing with imitation learning.

I. INTRODUCTION

  • Vision supplies global context but can degrade under occlusion, tactile sensors provide local contact information but not pre-contact perception, and proximity sensors lack spatial richness and precision.These complementary gaps make perception across pre-contact and post-contact phases challenging.Source: §I · p. 1

II. RELATED WORK

  • STS sensors let an embedded camera perceive visual signals such as object appearance and proximity alongside tactile signals such as contact and force.Prior work applied STS sensing to object perception, grasping, in-hand translation, and articulated-object manipulation.Source: §I · p. 2

I. INTRODUCTION

  • Many existing STS designs switch between tactile and visual modes, tactile markers can be hard to track against unpredictable backgrounds, and learning-based manipulation integration remains largely unexplored.Prior applications relied primarily on hand-crafted controllers.Source: §I · p. 2, §II · p. 2
  • TacThru combines a transparent elastomer, persistent illumination, keyline markers, and efficient tracking for simultaneous tactile-visual perception.The design is compatible with standard VBTS fabrication with minimal adoption.Source: §I · p. 2
  • Across five real-world tasks, TacThru-UMI achieved an average success rate of 85.5%, versus 55.4% for vision-only and 66.3% for tactile policies.The framework uses a Transformer-based Diffusion Policy to learn from simultaneous multimodal signals.Source: §I · p. 2

II. RELATED WORK

  • STS sensing has supported object-pose estimation, slip detection, material recognition, in-hand manipulation, and visual servoing, while learning-based integration remained underexplored.TacThru-UMI applies imitation learning to simultaneous multimodal perception for fine-grained and contact-rich manipulation.Source: §II · p. 2

III. TA CTH R U

TacThru prioritizes continuous visual access and robust tactile-marker tracking through a transparent elastomer, persistent illumination, and keyline markers. The design retains VBTS fabrication compatibility while trading depth perception for continuous visual perception.

III. TA CTH R U

  • TacThru uses a fully transparent elastomer, persistent illumination, and robust keyline markers to provide simultaneous tactile-visual perception while retaining compatibility with standard VBTS fabrication.The design differs primarily in elastomer material to support adoption.Source: §III · p. 2

A. Transparent Elastomer with Persistent Illumination

  • Compared with semi-transparent coatings, switchable illumination, or movable components, the transparent elastomer and persistent LED illumination trade depth perception for continuous visual perception.A 24-bulb LED array and diffusion films provide uniform, diffuse lighting while suppressing specular-reflection interference.Source: A. Transparent Elastomer with… · pp. 2–3
  • Contact detection remains available through light-reflection changes at contact interfaces and marker divergence caused by elastomer deformation.The design prioritizes global contact state and fingertip visual context over fine-grained surface-geometry reconstruction.Source: A. Transparent Elastomer with… · p. 3

B. Keyline Markers

  • Keyline markers use two concentric circles with contrasting colors so the inner edge remains detectable across backgrounds where conventional solid markers may disappear.The markers are formed by painting black inner circles and white outer circles on the transparent elastomer.Source: B. Keyline Markers · p. 3
  • Marker spacing is selected to exceed the maximum marker deviation and prevent tracking ambiguity.The figure shows sequential spraying of inner black and outer white markers on transparent elastomer, alongside the TacThru-equipped data collector and end-effector.Source: B. Keyline Markers · p. 3, Figure 2 · p. 3

C. Robust and Efficient Marker Tracking

TacThru tracks marker positions with a Kalman filter, combining image-based detections with motion continuity checks. Its pipeline thresholds images, detects blobs, associates detections with markers, and rejects implausible displacements.

C. Robust and Efficient Marker Tracking

  • A random-walk Kalman filter predicts each marker’s image position and covariance, then updates the estimate using detected-position measurements.The model uses direct position observations, with process and measurement noise parameterized by σ_v and Q.Source: C. Robust and Efficient Marker Tracking · p. 3
  • Image measurements are obtained by grayscale conversion, thresholding, blob detection, and nearest-blob association to each marker’s prior estimated position.Thresholding is intended to reveal marker edges while suppressing low-contrast background edges; nearest-blob matching rejects potential environmental noise.Source: C. Robust and Efficient Marker Tracking · p. 3
  • When marker displacement exceeds the continuity threshold, the measurement is rejected and the state estimate holds at its previous position while covariance accumulates.This check addresses temporary marker loss and mismatches to surrounding blobs under extreme deformation or ambient light.Source: C. Robust and Efficient Marker Tracking · p. 3
  • The tracking model uses a calibrated measurement-noise standard deviation of 0.42 px, with empirically selected intensity and displacement thresholds of 0.784 and 10 px.The measurement-noise ratio is tuned empirically to balance tracking responsiveness and stability.Source: C. Robust and Efficient Marker Tracking · pp. 3–4

D. Evaluation on Marker Tracking

In bottle-grasping tests, filtered keyline markers maintained tracking of all 64 markers with low latency, whereas solid markers missed detections and unfiltered keyline markers admitted false positives. Sensor-loop measurements also found that image read-in, rather than undistortion or tracking, was the main computational bottleneck.

D. Evaluation on Marker Tracking

  • The comparison used eight bottle-grasping trajectories with 1,628 frames per sensor, testing solid markers, keyline markers, and filtered keyline markers.Sensor positions were swapped for half the trajectories, and the bottle was rotated 90 degrees between collections to vary environmental exposure.Source: D. Evaluation on Marker Tracking · p. 4
  • Filtered keyline tracking maintained all 64 markers with an average latency of 6.08 ms, rejecting false positives that unfiltered keyline detection admitted.Solid markers frequently missed detections, leaving tactile information incomplete.Source: Figure 3 · p. 4, D. Evaluation on Marker Tracking · p. 4
  • Undistortion and marker tracking had consistently low computational load, while image read-in was the main sensor-loop bottleneck.The measurements were made on a workstation with an AMD Ryzen 9 5950X CPU.Source: D. Evaluation on Marker Tracking · p. 4, Figure 4 · p. 4

IV. LEARNING MANIPULATION WITH TA CTHRU-UMI

TacThru-UMI extends UMI and Diffusion Policy with simultaneous tactile-visual observations for fine-grained, contact-rich manipulation.

IV. LEARNING MANIPULATION WITH TA CTHRU-UMI

  • TacThru-UMI extends UMI and Diffusion Policy with multimodal tactile-visual observations for imitation-learning manipulation.The framework applies TacThru’s simultaneous tactile-visual perception to fine-grained and contact-rich tasks.Source: §IV · p. 4
  • Evaluations across multiple tasks highlight close-up visual observation of environments and objects alongside tactile feedback.The framework is described as compatible with UMI data collection, processing, and policy deployment.Source: §IV · p. 4
  • The evaluations report more precise manipulation than traditional single-modality approaches.Source: §IV · p. 4

A. System Setup

TacThru-UMI adapts UMI hardware and data collection, then uses a Transformer-based Diffusion Policy to map multimodal observations to robot actions.

A. System Setup

  • The data collector adapts UMI with TacThru sensors on extended linkages, streaming real-time images over USB during demonstrations and policy inference.The robot end-effector uses a low-cost gripper that mirrors the data collector’s body and TacThru fingers.Source: A. System Setup · p. 4

B. Data Collection and Processing

  • Wrist-camera, tactile-sensor, and proprioception streams are synchronized to wrist-camera timestamps and stored in Zarr for training.The pipeline replaces SLAM-based pose tracking with an HTC Vive Tracker; TacThru supports 120Hz marker tracking, while the system uses 30FPS.Source: B. Data Collection and Processing · p. 5

C. Policy Learning and Inference

  • A Transformer-based Diffusion Policy maps multimodal observations to robot actions, dynamically attending across visual, tactile, and proprioceptive signals.Wrist-camera and TacThru frames use DINOv2 encoders; marker deviations and proprioception use dedicated MLPs, and the tokens condition diffusion to produce action chunks.Source: C. Policy Learning and Inference · p. 5, Figure 5 · p. 5

V. EXPERIMENTS

Across five manipulation tasks, TacThru’s multimodal feedback supports perception throughout manipulation, including before contact and with thin, soft objects.

V. EXPERIMENTS

  • TacThru’s multimodal feedback enriches perception of object appearance, state, and the environment throughout the manipulation process.Source: §V · p. 5
  • The five tasks span pick-and-place, sorting, and insertion, assessing tactile information, visual perception, and their simultaneous operation.Source: §V · p. 5
  • The feedback provides cues before contact and when handling extremely thin, soft objects, supporting fine-grained and contact-rich manipulation while maintaining inference efficiency.Source: §V · p. 5

A. Task Settings

The experiments cover five manipulation scenarios and compare policies using different visual and tactile inputs under shared training trajectories.

A. Task Settings

  • PickBottle tests basic pick-and-place, PullTissue tests thin, soft object handling, and SortBolt tests visual discrimination by bolt shape and color.TacThru provides visual distinction where the wrist camera cannot distinguish the small bolts and traditional tactile sensing cannot separate geometrically identical bolts by color.Source: A. Task Settings · p. 5
  • HangScissors tests tactile discrimination of successful hanging versus missed attempts, while InsertCap tests millimeter-level alignment using visual servoing.Source: A. Task Settings · p. 5

B. Experimental Setup

  • The comparison equips one finger with TacThru and the other with a GelSight-type sensor, using identical training trajectories and 62–147 demonstrations per task.Source: B. Experimental Setup · p. 5
  • The four variants are wrist-camera-only, TacThru images with marker deviations, TacThru images without tracked markers, and GelSight images with marker deviations.All policies include wrist camera and proprioception as base inputs.Source: B. Experimental Setup · p. 6
  • Policies use two observation steps, predict 16 action steps, execute steps 3–8, and are evaluated with randomized rollouts matched to the data-collection distribution.Training runs for 150 epochs; evaluation uses 20 rollouts, except SortBolt uses 24 overall and 8 per bolt.Source: B. Experimental Setup · p. 6

C. Results

TacThru-UMI performs well on basic pick-and-place and shows task-specific benefits when manipulation depends on thin-object perception, visual discrimination, tactile feedback, or multimodal coordination.

Pick-and-place

  • All policies achieve at least 95% success on PickBottle, validating multimodal integration without compromising basic manipulation.Source: Figure 7 · p. 6, C. Results · p. 6

Thin-and-soft object perception

  • TacThru detects tissue displacement after slippage and triggers corrective retries, while GS-M and Wrist fail to notice it.Source: C. Results · p. 6

Visual discrimination

  • TacThru’s close view captures bolt geometry and color; its t-SNE clusters separate while VBTS embeddings overlap and GS-M confuses bolts B and C.Source: C. Results · p. 6

Tactile discrimination

  • Marker displacements in TT-M and GS-M reveal scissors-handle engagement that the occluded 2D wrist view cannot reliably identify, informing release timing.Source: C. Results · p. 6

Multimodal fusion

  • 15 / 20 visible cap-mount cases use visual servoing, while 5 / 20 degraded-view cases switch to tactile insertion using marker displacements.Source: C. Results · p. 6

D. Discussions

The discussion highlights structured features from TacThru images, context-dependent multimodal policies, and visual sensing for thin objects. It also notes encoder practicality and sensor failure boundaries.

Representative policy rollouts

  • Fig. 8 rollouts align third-person, wrist-camera, and magnified marker-deviation views over time, with annotations marking manipulation phases and sensing feedback.Source: Figure 8 · p. 7

Encoding TacThru images with DINOv2

  • Patch-token PCA heatmaps separate markers, manipulated objects, and the opposing finger across contact and non-contact views, despite minor noise.Source: D. Discussions · p. 7, Figure 10 · p. 8

Adaptive multimodal strategies

  • Policies learn vision-based alignment when visible and tactile insertion when occluded, without explicit programming.Source: D. Discussions · p. 7

Overcoming conventional tactile limitations

  • Thin objects such as tissue remain visually observable when their forces are insufficient for conventional tactile sensors.Source: D. Discussions · p. 7

Practical deployment viability

  • Pre-trained visual encoders support robust policy learning despite differences from transparent elastomer, marker overlays, and contact deformation.Source: D. Discussions · pp. 7–8

Bolt sorting performance analysis

  • TacThru produces separated bolt clusters and high performance, while GS-M confuses geometrically identical bolts B and C.Source: Figure 9 · p. 8

Sensor failure modes

  • Extreme loads or sharp indentation can delaminate the elastomer, while intense ambient light can reduce marker contrast and cause missed detection.Source: D. Discussions · p. 8

VI. CONCLUSIONS

The conclusion presents TacThru and TacThru-UMI as a simultaneous tactile-visual sensing and imitation-learning platform, with compatibility and future research directions.

  • TacThru provides simultaneous tactile and visual perception through transparent elastomer, persistent illumination, and keyline marker tracking.Source: §VI · p. 8
  • The platform maintains compatibility with standard vision pipelines while addressing limitations of existing tactile sensing.Source: §VI · p. 8
  • Future directions include large-scale data collection with synthetic tactile simulation and exploration of complex dexterous tasks.Source: §VI · p. 8
Loading 2512.09851v2…