PaperlayerView exact revision on arXiv

Source-linked AI summary

Touch in the Wild: Learning Fine-Grained Manipulation with a Portable Visuo-Tactile Gripper

Xinyue Zhu, Binghao Huang, Yunzhu Li

arXiv:2507.15062v2cs.ROcs.AIcs.LG

TL;DR

Portable visuo-tactile data collection faces limitations from unsuitable tactile hardware and challenges in learning across modalities. This paper pairs a tactile handheld gripper with cross-modal representation learning and reports successful policy transfer and robustness to environmental disturbances on fine-grained real-world tasks.

  • Problem

    Portable visuo-tactile data collection faces unsuitable tactile hardware and unresolved multimodal learning challenges, despite tactile feedback's role in precise manipulation.

    Source: §1 · pp. 1–2

  • Method

    The paper presents a lightweight handheld tactile gripper for in-the-wild data collection and a cross-attention masked-autoencoding framework that learns joint visual-tactile representations for downstream policy learning.

    Source: §1 · p. 2, §4.1 · p. 4, §4.3 · p. 6

  • Results

    Policies demonstrated successful transfer and robustness to environmental disturbances on real-world fine-grained tasks, including test tube insertion and pipette-based fluid transfer.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The work demonstrates the utility of a handheld visuo-tactile platform and pretrained joint encoder for fine-grained manipulation on a single-arm robot with a parallel gripper.

    Source: §6 · p. 10

Abstract

from arXiv · show

Handheld grippers are increasingly used to collect human demonstrations due to their ease of deployment and versatility. However, most existing designs lack tactile sensing, despite the critical role of tactile feedback in precise manipulation. We present a portable, lightweight gripper with integrated tactile sensors that enables synchronized collection of visual and tactile data in diverse, real-world, and in-the-wild settings. Building on this hardware, we propose a cross-modal representation learning framework that integrates visual and tactile signals while preserving their distinct characteristics. The learning procedure allows the emergence of interpretable representations that consistently focus on contacting regions relevant for physical interactions. When used for downstream manipulation tasks, these representations enable more efficient and effective policy learning, supporting precise robotic manipulation based on multimodal feedback. We validate our approach on fine-grained tasks such as test tube insertion and pipette-based fluid transfer, demonstrating improved accuracy and robustness under external disturbances. Our project page is available at https://binghao-huang.github.io/touch_in_the_wild/ .

1 Introduction

The paper addresses the lack of portable tactile sensing in handheld demonstration collection and the challenge of learning from distinct visual and tactile signals. It presents a visuo-tactile gripper, multimodal learning framework, and in-the-wild dataset for real-world manipulation.

1 Introduction

  • Portable visuo-tactile collection is challenged by bulky or fragile tactile hardware and by the difficulty of learning from local physical touch and broader visual context.Existing soft bubble sensors are described as too large for handheld use, while optical-based sensors can be bulky or less robust for long-term field use; vision-only systems also miss tactile feedback that complements visual sensing and is invariant to lighting and camera viewpoint.Source: §1 · pp. 1–2
  • A lightweight handheld gripper integrates flexible piezoresistive tactile sensors for collecting human manipulation demonstrations indoors and outdoors.The system is designed for portable, in-the-wild data collection.Source: §1 · p. 2
  • Masked autoencoding with cross-attention jointly learns visual and tactile inputs while preserving modality-specific characteristics, supporting more sample-efficient and accurate manipulation policies.The framework is designed to help policies interpret fine-grained tactile feedback.Source: §1 · p. 2
  • The in-the-wild dataset contains over 2.6 million visuo-tactile pairs from more than 2,700 demonstrations across 43 tasks and 12 indoor and outdoor environments.The dataset supports visuo-tactile pretraining, and its resulting encoder improves downstream policy learning.Source: §1 · p. 2
  • Real-world test tube insertion and pipette-based fluid transfer demonstrate successful policy transfer and robustness to environmental disturbances.The authors position the results as evidence for portable visuo-tactile platforms in complex, real-world settings.Source: §1 · p. 2

2 Related Works

Prior work identifies a need for scalable real-world visuo-tactile data, especially in unstructured settings where vision can fail and touch is valuable. This work addresses that need with a portable collection system, cross-modal learning, and a diverse dataset.

Scalable multi-sensory data and learning

  • Pretraining can align vision and touch while reducing task-specific data demands, but simulated tactile signals often mismatch real contact and multimodal collection adds hardware and synchronization burdens [5] [10] [12] [13] [14] [15] [16] [17] [18] [20] [21].Most existing datasets are limited to structured indoor environments [19].Source: §2 · p. 2
  • Prior in-the-wild systems show vision-only performance on simple tasks, but overlook touch’s complementary value when lighting or background clutter degrades vision [22] [23] [24] [25].Source: §2 · pp. 2–3
  • The proposed portable system synchronizes fisheye RGB and flexible tactile sensing for large-scale collection, while masked reconstruction learns cross-modal representations for tactile recovery and downstream policy learning.Unlike methods that merge vision and touch in a unified 3D space [19], it learns correlations between modalities for fine-grained manipulation.Source: §2 · p. 3

Visuo-tactile manipulation

  • Unlike typically rigid, bulky optical tactile sensors [15] [33] [39] [40] [41] [42] [43], thin flexible sensors measure contact-force distributions and provide consistent, object-agnostic representations suited to portable real-world learning [19] [45].Source: §2 · p. 3
  • The work publicly releases a diverse real-world dataset of synchronized RGB and tactile interactions across tasks and environments, filling a previously unmet large-scale in-the-wild data gap.The concurrent ViTaMIn project [47] shares related goals, but differs in system design, representation learning strategy, and task focus.Source: §2 · p. 3

3 Visuo-Tactile Data Collection System

This section presents a handheld gripper for scalable in-the-wild visuo-tactile data collection, combining flexible tactile sensors with synchronized visual and tactile recording.

3.1 Scalable Flexible Tactile Sensors

  • The gripper embeds matrix tactile pads in soft fin-shaped fingers, adapting 3D-ViTac [19] to the adaptive fin-shaped gripper [24].Each pad sandwiches a piezoresistive sensing layer between two flexible printed circuits.Source: §3.1 · p. 3
  • FPC electrodes provide each pad 12×32 taxels of 2×2mm2, while tool-free fabrication takes under five minutes and mounting takes two.The FPC design supports uniform trace pitch, improved robustness, and denser sampling along the flexible fingers.Source: §3.1 · p. 4

3.2 Portable Multi-Modal Sensing System

  • The battery-powered handheld unit weighs approximately 962 g and streams 12 × 32 tactile pads at 23 Hz for portable, high-throughput collection.Two custom Arduino-based boards sit beneath the palm, and data is transmitted over USB to a host device.Source: §3.2 · p. 4
  • A shared host-clock reference aligns camera video and 23 Hz tactile data by decoding a 30 Hz QR-code time sequence offline, without external synchronization hardware.QR codes displaying host time are shown before demonstrations and refreshed at 30 Hz; tactile packets carry host-clock timestamps.Source: §3.2 · p. 4

4 Visuo-Tactile Representation and Policy Learning

The section presents a two-stage approach: learn a joint yet modality-specific visuo-tactile representation through masked tactile reconstruction, then use it with proprioception in a diffusion policy for manipulation.

4.1 Problem Formulation

  • The formulation uses synchronized RGB images and tactile maps to train a self-supervised encoder whose fused representation preserves modality-specific structure for downstream manipulation.Source: §4.1 · p. 4

4.2 Stage 1: Visuo-Tactile Representation Learning

  • Masked autoencoding reconstructs missing tactile regions from partial tactile input plus visual context, retaining fine-grained, geometry-sensitive tactile information that contrastive alignment may suppress.Source: §4.2 · p. 5

4.3 Stage 2: Policy Learning via Behavior Cloning

  • At each step, the conditional diffusion policy takes the fused visuo-tactile embedding and proprioceptive state as its observation, and learns manipulation behaviors from human demonstrations.Source: §4 · p. 4, §4.1 · p. 5, §4.3 · p. 6

5 Experiments

The experiments examine how tactile feedback and visuo-tactile pretraining support fine-grained manipulation. Across four real-world tasks, the analyses highlight tactile state and force cues, coordinated multimodal representations, and pretraining benefits in low-data and low-epoch settings.

5.1 Large-Scale Visuo-Tactile Data and Pretraining

  • The pretraining dataset contains over 2.6 million visuo-tactile pairs from more than 2,700 demonstrations, spanning 43 tasks and 12 indoor and outdoor environments.It includes four core tasks, other indoor tasks, and over 30 in-the-wild tasks.Source: §5.1 · p. 6
  • Qualitative analyses show tactile reconstruction for in- and out-of-distribution inputs and attention to gripper–contact regions across varied backgrounds and objects.The evaluation masks tactile and RGB inputs for cross-modal reconstruction and visualizes final-layer ViT attention maps.Source: §5.1 · p. 6, Figure 5 · p. 7

5.2 Experimental Setup

  • The system is evaluated on four real-world tasks, with 20 trials per task under moderate initial-position and environmental randomization; policies are trained for 60 epochs.Task success is assessed using task-specific completion criteria.Source: §5.2 · p. 8, (2) Task Requiring Fine-Grained Force… · p. 9

(1) Task Requiring In-Hand State Information

  • Test Tube Collection requires picking up, reorienting, and inserting a tube without dropping or breaking it, while Pencil Insertion requires reorienting an upright-grasped pencil before inserting it into a sharpener.Success requires full tube insertion or accurate pencil insertion, respectively.Source: (1) Task Requiring In-Hand State… · p. 8

(2) Task Requiring Fine-Grained Force Information

  • Fluid Transfer tests pipette pressure control to move fluid without spilling, while Whiteboard Erasing tests controlled force to remove two strokes without exceeding damaging force limits.Success requires spill-free transfer or removal of all visible marker ink, respectively.Source: (2) Task Requiring Fine-Grained Force… · p. 8

5.3 Analysis and Discussion

  • The integrated tactile gripper and joint visuo-tactile encoder support manipulation by exposing in-hand state, detecting force-driven phase transitions, and coordinating vision with touch.Tactile feedback helped with object reorientation despite changes in visual appearance and sensed fluid-release transitions that vision alone could miss; cross-attention helped avoid excessive force in whiteboard erasing.Source: §5.3 · pp. 9–10
  • Pretraining particularly benefits Test Tube Collection with fewer than 60 demonstrations or fewer than 60 training epochs, improving trajectories and robustness to environmental variation.With 30 demonstrations, pretrained policies usually failed only at final insertion, while policies without pretraining could stall after grasping; in low-epoch trials, pretraining helped policies handle difficult initial configurations.Source: §5.3 · p. 10, Figure 7 · p. 10

6 Conclusion and Limitations

The work introduces a tactile-enabled handheld gripper and large-scale visuo-tactile dataset, and demonstrates their utility for fine-grained manipulation through joint-encoder pretraining and evaluation. The authors aim to extend the approach to multi-finger dexterous hands.

6 Conclusion and Limitations

  • The work presents a tactile-enabled handheld gripper and large-scale visuo-tactile dataset, then demonstrates their utility by pretraining a joint encoder and evaluating it on fine-grained tasks with a single-arm robot and parallel gripper.The robot setup mirrors the handheld setup.Source: §6 · p. 10
  • Future work aims to extend the approach to multi-finger dexterous hands, where tactile feedback could support richer and more dexterous manipulation skills.Source: §6 · p. 10

Supplementary Materials … B.1 Effect of Demonstration Quantity on Pretraining

New tactile sensors preserve nearly identical Test Tube Collection success rates across subtasks, while larger pretraining datasets improve reconstruction quality and focus attention more precisely on contact regions.

A Cross-Sensor Generalization and Consistency

  • Test Tube Collection success rates remained nearly identical across subtasks with newly fabricated tactile sensors, supporting cross-sensor consistency and interchangeability for downstream deployment.The evaluation replaced the robot’s sensors with new sensors not previously used in the project.Source: Table 2 · p. 15, A Cross-Sensor Generalization and… · p. 15

B.1 Effect of Demonstration Quantity on Pretraining

  • More pretraining demonstrations steadily reduce test MSE and stabilize training, producing cleaner tactile reconstructions and self-attention concentrated more precisely on gripper contact regions.These changes indicate stronger cross-modal representations.Source: Figure 8 · p. 16, §B.1 · p. 16

B.2 Impact of Removing Task-Specific Demonstrations on Pretraining · C Downstream Task Training Analysis

Removing task-specific demonstrations from pretraining yielded nearly identical downstream performance and preserved contact-focused attention, indicating that the learned visuo-tactile representations generalize to tasks absent from pretraining.

B.2 Impact of Removing Task-Specific Demonstrations on Pretraining

  • The study withheld all Test Tube Collection demonstrations in one experiment and all Whiteboard Erasing demonstrations in another.This tested whether the pretrained encoder generalized when task-specific examples were absent from pretraining.Source: §B.2 · p. 16
  • Pretraining used approximately 2,500 in-the-wild demonstration videos after task-data removal, down from 2,700 in the original dataset.The encoder was retrained on the reduced dataset, then fine-tuned and evaluated on the corresponding real-world task.Source: §B.2 · p. 16
  • Across both held-out tasks, ViT attention remained focused on gripper contact regions, preserving meaningful visuo-tactile correspondences.The attention pattern was consistent across training and held-out tasks and with results from the full pretraining dataset.Source: §B.2 · pp. 16–17
  • Policies with pretraining converged faster and had consistently higher success rates than policies without pretraining on Test Tube Collection and Pencil Insertion.Figure 9 shows training-loss curves from epochs 5 to 60, with rollout success rates indicated along the curves.Source: Figure 9 · p. 17
  • Downstream policies achieved nearly identical performance with or without the corresponding task demonstrations in pretraining, supporting generalization to unseen manipulation tasks.Table 3 compares policies after excluding Test Tube Collection or Whiteboard Erasing demonstrations; the supplied passages do not provide the table’s exact values.Source: Table 3 · p. 17, §B.2 · p. 17

C.1 Impact of Pretraining on Training Loss Convergence · C.2 Policy Self-Attention Maps Before vs. After Downstream Fine-Tuning

Visuo-tactile pretraining improves training-loss convergence, while real-world success and attention-map analyses suggest it also supports more reliable manipulation by focusing policies on contact-relevant information.

C.1 Impact of Pretraining on Training Loss Convergence

  • Across Test Tube Collection and Pencil Insertion, pretrained policies maintain lower training loss from epochs 5–60, indicating faster convergence and greater training stability.The analysis also considers Test Tube Collection success rates under low-demonstration and low-epoch regimes.Source: §C.1 · p. 17
  • Even when pretrained and non-pretrained policies converge to similar losses, pretrained policies achieve higher success rates in real-world rollouts.Training loss measures open-loop behavior, whereas physical rollouts evaluate closed-loop execution.Source: §C.1 · p. 17
  • Real-world object-position, environmental-dynamics, and contact-condition variations help explain why training loss alone does not capture rollout performance.These unpredictable variations affect downstream physical execution.Source: §C.1 · p. 17
  • Pretrained policies rely on tactile cues that generalize across scenarios, producing more reliable and consistent behavior, particularly under out-of-distribution conditions.This robustness is offered as an explanation for their stronger real-world performance.Source: §C.1 · p. 18

C.2 Policy Self-Attention Maps Before vs. After Downstream Fine-Tuning

  • In Test Tube Collection, pretrained policies attend to gripper-object contact, while non-pretrained policies often focus on irrelevant backgrounds such as table edges or nearby objects.The attention visual shows the same contrast between task-relevant contact regions and background features.Source: §C.2 · p. 18, Figure 10 · p. 18
  • Pretraining preserves spatial priors critical for manipulation; misplaced attention can degrade performance in cluttered or visually diverse environments.The attention patterns offer insight into why pretrained policies learn more efficiently and generalize better across task variations.Source: §C.2 · p. 18

D Baseline Failure Analysis

Across four fine-grained tasks, baseline failures expose the limits of vision-only and non-cross-attention policies, underscoring the importance of integrated visuo-tactile feedback. Their errors vary by task, from phase and alignment uncertainty to inappropriate force and orientation estimates.

Test Tube Collection

  • In Test Tube Collection, vision-only policies hesitate between reorientation and insertion, while policies without cross-attention may maintain excessive contact or fail to visually align the tube with narrow rack slots.The latter may continue reorienting without progress, even though they transition between phases more decisively.Source: D Baseline Failure Analysis · pp. 18–19

Fluid Transfer

  • In Fluid Transfer, vision-only policies may squeeze too hard or too long, overfill, and fail to release fluid fully; they may also skip expelling fluid when it is not clearly visible.These failures reflect reliance on ambiguous visual cues and sensitivity to subtle environmental variations.Source: D Baseline Failure Analysis · p. 19

Whiteboard Erasing

  • In Whiteboard Erasing, incomplete visual feedback about eraser contact can leave marks, while policies without cross-attention may press too hard, trigger safety stops, and terminate the task.Source: D Baseline Failure Analysis · p. 19

Pencil Insertion

  • In Pencil Insertion, occlusion or a low, slightly inclined pencil can impair vision-only orientation estimates, causing the policy to misjudge its orientation and fail insertion.Source: D Baseline Failure Analysis · p. 19

E Training Hyperparameters for Pretraining · F Detailed Hyperparameters for Downstream Task Training · G Extended Task Descriptions

The appendices provide pretraining and policy-learning configuration details, then describe four tasks through their motions and step-level evaluation metrics to show tactile-sensor capabilities.

E Training Hyperparameters for Pretraining

  • Table 4 reports pretraining hyperparameters and configuration, defining tactile masking ratio as random patch masking during pretraining.Source: Table 4 · p. 20

F Detailed Hyperparameters for Downstream Task Training

  • Table 5 presents policy-learning hyperparameters, separating observation settings from optimization parameters.Source: Table 5 · p. 20

G Extended Task Descriptions

  • The extended task descriptions cover four tasks, detailing motions and evaluation metrics for each step to demonstrate tactile-sensor capabilities.Source: G Extended Task Descriptions · p. 20

G.1 Details for the Test Tube Collection Task … G.4 Details for the Pencil Insertion Task

These task descriptions specify how randomized starting conditions create manipulation uncertainty and how tactile feedback supports precise contact, force control, and alignment. They define success through task-specific completion criteria across test tube collection, fluid transfer, whiteboard erasing, and pencil insertion.

G.1 Details for the Test Tube Collection Task

  • The robot grasps a partially exposed test tube amid randomized pose, orientation, clutter, and cork color, with a thin flexible tactile sensor supporting constrained access.The tube is embedded among objects in a box, requiring precise spatial reasoning and tactile feedback.Source: §G.1 · p. 20
  • After grasping, it uses the rack edge to rotate the transparent tube about 70 degrees, then aligns and fully inserts it into a tight slot using tactile contact cues.A single correct reorientation is required; insertion succeeds when the tube is fully and securely seated.Source: §G.1 · pp. 20–21

G.2 Details for the Fluid Transfer Task

  • The robot gently squeezes the pipette bulb to draw an appropriate amount of liquid, using tactile localization and force feedback to manage occlusion, volume, and spill risk while lifting.The initial setup varies pipette and beaker positions, water amount, and water appearance; success requires lifting without spilling.Source: §G.2 · p. 21
  • During transfer, it maintains bulb pressure and uses vision for coarse positioning, then applies tactile force signals to verify full liquid release before lifting cleanly.Success requires reaching above the second beaker without spilling and transferring all liquid without residual water or unintended spills.Source: §G.2 · p. 21

G.3 Details for the Whiteboard Erasing Task

  • For whiteboard erasing, the robot rotates the brush 90 degrees for a vertical T stroke, then reorients it for a horizontal ac stroke, using tactile cues to confirm contact and pressure.The setup randomizes robot pose, board height and position, and brush location; both strokes require controlled contact without slipping.Source: §G.3 · pp. 21–22

G.4 Details for the Pencil Insertion Task

  • The robot uses the sharpener surface to tactually align a partly occluded pencil parallel to the gripper, then detects hole contact and inserts without slipping, jamming, or realignment.Pencil pose, sharpener placement and height, hole size and alignment, and robot pose vary across episodes.Source: §G.4 · p. 22
Loading 2507.15062v2…