PaperlayerView exact revision on arXiv

Source-linked AI summary

Touch and Go: Learning from Human-Collected Vision and Touch

Fengyu Yang, Chenyang Ma, Jiacheng Zhang, Jing Zhu, Wenzhen Yuan, Andrew Owens

arXiv:2211.12498v2cs.CV

TL;DR

High-quality visuo-tactile data has been difficult to collect, despite the importance of linking touch and sight to understanding objects. Touch and Go pairs human-collected tactile recordings with video from natural environments. The dataset supports several learning tasks, including tactile-driven stylization and improved future-touch prediction using visual input.

  • Problem

    High-quality visuo-tactile data has been difficult to collect, limiting the data available for learning how touch and sight relate to object properties.

    Source: §1 · p. 2

  • Method

    Touch and Go pairs tactile recordings made by human collectors probing natural objects with simultaneously recorded video.

    Source: §1 · p. 2, §3 · p. 3, §3.1 · p. 4

  • Results

    The dataset supports self-supervised feature learning, tactile-driven image stylization, and future tactile prediction, with visual input improving predictions over touch alone.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors present Touch and Go as a diverse real-world dataset demonstrated on applications involving robotic manipulation, material understanding, and image synthesis.

    Source: §6 · p. 10

    Collection was concentrated near the University of Michigan campus, so the data may not generalize to all spaces; human collection may also introduce sampling and population biases.

    Source: §6 · p. 10

Abstract

from arXiv · show

The ability to associate touch with sight is essential for tasks that require physically interacting with objects in the world. We propose a dataset with paired visual and tactile data called Touch and Go, in which human data collectors probe objects in natural environments using tactile sensors, while simultaneously recording egocentric video. In contrast to previous efforts, which have largely been confined to lab settings or simulated environments, our dataset spans a large number of "in the wild" objects and scenes. To demonstrate our dataset's effectiveness, we successfully apply it to a variety of tasks: 1) self-supervised visuo-tactile feature learning, 2) tactile-driven image stylization, i.e., making the visual appearance of an object more consistent with a given tactile signal, and 3) predicting future frames of a tactile signal from visuo-tactile inputs.

1 Introduction

Touch–sight learning matters for understanding object properties, but collecting high-quality visuo-tactile data has been difficult. Touch and Go uses human-collected recordings from natural settings and demonstrates them across several learning tasks.

1 Introduction

  • Correlating touch with sight is presented as essential to understanding physical properties of surrounding objects, yet high-quality visuo-tactile data has been difficult to collect.Source: §1 · p. 2
  • Prior robot-collected data is often lab-bound and limited to objects robots can manipulate, while simulation struggles with real objects’ deformation and microgeometry.These constraints limit the range of visuo-tactile properties represented in those approaches.Source: §1 · p. 2
  • Human collection in natural environments produces a more diverse dataset than prior efforts, supporting self-supervised learning and tasks involving material properties.Collectors probe objects while recording video across indoor and outdoor environments.Source: §1 · p. 2
  • The authors apply Touch and Go to self-supervised feature learning, tactile-driven image stylization, and future tactile-frame prediction.Visual input improves future-touch predictions over touch alone.Source: §1 · p. 2

2 Related Work

Prior work spans robotic, simulated, and human-collected visuo-tactile data, as well as multimodal feature learning and image prediction. Touch and Go adapts existing multimodal methods to natural touch recordings and tactile-conditioned synthesis.

Robotic vision and touch; Simulated vision and touch

  • Robot-collected datasets are often confined to specific environments and small sets of robot-interactable objects; simulation-based work also has limits for deformable objects and complex microgeometry.Simulated datasets described here include rigid-object settings and CAD-derived data with only rigid deformation.Source: §2 · pp. 2–3

Human-collected multimodal data

  • Human-collected precedents include grasping and tactile datasets, while Touch and Go emphasizes rich multimodal data, indoor and outdoor scenes, GelSight, and cross-modal prediction.Related examples include human-collected grasping, audio-visual probing, and a lab dataset of 26 objects.Source: §2 · p. 3

Multimodal feature learning

  • The authors adapt Tian et al.’s contrastive model to learn visuo-tactile features from correlated images and touch.Source: §2 · p. 3

Multimodal image prediction; Cross-modal image stylization

  • They adapt audio-driven stylization to tactile inputs and contrast natural-object future-touch prediction with prior CNC-controlled prediction.The paper reports that visual information improves future tactile predictions.Source: §2 · p. 3

3 The Touch and Go Dataset

Touch and Go pairs human-collected tactile recordings with video of objects in natural locations. The dataset includes annotations and spans varied scenes and materials, with more object instances than prior datasets described in the paper.

3 The Touch and Go Dataset

  • The dataset pairs human-collected tactile data from objects in their natural locations with video, and includes material labels and within-press frames.Source: §3 · p. 3

3.1 Collecting a natural visuo-tactile dataset

  • Two collectors record each interaction: one presses the tactile sensor while the other records approximately egocentric video as they move among objects.The recorded objects vary in material properties, geometry, and semantics.Source: §3.1 · p. 3, Figure 2 · p. 4
  • Collection spans indoor and outdoor scenes and includes rigid and deformable objects; GelSight recordings and visual video are captured concurrently at about 27 Hz.The sensor’s gel deformation changes reflected illumination, from which surface orientation, depth, and shear can be estimated.Source: §3.1 · p. 4

3.2 Annotating the dataset

  • The dataset provides material-category labels and frames within each press; a press detector reaches 97% accuracy on 2k hand-labeled test frames.Five online workers label each image, and workers review and correct detected press frames.Source: §3.2 · pp. 4–5

3.3 Dataset analysis

  • The dataset contains approximately 13.9k detected touches and 3,971 estimated object instances, with 52.2% indoor and 47.8% outdoor scenes.Some materials occur only indoors and others only outdoors; prior robot-collected datasets are confined to one lab space, while the qualitative comparison notes limits of robotic and synthetic data.Source: §3.3 · p. 5
  • Touch and Go contains approximately 4× as many object instances as ObjectFolder 2.0 and 11× as many as Burka et al..It also contains 1.15× more touches than VisGel and 1.5× more than The Feeling of Success.Source: §3.3 · p. 5, Table 1 · p. 5

4 Applications

Applications span representation learning, image synthesis, and future prediction, probing several uses of multimodal data.

4.1 Multimodal self-supervised representation learning

  • CMC learns tactile representations by matching paired visual–tactile examples against mismatches in both visual-to-tactile and tactile-to-visual directions.The downstream questions are whether these features convey material properties and help robotic tasks using separately collected data.Source: §4.1 · pp. 5–6

4.2 Tactile-driven image stylization

  • Tactile-driven stylization conditions image translation on a GelSight input, preserving image structure while shifting texture toward what co-occurs with that touch.Source: §4.2 · p. 6

4.3 Multimodal future touch prediction

  • Future-touch prediction tests whether vision improves estimates of upcoming tactile signals; an adapted video model predicts autoregressively from paired visual–tactile history using L1 and perceptual losses.Source: §4.3 · p. 7

5 Experiments

The experiments evaluate self-supervised tactile features on grasping and material tasks, test tactile-conditioned image stylization, and compare tactile-only with visuo-tactile future-touch prediction. Across these tasks, the reported results favor the learned features, CUT-based stylization, and multimodal prediction, respectively.

5.1 Self-supervised feature learning

  • Feature evaluation uses a linear classifier for grasp success and material labels, with video-level splits to keep objects separate across material-training and test sets.The grasping split contains 68 training, 16 validation, and 21 test objects; material evaluation separates unlabeled, labeled-training, and labeled-test touches.Source: §5.1 · p. 7
  • Learned tactile features significantly outperform supervised ImageNet and visual CMC features on robotic grasping and material understanding.The authors attribute the improvement to learning from real tactile data.Source: §5.1 · p. 8

5.2 Tactile-driven image stylization

  • CUT-based stylization has higher CMC similarity than CycleGAN and consistently outperforms it on material-classification consistency.Table 3 reports quantitative stylization results, with the original image serving as the baseline.Source: Table 3 · p. 8, §5.2 · pp. 8–9
  • Stylized images reproduce tactile roughness or smoothness and can combine material cues, even without the corresponding tactile example image.The qualitative results also include failure cases.Source: Figure 4 · p. 9, §5.2 · p. 9

5.3 Multimodal video prediction

  • Visuo-tactile models improve future-touch prediction across metrics, baselines, and both tested settings, with a larger gap at longer horizons.The experiment uses three context frames and tests predictions skipping three or five frames.Source: §5.3 · pp. 9–10, Table 4 · p. 10

6 Discussion

Touch and Go is a human-collected visuo-tactile dataset from real-world objects, applied to tasks spanning robotic manipulation, material understanding, and image synthesis. The authors note geographic and human-collector limitations, and provide a webpage for the dataset.

6 Discussion

  • The dataset uses real-world objects and is reported as more diverse than prior datasets, with applications in manipulation, material understanding, and image synthesis.The authors frame it as a step toward human-collected multimodal data beyond robotics-centric domains.Source: §6 · p. 10
  • Collection near the University of Michigan campus may limit generalization across spaces, and human probing choices, force, and demographics may introduce bias.The authors note that collectors may probe unrepresentative object parts and may not represent the general population.Source: §6 · p. 10

A Project webpage

B Dataset file structure

The dataset package provides videos, timestamps, extracted frames, and material-label metadata. Its supplementary description also covers the egocentric recording setup, category labels, and dataset access.

B Dataset file structure

  • The dataset is currently available through the project webpage, with long-term maintenance planned on University of Michigan EECS servers after acceptance.Source: B Dataset file structure · p. 16
  • The package includes material-label files, extracted frame folders, and qualitative video examples available on the project page.The frame folders are generated by running extract_frame.py.Source: B Dataset file structure · p. 16
  • Each raw-video folder contains the RGB interaction video, GelSight tactile video, and a recording-time array for each video.The RGB video records human probing; the GelSight video records tactile signals.Source: B Dataset file structure · p. 16

C Egocentric recording setup

  • Two people typically collect data together: one presses the sensor while the other records approximately egocentric video.A one-person recording alternative is also described, with another person holding the computer.Source: C Egocentric recording setup · p. 16

D Category list

E Implementation details for self-supervised learning

The appendix gives training settings for self-supervised learning and tactile-driven image stylization. The stylization method combines image and tactile encoders and uses a contrastive loss to preserve source-image structure while changing texture.

E Implementation details for self-supervised learning

  • The self-supervised CMC model is trained for 240 epochs with batch size 128, while downstream linear classifiers are trained for 60 epochs with batch size 256.CMC uses SGD with learning rate 0.03; linear probing uses fixed pretrained backbones and global average pooling.Source: E Implementation details for… · p. 16
  • Figure 6 depicts two humans collecting data in the wild.Source: Figure 6 · p. 17
  • Table 5 provides statistics for the dataset’s material categories, without category counts specified in the supplied caption.Source: Table 5 · p. 17

F Details for Tactile-driven image stylization

  • The stylization generator comprises an image encoder, a tactile encoder, and a decoder that produces an output image from the image and tactile inputs.The architecture also includes a tactile-visual texture discriminator and a patch-wise structure discriminator.Source: F Details for Tactile-driven image… · p. 17
  • A contrastive NCE loss encourages corresponding patches in the source and generated images to match, helping preserve source structure during restyling.The method selects a generated-image query patch, one positive source-image patch, and negative source-image patches.Source: F Details for Tactile-driven image… · pp. 17–18
  • The NCE objective uses feature stacks from multiple image-encoder layers, with a multilayer perceptron mapping those features into vectors.The implementation extracts features from five layers, including the input, downsampling layers, and residual blocks.Source: F Details for Tactile-driven image… · p. 18

G More results for tactile-drive image stylization

The appendix shows tactile-driven stylization examples and describes a multimodal model for future tactile-frame prediction. The prediction setup combines visual and tactile inputs and evaluates generated frames with three image metrics.

G More results for tactile-drive image stylization

  • Figure 7 displays input images beside tactile-conditioned stylizations, with the corresponding tactile example and its reference image at the right.The reference image is not used by the model.Source: Figure 7 · p. 19

H Details for multimodal future touch prediction

Q1. For what purpose was the dataset created?

Touch and Go was created to train multimodal systems to associate objects’ visual appearance with tactile signals, using in-the-wild recordings from indoor and outdoor scenes.

Q1. For what purpose was the dataset created?

Q5. What do the instances that comprise the dataset represent (e.g., documents, photos, people, countries)?

Q6. How many instances are there in total (of each type, if appropriate)?

Q9. Is there a label or target associated with each instance?

Q11. Are relationships between individual instances made explicit (e.g., users’ movie ratings, social network links)?

Q12. Are there recommended data splits (e.g., training, development/validation, testing)?

Q13. Are there any errors, sources of noise, or redundancies in the dataset?

Q18. How was the data associated with each instance acquired?

  • Two authors collected data while walking through environments: one pressed a tactile sensor onto objects, while the other recorded video; the dataset uses an RGB camera and GelSight sensor.Source: Q18. How was the data associated with… · p. 21

Q31. What (other) tasks could the dataset be used for?

The authors identify possible uses in visual-tactile tasks, while cautioning that the dataset’s collection scope may limit generalization and its intended domain is visuo-tactile learning.

Q31. What (other) tasks could the dataset be used for?

Q33. Are there any tasks for which the dataset should not be used?

Q34. Any other comments?

Q41. Who will be supporting/hosting/maintaining the dataset?

  • The dataset was hosted on a public Google Drive directory, with a University of Michigan web-server mirror planned for indefinite availability.Source: Q41. Who will be… · p. 23

Q43. Is there an erratum?

Q46. Will older versions of the dataset continue to be supported/hosted/maintained?

Loading 2211.12498v2…