PaperlayerView exact revision on arXiv

Source-linked AI summary

TouchScale: 500 Hours of Human Vision and Touch for Visual-Tactile Learning

Dayou Li, Hao Wang, Qianqian Yang, Zihao Zhu, Haoquan Fang, Ziyao Zeng, Yan Han, Zihan Wang, Yan Wang, Baoru Huang, Dilin Wang, Kenji Shimada, Yiyue Luo, Manling Li, Teresa Lv, Mustafa Mukadam, Rakesh Ranjan, Ruohan Zhang, Qi He, Changliu Liu, Xu Chen, Marco Pavone, Bangya Liu, Jiachen Li, Masayoshi Tomizuka, Zhiwen Fan

arXiv:2610.10288v1cs.CV

TL;DR

Egocentric video captures hand motion but misses contact and pressure, while existing visual–tactile datasets remain comparatively small and can confound scale with sensing differences. TouchScale pairs 500 hours of human interaction with synchronized visual and tactile recordings from a unified wearable setup. Its evaluations report gains in tactile prediction and robot manipulation, with positive scaling trends under fixed sensing and collection protocols.

  • Problem

    Egocentric video leaves contact and pressure unrecorded, and existing visual–tactile datasets are smaller than human video, making data-scale effects difficult to isolate when sensor and collection procedures differ.

    Source: §1 · pp. 1–2

  • Method

    TouchScale is a 500-hour dataset pairing diverse human interaction with synchronized egocentric RGB-D, wrist RGB, and dense full-hand bimanual tactile measurements collected through a unified wearable pipeline.

    Source: §1 · pp. 2–3

  • Results

    Increasing data from one tenth to all of TouchScale raises zero-shot cIoU on EgoTactile from 0.311 to 0.383, and TouchScale mid-training raises average real-world success across four contact-rich tasks from 22.5% to 57.5%.

    Source: §1 · p. 3

  • Takeaways & Limitations

    With the sensor and collection protocol fixed, zero-shot tactile prediction and real-world manipulation success show overall upward trends as more TouchScale data is used.

    Source: §1 · p. 3

    The robot evaluation is limited to one platform and four manipulation tasks, and TouchScale does not include action labels.

    Source: §5 · p. 10

Abstract

from arXiv · show

Large-scale egocentric human interaction data is becoming an important source of physical supervision for embodied learning, yet video alone leaves the contact and pressure that characterize physical interaction unrecorded. Recent visual-tactile datasets provide this missing supervision, but their synchronized tactile data remain far smaller in volume than human video. Moreover, the largest resources often merge recordings from different sensors or annotation procedures, which makes the effect of data scale difficult to isolate. We therefore introduce TouchScale, a 500-hour dataset of contact-rich human interaction recorded with a single unified wearable setup. Its approximately 2K predefined task descriptions span everyday activities and structured manipulation, and each recording temporally aligns egocentric RGB-D video with wrist RGB video and dense full-hand bimanual tactile measurements. Compared with prior tactile data, training on the full TouchScale raises zero-shot contact IoU on data from an unseen tactile sensor from 0.134 to 0.383. Pretraining a visual encoder on TouchScale also yields the highest action recognition accuracy on three benchmarks among the compared visual-tactile datasets. Used for visual-tactile mid-training of a robot policy, TouchScale improves the average real-world success rate across four contact-rich manipulation tasks from 22.5% to 57.5%. With the sensor and collection protocol held fixed, both zero-shot tactile prediction and robot success show an overall upward trend as more TouchScale data is used. These results suggest that human visual-tactile data collected at scale with consistent sensing benefits both perception and robot manipulation. We will publicly release TouchScale, including all synchronized visual-tactile recordings and reconstructed object models, to support future research on scalable visual-tactile learning.

1 Introduction

TouchScale addresses the limited scale and inconsistent sensing of existing visual–tactile data with a 500-hour dataset collected through a unified wearable pipeline. Its evaluations report improvements in tactile prediction and robot manipulation, with performance trending upward as more data is used.

1 Introduction

  • Existing egocentric video captures hand motion but not contact or pressure, while most visual–tactile datasets contain fewer than 30 hours and pooling datasets confounds scale with sensing and protocol differences.The paper asks whether benefits extend to visual–tactile learning and manipulation when data are collected at hundreds-of-hours scale under a unified pipeline.Source: §1 · pp. 1–2
  • TouchScale pairs 500 hours of diverse human interaction with synchronized egocentric RGB-D, wrist RGB, and dense full-hand bimanual tactile measurements from a consistent wearable pipeline.The collection spans approximately 2K task descriptions, more than 1.5K objects, nine collection settings, and over 800 scene configurations.Source: §1 · pp. 2–3
  • On a size-matched 16-hour comparison, TouchScale reaches 0.181 cIoU on EgoTactile versus 0.134 for the EgoTouch training split, and its pretraining yields higher action-recognition accuracy across three benchmarks.The action-recognition comparison holds under both linear probing and end-to-end fine-tuning.Source: §1 · p. 2
  • Increasing TouchScale data from one tenth to the full dataset raises zero-shot cIoU on EgoTactile from 0.311 to 0.383, while expanding mid-training data from one fifth to full lifts robot success from 30.0% to 57.5%.Across four contact-rich tasks, TouchScale mid-training raises average real-world success from 22.5% to 57.5%.Source: §1 · p. 3
  • The authors will publicly release TouchScale recordings and reconstructed object models, and report that both tactile prediction and robot success trend upward with more data under fixed sensing and collection protocols.The dataset is intended to support scalable visual–tactile learning.Source: §1 · p. 3

2 Related Work

Prior egocentric visual–tactile datasets add contact and pressure supervision to human video, while work on robot learning explores tactile transfer and mid-training. TouchScale combines consistent human visual–tactile data with vision-to-tactile mid-training without human action labels.

2.1 Egocentric Visual–Tactile Datasets

  • Existing egocentric visual–tactile datasets study pressure prediction, tactile representations, retrieval, force supervision, and related tasks, while TouchScale contributes 500 hours collected through a consistent sensing pipeline.Prior resources include datasets focused on hand–surface contact, grasping, bimanual manipulation, and glove recordings paired with generated bare-hand video.Source: §2.1 · pp. 3–4

2.2 Human Visual–Tactile Data for Robot Learning

  • Prior robot-learning work uses human video, portable tactile grippers, shared human–robot tactile representations, or tactile policy mid-training; TouchScale uses human recordings for vision-to-tactile mid-training without human action labels or retargeting.The paper evaluates whether this supervision improves real-robot task success after robot-specific training.Source: §2.2 · p. 4

3 TouchScale Dataset

TouchScale is a 500-hour human visual–tactile dataset collected with a unified wearable setup. Its task and scene coverage, varied interactions, and quality-control process support a broad collection of synchronized recordings.

3 TouchScale Dataset

  • TouchScale records synchronized egocentric RGB-D, bimanual wrist RGB, and bimanual tactile measurements using one wearable setup.Figure 2 depicts the capture hardware and aligned observations across stages of an example interaction.Source: §3 · p. 4, §3.1 · p. 4, Figure 2 · p. 5

3.1 Data Collection Setup

  • The setup combines an egocentric RGB-D camera, two wrist-mounted RGB cameras, and bimanual tactile gloves; the head camera captures the overall interaction while wrist cameras provide close hand–object views.The cameras record at 30 Hz; the egocentric sensor also captures metric depth geometry.Source: §3.1 · p. 4

3.2 Data Collection and Task Design

  • Approximately 2K natural-language task descriptions span nine collection settings and 800+ scene configurations, covering everyday activities and contact-rich single-hand and bimanual manipulation.Instructions specify an action, object or objects, and intended interaction or target state, with further execution details when needed.Source: §3.2 · p. 5, Figure 3 · p. 6
  • Variation across 1.5K+ objects, object configurations, and approximately 20 participants produces different hand–object poses and tactile contact patterns across repeated tasks.The collection also provides reconstructed 3D objects from object-centric RGB captures.Source: §3.2 · p. 5
  • An automated quality-control pipeline crosschecks tactile recordings against wrist video, rejecting clear sensor failures while sending ambiguous individual-finger responses for manual review.It separately screens contact-free periods for persistent tactile activity and checks whether apparent activity reflects noise or unobserved contact.Source: §3.2 · p. 6

4 Experiments

The experiments test TouchScale across zero-shot tactile prediction, visual action recognition, and action-free robot-policy mid-training, including how results change with data scale. TouchScale improves performance across these settings, with gains also observed as more data is used.

4.1 Tactile Prediction

  • Zero-shot cIoU on EgoTactile rises from 0.134 with EgoTouch training to 0.181 with TouchScale training.TouchScale also improves vIoU from 0.185 to 0.243, CoP localization error from 0.444 to 0.416, and regional pressure magnitude error from 0.184 to 0.148.Source: §4.1 · p. 7, Table 2 · p. 7
  • As TouchScale training data grows from 10% to 100%, EgoTactile cIoU rises from 0.311 to 0.383 and the other three reported metrics also improve.The model and evaluation protocol are held fixed.Source: §4.1 · p. 7, Figure 4 · p. 7

4.2 Visual Representation Learning

  • TouchScale achieves the highest accuracy on all three action-recognition benchmarks under linear probing and end-to-end fine-tuning.Average accuracy is 24.45% for linear probing and 49.62% for fine-tuning.Source: Table 3 · p. 8, §4.2 · p. 8

4.3 Transfer to Robotic Manipulation

  • 57.5% average success across four real-world tasks with TouchScale mid-training, versus 22.5% without it.The comparison covers material-dependent sorting, bottle-cap removal, test-tube transfer, and whiteboard wiping.Source: §4.3 · p. 8, Table 4 · p. 9
  • Average robot success rises from 22.5% without mid-training to 57.5% with the full TouchScale dataset, with an overall positive trend across data scales.The intermediate results are 30.0%, 32.5%, 50.0%, and 50.0% at 20%, 40%, 60%, and 80% of TouchScale, respectively.Source: §4.3 · p. 9, Figure 6 · p. 10
  • Future-tactile prediction accuracy on unseen robot data improves with TouchScale mid-training, though the increase from 20% to 100% of the dataset is modest.With the full dataset, accuracy is 3.31%, compared with 0.18% without TouchScale mid-training.Source: §4.3 · pp. 9–10

5 Conclusion and Limitations

TouchScale is presented as a consistent-sensing, 500-hour dataset whose visual-tactile supervision benefits perception and robot manipulation. The robot evaluation is limited in scope, and the role of future-tactile prediction in control remains unresolved.

5 Conclusion and Limitations

  • TouchScale is a 500-hour human visual-tactile dataset collected with a consistent sensing and synchronization pipeline.The paper reports benefits for zero-shot tactile prediction, action recognition, and real-world robot manipulation.Source: §5 · p. 10
  • Robot evaluation covers one platform and four manipulation tasks; broader evaluation across embodiments and contact-rich tasks remains future work.Source: §5 · p. 10
  • Future-tactile and action prediction errors show only a weak association, leaving the role of future-tactile prediction in robot control open.Source: §5 · p. 10

A Tactile Prediction Evaluation Protocol

The protocol maps sensor-specific tactile layouts into a shared anatomical space for cross-dataset evaluation while preserving native layouts during training. It evaluates contact and pressure at the region level using four sequence-averaged metrics.

A.1 Cross-sensor anatomical mapping

  • The shared evaluation space maps three sensor layouts to 12 anatomical regions without resizing native grids, which remain unchanged during training.Assignments follow taxel anatomy, with middle and base finger taxels merged when separately represented; the same mapping is applied to predictions and ground truth.Source: §A.1 · p. 14

A.2 Region-level tactile representation

  • Regional contact uses normalized taxel pressure thresholds and a minimum active-taxel fraction, reducing sensitivity to differences in sensor density.The protocol also computes regional mean pressure and center of pressure from the mapped tactile measurements.Source: §A.2 · pp. 14–15

A.3 Evaluation metrics

  • Four sequence-level metrics are macro-averaged across test sequences: cIoU and vIoU reward higher values, while CoP Loc. and Press. Mag. reward lower values.The metrics assess contact overlap, pressure overlap, pressure localization, and pressure magnitude; only regions with applicable measurements are included in the relevant averages.Source: §A.3 · pp. 15–16

A.4 Qualitative Results of Tactile Prediction

TouchScale-trained predictions reproduce ground-truth tactile contact regions and pressure distributions across varied interactions. The examples include both sparse fingertip contacts and broader multi-region patterns on both hands.

A.4 Qualitative Results of Tactile Prediction

  • Across diverse contact-rich interactions, predictions recover dominant contact regions on both hands and closely follow ground-truth pressure distributions.The qualitative examples span different objects and interaction patterns.Source: §A.4 · p. 16, Figure 8 · p. 17
  • Predictions capture both sparse fingertip contacts and broader multi-region contact patterns.These examples illustrate varied contact configurations rather than a single interaction pattern.Source: §A.4 · p. 16
  • Figure 8 presents representative tactile predictions alongside ground truth for varied human interactions.Source: §A.4 · p. 16, Figure 8 · p. 17

B Tactile Glove Details

TouchScale records full-hand normal pressure with flexible Tachin gloves covering the fingers and palm. The wearable setup supports wireless or wired acquisition and more than 10 hours of battery operation.

B Tactile Glove Details

  • Each glove has 880 sensing taxels across five fingers and the palm, with below-2-mm spatial resolution and normal-pressure sensing.Each sensing unit measures 1.5 × 1.5 mm; the measurement range is 0.2–30 N/cm2 with 0.05 N/cm2 resolution.Source: B Tactile Glove Details · p. 17
  • The wearable system supports wireless acquisition at 50 Hz over WiFi or wired acquisition at 100 Hz through USB Type-C.The battery supports more than 10 hours of continuous operation.Source: B Tactile Glove Details · p. 17

C Tactile Data Quality Control

Quality control combines stream-integrity checks with independent hand-level review of missing signals and tactile noise. The more severe left- or right-hand result determines each episode’s final Pass, Review, or Reject label.

C Tactile Data Quality Control

  • Episodes receive batch-level checks for required streams, timestamps, tactile recordings, and metadata, plus integrity statistics and synchronized visualizations.Integrity statistics are stored for inspection but do not directly determine data quality; final episode labels follow the more severe hand-level result.Source: C Tactile Data Quality Control · p. 18
  • A fully silent hand is rejected only when wrist-video analysis confirms interaction with an object.The visual analysis identifies grasp and release intervals, involved or uncertain fingers, and visually free intervals without tactile input.Source: C Tactile Data Quality Control · p. 18
  • Missing-signal rejection requires at least two visually confirmed contact fingers to remain silent during an interval and throughout the episode.A persistently unresponsive finger without that multi-finger evidence leads to Review; a finger that responds elsewhere is treated as uncertain.Source: C Tactile Data Quality Control · p. 18
  • Tactile noise candidates are responses above threshold during visually contact-free intervals, but noise does not independently set the episode label.The strongest five candidates are verified using synchronized tactile and wrist-view clips; confirmed contact-free responses are recorded for quality analysis.Source: C Tactile Data Quality Control · pp. 18–19

D Future-Tactile Prediction Accuracy

This section evaluates whether Stage-1 future-tactile prediction transfers to robot observations and how its accuracy changes with TouchScale data scale. TouchScale training improves the primary Train Top-10 metric over the checkpoint without human mid-training, with a modest additional gain from more data.

D Future-Tactile Prediction Accuracy

  • Stage-1 predicts latent tactile tokens for future tactile change, and evaluation ranks each prediction against target representations by cosine similarity.A correct prediction should rank its corresponding future tactile target above targets from other samples.Source: D Future-Tactile Prediction Accuracy · p. 19
  • Top-K accuracy measures whether the matching future tactile target ranks among the K highest candidates; Top-10 is the primary metric.The section also reports MRR and Pool NCE; higher MRR and lower Pool NCE indicate better ranking and separation, respectively.Source: D Future-Tactile Prediction Accuracy · p. 19
  • Evaluation uses 8,539 robot-observation samples from trajectories not used during Stage-1 mid-training, with Train Top-10 as the primary metric.The pooled set combines 55 Train episodes and 13 Val episodes.Source: D Future-Tactile Prediction Accuracy · p. 19
  • 2.87–3.31% Train Top-10 with TouchScale Stage-1 training, versus 0.18% without it.Increasing TouchScale data from 20% to 100% raises Train Top-10 from 2.87% to 3.31%.Source: D Future-Tactile Prediction Accuracy · p. 20, Table 5 · p. 20

E.1 Qualitative Results

The qualitative results show successful TouchScale mid-training rollouts across four contact-rich tasks and contrast them with baseline Whiteboard Wipe failures. The examples trace interaction from initial contact through task completion, or failure to complete.

E.1 Qualitative Results

  • TouchScale mid-training rollouts show successful completion across Bottle-Cap Removal, Soft/Hard Sorting, Test-Tube Transfer, and Whiteboard Wipe.The examples span object compliance, small-object handling, precise transfer, and continuous contact during wiping.Source: §E.1 · p. 20, Figure 11 · p. 21
  • Baseline N0-VTLA Whiteboard Wipe rollouts without TouchScale mid-training initiate contact but fail to complete the task.The figure presents representative failure progressions.Source: §E.1 · p. 20, Figure 12 · p. 21
  • The successful examples follow each rollout from initial interaction to task completion, while the baseline examples show unsuccessful wiping progressions.The successful rollouts cover all four manipulation tasks; the failure examples are from Whiteboard Wipe.Source: §E.1 · p. 20, Figure 11 · p. 21, Figure 12 · p. 21

E.2 Implementation Details

TouchScale mid-training adapts the tactile pathway while keeping the vision–language–action backbone frozen, using future-tactile prediction and reconstruction objectives. The section also describes timestamp-based stream alignment, controlled robot post-training, and multi-view object reconstruction.

E.2 Implementation Details

  • TouchScale mid-training initializes from N0-VTLA, freezes its vision–language–action backbone, and updates the tactile projection, predictor, and reconstruction head without human action labels.The predictor uses five latent tactile tokens to represent expected tactile change over a 50-frame future horizon.Source: §E.2 · p. 22
  • The Stage-1 objective combines symmetric InfoNCE on predicted and target tactile representations with an ℓ1 reconstruction loss weighted by λrec = 0.5.The reconstruction head maps predicted tactile tokens to an 8 × 8 representation of future tactile change.Source: §E.2 · p. 22
  • Training converts three RGB and two tactile streams to a common 30 Hz timeline using timestamp-based nearest-neighbor association, without changing the released recordings.Fixed tactile normalization statistics estimated from the training split are applied throughout mid-training.Source: §E.2 · p. 22
  • The TouchScale and baseline policies use the same 50 robot demonstrations per task and post-training protocol, differing only in TouchScale human visual–tactile mid-training.Approximately 123.8M parameters are updated during mid-training while 3.71B base-policy parameters remain frozen.Source: §E.2 · p. 22

F 3D Object Reconstruction

  • TouchScale object models are reconstructed from segmented multi-view RGB images selected from short object-centric videos.The reconstructed models are provided with the dataset.Source: F 3D Object Reconstruction · p. 22

Related papers

Loading 2610.10288v1…