PaperlayerView exact revision on arXiv

Source-linked AI summary

TouchAnything: A Dataset and Framework for Bimanual Tactile Estimation from Egocentric Video

Jianyi Zhou, Ziteng Gao, Feiyang Hong, Zirui Liu, Guannan Zhang, Weisheng Dai, Ruichen Zhen, Chuqiao Lyu, Haotian Wu, Yinian Mao, Xushi Wang, Yuxiang Jiang, Wenbo Ding, Shuo Yang

arXiv:2605.13083v1cs.RO

TL;DR

Egocentric datasets rarely include tactile supervision, although tactile signals convey contact, force, and pressure and are costly to collect at scale. EgoTouch pairs multi-view video with tactile and pose data, and TouchAnything flexibly uses egocentric and wrist views. Wrist views improve tactile prediction, including on seen objects.

  • Problem

    Egocentric datasets typically lack direct tactile signals such as contact force and pressure, while collecting high-quality tactile data at scale is expensive and difficult.

    Source: §1 · pp. 1–2

  • Method

    EgoTouch pairs multi-view egocentric and wrist video with synchronized tactile pressure maps and bimanual hand pose, while TouchAnything flexibly predicts touch from available camera views.

    Source: §1 · pp. 2–3

  • Results

    Wrist views improve tactile prediction; on seen objects, Contact IoU rises from 0.4792 to 0.5030.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    EgoTouch provides multi-view tactile supervision for bimanual hand-object interaction across 208 tasks and 1,891 episodes.

    Source: §1 · p. 2

    Training data use tactile gloves, which may introduce glove-specific appearance bias and limit generalization to bare-hand tactile estimation; the benchmark focuses primarily on tactile estimation.

    Source: §6 · p. 10

Abstract

from arXiv · show

Egocentric human video data, which captures rich human-environment interactions and can be collected at scale, has become a key driver of embodied intelligence research. However, existing egocentric datasets typically lack tactile sensing, a critical modality that provides direct cues about contact, force, and pressure in human-object interaction. Without such signals, models struggle to learn physically grounded representations of real-world interaction dynamics. While tactile sensors provide these cues, deploying high-quality tactile hardware at scale remains expensive and cumbersome. This raises a central question: can tactile feedback be inferred directly from visual observations, enabling scalable tactile supervision for egocentric video data and supporting physically grounded embodied learning? To enable research in this direction, we introduce EgoTouch, a large-scale multi-view egocentric dataset with dense tactile supervision for bimanual hand-object interaction. EgoTouch comprises 208 manipulation tasks spanning 1,891 episodes in diverse indoor and outdoor environments, with synchronized multi-view RGB (head-mounted egocentric and dual wrist-mounted cameras), bimanual 3D hand pose, and continuous pressure maps from wearable tactile sensors. Building on EgoTouch, we introduce TouchAnything, a baseline multi-view vision-to-touch prediction framework that uses the egocentric view as the primary input and flexibly leverages available wrist-mounted views at inference time. Experiments show that incorporating wrist-mounted views generally improves tactile prediction over egocentric-only input, achieving up to 5.0% relative improvement in Contact IoU and 6.1% relative improvement in Volumetric IoU. We will publicly release the dataset, code, and benchmark.

1 Introduction

Egocentric vision-to-touch research lacks scalable tactile supervision and is challenged by contact regions occluded in bimanual interactions. EgoTouch provides multi-view tactile data, and TouchAnything uses flexible camera inputs, with wrist views improving prediction.

1 Introduction

  • Egocentric video is scalable, but it lacks direct tactile signals such as contact force and pressure, while high-quality tactile data collection is costly and difficult to scale.This gap motivates inferring tactile signals from visual observations.Source: §1 · pp. 1–2
  • Prior tactile datasets are limited by single-view capture or narrow interaction settings, while bimanual hand-object contact is often occluded from the head-mounted view.These constraints limit study of realistic bimanual vision-to-touch prediction.Source: §1 · p. 2
  • EgoTouch comprises 208 tasks and 1,891 episodes, pairing multi-view video with bimanual pose and dense tactile pressure maps.The dataset spans diverse indoor and outdoor environments.Source: §1 · pp. 2–3
  • On seen objects, adding wrist views raises Contact IoU from 0.4792 to 0.5030 and Volumetric IoU from 0.4311 to 0.4575.On unseen objects, Contact IoU rises from 0.4396 to 0.4496 and Volumetric IoU from 0.3743 to 0.3852.Source: §1 · p. 2
  • TouchAnything is a baseline vision-to-touch model that flexibly accepts single- or multi-camera inputs, with wrist views improving tactile prediction when available.Its benchmark evaluates seen and unseen objects and different camera-view configurations.Source: §1 · p. 3

2 EgoTouch Dataset

EgoTouch is a synchronized, multi-view dataset of bimanual manipulation, combining RGB video, hand pose, tactile pressure, and tracker data across diverse tasks and environments.

2 EgoTouch Dataset

  • EgoTouch contains 20 hours of video at 30 Hz across 1,891 episodes and 208 tasks, covering over 1,000 objects indoors and outdoors.Its synchronized annotations include multi-view RGB, bimanual 3D hand pose, and dense pressure maps for both hands.Source: §2 · p. 3

2.1 Data Collection Setup

  • The wearable capture setup pairs a head-mounted RGB camera and two wrist cameras with motion-capture gloves, pressure-sensing gloves, and head and wrist trackers.All streams are synchronized onto a shared 30 Hz timeline.Source: §2.1 · p. 3, Figure 2 · p. 4

2.2 Data Modalities

  • Each frame includes head and wrist RGB views, 42-joint bimanual 3D hand pose, bilateral tactile pressure maps, tracker poses, and task and scene metadata.The tactile arrays are recorded at 16 × 16 and remapped to 21 × 21 hand-shaped grids; modalities are frame-aligned.Source: §2.2 · pp. 3–4

2.3 Task Taxonomy

  • The 208 tasks are organized into five environment-based categories: Home, Workbench, Office, Retail, and Outdoor.The categories cover household, tool-based, workspace, consumer, and open-environment interactions.Source: §2.3 · pp. 4–5

3 METHOD: Tactile Prediction Framework

TouchAnything maps video from available egocentric and wrist views, together with bimanual hand pose, to bilateral pressure maps. Its shared multi-view fusion and pose-aware decoding are paired with view dropout so inference can accommodate missing wrist views.

3.1 Problem Formulation

  • The task maps a subset of egocentric and wrist-camera video views plus bimanual hand pose to bilateral tactile maps at each timestep.The maps use a canonical 21 × 21 hand-shaped grid.Source: §3.1 · pp. 5–6

3.2 Multi-View Tactile Prediction Framework

  • A shared visual encoder, cross-view attention, and gated fusion integrate camera features, while pose-aware fusion and temporal modeling guide bilateral pressure-map decoding.Joint-level features attend to relevant visual regions to associate tactile signals with hand regions.Source: §3.2 · p. 6

3.3 View Dropout and Training Objective

  • Training randomly drops wrist views while retaining the egocentric view, allowing inference with only egocentric input or with additional available views.The view combinations vary during training without requiring architectural changes at inference.Source: §3.3 · p. 6
  • The training objective combines pixel-wise reconstruction losses with spatial regularization and upweights contact regions where pressure exceeds 0.1.The higher contact-region weight addresses sparse tactile maps and discourages all-zero predictions.Source: §3.3 · p. 6

4 Experiments

Experiments evaluate tactile prediction and contact detection across episode-level dataset splits and seen- and unseen-object test subsets. Wrist views generally improve tactile prediction over egocentric-only input, while ablations examine view dropout and data scaling and qualitative results illustrate occlusion-related differences.

4.1 Experimental Setup

  • The dataset is split 80%/10%/10% by episode, with test data divided into seen- and unseen-object subsets; benchmarks include pressure-map prediction and thresholded contact detection.The split is intended to avoid temporal data leakage and evaluate generalization to novel object instances.Source: §4.1 · p. 7

4.2 Main Results

  • On seen objects, adding both wrist views raises Contact IoU from 0.4792 to 0.5030 and Volumetric IoU from 0.4311 to 0.4575; on unseen objects, the respective changes are 0.4396 to 0.4496 and 0.3743 to 0.3852.Table 2 compares multi-view tactile prediction across five scenarios against the Ego-only baseline.Source: §4.2 · p. 7, Table 2 · p. 8

4.3 Ablation Studies

  • With view dropout, the seen-object relative performance drop between all-view and ego-only inference improves from -27.20% to -5.78%.Without view dropout, performance drops substantially when wrist views are unavailable at inference.Source: §4.3 · p. 8, Table 3 · p. 9
  • Contact IoU and Volumetric IoU improve steadily as training data increases from 25% to 100%, with no saturation reported at the largest data regime.The study reports consistent improvement across metrics as more training data is used.Source: §4.3 · p. 9, Figure 6 · p. 9

4.4 Qualitative Results

  • With egocentric-only input, occluded contact interfaces can yield missing predicted contact regions; wrist views can reveal those interfaces and recover contact location and intensity.The qualitative comparison attributes the missing regions to insufficient visual evidence in the egocentric view.Source: §4.4 · p. 9

5 Conclusion

EgoTouch pairs multi-view video with bimanual pose and dense tactile pressure maps across a broad set of manipulation episodes. TouchAnything uses flexible camera inputs, and wrist views improve tactile prediction, particularly where egocentric views occlude contact.

5 Conclusion

  • EgoTouch provides synchronized head- and wrist-mounted RGB, bimanual 3D hand pose, tracker poses, and continuous pressure maps across 208 tasks and 1,891 episodes.The dataset targets bimanual hand-object interaction.Source: §5 · p. 9
  • TouchAnything combines cross-view fusion and view dropout to support egocentric-only or multi-view inference.The dataset and benchmark are intended to support research on tactile-grounded embodied perception, manipulation, and learning from egocentric interaction data.Source: §5 · pp. 9–10
  • Wrist-mounted views provide complementary contact evidence and improve tactile prediction, especially contact localization and pressure estimation under egocentric occlusion.Figure 7 contrasts missed occluded contacts in ego-only predictions with recovered pressure distributions from multi-view predictions.Source: §5 · p. 9, Figure 7 · p. 10

6 Limitations and Future Work

The paper identifies glove-specific bias, unsaturated data scaling, and the benchmark’s focus on tactile estimation as limitations. Future work includes expanding the dataset and broadening research toward embodied tasks.

6 Limitations and Future Work

  • Training data use tactile gloves, which may introduce glove-specific appearance bias and limit generalization to bare-hand tactile estimation.The authors propose glove-to-bare-hand retargeting and augmentation as future work.Source: §6 · p. 10
  • Contact IoU and Volumetric IoU continue to improve with more training data, so performance has not yet saturated.The authors plan to add more diverse objects, environments, contact patterns, and manipulation behaviors.Source: §6 · p. 10
  • The current benchmark focuses primarily on tactile estimation, while broader tactile-grounded embodied tasks remain future research directions.Examples named by the authors include contact-aware manipulation, grasp stability prediction, affordance learning, tactile-enhanced world models, and robot policy learning.Source: §6 · p. 10

A.1 Egocentric Hand-Object Interaction Datasets

Prior egocentric datasets and vision-to-touch methods have limited combinations of viewpoints, tactile labels, and interaction breadth. EgoTouch and wrist-mounted views address gaps in contact visibility and tactile prediction for realistic hand-object interaction.

A.1 Egocentric Hand-Object Interaction Datasets

  • Ego4D [10] and EPIC-KITCHENS [5] lack paired 3D hand pose and dense tactile annotations, while EgoDex [14], EgoPressure, and OpenTouch [25] remain limited in viewpoint or interaction scope.EgoPressure focuses on single-hand hand-surface interactions; OpenTouch remains single-hand and single-view.Source: §A.1 · p. 14

A.2 Vision-to-Touch Prediction

  • Existing vision-to-touch approaches face frequent occlusion of hand-object contact regions, and prior datasets offer limited support for realistic bimanual interactions.The paper argues that complementary viewpoints are needed to observe contact regions that standard visual inputs miss.Source: §A.2 · p. 14

A.3 Multi-View Learning for Hand Interaction

  • Multi-view methods have mainly targeted geometry or action understanding, while their viewpoints often do not directly capture the hand-object contact interface.Wrist-mounted cameras provide complementary, contact-aware views for tactile estimation under occlusion and viewpoint variation.Source: §A.3 · pp. 14–15

B Additional Dataset Details

The additional details describe synchronized multimodal collection, tactile-grid preprocessing, and compact storage for training. TouchAnything accepts available camera views, combines visual and pose information over time, and decodes bilateral pressure maps.

B.1 Data Collection Setup

  • Three cameras capture head/chest and wrist views alongside tracker poses, bimanual hand kinematics, and dense tactile measurements at a target rate of 30 Hz.Wrist cameras provide close-range views of contacts often occluded from the egocentric camera; pressure gloves record 256 channels per hand.Source: §B.1 · p. 15

B.2 Tactile Grid Mapping and Preprocessing

  • Raw 256-value tactile vectors are remapped to sparse 21 × 21 hand-shaped grids, with the right hand mirrored into a canonical coordinate system.Preprocessing can remove a contact-free first-frame baseline, repair broken columns, and normalize tactile and bending sensors separately.Source: §B.2 · p. 16

B.3 HDF5 Dataset Conversion

  • HDF5 conversion stores temporally aligned camera streams, hand and tracker poses, processed pressure grids, and metadata in a stable modality hierarchy.The final frame count is set to the minimum valid length across the three camera streams.Source: §B.3 · pp. 16–17

C Additional Implementation Details

  • The model accepts any subset of three camera views and bimanual hand pose, predicting dense bilateral pressure maps despite missing views.View dropout during training supports flexible inference with different camera configurations.Source: C Additional Implementation Details · p. 17

C.1 Problem Formulation

  • For each timestep, the task is to predict bilateral pressure maps for both hands from available video views and the corresponding bimanual hand-pose sequence.Source: §C.1 · p. 17

C.2 Multi-View Vision Encoder

  • A shared DINOv2 backbone adds camera-specific view embeddings, then cross-view attention and gating fuse information from available views.Sharing the backbone reduces parameters from 3 × 86M to 86M + 3 × 768; cross-view attention exchanges information between view summaries.Source: §C.2 · p. 17

C.3 Temporal Modeling and Pose-Vision Fusion

  • A windowed temporal transformer models manipulation dynamics, while pose-vision cross-attention lets each hand-joint token attend to relevant visual patches.The pose encoder produces per-joint features from the bimanual hand-pose sequence.Source: §C.3 · pp. 17–18

C.4 Joint-Level Tactile Decoder

  • The decoder splits 42 fused joint features into left- and right-hand groups and maps each group to a 21 × 21 pressure map.A sigmoid bounds the outputs to [0, 1].Source: §C.4 · p. 18

C.5 View Dropout Training Strategy

View dropout trains TouchAnything to work with available camera subsets, while its objective combines pressure-map fitting with spatial regularization and its shared evaluation protocol compares view configurations.

C.5 View Dropout Training Strategy

  • Randomly dropping each wrist view with probability p = 0.3 trains the model for four input configurations, and the same model accepts available views at inference without architectural changes or fine-tuning.The egocentric view is always retained.Source: §C.5 · p. 18

C.6 Training Objective and Optimization

  • The objective combines pixel-wise regression, sparsity-aware pressure fitting, and spatial regularization, with pixels above 0.1 weighted more to reduce all-zero pressure predictions.The contact-region weight is 3.0; λmse = 1.0, λl1 = 0.5, and λtv = 0.01.Source: §C.6 · p. 18

C.7 Inference and Evaluation Protocol

  • The evaluation protocol compares ego-only, single-wrist, and full multi-view inputs using the same checkpoint, split, preprocessing, and metric implementation.Metrics include Temporal Accuracy, Contact IoU, Volumetric IoU, and MAE.Source: §C.7 · p. 19

D Additional Qualitative Results

The qualitative examples show tactile predictions tracking contact locations, pressure patterns, and temporal change across varied interactions.

D Additional Qualitative Results

  • The examples show predicted maps capturing contact locations, pressure intensity, and temporal dynamics across a wide range of interactions.Each visualization places egocentric RGB input, ground-truth maps, and predicted maps in successive rows.Source: D Additional Qualitative Results · p. 19
  • Predicted pressure maps remain spatially consistent with hand pose and visual observations while capturing contact evolution during manipulation.Source: D Additional Qualitative Results · p. 19
  • Examples span diverse object categories, manipulation strategies, and contact configurations.Source: D Additional Qualitative Results · p. 19

D.1 Failure Cases and Limitations

The qualitative results include accurate contact predictions across varied tasks, but low-contrast hand-object appearance can produce false contact predictions; the authors identify several possible improvements.

D.1 Failure Cases and Limitations

  • Color similarity can mislead the model when it obscures hand-object boundaries, reflecting its reliance on visual proximity and occlusion cues for contact.Source: §D.1 · p. 19
  • The authors suggest depth or hand-object segmentation, more challenging color combinations in training data, and temporal consistency to suppress isolated first-frame false positives.Source: §D.1 · p. 19
  • Other examples show fine-grained contact patterns and bimanual coordination, including contact-location accuracy in precision manipulation and gameplay.Source: Figure 9 · p. 21, Figure 10 · p. 22
  • Additional examples show pressure maps consistent with hand pose and visual observations while capturing temporal contact evolution.Source: Figure 11 · p. 23
  • A black glove against black shorts causes a first-frame false contact prediction despite no contact having occurred.The low-contrast appearance resembles contact and leads the model to hallucinate pressure.Source: §D.1 · p. 19, Figure 12 · p. 24

Related papers

Loading 2605.13083v1…