PaperlayerView exact revision on arXiv

Source-linked AI summary

TACROSS: An Efficient and Low-Cost Scalable Human Touch System Across Heterogeneous Tactile Sensors for Dexterous Robot Learning

Bo Chen, Huanzhang Hu, Junyang Ma, Bo Yue, Fangdi Yu, Haijier Chen, Xianxin Lai, Shuyu Pan, Zhen Yang, Xiaoquan Sun, Wenze Cui, Zhongliang Jiang, Shaopeng Liu, Jiayu Chen

arXiv:2610.11945v1cs.ROcs.LG

TL;DR

Human tactile demonstrations could reduce the cost and time of robot data collection, but heterogeneous human and robot sensors make raw-channel alignment ill-posed, and retargeted human motion is not ground-truth robot action. TACROSS aligns contact semantics and grounds policy learning in robot actions. With 30 robot demonstrations, it reports 91.9% mean success across four tasks versus 51.3% for Robot-only training.

  • Problem

    Human tactile data is cheaper and faster to collect, but sensor heterogeneity makes raw-channel matching ill-posed, while retargeted human hand motion cannot provide ground-truth robot actions.

    Source: §I · p. 2

  • Method

    TACROSS maps heterogeneous touch into a shared contact-semantic latent and uses executed robot actions as ground-truth labels, with confidence-weighted, validity-masked retargeted human targets as auxiliary supervision.

    Source: §I · p. 2, 1) Contact Canonicalization across… · p. 4

  • Results

    With 30 robot demonstrations, full TACROSS with 150 human demonstrations achieves 91.9% mean success across four tasks versus 51.3% for Robot-only training (+40.6 percentage points).

    Source: C. Ego Data Gains and Requirements for… · p. 6

  • Takeaways & Limitations

    TACROSS provides a practical way to scale tactile data collection without requiring continuous robot operation while supporting contact-rich dexterous manipulation.

    Source: §V · p. 8

    Transfer still relies on robot demonstrations and retargeting and calibration specific to each embodiment.

    Source: §V · p. 8

Abstract

from arXiv · show

Collecting tactile demonstrations on robots is costly and slow, motivating the use of lower-cost human tactile gloves for scalable data collection. However, human capacitive/piezoresistive gloves and robotic tactile sensors differ fundamentally in transduction principle, sensor layout, spatial resolution, and dynamic response, making alignment of raw sensor channels ill-posed. To address this problem, we present TACROSS, a scalable system for learning from human touch and transferring it to robots that bridges this heterogeneity by aligning tactile streams at the level of contact events rather than raw sensor values. The hardware component of TACROSS integrates a piezoresistive glove with five layers and a cost of USD 10.86 with 285 sensing points. To align contact semantics, we design canonicalizers and residual adapters that map heterogeneous signals into a shared tactile latent with 256 dimensions via a temporal Transformer with attention across fingers. We further introduce a robot-grounded policy learning scheme in which robot demonstrations provide the sole source of ground-truth action supervision, while human demonstrations support tactile representation learning and provide confidence-weighted auxiliary supervision through valid retargeted hand targets. We evaluate our system on four contact-rich manipulation tasks. Compared to conventional teleoperation, our proposed system achieves a 3.5-fold efficiency improvement while reducing demonstration acquisition equipment cost by 95.7%. We will open-source the TACROSS hardware and software system and publicly release a tactile dataset comprising over 150 hours of recordings. Project page: https://tacross-touch-project.github.io/.

I. INTRODUCTION

TACROSS addresses the sensing and action mismatches that complicate transfer from low-cost human tactile demonstrations to robot manipulation. It combines cross-sensor contact-semantic alignment with robot-grounded policy learning and lower-cost human data collection.

I. INTRODUCTION

  • 112 and 140 episodes/h from Ego-R and Ego-V are 2.80× and 3.50× the 40 episodes/h teleoperation rate.The human systems yielded 159.09 h of trajectories, including 23,677 recorded and 21,412 valid demonstrations.Source: §I · p. 2
  • The Sensing Gap arises because the piezoresistive human glove and capacitive robot sensors differ in transduction, layout, resolution, and response, making raw-channel matching ill-posed.The Action Gap is that retargeted human motion is not an executed robot action and can contain errors, so pseudo-labels cannot serve as ground-truth robot actions.Source: §I · p. 2
  • TACROSS uses canonicalizers, validity masks, residual adapters, and a shared causal temporal encoder with attention across fingers to align human and robot touch in a 256-dimensional latent.The canonicalizers represent contact events rather than matching raw channels; robot-executed actions provide the sole ground-truth action supervision, while valid retargeted human targets provide confidence-weighted auxiliary supervision.Source: §I · p. 2
  • TACROSS will open-source its glove designs, robot configurations, and software, and release 183.95 h of trajectories: 159.09 h human and 24.86 h robot data.The hardware–software system is presented as an efficient, low-cost acquisition system evaluated for collection throughput and downstream robot performance.Source: §I · p. 2

II. RELATED WORK

Prior work explores human tactile capture, cross-sensor alignment, and learning from human and robot demonstrations. TACROSS combines these directions using regional contact semantics and robot-action supervision.

A. Human Tactile Demonstration Acquisition

  • Human tactile systems enable distributed capture through inexpensive sensing or personalized fabrication, while TACROSS links a piezoresistive human glove to capacitive robot touch.Related approaches include OSMO's shared glove and DexViTac's matched tactile hardware.Source: A. Human Tactile Demonstration… · p. 2

B. Cross-Sensor Tactile Alignment

  • Cross-sensor methods use domain-invariant features, masked maps with paired interactions, or rectified-flow mappings; TACROSS instead encodes regional contact state with field-level validity masks.TACROSS uses sensor-specific adapters and shared causal temporal and inter-finger encoders, with positive windows matched by task/object metadata and contact semantics rather than synchronized frames.Source: B. Cross-Sensor Tactile Alignment · p. 2

C. Learning from Human and Robot Demonstrations

  • TACROSS uses human touch for representation learning, executed robot actions for ground-truth supervision, and confidence-weighted, validity-masked retargeted hand targets as auxiliary pseudo-labels.The robot policy takes tactile latent, RGB, and proprioception at deployment; arm targets require separate supervision.Source: C. Learning from Human and Robot… · pp. 2–3

A. Problem Formulation and Overview

  • Training uses human and robot episodes; each episode records RGB, kinematics/proprioception, tactile observations, and native timestamps, while executed robot actions supply ground-truth labels.The tactile path canonicalizes regional contact fields, applies domain-specific adapters, and uses shared encoders for temporal dynamics and finger interactions.Source: A. Problem Formulation and Overview · p. 3

B. Human Tactile Acquisition

The human tactile acquisition system uses a layered glove with distributed sensing, calibrated readout, and timestamp-based causal synchronization.

B. Human Tactile Acquisition

  • The five-layer glove has 285 physical sensing locations across a 12 × 15 palm array and five 3 × 7 finger arrays; it streams 256-value frames at approximately 150 Hz.Matrix multiplexing, a 12-bit ADC, and a wrist controller support serial/BLE streaming; physical locations differ from streamed values, and defective channels retain invalid flags.Source: B. Human Tactile Acquisition · p. 3, Figure 2 · p. 3
  • For each modality, TACROSS selects the latest sample whose timestamp is no later than the current time and stores sample ages, masks, and valid pseudo-actions with each episode.An invalid channel is suppressed by its validity flag, while a valid no-contact reading remains distinct; calibration clips readings to a bounded scale.Source: B. Human Tactile Acquisition · p. 3
  • Table I reports hardware specifications and characterization of the TACROSS glove.Its caption defines F.S. as full scale.Source: Table I · p. 3

C. Contact-Semantic Alignment

TACROSS aligns regional contact semantics to reduce domain differences that are irrelevant to control while retaining contact information.

C. Contact-Semantic Alignment

  • Calibration standardizes some numerical preprocessing, but sensor layout, contact distribution, and dynamic response still differ, so TACROSS aligns regional contact semantics.The alignment is intended to reduce domain differences irrelevant to control while retaining contact information.Source: C. Contact-Semantic Alignment · p. 3

1) Contact Canonicalization across Embodiments:

TACROSS maps human and robot measurements to shared anatomical contact fields rather than matching channels directly. Validity masks distinguish unavailable measurements from valid no-contact observations.

1) Contact Canonicalization across Embodiments:

  • Anatomical correspondence, configured separately for each sensor domain, maps channels to shared regions rather than matching channel numbers.The canonicalizers stack regional field vectors and their componentwise validity masks in the same order for both domains.Source: 1) Contact Canonicalization across… · pp. 3–4
  • Each canonical field encodes contact probability, relative intensity and change, duration, area, center of pressure, instability proxy, and calibration confidence.Four intensity levels and six causal phases describe contact evolution.Source: 1) Contact Canonicalization across… · p. 4
  • Unsupported fields or regions receive zero values and masks, while valid readings without contact retain valid masks.Relative tactile intensity is not calibrated absolute force, and the instability value is a proxy rather than a direct slip measurement.Source: 1) Contact Canonicalization across… · p. 4

2) Adapters for Each Sensor:

Sensor-specific adapters map masked contact fields into regional tokens, then shared temporal and cross-finger Transformers encode both domains. Contrastive and predictive objectives align compatible human and robot contact while preserving contact information.

2) Adapters for Each Sensor:

  • Separate shallow human and robot adapters map sensor-dependent fields into regional tokens through a common interface.Inputs include masked fields, masks, intensity and phase embeddings, regional context, and anatomical identity; domain identity selects the adapter, not the shared encoder.Source: 2) Adapters for Each Sensor · p. 4

3) Shared Spatiotemporal Encoding:

  • A shared encoder processes 16-frame causal windows with temporal and cross-finger attention, producing a 256-dimensional hand latent.Temporal attention models contact evolution, while attention across fingers models coordination; padding masks handle missing fingers.Source: 3) Shared Spatiotemporal Encoding · p. 4

4) Objectives for Alignment across Domains:

  • Multi-positive contrastive learning attracts semantically compatible human–robot windows and separates negatives, alongside contact, force, phase, temporal, and domain-adversarial objectives.Compatible pairs use task or object, primitive, active-finger metadata, intensity, and phase without requiring synchronized frames.Source: 4) Objectives for Alignment across… · p. 5

D. Robot-Grounded Policy Learning

Robot demonstrations supply ground-truth action supervision, while retargeted human hand targets provide confidence-weighted auxiliary supervision. The policy uses robot observations and predicts action chunks through ACT.

D. Robot-Grounded Policy Learning

  • Robot demonstrations are the sole source of ground-truth action supervision; retargeted human hand targets provide confidence-weighted auxiliary supervision masked by validity.Human targets are generated from HaMeR keypoints and a calibrated geometric retargeter.Source: D. Robot-Grounded Policy Learning · p. 5
  • The calibrated retargeter maps human finger closure and thumb opposition features to Revo2’s six active control channels.User calibration poses and robot safe endpoints determine offsets, scales, and directions; invalid poses trigger a brief hold and then the configured safe state if loss persists.Source: D. Robot-Grounded Policy Learning · p. 5
  • ACT receives three RGB views, robot proprioception, and a projected tactile latent, then predicts 60-action chunks with 8- or 13-dimensional interfaces.The tactile encoder is frozen for 20,000 steps before further training at a lower learning rate; deployment uses robot observations.Source: D. Robot-Grounded Policy Learning · p. 5

IV. EXPERIMENTS

The experiments evaluate four real-robot contact-rich tasks and compare data collection configurations and equipment costs. Human glove systems record demonstrations faster than robot-only teleoperation, with substantially lower listed equipment investment.

A. Experimental Setup

  • The evaluation covers four real-robot tasks: fixed-point dispensing, liquid transfer, precision pipetting, and sequential fruit pick-and-place.Success requires completing all required stages within the time limit and satisfying task constraints; T4 requires the full fruit sequence.Source: A. Experimental Setup · p. 5

B. Data Acquisition Cost and Efficiency

  • Ego-R and Ego-V record approximately 112 and 140 episodes/h, respectively, versus approximately 40 episodes/h for robot-only teleoperation.Across the three collection configurations, the corpus totals 183.95 h, 24,913 recorded episodes, and 22,486 valid episodes; these totals differ from policy-training subsets.Source: B. Data Acquisition Cost and Efficiency · pp. 5–6
  • The full-hand glove has a USD 10.86 component BOM, while Robot-only, Ego-R, and Ego-V configurations cost approximately USD 12,000, USD 2,150, and USD 520.The listed Ego-R and Ego-V equipment investments are 82.1% and 95.7% lower than Robot-only; these costs cover demonstration acquisition equipment.Source: B. Data Acquisition Cost and Efficiency · p. 6

C. Ego Data Gains and Requirements for Robot Demonstrations

Human Ego demonstrations improve task success when added to a fixed robot-demo set and can reduce robot-data needs under several collection settings. Results also show that the benefit depends on the human-to-robot mix, with lower success at higher Ego proportions in some comparisons.

Adding Ego data at a fixed robot count

Comparison of full-hand tactile gloves

  • The glove-cost comparison warns that prototype bills of materials and commercial-system prices have different cost bases.The Tekscan Grip 4256E cost and derived cost per taxel are estimates, not a verified manufacturer quotation.Source: C. Ego Data Gains and Requirements for… · p. 7

Increasing human data with fixed robot episodes

Reducing the robot count

  • Reducing the allocation from 30 robot/150 human to 16 robot/160 human cuts robot demonstrations by 46.7%, while mean success falls from 91.9% to 88.3%.The reported success decrease is 3.6 percentage points.Source: C. Ego Data Gains and Requirements for… · p. 7

Mixing ratios under fixed budgets for collection time

  • At 1:5, Rokoko-based and vision-IK Ego achieve 93.5% and 91.5% macro success, respectively, versus 92.2% for Robot-only; at 1:10, they achieve 87.5% and 86.5%.The comparisons use fixed per-task collection-time budgets.Source: C. Ego Data Gains and Requirements for… · p. 7

Sequential Fruit Pick-and-Place study of data budgets

  • On T4, 50 robot and 50 Ego demonstrations achieve 37/40 successes (92.5%), versus 38/40 (95.0%) with 92 robot demonstrations, reducing robot demonstrations by 45.7%.At 75% and 90% Ego, success is 36/40 (90.0%) and 34/40 (85.0%), respectively.Source: C. Ego Data Gains and Requirements for… · p. 7

Estimated acquisition time

  • For fruit allocations, 50 robot and 50 Ego demonstrations reduce estimated collection time from 158.8 to 110.3 minutes (30.6%) relative to 92 robot demonstrations.The estimate uses validity rates of 1,074/1,236 for Robot-only and 9,743/10,893 for Vision-IK.Source: C. Ego Data Gains and Requirements for… · p. 7

D. Ablation Analysis

Ablations compare full TACROSS with variants that remove human tactile input or contact-semantic alignment. Both removals are associated with lower mean success in each tested allocation.

Ablation analysis

  • With 30 robot and 150 human demonstrations, full TACROSS achieves 91.9% mean success, versus 73.1% without human tactile input and 49.4% without contact-semantic alignment.The corresponding values with 16 robot and 160 human demonstrations are 88.3%, 71.3%, and 46.9%.Source: D. Ablation Analysis · p. 8
  • With 16 robot and 160 human demonstrations, removing human tactile input or contact-semantic alignment corresponds to success rates of 71.3% and 46.9%, versus 88.3% for full TACROSS.Both ablated variants have lower success than the full model in this training setting.Source: D. Ablation Analysis · p. 8

V. CONCLUSION

TACROSS combines human tactile acquisition, cross-sensor contact-semantic alignment, and policy learning grounded in robot demonstrations. The authors present it as a practical way to scale tactile data collection without continuous robot operation, while noting embodiment-specific dependencies.

V. CONCLUSION

  • TACROSS integrates tactile acquisition across the hand, cross-sensor contact-semantic alignment, and policy learning grounded in robot demonstrations.It combines synchronized human visual, motion, and tactile demonstrations with robot supervision.Source: §V · p. 8
  • The system supports faster, lower-cost acquisition for contact-rich dexterous manipulation without requiring continuous robot operation.Source: §V · p. 8
  • Transfer still relies on robot demonstrations and retargeting and calibration specific to each embodiment.The authors identify extending to additional robotic hands and tasks as future work.Source: §V · p. 8

Related papers

Loading 2610.11945v1…