PaperlayerView exact revision on arXiv

Source-linked AI summary

SoTa: Soft Tactile Skins for Dexterous Manipulation

Jingyun Yang, Baiyu Shi, Timothy Yu, Haitian Liu, Alberta Longhini, Weichen Wang, Rika Antonova, Zhenan Bao, Jeannette Bohg

arXiv:2610.02338v1cs.ROcs.LG

TL;DR

Dexterous visuo-tactile robot data is scarce because demonstrations require costly teleoperation, while human demonstrations need corresponding tactile signals to supplement robot data. SoTa provides custom-fit, full-hand capacitive skins with a shared 202-taxel layout and uses them for human-robot co-training. Across evaluated conditions, adding human demonstrations raises mean success from 22.8% to 45.9%.

  • Problem

    Dexterous robot demonstrations require costly teleoperation, and human demonstrations can supplement them only when human and robot tactile signals correspond.

    Source: §I · p. 1

  • Method

    SoTa uses custom-fit capacitive skins with a shared 202-taxel layout and co-trains robot policies using human and robot visuo-tactile demonstrations with auxiliary language supervision.

    Source: §I · p. 1, C. Policy Learning Architecture · p. 5, A. Experimental Setup · p. 5

  • Results

    With a fixed robot-demonstration budget, human visuo-tactile demonstrations raise mean success from 22.8% to 45.9% and improve every reported condition.

    Source: §I · p. 2, C. Auxiliary Supervision and… · p. 6, §V · p. 7

  • Takeaways & Limitations

    The results suggest that tactile human demonstrations are a practical way to supplement costly robot teleoperation.

    Source: §V · p. 7

    The study uses 150 human demonstrations per task; effects of greater dataset size or diversity and how far human demonstrations can replace robot data remain unmeasured.

    Source: §V · p. 7

Abstract

from arXiv · show

A growing body of work suggests that tactile sensing gives robot policies contact information that complements vision in dexterous manipulation. However, visuo-tactile robot data remains scarce: dexterous demonstrations require teleoperating robots, which limits dataset scale. Human demonstrations are far cheaper to collect and offer a path to scale this data, but only if human and robot hands carry tactile sensors with corresponding signals. This requires sensors that conform to different hand geometries, cover the full hand, and share a common layout across embodiments. We present SoTa, a low-cost capacitive tactile skin that provides full-hand coverage on humans and robots while preserving a shared layout of 202 taxels across corresponding finger and palm regions. Our multilayer design with fabric electrodes enables in-house fabrication of thin, soft skins with customizable geometry for under $10 in materials per skin. The sensor retains over 97% of its initial response span after 10,000 loading-unloading cycles with traces retaining continuity through 1,280 tight-fist folding cycles. The shared taxel layout supports human-robot co-training with a common tactile encoder and no learned cross-sensor mapping. Across three contact-rich manipulation tasks, tactile observations improve in-distribution success over vision-only policies. With a fixed robot demonstration budget, adding human demonstrations more than doubles mean success across eight evaluation conditions, from 22.8% to 45.9%, improving success in all five out-of-distribution conditions. We plan to open-source the resources needed to fabricate and operate these skins.

I. INTRODUCTION

SoTa addresses the scarcity of visuo-tactile robot data with a full-hand sensor that preserves corresponding taxel layouts across human and robot hands. The paper combines that shared interface with human-robot training and reports higher success when human demonstrations supplement a fixed robot-data budget.

I. INTRODUCTION

  • Dexterous robot demonstrations require slow, costly teleoperation, while human demonstrations are cheaper but need corresponding tactile signals across embodiments.Existing systems often sense only fingertips or serve a single embodiment, leaving phalange and palm contacts uncovered in tasks such as wiping and plug insertion.Source: §I · p. 1
  • SoTa is a low-cost capacitive tactile skin with geometry fitted to each hand and a shared layout of 202 taxels, whose corresponding indices map to the same semantic hand regions.Each skin costs under $10 in materials, responds to forces from 20mN to 10N, retains over 97% of its initial response span after 10,000 loading–unloading cycles, and keeps trace continuity through 1,280 tight-fist folding cycles.Source: §I · p. 1
  • Human demonstrations without action labels contribute through an auxiliary objective that predicts task phases from human and robot visuo-tactile observations; action supervision comes only from robot data.The shared layout allows a common tactile encoder without a learned cross-sensor mapping.Source: §I · pp. 1–2
  • With a fixed robot-demonstration budget, adding human visuo-tactile demonstrations raises average success from 22.8% to 45.9% and improves every evaluated condition.Tactile observations also improve in-distribution success on all three tasks, but those benefits do not consistently extend to out-of-distribution conditions.Source: §I · p. 2
  • The paper contributes sensor characterization and a language-supervised pipeline for incorporating human visuo-tactile demonstrations into robot policy learning.The authors plan to release resources for fabricating and operating the skins.Source: §I · p. 2

II. RELATED WORK

Prior work combines vision with touch or force for contact-rich manipulation, but often uses sparse sensing and costly robot data. SoTa pairs full-hand sensing across human and robot hands with a co-training approach that uses shared tactile encoding and task-phase supervision.

II. RELATED WORK

  • Most visuo-tactile policies use localized or sparse sensing, and dexterous robot demonstrations are costly to collect; SoTa provides full-hand coverage with a shared layout.Earlier human-data approaches use limited sensing or heterogeneous sensors with sensor-specific encoders.Source: §II · p. 2
  • Prior human-robot co-training aligns tactile representations with paired demonstrations or pseudo-pairs, whereas SoTa uses task-phase labels without cross-dataset observation pairs, motion retargeting, or human action labels.Human demonstrations contribute to an auxiliary objective, while the paper reports improvements over robot-only training.Source: §II · p. 2

III. THE SOTA SENSOR

  • The paper describes the SoTa sensor, human and robot data collection, and a policy-learning architecture with a shared tactile encoder across embodiments.This is the stated organization of the sensor-method section.Source: §III · p. 2

A. Sensor Design and Fabrication

SoTa combines custom-fit skins with a shared taxel structure and a multilayer fabric-electrode construction. Its characterization reports cyclic stability and load response, while the interface comparison table specifies dimensions used to classify prior systems.

A. Sensor Design and Fabrication

  • The human and robot skins share the same capacitive sensing principle, taxel size, taxel count, and region-level semantic correspondence, while spacing varies with hand geometry.The current design has 202 taxels: 40 on the thumb, 24 on the index finger, 16 on each other finger, and the remainder on the palm.Source: A. Sensor Design and Fabrication · p. 2
  • Table I compares tactile interfaces by coverage, human or robot embodiment, human-robot alignment, and whether the design is custom-fit.Its upper block covers robot-mounted skins and its lower block covers wearable interfaces with a human side.Source: Table I · p. 3
  • The thin-film skin uses laser-patterned fabric electrodes around an SEBS dielectric to form a multilayer parallel-plate piezocapacitive stack.The electrode layers are selectively ablated to define electrically isolated traces, then aligned and sealed.Source: A. Sensor Design and Fabrication · p. 3
  • At a peak force of 10 N, mean per-cycle maximum hysteresis was 11.05 ± 0.13% across three independently fabricated skins, versus 17.5 ± 2.8% reported for STAG [25].The skins showed capacitive responses to applied loads from 20 mN to 10 N.Source: A. Sensor Design and Fabrication · p. 3
  • After 10,000 loading-unloading cycles at a peak force of 8 N, the sensor retained 97.31% of its initial response span.The tests spanned 26.5 hours; endpoint peak and baseline capacitance drifts were −3.04% and −0.35% of full-scale output, respectively.Source: A. Sensor Design and Fabrication · p. 3
  • Skin materials cost less than $10 per device, and readout circuitry costs approximately $30 at an order quantity of 100 units.The authors commit to open-sourcing the sensing stack.Source: A. Sensor Design and Fabrication · p. 3

B. Data Collection Setup

The collection pipeline records synchronized visual and tactile observations for both human and robot demonstrations, with robot demonstrations also providing proprioception and actions. Human data enter alongside robot data without a per-sensor alignment step, and the tactile signals undergo per-taxel post-processing.

B. Data Collection Setup

  • The pipeline records synchronized visual and tactile observations for both embodiments, plus robot proprioception and actions for robot demonstrations.The human collection setup is portable and supports collection outside the laboratory.Source: B. Data Collection Setup · p. 3
  • Robot teleoperation combines wrist tracking and per-finger glove tracking to command the dexterous hand, while the camera and tactile skins record observations.The streams are recorded at 30 Hz; the front-facing camera supplies a single left image stream without stereo or a wrist-mounted view.Source: B. Data Collection Setup · pp. 3–4, Figure 3 · p. 4
  • Human demonstrations use a human-fit tactile skin and camera while the operator performs the same tasks without the robot.Human tactile observations are logged at 30 Hz, and the shared layout lets them enter training without per-sensor alignment.Source: B. Data Collection Setup · p. 4, Figure 3 · p. 4
  • Per-taxel post-processing subtracts a baseline, suppresses spikes, normalizes and transforms signals, then rescales and clips values to [0, 1].The baseline is computed from the first 30 frames, which are set to zero.Source: B. Data Collection Setup · p. 4

C. Policy Learning Architecture

The policy extends FTP-1 with SoTa tactile encoders and a training-only language expert for task-phase supervision. Robot actions are learned through flow matching, while human and robot demonstrations contribute to a two-stage auxiliary-supervision procedure.

C. Policy Learning Architecture

  • FTP-1 combines a vision-language backbone, tactile transformer, and flow-matching action expert; SoTa skins share tactile encoders and a functional-area representation.The shared model components are initialized from FTP-1, while new tactile encoders are trained from scratch.Source: C. Policy Learning Architecture · p. 4
  • A training-only language expert predicts the current task-phase annotation from visual-language and tactile features, with cross-entropy supervision over the pretrained language vocabulary.The prediction branch attends to vision-language and tactile representations, not the action branch.Source: C. Policy Learning Architecture · pp. 4–5
  • The action expert conditions on image, prompt, and tactile observations to generate actions through flow matching.It does not attend to the language expert, whose auxiliary branch is omitted at inference.Source: Figure 4 · p. 5
  • Two-stage co-training first pretrains representations on human and robot demonstrations without action supervision, then jointly trains robot action and human-and-robot language objectives.Human demonstrations never contribute to the action loss; the auxiliary language loss has weight 1.0 in both stages.Source: C. Policy Learning Architecture · p. 5

IV. EXPERIMENTAL EVALUATION

The evaluation examines tactile feedback, auxiliary language supervision and human-robot co-training, and sensor durability and motion-induced artifacts.

IV. EXPERIMENTAL EVALUATION

  • The evaluation tests whether tactile observations improve contact-rich manipulation relative to vision-only policies.Source: §IV · p. 5
  • It examines auxiliary language supervision and human-robot co-training under a fixed robot-data budget.Source: §IV · p. 5
  • It also characterizes sensor durability and motion-induced artifacts.Source: §IV · p. 5

A. Experimental Setup

The experiments evaluate three contact-rich manipulation tasks using robot and human demonstrations, with in-distribution and held-out evaluation conditions. Policies are compared using fully successful rollouts, with partial completions counted as failures.

A. Experimental Setup

  • The tasks are box reorientation, cup picking, and plug insertion, involving sustained contact, delicate grasping, and precise alignment, respectively.Source: A. Experimental Setup · p. 5
  • Methods use robot demonstrations for action supervision, while human data contribute only to the auxiliary objective; two-stage methods add representation pretraining before joint action and language learning.Table II encodes inclusion and absence with green ticks and red crosses.Source: Table II · p. 5
  • Robot training uses 75 demonstrations each for box reorientation and plug insertion and 50 for cup picking; each human dataset has 150 demonstrations across three training objects.One human-training object overlaps with the robot-training object, which is the sole robot-training object for each task.Source: A. Experimental Setup · p. 5
  • In the method notation, R and H denote robot and human demonstrations, while V and T denote vision and touch; robot proprioception is used in all conditions.Source: A. Experimental Setup · p. 5
  • Evaluation includes the robot-training object and setup for each task plus five OOD conditions absent from both training datasets.There is one OOD condition for box reorientation and two each for cup picking and plug insertion; initial object positions are randomized.Source: A. Experimental Setup · p. 5
  • Success is the percentage of fully successful rollouts across 40 trials per policy, task, and condition; partial completions count as failures.Each method and task uses one training seed because of time and compute constraints.Source: A. Experimental Setup · p. 5

B. Effect of Tactile Feedback

Compared with vision-only policies, robot-data policies using tactile feedback improve in-distribution success across all three tasks, but OOD effects vary by task. The reported explanation relates transfer to how contact patterns vary across objects.

B. Effect of Tactile Feedback

  • The tactile-versus-vision comparison uses policies initialized from the same FTP-1 checkpoint and trained on the same task-specific robot dataset, holding other settings fixed.Source: B. Effect of Tactile Feedback · pp. 5–6
  • Tactile feedback improves in-distribution success on all three tasks and substantially improves box OOD success, but does not improve OOD generalization for cup picking or plug insertion.Cup OOD success is lower than with the vision-only policy; Figure 6 reports fully successful rollouts out of 40 trials, counting partial completions as failures.Source: B. Effect of Tactile Feedback · p. 6, Figure 6 · p. 6
  • The authors hypothesize that broad, sustained box contact transfers across box sizes, whereas localized cup and plug contacts may be more sensitive to object geometry.They relate inconsistent transfer to training on only one robot object instance per task.Source: B. Effect of Tactile Feedback · p. 6

C. Auxiliary Supervision and Human-Robot Co-Training

The section evaluates whether human demonstrations and tactile auxiliary supervision contribute to co-training gains. Full human-robot visuo-tactile co-training outperforms the robot-only reference across reported conditions; ablations and object coverage offer evidence about those gains.

Experimental setup

  • Each method used the same FTP-1 initialization and task-specific robot demonstrations; co-training additionally used 150 human demonstrations per task.Source: C. Auxiliary Supervision and… · p. 6

Results

  • Full human-robot visuo-tactile co-training beat the robot-only reference in every reported condition, raising mean success from 22.8% to 45.9%.The largest relative gains were OOD.Source: C. Auxiliary Supervision and… · p. 6

Ablations

  • Robot-only two-stage training did not reproduce the full method’s OOD gains: improvements over the reference were inconsistent across tasks, and cup ID performance decreased.Source: C. Auxiliary Supervision and… · p. 6
  • Vision-only human co-training underperformed the full method and reference in most conditions; this ablation evaluates tactile auxiliary supervision as a whole, not human touch alone.Source: C. Auxiliary Supervision and… · p. 6

Analysis

  • The human dataset covered three objects per task, versus one robot-training instance; this broader coverage may have helped policies generalize beyond training objects and configurations.Source: C. Auxiliary Supervision and… · p. 6

D. Tactile Sensor Durability and Motion Robustness

The section tests SoTa traces under repeated folding and assesses motion-induced responses during finger flexion. The traces retain continuity through the folding test, while motion responses remain modest under the reported conditions.

Folding durability

Test conditions

  • Thin-film samples were cycled across two Wuji index-finger joints into tight fist closure, imposing more severe bending than typical use; assembly encapsulation was omitted from thin-film samples.Source: D. Tactile Sensor Durability and… · pp. 6–7

Folding durability

  • Mean SoTa resistance rose from 9.1 Ω to 1.14 kΩ, remaining below the readout’s nominal 10 kΩ discharge resistance and approximately 1/560 of jumper wires’ 643 kΩ.A fresh baseline is acquired each time the glove is worn; under the RC readout model, baseline subtraction and normalization cancel resistance-induced scaling of measured capacitance.Source: D. Tactile Sensor Durability and… · p. 7

Motion robustness

  • Across five flexion conditions, motion responses were strongest on the flexed finger and modest; the largest mean regional RMS was 6.18%, comparable to the normalized response under a 0.2 N light load.Source: D. Tactile Sensor Durability and… · p. 7

V. DISCUSSION AND CONCLUSION

SoTa provides full-hand tactile coverage across human and robot hands, with corresponding taxel regions supporting a shared tactile encoder. Across the reported tasks, adding human demonstrations improves success, while dataset scaling and long-term wear remain open questions.

SoTa

  • SoTa is a low-cost, durable capacitive tactile skin providing full-hand coverage on human and robot hands, fitted to each hand while preserving taxel correspondence.Source: §V · p. 7

Results

  • 22.8% to 45.9%: adding human visuo-tactile demonstrations raised average success and improved every evaluated condition, with the largest gains out of distribution.Source: §V · p. 7

Evaluation

  • The discussion reports tactile success gains across box reorientation, cup picking, and plug insertion; Table III summarizes regional RMS responses across four experimenters and ten flex–relax cycles.Source: §V · p. 7, Table III · p. 7

Limitations

  • The human dataset was fixed at 150 demonstrations per task, leaving effects of greater size and diversity and the possibility of replacing robot data unmeasured.Long-term wear under sustained use also remains to be evaluated.Source: §V · p. 7

Resources and future work

  • The authors plan to release the fabrication workflow, sensor designs, readout-board Gerber files, and firmware; future work will study scaling and extension to other end effectors.Source: §V · p. 7

Related papers

Loading 2610.02338v1…