PaperlayerView exact revision on arXiv

Source-linked AI summary

Dexterity from Touch: Self-Supervised Pre-Training of Tactile Representations with Robotic Play

Irmak Guzey, Ben Evans, Soumith Chintala, Lerrel Pinto

arXiv:2303.12076v1cs.ROcs.AIcs.CVcs.LG

TL;DR

Vision-based dexterity struggles with contact forces and hand occlusion, while tactile learning faces simulation, calibration, and high-dimensionality challenges. T-DEX pretrains tactile representations from teleoperated play and combines tactile and visual observations for imitation. Across tasks, it improves average success rate over vision-only and torque-only models.

  • Problem

    Vision-based dexterity struggles with contact forces and hand occlusion, while tactile learning faces simulation, calibration, and high-dimensionality challenges.

    Source: §I · p. 1

  • Method

    T-DEX pretrains tactile representations from teleoperated play and combines tactile and visual observations for imitation.

    Source: §I · p. 2, C. How important is tactile sensing… · p. 6

  • Results

    Over 170% improvement in average success rate over vision-only and torque-only imitation models.

    Source: §I · p. 2

  • Takeaways & Limitations

    T-DEX picked up 3 of 4 new bowls and 2 of 4 unseen cups without retraining.

    Source: F. Generalization to Unseen Objects · p. 8

    T-DEX is currently limited to offline imitation, limiting policies’ ability to learn from failures.

    Source: §VI · p. 8

Abstract

from arXiv · show

Teaching dexterity to multi-fingered robots has been a longstanding challenge in robotics. Most prominent work in this area focuses on learning controllers or policies that either operate on visual observations or state estimates derived from vision. However, such methods perform poorly on fine-grained manipulation tasks that require reasoning about contact forces or about objects occluded by the hand itself. In this work, we present T-Dex, a new approach for tactile-based dexterity, that operates in two phases. In the first phase, we collect 2.5 hours of play data, which is used to train self-supervised tactile encoders. This is necessary to bring high-dimensional tactile readings to a lower-dimensional embedding. In the second phase, given a handful of demonstrations for a dexterous task, we learn non-parametric policies that combine the tactile observations with visual ones. Across five challenging dexterous tasks, we show that our tactile-based dexterity models outperform purely vision and torque-based models by an average of 1.7X. Finally, we provide a detailed analysis on factors critical to T-Dex including the importance of play data, architectures, and representation learning.

I. INTRODUCTION

Vision-based dexterity can struggle with contact-force reasoning and hand-occluded objects, while tactile learning for multifingered manipulation faces high-dimensional inputs without pretrained encoders. T-DEX pretrains tactile representations from robot play and uses them with vision and a small set of task demonstrations.

A. Dexterous Manipulation

  • Prior dexterous-manipulation approaches include physics-based control, simulation-to-real learning, and imitation from visual observations or vision-derived state estimates.The cited discussion notes that visual observations and state estimates can suffer under heavy object occlusion.Source: §I · p. 1, A. Dexterous Manipulation · p. 2

I. INTRODUCTION

  • Vision-based methods can struggle with contact-force reasoning and hand-occluded objects, while multifingered tactile readings are high-dimensional and lack pretrained encoders.The introduction identifies tactile simulation and force calibration as additional challenges.Source: §I · p. 1
  • T-DEX pretrains tactile encoders on 2.5 hours of teleoperated object manipulation, then learns policies from a handful of task demonstrations.Its non-parametric policy retrieves nearest-neighbor actions and combines tactile encodings with vision without additional training or sensor fusion.Source: §I · p. 2
  • Over 170% improvement in average success rate over vision-only and torque-only imitation models is reported for T-DEX.The paper also reports an average 58% improvement over tactile models that do not use play data.Source: §I · p. 2

B. Tactile Sensing

The related work describes tactile sensing and representation learning as relevant but distinct challenges for multifingered dexterity. T-DEX combines task-agnostic play data with offline nearest-neighbor imitation to learn from few task demonstrations.

B. Tactile Sensing

  • Prior tactile-learning methods had not been applied to multifingered hands and required substantial task-centric data, whereas T-DEX uses task-agnostic play data.The paper uses XELA uSkin sensors to cover its Allegro hand.Source: B. Tactile Sensing · p. 2

C. Representation Learning for Robotics

  • Self-supervised learning can produce concise representations from high-dimensional observations, and T-DEX extends this approach to tactile data collected through robot play.The paper notes that large pretrained models and internet-scale data are unavailable for tactile observations.Source: C. Representation Learning for Robotics · pp. 2–3

D. Exploratory and Play Data

  • T-DEX uses cheap, imperfect, tactile-rich exploratory interactions to improve representations and downstream task performance.Prior work cited in the paper explores using off-policy or exploratory data for downstream performance and robustness.Source: D. Exploratory and Play Data · p. 3

E. Offline Imitation Learning

  • Because collecting large demonstration sets for dexterous tasks is difficult, T-DEX uses offline nearest-neighbor imitation to learn from few demonstrations.The paper contrasts this with behavior cloning, which often requires hundreds to thousands of trajectories.Source: E. Offline Imitation Learning · p. 3

III. SYSTEM DETAILS AND ROBOT SETUP

T-DEX uses a teleoperated robot hand with tactile and visual sensing, then trains tactile representations from play before learning task policies from demonstrations. Its downstream nearest-neighbor approach combines tactile and visual features.

III. SYSTEM DETAILS AND ROBOT SETUP

  • The system pairs a 6-DoF Kinova arm and 16-DoF Allegro hand with 15 tactile sensors and two RGB cameras.The tactile readings total 720 dimensions and are uncalibrated; policies use information from one of the cameras.Source: §III · p. 3, Figure 2 · p. 3

IV. TACTILE-BASED DEXTERITY (T-DEX)

  • T-DEX first collects task-agnostic play data and trains a tactile encoder with self-supervised learning, then applies non-parametric imitation to task demonstrations.The framework has separate pretraining and downstream-learning phases.Source: §IV · pp. 3–4

A. Phase I: Pre-Training Tactile Representations from Play

  • The play dataset contains 2.5 hours of contact-rich teleoperated interactions, including failed examples and random behavior.Subsampling on movement thresholds reduces the dataset from 450k frames to 42k observations.Source: A. Phase I: Pre-Training Tactile… · p. 4
  • BYOL trains tactile representations from augmented views, and the trained encoder extracts features from observations.The tactile readings are arranged as a three-channel image and resized to 224x224 for standard image encoders.Source: A. Phase I: Pre-Training Tactile… · p. 4

B. Phase II: Non-parametric Learning

  • For each task, teleoperators collect six demonstrations; successful demonstrations take at most 10 minutes, though collection can take up to 30 minutes.The paper attributes the longer collection time to failures when teleoperators lack tactile feedback.Source: B. Phase II: Non-parametric Learning · pp. 4–5
  • The downstream policy retrieves demonstration actions using nearest neighbors over visual and tactile features, with visual features learned from demonstration images.Demonstrations are subsampled using a 2 cm movement threshold to reduce dataset size and filter noise.Source: B. Phase II: Non-parametric Learning · p. 5

V. EXPERIMENTS

The experiments are framed around three questions: whether touch improves policy performance, how play data affects representations, and which design choices matter for tactile feature learning.

V. EXPERIMENTS

  • The evaluation asks whether tactile information improves policy performance.Source: §V · p. 5
  • The evaluation tests how important play data is to the learned tactile representations.Source: §V · p. 5
  • The evaluation examines which design choices matter when learning tactile features.Source: §V · p. 5

A. Description of Dexterous Tasks

The section describes five contact-rich tasks requiring precise multi-finger control. Evaluation uses six demonstrations per task and tests new object configurations within the convex hull of those demonstrated.

1) Joystick Movement:

  • Joystick pulling requires moving down and pulling a joystick backward, with evaluation on positions and orientations unseen during training.A trial succeeds when the joystick is pulled within 60 seconds.Source: 1) Joystick Movement · p. 5

2) Bottle Opening:

  • Bottle opening tests lid removal using demonstrations that require either the thumb or the middle finger.Success means opening the lid within 120 seconds.Source: 2) Bottle Opening · p. 5

3) Cup Unstacking:

  • Cup unstacking requires contacting both cups before lifting the inner cup with the index finger, without dropping it or knocking the outer cup off the table.The smaller cup must be raised outside the larger one within 240 seconds.Source: 3) Cup Unstacking · p. 5

4) Bowl Unstacking:

  • Bowl unstacking requires multiple fingers to lift and stabilize the larger bowl, with success defined as lifting it within 100 seconds.Source: 4) Bowl Unstacking · p. 5

5) Book Opening:

  • Book opening requires three-finger contact, an arm pull, and continued contact until the book is fully open.The rollout visualization notes severe visual occlusion at object contact; success is opening the book within 300 seconds.Source: 5) Book Opening · p. 5, Figure 5 · p. 6

Evaluation setup

  • Six demonstrations per task cover configurations in a 10x15cm box, and evaluation tests new configurations within the convex hull of demonstrated ones.The authors describe this as standard practice for evaluating robotics representations.Source: 5) Book Opening · p. 6

B. Baselines for Dexterity

The baselines compare imitation and nearest-neighbor policies using different combinations or sources of tactile, image, and torque information. The section labels the learned-policy comparison as real-world success rates.

1) Behavior Cloning (BC) [56]:

2) Nearest Neighbors with Torque only (NN-Torque) [66]:

3) Nearest Neighbors with Image only (NN-Image) [53]:

4) Nearest Neighbors with Tactile only (NN-Tactile):

5) Nearest Neighbors with Tactile Trained on Task Data (NN-Task):

6) Nearest Neighbors with Tactile Trained on Play Data (T-DEX):

C. How important is tactile sensing for dexterity?

Across the reported tasks, T-DEX combines image-based localization with tactile manipulation and produces the strongest results. Play-data pretraining and the structure of tactile representations both affect performance.

C. How important is tactile sensing for dexterity?

  • T-DEX combines NN-Image’s coarse localization with NN-Tactile’s fine-grained manipulation, producing the strongest results across all tasks.NN-Tactile struggles to localize objects for first contact, while NN-Image struggles under hand occlusion.Source: C. How important is tactile sensing… · p. 6
  • Image-only neighbors can miss bottle contact, while tactile-only neighbors can miss robot position; combining both modalities captures both cues.Figure 6 visualizes the camera image, top-two activated tactile sensors, and their nearest neighbors.Source: Figure 6 · p. 7, C. How important is tactile sensing… · p. 7

D. Does pre-training on play improve tactile representations?

  • Task-data-only tactile pretraining works on simpler Joystick Movement but struggles on harder Unstacking and Bottle Opening tasks; adding failed Bowl Unstacking demonstrations yields 30% success with no improvement.The authors attribute the harder-task results to poor representational matches from limited task data.Source: D. Does pre-training on play improve… · p. 7
  • 20 minutes of play data yields 90% success on Book Opening, while Cup Unstacking improves steadily with larger amounts of play data.Figure 7 compares success rates on both tasks across play-data amounts.Source: D. Does pre-training on play improve… · p. 7, Figure 7 · p. 7

E. Importance of tactile representation

  • T-DEX and ResNet perform similarly on Book Opening, but ResNet uses significantly more computation; on Cup Unstacking, ResNet performs a little worse than T-DEX.A simpler 3-layer CNN also performs poorly and does not reach either larger model’s performance.Source: Table II · p. 8, E. Importance of tactile representation · p. 8
  • Channel-stacked and shared-pad tactile encoders perform poorly; the authors hypothesize that noise in high-dimensional raw data can produce incorrect neighbors and actions.Raw Tactile is roughly on par with the Stacked method.Source: E. Importance of tactile representation · p. 8
  • On Bowl Unstacking, PCA on raw tactile features reaches 40% success, summed 4x4 sensor activations fail on every task, and shuffled pads reach 20% success.The authors report that PCA misses fine-grained tactile information and that shuffled pads perform much worse than the structured layout.Source: E. Importance of tactile representation · p. 8

F. Generalization to Unseen Objects

T-DEX generalizes to some unseen bowls and cups, though its success is below performance on training objects. The paper also identifies offline imitation as a limitation because policies cannot learn from failures.

F. Generalization to Unseen Objects

  • The image-only baseline sometimes applied too much force and picked up both objects, or failed to detect incorrect contact.Source: Figure 9 · p. 8
  • T-DEX picked up 3 of 4 unseen bowls and 2 of 4 unseen cups, while NN-Image succeeded on one novel bowl.Tactile information helped find demonstration neighbors for unseen objects without retraining.Source: F. Generalization to Unseen Objects · p. 8

VI. LIMITATIONS AND CONCLUSION

  • Success on out-of-distribution objects remains below training-object performance, and offline imitation cannot learn from failures.The authors suggest online learning, improved tactile architectures, and better tactile-vision fusion as possible directions.Source: §VI · p. 8

APPENDIX

The appendix gives experiment and model details, including standardized starting configurations, input normalization, BYOL selection, and nearest-neighbor policy settings.

A. Experiment Details

  • The joystick, bottle, and book were secured to the table, while bowls and cups were left unsecured; each method used the same 10 starting positions per trial index.Source: A. Experiment Details · p. 13

B. Model Details

  • Image inputs were normalized by training-set mean and standard deviation, while tactile inputs were scaled to [0, 1].Source: B. Model Details · p. 13

1) BYOL Details:

2) Nearest Neighbors Details:

  • Nearest-neighbor distances weighted vision and touch equally except for bottle cap, where tactile distance had weight 2 and image distance weight 1.A reject buffer avoids cyclical neighbor-action loops; its size was 10 except for Joystick Pulling (3), and the buffer plus 2 cm spatial subsampling were described as critical to success.Source: 2) Nearest Neighbors Details · p. 13

3) BC Details:

The appendix describes behavior-cloning and torque baselines, PCA compression of tactile play data, a shuffled-pad experiment, and additional task rollouts.

3) BC Details:

  • Behavior cloning was trained end-to-end with action MSE loss, BYOL’s learning rate, and batch size 64.Source: 3) BC Details · p. 13

4) NN-Torque Details:

  • The torque baseline used desired torque targets as a proxy for contact: targets were higher when fingers pressed further into an object and lower without contact.Source: 4) NN-Torque Details · p. 13

5) PCA Details:

  • PCA on tactile play data used the top 100 components as features, capturing about 95% of variance; visualized reconstructions retained a majority of tactile information.The explained variance ratio is shown in Figure 10, and the reconstructions in Figure 11.Source: 5) PCA Details · p. 13, Figure 10 · p. 13

6) Shuffled Pad Details:

  • The shuffled-pad experiment permuted 15 4x4 tactile sensors consistently across pretraining and deployment, without using pad spatial locations.If spatial layout had no effect, the authors expected performance to match T-DEX.Source: 6) Shuffled Pad Details · p. 13

C. Additional Rollouts

  • Additional rollouts are shown for Bottle Opening, Book Opening, Bowl Unstacking, Cup Unstacking, and Joystick Pulling.Source: C. Additional Rollouts · p. 13

D. Tactile Image Visualization

The appendix shows tactile images and additional rollouts across the five tasks, alongside PCA reconstructions of tactile readings.

D. Tactile Image Visualization

Related papers

Loading 2303.12076v1…