Source-linked AI summary
VITaL Pretraining: Visuo-Tactile Pretraining for Tactile and Non-Tactile Manipulation Policies
Abraham George, Selam Gano, Pranav Katragadda, Amir Barati Farimani
TL;DR
Complex robotic manipulation remains challenging, and tactile information supports understanding objects and providing feedback during control. This paper pretrains vision and tactile encoders contrastively for imitation learning. Visuo-tactile pretraining improves both tactile and vision-only policies, with vision-only USB-plugging success rising from 20% to 85%.
Problem
Complex manipulation remains a challenge in robotics, and tactile information supports object understanding and closed-loop feedback.
Source: §I · p. 1Method
The paper contrastively pretrains vision and tactile encoders on multimodal demonstrations, then uses the pretrained vision encoder for vision-only imitation or both encoders for visuo-tactile imitation.
Source: §I · p. 1Results
Takeaways & Limitations
The authors suggest visuo-tactile pretraining of vision-only policies may be a better use of tactile data than deploying visuo-tactile policies, particularly where sensor durability and cost matter.
Source: §V · p. 6Pretraining uses task-specific data and requires tactile observations to be collected for each task, limiting the pretrained model's applicability.
Source: §V · p. 6
Abstract
from arXiv · showhide
Tactile information is a critical tool for dexterous manipulation. As humans, we rely heavily on tactile information to understand objects in our environments and how to interact with them. We use touch not only to perform manipulation tasks but also to learn how to perform these tasks. Therefore, to create robotic agents that can learn to complete manipulation tasks at a human or super-human level of performance, we need to properly incorporate tactile information into both skill execution and skill learning. In this paper, we investigate how we can incorporate tactile information into imitation learning platforms to improve performance on manipulation tasks. We show that incorporating visuo-tactile pretraining improves imitation learning performance, not only for tactile agents (policies that use tactile information at inference), but also for non-tactile agents (policies that do not use tactile information at inference). For these non-tactile agents, pretraining with tactile information significantly improved performance (for example, improving the accuracy on USB plugging from 20% to 85%), reaching a level on par with visuo-tactile agents, and even surpassing them in some cases. For demonstration videos and access to our codebase, see the project website: https://sites.google.com/andrew.cmu.edu/visuo-tactile-pretraining
I. INTRODUCTION
The paper investigates how tactile information can be incorporated into imitation learning for complex manipulation. It proposes visuo-tactile pretraining and reports improved performance for both visuo-tactile and vision-only policies.
I. INTRODUCTION
- Tactile information supports object understanding and closed-loop feedback, while prior robotic work has incorporated it into several control approaches and manipulation tasks.Prior examples include cable routing, object detection, peg insertion, and door opening.Source: §I · p. 1
- The approach pretrains vision and tactile encoders with a temporally informed multimodal contrastive loss, then uses them in visuo-tactile or vision-only imitation learning.For vision-only control, the tactile encoder is discarded while the pretrained vision encoder serves as the policy backbone.Source: Figure 1 · p. 1, §I · p. 1
- VITaL uses multimodal pretraining to give a vision-only policy implicit tactile understanding without tactile input at deployment.The method projects visual and tactile observations into a shared latent space to emphasize features present in both modalities, especially contact-related features.Source: §I · p. 1
- Visuo-tactile pretraining moderately improves visuo-tactile policies and significantly improves vision-only policies, whose success rates become comparable.The evaluation includes USB cable plugging and two block-stacking tasks.Source: §I · p. 2
II. RELATED WORKS
Prior work has used tactile sensing in robotic control and contrastive pretraining across multimodal tasks. The paper situates its contribution at the intersection of visuo-tactile pretraining and non-tactile imitation policies.
A. Learning Control Policies using Tactile Sensors
- Tactile sensing has supported classical, reinforcement-learning, and imitation-learning control, while nearest-neighbor imitation and residual policies have also been used for visuo-tactile manipulation.The cited works span cable and wire insertion, peg insertion, pick-and-place, door opening, bowl unstacking, book opening, and joystick movement.Source: A. Learning Control Policies using… · p. 2
B. Contrastive Pretraining
- Contrastive pretraining has linked modalities for recognition and control, but prior work had not examined it for non-tactile control policies.The cited control example used visuo-tactile pretraining for a simulated bottle-capping task.Source: B. Contrastive Pretraining · p. 2
C. Imitation Learning for Robotic Manipulation
- Imitation learning trains policies from observation-action pairs, but can struggle with complex action spaces, multimodal goals, and training-to-deployment distribution shifts.The section highlights ACT's temporal ensembling and Diffusion Policy's action-distribution modeling as approaches to these concerns.Source: C. Imitation Learning for Robotic… · p. 2
1) Action Chunking Transformers:
This section describes ACT's action-sequence prediction and temporal ensembling, then explains how Diffusion Policy models complex action distributions through conditioned denoising.
1) Action Chunking Transformers:
- ACT predicts action sequences from state observations using a transformer-backed CVAE, with its latent variable representing action-sequence style.At inference, ACT sets the latent variable to the prior mean, while a strong KL-divergence term discourages over-reliance on it.Source: 1) Action Chunking Transformers · p. 2
- ACT combines overlapping action-sequence predictions with exponentially decaying weights to reduce the effect of a single bad prediction.This is intended to mitigate errors often caused by out-of-distribution states.Source: 1) Action Chunking Transformers · p. 2
2) Diffusion Policy:
- Diffusion Policy generates observation-conditioned action sequences by iteratively denoising them, training a network to predict added noise.Its observation conditioning uses FiLM layers, and decoupled training and inference noise schedulers permit fewer denoising iterations at inference for speed.Source: 2) Diffusion Policy · pp. 2–3
- The diffusion update uses a learned noise-prediction network and scheduler parameters, with the noise variance at each iteration set by the scheduler.The section identifies the noise-prediction parameters and the iteration-dependent variance.Source: 2) Diffusion Policy · pp. 2–3
III. METHODS
The method pretrains vision and tactile encoders to align observations in a shared latent space, using robot position as additional tactile context. Temporally separated samples and camera-wise comparisons encourage representations that capture scene changes and align across views.
A. Pretraining
- Vision and tactile encoders map observations into a shared 512-dimensional latent space, with robot position also supplied to the tactile projection head.The projection heads are discarded after pretraining.Source: A. Pretraining · p. 3
- Training maximizes cross-modal similarity for observations from the same scene and minimizes it for observations from different scenes.Figure 2 depicts the same-timestep and different-timestep contrast as similar and dissimilar embeddings.Source: A. Pretraining · p. 3, Figure 2 · p. 3
- Sampling seven observations from one demonstration at least 10 timesteps apart makes the encoders learn temporal changes within a trajectory.At the stated 10 Hz collection rate, the minimum gap corresponds to one second.Source: A. Pretraining · p. 3
- The loss is calculated separately for each camera view, aligning tactile observations with each view and encouraging shared vision features across cameras.The vision encoder is shared across camera views.Source: A. Pretraining · p. 3
B. Imitation Learning Frameworks
The study implements ACT and Diffusion Policy to examine how visuo-tactile pretraining affects manipulation-task learning.
B. Imitation Learning Frameworks
- The study implements Action Chunking Transformer (ACT) and Diffusion Policy as its two imitation learning frameworks.The implementations are used to study the impact of visuo-tactile pretraining on manipulation-task learning.Source: B. Imitation Learning Frameworks · p. 3
- The frameworks are presented as two implementations for evaluating visuo-tactile pretraining in manipulation learning.Their network designs are illustrated in Figure 3.Source: B. Imitation Learning Frameworks · p. 3
- Figure 3 depicts the study's two imitation learning network implementations.The passage identifies ACT and Diffusion Policy as the frameworks shown.Source: B. Imitation Learning Frameworks · p. 3
1) Action Chunking Transformer:
ACT combines pretrained visual and tactile encoders with an action-sequence prediction architecture. The implementation predicts relative goal positions and temporally ensembles its action predictions at inference.
1) Action Chunking Transformer:
- The implementation replaces ACT's stock ResNet vision encoder with the pretrained vision encoder and adds the pretrained tactile encoder.The tactile and visual encodings are concatenated before entering the transformer decoder.Source: 1) Action Chunking Transformer · pp. 3–4
- ACT's autoencoder predicts action sequences, sets latent z to 0 at inference, and combines timestep predictions through weighted temporal ensembling.The network is queried at each timestep.Source: Figure 3 · p. 4
- ACT predicts relative rather than global goal positions, then adds the current position at inference to recover global coordinates.The paper says delta-position prediction avoids biases detrimental to visual servoing; temporal ensembling uses k = 0.25.Source: 1) Action Chunking Transformer · p. 4
2) Diffusion Policy:
Diffusion Policy conditions action-sequence denoising on visual, tactile, and positional observations. Demonstrations were collected by teleoperating a robot with multiple camera views and a protected GelSight sensor, whose processed strain maps were used as tactile input.
2) Diffusion Policy:
- Diffusion Policy uses a 1D temporal CNN to predict noise for action sequences conditioned on encoded visual, tactile, and positional observations.Pretrained vision and tactile encoders replace the stock vision encoder and supply tactile features.Source: 2) Diffusion Policy · p. 4
- At inference, the policy denoises for 10 steps and executes 8 of its 20 predicted actions before replanning.The observation horizon is 1, action prediction horizon is 20, and training uses 100 denoising steps.Source: 2) Diffusion Policy · p. 4
C. Data Collection
- Teleoperated demonstrations used a Franka Emika Panda, with a VR controller providing end-effector goals to an impedance controller.Operators viewed the workspace directly rather than through VR.Source: C. Data Collection · p. 4
- Six RGB camera views, robot end-effector state, and teleoperation goal state were recorded at 10 Hz.The cameras included a wrist-mounted view, four around the workspace, and one overview camera.Source: C. Data Collection · p. 4
- A 1 mm Ecoflex cover extended GelSight pad life by an order of magnitude, while reducing small-scale feature detection but preserving large-scale contact cues.The uncovered pad lasted around 30 cable-plugging episodes before tearing; the cover withstood a few hundred.Source: C. Data Collection · p. 4
- The imitation pipeline used 240 x 320 strain maps in x, y, and z; prior work reported processed strain outperformed GelSight RGB input.Figure 5 maps normal strain to brightness and tangential x/y strain to color.Source: C. Data Collection · pp. 4–5, Figure 5 · p. 5
A. Cable Plugging
Visuo-tactile pretraining substantially improved vision-only cable-plugging performance and modestly improved visuo-tactile policies. The evaluation also examined latent alignment, force, and sensitivity to policy and sensor conditions.
A. Cable Plugging
- The cable-plugging evaluation used 100 demonstrations, 80 for training, fixed cable and plug starts, 2.5 mm action noise, and 20 runs.Source: A. Cable Plugging · p. 5
- Pretraining aligned visual and tactile embeddings by timestep, with post-training clusters separating cable-grasped from ungrasped states.Source: A. Cable Plugging · p. 5, Figure 6 · p. 5
- Pretrained visuo-tactile ACT reached 95% success, versus 20% for vision-only ACT and 45% for vision-only Diffusion Policy.Source: A. Cable Plugging · p. 5
- Pretraining reduced median tangential strain by 8% and 13% for visuo-tactile ACT and Diffusion Policy, and 20% and 17% for vision-only policies.Source: A. Cable Plugging · p. 5
- Diffusion Policy was less sensitive than ACT to pretraining, but its stochastic, larger motions increased strain about 15% versus pretrained visuo-tactile ACT.Source: A. Cable Plugging · p. 5
- Tactile-data pretraining significantly improved vision-only performance: ACT rose from 20% to 85%, and Diffusion Policy from 45% to 75%, nearly matching visuo-tactile agents.Source: A. Cable Plugging · p. 5
- Tactile-only Diffusion policies never completed plugging, repeatedly changing direction or dropping the cable; training and evaluation used different GelSight pads.Source: A. Cable Plugging · p. 6
B. Block Stacking
Visuo-tactile pretraining improved block-stacking performance for both visuo-tactile and vision-only agents, with vision-only agents reaching results on par with visuo-tactile agents. Tactile input could nevertheless hurt performance in the simple cube task.
B. Block Stacking
- The evaluation included thin rectangular-block stacking requiring precise force control and cube stacking as an inherently non-tactile task.Source: B. Block Stacking · p. 6
- Pretraining improved both policy types across the stacking tasks, and pretrained vision-only agents achieved results on par with visuo-tactile agents.Source: B. Block Stacking · p. 6
- In simple cube stacking, tactile input appeared to reduce performance, especially without pretraining; the authors suggest overfitting as a likely explanation.Source: B. Block Stacking · p. 6
- GelSight wear can change sensor properties between data collection and evaluation, creating domain shift and contributing to overfitting.Source: B. Block Stacking · p. 6
V. CONCLUSIONS
The paper concludes that contrastive visuo-tactile pretraining can improve imitation-learning performance, including for vision-only agents trained with tactile data. Its pretraining data were task-specific, limiting applicability across tasks.
V. CONCLUSIONS
- Visuo-tactile pretraining improved performance, including cable-plugging success from 90% to 95% and cube-stacking success from 30% to 95% for visuo-tactile agents.Source: §V · p. 6
- Tactile-data pretraining significantly improved vision-only policies, which reached parity with or sometimes surpassed visuo-tactile counterparts; USB plugging rose from 20% to 85%.Source: §V · p. 6
- Task-specific pretraining data limit applicability and require collecting tactile observations for each task; broader pretraining is left for future work.Source: §V · p. 6