Source-linked AI summary
FTP-1: A Generalist Foundation Tactile Policy Across Tactile Sensors for Contact-Rich Manipulation
Chengbo Yuan, Zicheng Zhang, Mingjie Zhou, Wendi Chen, Yi Wang, Zhuoyang Liu, Dantong Niu, Shuo Wang, Hui Zhang, Wenkang Zhang, Yingdong Hu, Yuanqing Gong, Wanli Xing, Chuan Wen, Cewu Lu, Kaifeng Zhang, Yang Gao
TL;DR
Tactile policies have remained tied to fixed sensors and embodiments because tactile signals differ substantially across hardware. FTP-1 addresses this gap with unified morphology-aware tactile tokens and a shared tactile Transformer trained on heterogeneous data, improving seen-sensor performance and transferring to unseen sensors. It provides a shared model-level starting point for tactile policy learning, while remaining limited in tactile control coverage and dataset scale.
Problem
Tactile policies remain tied to limited sensor configurations, observation formats, and embodiments, leaving cross-sensor generalist learning underexplored.
Method
FTP-1 uses heterogeneous tactile encoders, morphology-aware tactile tokens, and a shared tactile expert trained on 26 sources spanning 21 sensors.
Results
+17.2% success-rate gain is reported on seen sensor setups, with a +31% success-rate gain on 2 previously unseen tactile sensors.
Takeaways & Limitations
FTP-1 establishes a shared model-level starting point for tactile policy learning across sensors and embodiments.
Takeaways & Limitations
FTP-1 does not yet address tactile- or force-based servoing and control, and its pretraining dataset remains limited in scale and diversity.
Abstract
from arXiv · showhide
Despite the success of vision-based generalist robotic policies, existing tactile-based policies remain tied to fixed embodiments and sensor setups. This is because tactile signals are highly heterogeneous across hardware, making cross-sensor generalization difficult. We present FTP-1,the first generalist foundation tactile policy pretrained to acquire transferable tactile manipulation abilities across diverse sensors and embodiments. FTP-1 supports varied tactile inputs, including image-, array-, and state-based signals, by using heterogeneous encoders to project them into unified morphology-aware latent tokens that are jointly modeled by a shared tactile Transformer expert. Pretrained on around 3,000 hours of tactile manipulation data aggregated from 26 data sources, spanning human and robot demonstrations across 21 sensors, FTP-1 learns tactile skills that transfer beyond the sensors seen during pretraining. Across downstream finetuning experiments spanning 5 hardware configurations, FTP-1 improves contact-rich manipulation on seen sensor setups by +17.2% and, surprisingly, transfers to two previously unseen tactile-sensor setups, achieving a +31% gain in success rate. FTP-1 establishes the first unified foundation baseline for tactile manipulation, providing future tactile policies with a shared model-level starting point. Pretrained models, datasets, training code and more visualization at https://ftp1-policy.github.io.
1 Introduction
FTP-1 addresses the underexplored problem of generalist tactile policy learning by combining heterogeneous tactile data and sensors into a transferable foundation policy. It reports improved contact-rich manipulation on seen setups and transfer to unseen sensors.
- Vision-based generalist policies have advanced scalable robot learning, but tactile policies remain tied to limited sensors, observation formats, and embodiments.Tactile heterogeneity in modality, resolution, morphology, and contact response makes cross-sensor representation sharing difficult.
- FTP-1 is presented as the first generalist foundation tactile policy for transferable manipulation across diverse sensors and embodiments.It supports image-, array-, and state-based tactile inputs.
- FTP-1 uses morphology-aware tactile tokenization and a shared tactile Transformer expert to learn reusable representations across sensors.MTTS provides a unified interface, while heterogeneous encoders project different tactile inputs into common tokens.
- FTP-1-Dataset aggregates 26 data sources and approximately 3,000 hours of human and robot demonstrations across 21 tactile sensors.The data are standardized through the MTTS interface for unified pretraining.
- +17.2% success-rate gain is reported on 3 seen sensor setups, while 2 unseen tactile sensors receive a +31% success-rate gain.The evaluation spans 5 hardware configurations and 14 contact-rich tasks.
2 FTP-1: Generalist Foundation Tactile Policy
FTP-1 unifies heterogeneous tactile observations and robot action spaces through morphology-aware tokens, sensor-specific encoders, shared tactile processing, and a unified action representation. Its architecture combines tactile, vision-language, and proprioceptive information for action generation.
- FTP-1 predicts action chunks from language, RGB observations, proprioception, and tactile observations.The policy maps these inputs to a predefined unified action space.
- MTTS interface: MTTS organizes tactile signals into functional-area tokens with shared embeddings, providing a common interface across sensor morphologies.The definition uses 24 functional areas and distinguishes left- and right-hand tactile tokens.
- Heterogeneous tactile encoders: Heterogeneous encoders tokenize image-, array-, and state-based tactile inputs into MTTS.The encoders use sensor-specific ViTs, CNNs, or Fourier-encoded MLPs according to observation type.
- Shared tactile expert: A shared tactile expert processes tactile tokens alongside vision-language and proprioceptive information for action generation.The independent tactile expert supports reuse on unseen sensors and avoids disturbing pretrained vision-language knowledge.
- FTP-1-Dataset: The pretraining dataset covers 26 sources and 21 tactile sensors spanning image-, array-, and state-type modalities.All tactile annotations are organized according to MTTS functional areas.
3 Finetuning Experiment
FTP-1 is evaluated through finetuning across simulation and real-robot embodiments, using matched baselines to isolate tactile input, integration architecture, and pretraining. It achieves the strongest reported performance in both simulation and real-world seen-sensor settings.
- Baselines: The comparison includes π0.5, Tactile-VLA, and FTP-π0.5 to separate the effects of tactile sensing, tactile-expert integration, and FTP-1 pretraining.All methods use comparable model scale, finetuning data, and training protocols.
- Evaluation setup: FTP-1 is evaluated on UniVTAC simulation and two real-world embodiments using six contact-rich simulation tasks and long-horizon, pressing, and force-control behaviors.The seen-sensor setups use GelSight-Mini and Sharpa DTC, both included during pretraining.
- Simulation results: 66.66% overall success and 59.5% excluding lift tasks make FTP-1 the best UniVTAC method, outperforming the second-best method by about +17.5% under both metrics.Lift Bottle and Lift Can are excluded in the second metric because they can largely be solved without tactile input.
- Real-world results: 62.5% average success makes FTP-1 the best method in real-world evaluation, ahead of π0.5 at 45.3%.The results concern the seen tactile-sensor setups reported in Table 2.
- Behavioral analysis: FTP-1 produces more stable and smoother actions than tactile baselines when contact conditions change, including pressing-force maintenance and reactive force adjustment.The comparison covers Sharpa&Dexmate tasks and bottle-cap interaction.
4 Experiment of Transferable Tactile Manipulation Abilities
The paper tests whether heterogeneous tactile pretraining transfers to sensors absent during pretraining and whether its gains reflect tactile knowledge rather than data-distribution similarity. FTP-1 performs best on unseen setups, while controlled comparisons support transferable tactile knowledge as the explanation.
- Motivation: The study asks whether large-scale heterogeneous pretraining transfers to unseen tactile sensors, a question left unclear by prior in-domain pretraining results.This extends evaluation beyond sensors included during pretraining.
- Unseen-sensor setup: Unseen-sensor finetuning reuses the tactile expert, shared image-tactile Transformer components, and functional-area embeddings while training the sensor-specific encoder from scratch.The setup evaluates FlexivXense image sensors and TactileUMI Contactile array sensors.
- Behavioral transfer: FTP-1 shows recovery and reactive insertion behaviors, including slowing insertion after tactile misalignment, while other models are less stable or fail to maintain contact and pressing force.The unseen-sensor tasks also indicate improved data efficiency under only 100 demonstrations for Insert USB.
- Controlled comparison: A no-tactile-pretraining control uses the same data and optimization settings as FTP-1 but adds tactile architecture only during finetuning, isolating tactile pretraining effects.The resulting NTP-1 model is evaluated on UniVTAC and unseen FlexivXense.
- Ablation results: On FlexivXense, FTP-1 exceeds NTP-1 by +37.5%, supporting the hypothesis that pretrained tactile knowledge—not only dataset similarity—improves transfer.NTP-1 can outperform FTP-π0.5 on UniVTAC, but still remains below FTP-1.
5 Related Work
Prior generalist policies transfer across tasks, embodiments, and environments but largely omit tactile sensing, while tactile methods typically remain tied to specific sensors, tasks, or embodiments. FTP-1 extends the generalist-policy paradigm to tactile manipulation and evaluates it across seen and unseen sensors.
- Existing approaches: Generalist robot policies demonstrate broad transfer but largely omit tactile sensing, leaving tactile manipulation outside the prevailing generalist paradigm.Existing tactile policy and VTLA methods address contact-rich manipulation but are usually sensor-, task-, or embodiment-specific.
- FTP-1’s positioning: FTP-1 treats tactile manipulation skills as pretrainable and transferable across heterogeneous sensors and embodiments rather than as a sensor-specific policy add-on.Its evaluation covers both seen and unseen tactile-sensor setups.
6 Conclusion
FTP-1 is presented as a generalist tactile foundation policy that transfers manipulation skills across diverse sensors and embodiments. The paper identifies hardware heterogeneity as a central challenge and positions FTP-1 as a shared starting point for tactile policy learning, while noting important remaining limitations.
- FTP-1 is a generalist foundation tactile policy pretrained for diverse sensors and embodiments.
- FTP-1 uses heterogeneous tactile encoders and a shared tactile expert to learn transferable tactile manipulation skills across sensors and embodiments.
- The method provides a shared model-level starting point for tactile policy learning.
- FTP-1 does not yet address tactile- or force-based servoing and control, and its pretraining dataset remains limited in scale and diversity.
- Tactile hardware heterogeneity makes cross-sensor representation sharing challenging, despite tactile signals improving contact-rich manipulation.
B Details of FTP-1 Architecture
FTP-1 unifies heterogeneous robot action spaces and tactile observations before joint policy modeling. Its architecture combines fixed-layout action prediction, modality-specific tactile encoders, functional-area embeddings, and proprioceptive conditioning.
- Unified Action Space (UAS): FTP-1 represents different robot commands in a fixed-length sparse Unified Action Space for unified training across embodiments.Unsupported action dimensions are masked from the loss.
- Unified Action Space (UAS): The Unified Action Space includes left-arm, right-arm, head-pose, and supplementary control slots, with supported dimensions filled for each embodiment.
- Unified Action Space (UAS): FAAS unifies control action spaces for different end-effectors, while canonical hand-joint slots provide a common hand-control layout.
- Heterogeneous Tactile Encoders: Image, array, and state tactile inputs are processed by different encoders into tactile tokens, which are normalized, augmented with function-area embeddings, and projected to the tactile expert dimension.
- Other Details: Proprioception is Fourier-encoded and injected through adaptive RMSNorm, while the tactile expert uses a Transformer trained with AdamW.The paper reports better generalization and robustness from adaptive RMSNorm than from an independent proprioceptive token.
C Details of FTP-1 Pretraining Datasets
FTP-1-Dataset aggregates heterogeneous tactile manipulation data across human and robot domains, sensors, tasks, and embodiments. Source resampling balances the mixture while preserving broad coverage of tactile modalities.
- The pretraining dataset aggregates 26 data sources spanning human demonstrations, dexterous-hand data, parallel-gripper data, and UMI-style data.
- FTP-1-Dataset spans 21 tactile sensors across image-, array-, and state-type modalities.
- The resampled mixture contains approximately 20% human-hand data, 30% dexterous-hand data, and 50% gripper data.
- The dataset comprises around 3,000 hours of tactile-based manipulation data covering diverse contact conditions, tasks, scenes, and embodiments.
- Table 4 groups sources by end-effector type and specifies sampling ratios used to address data imbalance across sources.
D Training Details
FTP-1 pretraining and finetuning use distributed GPU infrastructure and fixed optimization schedules. Evaluation covers multiple hardware setups, with unseen components explicitly identified relative to heterogeneous pretraining.
- Pretraining: FTP-1 pretraining runs for 50k steps on 48 NVIDIA H20 GPUs with global batch size 768 and a learning rate decayed from 1 × 10−4 to 5 × 10−5.
- Pretraining Infrastructure: The training infrastructure assigns different data domains to separate GPUs while merging gradients from shared modules for joint updates.
- Finetuning: Downstream finetuning uses 8 NVIDIA A800 GPUs for 20k steps per dataset, with batch size 64 and a learning rate decayed from 5×10−5 to 5×10−6.
- Evaluation: Evaluation uses five hardware setups, and components marked unseen are not covered by FTP-1 large-scale heterogeneous pretraining.
- Evaluation: The detailed hardware setups are provided in Table 5.
E.2 Evaluation Setups of Seen Sensors
The seen-sensor evaluation covers three embodiments and multiple contact-rich tasks across simulation and hardware. FTP-1 is finetuned using task-specific demonstrations, with evaluation protocols tailored to each setup.
- Evaluation scope: The evaluation uses UniVTAC simulation, Sharpa North, and Sharpa&Dexmate, covering GelSight-Mini and Sharpa DTC tactile sensors.Because these sensors appear in pretraining, both the tactile tokenizer and tactile expert are initialized from FTP-1 checkpoints.
- UniVTAC: UniVTAC tasks are finetuned with 50 official demonstrations per task, excluding Grasp Classify and Insert HDMI from evaluation.Grasp Classify was already perfectly solved, while Insert HDMI demonstrations provided insufficient contact feedback because of motion-planning-based collection.
- Sharpa North: Sharpa North evaluates Draw Balloon, Fix Hand, and Twist Cap using 250–400 demonstrations across long-horizon manipulation tasks.Fix Hand is evaluated in Tear and Finish stages, separating sticker removal from final finger attachment.
- Sharpa&Dexmate: Sharpa&Dexmate adds Flip Book and Wipe Dish, testing contact-rich pressing and force-control behaviors with 100 demonstrations per task.These tasks complement the evaluation suite with additional contact-rich behaviors.
- Rollout visualizations: Figures 11–13 show five key frames from successful rollouts for the seen-sensor evaluation tasks.Each row corresponds to one task.
E.4 Details of Similar Sensors for Unseen Sensor Evaluation
The unseen-sensor evaluation identifies the closest pretraining sensor for each downstream sensor and provides rollout visualizations for the corresponding setups. Additional material includes a tactile-knowledge ablation on UniVTAC.
- Similar sensors: GelSight-Mini is the closest pretraining counterpart for image-type Xense, while AetherGlove is the closest counterpart for array-type Contactile.These similarities are used to relate the unseen downstream sensors to sensors present during pretraining.
- Similar sensors: Figure 10 depicts the similar sensors between the pretraining dataset and downstream evaluation setups.The figure supports comparison of sensor relationships used in unseen-sensor evaluation.
- Unseen-sensor rollouts: Figures 14 and 15 provide representative rollout visualizations for the unseen-sensor evaluation setups.The visualizations cover FlexivXense tasks Insert Hanoi and Insert USB, and the TactileUMI Wipe Board task.
- Ablation: Table 6 reports success rates for an ablation of pretrained tactile knowledge on the UniVTAC simulation benchmark.The table also reports Avg. w/o Lifts, which excludes Lift Bottle and Lift Can, and highlights the best and second-best results.
- Reference rollouts: Figure 13 shows representative Sharpa&Dexmate rollouts for Flip Book and Wipe Dish, while Figure 12 covers Sharpa North tasks.The figures include Draw Balloon, Fix Hand (Tear), Fix Hand (Finish), Twist Cap, Flip Book, and Wipe Dish.