Source-linked AI summary
T-Rex: Tactile-Reactive Dexterous Manipulation
Dantong Niu, Zhuoyang Liu, Zekai Wang, Boning Shao, Zhao-Heng Yin, Anirudh Pai, Yuvan Sharma, Stefano Saravalle, Ruijie Zheng, Jing Wang, Ryan Punamiya, Mengda Xu, Yuqi Xie, Yunfan Jiang, Letian Fu, Konstantinos Kallidromitis, Matteo Gioia, Junyi Zhang, Jiaxin Ge, Haiwen Feng, Fabio Galasso, Wei Zhan, David M. Chan, Yutong Bai, Roei Herzig, Jiahui Lei, Li Fei-Fei, Ken Goldberg, Jitendra Malik, Pieter Abbeel, Yuke Zhu, Danfei Xu, Linxi Fan, Trevor Darrell
TL;DR
T-Rex addresses the scarcity of tactile data and the frequency mismatch between tactile-reactive control and conventional VLA models. It introduces a 100-hour tactile dataset, a variable-rate MoT with temporal tactile encoding, and a 12-task benchmark. The resulting policy achieves a 30% average success-rate advantage over existing baselines, while hardware limitations remain a scope boundary.
Problem
VLA manipulation models often lack tactile information, while tactile-reactive control requires high-frequency responses and diverse synchronized data.
Method
T-Rex combines a 100-hour tactile-synchronized robot dataset, variable-rate MoT experts, spatio-temporal tactile VQ-VAE encoding, and tactile-grounded mid-training.
Results
30% average success-rate improvement was achieved over existing dexterous and tactile-aware VLA baselines across 12 real-world tactile-reactive tasks.
Takeaways & Limitations
Tactile-grounded mid-training and asynchronous tactile refinement support contact-rich dexterous manipulation with improved data efficiency.
Takeaways & Limitations
Long-horizon tasks with precise contact coordination remain difficult to teleoperate, and hardware is limited by sensor distortion, calibration drift, and absent dense palm sensing.
Abstract
from arXiv · showhide
The ability to react dynamically to tactile signals has long been considered crucial to agile human-level dexterity. Yet contemporary learning-based Vision-Language-Action (VLA) models for robotic manipulation generally either overlook the tactile modality or are limited to encoders with static cues, due in part to the scarcity of diverse training data and standardized evaluation, architectural constraints in current VLA models, and limitations of static tactile encoders. In this paper, we push the frontier of tactile-reactive manipulation by addressing all of these limitations. We propose a large-scale, 100-hour tactile-rich dataset collected via a novel, data-efficient recipe that prioritizes elementary motor primitives. To effectively exploit naturally high-frequency touch signals without sacrificing the existing capabilities of existing VLAs, we introduce a variable-rate Mixture-of-Transformers (MoT) architecture equipped with a novel temporal tactile VQ-VAE encoder. We demonstrate the effectiveness of tactile-reactive policies on 12 manipulation tasks requiring delicate force control and deformable object manipulation, achieving over 30% higher average success rate than the strongest baseline.
1 Introduction
T-Rex addresses scarce tactile data and frequency-mismatched VLA architectures for tactile-reactive dexterous manipulation. It contributes a tactile-synchronized dataset, variable-rate MoT model, and benchmark showing substantially higher success than existing baselines.
- Tactile-reactive manipulation requires immediate closed-loop responses to fine-grained signals, beyond conventional vision-based control speeds.
- Tactile policy pre-training data remain scarce, motivating a 100-hour dataset organized around compositional motor primitives and object interactions.
- Variable-rate MoT separates low-rate action generation from high-rate tactile residual refinement and uses spatio-temporal VQ-VAE representations.
- 30% higher average success rate was achieved across 12 contact-rich dexterous manipulation tasks than existing dexterous-hand foundation models.
2 Related Work
Prior work has explored tactile representations, multimodal fusion, unified robot models, and human-video pre-training, but tactile-aware dexterous manipulation lacks a canonical large-scale pre-training and mid-training recipe.
- Earlier tactile policies used shallow MLPs, while later methods modeled poses, future observations, actions, and low-level tactile refinement.
- Recent VLA systems add touch as a modality, align tactile features with visual-language latents, or design architectures for force and contact signals.
- Existing work lacks a canonical tactile-aware recipe benefiting from large-scale pre-training and mid-training for dexterous manipulation.
- Unified multimodal robot models increasingly combine language, vision, video, and actions, with dedicated experts supporting continuous action generation and future prediction.
- Human egocentric video datasets support representation learning, affordance extraction, VLA pre-training, co-training, planning, and human-to-robot transfer.
3 The T-Rex Dataset
The T-Rex Dataset provides broad tactile-rich bimanual manipulation coverage through diverse objects, motor primitives, and synchronized multimodal recordings. Its statistics emphasize contact-rich behavioral diversity rather than narrow task demonstrations.
- The dataset contains 100 hours of tactile-synchronized bimanual manipulation data spanning 200+ daily objects and 22 motor primitives.
- The dataset is designed for broad coverage of contact-rich interactions through object categories, motor primitives, and long-tail individual-object distributions.
- Each episode synchronizes RGB observations, tactile signals, robot states, actions, and language instructions.
- Its coverage addresses the limited tactile-rich dexterous interaction coverage of datasets focused on parallel grippers or grasp-centric hands.
4 Tactile-Reactive Dexterous Manipulation
T-Rex combines multimodal flow-based action generation with asynchronous tactile refinement: visual-language context supports low-frequency planning, while tactile signals refine actions at higher frequency. A staged training recipe transfers human visuomotor priors into robot-executable contact control.
- Policy formulation: The policy conditions future action chunks on RGB observations, language, tactile force history, and tactile deformation maps.
- Model architecture: T-Rex uses a Mixture-of-Transformer-Experts backbone with latent, action, and tactile experts for visual prediction, action denoising, and tactile refinement.
- Tactile encoding: A per-finger VQ-VAE compresses recent force history, while current force and deformation-map encoders preserve instantaneous contact and spatial signals.
- Asynchronous refinement: The action expert denoises an upper flow segment, then the tactile expert reuses cached visual-language context to complete lower-segment denoising with real-time tactile observations.
- Asynchronous refinement: This cascaded mechanism enables dynamic responses to real-time tactile feedback for contact-rich, highly dexterous tasks.
- Training objective: Training jointly optimizes action and tactile experts against a shared flow target while adding future-frame visual prediction.
- Training recipe: The three-stage recipe uses 22,889 hours of human video pre-training, 100 hours of tactile robot mid-training, and approximately 100 task demonstrations for optional post-training.
5 Experiments
T-Rex is evaluated on 12 tactile-reactive tasks using randomized trials and average success rate, outperforming dexterous manipulation and VLA baselines. Ablations examine tactile representations, asynchronous refinement, split-step choices, dataset efficiency, and training stages.
- Evaluation Protocol: T-Rex is evaluated on 12 tactile-reactive tasks with randomized object positions and rotations, using 16 trials per task and average task success rate.Multi-stage tasks use progress-based rubrics to capture partial completion.
- Main Results: T-Rex achieves the highest average success rate across task categories, outperforming the strongest baseline by more than 30%.The benchmark covers delicate force control, deformable object handling, and joint force-deformation reasoning.
- Main Results: Large-scale pre-training improves dexterous manipulation, while pretrained VLA models still struggle with precise contact adjustment and force-sensitive behaviors.EgoScale outperforms methods pretrained only on task-specific real-robot data, but tactile feedback remains important for contact-rich manipulation.
- Main Results: Naively conditioning a pretrained VLA on tactile signals can degrade performance, whereas T-Rex combines large-scale pre-training, tactile-grounded mid-training, and tactile-reactive control.The result highlights the importance of how tactile information is integrated.
- Ablation Studies: Asynchronous tactile refinement improves performance over synchronous refinement by decoupling low-frequency visuomotor planning from high-frequency tactile control.An intermediate denoising split performs best; overly small or large splits limit either visuomotor priors or tactile integration.
- Ablation Studies: The 100-hour tactile-grounded dataset improves generalization, zero-shot transfer, and low-data performance compared with matched task-specific data and no mid-training.Varying post-training demonstrations from 10 to 200 shows reduced downstream data requirements for contact-rich manipulation.
- Ablation Studies: The training-recipe ablation isolates human pretraining as a source of semantic grounding and coarse visuomotor priors, alongside tactile-grounded mid-training.The study compares variants with and without human pretraining and tactile-grounded mid-training on six robot tasks.
6 Conclusion
T-Rex combines a Mixture-of-Transformer-Experts model, asynchronous tactile refinement, dynamic tactile VAE encoding, and a 100-hour tactile-synchronized dataset. Across 12 real-world tasks, it improves average success rate by 30% and data efficiency.
- Conclusion: T-Rex combines asynchronous tactile refinement and dynamic tactile VAE encoding in a Mixture-of-Transformer-Experts model.The training pipeline uses general human video pre-training followed by tactile-grounded mid-training.
- Conclusion: T-Rex is trained with an open-source 100-hour tactile-synchronized dexterous manipulation dataset and evaluated on 12 real-world tactile-reactive tasks.The dataset supports the model's tactile-grounded mid-training stage.
- Conclusion: T-Rex outperforms existing dexterous and tactile-aware VLA baselines by an average success rate of 30% and significantly improves data efficiency.The conclusion summarizes the reported benchmark and efficiency outcomes.
7 Limitation and Future Work
T-Rex's future work is bounded by difficult long-horizon contact coordination and hardware limitations. Proposed directions include interaction-based refinement and richer, unified tactile sensing.
- Future Work: Long-horizon tasks with precise contact coordination and tight tolerances remain settings where teleoperation is difficult.Future work could integrate reinforcement learning or online interaction-based refinement.
- Limitations: Hardware limitations include sensor distortion, calibration drift across devices, and the absence of dense palm sensing for whole-hand manipulation.Future work may explore unified representations across heterogeneous tactile sensors and richer whole-hand tactile hardware.
A Model and Training Details
T-Rex combines asynchronous slow visual-language and fast tactile experts with temporal tactile encoding for high-frequency manipulation. Training jointly optimizes action, tactile, and future-frame prediction objectives, while deployment reuses cached slow-stream computation.
- Asynchronous Cascaded Flow Matching: The cascaded flow-matching design separates low-frequency visual-language processing from high-frequency tactile-reactive inference.The inference procedure formalizes asynchronous slow and fast streams with shared intermediate state and key-value cache.
- Explicit Conditioning and Training Objectives: The action expert uses multimodal visual-language context, whereas the tactile expert uses tactile tokens and detached slow-stream state without raw visual observations.Both experts regress the shared velocity target while receiving distinct conditioning contexts.
- Explicit Conditioning and Training Objectives: The total objective combines action loss, weighted tactile loss, and future-frame visual prediction loss with λ_tac = 1.0 and λ_future = 0.5.The objective is L = L_act + λ_tacL_tac + λ_futureL_future.
- Delay Augmentation: Delay augmentation randomly shifts tactile-frame indices by δ ∈ {0, 4, 8, 12} to match deployment-time visual-tactile staleness.This addresses asynchronous fast ticks occurring at intra-chunk offsets.
- Runtime Synchronization: Deployment amortizes computation by reusing the visual tower, latent expert, and action expert during fast ticks, leaving the lightweight tactile expert to perform K_fast Euler steps.A single-threaded request socket and execution lock serialize expert access to shared state.
- Tactile Encoders: The temporal force encoder uses a VQ-VAE to discretize continuous multi-finger force sequences into compact tactile tokens.The encoder processes force windows of T = 16 frames and uses a learned codebook of size K = 64.
- Tactile Encoders: A separate deformation encoder represents fingertip skin-displacement maps, capturing contact geometry such as edges, slip, and shear patterns.It uses a lightweight single-channel convolutional encoder with shared convolutional weights across fingers.
E Implementation Details of Baselines
The evaluation compares T-Rex with six reproduced baselines across the same 12 contact-rich tasks. Baselines span visuo-tactile imitation, slow-fast tactile control, tactile-aware VLAs, and visual or tactile-conditioned foundation models.
- Baseline Coverage: T-Rex is compared with 6 baselines across 12 contact-rich manipulation tasks using reproduced implementations.The baselines are evaluated under unified task and protocol settings where specified.
- Visuo-Tactile Imitation: ViTacFormer uses visual-tactile fusion and auxiliary future tactile prediction in task-specific ACT policies.The reproduction trains separate policies for each task with 100 demonstrations and 100 training epochs.
- Visuo-Tactile Imitation: Reactive Diffusion Policy combines a low-frequency latent diffusion policy with a high-frequency tactile-reactive controller.It is reproduced as a separate task-specific baseline for each of the 12 tasks.
- Tactile-Aware VLAs: Tactile-VLA integrates touch through multimodal fusion and hybrid force-position control.Its tactile encoder is adapted from GelSight images to 6D force/torque vectors from all 10 fingers.
- Foundation-Model Baselines: EgoScale is reproduced from a pretrained GR00T N1.7 checkpoint and fine-tuned separately for each task.The reproduction uses relative end-effector actions for bimanual arms and 22-DoF hand joint actions.
- Foundation-Model Baselines: The π0.5 comparison includes both visual-only and tactile-conditioned variants.The tactile variant concatenates single-step 6D force/torque observations from all 10 fingers to the state input.
F Evaluation Tasks
The benchmark contains 12 contact-rich dexterous tasks covering force regulation, tactile deformation, insertion, object manipulation, and bimanual coordination. Tasks use additive rubrics that score intermediate actions and final outcomes.
- Benchmark Scope: The 12 tasks cover force-reactive and tactile-deformation-sensitive manipulation situations.Examples include fragile-object handling, controlled pressure, slip resistance, and tactile identification of objects or mechanisms.
- Single-Object Manipulation: Flip Page requires lifting one sheet, moving it across the spine, and smoothing it flat on the opposite side.Its rubric separately scores single-finger contact, turning the page, and completing the flip.
- Force-Sensitive Tasks: Transfer Egg tests fragile-object grasping, transport, and intact release between trays.The rubric awards separate credit for contact, lifting, transport, and release without cracking the egg.
- Bimanual and Deformable Interaction: Wipe Plate requires bimanual stabilization of a plate while a cloth contacts and removes its stains.The additive rubric emphasizes pressing the plate steady, wiping away all visible ink, and returning the cloth.
- Bimanual and Deformable Interaction: Apply Toothpaste tests coordinated grasping, squeezing a bead onto bristles, and returning both objects.The task requires separate hands to manipulate the toothbrush and toothpaste tube.
- Bimanual and Deformable Interaction: Split Cup requires stabilizing a nested stack while twisting and separating exactly one cup.The rubric scores grasping the stack, grasping the top cup, separation, and intact holding.
- Tactile Identification: Sort Mahjong requires tactilely identifying a face-down tile and placing it into the matching compartment.The rubric separately scores opening the correct compartment, placement, and closure.
- Precision Interaction: Open Lock tests key and padlock handling, keyhole insertion, and rotation to release the shackle.The task combines precise alignment with coordinated two-hand manipulation.
G T-Rex Dataset
The T-Rex Dataset is designed for large-scale mid-training of tactile-reactive dexterous policies through synchronized multimodal episodes, broad object–primitive coverage, and diverse scenes. Its pipeline includes data cleaning, language annotation, and controlled laboratory release practices.
- Dataset Schema: Each demonstration is stored as a time-aligned bundle of synchronized streams collected through the teleoperation stack.Modalities include RGB, bimanual proprioception, wrist poses, and per-fingertip deformation and force observations.
- Data Taxonomy: The taxonomy combines 207 household objects with 22 motor primitives, retaining physically feasible object–primitive pairs.The initial 207 × 22 design space contains 4,554 candidate combinations before infeasible pairs are pruned.
- Scene Diversity: Scene diversity uses six tabletop backdrops and more than 210 possible distractor objects, typically placing 0–5 distractors beside the target.These variations change surrounding-object arrangements across demonstrations.
- Quality Control: Data cleaning removes unstable tactile measurements, corrupted streams, teleoperation failures, extreme joint velocities, and other artifacts.The filtering stage targets quality and consistency for policy learning.
- Language Annotation: A commercial vision-language model generates one imperative instruction per episode from sampled head-camera frames and teleoperation labels.The labels identify the target object and motor primitive used in the episode.
- Ethical Considerations and Release: The released demonstrations are collected in a controlled laboratory environment without third-party human subjects in the RGB streams.Teleoperator hands visible during reset interactions are clipped, and commodity objects contain no personally identifying information.
H Failure Case Analysis
Failure analysis identifies recurring weaknesses in visual alignment, fine-grained dexterity, precise positioning, finger coordination, force control, and temporal tactile conditioning across contact-rich tasks.
- Object collision during screw-lightbulb insertion prevented subsequent rotation, indicating insufficient fine-grained visual alignment and overly rapid motion.
- The model grasped the key but failed to maintain a secure grip, revealing limited fine-grained dexterity for small-object manipulation.
- Imprecise egg placement exposed positioning deficiencies associated with behavioral-cloning distribution shift.
- Incorrect thumb placement caused unintended contact with a neighboring mahjong tile, highlighting insufficient individual-finger coordination.
- Excessive force while squeezing toothpaste caused failure to catch it with the toothbrush, constraining deformable-object manipulation.
- Nonuniform extraction force in the card task indicates a need for stronger temporal tactile conditioning during sliding motions.