Source-linked AI summary
ViTacFormer: Learning Cross-Modal Representation for Visuo-Tactile Dexterous Manipulation
Liang Heng, Haoran Geng, Kaifeng Zhang, Pieter Abbeel, Jitendra Malik
TL;DR
Dexterous manipulation needs tactile information for fine-grained control, but effective cross-modal representations for visuo-tactile manipulation remain lacking. ViTacFormer fuses vision and touch with cross-attention, predicts future tactile signals autoregressively, and trains with a two-phase curriculum. It reports roughly 50% higher success rates than strong baselines and completes an 11-stage task with over 2.5 minutes of continuous manipulation.
Problem
Effective cross-modal representations for visuo-tactile dexterous manipulation remain lacking despite the importance of tactile sensing for fine-grained control.
Method
ViTacFormer uses cross-attention to fuse visual and tactile inputs, autoregressively predicts future tactile signals, and applies a two-phase training curriculum.
Results
Approximately 50% higher success rates are reported across benchmarks, with 11 sequential stages and over 2.5 minutes of continuous manipulation on a real robot.
Takeaways & Limitations
The learned representation supports precise and adaptive manipulation with multi-fingered hands across challenging short- and long-horizon real-world tasks.
Takeaways & Limitations
The policy cannot autonomously generalize to novel tasks unseen during training and depends on time-consuming human teleoperation for data collection.
Abstract
from arXiv · showhide
Dexterous manipulation is a cornerstone capability for robotic systems aiming to interact with the physical world in a human-like manner. Although vision-based methods have advanced rapidly, tactile sensing remains crucial for fine-grained control, particularly in unstructured or visually occluded settings. We present ViTacFormer, a representation-learning approach that couples a cross-attention encoder to fuse high-resolution vision and touch with an autoregressive tactile prediction head that anticipates future contact signals. Building on this architecture, we devise an easy-to-challenging curriculum that steadily refines the visual-tactile latent space, boosting both accuracy and robustness. The learned cross-modal representation drives imitation learning for multi-fingered hands, enabling precise and adaptive manipulation. Across a suite of challenging real-world benchmarks, our method achieves approximately 50% higher success rates than prior state-of-the-art systems. To our knowledge, it is also the first to autonomously complete long-horizon dexterous manipulation tasks that demand highly precise control with an anthropomorphic hand, successfully executing up to 11 sequential stages and sustaining continuous operation for 2.5 minutes.
I. INTRODUCTION
ViTacFormer addresses limited generalization in dexterous manipulation by learning deep cross-modal representations that fuse vision and touch, predict future tactile states, and support imitation learning. Across real-world benchmarks, it improves success rates by roughly 50% and completes an 11-stage task with over 2.5 minutes of continuous manipulation.
- Existing dexterous manipulation methods often use simple hand configurations and underutilize tactile sensing, limiting generalization and fine-grained control.
- ViTacFormer learns cross-modal representations by fusing visual and tactile cues at every policy stage with cross-attention.
- A dedicated autoregressive tactile-prediction head anticipates future contact signals and uses them to generate actions.
- A two-phase curriculum uses ground-truth tactile inputs for the first 75% of training and predicted signals for the final 25%.
- Approximately 50% higher success rates are reported across benchmarks, including an 11-stage task completed with 2.5 minutes of continuous operation.
- The work contributes a real-world setup, dataset, and benchmark suite for evaluating visuo-tactile manipulation with bi-manual dexterous hands.
B. Manipulation with Tactile Signals
ViTacFormer targets the lack of effective cross-modal representations for visuo-tactile dexterous manipulation. It formulates imitation learning around multimodal observations and autoregressively predicts future tactile signals before generating actions.
- Prior tactile-manipulation studies often learn shallow tactile representations, leaving effective cross-modal learning for dexterous manipulation unresolved.
- The model is a conditional variational auto-encoder whose transformer encoder maps demonstrations and proprioception to an action-style variable.
- ViTacFormer combines cross-attention-based fusion with autoregressive future-tactile forecasting for action generation.
- The policy maps proprioception, visual observations, and fingertip tactile observations to actions while imitating expert trajectories.
- Evaluation measures manipulation-task success under diverse short- and long-horizon conditions.
B. Hardware Setup
The system combines bi-manual anthropomorphic hands, multi-view cameras, high-resolution fingertip sensing, and exoskeleton-based teleoperation. Its model fuses visual and tactile inputs through cross-attention and forecasts tactile dynamics to support action generation.
- Two robot arms use anthropomorphic five-digit hands with 17 degrees of freedom per hand and high-resolution fingertip tactile sensors.
- Visual sensing combines wrist-mounted fisheye cameras for local views with a top-mounted stereo camera for global scene awareness.
- An exoskeleton teleoperation system captures demonstrations while a VR interface provides stereo, wrist, and real-time tactile-overlay feedback.
- Naive token fusion does not account for relevant interactions between visual and tactile observations.
- Cross-attention computes visual keys and values from tactile queries and reciprocally, then concatenates the resulting features into hidden states.
- The training procedure uses ground-truth tactile tokens for the first 75% of epochs before switching to predicted tokens for the final 25%.
C. Neural Network Architecture and Learning Procedure
ViTacFormer combines multimodal transformer encoding with action-style conditioning, tactile prediction, and auxiliary pose supervision. Its implementation processes synchronized visual, proprioceptive, and tactile inputs to generate smoothed high-frequency action sequences.
- Architecture: The architecture maps proprioception and expert actions to a style variable, then fuses visual-tactile observations to predict future touch and generate actions.The style variable is sampled from demonstrations during training and set to zero during inference.
- Learning Procedure: The training objective combines KL divergence, action and tactile L1 losses, and an auxiliary end-effector loss.The arm loss supervises end-effector position with L2 loss and rotation with L1 loss.
- Inputs and Outputs: The model uses four camera views, a 58-dimensional proprioceptive state, and synchronized tactile force-torque sequences as inputs.The supplied implementation details specify a six-frame visual/proprioceptive horizon and 18 tactile frames with frame-wise deltas.
- Inputs and Outputs: The policy outputs action sequences of shape (100, 50), runs at 10Hz during deployment, and applies temporal smoothing for stable execution.Each task is trained from 50 expert demonstrations using Adam; short-horizon training takes up to 12 hours and long-horizon training up to 2 days.
- Evaluation: The experiments evaluate both algorithmic performance against imitation-learning baselines and the effectiveness of individual model components.The evaluation covers the questions posed for comparative performance and ablation effectiveness.
A. Benchmark and Environment Setup
The benchmark covers four short-horizon dexterous tasks and a long-horizon hamburger-making task, using success rates and Human Normalized Score to assess performance. ViTacFormer exceeds baseline success rates by over 50% and completes an 11-stage long-horizon sequence.
- Tasks and Environment: The benchmark includes Peg Insertion, Cap Twist, Vase Wipe, Book Flip, and a very long-horizon hamburger-making task.The short-horizon tasks test occluded insertion, rotational control, contour wiping, and friction-sensitive object handling.
- Algorithm Comparison: The benchmark uses 50 trajectories per task, and ablations show that each ViTacFormer component improves manipulation performance.The reported results also describe robust generalization to spatial perturbations with limited demonstrations.
- Metrics: Human Normalized Score evaluates stage-level manipulation quality using 0–3 scores, normalized across stages and weighted by tactile reliance.The metric uses N stages, stage weights w_i, and raw stage scores s_i.
- Baselines: The baselines are Diffusion Policy, HATO, ACT, and ACTw/T, with HATO and ACTw/T incorporating tactile signals through naive token fusion.DP and ACT omit tactile inputs, while HATO and ACTw/T use them as conditions or input tokens.
- Algorithm Comparison: Over 50% success rates separate ViTacFormer from the baselines across the four short-horizon tasks.The comparison reports ViTacFormer as the best-performing method, while tactile-input baselines outperform their non-tactile counterparts.
long-horizon manipulation tasks?
ViTacFormer completes an 11-stage hamburger-making task on a real robot, with stage-wise evaluation and strict end-to-end success metrics. Ablations and failure analyses attribute robustness to cross-modal fusion, tactile forecasting, and curriculum training.
- Long-horizon task: ViTacFormer completes an 11-stage hamburger-making task on a real robot using a single imitation-learning model.The comparison focuses on ACT-based baselines because DP and HATO rarely complete the full sequence.
- Evaluation: The long-horizon evaluation reports stage-wise human normalized scores and two success criteria, including strict completion without human intervention.A stage score below 1 indicates task failure for subsequent-stage testing.
- Ablation Study: Removing cross-attention causes performance drops in Peg Insertion and Cap Twist, where dynamic tactile weighting matters under visual ambiguity.The ablation replaces attention with naive concatenation and links the drop to weaker dense modality interaction.
- Ablation Study: Removing autoregressive forecasting impairs contact-dynamics modeling, while removing the two-phase curriculum produces unstable convergence and higher success-rate variance.The forecasting objective encodes contact trends; the curriculum bridges valid kinematics and noisy predicted tactile signals.
VI. LIMITATION AND FUTURE WORK
ViTacFormer remains constrained by imitation-learning requirements, including limited generalization to unseen tasks and labor-intensive teleoperated data collection. Its stability can also decline with extremely noisy or ambiguous tactile feedback, motivating simulation transfer and more generalizable tactile representations.
- Limitations: Imitation learning limits autonomous generalization to novel tasks unseen during training.The policy also depends on human teleoperation for data collection.
- Limitations: Teleoperated data collection remains time-consuming and labor-intensive.
- Limitations: Extremely noisy or ambiguous tactile feedback can reduce manipulation stability because of current sensor-resolution and representation-learning limitations.
- Future Work: Future work targets Sim-to-Real transfer to reduce reliance on human demonstrations and scalable tactile-data collection in physics-rich simulation.
- Future Work: Future work also explores generalizable tactile representations, including touch foundation models, to improve robustness to sensor noise and adaptation to novel objects.
A. Input Modalities
The system combines visual observations, robot proprioception, and tactile signals within a real dual-arm manipulation platform. Its training and deployment use high-frequency action prediction, temporal smoothing, standardized tabletop tasks, and controlled object-position perturbations.
- Input Modalities: ViTacFormer takes visual observations, robot proprioception, and tactile signals as multimodal inputs.
- Action Representation: The policy generates 100-frame, 50-dimensional action sequences, matching the robot’s full control dimension.
- Action Representation: A 100-frame horizon supports fine-grained dexterous motion across extended manipulation stages.
- Deployment: Deployment runs at 10Hz with temporal smoothing over predicted trajectories for stable execution on a real dual-arm platform.
- Task Setup: The four short-horizon tasks share a standardized tabletop workspace and common objects, with a 5cm grid for object placement.
- Task Setup: Generalization is tested by randomly perturbing each object within a circular region of half-grid radius, equal to 2.5cm.
1) Peg Insertion:
The short-horizon benchmark includes peg insertion and cap twist among standardized tabletop tasks, with performance assessed through stage-level criteria and task success. ViTacFormer achieves perfect peg-insertion success and demonstrates representative failure modes involving localization, grasp posture, and twisting.
- Peg Insertion: Peg insertion requires visual alignment, precise grasping, and tactile-guided insertion into a hole.
- Peg Insertion: Peg insertion is scored across grasping and insertion stages, with a weighted human normalized score and threshold-based success criterion.
- Peg Insertion: 0.93 HNS and 100% success rate are reported for peg insertion, the highest values in the comparison.
- Peg Insertion: Peg-insertion failures include inaccurate hole localization and imprecise grasp posture that causes thumb slippage during transport.
- Cap Twist: Cap twist requires rotating a tightened cap approximately 100 degrees from its open position before placing it on the table.
- Cap Twist: Cap-twist failure can result from slippage and insufficient rotation angle, preventing successful cap removal.
3) Vase Wipe:
The vase-wipe task combines bimanual vase and sponge handling with contact-driven wiping, while the book-flip task requires coordinated page manipulation. ViTacFormer reports strong performance on both, but failures arise from force-control and perception errors.
- Vase Wipe: Vase wiping requires perfect scores in both sponge-grasping and wiping stages for task success.
- Vase Wipe: 0.98 HNS and 90% success rate are reported for vase wiping, with contact-driven wiping highlighted as reliable.
- Vase Wipe: Vase-wipe failures involve insufficient force causing incomplete contact and excessive grasping force causing sponge slippage.
- Book Flip: Book flip requires the right middle finger to lift a page before the left hand presses it down.
- Book Flip: 0.93 HNS and 90% success rate are reported for book flip, outperforming all baselines.
- Book Flip: Book-flip failures arise from missing the page edge or applying excessive downward force before the flip.
F. Long-horizon task: Make Hamburger
The Make Hamburger task tests long-horizon dexterous manipulation through randomized object placement, 11 weighted stages, and strict whole-task success criteria. Failures chiefly arise from unstable grasps, inaccurate localization, and incorrect hand orientation during sequential execution.
- Workspace setup: Seven ingredients and tools are randomly initialized within predefined circular or rectangular zones to test spatial generalization.The setup applies controlled spatial variability during training and evaluation.
- Task sequence: The robot performs a multi-stage hamburger assembly sequence involving sign flipping, spatula use, ingredient placement, and final delivery.The sequence requires coordinated tool use and manipulation across multiple stages.
- Scoring: The task comprises 11 sequential stages covering symbolic interaction, tool use, ingredient assembly, and final delivery, with each stage scored from 0 to 3.Stage weights reflect task complexity and tactile sensitivity.
- Scoring: Whole-task success requires every stage to reach at least 1, while weighted stage scores determine the Human Normalized Score.The scoring criteria and weights are specified in Table XI.
- Failure analysis: Representative failures include unstable lettuce grasping, inaccurate insertion-hole localization, and incorrect sign-grasp orientation causing collisions.These errors can prevent placement or block progression despite partial task execution.