PaperlayerView exact revision on arXiv

Source-linked AI summary

TaF-VLA: Tactile-Force Alignment in Vision-Language-Action Models for Force-aware Manipulation

Yuzhe Huang, Pei Lin, Wanlin Li, Daohan Li, Jiajun Li, Jiaming Jiang, Chenxi Xiao, Ziyuan Jiao

arXiv:2601.20321v2cs.RO

TL;DR

Vision-dominant and tactile-vision-aligned VLA models do not adequately represent physical contact forces. TaF-VLA addresses this gap with synchronized tactile-force data, a temporal force-aligned adapter, and VLA integration, achieving higher success on force-critical manipulation tasks while retaining limitations in high-frequency and pre-contact control.

  • Problem

    VLA models and tactile-vision alignment methods lack force-grounded representations for contact-rich manipulation requiring precise physical interaction.

  • Method

    TaF-VLA combines an automated synchronized tactile-force data pipeline with a contrastive TaF-Adapter that aligns historical tactile observations to force signals in a discrete latent space.

  • Results

    TaF-VLA achieves a 64.8% average success rate, roughly doubling vision-only baselines and outperforming vision-tactile methods by 19–22 percentage points.

  • Takeaways & Limitations

    Force-aligned representations support language-conditioned force modulation and force-sensitive manipulation capabilities reported as intractable for vision-only or naive tactile-vision baselines.

  • Takeaways & Limitations

    Policy latency and pre-contact positioning errors cause failures in high-frequency dynamics and tasks where initial tactile contact is not established.

Abstract

from arXiv · show

Vision-Language-Action (VLA) models have recently emerged as powerful generalists for robotic manipulation. However, due to their predominant reliance on visual modalities, they fundamentally lack the physical intuition required for contact-rich tasks that require precise force regulation and physical reasoning. Existing attempts to incorporate vision-based tactile sensing into VLA models typically treat tactile inputs as auxiliary visual textures, thereby overlooking the underlying correlation between surface deformation and interaction dynamics. To bridge this gap, we propose a paradigm shift from tactile-vision alignment to tactile-force alignment. Here, we introduce TaF-VLA, a framework that explicitly grounds high-dimensional tactile observations in physical interaction forces. To facilitate this, we develop an automated tactile-force data acquisition device and curate the TaF-Dataset, comprising over 10 million synchronized tactile observations, 6-axis force/torque, and matrix force map. To align sequential tactile observations with interaction forces, the central component of our approach is the Tactile-Force Adapter (TaF-Adapter), a tactile sensor encoder that extracts discretized latent information for encoding tactile observations. This mechanism ensures that the learned representations capture history-dependent, noise-insensitive physical dynamics rather than static visual textures. Finally, we integrate this force-aligned encoder into a VLA backbone. Extensive real-world experiments demonstrate that TaF-VLA policy significantly outperforms state-of-the-art tactile-vision-aligned and vision-only baselines on contact-rich tasks, verifying its ability to achieve robust, force-aware manipulation through cross-modal physical reasoning.

I. INTRODUCTION

TaF-VLA addresses the force limitations of vision-only and tactile-vision-aligned VLA policies by aligning sequential tactile observations with physical force signals. It combines synchronized data collection, a force-aligned adapter, and VLA integration for force-critical manipulation.

  • Motivation: Vision-only VLA models lack direct force perception needed for fragile-object handling, precision assembly, and deformable-object interaction.Visual observations can also suffer from occlusion in contact-rich scenarios.
  • Motivation: Existing tactile-VLA methods treat tactile frames as auxiliary visual tokens, overlooking their relationship to local mechanical interactions and force dynamics.This alignment can fail to distinguish static surface features from dynamic force events.
  • Method: TaF-VLA constructs synchronized visuotactile, 6-axis force/torque, and matrix-force data through an automated acquisition device and TaF-Dataset.The device applies matched forces to tactile and force sensors while varying indenter stiffness and geometry.
  • Method: The TaF-Adapter uses contrastive learning, historical observations, and a discrete codebook to learn noise-robust, force-aligned representations of tactile dynamics.It aligns tactile embeddings with paired force profiles without requiring explicit force reconstruction.
  • Policy integration: TaF-VLA interleaves tactile-force-aligned tokens into a pretrained VLA backbone and conditions actions on language and tactile feedback.Force-aware language instructions are used during policy fine-tuning.
  • Results: Across 7 force-critical manipulation tasks, TaF-VLA improves average success rate by 22% over a state-of-the-art vision-tactile-aligned baseline.The reported advantage is strongest in contact-rich tasks requiring precise force perception and regulation.

II. RELATED WORK

Prior work supplies tactile or force signals to VLA policies but leaves a gap between visual-semantic representations and physical interaction dynamics. TaF-VLA addresses this gap through synchronized force-labeled data and temporal tactile-force alignment.

  • Motivation: VLAs generalize across manipulation tasks but struggle with the physical interaction details required when visual feedback is insufficient.RGB cameras lack direct perception of contact forces, friction, and compliance.
  • Tactile sensing: Vision-based tactile sensors provide high-resolution deformation images containing geometry, texture, local pressure, and slippage information.Their visual form makes them compatible with vision-based learning methods.
  • Force perception: Physics-driven and data-driven force estimation methods have evolved toward temporal modeling for history-dependent effects such as hysteresis and incipient slip.Earlier calibration approaches depend on sensor geometry and are sensitive to material aging and manufacturing variability.
  • Force sensing: Global wrist or joint force signals measure total interaction load but lose spatial pressure patterns and local contact geometry.These sensors are also described as expensive, fragile, and difficult to deploy broadly.
  • Tactile integration: Tactile-VLA methods commonly align tactile embeddings with scene images, which can miss temporal dependencies and lack force-anchored organization.The stated limitation is that tactile appearance alignment does not ensure force-aware or force-consistent representations.
  • TaF-VLA: TaF-VLA combines a receding-horizon temporal adapter, contrastive tactile-force alignment, and force-aware language conditioning.Its acquisition pipeline supplies the synchronized visuotactile and force data required for this framework.

1) Design Principle:

The acquisition system is designed to collect synchronized tactile and force responses under matched indentation conditions. Its modular hardware combines coordinated actuation, interchangeable indenters, and multiple sensing modalities.

  • Design Principle: Twin platforms use mechanically linked, identical motor-gear pairs to maintain synchronized indentation across corresponding degrees of freedom.The design aims to eliminate relative phase errors between tactile and force measurements.
  • Design Principle: The device uses a compact 3D-printed chassis and modular components to support rapid fabrication and replication.Its stated footprint is 25 × 20 × 45 cm, and the full unit can be assembled in under four hours.
  • Actuation: Five servomotors drive Z translation, Yaw, Roll, and Pitch through upper and lower actuation stages.A gear-driven transmission and simplified Stewart-platform configuration coordinate the platforms.
  • Actuation: A compliant Z-axis element maintains contact pressure and helps prevent overload during high-speed operation.The compliant mechanism is integrated into the upper-stage Z-axis transmission.
  • Contact interfaces: Quick-release mounts support interchangeable indenters with varied patterns, curvatures, and stiffness levels.Paired indenters simulate contacts ranging from point interactions to conformal surface patches.
  • Sensing: The sensing system combines vision-based tactile sensors, 6-axis force/torque sensing, and a 12 × 12 matrix force sensor.Force sensors are resampled to the tactile sensor's 30 Hz rate for paired acquisition.

B. TaF-Dataset Construction

TaF-Dataset construction combines automated, diverse tactile-force acquisition with force quantization and temporal tactile encoding. The TaF-Adapter aligns sequential tactile observations with discretized physical-force representations for downstream force-aware control.

  • Automated Generation Protocol: The acquisition system establishes synchronized measurements from tactile images, 6-axis force/torque, and matrix force sensors during automated contact exploration.The protocol uses Z-axis contact followed by press-and-perturb motion across rotational degrees of freedom.
  • TaF-Dataset: Over 10 million synchronized tactile-force pairs from six distinct VBTS units support cross-sensor visual-force alignment.The sensors include four custom-built variants and two commercial GelSight units.
  • Force Quantization: The force encoder quantizes pressure maps and 6-axis force/torque sequences into discrete codebooks representing force primitives.Separate modalities are encoded with dedicated MLPs, quantized, and concatenated into the final force embedding.
  • Force Quantization: Reconstruction and quantization losses constrain the force codebooks to preserve high-fidelity physical dynamics and support stable tactile alignment.Reconstruction is introduced to prevent codebook collapse and ensure the codebook spans force-dynamic variance.
  • Temporal Tactile Encoding: A causal Transformer summarizes sequential tactile images, preserving temporal information needed to distinguish static deformation from slip or hysteresis.A learnable summary token provides the holistic tactile-event representation.
  • Cross-Modal Alignment: Symmetric InfoNCE aligns each tactile event with its corresponding force code while separating force codes from other time steps.Sliding-window sampling teaches temporal force evolution directly from sequential tactile observations.

B. Force-aware VLA Policy Integration

TaF-VLA integrates a pretrained TaF-Adapter into a flow-matching VLA backbone, combining visual-language context with local tactile-force features. The resulting policy generates actions conditioned on both task semantics and tactile feedback.

  • Policy Integration: TaF-VLA integrates the pretrained TaF-Adapter into the π0.5 flow-matching generalist policy backbone.The adapter supplies force-aligned tactile features to the policy integration stage.
  • Policy Integration: The policy forms high-level conditioning from third-person images, wrist images, and language instructions.This unified vector provides task semantics and global scene context for action generation.
  • Tactile Conditioning: Sliding-window VBTS signals are fused with visual-language features to inject local contact cues into the action-generation process.The tactile-conditioned objective combines tactile features with the flow-matching action formulation and robot proprioception.
  • Action Generation: During inference, TaF-VLA integrates a learned velocity field from noise to a clean action trajectory.The update uses δ = 0.1 and integrates over τ ∈[0, 1].
  • Policy Integration: Conditioning on global visual-language features and local tactile latents produces controllable, contact-aware actions that can adjust to tactile feedback while following task directives.The action expert uses the combined conditioning to generate force-aware manipulation behavior.

V. EXPERIMENTS

The experiments evaluate TaF-VLA and TaF-Adapter through real-world contact-rich manipulation studies focused on performance, generalization, efficiency, and architectural contributions.

  • Performance Benchmarking: The evaluation asks whether explicit tactile-force alignment improves TaF-VLA over vision-only and tactile-vision-aligned baselines.This comparison is conducted across diverse contact-rich tasks.
  • Generalization & Robustness: The experiments test TaF-Adapter generalization across sensor hardware and its plug-and-play transfer to other policy architectures.These studies target robustness beyond a single sensor or backbone.
  • Training Efficiency: The study examines whether multimodal feedback improves training sample efficiency relative to vision-only baselines.The relevant comparison uses fewer demonstrations.
  • Ablation Studies: Ablation studies measure the effects of historical observations and discrete vector quantization on system performance.The design choices are evaluated as specific TaF-VLA components.

A. Experimental Setup

Policy training uses over 10,000 force-aware episodes spanning more than 20 scenarios, while quantitative evaluation targets eight real-world contact-rich tasks covering varied physical interactions.

  • Training Data: The training dataset contains over 10,000 force-aware episodes across more than 20 diverse scenarios.Scenarios range from fragile-object handling to complex tool use.
  • Evaluation Tasks: Quantitative evaluation uses eight real-world contact-rich tasks selected to span different types of physical interactions.The task protocols and success criteria are detailed separately in the paper.

1) Training and Evaluation Settings:

The evaluation compares TaF-VLA and adapter-augmented policies with vision-only and tactile-vision baselines across seven manipulation tasks, using common training data and success-rate evaluation.

  • Baselines: Five open-source baseline families include RGB-only ACT and Diffusion Policy, vision-tactile FreeTacMan, adapter-augmented ACT and Diffusion Policy, and vision-only π0.5.FreeTacMan treats tactile feedback as auxiliary visual tokens, while the adapter variants test tactile-force alignment across standard policy backbones.
  • Baselines: π0.5 is a state-of-the-art VLA foundation model fine-tuned using only visual observations and serves as TaF-VLA's vision-only backbone.
  • Evaluation protocol: The benchmark reports task success rates across seven manipulation tasks, with bold and underlined entries marking the best and second-best policies.
  • Evaluation protocol: All evaluated models use the same dataset, and real-world inference runs on an NVIDIA RTX 4090 workstation.
  • Evaluation protocol: Performance is measured by task-specific binary success criteria over 15 rollout trials per task.Success rate is computed as the proportion of successful attempts.

B. Comparison with Baselines

TaF-VLA achieves the strongest overall task performance, with its advantage largest on force-critical manipulation; tactile-force alignment also improves adapter-augmented policies and cross-sensor robustness.

  • Overall comparison: TaF-VLA achieves the highest success rate across all evaluated scenarios, with the largest margin over the strongest baseline on force-critical tasks.
  • Comparison with vision-only policies: Vision-only policies perform well on geometry-dominated Tube Insertion at 53.3% but all three cited vision-only baselines achieve 0% on Tweezer Weight Pick.The passage attributes failures on friction-dependent tasks to insufficient grasp force and visually imperceptible micro-slip cues.
  • Comparison with vision-only policies: Grounding tactile feedback in physical force enables detection of contact transitions and shear resistance for stable closed-loop control.
  • Impact of tactile-force alignment: TaF-VLA surpasses FreeTacMan by 26.6% on Tweezer Weight Pick and 33.3% on Whiteboard Erasing.The comparison supports explicit force grounding over treating tactile data as auxiliary vision for force-sensitive tasks.
  • Plug-and-play capability: Adapter augmentation improves ACT and Diffusion Policy by 6.7%–33.3% across tasks, including Jelly Slicing rising from 13.3% to 33.3% for Diffusion Policy.ACT + TaF-Adapter also exceeds FreeTacMan by 20.0% on Power Bank Extraction.
  • Cross-sensor robustness: On unseen CUSTOM-BUILT B, TaF-Adapter reaches 60.3% mean success versus 30.0% for force prediction, while unseen GELSIGHT reaches 53.3% versus 23.3%.

D. Pretrained TaF-Adapter Improves Data Efficiency

Pretrained tactile-force representations improve data efficiency and depend on temporal context, sufficient codebook capacity, decoupled modalities, and clean force supervision.

  • Data efficiency: TaF-VLA matches or exceeds π0.5's best performance on three contact-rich tasks using 100 demonstrations instead of 200.The comparison covers Tongs Food Pick, Power Bank Extraction, and Chocolate Grasping.
  • Data efficiency: π0.5 improves from 13.3% to 40.0% on Tongs Food Pick, 13.3% to 33.3% on Power Bank Extraction, and 60.0% to 73.3% on Chocolate Grasping as data grows from 100 to 200 episodes.
  • Ablations: Reducing the receding-horizon window from N = 5 to N = 1 substantially degrades performance because instantaneous deformation cannot distinguish static grasps from incipient slip.
  • Ablations: Reducing codebook size lowers success rates by collapsing distinct tactile-visual features into shared entries and losing representational granularity.
  • Ablations: A shared codebook for pressure maps and global 6-DoF force vectors performs poorly across all three benchmark tasks.
  • Ablations: Adding Gaussian noise to ground-truth force labels substantially decreases performance, demonstrating sensitivity to tactile-force data quality.

A. Key Findings

TaF-VLA addresses force-blindness in generalist robot policies through force-aligned tactile representations, a large synchronized dataset, and a tactile-force adapter. Evaluations report stronger performance and generalization, while identifying task, inference, sensor, hardware, and architectural boundaries.

  • Results: Force sensing is task-dependent: vision-only policies can match tactile approaches on gentle lightweight-object tasks but underperform on force-critical manipulation.The force-critical examples include Heavy Part Lifting and Power Bank Extraction.
  • Results: Discrete tactile representations consistently outperform continuous embeddings by suppressing high-frequency noise and hardware-specific artifacts while retaining physical cues.Zero-shot results on visually similar but structurally different sensors support hardware-agnostic tactile perception.
  • Failure Modes: Most TaF-VLA failures occur before contact when positioning errors leave an air gap, although insufficient interaction force also causes failures.The air gap prevents the tactile-force feedback loop from being triggered.
  • Limitations: Receding-horizon action output limits closed-loop correction frequency, causing latency during high-frequency events such as force spikes in Jelly Slicing.The authors identify faster inference as necessary for high-frequency force control.
  • Limitations: Generalization remains limited to relatively similar visuotactile sensors, while custom 3D-printed acquisition hardware may introduce measurement bias.The authors also identify deeper tactile integration and higher-precision hardware as future directions.
  • Results: 64.8% average success rate roughly doubles vision-only baselines and exceeds existing vision–tactile methods by 19–22 percentage points.The evaluation also reports fine-grained force-modulation commands previously described as intractable.

APPENDIX

The force-aware manipulation dataset combines real-world demonstrations, high-fidelity visual-tactile data, teleoperation, and force-grounded language annotations. Its tasks span grasping, tool use, insertion, extraction, surface interaction, cutting, and fragile-object handling under varied force requirements.

  • Dataset Construction: The dataset uses real-world demonstrations with high-fidelity visual and tactile modalities collected through a teleoperation pipeline.A full-scale exoskeleton controller streams low-latency joint-space commands to the robot.
  • Force-Regulated Manipulation: Tasks vary object stiffness, weight, and geometry, requiring grip-force modulation from physical feedback to maintain stable grasps.Fragile objects require minimal force, whereas heavy low-friction objects require higher force to prevent slippage.
  • Tool Use: Tool-use scenarios such as Tweezer Weight Pick require reasoning about force transmission, counterrotational moments, and coupled tool–hand dynamics.The tool acts as a lever that amplifies rotational moments at the fingertips.
  • Dynamic Surface Interaction: Dynamic surface tasks require continuous contact regulation, including steady normal force, shear compensation, and stiffness-transition detection.Whiteboard Erasing and Jelly Slicing exemplify these requirements.
  • Language Annotation: The dataset augments every demonstration with force-grounded language annotations generated through an LLM-based physics-informed Chain-of-Thought prompting pipeline.The prompt template for generating the dataset is shown in Fig. 10.
  • Evaluation Tasks: The evaluation includes precision insertion, friction-fit extraction, heavy lifting, compliant-tong manipulation, and fragile chocolate and crisp grasping.These tasks test collision avoidance, sustained grip force, variable friction, and near-zero tolerance for force overshoot.
Loading 2601.20321v2…