Source-linked AI summary
HapticVLA: Contact-Rich Manipulation via Vision-Language-Action Model without Inference-Time Tactile Sensing
Konstantin Gubernatorov, Mikhail Sannikov, Ilya Mikhalchuk, Egor Kuznetsov, Makar Artemov, Ogunwoye Faith Oluwatobi, Marcelino Fernando, Artem Asanov, Ziang Guo, Dzmitry Tsetserukou
TL;DR
VLAs lack explicit tactile awareness, while tactile hardware increases deployment cost and reduces reproducibility. HapticVLA combines safety-aware reward-weighted flow matching with tactile distillation so a conventional VLA can generate tactile-aware actions without inference-time sensors. In real-world experiments, it achieves an 86.7% mean success rate and outperforms the evaluated baselines.
Problem
VLAs lack explicit force and tactile awareness for safely manipulating soft and fragile objects, while tactile-sensor-based approaches limit reproducibility through hardware cost and compatibility.
Method
HapticVLA trains a safety-aware reward-weighted flow-matching expert with offline tactile rewards, then distills tactile-aware behavior into a conventional VLA student.
Results
86.7% mean success rate across three contact-rich tasks, outperforming the evaluated baseline VLAs.
Takeaways & Limitations
Tactile-aware manipulation can be deployed on robotic setups without tactile sensors at inference while retaining performance on fragile and soft-object tasks.
Abstract
from arXiv · showhide
Tactile sensing is a crucial capability for Vision-Language-Action (VLA) architectures, as it enables dexterous and safe manipulation in contact-rich tasks. However, reliance on dedicated tactile hardware increases cost and reduces reproducibility across robotic platforms. We argue that tactile-aware manipulation can be learned offline and deployed without direct haptic feedback at inference. To this end, we present HapticVLA, which proceeds in two tightly coupled stages: Safety-Aware Reward-Weighted Flow Matching (SA-RWFM) and Tactile Distillation (TD). SA-RWFM trains a flow-matching action expert that incorporates precomputed, safety-aware tactile rewards penalizing excessive grasping force and suboptimal grasping trajectories. TD further transfers this tactile-aware capability into a conventional VLA: we distill a compact tactile token from the SA-RWFM teacher and train a student VLA to predict that token from vision and state modalities, enabling tactile-aware action generation at inference without requiring on-board tactile sensors. This design preserves contact-rich tactile-aware reasoning within VLA while removing the need for on-board tactile sensors during deployment. On real-world experiments, HapticVLA achieves a mean success rate of 86.7%, consistently outperforming baseline VLAs - including versions provided with direct tactile feedback during inference.
I. INTRODUCTION
VLAs lack explicit force and tactile awareness, limiting safe manipulation of soft and fragile objects. HapticVLA addresses this by learning tactile-aware behavior offline and distilling it into a conventional VLA that requires no tactile sensors at inference.
- VLAs rely on vision and proprioceptive states but lack explicit force awareness, limiting safe contact-rich manipulation of soft and fragile objects.
- Vision-based encoding of tactile signals overlooks local mechanical interactions and can fail to extract contact-force information needed for tactile-aware manipulation.
- HapticVLA combines safety-aware reward-weighted flow matching with tactile distillation to encode tactile-aware manipulation without direct sensor measurements at deployment.The framework computes tactile rewards offline, trains an action expert, and distills its tactile-aware behavior into a conventional VLA.
- Existing tactile and force-enabled VLA approaches improve contact-rich manipulation but often depend on costly, hardware-specific sensors that limit reproducibility.
B. Contact-rich Manipulation without Force Sensors at Inference
Prior work shows that tactile data can improve manipulation and sometimes be removed at inference, while offline reward-weighted flow matching offers a route to incorporate contact-rich safety rewards. HapticVLA operationalizes this route through calibrated tactile-map rewards and distillation.
- Prior methods demonstrate benefits from adding tactile data during training, including improved vision-only insertion performance after tactile sensors are disabled at inference.
- Offline reinforcement learning for flow-matching VLAs motivates incorporating tactile data and contact-rich safety rewards directly into flow objectives.
- HapticVLA computes tactile-based safety rewards offline from tactile maps, using calibrated force, pressure, concentration, asymmetry, center-of-pressure, holding, and slip signals.
2) Per-step Reward and Episode Risk:
HapticVLA’s reward design evaluates unsafe tactile behavior both locally and across the full trajectory. Per-step penalties capture specific contact failures, while episode risk summarizes near-failure behavior during holding.
- The per-step reward penalizes over-force, underforce while holding, peak pressure, pressure concentration, force asymmetry, and slip.
- Episode-level risk captures near-failure behavior across the full trajectory by aggregating threshold exceedances during positive holding states.
- The framework overview places offline tactile-reward calculation before SA-RWFM training, where the reward conditions the action expert.
3) Episode Reward:
The episode reward combines step-level quality, task success, and safety-related penalties, then weights flow-matching training samples according to their computed rewards.
- The final episode reward combines averaged step reward, task outcome, and a safety risk penalty.
- The offline dataset records observations, action horizons, timestep, per-step reward, and episode reward for reward-weighted flow-matching fine-tuning.
- Training uses a masked per-sample flow-matching loss for the action sequences.
4) SA-RWFM Fine-Tuning:
SA-RWFM combines local chunk returns and global episode rewards, normalizes them within groups, and converts the resulting advantage-like scores into bounded training weights.
- Chunk returns provide local credit, while episode rewards provide global outcome credit.
- Robust group-wise normalization reduces scale bias across datasets, falling back to standard deviation when MAD is degenerate.
- Normalized episode and chunk scores are combined into an advantage-like score for weighting.
- Exponentiated weights are clipped between wmin and wmax and renormalized within each batch.
- The method uses α = 0.25, wmin = 0.25, and wmax = 4.0.
- The reward-weighted flow-matching objective uses these weights to prioritize training samples.
5) Anchor Regularization for Stability:
Anchor regularization stabilizes reward-weighted fine-tuning by keeping the policy near its initial imitation solution while the SA-RWFM procedure updates parameters.
- 5) Anchor Regularization for Stability:: Exponentiated weighting can cause drift and mode collapse in generative policies.
- 5) Anchor Regularization for Stability:: An anchor regularizer constrains parameters toward the initial imitation solution θ0.
- 5) Anchor Regularization for Stability:: The total objective combines the reward-weighted flow-matching loss with the anchor loss.
- 5) Anchor Regularization for Stability:: A warm-up schedule gradually increases α and adjusts λanchor to preserve baseline behavior early in training.
B. Tactile Distillation
Tactile Distillation transfers the SA-RWFM teacher’s tactile-aware behavior into a conventional VLA, removing tactile hardware requirements from deployment without architectural modifications.
- B. Tactile Distillation: The SA-RWFM teacher requires physical tactile sensors at inference, limiting deployment to hardware-equipped platforms.
- B. Tactile Distillation: Offline TD distills the teacher’s tactile-aware actions into a conventional VLA student with zero architectural modifications.
- B. Tactile Distillation: The teacher policy conditions on visual and language observations plus a state combining six joint variables with a 128-dimensional tactile embedding.
1) Problem Formulation:
The distillation formulation represents teacher and student policies with conditional flow matching, generates offline teacher action targets, and initializes the student by transferring non-tactile teacher parameters.
- 1) Problem Formulation:: Both policies generate action chunks through conditional flow matching with horizon K = 50 and per-arm joint dimension da = 6.
- 2) Stage 1: Offline Teacher Target Generation:: For each training sample, the tactile-conditioned teacher maps observations and full states to a predicted action chunk.
- 2) Stage 1: Offline Teacher Target Generation:: Teacher predictions are precomputed once over the full training set, making distillation completely offline.
- 2) Stage 1: Offline Teacher Target Generation:: These predictions form soft targets whose quality reflects the teacher’s tactile-conditioned force awareness and reward-weighted bias.
- 3) Stage 2: Teacher Backbone Initialization:: The student is initialized from the teacher’s weights with targeted adaptation to transfer visual and language representations.
- 3) Stage 2: Teacher Backbone Initialization:: All teacher parameters except the tactile encoder are copied into the student.
- 3) Stage 2: Teacher Backbone Initialization:: Because the state dimension changes, the student retains only the teacher projection columns corresponding to proprioception.
- 3) Stage 2: Teacher Backbone Initialization:: This preserves the teacher’s learned joint-position mapping while discarding tactile input columns.
4) Stage 3: Blended Target Training:
Blended target training combines demonstration actions with offline teacher predictions, balancing behavioral grounding against tactile-aware shaping. The training data also includes faulty demonstrations to sharpen safety-aware action learning.
- Blended Target Training: The student trains on blended action targets that interpolate between ground-truth demonstrations and teacher predictions.The blended target is used as the flow-matching target.
- Blended Target Training: With αd = 0.5, ground-truth anchoring limits teacher-error amplification while teacher shaping injects tactile-aware force modulation.Validation sets αd = 0 for an unbiased reconstruction-quality comparison.
- Safety-Aware Data: Successful and faulty demonstrations support explicit reward weighting based on tactile feedback and post-grasp object integrity.Faulty episodes involve excessive grasping force that causes permanent damage or significant deformation.
- Safety-Aware Data: Negative examples constrain and sharpen the high-reward region, helping distinguish safe, high-reward trajectories from unsafe ones.The dataset contains 310 real-world episodes across three tasks, including successful and faulty demonstrations.
IV. EXPERIMENTS
Experiments use a bimanual SO-101 platform with external and wrist-mounted cameras plus high-resolution fingertip tactile arrays. Evaluation covers three contact-rich pick-and-place tasks involving objects with different fragility and hardness.
- Platform: The platform uses two SO-101 manipulators, with the right arm modified by a parallel gripper carrying tactile sensors on each finger.The right arm uses higher-voltage servomotors to provide increased torque for precise grasping control.
- Perception: Three RGB streams combine an Intel RealSense D435 external camera with IMX335 wrist cameras, all operating at 640 × 480 pixels.The RealSense provides the primary external view, while each manipulator carries a wrist-mounted camera.
- Tactile Sensing: Each fingertip tactile array has 100 taxels in a 10 x 10 layout, detecting 1–9 N per point for 200 taxels total at 120 Hz.This resolution monitors force-distribution shifts, incipient slip, and localized pressure variations.
- Tasks and Evaluation: Evaluation covers Jar, Waffles, and Egg pick-and-place tasks involving objects with different fragility and hardness.Each model receives 20 test trials per task, and success requires intact objects placed at designated targets, with task-specific criteria.
- Tasks and Evaluation: Task-specific success criteria permit minor transient jar deformation but require intact waffles and eggs after placement.The jar must retain its initial shape without structural damage; waffles must avoid cracks or crumbles; eggs must remain intact in the correct tray slot.
C. Main Results
HapticVLA is evaluated against lightweight, cross-embodiment, and larger VLA baselines on contact-rich manipulation, with ablations isolating tactile distillation and SA-RWFM. It achieves the strongest reported performance while removing tactile sensing at inference.
- Main Results: The benchmark compares HapticVLA with SmolVLA, X-VLA, and VLA-0, fine-tuning all baselines on the same in-house dataset.The baselines represent a lightweight edge model, a compact cross-embodiment model, and a larger model.
- Main Results: 86.7% mean success rate: HapticVLA significantly outperforms all baselines across three contact-rich manipulation tasks.HapticVLA also improves egg manipulation by 45% absolute over the base SmolVLA.
- Main Results: 75% mean success rate: SmolVLA with the SA-RWFM action expert substantially exceeds the base model, reaching 85% in two of three tasks.X-VLA and VLA-0 each achieve zero-percent success in these experiments.
- Ablation Study: Tactile Distillation substantially improves contact-rich manipulation, and combining it with SA-RWFM yields the largest mean success rate across the three tasks.The ablation evaluates four configurations over three tasks with 20 independent trials per task.
- Ablation Study: Asynchronous inference can lower performance relative to synchronous configurations, attributed to temporal misalignment and increased effective latency.The comparison concerns configurations with and without tactile distillation and tactile feedback at inference.
- Conclusion: HapticVLA enables tactile-aware manipulation without tactile sensors at inference by combining SA-RWFM with distilled tactile representations from vision and state inputs.Real-world experiments report a mean success rate of 86.7% while manipulating fragile and soft objects without haptic feedback.