PaperlayerView exact revision on arXiv

Source-linked AI summary

ViTacPhys: Physical Property-Aware Grasping from Human Visual-Tactile Demonstrations

Yiwen Liu, Yujun Zhu, Kui Jia, Zhao Liao, Yangwei You, Shuaijun Wang

arXiv:2608.21355v1cs.RO

TL;DR

ViTacPhys addresses the limited use of explicit object physical properties in adaptive manipulation. It predicts mass, friction coefficient, and stiffness from human visual–tactile demonstrations using temporal multimodal fusion and a pre-contact VLM prior, then transfers these predictions to robot grasping. The conditioned policy is associated with higher clean-success rates than ACT on both in-distribution and out-of-distribution objects.

  • Problem

    Vision-based action models rarely use explicit object mass, stiffness, and friction properties to adapt manipulation policies.

  • Method

    ViTacPhys combines temporal visual–tactile modeling, interaction flow, cross-attention fusion, and a pre-contact VLM semantic prior, then transfers predictions to a robot policy.

  • Results

    The conditioned policy is associated with clean-success rates 12.5 percentage points higher on in-distribution objects and 38.9 percentage points higher on out-of-distribution objects than ACT.

  • Takeaways & Limitations

    The results demonstrate the feasibility of explicitly estimating and conditioning on object physical properties for real-world adaptive grasping.

  • Takeaways & Limitations

    The dataset contains 60 objects manipulated by one participant, and the tactile maps mainly capture normal pressure, limiting exact mass and friction regression.

Abstract

from arXiv · show

Recent vision-based action models have demonstrated strong capabilities in complex manipulation, but they rarely leverage explicit object physical properties to adapt their policies. We introduce ViTacPhys, a visual-tactile framework and data acquisition system that estimates object mass and friction-coefficient classes, together with continuous stiffness, from human manipulation demonstrations. Trained on data from 60 rigid and deformable objects, ViTacPhys combines temporal visual-tactile modeling, cross-attention multimodal fusion, and a semantic prior derived from a vision-language model. On seen objects, it achieves 97.2% mass classification accuracy, 98.8% friction-coefficient classification accuracy, and a stiffness mean absolute percentage error (MAPE) of 5.51%. On held-out objects from known categories, it achieves 87.5% mass accuracy, 97.5% friction-coefficient accuracy, and a stiffness MAPE of 9.08%. We transfer ViTacPhys from the human domain to the robot domain using limited robot teleoperation data, robot-style video augmentation, and human demonstrations with matched actions, and deploy it as an online module for adaptive grasping. The resulting physical-property-conditioned policy achieves total grasping success rates of 95.0% on in-distribution objects and 83.4% on out-of-distribution objects. For out-of-distribution objects successfully grasped by both methods, its force profiles are more consistent with human teleoperation than those produced by ACT. These results demonstrate the feasibility of explicitly estimating and conditioning on object physical properties for real-world adaptive grasping.

I. INTRODUCTION

ViTacPhys addresses the need for robots to use object physical properties alongside visual knowledge during manipulation. It learns these properties from human visual–tactile demonstrations and conditions grasping policies on the predictions.

  • Motivation: Mass and friction coefficient determine tangential force requirements, while stiffness determines normal force needed to avoid deformation.These properties matter because insufficient force can cause slipping, whereas excessive force can deform delicate objects.
  • Approach: Human demonstrations provide scalable manipulation experience for predicting mass, stiffness, and friction coefficient from temporally structured visual–tactile evidence.The framework uses visual cues, tactile maps, temporal interaction content, and flow from 1,800 demonstrations involving 60 objects.
  • Dataset contribution: ViTacPhys contributes an explicit physical-property dataset with mass, stiffness, and friction-coefficient labels for 60 objects.The dataset includes rigid and deformable everyday objects and complements prior datasets with these manipulation-relevant properties.
  • Results: 87.5% mass accuracy, 97.5% friction-coefficient accuracy, and 9.08% stiffness MAPE are achieved on held-out objects from known categories.The predictor combines visual and tactile content, interaction flow, and a pre-contact VLM semantic prior.
  • Robot transfer: 12.5 percentage points higher clean-success rates on in-distribution objects and 38.9 percentage points higher rates on out-of-distribution objects are associated with the conditioned policy relative to ACT.The system integrates predicted properties into a dexterous-robot grasping policy.

B. Learning Object Physical Properties

Prior approaches obtain physical-property information from visual–tactile data or task-specific interactions, but human demonstrations offer a promising route for transferring natural action adaptations to robots. ViTacPhys targets this gap with human visual–tactile learning and robot integration.

  • Existing approaches: Purely visual methods provide pre-contact physical-property priors, while tactile observations add information beyond visual appearance.Visual estimates can fail for disguised materials or unseen objects.
  • Existing approaches: Contrastive and adversarial learning align visual observations with tactile images or force signals to support synesthesia-like physical-property inference.Other implicit approaches use human poses, robot proprioception, or predefined physical experiments as self-supervised signals.
  • Research gap: Human demonstrations contain consistent action adaptations to physical-property combinations and are promising for transferring such knowledge to robots.Only a few prior works have explored learning physical properties from human data.
  • ViTacPhys: ViTacPhys learns physical properties from human grasping demonstrations and integrates the predictions into downstream adaptive grasping policies.The framework focuses on mass, stiffness, and friction coefficient.

III. HUMAN VISUAL–TACTILE PHYSICAL-PROPERTY DATASET

The ViTacPhys dataset synchronizes wearable visual, tactile, and motion measurements during natural grasping of varied everyday objects. Its annotation protocols measure mass, operational stiffness, and silicone-contact friction while reflecting sensing limitations.

  • Data acquisition: The wearable system synchronizes wrist RGB video, fingertip pressure images, and fingertip motion-capture measurements.Pressure arrays are mounted on the thumb, index, and middle fingertips, while the wrist camera supplies the visual stream.
  • Data annotation: Mass is measured five times with a 0.1 g-resolution scale, while stiffness is estimated during quasi-static pre-yield pinching from calibrated force and displacement.Motion capture measures relative thumb–index displacement and tactile readings provide calibrated normal force.
  • Data annotation: Stiffness is reported in N/mm as the mean of five trials, and friction coefficient is measured from the critical sliding angle using µs = tan(θ).The friction protocol uses a silicone-coated inclined plane matching the tactile contact material.
  • Dataset composition: 60 everyday objects across seven categories receive mass, stiffness, and silicone-contact friction-coefficient labels.The objects vary in shape, material, surface, filling, and rigidity.
  • Label design: Normal-pressure sensing constrains exact mass and friction regression, so those properties are represented as ordered low-medium-high targets while stiffness remains continuous.Dataset-level class boundaries are fitted once before evaluation splits and then fixed.
  • Interaction protocols: Two collection protocols—vertical grasping and shaking grasping—provide stable and dynamic interaction cues.The dataset contains 1,800 demonstrations collected from 15 trials per protocol for each object by one participant.

IV. METHOD

ViTacPhys predicts mass, friction, and stiffness by fusing temporal visual–tactile interaction evidence with a pre-contact VLM semantic prior. Cross-attention aligns modalities before task-specific prediction heads produce the physical-property outputs.

  • Problem formulation: The model maps synchronized visual and fingertip tactile sequences to mass logits, friction logits, and a continuous stiffness value.Frame-to-frame optical flow is computed from adjacent post-processed frames to capture object dynamics during grasping.
  • Temporal encoding: Each modality uses separate content and motion streams, followed by temporal GRUs that preserve interaction information across time.Visual and tactile features are formed from sampled frames and flow fields.
  • VLM-derived semantic prior: A semantic prior is generated from five RGB frames acquired strictly before contact and encodes visible category, material, texture, geometry, and fill-state cues.The VLM receives no tactile frames, contact deformation, or post-contact motion.
  • Probabilistic interpretation: The text prior conditions the physical-property posterior, while post-contact visual–tactile sequences provide interaction evidence rather than an additional training objective.The joint target is y = (m, s, f).

D. Multimodal Fusion and Prediction

ViTacPhys fuses visual, tactile, temporal, and VLM-derived semantic representations to predict mass, friction coefficient, and stiffness. It uses task-specific objectives, ordinal regression for ordered classes, and adaptive task balancing.

  • Multimodal fusion: The model cross-attends visual and tactile sequences, temporally pools them, then uses the result to query VLM-derived tokens.A learned gate regulates the text contribution, while the direct visual–tactile branch preserves sensor evidence when text influence is small.
  • Prediction heads: ViTacPhys predicts mass, friction coefficient, and stiffness through three task-specific heads after multimodal fusion.Mass and friction heads output scalar ordinal logits, while the stiffness head predicts a normalized continuous scalar.
  • Training objectives: The stiffness head uses mean-squared regression, while mass and friction use ordinal regression because their discretized labels retain low-medium-high ordering.Ordinal regression models relative class structure that standard cross-entropy treats as independent categories.
  • Training objectives: GradNorm adaptively balances the mass, stiffness, and friction task weights using gradient norms measured before the task-specific heads.This placement promotes balanced optimization across the three prediction tasks.

1) Human-to-Robot Transfer:

ViTacPhys transfers human-learned physical-property representations to dexterous-robot grasping by reducing sensor and visual domain gaps and conditioning an ACT-style policy online. Contact detection activates property inputs, which are stabilized and encoded alongside robot observations.

  • Human-to-robot transfer: The transfer uses matching visual and tactile sensor types, robot-style video augmentation, limited teleoperation, and human demonstrations with matched actions.Edited RGB videos remain paired with the original synchronized tactile observations.
  • Online deployment: Before contact, physical-property inputs are zero; after tactile threshold detection, a rolling queue enables an immediate first prediction.Repeated queue entries are progressively replaced by new observations after contact.
  • Property conditioning: Mass and friction predictions are represented as class indices, while continuous stiffness is discretized into ten fixed quantile-based bins.The bin boundaries are derived only from ground-truth stiffness distributions before downstream policy training and remain fixed during evaluation.
  • Property conditioning: Temporal voting selects the most probable recent prediction for each property before supplying the structured property token to the ACT-style policy.The token combines contact state with mass, stiffness, and friction embeddings.
  • Policy learning: The policy observes the property token, wrist RGB frame, and dexterous-hand proprioception, and predicts dexterous-hand actions while the arm is controlled separately.The property token links object-dependent properties with demonstrated finger-closure and grasp-force profiles.

V. VITACPHYS PREDICTION EVALUATION

ViTacPhys is evaluated across in-distribution, held-out-object, and one-shot protocols with object- or episode-level splits designed to test seen-object interaction, category-level generalization, and limited object diversity. Metrics cover classification, regression, correlation, and error tolerance, with results averaged over independent runs and balanced across episodes.

  • Evaluation protocols: The in-distribution protocol randomly splits episodes, allowing episodes from the same object across subsets to test new interactions with seen objects.This protocol uses episode-level rather than object-level separation.
  • Evaluation protocols: The held-out-object protocol assigns all episodes of each object to one subset, testing unseen objects from categories observed during training.Each coarse category retains validation and test-object coverage.
  • Evaluation protocols: The one-shot protocol trains on one randomly selected object per coarse category and tests on all remaining objects.It evaluates transferability of physical-property representations under severely limited object diversity.
  • Metrics: Classification uses accuracy and Macro-F1, while stiffness regression uses MAE, RMSE, MAPE, Pearson correlation, and Within-k.Macro-F1 weights classes equally, and Within-k measures predictions below a relative-error tolerance.
  • Reporting: Results are averaged over three independent runs and computed at the temporal-window level, with episode-balanced classification weighting.Longer episodes therefore do not receive greater total classification weight.

B. Physical-Property Prediction Results

ViTacPhys predicts physical properties accurately for seen objects and generalizes well to held-out objects within known categories, but performance declines when object diversity is severely restricted. Table II reports prediction results, while Table III provides input-modality ablations.

  • In-distribution results: 97.2% mass accuracy, 98.8% friction-coefficient accuracy, and 5.51% stiffness MAPE are achieved on seen objects.In-distribution stiffness predictions include 96.56% within 20% relative error, with Macro-F1 scores of 0.970 and 0.984 and Pearson r = 0.980.
  • Generalization results: 87.5% mass accuracy, 97.5% friction-coefficient accuracy, and 9.08% stiffness MAPE are achieved on held-out objects from known categories.Stiffness reaches Pearson r = 0.947 in this setting.
  • Generalization results: Under one-shot training, mass accuracy falls to 49.2%, friction-coefficient accuracy to 56.9%, and stiffness MAPE rises to 16.84%.The reported results indicate substantial degradation when training-object diversity is severely limited.

C. Ablation Studies

Ablations show that visual–tactile content, interaction flow, semantic priors, and ordinal objectives contribute unevenly across physical-property tasks. The downstream system uses synchronized sensing and online deployment to condition grasping on predicted properties.

  • Input Modalities: Removing visual or tactile content reduces mass accuracy by up to 21.5 percentage points and increases stiffness MAPE by up to 7.62 percentage points.Removing visual content also lowers friction-coefficient accuracy from 97.5% to 84.4%.
  • Flow Ablation: Removing tactile flow lowers mass and friction-coefficient accuracy by 13.9 and 8.7 percentage points, respectively, but improves stiffness MAPE from 9.08% to 8.54%.Removing both flow branches degrades all three tasks, while tactile flow does not consistently improve stiffness regression.
  • Textual Prior: Removing text lowers mass accuracy from 87.5% to 81.0% and increases stiffness MAPE from 9.08% to 12.43%, while friction-coefficient accuracy rises from 97.5% to 98.1%.The textual prior complements visual–tactile evidence for mass and stiffness but does not improve every metric.
  • Loss Weighting: GradNorm improves mass and friction-coefficient accuracy by 9.7 and 4.0 percentage points and reduces stiffness MAPE by 1.93 percentage points relative to manual weighting.Dynamic Weight Averaging provides no consistent improvement over manual weighting in this setting.
  • Ordinal Loss: Replacing ordinal regression with cross entropy reduces mass accuracy from 87.5% to 63.5%, while cross entropy is 0.4 percentage points more accurate for friction coefficient.The ordinal objective therefore benefits mass classification but not both classification tasks equally.
  • Real-Time Deployment: The online deployment combines independently trained ViTacPhys and policy networks, using cumulative-voted property predictions at 30 Hz.On an NVIDIA Jetson Orin, ViTacPhys and the policy require 9 ms and 10 ms per inference, respectively; the VLM initialization takes approximately 10 s before grasping.

B. Human-to-Robot Transfer Results

Human-to-robot transfer combines teleoperation with matched-action human demonstrations and robot-style augmentation, then evaluates property-conditioned grasping on in-distribution and out-of-distribution objects. The combined transfer strategy produces the strongest reported out-of-distribution property-prediction result, while grasping outcomes distinguish clean from total success.

  • Adaptive Grasping: Total success is reported as lift-and-hold success, summing clean and over-force outcomes, for ID and OOD adaptive-grasping evaluation.The figure distinguishes ground-truth-property conditioning from ViTacPhys-predicted-property conditioning, with clean success as the primary metric.
  • Property Prediction: 78.0% mass accuracy, 19.01% stiffness MAPE, and 82.7% friction-coefficient accuracy are achieved on OOD objects when matched-action human data is added to teleoperation and augmented data.The strongest transfer combines human pretraining, robot teleoperation, matched-action human data, and robot-style visual augmentation.
  • Transfer Ablation: Adding augmented data to teleoperation improves mass accuracy from 54.4% to 63.9%, but reduces friction-coefficient accuracy from 75.6% to 69.3%.Stiffness MAPE also increases from 20.29% to 21.54%, so augmentation alone is not uniformly beneficial.

C. Adaptive Grasping Results

ViTacPhys-conditioned grasping improves success relative to ACT, especially on visually similar but physically different objects, while producing force profiles closer to human teleoperation on shared successful OOD objects.

  • 12.5 and 38.9 percentage points higher clean-success rates were associated with ViTacPhys predictions than ACT on ID and OOD objects, respectively.Total-success rates were also 8.3 and 19.5 percentage points higher, respectively.
  • 14.7 percentage points was the clean-success drop from ID to OOD for ViTacPhys predictions, compared with 41.1 percentage points for ACT.ACT also produced over-force successes more frequently.
  • 19.7% lower absolute force error relative to ACT was achieved by ViTacPhys on common successful OOD objects, alongside Pearson r and pairwise-ranking improvements of 0.414 and 0.167.These force metrics were computed only on objects successfully grasped by both methods.
  • ViTacPhys followed human force trends more closely than baselines across evaluated successful property bins, despite noisy individual bins.Human teleoperation generally used larger forces for heavier and stiffer objects and smaller forces for higher-friction objects.
  • The system remains limited by sensing, data scale, latency, and the operational rather than intrinsic interpretation of stiffness.Its tactile maps mainly capture normal pressure; the dataset has 60 objects manipulated by one participant, and deployment requires a one-time pre-contact VLM call.
Loading 2608.21355v1…