Source-linked AI summary
Human-Centric Grasp State Assessment: Toward Transferring Subjective Evaluation to Robots
Ryohei Kobayashi, Kosei Isomoto, Yuga Yano, Yuichiro Tanaka, Hakaru Tamukoh
TL;DR
Robots struggle to transfer subjective human judgments of appropriate force to newly encountered deformable objects, while conventional approaches require extensive annotations. The paper combines VLM-based semi-automated supervision with a lightweight predictor using tactile and force time series. Experiments support feasibility in the evaluated setting, including human-rated appropriateness across three objects.
Problem
Robots must balance sliding, damage, and human expectations about deformable-object grasping, but conventional learning requires extensive object-specific annotations.
Method
A VLM propagates criteria from minimally annotated trials, while a lightweight predictor sequentially estimates grasp state from tactile and grasping-force measurements.
Results
The framework demonstrated feasibility for human-aligned grasp-force adjustment in experiments on three deformable objects and human evaluation by 25 participants.
Takeaways & Limitations
Within the evaluated task setting, VLM-assisted supervision and lightweight object-wise adaptation can transfer human-perceived grasp appropriateness to robotic force adjustment.
Takeaways & Limitations
Evaluation was limited to three objects and a small number of sequences, so the results provide feasibility evidence rather than proof of broad generalization.
Abstract
from arXiv · showhide
We propose a framework that transfers tacit human subjective criteria to robotic systems for the appropriate grasping of deformable objects. Achieving such behavior is challenging because a semantic gap exists between qualitative human expectations and quantitative robotic measurements. Conventional deep learning approaches for bridging this gap also require prohibitive amounts of manually annotated data for each newly encountered object. To address these challenges, our framework integrates a Vision-Language Model (VLM)-based semi-automated supervisor generator with a lightweight grasp state predictor, using a minimal set of human-annotated trials as contextual anchors to propagate subjective criteria to unannotated data. The prediction model then enables rapid online adaptation by sequentially estimating the grasp state from time-series tactile and grasping force measurements. Through experiments on three representative deformable objects and a human evaluation study with 25 participants, we demonstrate the feasibility of the proposed framework for adjusting grasping force according to human-perceived grasp appropriateness in the evaluated task setting.
I. INTRODUCTION
Robots must grasp diverse deformable objects while balancing physical stability with subjective human expectations about appearance and tactile integrity. The framework reduces annotation and retraining costs by transferring human criteria through VLM-assisted supervision and lightweight sequential prediction.
- Deformable-object grasping requires balancing insufficient force, which causes sliding, against excessive force, which causes structural damage.
- Human-perceived appropriateness also depends on object-specific material and stiffness, which numerical robot measurements do not directly capture.
- Manual physical thresholds do not scale to diverse objects, while deep learning approaches require large annotated datasets for newly encountered objects.
- The framework reduces annotation cost by using a VLM initialized with a small number of human subjective annotations to generate training data.
- A lightweight time-series learning model supports sequential object-wise adaptation for transferring human evaluation criteria to robotic grasping.
II. RELATED WORK
Prior grasping methods mainly target physical stability, whereas this work addresses human subjective judgments of grasp appropriateness under object-dependent deformation. The proposed approach links those judgments to tactile and force measurements to reduce data demands and speed adaptation.
- Force-control and tactile-feedback methods primarily prevent slip or excessive force but may miss inappropriate visual deformation or loss of appearance.
- Visual-tactile deep networks classify sliding, appropriate, and excessive states but depend on large-scale prior datasets and substantial training time.
- The framework correlates subjective evaluations with tactile and gripper-force measurements, allowing lightweight inference from mechanical data for rapid adaptation.
C. Supervisor Generation using VLMs
The framework uses a VLM-based Supervisor Generator to propagate human grasp judgments from a few annotated examples to unannotated trials. A lightweight Grasp State Predictor then estimates sliding, appropriate, or excessive states from time-series mechanical signals.
- C. Supervisor Generation using VLMs: Structured outputs and human subjective anchoring stabilize VLM-generated training data and reduce manual intervention for newly encountered objects.
- C. Supervisor Generation using VLMs: The framework combines an SG that generates training data from limited annotations with a GSP that predicts three grasp states one step ahead.
- C. Supervisor Generation using VLMs: The GSP uses time-series tactile and grasping-force data, keeping online adaptation computationally practical for continuous operation.
A. Dataset Preparation
Dataset preparation converts multimodal grasp recordings into timestamp-based labels, VLM video inputs, and cleaned mechanical time series. Human annotators mark grasp completion and deformation onset, defining sliding, appropriate, and excessive intervals.
- A. Dataset Preparation: Human annotators label grasp completion and deformation-onset timestamps from RGB videos to establish ground-truth grasp states.
- A. Dataset Preparation: The three labels are sliding before grasp completion, appropriate between grasp completion and deformation onset, and excessive after deformation begins.
- A. Dataset Preparation: For SG training, hand-camera RGB images are horizontally concatenated with tactile z-axis displacement heatmaps into a single video stream.
- A. Dataset Preparation: For GSP training, tactile data are cleaned and combined with grasping force into three-dimensional feature vectors representing mechanical state transitions.
- A. Dataset Preparation: The grasping force increases from 0.1 N to 1.4 N in 0.1 N increments, producing 14 discrete time steps for the mechanical sequence.
B. Supervisor Generator (SG)
The Supervisor Generator uses few-shot human-anchored VLM prompting to transfer object-specific subjective grasp criteria to unannotated trials. Its structured output supports automated integration with the grasp state predictor while reducing manual annotation.
- Few-shot subjective grounding: The method addresses VLM hallucinations and unstable boundary judgments by using contextual anchors rather than generic linguistic instructions alone.This is particularly relevant to continuous physical transitions and object-dependent excessive-state perception.
- Few-shot subjective grounding: Few-shot human-annotated reference videos calibrate the VLM to propagate object-specific subjective grasp criteria across unannotated trials.The references pair videos with human timestamp labels and serve as explicit baselines for the VLM.
- Pipeline integration: The SG output is constrained to predefined JSON, enabling automated parsing of timestamps and reasoning strings for a robust data pipeline.
- Training-data generation: The SG converts minimal human intervention into object-wise training data, reducing manual labor for human-centric control.
C. Grasp State Predictor (GSP)
The Grasp State Predictor is a lightweight sequential module that anticipates grasp-state transitions from tactile and force signals. The evaluation plan covers adaptability, predictability, real-time stability, and alignment with human judgments.
- GSP design: The GSP predicts the grasp state one step ahead, transferring qualitative human criteria into quantitative robot execution through material-specific mechanical signatures.
- Sequential inference: At each time step, the model uses left- and right-tactile values plus gripper force to anticipate upcoming state transitions.An echo state network provides a nonlinear temporal reservoir for interaction dynamics and history.
- Evaluation criteria: The evaluation examines adaptability, predictability, real-time stability, and agreement with human subjective evaluations.
- Experimental scope: Experiments use multimodal data collected from three target objects, with the target-object set shown in Fig. 5.
A. Experimental Setup and Dataset Collection
The dataset comprises multimodal grasping trials on three representative deformable objects, with force varied systematically and all sequences assigned human ground-truth states. Supervisor-generator evaluation varies the amount of annotated context under cross-validation.
- Multimodal sensing: The setup records RGB images, tactile measurements, and grasping-force data using sensors mounted on a Toyota Human Support Robot gripper.
- Target objects: The target objects are a rice ball, sandwich, and paper cup, selected as representative deformable items in unstructured environments.
- Dataset collection: The dataset contains 12 sequences from four trials per object, with grasping force incremented from 0.1 N to 1.4 N and three states assigned by a human annotator.The labels define the baseline for inference-accuracy evaluation.
- SG evaluation setup: The SG comparison provides zero, one, or two annotated example videos per object, reducing evaluated target sequences from 12 to 9 and 6, respectively, with 4-fold cross-validation.
- SG results: Two annotated examples yielded higher accuracy and F1 scores for the rice ball and sandwich, while the paper cup reached 0.97 accuracy and 0.94 F1.One example improved overall scores but left adjacent-state boundary confusions; two examples resolved those ambiguities.
C. Evaluation of Grasp State Predictor
The GSP evaluation isolates prediction performance from supervisor-generator timestamp errors and tests fixed-hyperparameter behavior, adaptability, and predictability. In the adaptability experiment, increasing training data from one to three sequences stabilized state-transition predictions.
- Evaluation scope: GSP scores represent upper-bound prediction performance under human-annotated labels because supervisor-generator timestamp errors are excluded.
- Experimental control: Hyperparameters are optimized using Optuna for test-data waveform-tracking RMSE and then fixed across GSP experiments to avoid tuning bias.
- Adaptability: The adaptability experiment incrementally updates object-specific readout weights while expanding training data from one to three sequences per target object.
- Predictability: The GSP predictability assessment includes accuracy, F1 score, and IoU@appropriate.
- Adaptability: Increasing training data from one to three sequences made the sandwich prediction successfully match ground truth after unstable transition boundaries with one sequence.This supports rapid adaptation with minimal object-specific training data.
2) Evaluation of Predictability:
The grasp state predictor reproduced human-annotated grasp-state boundaries from tactile and grasping-force signals in the controlled object-wise adaptation setting, while the limited sequences constrain claims about generalization.
- 2) Evaluation of Predictability:: IoU@Appropriate measured overlap between ground-truth and predicted time steps classified as appropriate.The metric uses the intersection over union of the ground-truth and predicted appropriate-state intervals.
- 2) Evaluation of Predictability:: 1.0 across all evaluation metrics and objects demonstrated that the GSP reproduced human-annotated grasp-state boundaries in the controlled adaptation setting.The evaluation used Accuracy, F1, and IoU@Appropriate with four-fold cross-validation.
- 2) Evaluation of Predictability:: The results provide feasibility evidence for the evaluated task setting rather than proof of broad generalization because each fold contained few sequences.This limitation qualifies the perfect evaluation scores.
3) Evaluation of Stability:
The GSP selected grasp forces for real-robot transport by stopping before predicted excessiveness, achieving stable transport while exposing sensitivity to object-specific physical variation.
- 3) Evaluation of Stability:: The robot incrementally increased force, selected the preceding force predicted as appropriate, and used it for transport.Force increments were 0.1 N, and the stopping rule was based on the next step entering the excessive state.
- 3) Evaluation of Stability:: Rice-ball errors arose from asymmetrical geometry, non-uniform density, environmental stiffness variation, and criteria learned from limited uniform data.These factors caused the model to interpret an unlearned resistance increase conservatively as reaching the deformation threshold.
- 3) Evaluation of Stability:: The robot successfully transported objects without dropping or crushing them while maintaining force within the ground-truth appropriate interval.The prediction-based motion stop timing remained safely within the appropriate interval despite the rice-ball discrepancy.
4) Subjective Human Evaluation:
A 25-participant study found that the robot’s force-adjusted grasps were judged appropriate in nearly all trials, while the evaluation remained limited by object and sequence diversity.
- 4) Subjective Human Evaluation:: The subjective evaluation used participant judgments of nine recorded robot trials, categorized as sliding, appropriate, or excessive.The questionnaire was administered through Google Forms.
- 4) Subjective Human Evaluation:: More than 90% of participants judged the grasp appropriate in every physical-robot trial except S1.The nine trials covered rice balls, sandwiches, and paper cups, with three sequences per object.
- 4) Subjective Human Evaluation:: S1 received divided evaluations, which the authors attributed to an off-center grasp that made the object appear slightly distorted.Some participants consequently judged the grasp as excessive.
- 4) Subjective Human Evaluation:: The study’s feasibility evidence is restricted by three objects and few sequences, and the current three-state representation only approximates human-centric appropriateness.The authors identify broader objects, users, scenarios, and personalized or probabilistic preference models as future directions.