PaperlayerView exact revision on arXiv

Source-linked AI summary

Temporal Tactile Encoding and Compliance for Intent-Aware Robot-to-Human Bimanual Handover

Pasquale Marra, Stefano Berti, Gabriele Mario Caddeo, Lorenzo Natale

arXiv:2609.05282v1cs.RO

TL;DR

Robot-to-human handover must identify when a person is ready to receive an object, because visual observations can confuse deliberate taking with ambiguous contact. The paper combines a VLA policy, temporally encoded tactile feedback, proprioception, and compliant control, then evaluates full and ablated systems in human-subject trials. The full system achieves the strongest reported objective and subjective handover performance, while tactile history and compliance provide complementary benefits.

  • Problem

    Robot-to-human handover requires timely release, but vision alone may not distinguish deliberate taking intent from accidental, weak, wrong-direction, or transient contact.

  • Method

    The approach fine-tunes a VLA model from human demonstrations using RGB observations, approximately one second of tactile history, proprioception, and a compliance controller.

  • Results

    P1 achieves 93% raw success and 92% success at Tr = 2.8 s, compared with 80% for P3 and 45% for P2.

  • Takeaways & Limitations

    Temporal tactile encoding accounts for most release-robustness improvement, while compliance further improves comfort and smoothness; together they are complementary.

Abstract

from arXiv · show

Reliable robot-to-human handover requires the robot to infer when the person is ready to receive the object, and release it safely, comfortably, and at the right time. This is challenging because visual observations alone may not disambiguate clear taking intent from accidental contact, weak grasping, wrong-direction forces, or transient interactions. In this work we treat human-robot handover as an intrinsically multimodal problem. Our approach couples a VLA model with a compliance controller that reduces interaction forces during object transfer. We finetune the VLA model with human demonstrations using RGB observation, temporally encoded tactile feedback and proprioception. We evaluate the complete system in a human-subject study against two baselines: one without tactile feedback and one using tactile feedback without compliance control. We hypothesize that combining compliance and temporal tactile encoding yields the most reliable and comfortable handovers, as compliance facilitates physical interaction while tactile history captures sustained taking intent. Performance is measured through objective metrics and an ad-hoc questionnaire. The results show that the two components provide complementary benefits and substantially outperform the baselines. Code and data will be released upon acceptance.

I. INTRODUCTION

Robot-to-human handover requires reliable inference of taking intent and appropriately timed release, but vision alone may not distinguish deliberate transfer from ambiguous contact. This work combines temporal tactile encoding, compliance control, and multimodal imitation learning, then evaluates their roles with human subjects.

  • Motivation: Vision alone may not distinguish deliberate grasp-and-pull actions from accidental contact, wrong-direction forces, or transient interactions.Taking intent depends on visual motion, contact, grasp formation, pulling force, and load transfer.
  • Motivation: Compliance reduces resistance during pulling and limits forces during unintended or prolonged contact, but robot yielding can make release behavior ambiguous.Tactile feedback helps separate sustained taking intent from motion caused by compliant response.
  • Approach: The proposed approach uses multimodal imitation learning with RGB observations, fingertip tactile feedback, temporal tactile encoding, proprioception, and compliance control.The model is fine-tuned from demonstrations collected with 8 human subjects, while approximately one second of tactile history captures contact and pulling evolution.
  • Evaluation: The study ablates tactile history and compliance by comparing a full multimodal policy with no-tactile and no-compliance variants.This design isolates the roles of the two components in learned handover behavior.
  • Evaluation: Human-subject evaluation combines objective handover metrics with ratings of safety, comfort, reliability, responsiveness, and preference.The evaluation is designed to assess both physical performance and subjective interaction quality.

II. RELATED WORK

Prior work provides model-based control, visuo-tactile manipulation, and multimodal handover methods, but learned tactile release for human-facing handover remains underexplored. This work addresses that gap with temporal fingertip sensing and an ablation of tactile history and compliance.

  • Model-based bimanual manipulation: Model-based bimanual methods provide stability and safety guarantees through explicit force-distribution, admittance, and impact-aware control laws.These approaches do not learn release behavior from multimodal demonstrations.
  • Visuo-tactile learning: Visuo-tactile learning has improved contact-rich bimanual robot-object tasks such as assembly, pushing, reorientation, and language-conditioned manipulation.These works primarily address robot-object manipulation rather than human-facing release decisions.
  • Human-robot handover: Human-robot handover research covers release timing, intent prediction, role adaptation, mobility, affordance, motion prediction, and soft-hand interaction.Related learning-based work combines demonstrations with impedance control for safe physical interaction.
  • Contact sensing for release: Existing contact-sensing approaches include direction-gated release, tactile-glove classification, and proprioceptive filtering with adaptive release laws.The cited approaches commonly rely on rule-based release logic or specialized controllers.
  • Research gap and positioning: High-frequency fingertip tactile feedback remains underexplored in humanoid foundation-model handover policies.The paper evaluates dense tactile streams with temporal encoding in a multimodal imitation-learning policy against no-tactile and no-compliance variants.

III. METHOD

The method studies bimanual package handover on the humanoid ergoCub, using egocentric vision and fingertip tactile sensing to decide when the human has taken over the load. Demonstrations use teleoperation, while evaluation uses human-subject trials that include misleading contact patterns.

  • Task: The robot holds a box with both hands and releases it once the human receiver has taken over the load.The task uses an egocentric RGB camera and fingertip tactile sensors on both hands.
  • Data collection and evaluation: Teleoperation collects demonstrations, and human-subject trials evaluate the learned handover behavior.The interaction may represent deliberate transfer or misleading contact patterns that resemble one.

B. Teleoperation System and Data Collection

Demonstrations combine teleoperated robot references, visual observations, robot states, and separately sampled fingertip taxel measurements. The tactile pipeline estimates contact forces and trains an autoencoder with reconstruction and directional objectives while excluding near-zero-force directions.

  • Teleoperation and recording: Teleoperation records hand poses, headset orientation, robot states, and visual observations at 10 Hz, while raw Xela fingertip measurements are recorded at approximately 70 Hz.The measurements are timestamped for subsequent alignment, and data are collected from 8 participants.
  • Tactile sensing: Each fingertip sensor uses seven magnetic taxels that displace in three directions, with finger-specific estimators producing 3-axis contact forces.Only four non-thumb fingers on each hand are used because the thumbs do not contact the object during the task.
  • Tactile encoding: The tactile autoencoder is trained with a combined reconstruction and directional-loss objective.The auxiliary losses compare force vectors and force changes using cosine distance.
  • Tactile encoding: Force-direction losses exclude indices whose target force magnitude or target force difference is at most 0.2 N.This removes near-zero vectors for which direction is not well defined.
  • Tactile encoding: The directional-loss weights are λrec = 0.25, λdir = 2, and λ∆ = 1, with temporal interval d = 8 for force differences.These values define the relative contribution of reconstruction, direction, and change-direction terms.

D. Compliance and upper body control

The compliance module uses fingertip forces and hand references to compute compliant hand positions and velocities, allowing the robot to yield during interaction. These outputs are then passed to the torso and arm controller, while the head controller compensates for torso motion.

  • The compliance module receives reference hand positions and velocities together with fingertip forces, then computes compliant hand positions and velocities.It yields along external-force directions so human motion carries the robot’s hands rather than being resisted.
  • Kp and Kd are positive scalar gains set to Kp = 50 and Kd = 40 in the reported controller.
  • The resulting compliant references are provided to the torso and arms controller, which calculates the robot’s torso and arm joint references.
  • The head controller regulates desired orientation while compensating for torso motion to generate head joint references.

E. Policy

The policy is a post-trained VLA model that combines egocentric RGB images with proprioceptive and tactile state inputs to predict short-horizon Cartesian actions for the robot’s hands and head.

  • All policies use the NVIDIA Isaac GR00T-N1.5-3B backbone post-trained for the handover task with a task-specific prompt.The prompt is “Get the box and pass it to human”.
  • The policy receives egocentric RGB images and a proprioceptive/tactile state vector, then outputs 16 Cartesian target actions for both hands and the head.
  • The 16-action predictions correspond to a 1.6 s horizon at 10 Hz, with the first 8 actions executed before the next policy query.
  • All ablated configurations share the same backbone, training recipe, and action space.

A. Stage 1: Instantaneous Force vs. Temporal Tactile Encoding

Temporal tactile encoding reduces false release predictions compared with instantaneous force measurements on a held-out set containing misleading disturbances. The TTE-based policy remains stable until the final genuine pull, whereas the instantaneous-force policy often opens prematurely.

  • A1 uses instantaneous force measurements, while A2 receives the output of temporal tactile encoding under otherwise identical training conditions.
  • A1 repeatedly predicts release after non-causal pulls, whereas TTE-based A2 detects release only during the final genuine pull.
  • 51 threshold crossings for A1 versus 3 for A2 were observed across 10 held-out Challenging Test Set episodes.The set contained 3,776 frames and included shakes, taps, and wrong-direction pushes.
  • 4 sustained false openings longer than 1.5 s occurred for A1, while A2 produced none during the hold phase.

B. Stage 2: Human-Subject Evaluation

Stage 2 isolates tactile-history and compliance contributions by comparing a full policy with variants that remove one component. The no-compliance policy is trained on a separately collected, frame-matched dataset for a fair comparison.

  • The three policies are P1 with TTE and compliance, P2 with compliance only, and P3 with TTE only.Comparing P1 with P2 isolates tactile history, while comparing P1 with P3 isolates compliant control.
  • P3 is trained on a separate no-compliance dataset whose episode count is matched to the compliant dataset.
  • Because episode lengths differ, the compliant dataset is subsampled to match the number of training frames used for P3.

2) Protocol:

The protocol evaluates three policies across release-expected and hold-expected interactions designed to test handover success, intent sensitivity, and robustness to misleading contact. Success is measured both by expected trial outcome and by release timing within a learned threshold.

  • Participants and trial design: Ten participants each evaluated all three policies using 5 release-expected and 5 hold-expected trials per policy.Policy order was counterbalanced across participants.
  • Release-expected trials: Release-expected trials tested standard, repeated, weak, slow, and motion-perturbed handovers requiring release after clear taking evidence.These trials assessed success, consistency, weak-intent sensitivity, and whether release followed actual pulling.
  • Temporal evidence: Equal peak force did not trigger release when pulls were brief, whereas sustaining the same force for ∼1.5 s triggered release.The brief pulls lasted approximately 0.2 s, showing that persistence rather than magnitude governed the observed release decision.
  • Hold-expected trials: Hold-expected trials tested accidental contact, taps, wrong-direction forces, robot-induced motion, and transient pulls requiring the robot to keep holding.Bending trials specifically tested whether compliance-related robot motion was mistaken for taking intent.
  • Metrics: Each policy was evaluated over 100 trials, with success defined as release in R trials and no release during the initial 30 s disturbance phase in H trials.Timing-constrained success counted release attempts across both R trials and final H-trial release phases, using threshold Tr = 2.8 s.

3) Results:

The full system achieved the strongest objective and subjective handover performance, while the ablations exposed distinct roles for tactile history and compliance. Tactile history supported reliable release decisions, whereas compliance improved force requirements, smoothness, and comfort.

  • Objective performance: P1 achieved 93% raw success and 92% timing-constrained success at Tr = 2.8 s, exceeding P3 at 80% and P2 at 45%.P1 also had the fastest median release delay, 1.70 s, and lowest peak pulling force, 4.07 N.
  • Trial-type breakdown: P1 reached 100% success on seven of ten trial types, with lower performance on T6 at 80%, T10 at 90%, and T9 at 60%.T9 was the most difficult condition because impulsive pulls could match genuine force magnitudes while remaining shorter than sustained release-triggering pulls.
  • Ablation comparison: P2 performed worst on almost all objective metrics, including 10% success on T4 and 30% on T7 hold-expected trials.Without tactile input, compliant robot motion and incidental human movements could be interpreted as intentional taking.
  • Ablation comparison: P3 achieved 82% success versus P1’s 93% but required higher forces, responded more slowly, and performed worse on some release-expected trials.Removing compliance left tactile history intact but prevented the robot from yielding to human motion.
  • Questionnaire results: P1 received the highest smoothness score, 6.2, and overall satisfaction score, 6.2, while 9 of 10 participants ranked it first.P2 and P3 scored 3.8 and 3.2 for smoothness, and 3.3 and 4.7 for satisfaction, respectively.

V. CONCLUSION

The conclusion identifies temporal tactile encoding and compliance as complementary design choices for learned humanoid handover. Tactile history mainly improves release robustness, while compliance improves comfort and smoothness through a faster physical response.

  • Tactile representation: Temporal tactile encoding was more robust than instantaneous force readings for preventing false releases.The conclusion frames tactile representation as a central design choice in learned handover policies.
  • Combined design: Combining temporal tactile encoding with compliant control produced the most reliable, responsive, and preferred handover behavior.The conclusion reports this outcome across the two-stage evaluation.
  • Complementary roles: Tactile history accounts for most release-robustness improvement, whereas compliance further improves comfort and smoothness.The two components are described as complementary rather than redundant.
  • Timescale separation: Compliance reacts within a low-level control cycle to reduce resistance, while the learned release decision follows after temporal tactile integration.This suggests a hierarchy in which fast physical response and slower evidence integration serve different timescales.
Loading 2609.05282v1…