PaperlayerView exact revision on arXiv

Source-linked AI summary

From Wearable Interfaces to Dexterous Policies: Contact Shifts and Tactile Representations

Ruitong Tian, Fang Xu, Noah B. Wilson, Xianyao Li, Eric Jing Du

arXiv:2610.08870v1cs.RO

TL;DR

Wearable collection interfaces allow direct object interaction, but their geometry can shape both demonstrations and recorded tactile evidence. The paper compares a controlled hand-side revision and two tactile inputs in matched robot evaluations. Revised-interface demonstrations and spatial-plus-force inputs each improve pooled policy success.

  • Problem

    Wearable hardware participates in demonstration-generating interactions, but how interface geometry and tactile representation relate to downstream policy performance is an open question.

    Source: §I · p. 1

  • Method

    The study compares two hand-side geometries with shared robot commands, mapping, calibration, and tactile modules, then trains matched policies using binary aggregate or spatial-plus-force inputs.

    Source: §I · pp. 1–2, C. Recorded Data and Tactile Policy… · p. 4

  • Results

    In matched robot rollouts, revised-interface demonstrations and spatial-plus-force inputs each significantly increased pooled success; spatial-plus-force success rose from 0.35 to 0.52 (common odds ratio 2.1, p = 0.010).

    Source: §VI · p. 6, C. Policy Performance · p. 6

  • Takeaways & Limitations

    Wearable collection systems should be evaluated for the interactions they make recordable and the tactile information available for learning, as well as their human-to-robot motion mapping.

    Source: §VII · p. 7

    The reported contact-shift findings concern lid twisting and egg carton opening, so those task-specific patterns are not established here for other tasks.

    Source: §VI · p. 6

Abstract

from arXiv · show

Unlike conventional teleoperation, wearable interfaces allow operators to collect dexterous demonstrations through their own hand motions while directly interacting with task objects. This direct interaction reduces dependence on the target robot during collection, but it also makes the collection hardware part of the physical process that generates each demonstration. Interface geometry can influence both how a task is performed and what tactile observations are recorded for learning. We study two versions of a DexUMI-family exoskeleton that share the same robot command definition, mapping procedure, and tactile module type but differ in hand-side geometry. The revised interface reduces reported physical demand, improves selected device ratings, enables tactile access in a precision grasp that is mechanically blocked by the baseline, and produces task-dependent changes in recorded contact. Lid twisting primarily exhibits a change in contact location, whereas egg carton opening primarily exhibits a change in contact occurrence. We then train matched policies using either a binary aggregate input or a spatial-plus-force input. On lid twisting and egg carton opening, policies trained on revised-interface demonstrations achieve higher success than those trained on baseline demonstrations, with a significant pooled interface effect. Across these and two additional tasks, USB insertion and soldering tool pick and place, the spatial-plus-force input likewise outperforms binary aggregation. These results show that wearable collection hardware is part of the data-generation process for robot learning. Such interfaces should therefore be evaluated not only through operator experience, but also through the tactile interactions they make recordable and the policy performance their demonstrations support.

I. INTRODUCTION

Wearable interfaces enable direct object interaction during demonstration collection, but their geometry can shape both operator motion and recorded tactile contact. This study tests how a controlled hand-side revision and tactile representation relate to downstream policy success.

  • Robot-aligned wearable constraints can restrict natural finger motion and create localized discomfort, while surrounding fingertip material can block tactile contact.The interface also determines whether contact is accessible, where it occurs on the pad, and how often it is detected.Source: §I · p. 1
  • The study holds robot commands, mapping, calibration, and tactile modules fixed while comparing hand-side geometry, task-dependent contact changes, and policies using two tactile inputs.Binary aggregation retains per-finger contact occurrence but discards within-pad location and force; spatial-plus-force retains location and aggregate fingertip force.Source: §I · pp. 1–2
  • Matched real-robot rollouts show that revised-interface demonstrations and spatial-plus-force inputs each increase dexterous policy success in pooled comparisons.Source: §I · p. 2

II. RELATED WORK

Demonstration interfaces trade off robot dependence, direct object interaction, and correspondence to the deployment hand. This work compares a controlled hand-side revision while treating operator experience and recorded contact as consequences of interface design.

A. Demonstration Collection Interfaces

  • Robot-mediated collection provides deployment-proximal commands but depends on robot hardware, while human video and motion capture still require conversion to robot-feasible actions.Source: A. Demonstration Collection Interfaces · p. 2
  • Wearable interfaces support direct object manipulation with correspondence to a target hand, but place collection hardware inside the physical interaction.Unlike comparisons of complete exoskeleton systems, this study holds robot command, calibration, and sensing fixed while comparing tactile data and policies.Source: A. Demonstration Collection Interfaces · p. 2

B. Mechanical Correspondence and Human Interaction

  • Robot-derived finger spacing, joint geometry, and clearances can constrain natural trajectories or create pressure, and those constraints can also alter grasp data or block measurable contact.The study therefore treats operator experience and recorded contact as two consequences of the same hand-side revision.Source: B. Mechanical Correspondence and Human… · p. 2

C. Tactile Representation and Downstream Learning

Tactile inputs preserve different aspects of contact, and wearable geometry can affect which interactions are recorded. The study frames its hand-side revision as an integrated comparison with the robot command and tactile-sensing pipeline held constant.

C. Tactile Representation and Downstream Learning

  • Binary aggregate signals preserve per-finger contact occurrence but discard within-pad location and force, whereas spatial tactile representations retain those distinctions.A difference present in recorded tactile arrays may disappear before reaching a policy.Source: C. Tactile Representation and… · p. 2

III. INTERFACE REVISION AND POLICY INPUTS

  • The baseline and revised interfaces share the 12-DoF XHand control and tactile-sensing pipeline but differ in finger clearance, fingertip housing, and thumb geometry.Source: §III · p. 2

A. Hand-Side Interface Revision

  • The revision increases clearance for all four fingers and makes the fingertip housing approximately flush with the tactile pad; the changes are evaluated as one integrated revision.The baseline housing extends approximately 5.5 mm beyond the pad, and the tactile modules remain unchanged.Source: A. Hand-Side Interface Revision · p. 3

B. Shared XHand Command and Collection Pipeline

The collection pipeline keeps robot commands and tactile hardware consistent across interface versions, while constructing two tactile inputs that retain different contact information. The spatial-plus-force input preserves more tactile detail than the binary aggregate input.

B. Shared XHand Command and Collection Pipeline

  • Both interfaces use the same 12-DoF XHand command definition, encoder assignment, mapping code, calibration, control frequency, tactile module type, and robot platform.Visual processing is also identical across policy comparisons, leaving interface version and tactile representation as the experimental factors.Source: B. Shared XHand Command and Collection… · p. 3

C. Recorded Data and Tactile Policy Inputs

  • Each demonstration records wrist-camera frames, wrist pose, encoder angles, and a 3×120×3 tactile array from the thumb, index, and middle fingertips.Each taxel measures three-axis force.Source: C. Recorded Data and Tactile Policy… · p. 3
  • The binary aggregate input retains one contact flag per instrumented finger but discards within-pad location and force.The spatial-plus-force input instead retains occurrence, within-pad location, and aggregate three-axis fingertip force.Source: C. Recorded Data and Tactile Policy… · pp. 3–4

IV. EVALUATION PROTOCOL

The evaluation combines operator, tactile-data, and robot-policy tests to examine the effects of interface revision and tactile representation. It includes participant ratings, contact analyses, a dataset-separability diagnostic, and matched robot rollouts.

IV. EVALUATION PROTOCOL

  • Matched robot rollouts test how collection interface and tactile representation each affect policy success.The broader protocol also includes participant experience, tactile accessibility, shared-task contact analyses, and dataset separability.Source: §IV · p. 4
  • The participant study evaluates operator burden and device experience, while the accessibility test checks whether the revision changes recordable fingertip contacts.Shared-task demonstrations are analyzed for contact location and occurrence.Source: §IV · p. 4
  • A dataset-separability diagnostic tests which tactile feature family most clearly distinguishes the two interface datasets.Source: §IV · p. 4

A. Operator Experience and Tactile Accessibility

The study assesses operator experience, tactile accessibility, and the timing of contact during revised-interface demonstrations. Twenty participants evaluated both interfaces, and a precision-grasp test checked fingertip-pad access.

A. Operator Experience and Tactile Accessibility

  • Twenty right-handed participants evaluated both interfaces in a randomized within-subject study, completing lid twisting and egg carton opening before rating workload and device experience.Interface order was assigned baseline-first or revision-first, with version labels and design intent concealed.Source: A. Operator Experience and Tactile… · p. 4
  • Physical Demand was the prespecified primary workload outcome, assessed alongside six raw NASA-TLX dimensions and six custom device-experience items.Paired differences used two-sided Wilcoxon signed-rank tests, with paired sign-flip sensitivity analyses and FDR correction for remaining outcome families.Source: A. Operator Experience and Tactile… · p. 4
  • The soldering-tool precision-grasp test checks whether the revised fingertip housing lets the tactile pad reach the handle before the surrounding housing.Source: A. Operator Experience and Tactile… · p. 4
  • Fig. 3 shows wrist-camera views from one revised-interface demonstration per task, ordered from before contact through early and late contact.Source: Figure 3 · p. 4

B. Demonstration Collection and Contact Measures

Matched demonstrations support contact analyses for lid twisting and egg carton opening, with sessions as the unit of inference. Contact measures distinguish pad location from contact occurrence and are normalized by demonstration length.

B. Demonstration Collection and Contact Measures

  • Contact analysis uses 20 sessions per interface for lid twisting and 16 baseline versus 17 revised sessions for egg carton opening.Baseline egg-carton sessions recorded before an encoder repair were excluded; soldering-tool and USB demonstrations were collected only with the revised interface for robot evaluation.Source: B. Demonstration Collection and… · p. 4
  • The collection session is the inference unit, with session-level permutation tests and standardized differences computed on session means.An automated screen checked demonstrations for encoder-reference drift, encoder or tactile dropouts, flat channels, and insufficient camera overlap.Source: B. Demonstration Collection and… · p. 4
  • Contact location uses pad occupancy, centroid coordinates, and between-interface centroid displacement, while contact occurrence uses per-finger rate and, when needed, active-taxel count.Occupancy and rate are normalized by demonstration length; reported contact rate thresholds normal force, unlike the policy flag's threshold on largest three-axis taxel force magnitude, though both use τ.Source: B. Demonstration Collection and… · pp. 4–5

C. Dataset Separability Diagnostic

The diagnostic compares tactile feature families for separating interface versions, while the policy evaluation crosses collection interface and tactile representation on shared tasks. It also evaluates tactile representations on two additional tasks using revised-interface demonstrations.

C. Dataset Separability Diagnostic

  • The diagnostic holds out complete collection sessions to compare how contact-location and contact-amount-and-timing features distinguish interface versions against shuffled-label chance.It uses a linear classifier on demonstration-level tactile summaries and AUROC; these features are distinct from policy inputs.Source: C. Dataset Separability Diagnostic · p. 5

D. Policy Evaluation

  • Diffusion Policy predicts 16 future wrist-pose and hand-motor actions from one image and a tactile input, without proprioception.The four policies cross collection interface with binary aggregate or spatial-plus-force inputs; policies for USB insertion and soldering use revised-interface demonstrations.Source: D. Policy Evaluation · p. 5
  • Lid twisting, egg carton opening, soldering-tool pick and place, and USB insertion are evaluated using task-specific success criteria.Each policy–task cell is evaluated in 20 rollouts; the shared tasks use matched demonstration budgets across interfaces.Source: D. Policy Evaluation · p. 5
  • The evaluation contrasts tactile representations within task and interface, and revised versus baseline demonstrations within each shared task and input.It summarizes comparisons with Mantel–Haenszel common odds ratios and tests them using exact conditional tests stratified by comparison.Source: D. Policy Evaluation · p. 5

V. RESULTS

The revised interface reduced reported physical burden and restored tactile-pad access in a precision grasp blocked by the baseline housing. The policy evaluation compares success rates across interface and tactile-input conditions.

A. Operator Experience and Tactile Accessibility

  • Physical Demand fell from 11.0±5.5 under baseline to 7.5±4.3 under revised, while Comfort rose from 2.0 ± 0.9 to 3.5 ± 1.0.Effort and overall workload also decreased, and self-rated Performance improved; Mental Demand, Temporal Demand, and Frustration did not differ after correction.Source: A. Operator Experience and Tactile… · p. 5
  • The revised flush fingertip hood lets the tactile pad contact the soldering-tool handle directly, unlike the baseline housing edge that blocks pad contact.This access enabled the soldering-tool demonstrations used in the robot evaluation.Source: A. Operator Experience and Tactile… · p. 5
  • Table II compares interface and tactile-input combinations on shared tasks, then compares tactile inputs for USB insertion and soldering using revised-interface demonstrations.Shared-task conditions use matched demonstration budgets per interface and the same 20-entry task-specific initial-state list.Source: Table II · p. 6, A. Operator Experience and Tactile… · p. 6

B. Task-Dependent Contact Changes

Contact changes depended on the task: lid twisting primarily shifted thumb contact location, while egg carton opening primarily changed index-contact occurrence. In robot rollouts, revised-interface demonstrations and spatial-plus-force inputs each improved pooled success, alongside remaining hardware and study-scope constraints.

B. Task-Dependent Contact Changes

  • Lid twisting shifted thumb contact centroid 8.6 mm, including 7.4 mm along the pad long axis, while carton index-contact frame rate rose from 0.09 to 0.18.Active taxels during carton contact changed little, from 2.7 to 3.0; Fig. 5 visualizes the contact centroids and frame rates.Source: B. Task-Dependent Contact Changes · p. 6, Figure 5 · p. 7
  • Separability diagnostics reinforce the task difference: lid location features reached AUROC 0.99 versus 0.84 for amount-and-timing, while carton feature families reached 0.82 and 0.80.Source: B. Task-Dependent Contact Changes · p. 6

C. Policy Performance

  • Pooled success rose from 0.39 to 0.59 for revised versus baseline demonstrations and from 0.35 to 0.52 for spatial-plus-force versus binary input.The interface comparison had common odds ratio 2.5 (p = 0.013); the input comparison had common odds ratio 2.1 (p = 0.010).Source: C. Policy Performance · p. 6

VI. DISCUSSION AND LIMITATIONS

  • Revised geometry changes which interactions are recordable and what tactile evidence enters the dataset, while binary aggregation drops within-pad location and force retained by spatial-plus-force input.Source: §VI · p. 6
  • Robot-hand precision and exoskeleton mechanics still limit which dexterous tasks can be collected and reproduced on the robot.Backlash and friction make fingertip position depend on motion direction, and the command-to-fingertip mapping shifts over time and across reboots.Source: §VI · pp. 6–7
  • The study covers evaluated objects and tasks; its integrated revision, single policy instance per condition, and combined spatial-plus-force input constrain component-level attribution.Source: §VI · p. 7

VII. CONCLUSION

This work studies wearable collection hardware as part of the data-generation process for dexterous robot learning. A controlled hand-side revision changes recordable contact, and both revised-interface demonstrations and spatial-plus-force inputs increase pooled policy success in matched evaluation.

VII. CONCLUSION

  • The study treats wearable collection hardware as part of the data-generation process for dexterous robot learning.Source: §VII · p. 7
  • A controlled hand-side revision reduces reported operator burden, restores tactile-pad access in a precision grasp, and changes recorded contact in task-dependent ways.Source: §VII · p. 7
  • Matched robot evaluation finds significant pooled policy-success increases for both revised-interface demonstrations and spatial-plus-force input.Source: §VII · p. 7

Related papers

Loading 2610.08870v1…