PaperlayerView exact revision on arXiv

Source-linked AI summary

ExoBridge: Learning a Bare Hand to Hand-Worn Exoskeleton Mapping through Human Limb Coupling

Ruitong Tian, Xianyao Li, Noah B. Wilson, Fang Xu, Eric Jing Du

arXiv:2610.07692v1cs.RO

TL;DR

Bare-hand video preserves natural interaction but does not directly provide the encoder and tactile measurements of a robot-oriented interface. ExoBridge learns to predict contralateral exoskeleton state from coordinated bimanual demonstrations, with held-out-session evaluation showing predictive contact and intensity across four tasks. The resulting bridge maps bare-hand observations into an exoskeleton state space containing motion and tactile measurements.

  • Problem

    Bare-hand video supports human manipulation demonstrations, but does not directly provide the encoder state and tactile measurements recorded by robot-oriented interfaces.

    Source: §I · p. 1

  • Method

    ExoBridge trains a temporal visual model on coordinated demonstrations, pairing bare-hand RGB video with synchronized motion and tactile measurements from a contralateral exoskeleton.

    Source: §I · p. 1

  • Results

    Across four manipulation tasks, held-out-session evaluation yields a pooled any contact AUROC of 0.916, and relative-motion correspondence is strongest in active channels.

    Source: §I · p. 2, A. Contact Supervision and Prediction · p. 5

  • Takeaways & Limitations

    ExoBridge establishes a learned connection from bare-hand visual observations to an exoskeleton state space containing motion and tactile interaction measurements.

    Source: §VII · p. 7

    Evaluation covers four tabletop tasks and one fixed camera arrangement, and closed-loop use of bridge-predicted states remains untested.

    Source: §VI · p. 7

Abstract

from arXiv · show

Human video offers a scalable source of experience for dexterous robot learning, but obtaining motion and tactile supervision while preserving bare hand interaction remains challenging. We present ExoBridge, a framework that leverages human limb coupling to learn a bridging function from bare hand video to the motion and tactile state of a sensorized exoskeleton. Our central idea is to use coordinated bimanual behavior to connect an uninstrumented visual source with a measured manipulation interface. During collection, one hand remains bare and provides visual observations, while the opposite hand wears the exoskeleton and supplies synchronized motion and tactile measurements. These paired demonstrations train a temporal visual model to predict fingertip contact, continuous tactile intensity, and relative encoder motion from bare hand video alone. The exoskeleton defines an intermediate state space whose motion coordinates are linked to a dexterous robot hand through existing calibration. Evaluation on 1,215 demonstrations across four manipulation tasks uses held out collection sessions and yields a pooled any contact AUROC of 0.916 and a Pearson correlation of 0.790 for tactile intensity. The learned bridge also predicts relative changes in exoskeleton configuration from bare hand video. These results demonstrate that human limb coupling can turn exoskeleton measurements into supervision for bare hand video, establishing a learned bridge between human visual demonstrations and a robot oriented manipulation interface.

I. INTRODUCTION

Bare-hand video preserves uninstrumented interaction but does not directly provide the motion and tactile measurements used by robot-oriented interfaces. ExoBridge uses coordinated bimanual demonstrations to learn those measured states from bare-hand video, and evaluates the bridge across four manipulation tasks.

I. INTRODUCTION

  • Bare-hand video captures human manipulation without a robot during every demonstration, but does not directly reveal interface encoder states or tactile measurements.A sensorized exoskeleton records both signals and has an existing configuration mapping to a target robot hand.Source: §I · p. 1
  • ExoBridge pairs a bare observed hand with a contralateral sensorized exoskeleton hand performing the same objective on a separate object copy.The hands are coordinated without requiring identical joint trajectories; synchronized measurements provide training targets.Source: §I · p. 1
  • From bare-hand RGB video alone, the temporal model predicts fingertip contact, continuous tactile intensity, and changes in exoskeleton encoder configuration.The motion output is relative to the first sampled state, and existing calibration connects exoskeleton configuration to a chosen robot hand.Source: §I · p. 1
  • Held-out-session evaluation across four tasks shows predictive contact and tactile-intensity information, with relative-motion correspondence strongest in active channels.The introduction reports a pooled any-contact AUROC of 0.916; the tasks are egg carton opening, lid twisting, soldering tool orientation, and USB insertion.Source: §I · pp. 1–2
  • The paired dataset contains 1,215 coordinated bimanual demonstrations with synchronized RGB, tactile, and encoder measurements.Session-level splits and evaluation code support reproducible study of cross-hand state prediction.Source: §I · p. 2

II. RELATED WORK

ExoBridge relates to work that uses human demonstrations for dexterous robot learning, but focuses on predicting measured interaction state from the opposite hand’s exoskeleton.

A. Human Demonstrations for Dexterous Robot Learning

  • Human demonstration systems make the representation connecting human observations and robot actions an explicit part of learning.Examples include UMI, DexCap, EgoMimic, and EgoScale.Source: A. Human Demonstrations for Dexterous… · p. 2
  • ExoBridge studies this connection at the state-estimation level, predicting measured fingertip interaction as well as relative motion.Its supervision comes from an exoskeleton worn on the opposite hand.Source: A. Human Demonstrations for Dexterous… · p. 2
  • The predicted state combines tactile interaction with relative motion rather than limiting the mapping to human motion retargeting.The passage contrasts ExoBridge’s output with demonstration approaches that capture or retarget motion to a robot hand.Source: A. Human Demonstrations for Dexterous… · p. 2

B. Wearable Interfaces and Exoskeleton State

ExoBridge uses an exoskeleton as a measured target interface while retaining bare-hand video as input. Its formulation predicts distinct tactile and motion outputs from synchronized visual history.

B. Wearable Interfaces and Exoskeleton State

  • Unlike wearable demonstration systems that place an interface in the demonstration process, ExoBridge uses exoskeleton encoder and fingertip measurements as targets for a separate bare-hand visual stream.The encoder coordinates define motion output, while fingertip sensors define tactile output.Source: B. Wearable Interfaces and Exoskeleton… · p. 2

C. Visual Tactile Estimation and Bilateral Supervision

  • Visual tactile methods estimate measured interaction from appearance, while bilateral approaches pair a visible bare hand with synchronized sensing from another hand.ExoBridge maps that supervision to relative exoskeleton encoder motion and fingertip tactile state.Source: C. Visual Tactile Estimation and… · p. 2

A. Problem Formulation

  • The bridging function takes a bare-hand RGB history paired with contralateral exoskeleton measurements and predicts contact probabilities, tactile intensities, and relative encoder motion.These outputs describe distinct aspects of the synchronized exoskeleton state: fingertip engagement, sensor-response magnitude, and configuration change.Source: A. Problem Formulation · p. 3

B. Coordinated Bimanual Demonstrations

The collection pairs bare-hand video of one hand manipulating an object with synchronized measurements from the opposite hand wearing an exoskeleton and manipulating a separate copy. It covers four tasks and includes 1,215 episodes from 31 collection sessions.

B. Coordinated Bimanual Demonstrations

  • The bare left hand manipulates one object copy in view while the right exoskeleton-wearing hand performs a coordinated instance of the same task on a separate copy.The hands share a manipulation objective without a prescribed identical joint trajectory, and the exoskeleton remains outside the visual model view.Source: B. Coordinated Bimanual Demonstrations · p. 3
  • Synchronized RGB, encoder, and tactile recordings cover egg carton opening, lid twisting, soldering tool orientation, and USB insertion.The tasks span enclosure opening and release, sustained rotation, pickup and reorientation, and precision grasping and alignment.Source: B. Coordinated Bimanual Demonstrations · p. 3
  • 1,215 episodes from 31 distinct collection sessions form the final paired dataset, with session identifiers retained across tasks for evaluation splits.A session is a consecutive block of demonstrations recorded in one sitting; task-level session counts can overlap.Source: B. Coordinated Bimanual Demonstrations · p. 3
  • Figure 2 shows third-person views of the four tasks and bare-hand camera frames ordered top to bottom through one episode per task.Table I summarizes paired bimanual recordings by task.Source: Figure 2 · p. 3, Table I · p. 3

C. Exoskeleton State Targets

The exoskeleton targets encode tactile response relative to each episode's first sampled frame and configuration change relative to its initial encoder state. The model predicts contact, continuous fingertip intensity, and relative encoder motion.

C. Exoskeleton State Targets

  • Contact targets use a fixed threshold on the maximum referenced taxel response for each fingertip, and any contact is the union of the three fingertip targets.The three instrumented fingertips are thumb, index, and middle finger.Source: C. Exoskeleton State Targets · p. 4
  • Continuous tactile intensity retains each fingertip's aggregate referenced taxel response in logarithmic form, with the sum truncated at zero.The learned output includes three scalar fingertip intensities, while the full taxel measurements remain available in the recordings.Source: C. Exoskeleton State Targets · p. 4
  • Relative encoder motion is configuration change from the episode's initial sampled state, expressed in the collection system's fixed reference encoder coordinates.The model predicts all 12 channels, but primary motion evaluation uses channels 0–9; channels 10 and 11 are excluded because their robot mapping differs.Source: C. Exoskeleton State Targets · p. 4

D. Temporal Visual Model

The temporal visual model combines a hand-focused crop with surrounding scene context, then predicts exoskeleton targets from the fused sequence. At inference it uses only current and past bare-hand RGB frames.

D. Temporal Visual Model

  • The model encodes a hand crop and a masked surrounding-scene stream with frozen DINOv2 features, then projects and fuses them in a shared 512-dimensional space.The hand crop is based on a detected bounding box expanded by 25 percent, with normalized box coordinates added to the hand representation.Source: D. Temporal Visual Model · p. 4
  • Six causal residual temporal convolution blocks use kernel size three and dilations 1, 2, 4, 8, 16, and 32, spanning 127 sampled frames.Prediction heads estimate any contact, per-fingertip contact, continuous tactile intensity, and relative encoder motion.Source: D. Temporal Visual Model · p. 4
  • Training uses class-weighted binary cross entropy for contact and standardized Smooth L1 regression for intensity and motion, jointly across the four tasks.The model is trained for 30 epochs with AdamW; exoskeleton measurements provide training targets, while inference uses bare-hand RGB alone.Source: D. Temporal Visual Model · p. 4

E. Evaluation Protocol

Evaluation holds out entire collection sessions and tests contact, tactile intensity, and relative motion prediction on new sessions within the recorded task set. Metrics are averaged across folds and random seeds.

E. Evaluation Protocol

  • Five session-separated folds keep all recordings from a session together, and each held-out episode is predicted by a model trained without that session.The procedure is repeated with three random seeds and evaluates new collection sessions within the recorded task set.Source: E. Evaluation Protocol · p. 4
  • Contact is measured by AUROC; tactile intensity is measured by Pearson correlation in logarithmic target space, normalized MAE, and MAE in recorded sensor units.Intensity normalization uses the held-out target range between the fifth and ninety-fifth percentiles.Source: E. Evaluation Protocol · p. 4
  • Relative motion is evaluated by mean absolute error and Pearson correlation for each channel, alongside a zero-change reference predicting ∆q_t = 0.The reference error measures motion away from the initial state, enabling comparison with remaining at that state.Source: E. Evaluation Protocol · p. 4

IV. EXPERIMENTS

Held-out session experiments test whether bare-hand video predicts measured exoskeleton contact and whether that prediction depends on the paired bimanual recordings. ExoBridge achieves pooled any contact AUROC 0.916, while a decoupled-pairing comparison supports information beyond task progress alone.

IV. EXPERIMENTS

  • The experiments assess contact, continuous intensity, relative motion, and an aligned demonstration, with quantitative results summarized under common folds, seeds, and model configuration.The evaluation uses held-out collection sessions.Source: §IV · pp. 4–5

A. Contact Supervision and Prediction

  • Among 85 held-out frames positive under the exoskeleton threshold, 81 (95.3%) were labeled as visible bare-hand contact from video alone.The 95% frame-level bootstrap percentile interval is 90.2–98.9% over 2,000 replicates; this checks conditional correspondence independently of model prediction.Source: A. Contact Supervision and Prediction · p. 5
  • 0.916 pooled any contact AUROC, with task values from 0.888 to 0.939, shows discrimination of exoskeleton contact targets across four manipulation settings.Evaluation uses held-out collection sessions.Source: A. Contact Supervision and Prediction · p. 5
  • 0.84, 0.82, 0.75, and 0.80 pair AUC for true pairing exceed decoupled pairing by 0.18, 0.24, 0.08, and 0.12 across the four tasks.Paired session bootstrap intervals exclude zero for every task and seed; task-progress-only pair AUC ranges from 0.48 to 0.60.Source: A. Contact Supervision and Prediction · p. 5

B. Continuous Tactile Intensity Prediction

ExoBridge predicts continuous fingertip intensity and relative exoskeleton motion from bare-hand video. Intensity predictions correlate with measurements, while motion prediction reduces error relative to a static initial-state reference, with stronger motion evidence in active channels.

B. Continuous Tactile Intensity Prediction

  • 0.790 Pearson correlation, 0.153 normalized mean absolute error, and 14.8 LSB absolute error characterize pooled continuous intensity prediction across tasks.The target is aggregate fingertip response rather than contact probability alone.Source: B. Continuous Tactile Intensity… · p. 5
  • Soldering tool orientation has higher intensity correlation than lid twisting, 0.821 versus 0.737, but larger sensor-unit error, 19.7 versus 15.1 LSB.Correlation and magnitude error capture different aspects of agreement with the continuous target.Source: B. Continuous Tactile Intensity… · p. 5

C. Relative Encoder Motion Prediction

  • 4.15 pooled mean absolute error across encoder channels 0–9, versus 7.51 for the zero-change reference, a 44.7% reduction.The learned prediction improves over the reference in each task; Table II pools five session folds and three seeds.Source: C. Relative Encoder Motion Prediction · p. 6, Table II · p. 6
  • 0.82, 0.75, 0.71, 0.84, and 0.80 correlations are reported for the thumb, index, middle, ring, and little-finger channels with the largest zero-change reference errors.These are encoder channels 1, 4, 6, 8, and 10; channel 10 is excluded from the pooled channels 0–9 error.Source: C. Relative Encoder Motion Prediction · p. 6

D. Tactile and Motion Outputs in a Held Out Demonstration

An aligned held-out egg-carton episode illustrates tactile and motion outputs over the same interaction timeline. The discussion connects the exoskeleton target space to robot-hand calibration and clarifies that the reported policy results use exoskeleton demonstrations directly, not bridge predictions.

D. Tactile and Motion Outputs in a Held Out Demonstration

  • The held-out egg-carton example aligns bare-hand video, contact, continuous intensity, and relative motion on one timeline at contact onset, peak intensity, and the final sampled state.The final state follows carton opening and hand withdrawal; Fig. 5 trajectories provide temporal context for the three instants.Source: D. Tactile and Motion Outputs in a… · p. 6
  • Predicted relative encoder state is added to the measured initial configuration and rendered through the existing exoskeleton-to-XHand calibration; predicted pads show scalar intensity, not taxel-level responses.Prediction and reference pads use separate color scales.Source: D. Tactile and Motion Outputs in a… · p. 6

V. DISCUSSION

  • Policies trained on exoskeleton demonstrations reached 0.80 success on egg-carton opening and 0.55 on lid twisting over 20 autonomous rollouts.The policies used exoskeleton demonstrations directly, rather than bridge predictions.Source: §V · p. 7, Figure 6 · p. 7
  • The existing calibration links exoskeleton configuration to a robot hand, while the present evaluation establishes the prediction stage rather than downstream use of predicted state for robot learning and control.The discussion identifies a downstream policy study as a next test.Source: §V · p. 7
  • The authors plan to release paired RGB, tactile and encoder measurements, session identifiers, split assignments, and evaluation code, subject to participant consent and data-sharing requirements.The materials are intended to support reproduction of session-separated evaluation and comparison of alternative bridge implementations.Source: §V · p. 7

VI. LIMITATIONS

ExoBridge predicts contact, tactile intensity, and relative encoder motion in held-out sessions, but the evidence is bounded to a fixed-camera tabletop setting and offline state prediction.

VI. LIMITATIONS

  • Evaluation covers four tabletop tasks and one fixed camera arrangement, with held-out generalization confined to collection sessions in that setting.Source: §VI · p. 7
  • All-sample intensity evaluation does not isolate amplitude variation during contact.Source: §VI · p. 7
  • State prediction is offline; rollouts use exoskeleton demonstrations, so closed-loop use of bridge predictions remains untested.Source: §VI · p. 7

VII. CONCLUSION

  • Across four manipulation tasks, ExoBridge predicts measured contact, tactile intensity, and relative encoder motion on held-out sessions, with strongest motion correspondence in active channels.Source: §VII · p. 7
Loading 2610.07692v1…