PaperlayerView exact revision on arXiv

Source-linked AI summary

TactAlign: Human-to-Robot Policy Transfer via Tactile Alignment

Youngsun Wi, Jessica Yin, Elvis Xiang, Akash Sharma, Jitendra Malik, Mustafa Mukadam, Nima Fazeli, Tess Hellebrekers

arXiv:2602.13579v1cs.RO

TL;DR

TactAlign addresses the challenge of transferring human tactile signals to robots with different sensors and embodiments, where prior approaches often require paired data or shared sensing. It learns modality-specific latent representations and aligns them with rectified flow guided by interaction-derived pseudo-pairs, improving transfer across contact-rich tasks, unseen objects and tasks, and zero-shot light-bulb screwing.

  • Problem

    Human tactile feedback is valuable for dexterous H2R learning, but existing tactile approaches often assume identical sensors, limited embodiment gaps, or paired demonstrations.

  • Method

    TactAlign independently learns human and robot tactile representations, then uses rectified flow and hand-object-interaction pseudo-pairs to align their latent spaces without explicitly paired datasets.

  • Results

    TactAlign improves H2R transfer across pivoting, insertion, and lid closing, generalizes with ≤5 minutes of human data, and achieves +100% improvement on zero-shot light-bulb screwing over policies without tactile input or alignment.

  • Takeaways & Limitations

    The method supports cross-embodiment tactile policy transfer across heterogeneous sensors, objects, and tasks using limited human demonstrations.

  • Takeaways & Limitations

    Evaluation is limited to a single glove-robot pairing, and tactile alignment alone does not address visual discrepancies between human and robot embodiments.

Abstract

from arXiv · show

Human demonstrations collected by wearable devices (e.g., tactile gloves) provide fast and dexterous supervision for policy learning, and are guided by rich, natural tactile feedback. However, a key challenge is how to transfer human-collected tactile signals to robots despite the differences in sensing modalities and embodiment. Existing human-to-robot (H2R) approaches that incorporate touch often assume identical tactile sensors, require paired data, and involve little to no embodiment gap between human demonstrator and the robots, limiting scalability and generality. We propose TactAlign, a cross-embodiment tactile alignment method that transfers human-collected tactile signals to a robot with different embodiment. TactAlign transforms human and robot tactile observations into a shared latent representation using a rectified flow, without paired datasets, manual labels, or privileged information. Our method enables low-cost latent transport guided by hand-object interaction-derived pseudo-pairs. We demonstrate that TactAlign improves H2R policy transfer across multiple contact-rich tasks (pivoting, insertion, lid closing), generalizes to unseen objects and tasks with human data (less than 5 minutes), and enables zero-shot H2R transfer on a highly dexterous tasks (light bulb screwing).

I. INTRODUCTION

TactAlign addresses the underexplored problem of transferring human tactile feedback to robots with heterogeneous embodiments. It uses self-supervised modality-specific representations and rectified-flow alignment from noisy pseudo-pairs, improving cross-task and cross-object H2R transfer with limited human data.

  • Human tactile feedback remains underexplored in H2R learning despite its role in dexterous, contact-rich manipulation.
  • Existing tactile H2R methods often assume identical sensors, minimal embodiment gaps, or strictly paired human-robot demonstrations.These assumptions limit applicability across diverse robot hands and can make correspondence maintenance prohibitively difficult.
  • TactAlign independently self-supervises human and robot tactile encoders, then aligns their latent spaces with rectified flow guided by hand-object-interaction pseudo-pairs.The method targets heterogeneous tactile sensors without explicitly paired datasets.
  • +59% H2R co-training success versus no tactile and +51% versus no alignment across pivoting, insertion, and lid closing, using ≤5 minutes of human data.
  • +100% improvement enables zero-shot dexterous light-bulb screwing transfer over policies without tactile input or alignment.

III. METHODOLOGY

The methodology learns modality-specific tactile representations from separate human and robot demonstration datasets, then maps them into a shared latent space. Cross-attention pooling produces fixed-dimensional features despite heterogeneous tactile signal dimensions.

  • A. Problem Statement: TactAlign assumes offline human-glove and heterogeneous-robot demonstration trajectories, with the robot dataset smaller because robot data collection costs more.
  • A. Problem Statement: The datasets represent variable-length trajectories containing fingertip tactile observations, fingertip poses, object poses, and wrist poses at each time step.
  • A. Problem Statement: For same-task trajectories sharing start and end object states, object pose estimates establish initial cross-domain correspondences without privileged information.
  • B. Tactile Self-supervised Learning: Human and robot tactile observations are encoded independently into latent vectors using modality-specific encoders and decoders trained with self-supervised reconstruction.The architecture uses a JEPA-based design with mean squared error reconstruction loss.
  • B. Tactile Self-supervised Learning: Cross-attention pooling gives both modalities a consistent fixed-dimensional latent representation in R^d despite differing tactile dimensions.

C. Pseudo-Pair Extraction from Demonstrations

TactAlign constructs noisy tactile pseudo-pairs from similar human and robot hand-object transitions, then filters them using contact consistency. These correspondences guide rectified flow toward useful cross-domain transport while tolerating noise.

  • Human and robot transitions are compared in normalized pose space using position and object-pose differences, with λ balancing pose and velocity terms.
  • Pseudo-pairs are formed from same-task, same-object demonstrations sharing reset and goal states, subject to a global similarity threshold δ.The pairs provide only an initial alignment guide because they are inherently noisy.
  • Pseudo-pairs guide rectified flow toward desired source-target correspondences rather than the random pairings used by standard rectified flow.The learned transport remains robust despite noise in the pseudo-pairs.
  • Binary contact filtering retains only contact-to-contact and non-contact-to-non-contact transition pairs.This helps distinguish subtle contact transitions that configuration-space similarity may miss.

D. Tactile Alignment via Rectified Flow

TactAlign formulates cross-embodiment tactile alignment as rectified flow over latent tactile states, using pseudo-pairs to learn a velocity field that transports human features into robot space. The transformed features support a shared human–robot policy.

  • Formulation: Rectified flow models tactile alignment as a conditional distribution p(x | t, z), where x evolves over normalized time t ∈[0, 1].The method learns a latent transport process conditioned on variables derived from pseudo-pairs.
  • Pseudo-pair guidance: Pseudo-pairs from offline hand-object interactions provide coarse correspondences across tasks instead of the random pairings used by original rectified flow.These noisy pseudo-pairs guide cross-sensor latent transport without requiring strict paired demonstrations.
  • Velocity-field learning: TactAlign learns a velocity field vθ through least-squares regression along interpolated latent states between human and robot tactile features.For each pseudo-pair, the method defines an interpolated state and trains the velocity field over t ∈[0, 1].
  • Inference: At inference, simulating dxt = vθ(xt, t) transforms a human tactile feature hi into the robot-space feature ˆhi.The implementation solves the ODE with a vanilla Euler method using a constant step size.
  • Policy learning: A shared H2R policy consumes human or robot tactile latents and predicts action chunks containing fingertip locations and wrist orientation.Human wrist orientation is adjusted by a constant offset, and the policy runs at 10–30 Hz during execution.

IV. EXPERIMENTS AND RESULTS

Experiments use heterogeneous glove and robot fingertip sensors, self-supervised tactile encoders, and rectified-flow alignment data collected from contact-rich tasks. The aligned latent features nearly overlap across domains and preserve a cross-domain trend in contact-force magnitudes without using force during training.

  • Hardware: The shared H2R action policy combines robot, human, and shared modules, with human glove features passed through an ODE solver and proprioception encoding fingertip locations and wrist orientation.Only the yellow modules are trained; the remaining modules are frozen.
  • Hardware: OSMO glove and Xela robot sensors differ in sensing mechanisms, signal characteristics, and spatial resolution despite both being magnetic-based.OSMO provides 1 × 3 fingertip sensing, whereas Xela provides 30 × 3 sensing.
  • Data and training: Tactile encoders are pretrained with approximately 10 minutes of play data plus an in-domain tactile alignment dataset.Human and robot encoders are trained independently before cross-sensor alignment.
  • Data and training: Rectified-flow alignment uses 100 robot and 200 human demonstrations from pivoting and insertion, while policy co-training evaluates pivoting, insertion, and lid closing.The co-training setup includes shared objects, human-only objects, and a single robot training object.
  • Alignment results: 78%: EMD between human and robot tactile distributions decreases from 0.091 to 0.020 after alignment.UMAP projections show human glove features moving toward and nearly overlapping robot features.
  • Alignment results: Aligned features preserve a consistent cross-domain trend in normalized force magnitudes, although force is never explicitly used during training.Higher-force glove features tend to map to higher-force robot features, and vice versa.

D. Human-Robot Policy Co-Training

TactAlign combines robot and human demonstrations with tactile alignment to train policies for contact-rich tasks and generalize across objects and tasks. Removing tactile input or alignment substantially weakens performance, while the full method achieves strong success across pivoting, insertion, and lid closing.

  • Evaluation tasks: The evaluation covers pivoting, insertion, and lid closing, requiring contact detection and force reasoning from initially non-contact states.Pivoting and insertion test unseen objects within the same task; lid closing additionally tests an unseen task.
  • Robot-only baseline: Human demonstrations, collected approximately 4× faster than robot data in this setting, markedly improve generalization to unseen objects.Without tactile modality, performance improves +10% on seen-by-both objects, +59.3% on human-only objects, and +54.4% on held-out objects.
  • Without Tactile baseline: +59% average success results from incorporating tactile input across the three contact-rich tasks compared with the no-tactile baseline.The largest gap appears in pivoting, where success improves by up to +100%.
  • Without Alignment baseline: Removing tactile alignment causes a −51% drop in average success rate compared with full TactAlign.Non-aligned tactile features can produce near-complete failure on seen-by-both pivoting and insertion objects.
  • Without Alignment baseline: The alignment-free performance gap is smaller for lid closing, with a −23% drop, because the task may admit broader successful contact strategies.This task was not used during alignment training and may benefit from coarse contact cues in raw tactile signals.
  • TactAlign results: TactAlign reaches 76%, 72%, and 74% success on pivoting, insertion, and lid closing, respectively, while averaging 100%, 71%, and 65.5% across seen-by-both, human-only, and held-out objects.The results cover tasks used for alignment and lid closing, which was not used during alignment or encoder training.

E. Dexterous Robot Policy Learning with Human Data Only

TactAlign transfers human tactile guidance to a dexterous robot policy without robot data for light-bulb screwing and evaluates whether aligned features preserve force information. The policy succeeds zero-shot, while alignment substantially closes the cross-sensor force-prediction gap but remains limited to one glove-robot pairing.

  • Dexterous task: TactAlign achieves a 100% success rate on light-bulb screwing using human demonstrations only and zero robot data.The task is dexterous and occlusion-heavy, making teleoperation difficult without tactile feedback and reliable visual cues.
  • Baseline comparison: Without tactile input or alignment, light-bulb screwing success falls to 0%.Without tactile input, fingertips fail to establish stable contact; without alignment, jamming prevents recovery.
  • Force evaluation: The force-prediction evaluation tests whether aligned latent features preserve physically meaningful information across heterogeneous tactile sensors without using tactile observations or force measurements during training.A robot force decoder is trained on frozen TactAlign features and evaluated under human-to-robot and robot-to-robot settings.
  • Force evaluation: TactAlign reduces ℓ1 force-prediction error by approximately 98%, 99%, and 93% along Fx, Fy, and Fz, respectively.The aligned human-to-robot setting also significantly reduces variance across runs.
  • Force evaluation: Aligned human-to-robot performance comes within 2% of the robot-to-robot baseline on Fx and within 13% on Fy, while Fz remains 0.7N higher.Robot-to-robot evaluation represents the best-case upper bound.
  • Limitations: The evaluation is limited to a single glove-robot pairing, and tactile alignment does not address visual discrepancies between human and robot embodiments.Additional tactile modalities, multihand or full-palm sensing, and unified multimodal policies remain future directions.

APPENDIX A DATASET

The dataset treats the human hand as a dexterous, embodiment-independent data source and collects aligned human–robot demonstrations, force references, poses, and diverse evaluation objects.

  • A. Wearable Devices: The human hand offers up to 23 degrees of freedom and is proposed as a universal embodiment for collecting reusable dexterous data.This is contrasted with exoskeleton-based interfaces that constrain natural motion and tend to produce embodiment-specific data.
  • A. Wearable Devices: The human dataset uses an OSMO tactile glove that preserves full dexterity while capturing shear and normal tactile signals.
  • Dataset Organization: Pivoting and insertion reuse seen-to-both object demonstrations from the cross-sensor tactile alignment dataset.
  • Force Data Collection: An ATI Gamma force-torque sensor is fixed beneath the table, while glove and Xela fingertips are aligned along the sensor frame’s +x direction.The setup provides a stable reference frame and consistent contact orientation across trials.
  • Force Dataset: Robot and glove train/test sets are sampled from force distributions with comparable mean x- and y-components, while robot mean force is approximately 0.5 N larger.
  • Evaluation Objects: Evaluation objects span diverse geometries and physical properties, with objects categorized as seen-by-all, human-only, or heldout.The categories support testing generalization across manipulation settings.
  • Pose Extraction: Fingertip and wrist poses are extracted with WiLoR and SAM3, refined using ICP and depth maps, then temporally smoothed with a Savitzky–Golay filter.
  • Object Pose Extraction: Object poses use a mesh-less pipeline combining FoundationPose with FoundationStereo depth and automatically reconstructed meshes from RGB segmentation.Initial-frame estimation is followed by tracking for temporal consistency, with texture generation helping disambiguate symmetry.

B. Signal Post Processing

Signal post-processing removes sensor baselines, compensates for glove sign ambiguity and robot drift, and applies robot-derived task normalization to both embodiments.

  • Human Signals: Human tactile signals are baseline-corrected using non-contact measurements, converted to absolute values, and collected at 30 fps.Absolute values address sign flips caused by the glove’s magnetic skin quadrants under identical applied forces.
  • Robot Signals: Robot tactile signals are collected at 100 fps and baseline-corrected before each data-collection session or rollout to compensate for drift.
  • Task Normalization: For each task, normalization statistics are computed from robot data and applied identically to human and robot observations.The procedure subtracts the mean and divides by the maximum standard deviation across axes.
  • Fingertip Positions: Robot fingertip positions are normalized by subtracting task-level robot means and dividing by the maximum per-axis standard deviation.

2) Object pose normalization:

Object positions and orientations are normalized separately in task-specific robot-derived spaces, with orientations represented as rotation vectors.

  • 2) Object pose normalization:: Object pose normalization treats object pose components separately for both human and robot embodiments.
  • Object Positions: Object positions are normalized by subtracting task-level robot means and dividing by the maximum per-axis standard deviation.
  • Object Orientations: Object orientations are represented as rotation vectors and normalized with the same mean-and-maximum-standard-deviation procedure.

B. Balancing Term

The alignment objective uses a fixed balancing term and thresholded nearest-neighbor pseudo-pairs, while sensitivity analyses assess robustness to both choices and the tactile contact threshold.

  • B. Balancing Term: λ = 1 is used across tasks and embodiments, and alignment remains stable over a reasonably wide range of λ values.Table IV reports EMD reduction rates before and after alignment for different balancing terms.
  • Pseudo-Pair Construction: Pseudo-pairs match human and robot transitions from the same task, object, reset state, and goal state using a global similarity threshold δ.Nearest-neighbor matching selects N robot transitions, while δ excludes degenerate matches with large pose discrepancies.
  • Threshold Sensitivity: Within a reasonable range, pseudo-pair construction and alignment performance remain largely consistent as δ changes; experiments use N = 3 and δ = 2.0.Values that are too large can admit low-quality or semantically meaningless pairs.
  • Contact Thresholds: Non-contact thresholds are selected from separated raw-signal norm distributions, capturing the near-zero non-contact peak while excluding meaningful contact signals.
  • Implementation: The rectified-flow velocity model uses a three-hidden-layer multilayer perceptron with width 1024 and takes about 10 minutes to train on one RTX 4090.Training uses 100 discretized time steps for 200,000 epochs at a learning rate of 5×10^-5.

B. H2R Policy – Pivoting

The policy configurations vary by task, while the co-training setup uses tactile and proprioceptive inputs with low-dimensional wrist- or fingertip-based actions. Inference executes action chunks at task-specific rollout frequencies and converts predictions into robot commands.

  • Pivoting uses index-finger tactile and proprioceptive signals to predict a 6-dimensional index-fingertip position and wrist-rotation action in the robot base frame.
  • Insertion uses tactile and proprioceptive signals from the thumb, index, and middle fingers, with the same 6-dimensional index-fingertip and wrist-rotation output.
  • Lid closing uses all four fingertips as policy inputs, while its 6-dimensional output specifies index-fingertip position and wrist rotation for wrist-only control.
  • The glove latent evolves toward the robot tactile distribution through rectified flow, as illustrated by the shared color coding across the latent-evolution figures.
  • The policy runs at 10 Hz with 32-step action chunks for co-training tasks, executing 4, 2, and 8 actions per rollout for pivoting, insertion, and lid closing.Predicted representations are converted into joint commands through inverse kinematics.
Loading 2602.13579v1…