PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Supervised Anchoring of Fingertip Sensing to Proprioception and Proactive Actions for Robot Imitation Learning

Tomohiro Motoda, Masaki Murooka, Keisuke Shirai, Hanbit Oh, Ryoichi Nakajo, Shotaro Miwa, Roman Mykhailyshyn, Hugo Duarte, Yukiyasu Domae

arXiv:2609.29822v1cs.RO

TL;DR

Fingertip sensors provide complementary information before and after contact, but their phase-dependent signals are difficult to exploit from limited demonstrations. PROPRA independently aligns each sensor history with proprioception and action sequences; it achieves the highest reported average success rate among the compared pretraining conditions, though its advantage is not uniform across tasks.

  • Problem

    Phase-dependent fingertip signals and limited demonstrations make it challenging to learn representations that capture when each sensor carries useful manipulation information.

    Source: §I · p. 1, B. Interval Structure of Sensor… · p. 3

  • Method

    PROPRA independently aligns tactile and proximity histories with proprioception and action sequences as a common sensorimotor anchor.

    Source: §I · pp. 1–2, B. Encoders and Learning Objective · p. 4

  • Results

    PROPRA achieved the highest average success rate (75.0%, vs. 71.7% for the image anchor and 72.1% with the direct pair), although reported PickCup and PickSponge differences were within variation across seeds.

    Source: signals help, compared with image… · p. 6

  • Takeaways & Limitations

    The results indicate that sensorimotor signals can anchor sensors whose informative responses are confined to particular intervals without directly aligning observation modalities.

    Source: §VII · p. 8

    The study targets grasp-oriented manipulation with two retrofitted sensors and limited demonstrations; PROPRA’s advantage is less apparent on OpenLid, where sensor responses vary over a long trajectory.

    Source: D. Limitations and future perspective · p. 8

Abstract

from arXiv · show

Robotic imitation learning often relies on external cameras, yet local interaction cues such as object proximity, contact onset, and grasp state are difficult to observe near the fingertips because of occlusion and limited temporal resolution. We study how to effectively incorporate complementary fingertip sensing into imitation learning using pressure-sensitive tactile and reflective proximity sensors, along with pretrained sensor encoders. The two modalities provide information at different manipulation phases: proximity sensing is informative before contact, whereas tactile sensing becomes informative after contact. However, naively adding these signals to a policy does not consistently improve performance and can even underperform vision-only policies, suggesting that sparse, phase-dependent sensor signals are difficult to exploit from limited demonstrations. We therefore propose a proprioception-anchored pretraining method, PROprioceptive-and-PRoactive Anchoring (PROPRA), which independently aligns each fingertip sensor history with proprioceptive and action segments. This provides a continuously available sensorimotor reference, allowing each sensor to be aligned independently during its informative phases. Experiments on real-world manipulation tasks show that our pretraining method improves average success rates over vision-only policies and image-anchored pretraining baselines. Representation analysis further shows that it preserves richer information about pre-contact states, enabling more effective use of complementary fingertip sensing. Please refer to our project page: https://tomohiromotoda.github.io/nia.propra/

I. INTRODUCTION

Fingertip sensors provide complementary cues across manipulation phases, but sparse, phase-dependent signals make them difficult to exploit through naive fusion or direct alignment. PROPRA independently aligns each sensor history with proprioception and action chunks, and experiments report higher average success rates than vision-only and image-anchored policies.

I. INTRODUCTION

  • Fingertip sensing is motivated by human multimodal feedback, which supports precise movement and reliable grasping when visual information is partially or temporarily unavailable.The paper presents robotic fingertip sensing as a direction toward more dexterous and robust manipulation.Source: §I · p. 1
  • Phase-dependent fingertip signals make representation learning challenging: direct modality alignment can provide weak supervision, and naive concatenation can degrade policy performance.Proximity and tactile sensing become informative at different manipulation stages, so temporal co-occurrence may be unreliable.Source: §I · p. 1
  • PROPRA uses proprioception and action chunks, available throughout manipulation, as a common anchor for independently aligning fingertip sensor histories.The approach associates each modality with states and motion during its informative phase rather than forcing alignment between modalities that respond at different times.Source: §I · pp. 1–2
  • The proposed fingertip system combines pre-contact proximity sensing with post-contact tactile sensing to capture complementary interaction states during real-robot grasping.The two sensors are integrated on the same fingertip.Source: §I · p. 2
  • The authors report higher average success rates than vision-only policies and image-anchored pretraining while retaining pre-contact information in learned representations.They also report this retention on recordings from policy execution.Source: §I · p. 2

II. RELATED WORK

Prior work uses tactile sensing for post-contact interaction and proximity sensing for pre-grasp state, while learning useful representations from limited demonstrations remains a challenge. Existing approaches align modalities through shared anchors, motivating PROPRA’s sensorimotor anchor for phase-complementary fingertip signals.

II. RELATED WORK

  • Tactile sensing has been used to improve dexterity by directly measuring robot–object interaction, including contact geometry, force, slip, and object pose –.Prior learning work also includes self-supervised visuotactile learning and visual–tactile alignment [12]–[14].Source: §II · p. 2
  • Tactile sensing captures post-contact interaction properties, while proximity sensing estimates fingertip–object distance and configuration before contact,, [5].Combining them provides complementary observations over a broader part of grasping.Source: §II · p. 2
  • Learning useful sensor representations from limited demonstrations is a key challenge; prior work includes action-conditioned future tactile prediction, future-visual supervision, and cross-sensor tactile representations [18]–[21].Other work explores visual, tactile, and visuotactile pretraining,,.Source: §II · p. 2
  • Shared-anchor approaches align modalities to images or a common latent space, while fixed anchors can limit information and modality-specific or inter-modality relationships –[24].ImageBind uses images as an anchor; Uni-Touch applies a similar strategy to tactile learning.Source: §II · p. 2
  • PROPRA instead uses proprioception and action sequences as a sensorimotor anchor for complementary pre-grasp and contact-related representations.The motivation is that fingertip sensors are informative at different interaction phases, making direct same-time alignment potentially insufficient.Source: §II · p. 2

III. PROBLEM FORMULATION

The study formulates grasping as a setting where proximity and tactile signals become informative at different stages, then describes a retrofittable fingertip system and the phase structure used to analyze sensor responses. It also identifies limitations of directly adding sensor histories to a policy trained from limited demonstrations.

III. PROBLEM FORMULATION

  • Proximity sensing tracks fingertip–object distance during approach, while tactile sensing captures contact onset and subsequent grasp state, providing observations from approach through holding.The authors integrate both sensors on the same fingertip.Source: §III · p. 2

A. Hardware design

  • The fingertip module uses a force-sensitive resistor for tactile sensing and a reflective optical distance-sensing circuit for proximity sensing.Signals are acquired synchronously at 100 Hz, and both components can be retrofitted to an existing parallel gripper.Source: A. Hardware design · p. 2

B. Interval Structure of Sensor Responses

  • Contact onset is defined as the tactile signal’s rise from its stationary baseline; the preceding 60 control steps, about 2 seconds, form the approach interval.Earlier pre-grasp time is labeled pre, and time from contact onset onward is contact; these labels are for analysis and evaluation, not training.Source: B. Interval Structure of Sensor… · p. 3
  • During approach, proximity varies and is most discriminative just before contact while tactile remains near baseline; tactile rises during contact.During the earlier pre interval both signals remain near their stationary baselines.Source: B. Interval Structure of Sensor… · p. 3
  • At least one fingertip sensor remains near its stationary value for more than half of pretraining samples, reflecting each sensor’s limited informative interval.The sensors are complementary, but neither is discriminative throughout the episode.Source: B. Interval Structure of Sensor… · p. 3
  • Directly concatenating sensor histories does not ensure effective use: with limited demonstrations, performance falls below the vision-only policy on some tasks.The stated challenge is learning both what each signal means and when to use it; Table I reports success rates for naive concatenation without pretraining.Source: B. Interval Structure of Sensor… · p. 3, Table I · p. 3

C. Missing Supervision in Contrastive Pretraining

Direct contrastive alignment can provide little useful supervision when paired sensors are temporally out of phase, especially during approach. The section motivates an anchor that is available throughout manipulation and meaningfully coupled to the robot’s interaction state and motion.

C. Missing Supervision in Contrastive Pretraining

  • When one member of a same-time contrastive pair stays constant, the loss need not preserve temporal distinctions in the other modality.Because tactile sensing remains near baseline during approach while proximity varies, direct tactile–proximity alignment provides limited supervision there.Source: C. Missing Supervision in Contrastive… · p. 3
  • Removing degenerate samples does not resolve the supervision gap: remaining direct sensor pairs concentrate near contact and leave most of approach without useful supervision.In the demonstration data, tactile observations cannot predict the corresponding proximity variation during approach.Source: C. Missing Supervision in Contrastive… · p. 3
  • A suitable anchor should exist throughout manipulation, change with its progress, covary with sensor response intervals, and provide context beyond a deterministic copy of one sensor.These requirements keep the reference available when individual sensors do not respond while avoiding identity-like alignment.Source: C. Missing Supervision in Contrastive… · p. 3
  • Images are available throughout, but their connection to local fingertip quantities is indirect, and their relation to interaction state depends on what the image reveals.Tactile and proximity observations do not themselves satisfy the throughout-manipulation availability requirement when used as each other’s anchors.Source: C. Missing Supervision in Contrastive… · p. 3

IV. METHOD

PROPRA aligns each fingertip-sensor history independently with a shared robot-side anchor comprising current proprioception and forthcoming actions. This temporally links past sensing to the current state and near-future motion without directly pairing sensor modalities.

IV. METHOD

  • Each sensor history is aligned independently with an anchor embedding of the current proprioceptive state and an H-step action chunk; the anchor is not transferred to policy inference.The anchor is available throughout the trajectory and relates past sensor input to current state and subsequent motion.Source: A. Proprioceptive and Proactive… · p. 3
  • PROPRA uses no direct sensor-to-sensor objective; cross-sensor correspondence emerges through each modality's alignment to the shared anchor.Each additional modality adds one anchor–modality term, so the framework scales linearly with the number of sensors.Source: A. Proprioceptive and Proactive… · pp. 3–4
  • Unlike image anchoring concentrated in the brief contact window, PROPRA aligns sensor histories to the temporally shared current state and forthcoming action chunk at every time step.The action chunk is defined and varies at every time t, and PROPRA uses no images.Source: Figure 2 · p. 4, A. Proprioceptive and Proactive… · p. 4

B. Encoders and Learning Objective

Modality-specific Transformer encoders map sensor histories to normalized embeddings, while an MLP embeds the shared anchor. Training independently aligns each sensor embedding with the anchor using a symmetric contrastive objective.

B. Encoders and Learning Objective

  • Each sensor has a separate four-layer prenorm Transformer encoder that projects its history to a 512-dimensional embedding; the anchor is encoded by an MLP.The sensor encoders are not shared because the modalities differ in physical quantity and response characteristics.Source: B. Encoders and Learning Objective · p. 4
  • L2-normalized anchor and sensor embeddings make their inner product correspond to cosine similarity.Sensor channels are normalized using their respective means and standard deviations.Source: B. Encoders and Learning Objective · p. 4
  • Within each batch, same-time anchor and sensor embeddings are positive pairs and other sensor embeddings are negatives in a symmetric InfoNCE objective.The tactile and proximity losses are computed independently against the same anchor and averaged; no term couples the two sensor modalities.Source: B. Encoders and Learning Objective · p. 4
  • The image–sensor and direct sensor–sensor baselines use weighted alignment terms, with cw = 0 for image anchoring alone and cw = 1 for adding the direct pair.Source: B. Encoders and Learning Objective · p. 4

C. Integration with Policy Learning

The pretrained sensor encoders are fine-tuned within a diffusion policy, while the anchor encoder is discarded for deployment. Experiments use four real-robot tasks with 50 demonstrations per task and compare training conditions that include a non-uniform sampling recipe for PROPRA.

C. Integration with Policy Learning

  • Only pretrained sensor encoders transfer to the diffusion policy and are fine-tuned end-to-end; the anchor and its encoder are not used at inference.At inference, the robot uses images, proprioceptive state, and sensor histories.Source: C. Integration with Policy Learning · p. 5

V. EXPERIMENTAL SETUP

  • Experiments use a UR5e with a fingertip module publishing three force and six proximity channels at 100 Hz, alongside two RealSense cameras.Robot state comprises a 6-D end-effector pose and 2-D gripper opening; actions comprise seven commanded joint positions and one gripper command.Source: §V · p. 5, Figure 4 · p. 5

A. Data Collection

  • Four real-robot tasks—PickCup, PickSponge, MovePen, and OpenLid—are each trained from 50 demonstrations with randomized initial object positions.Source: A. Data Collection · p. 5

B. Model Training

  • PROPRA pretraining oversamples low-proximity and tactile-amplitude threshold-crossing steps fourfold and draws 25% of each batch as hard negatives; image-anchored conditions use uniform sampling.Thus, the training-recipe comparison also differs in its sampling rule.Source: B. Model Training · p. 5
  • Each policy seed is evaluated over 20 trials; policy training uses an action horizon of 16 and eight inference denoising steps.The encoder conditions use AdamW for 1000 epochs with three seeds, and checkpoints are selected by validation loss.Source: B. Model Training · p. 5

C. Evaluation Metrics

Evaluation covers manipulation success and representation properties, including pre-contact time-to-contact decoding and held-out cross-sensor retrieval. The results section frames comparisons against vision-only, unpretrained, and image-anchored conditions and poses three research questions.

C. Evaluation Metrics

  • Pre-contact time-to-contact is decoded from frozen proximity embeddings with ridge regression and reported as R2; cross-sensor retrieval reports chance-normalized Recall@1.Source: C. Evaluation Metrics · p. 6, §VI · p. 6

VI. RESULTS AND DISCUSSION

  • The comparison conditions include vision-only policies, sensor concatenation without pretraining, and image-anchored pretraining with or without direct tactile–proximity alignment.Source: §VI · p. 6
  • The results section asks whether robot-side anchoring outperforms image anchoring, whether cross-sensor correspondence emerges without direct alignment, and whether pre-contact information is retained during policy execution.Source: §VI · p. 6
  • Fig. 5 evaluates retrieval of the same-time proximity embedding from tactile embeddings during contact on held-out episodes, with accuracy normalized by chance.The caption identifies the chance reference as 1 and reports means over three seeds with standard-deviation error bars.Source: Figure 5 · p. 6

A. RQ1: Does anchoring to past, present, and future body

Pretraining improves the usefulness of fingertip signals, and PROPRA has the highest average success rate among the compared pretrained conditions. Its average matches or exceeds image-anchored pretraining without camera images during pretraining.

A. RQ1: Does anchoring to past, present, and future body

  • With pretraining, average success exceeds 70% for all conditions, and MovePen and OpenLid surpass the vision-only baseline.Without pretraining, directly concatenating sensor histories raises average success from 30.4% to 40.4%, but those two tasks remain below vision-only performance.Source: signals help, compared with image… · p. 6
  • 75.0% average success rate for PROPRA, versus 71.7% for the image anchor and 72.1% with direct pairing.The reported task-level differences are within variation across seeds; MovePen is identical, and the image anchor is 3.4 points higher on OpenLid.Source: signals help, compared with image… · p. 6
  • Adding direct tactile–proximity alignment changes average success by only 0.4 points, with no clear benefit from explicitly coupling the representations.PROPRA aligns each sensor independently rather than directly pairing the two representations.Source: signals help, compared with image… · p. 6
  • Robot-side anchoring matches or exceeds image-anchored pretraining on average without requiring camera images during pretraining.Source: signals help, compared with image… · p. 6

B. RQ2: Do tactile and proximity representations corre-

The findings show that cross-sensor correspondence can emerge without direct pairing, while proximity representations retain pre-contact information in additional policy-execution recordings. Results vary by task and by whether the predictor is refit.

spond without direct pairing?

  • 6–12× chance retrieval on PickCup, MovePen, and OpenLid exceeds the untrained encoder’s 1–2× for every seed combination.On PickSponge, PROPRA reaches only 3.6× chance, comparable to the untrained encoder; the image anchor with direct pairing exceeds PROPRA on PickSponge and MovePen.Source: spond without direct pairing? · pp. 6–7

mation, also beyond the demonstration data?

  • On three of four tasks, PROPRA has the highest pre-contact time-to-contact R2; on PickSponge it reaches 0.440 versus 0.377 for the image anchor.The image anchor with direct pairing is highest on MovePen, while most of the gain over raw signals on MovePen and OpenLid is already present without pretraining; tactile R2 is ≤0.02.Source: Table III · p. 7, Figure 6 · p. 7, mation, also beyond the demonstration… · p. 7
  • On 100 policy-execution episodes per task, a refitted linear predictor retains 65–90% of PROPRA’s demonstration R2, with a 95% CI within ±4%.The recordings were used for neither pretraining nor policy learning, and PROPRA outperforms both image-anchor conditions in 20 of 24 seed-wise comparisons after initial-value alignment for zero-point drift.Source: mation, also beyond the demonstration… · p. 7
  • With the demonstration-fitted predictor and no refitting, PROPRA stays positive on every task and seed, while both image-anchor conditions are higher on MovePen and OpenLid.The raw signal has R2 values of −8.0 on PickCup and −25.5 on PickSponge; both image-anchor conditions turn negative on PickSponge.Source: mation, also beyond the demonstration… · p. 7, Figure 7 · p. 7

D. Limitations and future perspective

PROPRA’s advantage is not uniform: the paper reports an OpenLid case favoring image anchoring and frames the study around a specific grasp-oriented, two-sensor setting. The conclusion identifies broader tools and sensing modalities as future work.

D. Limitations and future perspective

  • On OpenLid, image anchoring reaches 76.7% versus 73.3% for PROPRA, where repeated lid contacts make informative sensor responses less sparse.The authors note that image-based alignment may already have enough correspondence when sensor information is rich across much of a trajectory.Source: D. Limitations and future perspective · p. 8
  • The study targets grasp-oriented manipulation with two sensors retrofitted to an existing gripper and imitation learning from limited demonstrations.It does not compare against large-scale tactile foundation models; scalability to broader sensing configurations is left for future work.Source: D. Limitations and future perspective · p. 8

VII. CONCLUSION

  • Future work will test other tools and modalities, including force, torque, contact sound, and vibration, and assess transfer across differing sensor characteristics and response patterns.Source: §VII · p. 8
Loading 2609.29822v1…