BIFTA: Brain-Inspired Few-Shot Tactile Adaptation for Unknown Sensors

Boheng Liu; Ziyu Li; Xia Wu

Source-linked AI summary

BIFTA: Brain-Inspired Few-Shot Tactile Adaptation for Unknown Sensors

Boheng Liu, Ziyu Li, Xia Wu

arXiv:2609.08673v1Sep 8, 2026cs.ROcs.AI

TL;DR

Unknown tactile sensors can cause abrupt performance collapse because their optical, mechanical, and imaging properties differ from those seen during pretraining. BIFTA adapts a frozen tactile encoder from a small labeled support set using statistical memory, support-conditioned spectral geometry, and reliability-gated recurrent inference, substantially improving transfer across datasets, backbones, and tactile tasks. On SITR, it raises Sparsh accuracy from 6.86% to 87.09% with 10% labeled target data, exceeding the strongest comparison by 47.22 percentage points.

Problem
Differences among tactile sensors can make models trained on known hardware perform poorly on unknown sensors, motivating rapid adaptation from few labeled contacts.
Method
BIFTA adapts a frozen encoder through dual-view statistical memory, support-conditioned spectral geometry, and reliability-gated recurrent inference without encoder updates.
Results
87.09% Sparsh accuracy is reached on SITR with 10% labeled target data, up from 6.86% for the frozen source classifier and 47.22 points above the strongest comparison, with gains extending across datasets, backbones, and tactile tasks.
Takeaways & Limitations
BIFTA supports data-efficient adaptation of tactile models to heterogeneous unknown sensors while preserving frozen pretrained encoders.
Takeaways & Limitations
Future work is needed to extend BIFTA to online manipulation, material understanding, distributed tactile skins, and more computationally efficient real-time inference.

Abstract

from arXiv · show

Advances in tactile sensing have made contact-rich perception possible, accelerating progress in robotic manipulation, material understanding, and embodied interaction. However, because optical design, elastomer mechanics, and imaging geometry differ substantially across tactile sensors, models trained on known sensor types can suffer an abrupt performance collapse on unknown sensors. To address this problem, we propose the Brain-Inspired Few-Shot Tactile Adaptation (BIFTA) framework; it draws on the brain's rapid sensory adaptation mechanism to adapt a frozen encoder to an unknown tactile sensor from a small labeled support set. BIFTA preserves pretrained representations through dual-view statistical memory, constructs support-conditioned spectral graphs to repair sensor-dependent feature neighborhoods, and applies uncertainty-gated recurrent propagation to strengthen reliable cross-query evidence. Extensive benchmarks across three tactile datasets show that BIFTA substantially improves adaptation to unknown sensors: with only 10\% labeled target data on SITR, it raises mean Sparsh accuracy from 6.86\% for the frozen source classifier to 87.09\%, exceeding the strongest implemented prior comparison by 47.22 percentage points, and these gains generalize across datasets, pretrained backbones, and tactile tasks. These results validate BIFTA for data-efficient adaptation to unknown tactile sensors and offer a promising route toward tactile models that transfer across heterogeneous hardware.

1 Introduction

Unknown tactile sensors can cause severe transfer failures because sensor-specific observations disrupt pretrained feature neighborhoods. BIFTA adapts frozen encoders from few labeled contacts and substantially restores unknown-sensor performance across datasets and backbones.

Sensor differences in optics, marker layout, field of view, and elastomer mechanics cause tactile models trained on known hardware to generalize poorly to unseen sensors.
Few-shot adaptation can use optimization-based classifiers or relation-based prototypes, caches, nearest neighbors, and query graphs, but these approaches respectively risk overfitting or depend on reliable feature metrics.
BIFTA combines dual-view statistical memory, support-conditioned spectral geometry, and reliability-gated recurrent inference without gradient-based encoder updates.
BIFTA’s gains extend across three tactile datasets, two frozen backbones, and classification and retrieval tasks.

2 Related Work

Tactile foundation models have progressed from sensor-specific learning toward reusable multimodal and self-supervised representations. Few-shot adaptation methods then address new sensors through optimization-based updates or relation-based inference over frozen features.

Tactile foundation models evolved from task-specific supervision toward reusable multimodal and self-supervised representations, including TVL and UniTouch.
Optimization-based few-shot methods fit lightweight classifiers, adapters, or test-time states to adjust decision boundaries, but iterative updates can overfit scarce support examples.
Relation-based methods retain frozen features and classify through support-derived relations rather than updating the representation.

3 Method

BIFTA adapts a frozen encoder to an unknown tactile sensor by combining support-derived class evidence, sensor-conditioned query geometry, and reliability-gated recurrent inference. The method preserves stable evidence while recalibrating sensor-dependent relationships for classification or retrieval.

Problem formulation: BIFTA assumes a labeled support set and unlabeled query batch, using a fixed pretrained encoder with two feature readouts and no encoder updates.Every class appears in the support set, and queries are predicted jointly without source-data replay or query annotations.
Rapid support memory: Rapid support memory standardizes each readout and fits a shrinkage-LDA memory to produce readout-specific class probabilities fused into the probability anchor P0.The anchor carries direct support-supervised class evidence into final inference.
Support-conditioned spectral graph: Support-conditioned spectral geometry downweights high-variance feature directions before constructing nearest-neighbor graphs from transformed query features.The transform changes query-connection geometry while leaving the probability anchor unchanged.
Support-conditioned spectral graph: The method retains relations supported by both readouts, producing a row-stochastic affinity matrix G that encodes support-adapted query geometry.Shared cross-readout edges are retained after graph symmetrization, with a fallback for queries lacking shared edges.
Reliability-gated inference: A reliability gate combines anchor ambiguity and cross-readout disagreement to control each query’s access to graph evidence.Higher recurrence weights permit more relational evidence, while lower values keep predictions closer to the support-derived anchor.
Anchored recurrent inference and output: Anchored recurrent inference repeatedly restores support evidence and propagates neighborhood evidence through G in proportion to query reliability.After the iterations, classification selects the highest-probability class, while retrieval ranks enrolled identities by final class probabilities.

4 Experiments

BIFTA is evaluated for few-shot adaptation to unknown tactile sensors across three datasets, two frozen backbones, classification, and identity ranking. It consistently improves performance across label budgets and sensor shifts, with ablations identifying spectral normalization and consensus recurrence as important components.

Experimental setup: SITR uses DIGIT as source and six targets, with 1%, 5%, and 10% support budgets; TacVerse uses five targets, while TacQuad evaluates two targets for identity ranking.TVL and Sparsh remain frozen throughout the experiments.
Cross-sensor classification: BIFTA improves SITR accuracy at every backbone–budget setting, reaching 85.71% with TVL at 10% support and 87.09% with Sparsh.With Sparsh, gains over LaplacianShot are 29.34, 43.87, and 47.22 percentage points across the three budgets.
Cross-sensor classification: 83.58% Sparsh accuracy on TacVerse at 10% support exceeds the strongest comparison by 27.60 percentage points, while gains remain above 13 points at 1% support.With TVL at 10%, BIFTA reaches 79.13% and exceeds the strongest comparison by 22.46 percentage points.
Cross-sensor classification: 55.6 and 41.6 percentage points are BIFTA’s largest sensor-wise gains on GelSight Hex in SITR and MagicTac in TacVerse, respectively.Figure 3 reports gains over the strongest comparison for each unknown sensor and label budget.
Closed-set identity ranking: BIFTA improves TacQuad identity-ranking MRR over the strongest comparison by 8.45 points with TVL on average and by 7.55, 12.89, and 14.86 points with Sparsh.The Sparsh values correspond to 10%, 20%, and 30% support budgets.
Ablation study: Consensus recurrence adds 2.65–4.71 accuracy points, while removing spectral normalization reduces accuracy by 5.36 points on average and up to 7.06 points.The ablation also reports declines for view-1-only and view-2-only configurations relative to full BIFTA.

5 Conclusion

BIFTA adapts frozen tactile encoders to unknown sensors through statistical memory, support-conditioned spectral geometry, and reliability-gated recurrent inference. Across three datasets, two backbones, and multiple tasks, it delivers consistent cross-sensor gains while leaving online and real-time deployment for future work.

Conclusion: BIFTA transfers frozen tactile encoders to unknown sensors through dual-view statistical memory, support-conditioned spectral geometry, and reliability-gated recurrent inference.The pipeline extracts two readouts, builds readout-specific memories and graphs, fuses them, and iteratively propagates reliability-weighted evidence.
Conclusion: 10% labeled target data raises Sparsh accuracy on SITR from 6.86% for the frozen source classifier to 87.09%, exceeding the strongest comparison by 47.22 percentage points.The result is reported as a representative cross-sensor outcome in the conclusion.
Conclusion: Experiments across three tactile datasets, two pretrained backbones, and classification and ranking tasks demonstrate consistent cross-sensor gains.The conclusion also reports complementary contributions from memory, geometry correction, and recurrent evidence integration in ablations.
Future work: Future work targets online robotic manipulation, visual–tactile material understanding, distributed tactile-skin perception, and more efficient real-time inference.The proposed extensions include sparse neighborhoods and streaming updates for large query streams.

A.2 Dataset splits and sample accounting

The experiments use defined source–target splits across SITR, TacVerse Shape, and TacQuad, with support budgets and query sets specified per task. Images are standardized differently for TVL, Sparsh, and TacQuad, while both encoders remain frozen during feature extraction.

SITR: SITR uses DIGIT as source and six target sensors, with 1%, 5%, and 10% budgets corresponding to 8, 40, and 80 support images per class.The 16-class subset provides 3,200 evaluation queries per target.
TacVerse Shape: TacVerse Shape contains seven sensors and nine shape classes, with 27, 135, and 270 support images per target under the three budgets.The ordered split provides 2,700 training, 900 validation, and 900 test images per sensor.
TacQuad: TacQuad retains 56 trial identities with complete 20-frame sequences across three RGB sensors, using 112, 224, or 336 target-support frames and 224 query frames.The force-field sensor Tac3D is excluded from the RGB encoder experiment.
Preprocessing: SITR and TacVerse images are resized to 224 × 224, with TVL-specific normalization, no additional Sparsh channel normalization, and task-specific TacQuad resizing.Sparsh concatenates each image with itself to obtain six channels.
Encoders: TVL and Sparsh extract frozen evaluation-mode features without gradient updates, using global/projected features for TVL and mean-pooled final-block patch tokens for Sparsh.The archived frozen parameter counts are 21,961,344 for TVL and 21,894,144 for Sparsh.

A.4 Baseline Implementations

The baseline suite combines frozen-feature few-shot methods with lightweight-head adaptation settings. SITR and TacVerse share one configuration, whereas TacQuad uses a task-specific configuration.

Baseline implementations: Lightweight-head methods optimize compact feature modules and classifiers while keeping the tactile encoder frozen.Training uses AdamW, class-balanced target sampling, target cross-entropy, source-prototype preservation, and within-class compactness.
Baseline implementations: Frozen-feature methods transfer normalized source or support features through class prototypes, support caches, or query graphs without updating the backbone.These methods rely on fixed representations rather than backbone optimization.
Configurations: SITR and TacVerse share one hyperparameter configuration, while TacQuad uses a separate configuration because it evaluates identity ranking.The fixed values for the two settings are listed in Table 6.

B.1 Per-Sensor Accuracy Results

BIFTA achieves the strongest overall performance across sensors and backbones, with gains generally increasing at larger support budgets. Its advantage also appears in Macro-F1 and class-wise prediction structure, indicating balanced recognition under sensor shifts.

Per-sensor accuracy: BIFTA achieves the strongest overall accuracy across sensors and backbones, with gains generally increasing as the support budget grows.The reported pattern covers the per-sensor accuracy tables for SITR and TacVerse Shape.
Macro-F1: BIFTA remains the strongest method on Macro-F1 across SITR and TacVerse Shape, with a more balanced advantage across classes than accuracy alone.The reported Macro-F1 pattern indicates greater robustness to class-dependent sensor shifts.
Class-wise prediction structure: On TacVerse queries with 10% support, BIFTA produces more diagonal concentration and stronger cross-class balance than SimpleShot.Figure 4 compares row-normalized confusion matrices over 13,500 predictions from five target sensors, three seeds, and 900 queries per target.
Identity ranking: The evaluation also includes TacQuad identity-ranking results with TVL and Sparsh, reporting MRR and R@1 across three random seeds.Tables 15 and 16 organize the identity-ranking evaluation by backbone.

C Additional identity-ranking results

BIFTA consistently improves cross-sensor identity ranking on TacQuad across pretrained backbones and support budgets.

BIFTA achieves higher R@1 and MRR than competing methods across both backbones and all support budgets.R@1 measures first-rank identity accuracy and is equivalent to closed-set identity classification accuracy.

D Complete component evidence

Component evidence indicates that spectral normalization and uncertainty gating are central to BIFTA’s class-balanced discrimination and evidence propagation.

Spectral normalization consistently strengthens class-balanced discrimination in the SITR ablation study.Removing spectral normalization produces the largest decline among full-model variants.
The uncertainty gate regulates anchor and graph evidence to reduce unreliable propagation.Disagreement matches the complete gate in several settings, whereas entropy contributes less consistently.

E.1 Training-only inner validation on SITR

Training-only inner validation separates configuration selection from evaluation and identifies which parameters most affect graph structure and recurrent evidence propagation.

E.1 Training-only inner validation on SITR: Configuration selection on one training-only sensor improves fairness by separating tuning from six-sensor evaluation.The validation protocol uses a 5% support budget on Mini_3.
E.1 Training-only inner validation on SITR: The disagreement weight and iteration count remain comparatively stable around their selected values on Mini_3.This stability contrasts with the greater sensitivity of graph temperature and recurrence range.
E.1 Training-only inner validation on SITR: Graph temperature and recurrence range are more sensitive because they control edge concentration and recurrent evidence propagation strength.These parameters directly affect how evidence is concentrated and propagated through the graph.
E.1 Training-only inner validation on SITR: Spectral normalization improves TacQuad MRR over spectral exponent zero across all six backbone–budget settings at exponent 0.25.Neighborhood size and disagreement weight are comparatively stable, while spectral exponent and recurrence length have larger effects under the lowest support budget.
E.1 Training-only inner validation on SITR: Tables 25–29 report TacQuad MRR sensitivity to neighborhood size, spectral exponent, recurrence range, disagreement weight, and iteration count.The reported metric is mean ± sample standard deviation over three random seeds.
E.1 Training-only inner validation on SITR: Table 30 defines adaptation time and end-to-end query throughput for SITR, TacVerse, and TacQuad under their stated support budgets.SITR and TacVerse use 10% support, while TacQuad uses 30% support.