Source-linked AI summary
GelSlim 5.0: Unifying Vision and Touch with a Single Fingertip Camera
Xiangdong Wang, I-Hsuan Huang, Xiaohao Xu, Xili Yi, Nima Fazeli, Xiaonan Huang
TL;DR
Contact-site occlusion and viewpoint differences complicate relating external vision to tactile measurements of contact geometry. GelSlim 5.0 acquires both observations with one camera, and VisTacFusion combines them for pose and geometry prediction. In validation on known objects, paired observations yield lower depth and pose errors than either modality alone.
Problem
External cameras can be occluded at the contact site, while viewpoint differences complicate associating visual appearance with tactile measurements.
Source: §I · p. 1Method
GelSlim 5.0 switches UV illumination to acquire visual and tactile images with one fixed camera, and VisTacFusion uses separate pose and geometry prediction paths.
Source: §I · p. 1, §III · p. 2Results
Within-session validation on 12 known objects shows lower contact-depth and pose errors for paired observations than for either input alone.
Source: §I · p. 1Takeaways & Limitations
The results suggest that paired vision and touch can resolve each other’s ambiguities, with proximity vision providing global context and touch direct local geometry measurements.
Source: A. Discussion · p. 6The best real-to-simulation training ratio is specific to the current simulator, dataset, and training configuration.
Source: A. Discussion · p. 7
Abstract
from arXiv · showhide
Vision and touch provide complementary cues for object pose and local contact geometry, but external cameras are often occluded at the contact site. We present GelSlim 5.0, a fingertip sensor that acquires visual and tactile observations with one fixed camera and a continuous UV-responsive contact layer. By switching illumination, the camera sequentially acquires through-membrane visual images and contact-sensitive tactile images from a shared viewpoint, avoiding inter-camera extrinsic calibration. To combine these observations, we introduce VisTacFusion, a multimodal transformer with separate prediction paths for planar object pose and contact geometry. Asymmetric bottleneck fusion incorporates visual context while preserving tactile spatial features for depth and surface-normal estimation. Modality dropout enables inference with paired inputs or either modality alone. On real validation data from 12 known objects, paired inputs reduce depth MSE, angular error, and translation $L_1$ by 61.2%, 72.3%, and 61.3%, respectively, relative to tactile-only inference with the same model. Relative to visual-only inference, the corresponding reductions are 93.7%, 6.1%, and 8.6%.
I. INTRODUCTION
GelSlim 5.0 places visual and tactile sensing at the fingertip to address contact-site occlusion and relate object appearance to local contact geometry. VisTacFusion combines these observations for pose and geometry estimation, and paired inputs lower errors than either modality alone in within-session validation.
I. INTRODUCTION
- The continuous UV-responsive interface lets one fixed camera acquire visual and tactile images sequentially from a shared viewpoint, without inter-camera extrinsic calibration.The observations are acquired under approximately stationary contact.Source: §I · p. 1, Figure 1 · p. 1
- VisTacFusion uses separate paths for planar pose and contact geometry, with visual context conditioning both while tactile features preserve spatial detail.Modality dropout trains inference with paired inputs or either modality alone.Source: §I · p. 1
- Paired observations lower contact-depth and pose errors than either input alone in within-session validation on 12 known objects.The evaluation also covers modality complementarity, encoder and fusion choices, and training configuration.Source: §I · p. 1
II. RELATED WORK
- Prior visuotactile sensors use varied scene-access strategies; GelSlim 5.0 is characterized by a shared camera and continuous UV-responsive film.The paper positions GelSlim 5.0 among prior interfaces and retains the GelSlim 4.0 body design.Source: §II · pp. 1–2, §I · p. 1
UV LEDs
The supplied passages frame sensor comparisons around construction, acquisition, contact representation, and physical size, rather than shared accuracy benchmarks.
UV LEDs
- The size guide distinguishes sensing-head envelope from contact-interface dimensions and excludes components such as back covers, cables, and robot adapters from the stated envelope.It also notes that reconstruction is evaluated on known-object, within-session validation samples.Source: UV LEDs · p. 2
- Table I summarizes representative sensor designs and physical sizes, not a common accuracy benchmark.Its rows describe cited implementations.Source: Table I · p. 2
- GelSlim 5.0 uses contact-sensitive image appearance rather than marker displacement and does not spatially partition the film into clear and coated windows.The table separates head dimensions from contact-interface measurements.Source: UV LEDs · p. 2
III. GELSLIM 5.0: COMMON-VIEW SENSING
GelSlim 5.0 uses one camera and UV-controlled illumination to acquire visual and tactile observations through a continuous contact membrane. The fixed camera preserves shared image coordinates, although refraction, deformation, and motion can affect pixelwise alignment.
III. GELSLIM 5.0: COMMON-VIEW SENSING
- The sensor combines a compliant UV-responsive membrane, an illumination ring, and a 3D-printed body housing one camera and a custom driver PCB.The prototype retains the GelSlim 4.0 gripper-body CAD and uses a two-layer silicone membrane.Source: §III · p. 2
- UV off records a through-membrane visual image, while UV on records a contact-sensitive tactile image.The layer contacts the object directly without a separate protective overlayer.Source: §III · p. 3
- Sequential captures under approximately stationary contact share viewpoint and image coordinates, avoiding inter-camera extrinsic calibration, but refraction, membrane deformation, and residual motion can affect pixelwise alignment.Camera intrinsics and object-pose calibration remain relevant to estimation.Source: §III · p. 3
IV. LEARNING OBJECT POSE AND CONTACT GEOMETRY
VisTacFusion combines visual context and tactile signatures through separate pose and geometry paths. Cross-attention associates the modalities without requiring pixelwise correspondence.
IV. LEARNING OBJECT POSE AND CONTACT GEOMETRY
- The pose path aggregates features into a pose vector, while the geometry path preserves spatial detail for depth and surface-normal prediction.The paths combine visual context with tactile signatures.Source: §IV · p. 3
- Cross-attention associates visual and tactile modalities without requiring pixelwise correspondence.This association is used in VisTacFusion's separate pose and geometry prediction paths.Source: §IV · p. 3
A. Problem Formulation
The estimator takes a known object and visual input, tactile input, or both, and predicts sensor-relative planar pose and contact geometry.
A. Problem Formulation
- For a known object, the estimator accepts visual input, tactile input, or both, depending on the selected mode.The modes are Vis, Tac, and Both, respectively.Source: A. Problem Formulation · p. 3
- Planar pose is encoded by rotation components (cos θ, sin θ) and translations (tx, ty), while depth and surface normals describe contact geometry.The geometry is represented on the tactile image grid.Source: A. Problem Formulation · p. 3
B. Modality-Specific Features
Frozen tactile and visual encoders produce features that are projected into a shared-width representation for downstream prediction.
B. Modality-Specific Features
- Frozen SITR-B/16 encodes touch, while frozen MAE-L/16 encodes vision.At 224 × 224 resolution, each produces 196 patch tokens and one CLS token, with widths 768 and 1024, respectively.Source: B. Modality-Specific Features · p. 3
- SITR calibration tokens are discarded before downstream prediction.The tactile encoder uses a fixed set of 18 calibration references.Source: B. Modality-Specific Features · p. 3
- Separate projections map both streams to width 768, and modality embeddings distinguish them.Tactile patches receive spatial embeddings and initialize 196 spatial queries; projected visual tokens form a 197-token memory.Source: B. Modality-Specific Features · p. 3
C. Object Pose Estimation
VisTacFusion uses separate pose and geometry paths: tactile-initialized queries read visual context for pose, while spatial tactile features support contact geometry estimation.
C. Object Pose Estimation
- The pose head combines the final pose token with the mean of the spatial-token states before predicting rotation and translation.A layer-normalized three-layer MLP maps this combined vector to two rotation components and two translations.Source: C. Object Pose Estimation · p. 3
- The pose path uses tactile-initialized queries and a 16-token visual bottleneck to condition the prediction of planar pose.Four layers update the bottleneck from visual memory, then apply query-to-bottleneck cross-attention and query self-attention.Source: C. Object Pose Estimation · p. 3, Figure 3 · p. 4
D. Contact Geometry Estimation
- The geometry path retains tactile feature maps from encoder blocks 3, 6, 9, and 12 instead of relying on the pose-query trunk.Each map receives visual context from the final bottleneck through a separately gated cross-attention module.Source: D. Contact Geometry Estimation · p. 4
- Learned scalar gates make visual conditioning a residual update to the tactile features, and both pose and geometry losses supervise the shared visual summary.The geometry decoder uses the conditioned spatial features directly, without depending on pose predictions.Source: D. Contact Geometry Estimation · p. 4
- A DPT-style decoder reassembles the four conditioned maps into multiscale features and fuses them into a shared 112×112 feature map.Separate convolutional heads predict depth and surface normals at 224 × 224.Source: D. Contact Geometry Estimation · p. 4
E. Input Availability and Joint Training
Modality dropout trains the model to operate with paired inputs or either modality alone, while all modes receive pose and geometry supervision.
E. Input Availability and Joint Training
- Modality dropout trains the same model in Both, Vis, and Tac modes.In Vis, learned mask queries replace tactile queries; in Tac, fusion cross-attention and visual conditioning in the geometry path are skipped.Source: E. Input Availability and Joint Training · p. 4
- All three modes receive pose and geometry supervision.For inputs including vision, geometry-path visual conditioning is independently enabled during training with probability pinject and always enabled at inference.Source: E. Input Availability and Joint Training · p. 4
- Depth and normal losses use full-image MSE, rotation uses 1 − cos(θ̂ − θ), and translation uses coordinate-averaged L1 error in normalized units.The training objective uses uncertainty weighting, with each learned scalar s_k weighting one task.Source: E. Input Availability and Joint Training · p. 4
V. EXPERIMENTAL EVALUATION
The evaluation examines modality complementarity, encoder and fusion choices, and training-data composition under a shared real-data validation protocol.
A. Sensor Observations and Evaluation Protocol
- More than 45,000 normal indentations at 20 N produced no observed cracking, delamination, or loss of the imaging-state change.Source: A. Sensor Observations and Evaluation… · pp. 4–5
- Fig. 4 shows sequential UV-on tactile and UV-off proximity images from the same fixed camera under quasi-static contact.Source: Figure 4 · p. 5
- The endurance result is limited to the tested normal-loading protocol; sliding wear, contamination, and optical drift were not quantified.Source: A. Sensor Observations and Evaluation… · p. 5
- The robot collects approximately 400 contacts for each of 12 objects, recording both image states and end-effector pose at every contact.Pose and geometry generate depth references; real normal maps are computed by finite differences, and simulated tactile images are generated from rendered depth maps.Source: A. Sensor Observations and Evaluation… · p. 5
- Every tenth sample within each real session is held out for validation and checkpoint selection, with objects and sessions shared between training and validation.Reported scores therefore measure within-session generalization on known objects.Source: A. Sensor Observations and Evaluation… · p. 5
- Depth and normal MSE average over all pixels, while rotation and translation use distinct error conventions and task-selected checkpoints.Fusion checkpoints are evaluated in all three input modes without reselection; main fusion runs use real–simulation co-training at a 1:1.27 ratio.Source: A. Sensor Observations and Evaluation… · p. 5
B. Complementary Contributions of Vision and Touch
Paired vision and touch yield the strongest VisTacFusion results, while representation, fusion, and training choices trade off geometry and pose accuracy.
B. Complementary Contributions of Vision and Touch
- Paired VisTacFusion reduces depth MSE to 0.0066 mm2 (61.2% vs Tac; 93.7% vs Vis) and angular error by 72.3% vs Tac, with lowest error on all five metrics within the model.Source: B. Complementary Contributions of… · p. 6
- Among paired-input models, VisTacFusion has the lowest error across all five metrics and reduces depth MSE by 38.3% and normal MSE by 26.7% versus Sparsh-X+MAE.Translation L1 falls from 0.0055 to 0.0053, while angular errors are similar: 0.652° versus 0.655°.Source: B. Complementary Contributions of… · p. 6
- Dedicated single-input models remain competitive: SITR leads Tac depth and normal accuracy, Sparsh-X leads angular accuracy, and DAv2 leads translation accuracy.With Vis, T3 has the lowest depth, normal, and angular errors, while VisTacFusion has the lowest translation error.Source: B. Complementary Contributions of… · p. 6
C. Representation and Training Design
- MAE gives the lowest depth and angular errors for every tactile encoder; SITR+MAE reaches depth MSE 0.0067, while asymmetric fusion lowers angular error from 0.747° to 0.652° and translation L1 from 0.0054 to 0.0053 versus symmetric MBT.Symmetric MBT instead achieves lower depth and normal MSEs and the lowest errors across all five Tac metrics; Table III compares the fusion configurations with identical SITR+MAE encoders and heads.Source: C. Representation and Training Design · p. 6, Table III · p. 6
- At a 1:1.27 real:sim ratio, depth MSE is 0.0066, angular error 0.652°, and translation L1 0.0053, versus 0.0079, 0.720°, and 0.0056 for real-only training.The 1:1.27 configuration has the lowest angular and translation errors among plotted configurations; real-only and mixed-data runs use 100 and 150 epochs, respectively.Source: C. Representation and Training Design · p. 6
- More frequent visual conditioning decreases depth MSE, while paired-only training trades lower depth MSE (0.0058) for higher angular error (0.675°); tactile input recovers V-shape and corner boundaries.Paired input also reduces the irregular rod depth profile seen with Tac alone.Source: C. Representation and Training Design · p. 6
VI. DISCUSSION AND FUTURE WORK
GelSlim 5.0 and VisTacFusion frame shared-view visual–tactile sensing as a way to combine complementary cues for pose and contact geometry. The discussion reports a real–simulation trade-off and identifies hardware and perception questions for future work.
A. Discussion
- Paired observations suggest cross-modal synergy: proximity vision provides a global prior, while touch measures local contact geometry, motivating task-specific fusion and state-dependent mode selection.The paper presents this as a motivation for dexterous manipulation, not as a demonstrated general result beyond its evaluation.Source: A. Discussion · p. 6
- 1:1.27 real-to-simulation is the selected ratio, providing the best overall pose-estimation trade-off while retaining competitive geometry accuracy; adding simulation improves real-world validation over real-only training.The comparison is specific to the current simulator, dataset, and training configuration.Source: A. Discussion · pp. 6–7, Figure 6 · p. 7
- Illumination, blur, and background differences can introduce simulation-specific cues that may bias representations and cause negative transfer as the simulated-data fraction grows.This helps explain why additional simulation does not produce monotonic gains and why an intermediate mixing ratio is optimal.Source: A. Discussion · p. 7
B. Future Work
- The current prototype retains the GelSlim 4.0 body; future work will co-design the camera–lens module, illumination, electronics, and contact stack for a smaller, more power-efficient fingertip.The design target is to preserve field of view and contact sensitivity while quantifying package size, energy per paired observation, and contact-surface temperature under matched acquisition conditions.Source: B. Future Work · p. 7
- Future perception evaluation will test cross-session generalization and validate local contact geometry with contact-region metrics and direct membrane-shape measurements.Source: B. Future Work · p. 7