Source-linked AI summary
Toward Trustworthy Robot-Assisted Sliding Palpation for Shallow Vessel Localisation with a Calibrated Digital Twin
Piotr Blaszyk, Wen Fan, Kaizhong Deng, Daniel Elson, Dandan Zhang
TL;DR
Shallow-vessel localisation matters for safe robot-assisted procedures, but collecting diverse tactile data is costly and can degrade soft sensors. This paper calibrates a digital twin to generate labelled palpation data for an ST-GNN and interpretable vessel maps, achieving 0.50 mm marker-alignment MAE and 1.2 mm average map distance on Sim→Silicone, while Sim→Meat exposes transfer limits.
Problem
Shallow-vessel localisation supports safe robot-assisted manipulation and clinical procedures, but physical tactile-data collection is slow, labour-intensive, and can degrade soft sensors.
Method
A calibrated physics-based digital twin generates labelled, domain-randomised sliding-palpation sequences for an ST-GNN that classifies marker nodes and reprojects predictions into top-view vessel maps.
Results
The twin achieves 0.50 mm marker-alignment MAE across four validation trajectories; Sim→Silicone places predicted vessel pixels 1.2 mm from annotated pixels, while Sim→Meat reaches 5.5 mm.
Takeaways & Limitations
Calibrated simulation supports interpretable cross-domain tactile localisation, while the meat phantom identifies current limits of simulation transfer.
Takeaways & Limitations
Evaluation remains constrained by limited trajectories and simplified phantoms, approximately 2 mm marker spacing, calibration scalability, optimistic map selection, and incomplete component isolation.
Abstract
from arXiv · showhide
Reliable localisation of shallow subsurface vessels is important for safe robot-assisted venous access and vessel-aware manipulation, but collecting diverse tactile data on physical hardware is costly, time-consuming, and can degrade soft vision-based tactile sensors. We present a robot-assisted sliding-palpation framework in which a calibrated digital twin generates labelled tactile sequences, reducing reliance on real-world data. The twin models sensor-vessel contact, is calibrated against real palpation trajectories using Bayesian-optimisation-based domain adaptation, and is randomised over sliding direction and contact conditions. A spatio-temporal graph neural network trained on simulated marker trajectories performs per-node vessel classification and produces a human-verifiable top-view localisation map through 2D-to-3D-to-2D geometric projection. We evaluate three datasets: Sim, Silicone, and Meat, the latter a raw-meat phantom with vessel models at nominal depths of 0 to 30 mm, using four train-to-test configurations: Sim to Sim, Sim to Silicone, Sim to Meat, and Meat to Silicone. The calibrated twin achieves a simulated-to-real marker-alignment mean absolute error of 0.50 mm at deepest contact across four canonical interactions. After reprojection onto a 1 mm top-view grid, predicted vessel pixels lie on average 1.05 to 5.49 mm from the nearest true vessel pixel across the four models, with 1.05 to 1.31 mm for all except Sim to Meat. The larger error for Sim to Meat reflects the greater domain shift and current limit of simulation transfer. These results demonstrate progress toward trustworthy tactile palpation through calibrated simulation, interpretable localisation, and transparent cross-domain evaluation. Code, model weights, and data are publicly available on GitHub and Zenodo.
1 Introduction
Shallow-vessel localisation supports safer robotic and clinical procedures, but conventional imaging is costly and tactile learning requires diverse data. This work combines a vision-based tactile sensor, calibrated digital twin, domain-randomised simulation, and interpretable top-view mapping for sliding palpation.
- Motivation: Shallow-vessel detection supports vessel-aware manipulation, venous access, blood sampling, and interventional procedures.It can help prevent accidental vessel injury and inform clinical actions such as catheterisation and tumour-treatment electrode placement.
- Motivation: MRI, CT, ultrasound, NIR, and OCE provide vessel information but require specialised imaging equipment.The passage identifies ultrasound as the clinical gold standard for intra-operative detection of shallow and deep vessels.
- Motivation: NIR and ultrasound probes cost approximately $1,800 and $4,800, while ophthalmic OCT probes were reported at $40,000–$150,000 in 2021.Maintenance, infrastructure, and trained operators further hinder deployment in resource-constrained settings.
- Motivation: Vision-based tactile sensors offer compact, comparatively low-cost, high-resolution tactile perception from contact-induced deformation.Commercial examples cited include the GelSight Mini at approximately $500 and DIGIT at approximately $350.
- Clinical workflow: A brief tactile pre-scan can produce a human-verifiable vessel map before needle insertion or instrument interaction.The map is intended to support automated venous access and vessel-aware manipulation while retaining clinician oversight.
- Approach: Digital twins generate diverse, automatically labelled tactile interactions without physical wear, while domain adaptation and randomisation address sim-to-real differences.The paper motivates these strategies because physical data collection is slow, labour-intensive, and can degrade soft tactile sensors.
- Contributions: The study combines sliding palpation, a vision-based tactile sensor, and a digital twin for shallow-vessel detection.It formulates localisation from continuous robot-executed sliding and evaluates transfer to silicone and raw-meat phantoms, including nominal vessel depths of 0–30 mm.
- Contributions: Tactile predictions are transformed through robot kinematics into a top-view vessel map evaluated per pixel across three datasets.The map is designed as spatial input for subsequent robot planning.
2 Related work
Prior tactile-palpation studies use deformation maps or marker graphs, while digital twins provide diverse, labelled contact data without physical hardware costs and wear.
- Related work covers VBTS tumour palpation and vessel detection using tactile deformation, tapping, sliding, and alternative sensing modalities.The reviewed studies include modular deformation-map classifiers, end-to-end marker-graph models, and vessel-line or distance-based evaluation.
- Digital twins simulate sensing, dynamics, and interaction to generate large, diverse tactile datasets with automatic labels and reduced hardware wear.Physics-based simulation paired with data-driven models can approximate tactile deformation and sensor noise across interaction conditions.
3 Methods
The method combines a ViTacTip-based tactile sensor, graph representation of marker motion, a calibrated physics digital twin, and a temporal ST-GNN for vessel classification.
- Sensor and graph representation: The ViTacTip uses a transparent elastomeric skin and internal camera to track marker strain and shear during sliding palpation.Its visual channel also supports identifying vessel locations for ground-truth construction.
- Sensor and graph representation: Marker displacements form a graph whose nodes are tactile markers and whose edges represent spatial neighbourhoods, enabling GNN-based tactile perception.GNNs explicitly model inter-marker dependencies in marker-based tactile tasks.
- ST-GNN training and evaluation: A temporal-window ablation selects five frames because one frame is insufficient and seven frames is less stable across seeds.The vessel signature is represented as temporal change in marker motion rather than a static pattern.
- Digital twin and calibration: The calibrated twin is validated on press, slide, twist-x, and twist-z trajectories using marker-position error at deepest contact.The validation covers four real trajectories after calibration.
- ST-GNN training and evaluation: 500 simulated trajectories train an ST-GNN for binary per-node vessel classification using five-frame clips and central-frame testing.The dataset contains 250 vessel-present and 250 vessel-absent trajectories, with deep supervision across markers and frames.
- Digital twin and calibration: The digital twin simulates sensor–vessel contact with a deformable FEM sensor, rigid vessel and phantom bodies, and camera projection into image space.The phantom is visualised but does not participate in the enabled sensor–vessel contact pair.
- Digital twin and calibration: Bayesian optimisation adapts Young’s modulus and normal contact stiffness using real sliding data while keeping the remaining contact parameters fixed.The objective combines marker-position error on vessel-absent contact with sufficient deformation during vessel contact.
4 Experiments
Experiments compare simulated and physical phantom domains, including a layered raw-meat phantom, graph-based processing, alignment validation, and annotation procedures.
- Datasets and evaluation: The evaluation uses Sim, Silicone, and Meat datasets across Sim→Sim, Sim→Silicone, Sim→Meat, and Meat→Silicone configurations.Sim is generated in the digital twin; Silicone and Meat are physical phantoms with shallow or varying-depth vessels.
- Model architecture: The ST-GNN processes marker and global nodes with spatial, temporal, and global edge types through three GINE layers and an MLP classification head.Skip connections fuse the input with all three layer representations, giving a three-hop receptive field.
- Calibration evaluation: Alignment is assessed for simulated and real marker positions after adaptation across press, z-axis twist, x-axis twist, and slide.Each interaction is shown at deepest contact.
- Phantom designs: The raw-meat phantom provides a mechanically complex domain with adjustable inclusion depth through a ten-layer stacked design.The study contrasts this substrate with silicone and reports qualitatively lower sliding resistance in meat.
- Phantom designs: Meat trials place metal or silicone straws beneath 0 to 30 mm nominal depths, including single-vessel, multiple-vessel, and control conditions.The phantom uses ten approximately 5 mm pork medallions, with ground truth derived from robot kinematics and sensor video.
- Annotations: Silicone frames use manual vessel annotations and centrelines, whereas Meat labels mark vessel presence or absence from robot kinematics.The Meat labels have minor misalignment with the actual metal straw.
5 Results and Analysis
Calibration aligned simulated and real marker trajectories, while geometric reprojection produced interpretable vessel maps across simulation and physical phantoms. Performance was strongest in-domain and on Silicone, whereas Sim→Meat exposed the larger domain shift.
- Calibration: 0.50 mm mean marker-alignment MAE was achieved across four validation trajectories after Bayesian-optimisation calibration of the digital twin.The adopted configuration used E = 881 kPa and kn = 9.5 × 10^4 N/m.
- Semantic segmentation: Sim→Silicone matched or slightly exceeded Meat→Silicone, with AP 0.32 ± 0.00 versus 0.30 ± 0.04 over seeds.Foreground IoU was 0.24 versus 0.16 for the best instances, while Meat→Silicone showed greater seed spread.
- Temporal context: A five-frame temporal window was selected because a single frame was insufficient and longer context became less stable across seeds.Performance increased through five frames, while seven frames was slightly worse and markedly less stable.
- 2D→3D→2D reprojection: The reprojection pipeline lifted per-node predictions through robot pose and camera geometry onto a 1 mm/pixel top-view grid.Pixel scores used the highest probability among markers landing on each grid cell, with ground truth processed through the same route.
- Operating point: A model-specific threshold maximised recall while maintaining map precision at or above 0.9, except Sim→Meat, whose peak precision was 0.87.Sim→Meat was therefore shown at its F1-optimal threshold, while reported statistics used exact-pixel agreement.
- Localisation maps: Predicted vessel pixels averaged about 1.2–1.3 mm from true pixels on Silicone but 5.5 mm on Meat, where transfer exposed substantial localisation error.In-domain predictions traced the vessel closely, whereas Meat predictions often fell away from the straws despite recovering them.
6 Discussion
The discussion positions the method as comparable to prior vessel-detection systems while emphasizing that the evaluation measures reprojected localisation rather than conventional map accuracy. Practical limitations include constrained speed evidence, simplified phantoms, limited trajectories, and optimistic operating-point selection.
- Comparison with baselines: 1.0–1.3 mm mean predicted-to-true vessel-pixel distance was obtained for all models except Sim→Meat, whose 5.5 mm was closer to the most difficult comparator.Direct comparison remains limited because the study uses different phantoms and evaluates reprojected localisation rather than per-trial classification.
- Comparison with baselines: Map accuracy was not reported because the overwhelmingly vessel-free grid would make true negatives dominate that metric.The study instead emphasizes mean distance from predicted vessel pixels to the nearest true vessel pixel.
- Sliding speed: The Meat experiments used approximately 10 mm/s sliding, while higher speeds such as 40 mm/s remain feasible only as an untested possibility.The authors state that higher speeds need investigation.
- Limitations: The evaluation is bounded by simplified visible-vessel Silicone phantoms, limited Silicone and Meat trajectories, approximately 2 mm marker spacing, and a calibration step whose scalability remains future work.These limitations motivate more realistic phantoms, broader interactions, finer sensing, and more automated calibration.
- Limitations: The map operating point is chosen on test data and maps use the best-of-five seed, making those figures optimistic by construction.Only seed-averaged metrics are free of this selection.
7 Conclusions and Future Work
The work is presented as a proof of concept for calibrated, interpretable tactile vessel localisation rather than clinical autonomy. Its next step is an uncertainty-aware tactile world model evaluated across broader motions, anatomies, and repeated trials.
- Conclusions: The framework combines a calibrated digital twin, domain-randomised tactile training, an ST-GNN, and kinematic reprojection into a human-verifiable vessel map.The twin achieved 0.50 mm marker-alignment MAE, while Sim→Silicone reached 1.2 mm mean map distance and Sim→Meat degraded to 5.5 mm.
- Conclusions: Trustworthiness is defined narrowly through physical calibration, explicit sim-to-real evaluation, seed variability, per-pixel statistics, and human-verifiable output—not clinical safety or autonomous intervention readiness.The conclusion explicitly limits the scope of the trustworthiness claim.
- Future work: The digital twin is currently an offline data generator; a future tactile world model would predict observations and map updates for uncertainty-aware closed-loop palpation.The proposed system could abstain on low-confidence predictions, but this capability is not part of the current framework.
- Future work: Broader evaluation should include repeated trials, held-out calibration trajectories, varied forces and speeds, branching vessels, non-planar surfaces, and more representative compliant fluid-filled phantoms.Human studies are proposed only after these evaluations and appropriate ethical approval.