PaperlayerView exact revision on arXiv

Source-linked AI summary

PICO: Projection-Informed Consistency Optimisation for 6DoF Surgical Tool Pose Estimation

Lucy Fothergill, Pietro Valdastri, Dominic Jones, Duygu Sarikaya

arXiv:2609.30989v1cs.CV

TL;DR

Accurate surgical-tool pose is important for safe interaction, but kinematic estimates can accumulate error and marker-based vision faces workflow and visibility constraints. PICO combines pose regression with segmentation, depth, and 2D/3D consistency losses. On SurgRIPE, it ranks second in rotation across four test datasets, including occluded cases, and remains competitive in translation.

  • Problem

    Accurate tool pose supports safe surgical interaction, while kinematics can accumulate error and marker-based methods require procedures and line of sight that may be unavailable.

    Source: §1 · p. 2

  • Method

    PICO jointly predicts segmentation and depth and regresses translation and rotation, using projection and point-to-point losses to enforce 2D and 3D geometric consistency.

    Source: §1 · p. 3, §3 · p. 5

  • Results

    PICO ranks second in rotational accuracy across all four SurgRIPE test datasets, including occluded cases, and remains competitive in translation, especially under occlusion.

    Source: §1 · p. 3, §5 · p. 10

  • Takeaways & Limitations

    The results support PICO as a robust and accurate approach to 6DoF surgical tool pose estimation, including in occluded scenarios.

    Source: §6 · p. 12, §7 · p. 13

    SurgRIPE remains the only publicly available surgical-domain dataset with ground-truth pose annotations, limiting the available evaluation scope.

    Source: §6 · p. 13

Abstract

from arXiv · show

Purpose: Accurate 6 DoF pose estimation of surgical tools is critical for automa- tion, robotic proprioception, and safe interaction with the tissue operated on. Kinematics-based approaches suffer from accumulated errors due to the cable- driven nature of robotic arms, while vision-based methods often rely on external markers or trackers. Although more recent vision-based advances have been pro- posed, these two-stage pose estimation methods often lack real-time robustness due to accumulated errors and computational overhead. Methods: We propose a novel end-to-end trainable model, PICO. Our model employs a multi-task learning architecture to predict segmentation and depth maps, alongside regression of translation and rotation parameters. We define two proxy tasks that enforce geometric consistency in both 2D and 3D spaces, improving accuracy and robustness. For this, we propose a projection loss, and a point-to-point loss. Results: We evaluate our method on the SurgRIPE dataset, benchmarking its performance against state-of-the-art approaches using standard 6DoF pose esti- mation metrics. Our results demonstrate consistently strong performance across all four subsets, specifically in rotation, ranking second even under occlusion. It also demonstrates comparable translational performance, remaining competitive, especially in occluded cases. Conclusion: PICO demonstrates the effectiveness of multi-task learning and geometry-aware proxy tasks for robust and reliable surgical tool pose estimation, especially in occluded scenarios, highlighting potential for future applications.

1 Introduction

Accurate tool pose supports safe robotic surgery, but kinematic estimates can err and marker-based vision is impractical when line of sight is blocked. PICO combines pose regression with segmentation, depth, and 2D/3D geometric consistency tasks; it ranks second in rotation across four test sets and remains competitive in translation.

1 Introduction

  • Kinematics-based pose estimates can have discrepancies up to 1.02 mm from cable slack, pulley-cable friction, and accumulated joint errors.The text notes that small errors can lead to unintended incisions or damage to critical structures.Source: §1 · p. 2
  • Marker-based methods require time-consuming procedures and clear camera-to-marker line of sight, which blood, tissue, smoke, or other instruments can obstruct.The introduction therefore favors markerless methods for surgical settings.Source: §1 · p. 2
  • PICO jointly predicts segmentation and depth maps while regressing translation and rotation, with proxy tasks enforcing consistency in 2D and 3D.Its projection loss aligns predicted pose with predicted segmentation; its point-to-point loss minimizes RMSE between transformed 3D model points.Source: §1 · pp. 2–3
  • PICO ranks second in rotational accuracy across all four test datasets, including occluded cases, while remaining competitive in translation, especially under occlusion.The evaluation uses the SurgRIPE dataset and standard 6DoF pose metrics.Source: §1 · p. 3, Figure 1 · p. 3

2 Related Work

Prior 6D pose methods commonly estimate intermediate features or an initial pose before refinement, while direct pose prediction is less common in surgical tool estimation. These pipelines face computational overhead and sensitivity to intermediate predictions, motivating direct estimation with geometric consistency.

2 Related Work

  • Object-pose methods include semidense template matching followed by optical-flow refinement, or initial pose estimates refined through dense optical flow or render-and-compare [10].GDR-Net uses dense 2D–3D correspondences and differentiable PnP, while PVNet predicts keypoints for a PnP solver.Source: §2 · p. 4
  • Surgical methods often infer pose from segmentation, keypoints, or depth before a later solver; direct pose estimation is less common [21].Examples of direct approaches include Spektor et al. and Fan et al..Source: §2 · p. 4
  • Multi-stage surgical pipelines add computation and depend on intermediate predictions, while correspondence generation, iterative refinement, or PnP can limit real-time suitability.The text also notes that articulation and metallic-surface reflections challenge generalization of object-pose methods to surgery.Source: §2 · p. 4

3 Methodology

PICO estimates pose and auxiliary image representations from a cropped monocular RGB image using a shared encoder and task-specific heads. Its joint losses supervise image predictions and pose while adding projection-based 2D alignment and point-to-point 3D consistency.

3 Methodology

  • PICO jointly predicts segmentation masks and depth maps while estimating 6DoF pose as rotation and translation from a cropped monocular RGB image.Rotation is regressed as a 3×3 matrix and projected onto the nearest valid rotation matrix in SO(3) using SVD; translation is expressed as image-frame (x,y) and camera-depth z.Source: §3 · p. 5
  • A shared encoder feeds a decoder for segmentation and depth, and regression heads for rotation and translation; translation is split into image coordinates (x,y) and depth z.The U-Net-style architecture uses a ResNet-50 encoder; pseudo-depth maps provide additional 3D supervision beyond the predicted point.Source: §3 · p. 5

3.1 Loss Functions

  • The end-to-end multi-task objective sums segmentation, depth, projection, point-to-point, rotation, and translation losses.Image-based segmentation and depth losses are combined with losses that encourage geometric and pose consistency.Source: §3.1 · pp. 6–7
  • Pseudo-depth maps project randomly sampled tool-mesh points using the ground-truth pose; collisions retain the nearest depth, and holes are filled with neighboring-pixel averages.The method uses the Scale-Invariant Loss proposed for monocular depth estimation.Source: §3.1 · p. 6
  • The projection loss compares the predicted segmentation with a projected binary mask formed from the pose-transformed 3D tool points.The projected mask acts as pseudo-ground-truth supervision for alignment between 2D appearance and 3D model space.Source: §3.1 · p. 6
  • The point-to-point loss minimizes RMSE between corresponding 3D model points transformed by predicted and ground-truth poses.Rotation uses geodesic angular distance rather than Euclidean distance between matrix elements; translation RMSE is minimized separately for (x,y) and z.Source: §3.1 · pp. 6–7

4 Experiments

Experiments use monocular RGB images from SurgRIPE, evaluating two surgical tools under occluded and unoccluded conditions with translation, rotation, and ADD metrics. During inference, the decoder is discarded and a detector supplies tool crops.

4 Experiments

  • SurgRIPE provides 1,147 large needle driver and 1,109 Maryland bipolar forceps images, split 90:10 for training and validation, plus occluded and unoccluded test sets for each tool.The study compares against SurgRIPE challenge results.Source: §4 · p. 8
  • Evaluation uses translation error, rotation error, and ADD, which measures the percentage of instances below 10% of instrument diameter in average transformed-point distance.The test images include both LND and MBF, each with occluded and unoccluded sets.Source: §4 · p. 8
  • At inference, the decoder is discarded and YOLOv5 predicts the tool bounding box before a 30-pixel-padded crop is resized to 224 × 224.The network is trained for 50 epochs with batch size 8 and Adam at learning rate 10^-4, with stopping after 10 epochs without validation-loss improvement.Source: §4 · p. 8

5 Results

PICO ranked second in rotational accuracy across all four test datasets, including occluded cases, while remaining competitive in translation. The results also report inference speed, assess loss-term ablations, and identify depth error as the primary limiting factor for ADD.

5 Results

  • Across all four test datasets, PICO ranked second in rotational accuracy, including under occlusion, and remained translationally competitive; only IMF achieved lower rotation error.For LND-occluded cases, PICO’s translation error was 8.87 mm versus 28.09 mm for PVNet.Source: Table 1 · p. 9, §5 · p. 10

5.1 Runtime Analysis

  • At 33.3 FPS (~30 ms/image) on a Tesla T4, PICO exceeded PVNet’s reported ~25 FPS on a GTX 1080ti, though tested on different GPUs.Source: §5.1 · p. 10

5.2 Ablation Studies

  • The full loss improves both metrics over segmentation–depth on LND, MBF, and MBF-occluded, and translation on LND-occluded, coupling signals through one predicted transformation.Depth supervision yields the lowest translation error on LND, LND-occluded, and MBF; segmentation improves rotation on LND and MBF, but combining the tasks does not simply add their gains.Source: §5.2 · pp. 10–11, Table 2 · p. 11

5.3 Error Decomposition and the ADD Metric

  • Across LND, LND-occluded, MBF, and MBF-occluded, z-error/ADD Spearman correlations were 0.955/0.978/0.963/0.982 versus xy-error correlations of 0.449/0.650/0.602/0.847, identifying depth error as ADD’s primary limit.The figure plots xy and z translation errors against ADD distance; z error exceeds xy error across all four test sets.Source: §5.3 · pp. 11–12, Figure 2 · p. 12

6 Discussion

PICO remains competitive with multi-stage methods while substantially improving rotational accuracy over the other single-stage method. Ablations show that geometric losses recover performance lost when depth and segmentation tasks interfere, while depth estimation remains a limitation.

6 Discussion

  • PICO substantially improves rotational accuracy over the other single-stage method without compromising translation, remains competitive with multi-stage frameworks, and runs at ≈33 FPS on a consumer-level NVIDIA Tesla T4 GPU.The results suggest that end-to-end single-stage networks can match iterative refinement approaches when guided by geometric consistency losses.Source: §6 · p. 12
  • Depth primarily limits ADD scores, while the auxiliary depth task reduces translation error from 13.28 to 8.36 mm on LND occluded.The authors state that closing the remaining gap requires more accurate depth supervision than their current pseudo-depth maps provide.Source: §6 · p. 12
  • Depth and segmentation auxiliary tasks address translation and orientation, respectively, but interfere when jointly optimised; projection and point-to-point losses couple both signals to one transformation and recover lost performance.The authors attribute the full model's improvement to these geometric losses.Source: §6 · p. 12
  • Evaluation is limited to SurgRIPE comparisons reported in the benchmark challenge paper, and pseudo-depth approximations and crop resizing may introduce depth or aspect-ratio errors.The authors also identify generalisation to unseen instruments and surgical environments as an ongoing challenge.Source: §6 · p. 13

7 Conclusion

PICO is a real-time, end-to-end approach to monocular RGB surgical tool pose estimation that combines multi-task prediction with geometric consistency proxy tasks. On SurgRIPE, it is competitive in translation and rotation, including occluded settings, but poor depth estimation limits ADD performance.

7 Conclusion

  • PICO jointly predicts segmentation and depth while regressing rotation and translation, using projection and point-to-point proxy tasks to enforce geometric consistency in 2D and 3D.The model is an end-to-end trainable approach to 6DoF pose estimation from monocular RGB images.Source: §7 · p. 13
  • On SurgRIPE, PICO achieves competitive translational and rotational accuracy, especially in occluded settings, and significantly improves rotation over MVL_3S.It remains competitive with multi-stage methods.Source: §7 · p. 13
  • PICO operates at approximately 33 FPS and is presented as suitable for real-time surgical applications.The authors describe it as a promising path toward real-time, markerless 6DoF pose estimation for surgical robotics.Source: §7 · p. 13
  • Poor depth estimation limits ADD performance; preprocessing distortions and few publicly released surgical pose datasets also constrain evaluation and reproducibility.The authors identify improved depth modelling and generalisation to unseen tools and environments as future directions.Source: §7 · p. 13

Related papers

Loading 2609.30989v1…