Source-linked AI summary
Automated Surgical Skill Assessment in RMIS Training
Aneeq Zia, Irfan Essa
TL;DR
Manual RMIS training feedback is time-consuming and subjective, and automated feedback for basic surgical tasks is limited. The paper evaluates holistic robot-kinematic features with weighted fusion for skill assessment and score prediction. These features outperform prior HMM methods for classification, while fusion improves score prediction to up to 0.61 average Spearman correlation.
Problem
Manual feedback in basic RMIS training is time-consuming and subjective, while automated feedback for basic surgical tasks is limited.
Method
The framework extracts SMT, DFT, DCT, and ApEn features from robot kinematics, then uses individual or weighted-fused features for classification, score prediction, and task highlights.
Results
Holistic features outperform previous HMM state-of-the-art methods for JIGSAWS skill classification, and fusion improves score prediction to up to 0.61 average Spearman correlation.
Takeaways & Limitations
Global information from robot kinematics can evaluate surgeon skill in basic da Vinci tasks and potentially support real-time score feedback in RMIS training.
Takeaways & Limitations
The task-highlight impact plots have no ground-truth highlights for comparison.
Abstract
from arXiv · showhide
Purpose: Manual feedback in basic RMIS training can consume a significant amount of time from expert surgeons' schedule and is prone to subjectivity. While VR-based training tasks can generate automated score reports, there is no mechanism of generating automated feedback for surgeons performing basic surgical tasks in RMIS training. In this paper, we explore the usage of different holistic features for automated skill assessment using only robot kinematic data and propose a weighted feature fusion technique for improving score prediction performance. Methods: We perform our experiments on the publicly available JIGSAWS dataset and evaluate four different types of holistic features from robot kinematic data - Sequential Motion Texture (SMT), Discrete Fourier Transform (DFT), Discrete Cosine Transform (DCT) and Approximate Entropy (ApEn). The features are then used for skill classification and exact skill score prediction. Along with using these features individually, we also evaluate the performance using our proposed weighted combination technique. Results: Our results demonstrate that these holistic features outperform all previous HMM based state-of-the-art methods for skill classification on the JIGSAWS dataset. Also, our proposed feature fusion strategy significantly improves performance for skill score predictions achieving up to 0.61 average spearman correlation coefficient. Conclusions: Holistic features capturing global information from robot kinematic data can successfully be used for evaluating surgeon skill in basic surgical tasks on the da Vinci robot. Using the framework presented can potentially allow for real time score feedback in RMIS training.
1 Introduction
Basic RMIS training relies on manual assessment that is subjective and inefficient, while automated feedback remains limited. The paper proposes a framework combining holistic kinematic features, weighted fusion, and task highlights for more directed assessment.
- Manual assessment of dry-lab RMIS exercises is subjective and reduces training efficiency.
- Prior work emphasized gesture recognition or HMM-based task modeling, whereas this paper investigates global features from complete task data.
- Texture-, frequency-, and entropy-based features outperform previous HMM state-of-the-art methods for JIGSAWS skill classification using kinematic data.
- Weighted feature fusion is proposed for skill-score prediction, alongside analysis of how different features contribute to performance.
- A task-highlight technique identifies parts of a task with the most positive or negative impact on predicted scores.
2 Background
Research on automated surgical-skill assessment spans gesture recognition, motion-texture, frequency, entropy, and HMM-based approaches. This paper addresses a gap by applying repeatability- and predictability-oriented holistic features to robot kinematic data.
- Video-based studies have used SMT texture features, DCT and DFT frequency features, and approximate entropy to represent surgical motion.
- Prior RMIS work includes sparse HMMs, gesture-recognition models, SVMs using basic motion metrics, and other skill-prediction techniques.
- Earlier RMIS studies did not examine frequency- and entropy-based features capturing repeatability and predictability directly from robot kinematics.
- The proposed framework is motivated by the hypothesis that smoother, more predictable expert motions contain more skill-relevant information.
3 Methodology
The methodology extracts holistic representations from robot kinematic time series, reduces dimensionality, and predicts skill or scores using learned models. It also reconstructs omitted segments with inverse DCT operations to identify task highlights.
- 3.1 Skill Classification/Score Prediction: Four holistic feature types—SMT, DFT, DCT, and ApEn—are extracted from robot kinematic time series before PCA-based dimensionality reduction.
- 3.1 Skill Classification/Score Prediction: Feature-specific outputs are concatenated and combined through least-squares learning of an optimal weight vector.
- 3.1 Skill Classification/Score Prediction: Nearest-neighbor classification predicts skill levels, while linear SVR predicts exact skill scores.
- 3.2 Task Highlights: Task highlights are motivated as a way to show which segments most affect final score predictions and guide focused training.
- 3.2 Task Highlights: For each kinematic dimension, DCT features are transformed after removing a frame range, yielding an inferred representation for score prediction.
4 Experimental Evaluation
The framework is evaluated on JIGSAWS kinematic data from three robotic surgical tasks, using holistic features with PCA-based dimensionality reduction and task-highlight settings.
- Dataset: JIGSAWS provides kinematic data from 8 participants performing Suturing, Knot Tying, and Needle Passing for evaluation.The analysis uses only kinematic data with LOSO and LOUO cross-validation setups.
- Feature processing: ApEn uses six radius values from 0.1 to 0.25, producing a 6D-dimensional feature vector with m = 1 and τ = 1.
- Feature processing: The evaluation extracts SMT, DFT, DCT, and ApEn features from robot kinematic time series before classification or prediction.The features are reduced using PCA before being passed to the classifier or regression model.
- Parameter selection: PCA component counts are evaluated from 10 to 3000, with the value yielding the highest average performance across tasks selected.For score prediction, the SVR regularization parameter C is also estimated.
- Task highlights: Task highlights use the 50 lowest DCT features with a running window of length 100.
5 Results and Discussion
Holistic kinematic features outperform prior HMM methods for skill classification, while weighted fusion improves score prediction across tasks and supports interpretable task highlights.
- Skill classification: Holistic features significantly outperform previous HMM variants for self-proclaimed skill classification on JIGSAWS, with ApEn performing best.The paper relates ApEn’s advantage to experts’ smoother, more predictable motions.
- Score prediction: Weighted fusion performs better than individual features, although performance is comparatively lower under LOUO and for needle-passing.The authors attribute lower needle-passing performance to its relatively low repetitiveness.
- Score prediction: DCT+DFT+ApEn performs best on average across the three tasks for both OSATS and GRS score prediction.The comparison aggregates ρ values across all three tasks.
- Feature fusion: DCT receives the highest learned fusion weight in most cases, whereas DFT and ApEn receive similar weights and SMT remains low-weighted.The weight heatmaps show feature contributions for OSATS criteria and GRS under different tasks and validation schemes.
- Task highlights: Task highlights use impact scores over frames and gestures to identify segments with positive or negative effects on predicted skill scores.Examples include G3 in suturing and G15 in knot tying, but no ground-truth highlights are available for comparison.
6 Conclusion
The framework uses holistic kinematic features and weighted fusion for automated RMIS skill assessment, with real-time feedback and task highlights for more focused training.
- Holistic SMT, DCT, DFT and ApEn features outperform existing HMM-based approaches for RMIS skill assessment.
- Weighted feature combination further improves score prediction performance.
- The framework is computationally feasible for real-time feedback and can generate task highlights for more focused surgical training.