Source-linked AI summary
Evaluating surgical skills from kinematic data using convolutional neural networks
Hassan Ismail Fawaz, Germain Forestier, Jonathan Weber, Lhassane Idoumghar, Pierre-Alain Muller
TL;DR
Manual and kinematic-data-based surgical skill assessments have limits in objectivity, interpretability, and actionable feedback. This paper uses a hierarchical CNN on robotic-surgery kinematics and class activation maps to visualize task regions relevant to skill predictions. It reports 100% accuracy for suturing and needle passing on JIGSAWS, while knot tying is less accurate.
Problem
Existing surgical skill assessment can be subjective, and kinematic approaches have lacked interpretable feedback while some require manually annotated gesture boundaries.
Source: §1 · pp. 1–2Method
A hierarchical one-dimensional CNN classifies surgical skill from kinematic time series, with class activation maps highlighting task regions contributing to predictions.
Source: §1 · p. 2, §2.2 · p. 3, §2.4 · p. 5Results
100% accuracy was reported for suturing and needle passing; for knot tying, the CNN reported 92.1% micro and 93.2% macro configurations.
Source: §3.1 · p. 5Takeaways & Limitations
The activation maps visualize movements contributing to a skill classification and can support trainee feedback about movements associated with novice or expert classifications.
Source: §3.2 · p. 6The channel subclusters are based on hypothesized correlations among kinematic variables.
Source: §2.2 · p. 4
Abstract
from arXiv · showhide
The need for automatic surgical skills assessment is increasing, especially because manual feedback from senior surgeons observing junior surgeons is prone to subjectivity and time consuming. Thus, automating surgical skills evaluation is a very important step towards improving surgical practice. In this paper, we designed a Convolutional Neural Network (CNN) to evaluate surgeon skills by extracting patterns in the surgeon motions performed in robotic surgery. The proposed method is validated on the JIGSAWS dataset and achieved very competitive results with 100% accuracy on the suturing and needle passing tasks. While we leveraged from the CNNs efficiency, we also managed to mitigate its black-box effect using class activation map. This feature allows our method to automatically highlight which parts of the surgical task influenced the skill prediction and can be used to explain the classification and to provide personalized feedback to the trainee.
1 Introduction
Existing surgical skill assessments rely on expert observation, outcome measures, or global motion features, with limits in objectivity, data availability, and actionable feedback. The paper proposes a CNN that learns from kinematic data and uses class activation maps to make skill predictions more interpretable.
1 Introduction
- Current checklist-based assessment remains vulnerable to subjective factors, including checklist development, inter-rater reliability, and evaluator bias.OSATS is used in clinical practice, but it is still an observational evaluation.Source: §1 · p. 1
- Outcome-based assessment is difficult to apply in training because surgical outcomes also depend on patient physiology and are difficult to acquire.Technical skill has been associated with postoperative outcomes.Source: §1 · pp. 1–2
- Global motion features can classify skill, but their use for detailed, constructive trainee feedback is unclear.Examples include task completion time, speed, curvature, and motion smoothness.Source: §1 · p. 2
- Gesture-based methods can be accurate, but require many labeled gestures; HMM approaches also require human-defined boundaries.The paper identifies both a lack of interpretable evaluation and time-consuming, reliability-prone annotation as limits of kinematic-data approaches.Source: §1 · p. 2
- The proposed CNN learns hierarchical motion representations and uses class activation maps to highlight task regions contributing to skill classification.The architecture avoids predefining gesture boundaries and is evaluated on JIGSAWS.Source: §1 · p. 2
2 Method
The method treats each JIGSAWS trial as a multivariate kinematic time series and classifies skill with a hierarchical one-dimensional CNN. Its channel grouping, training setup, and global-average-pooling-based activation maps define how the model learns and visualizes task regions relevant to predictions.
2.1 Dataset
- Each trial contains 76 kinematic variables from four manipulators, recorded at 30 frames per second and treated as a multivariate time series.JIGSAWS includes eight right-handed subjects, three skill levels, three tasks, and five trials per subject per task.Source: §2.1 · p. 3
2.2 Architecture
- The CNN accepts variable-length inputs with 76 channels and uses domain knowledge to group channels for separate convolutions across its hierarchical layers.The 76 variables are divided into four 19-variable manipulator clusters, then into five subclusters based on hypothesized correlations; convolution choices differ by subcluster in layer one, by manipulator cluster in layer two, and are shared across channels in layer three.Source: §2.2 · pp. 3–4
2.4 Class Activation Map
- A global average pooling layer replaces the fully connected layer after the third convolution, preserving channel count while reducing time length to one and enabling class activation maps.Zero padding preserves the original time-series length through the convolutions, allowing the activation map to align with the task timeline.Source: §2.2 · p. 4, §2.4 · p. 5
2.3 Training & Testing
- The model is trained separately for each task with the same architecture and hyperparameters, selecting epochs using a random non-seen split from the training set.Training uses 1,000 epochs, Adam, cross-entropy, and l2 regularization of 10^-5; testing follows five-fold Leave One Super Trial Out.Source: §2.3 · pp. 4–5
3 Results
Across three surgical tasks, the CNN achieved 100% accuracy on suturing and needle passing, with lower micro and macro results on knot tying. Class Activation Maps (CAMs) visualize movements contributing to skill classifications and can support trainee feedback.
3.1 Surgical skill classification
- Compared with Sparse Hidden Markov Models, the CNN achieved much higher accuracy while still providing interpretable skill evaluation.The S-HMM approach uses gesture boundaries during training.Source: §3.1 · p. 5
- Approximate Entropy and the CNN both achieved 100% accuracy for suturing and needle passing, but the authors note ApEn’s feedback extension is unclear.They hypothesize cross-validation and hyperparameter tuning could improve CNN accuracy, especially for knot tying.Source: §3.1 · p. 6
- The CNN is reported as more accurate and equally interpretable than the sliding-window bag-of-words approach.That approach uses discretized motion segments and cosine similarity to classify trials.Source: §3.1 · p. 6
- 100% accuracy was reported for suturing and needle passing; knot tying reached 92.1% micro and 93.2% macro accuracy.The authors attribute the lower knot-tying accuracy to that task’s complexity.Source: §3.1 · p. 5, Table 1 · p. 6
3.2 Feedback visualization
- CAM heatmaps identify movements behind a skill classification, including movements that led a novice trial to be classified as novice.The authors suggest this can help trainees identify good and bad movements; Figure 2 illustrates the feedback.Source: §3.2 · p. 6, Figure 2 · p. 7
4 Conclusion
The paper presents a CNN for classifying surgical skills and reports 100% accuracy alongside interpretable evaluation. Future work aims to combine video and kinematic data in a unified CNN framework.
4 Conclusion
- The authors report 100% accuracy and interpretability that helps explain skill evaluations and reduces the CNN’s black-box effect.The conclusion does not specify the tasks associated with this accuracy.Source: §4 · p. 7
- Future work aims to develop a unified CNN using video and kinematic data for skill classification and interpretable trainee feedback.Source: §4 · p. 7