Source-linked AI summary
Deep Learning with Convolutional Neural Network for Objective Skill Evaluation in Robot-assisted Surgery
Ziheng Wang, Ann Majewicz Fey
TL;DR
Existing surgical skill assessment often relies on engineered features or gesture segmentation, motivating more direct analysis of complex robot-motion data. The paper evaluates an end-to-end convolutional framework that maps multivariate motion windows to skill levels, achieving competitive performance and supporting short-window online assessment. The authors identify ground-truth skill labeling as an important limitation of supervised evaluation on JIGSAWS.
Problem
Existing surgical skill assessment often requires engineered intermediate features or gesture segmentation, while complex robot-motion data remain difficult to analyze for online assessment.
Method
A deep convolutional neural network maps multivariate robot-motion time series directly to surgical skill classifications using end-to-end learning.
Results
The model achieved competitive performance across Suturing, Needle-passing, and Knot-tying and decoded skills from 1–3 second motion windows.
Takeaways & Limitations
Raw motion profiles can support online objective skill assessment without engineered features or carefully tuned gesture segmentation.
Takeaways & Limitations
Supervised accuracy depends heavily on labeled samples, while JIGSAWS lacks consensus ground-truth skill annotations and commonly accepted GRS cutoffs.
Abstract
from arXiv · showhide
With the advent of robot-assisted surgery, the role of data-driven approaches to integrate statistics and machine learning is growing rapidly with prominent interests in objective surgical skill assessment. However, most existing work requires translating robot motion kinematics into intermediate features or gesture segments that are expensive to extract, lack efficiency, and require significant domain-specific knowledge. We propose an analytical deep learning framework for skill assessment in surgical training. A deep convolutional neural network is implemented to map multivariate time series data of the motion kinematics to individual skill levels. We perform experiments on the public minimally invasive surgical robotic dataset, JHU-ISI Gesture and Skill Assessment Working Set (JIGSAWS). Our proposed learning model achieved a competitive accuracy of 92.5%, 95.4%, and 91.3%, in the standard training tasks: Suturing, Needle-passing, and Knot-tying, respectively. Without the need of engineered features or carefully-tuned gesture segmentation, our model can successfully decode skill information from raw motion profiles via end-to-end learning. Meanwhile, the proposed model is able to reliably interpret skills within 1-3 second window, without needing an observation of entire training trial. This study highlights the potentials of deep architectures for an proficient online skill assessment in modern surgical training.
1 INTRODUCTION
Robot-assisted surgery generates complex motion data that could support objective skill assessment, but existing approaches often depend on engineered features and intermediate representations. This paper proposes an end-to-end deep-learning framework for online assessment from multivariate motion data.
- Motivation: Robot-assisted systems collect high-volume sensory data that may reveal operator skill and proficiency.
- Motivation: Nonlinear, non-stationary, stochastic motion profiles exhibit substantial variability and high dimensionality, challenging accurate skill assessment.
- Existing approaches: Existing predictive approaches commonly transform raw motion data into intermediate interpretations or summary features before assessment.
- Proposed direction: Deep learning can automatically discover hierarchical abstract representations, yet relatively little work had explored it for surgical skill assessment.
- Proposed approach: The proposed framework directly processes multivariate motion data and outputs skill classifications without manual feature extraction or selection.
- Evaluation: The study evaluates three JIGSAWS training tasks—Suturing, Needle-passing, and Knot-tying—and examines augmentation and validation schemes for deep skill modeling.
2 DEEP SURGICAL SKILL CLASSIFICATION MODEL
The proposed surgical skill classifier is a convolutional neural network that formulates assessment as supervised three-class classification of motion-kinematics time series. It processes fixed-length multichannel windows through convolution-pooling stages and is trained end-to-end with regularization.
- Model overview: The model uses a convolutional neural network for end-to-end classification of surgical skills from motion data.
- Input and architecture: The architecture uses three convolution-pooling stages, followed by flatten, fully connected, and softmax layers.
- Problem formulation: Assessment is formulated as supervised three-class classification of multivariate time series from robot end-effectors into Novice, Intermediate, or Expert labels.
- Input and architecture: The network receives a W × C matrix representing a length-W sensory window with C input channels.
- Training: The network is trained from scratch with Xavier initialization, Adam optimization, mini-batch gradient descent, and stochastic dropout regularization.
3 EXPERIMENT SETUP
The experiment uses JIGSAWS robot kinematics from three surgical training tasks, normalized and augmented into sliding-window samples for deep skill classification. Model performance is assessed with LOSO and hold-out validation using standard classification metrics and runtime.
- Inputs: Each task is represented by 19 measurements per robot end-effector, including positions, rotations, linear and angular velocities, and gripper angle.These sensory variables form the multivariate time-series inputs for each operation trial.
- Dataset: JIGSAWS records da Vinci kinematics from eight surgeons performing Suturing, Knot-tying, and Needle-passing tasks five times each.The robot data are multivariate time series captured at 30 Hz from end-effectors.
- Preprocessing: Z-normalization independently rescales each raw sensory channel using its mean and standard deviation before neural-network input.The normalization can be performed online on batches of sensory data.
- Augmentation: Data augmentation separates master and patient-side manipulator streams and extracts label-preserving fixed-size crops with a sliding window.The procedure enlarges the small dataset while preserving each trial’s class label.
- Augmentation: Using window width W = 60 and step size L = 30 produced 6290, 6780, and 3542 crops for Suturing, Needle-passing, and Knot-tying, respectively.Crop counts differ because the original JIGSAWS recordings have different lengths.
- Evaluation: The study compares LOSO cross-validation with a single train/test hold-out split for each surgical task and evaluates accuracy, precision, recall, F1-score, and runtime.LOSO averages results across five folds, while hold-out reserves one trial from each surgeon for testing.
4 RESULTS
The results evaluate self-proclaimed and GRS-based skill classification with LOSO and compare validation strategies. Performance varies across tasks and improves when the sliding window is increased from two to three seconds.
- Classification results: The model’s confusion matrices report self-proclaimed and GRS-based three-class skill classifications under LOSO evaluation.Rows encode true labels, columns encode predicted labels, and the diagonal represents correct predictions.
- Window-size analysis: 0.75%, 0.56%, and 2.38% average-accuracy improvements occur for Suturing, Needle-passing, and Knot-tying when W3 = 90 replaces W2 = 60.The three-second window contains 90 time steps and performs better than the two-second window.
- Validation comparison: The study also compares self-proclaimed skill classification under LOSO cross-validation and the Hold-out strategy.This comparison examines the validation strategy used for system development.
5 DISCUSSION
The discussion presents a deep learning approach for online surgical skill assessment, emphasizing competitive performance, efficient short-window decoding, validation choices, and limitations from skill-label quality and model interpretability.
- 5 DISCUSSION: The study addresses complex surgical data as an open problem and evaluates a deep learning approach for online skill assessment.The model reduces dependence on manual feature design and carefully tuned gesture segmentation.
- 5.1 Validity of our deep learning model for objective skill assessment: Suturing and Needle-passing outperform Knot-tying in skill classification, indicating that Knot-tying is more difficult to assess.This pattern appears in both self-proclaimed and GRS-based skill classification.
- 5.1 Validity of our deep learning model for objective skill assessment: 0.34 of self-proclaimed Intermediate samples were misclassified as Novice during Knot-tying.The authors attribute this pattern to self-proclaimed labels based on hours of robot operation.
- 5.1 Validity of our deep learning model for objective skill assessment: 133.88 ms, 172.87 ms, and 214.14 ms were required to classify the testing dataset for W1, W2, and W3, respectively.Larger windows can improve accuracy but increase computational effort and delay.
- 5.2 Comparison of Validation Schemes: LOSO cross-validation gives a reliable performance estimate, whereas Hold-out validation has larger result variance but lower computational cost.The authors note that Hold-out may be preferable for large datasets and complex architectures.
- 5.3 Limitations: Supervised deep learning depends heavily on labeled samples, while JIGSAWS lacks strict consensus on ground-truth skill annotations.The study also identifies limited interpretability from the black-box nature of deep learning models.
6 CONCLUSION
The conclusion highlights an end-to-end deep architecture for surgical skill classification, emphasizing raw-data representation learning, short-window efficiency, validation strategies, and online applicability.
- 6 CONCLUSION: The study contributes a data-driven deep architecture for end-to-end surgical skill classification.It also examines accuracy, time efficiency, data augmentation, and validation schemes for deep skill modeling.
- 6 CONCLUSION: 1−3 second windows with 30−90 time steps were computationally feasible for online assessment and immediate training feedback.An appropriate window size increased Knot-tying accuracy by 2.24%.
- 6 CONCLUSION: The model automatically learns abstract representations from raw sensory data with high predictive accuracy and fast processing speed.The authors state that it could be integrated into robot-assisted surgical systems for personalized training feedback.
Ethical approval
The paper states that formal consent is not required for this type of study.
- Ethical approval: Formal consent is not required for this type of study.
Informed consent
The paper states that it does not contain patient data.
- Informed consent: The article does not contain patient data.