Source-linked AI summary
Video-based surgical skill assessment using 3D convolutional neural networks
Isabel Funke, Sören Torge Mees, Jürgen Weitz, Stefanie Speidel
TL;DR
Automatic surgical skill assessment is difficult because motion-based methods require specialized tracking or robotic systems, while video is readily collected. The paper directly learns from surgical video using a pretrained 3D ConvNet within a Temporal Segment Network. On JIGSAWS dry-lab training tasks, it achieves at least 95% video-level classification accuracy and demonstrates feasibility, though broader evaluation requires more annotated data.
Problem
Existing automated skill-assessment methods often require robot kinematics or specialized tracking, motivating video-only assessment for surgical training.
Method
The method fine-tunes a pretrained 3D ConvNet within a Temporal Segment Network on stacks of video frames or optical-flow fields.
Results
At least 95% video-level classification accuracy is achieved on the JIGSAWS dataset for basic robot-assisted minimally invasive surgery tasks.
Takeaways & Limitations
The results indicate that 3D ConvNets can learn meaningful patterns from surgical video that differentiate expert, intermediate, and novice surgeons.
Takeaways & Limitations
The approach cannot explicitly model long-term temporal relations and requires a comprehensive video database with varied skill levels.
Abstract
from arXiv · showhide
Purpose: A profound education of novice surgeons is crucial to ensure that surgical interventions are effective and safe. One important aspect is the teaching of technical skills for minimally invasive or robot-assisted procedures. This includes the objective and preferably automatic assessment of surgical skill. Recent studies presented good results for automatic, objective skill evaluation by collecting and analyzing motion data such as trajectories of surgical instruments. However, obtaining the motion data generally requires additional equipment for instrument tracking or the availability of a robotic surgery system to capture kinematic data. In contrast, we investigate a method for automatic, objective skill assessment that requires video data only. This has the advantage that video can be collected effortlessly during minimally invasive and robot-assisted training scenarios. Methods: Our method builds on recent advances in deep learning-based video classification. Specifically, we propose to use an inflated 3D ConvNet to classify snippets, i.e., stacks of a few consecutive frames, extracted from surgical video. The network is extended into a Temporal Segment Network during training. Results: We evaluate the method on the publicly available JIGSAWS dataset, which consists of recordings of basic robot-assisted surgery tasks performed on a dry lab bench-top model. Our approach achieves high skill classification accuracies ranging from 95.1% to 100.0%. Conclusions: Our results demonstrate the feasibility of deep learning-based assessment of technical skill from surgical video. Notably, the 3D ConvNet is able to learn meaningful patterns directly from the data, alleviating the need for manual feature engineering. Further evaluation will require more annotated data for training and testing.
1 Introduction
Surgical skill assessment supports novice training and certification, but existing automated approaches often rely on specialized motion or kinematic data. The paper proposes learning directly from surgical video with a 3D ConvNet and Temporal Segment Network.
- Motivation: Skill assessment enables targeted feedback for novice surgeons and supports certification.Assessment tells novices how to improve and helps certify prospective surgeons.
- Problem: Manual expert evaluation and standardized protocols are time-consuming and expensive.Automating evaluation could support training without continuous human supervision.
- Problem: Most automated methods analyze robot kinematics or tool motion, which generally require robotic systems or specialized tracking equipment.Video is easier to acquire but is higher-dimensional and more complex than short motion-variable sequences.
- Related work: Prior video approaches commonly track surgical tools or transform video into engineered representations such as BoW descriptors.Examples include tool usage patterns, movement range, economy of motion, and STIP descriptors built from HOF and HOG features.
- Contribution: The proposed method uses a 3D ConvNet to learn feature representations directly from video, avoiding manually engineered intermediate representations.The authors present this as an automatic, objective surgical skill classification approach using video data only.
- Contribution: The method combines an inflated 3D ConvNet, Kinetics pretraining, and a Temporal Segment Network for surgical video classification.The TSN considers multiple segment-sampled snippets to resolve ambiguities in individual snippets.
- Evaluation: Evaluation uses JIGSAWS recordings of three elementary robot-assisted surgery tasks performed on dry lab bench-top models.The dataset represents surgical training scenarios but does not contain intraoperative recordings or real surgical scenes.
2 Methods
The method classifies surgical skill from short video snippets aggregated across a video using a Temporal Segment Network with an inflated 3D ConvNet. It uses spatiotemporal modeling, pretrained initialization, and segment-based sampling for video-level classification.
- Problem formulation: Each surgical-training video is classified as expert, intermediate, or novice based on the demonstrated skill level.The videos depict performances of tasks such as suturing or knot tying and typically last one to five minutes.
- Temporal Segment Network: A Temporal Segment Network classifies multiple short snippets and aggregates their predictions into an overall video classification.Consensus can compensate for contradictions and ambiguities in individual snippets.
- Temporal Segment Network: During training, each video is divided into K nonoverlapping temporal segments and one snippet is randomly sampled from each segment.This segment-based strategy provides K snippets for consensus learning.
- Snippet structure: The method uses 64 consecutive frames per snippet sampled at 10 Hz, representing more than six seconds of surgical video.Input snippets can use RGB frames or optical-flow fields.
- ConvNet architecture: A 3D ConvNet models temporal as well as spatial information in stacks of video or optical-flow frames.The selected Inception-v1 I3D architecture is an inflated 3D version of Inception.
- Output: The network outputs three one-hot encoded skill classes: expert, intermediate, and novice.The output layer is adapted specifically for surgical skill classification.
- Training: The 3D ConvNet is initialized from Kinetics pretraining before fine-tuning on surgical video.Kinetics contains 400 action classes and at least 400 video clips per class, providing a large related-domain initialization.
- Training: Freezing earlier layers reduces the number of trainable parameters and allows fine-tuning with one GPU containing 8 GB of video RAM.The implementation uses PyTorch and standard video augmentations including corner cropping, scale jittering, and horizontal flipping.
3 Evaluation
The evaluation uses JIGSAWS surgical-task videos to assess skill classification with LOSO cross-validation and reports accuracy, average recall, and average F1. Ablation studies examine the 3D ConvNet, TSN training, and Kinetics pretraining, while LOUO results are limited by the dataset’s participant count.
- Dataset: JIGSAWS contains suturing, needle passing, and knot tying recordings from robot-assisted tasks performed on a dry lab bench-top model.The dataset includes laparoscopic stereo video and synchronized robot kinematics; this evaluation uses video-only input.
- LOSO evaluation: The method is evaluated separately for each task using five-fold LOSO cross-validation, with test videos classified by models trained without those videos.Accuracy, average recall, and average F1 score are computed from the predictions.
- LOSO evaluation: Results are averaged over four evaluation runs for RGB and optical-flow modalities, with mean and standard deviation reported for the 3D ConvNet method.Comparisons include state-of-the-art methods that use robot kinematics, whereas no other JIGSAWS video-based method was identified.
- Ablation studies: The ablation results indicate that combining a 3D ConvNet with TSN-based training benefits skill classification by learning spatiotemporal features and aggregating snippet predictions.Consensus across multiple snippets helps resolve contradictions and ambiguities in individual snippets.
- Ablation studies: Ablations on knot tying with optical flow test the benefits of 3D versus 2D convolution, TSN training, and Kinetics pretraining.The experiments use modified training procedures while focusing on the most challenging task.
- LOUO evaluation: LOUO cross-validation is considered unreliable on JIGSAWS because the dataset has only two experts and two intermediates, limiting intra-class variability in training splits.The resulting model struggles particularly with recognizing intermediate and expert performances.
4 Discussion
The video-only method achieves high skill-classification performance and learns discriminative patterns without manual intermediate representations. Its performance depends on pretraining and combined spatiotemporal modeling, while task difficulty and data limitations remain important boundaries.
- The method achieves high classification accuracies comparable to state-of-the-art approaches, with RGB and optical-flow inputs performing similarly overall.Optical flow reaches 100% accuracy on needle passing, suggesting a slight modality advantage for that task.
- Raw surgical video enables the model to learn discriminative patterns despite the greater complexity of video compared with 76-variable robot-kinematic data.This could extend assessment to low-cost trainers without sophisticated data-acquisition systems.
- The model attains its lowest accuracy on knot tying, mostly misclassifying intermediate surgeons as novices.Because JIGSAWS labels are based on robotic-surgery experience hours, intermediate performers may display novice-like motion patterns on this challenging task.
- The handcrafted approximate-entropy feature still outperforms the proposed approach, although the authors see learning-based methods as promising for detecting procedural errors.They note that dexterity features alone cannot detect such errors.
- The 3D ConvNet and TSN combination benefits performance by learning spatiotemporal features and aggregating multiple snippets to resolve local ambiguities.TSN training finds consensus across snippets rather than assigning each snippet the video-level skill label explicitly.
- Kinetics pretraining is the most important training step; training a TSN from scratch yields only 41% accuracy on knot tying and overfits the training data.The model fit training examples perfectly but failed to generalize to unseen examples.
- The approach cannot explicitly model long-term temporal relations and requires a comprehensive database of annotated performances spanning surgical skill levels.Future work may address data limitations through semi-supervised or self-supervised learning.
5 Conclusion
The paper presents a video-only deep-learning approach for assessing technical surgical skill with a pretrained 3D ConvNet trained through TSN. On JIGSAWS, it achieves at least 95% video-level accuracy, while larger annotated datasets and richer temporal modeling remain future needs.
- The approach assesses technical surgical skill using video data only.
- A pretrained 3D ConvNet is fine-tuned through TSN on stacks of video frames or optical-flow fields for skill classification.
- At least 95% video-level classification accuracy is achieved on JIGSAWS recordings of basic robot-assisted minimally invasive surgery tasks.The model differentiates expert, intermediate, and novice surgeons from learned patterns in the data.
- Future work must expand and annotate surgical-skill datasets and investigate combining RGB video with optical flow.The broader goal includes modeling interactions with the environment and long-range temporal structure.