Source-linked AI summary
Tool Detection and Operative Skill Assessment in Surgical Videos Using Region-Based Convolutional Neural Networks
Amy Jin, Serena Yeung, Jeffrey Jopling, Jonathan Krause, Dan Azagury, Arnold Milstein, Li Fei-Fei
TL;DR
Surgical skill assessment is currently manual, subjective, and time consuming, while spatial tool localization is needed for richer movement analysis. The paper introduces an R-CNN-based approach and the m2cai16-tool-locations dataset, showing effective spatial detection, stronger frame-level presence detection, and surgical-quality assessment from tool movements.
Problem
Manual expert assessment of surgical performance is subjective and time consuming, and existing work largely detects tool presence without spatial bounds needed for movement analysis.
Method
The authors combine Faster R-CNN-based spatial tool detection with the m2cai16-tool-locations dataset, which extends m2cai16-tool with tool bounding boxes.
Results
The method effectively detects tool bounds, outperforms previous frame-level presence detection, and supports surgical-quality assessment using tool usage patterns, movement range, and economy of motion.
Takeaways & Limitations
Spatial tool detections enable richer automated analysis of surgical quality in real-world laparoscopic videos.
Takeaways & Limitations
Future work remains to automatically extract additional information such as motion smoothness, tissue damage, repeated movements, and procedure phase, and to develop an automated GOALS rating system.
Abstract
from arXiv · showhide
Five billion people in the world lack access to quality surgical care. Surgeon skill varies dramatically, and many surgical patients suffer complications and avoidable harm. Improving surgical training and feedback would help to reduce the rate of complications, half of which have been shown to be preventable. To do this, it is essential to assess operative skill, a process that currently requires experts and is manual, time consuming, and subjective. In this work, we introduce an approach to automatically assess surgeon performance by tracking and analyzing tool movements in surgical videos, leveraging region-based convolutional neural networks. In order to study this problem, we also introduce a new dataset, m2cai16-tool-locations, which extends the m2cai16-tool dataset with spatial bounds of tools. While previous methods have addressed tool presence detection, ours is the first to not only detect presence but also spatially localize surgical tools in real-world laparoscopic surgical videos. We show that our method both effectively detects the spatial bounds of tools as well as significantly outperforms existing methods on tool presence detection. We further demonstrate the ability of our method to assess surgical quality through analysis of tool usage patterns, movement range, and economy of motion.
1. Introduction
The paper targets subjective, time-consuming surgical skill assessment by detecting and localizing tools in laparoscopic videos, then analyzing their movements to assess surgical quality.
- Five billion people lack access to safe surgical care, while inadequate training and feedback contribute to preventable surgical complications.
- Manual assessment requires expert supervision and is subjective and time consuming, motivating real-time automated surgical video analysis.
- Spatial tool bounds are needed alongside tool presence to analyze movement range and economy of motion.
- The authors introduce an R-CNN approach and m2cai16-tool-locations dataset for spatial tool detection in real-world laparoscopic videos.
- The method significantly outperforms previous work on tool presence detection while effectively detecting spatial bounds.
- Tool usage patterns, movement range, and economy of motion are used to assess surgical quality, with surgeon reviews substantiating the findings.
2. Related Work
Prior work established frame-level tool detection and related surgical video-analysis tasks, but this paper extends the focus to spatial localization and comprehensive quality assessment in real-world operations.
- Earlier approaches used RFID tags, segmentation, contour processing, 3D modeling, and Viola-Jones detection for surgical tool analysis.
- Deep learning methods achieved state-of-the-art performance on surgical tool presence detection and phase recognition through the M2CAI benchmarks.
- Existing research also addressed tool localization in robot-assisted training tasks and components such as surgical phase and activity recognition.
- This work uses R-CNNs to detect tool bounds in real-world laparoscopic cholecystectomies and analyzes unedited, full-length operations rather than only short or simulated segments.
3. Dataset
The paper introduces m2cai16-tool-locations, a dataset adding spatial tool annotations to an existing frame-level presence dataset for real-world laparoscopic surgery.
- Existing m2cai16-tool and Cholec80 datasets primarily provide binary frame-level annotations indicating tool presence.
- Because no suitable real-world laparoscopic dataset with spatial tool bounds was known, the authors collected and introduced m2cai16-tool-locations.
- m2cai16-tool contains 23,000 downsampled frames from 15 cholecystectomy videos, labeled for presence or absence of seven surgical tools.
- The new dataset labels 2,532 frames with surgeon-supervised spatial bounding boxes and uses 50%, 30%, and 20% training, validation, and test splits.
- Frames with one, two, and three tools were annotated to increase the number of labeled instances per tool class.
4. Approach
The approach uses Faster R-CNN to detect and localize surgical tools in video frames, producing bounding boxes that support downstream movement analysis. Its architecture combines VGG-16 features, region proposals, pooled regions, and classification with bounding-box refinement.
- Spatial tool detection: Faster R-CNN takes a surgical-video frame as input and outputs spatial bounding boxes for detected surgical tools.The model targets seven surgical instruments and supports tracking tool usage and movement.
- Network architecture: VGG-16 extracts visual features, while a region proposal network generates likely object regions using shared convolutional features.The RPN produces proposals at sliding-window positions for subsequent region processing.
- Training objective: The RPN loss combines binary objectness classification with bounding-box regression, normalized and weighted by λ.Anchors receive objectness labels based on overlap with ground-truth boxes, while regression predicts box coordinates.
- Network architecture: Features pooled over proposed regions are passed to classification and bounding-box refinement networks.The final networks identify tools and refine their spatial coordinates.
- Training procedure: The network is pre-trained on ImageNet and then fine-tuned on the smaller m2cai16-tool-locations surgical-tool dataset.This transfers general visual features before adapting the model to surgical instruments.
- Training procedure: Training uses stochastic gradient descent, 40K iterations, mini-batches of 50, horizontal-flip augmentation, and a learning rate reduced every 10K iterations.Deployment processing speed is 5 fps, described as real-time surgical tool detection.
5. Results
The model performs spatial tool detection in laparoscopic videos and converts those detections into stronger frame-level presence detection. Tool trajectories, usage timelines, heat maps, and distance traveled support qualitative and quantitative assessment of surgical performance.
- 5.1. Spatial detection: 63.1 overall mAP indicates strong spatial detection performance on m2cai16-tool-locations.A detection is correct when its IoU with a ground-truth box is 0.5 or higher.
- 5.1. Spatial detection: The model detects, classifies, and localizes instruments across varying positions, angles, and partial occlusions, although false positives and missed tools occur.Reported errors include confusing clipper with scissors, mistaking anatomy or tool poles for irrigators, and missing a grasper.
- 5.2. Frame-level presence detection: 28% improvement in mAP, from 63.8 to 81.8, was achieved for frame-level presence detection using 2,500 of 23,000 frames.The approach significantly outperformed winners of the 2016 M2CAI Tool Presence Detection Challenge and surpassed EndoNet.
- 5.3. Assessment of Surgical Performance: Spatial detections enabled analysis of tool usage patterns, movement range, economy of motion, and surgical quality in full-length operations.The analyses used timelines, heat maps, trajectories, and total tool distance traveled, with surgeon review supporting the findings.
- 5.3. Assessment of Surgical Performance: Testing video 2 showed the most technically excellent surgery through focused instrument use, minimal switching, deft trajectories, and greatest economy of motion.Trajectory maps and distance measurements linked these patterns to bimanual dexterity, efficiency, and tool handling.
6. Conclusion
The paper presents an R-CNN approach and dataset for real-time spatial tool detection in laparoscopic videos, supporting automated surgical assessment. Future work targets additional video-derived measures and automated GOALS ratings.
- 5fps real-time processing supports automated analysis of spatial tool detections in laparoscopic surgical videos.
- The method extracts tool usage patterns, movement range, and economy of motion that correlate with experienced surgeons’ independent assessments.
- Future work will extend automatically extracted information to motion smoothness, tissue damage, repeated movements, and procedure phase.
- The authors also plan to develop an automated GOALS rating system.