Source-linked AI summary
Detection and Localization of Robotic Tools in Robot-Assisted Surgery Videos Using Deep Neural Networks for Region Proposal and Detection
Duygu Sarikaya, Jason J. Corso, Khurshid A. Guru
TL;DR
The paper addresses the lack of deep-learning approaches for detecting and localizing surgical tools in RAS videos, a step needed for modeling surgeon motion. It proposes a strictly vision-based multimodal CNN architecture with an RPN and reports 91% AP at about 0.1 seconds per test frame, alongside the ATLAS Dione dataset.
Problem
Deep neural networks had not yet been used, to the authors’ knowledge, for surgical-tool detection and localization in RAS video understanding, despite the importance of tool motion for modeling surgeon gestures and skills.
Method
The paper combines an RPN with a two-stream CNN that processes RGB frames and optical-flow representations, then fuses features for object detection and localization.
Results
91% Average Precision (AP) and 0.1 seconds per test frame were achieved, and the multimodal architecture improved on the architecture proposed by Girschick et al.
Takeaways & Limitations
The study introduces ATLAS Dione, a dataset of ten subjects performing six surgical tasks with per-frame robotic-tool annotations, and supports multimodal cues for RAS tool localization.
Abstract
from arXiv · showhide
Video understanding of robot-assisted surgery (RAS) videos is an active research area. Modeling the gestures and skill level of surgeons presents an interesting problem. The insights drawn may be applied in effective skill acquisition, objective skill assessment, real-time feedback, and human-robot collaborative surgeries. We propose a solution to the tool detection and localization open problem in RAS video understanding, using a strictly computer vision approach and the recent advances of deep learning. We propose an architecture using multimodal convolutional neural networks for fast detection and localization of tools in RAS videos. To our knowledge, this approach will be the first to incorporate deep neural networks for tool detection and localization in RAS videos. Our architecture applies a Region Proposal Network (RPN), and a multi-modal two stream convolutional network for object detection, to jointly predict objectness and localization on a fusion of image and temporal motion cues. Our results with an Average Precision (AP) of 91% and a mean computation time of 0.1 seconds per test frame detection indicate that our study is superior to conventionally used methods for medical imaging while also emphasizing the benefits of using RPN for precision and efficiency. We also introduce a new dataset, ATLAS Dione, for RAS video understanding. Our dataset provides video data of ten surgeons from Roswell Park Cancer Institute (RPCI) (Buffalo, NY) performing six different surgical tasks on the daVinci Surgical System (dVSS R ) with annotations of robotic tools per frame.
I. INTRODUCTION
The paper frames robotic-tool detection and localization as a necessary first step for modeling surgeon motion in RAS videos, addressing limitations in training and assessment with deep learning.
- Motivation: RAS training has a steep learning curve, limited haptic sensation, and traditionally relies on observational learning and subjective expert evaluation.Surgeons reportedly become comfortable only after procedures on 12–18 patients.
- Motivation: Automated, universally accepted metrics could support quantitative skill assessment, personalized training, automated feedback, and human-robot collaborative surgery.
- Research problem: Tool detection and precise localization are treated as prerequisites for tracking tool motion and modeling surgeons’ gestures and skills.
- Contribution: The study proposes deep convolutional neural networks for fast surgical-tool detection and localization, addressing an area the authors identify as underused in medical imaging.
- Contribution: The architecture combines an RPN with multimodal CNN processing of RGB frames and optical-flow representations before fusing stream features for classification.
II. RELATED WORK
Earlier approaches used markers, initialization, additional sensing, or sliding-window vision methods, whereas this paper presents a strictly vision-based multimodal CNN system with RPN-based proposals.
- Earlier approaches: Marker-based approaches simplify tool detection through color segmentation or thresholding but require added markers or instruments that raise manufacturing, biocompatibility, and surgical-setting concerns.
- Earlier approaches: Some tracking methods require per-video initialization, including an instrument pose or bounding box in the first frame, while others fuse region constraints with additional data.
- Earlier approaches: A strictly vision-based sliding-window classifier evaluates tool-part features across image locations and multiple scales, with early stopping used to reduce computation time.
- Proposed approach: The proposed system uses separate CNN streams for RGB frames and optical-flow cues, with an RPN generating proposals from RGB features and fused stream features supporting detection.
- Proposed approach: The method requires no initialization and is intended to detect and localize tools on new video data without retraining, potentially initializing tracking algorithms automatically.
III. DATASET
ATLAS Dione addresses limited public RAS datasets by providing a more challenging, annotated collection spanning varied tools, tasks, and surgical skill levels.
- RAS videos challenge understanding because endoscopic views are limited, tools vary in pose and articulation, and tissue or other objects can occlude them.
- ATLAS Dione was created because existing public RAS datasets lacked tool annotations, while JIGSAWS also lacked artifacts, camera movement, zoom, and broad free movement.
- The dataset contains videos of ten subjects performing six surgical tasks on the dVSS, with task difficulty and complexity varying across activities.
- Tool annotations identify both left and right tools with exact bounding boxes, supporting detection and localization research.
- Tasks are grouped into basic, intermediate, and advanced robotic skills according to their difficulty and complexity.Basic tasks include ball placement, suture pass, and ring peg transfer; intermediate tasks include knot tying; advanced tasks involve urethrovesical anastomosis.
A. Tool Annotation
The dataset uses expert-guided manual annotation to mark robotic tools frame by frame and distributes frames with standardized XML metadata.
- Both left and right tools are manually annotated with bounding boxes guided by an expert RAS surgeon.The annotation workflow was adapted from tools provided by the Caltech Pedestrian Detection Benchmark.
- Each RAS video frame is supplied as an 854x480 JPEG image, with annotations stored in VOC-format XML templates.
B. Subject Demographics
The dataset includes ten RPCI subjects spanning training and practicing surgeons, grouped by expertise under the Dreyfuss model, and is released for further research.
- Ten surgeons from RPCI performed the specified tasks on the dVSS.
- The participants included two residents, three fellows, and five practicing robotic surgeons.
- Participants varied in experience, including surgeons with more than 10 years, two to five years, and ongoing training.Two subjects had performed over 500 robot-assisted procedures.
- Subjects were assigned to beginner, combined competent and proficient, and expert groups based on the Dreyfuss model.
- The dataset with tool annotations is available for download to encourage further research in RAS video understanding.
IV. METHOD
The method combines RGB appearance and optical-flow motion in two CNN streams, uses an RPN for proposals, and jointly trains proposal and detection networks.
- The architecture processes RGB video frames and RGB optical-flow representations through separate CNN streams before fusing their features.
- A Region Proposal Network generates object proposals from convolutional features of the RGB input.Each proposal receives an objectness score.
- Region proposal boxes and both-modal convolutional features are passed to ROI pooling for each stream.
- Features from the two streams are concatenated and fused in a fully connected layer before the loss layers.
- The region proposal and multimodal object detection networks are trained jointly using approximate joint training.
A. Loss function for Learning
The model uses a multitask objective that jointly learns object classification and bounding-box regression, with regression applied only to positive anchors and balanced against classification loss.
- A. Loss function for Learning: The multitask objective jointly trains classification and bounding-box regression across the full network.The loss follows a Faster R-CNN-style multi-task formulation.
- A. Loss function for Learning: The RPN generates region proposals and assigns each an objectness score indicating whether it contains a tool.Sliding windows over convolutional feature maps produce fixed-dimensional features for proposal prediction.
- A. Loss function for Learning: The classification layer predicts whether each anchor represents an object, while the regression layer predicts bounding-box offsets.The RPN produces 2k classification scores and 4k regression values for k anchors.
- A. Loss function for Learning: Regression loss is activated only for positive anchors and is normalized separately from classification loss.The two losses use Ncls and Nreg, with balancing weight λ set to 10.
- A. Loss function for Learning: Bounding-box regression refines proposal boundaries using anchors covering different scales and aspect ratios.Each proposal is associated with k learned bounding-box regressors.
B. Optimization
Optimization trains the RPN end-to-end with stochastic gradient descent, combining pretrained initialization for RGB features with Gaussian initialization for new optical-flow layers.
- B. Optimization: The RPN is trained end-to-end using back-propagation and stochastic gradient descent.Training samples images and anchors while maintaining a 1:1 ratio of positive and negative anchors.
- B. Optimization: Pretrained ImageNet weights initialize existing layers, while newly introduced layers use zero-mean Gaussian initialization.The optical-flow stream lacks a suitable pretrained dataset model for its new layers.
- B. Optimization: The optical-flow initialization caused convergence fluctuations, although the final model improved accuracy.The RGB pretrained weights helped address overfitting and reduce convergence fluctuations.
- B. Optimization: 70k iterations were selected because they produced results comparable to 120k iterations while requiring less training time.The learning rate was 0.001, momentum 0.9, and weight decay 0.0005.
C. Detection at test time
At test time, the fully convolutional RPN processes the entire image, suppresses redundant proposals, and sends the highest-ranked proposals for detection.
- C. Detection at test time: The fully convolutional RPN is applied to the entire test image to generate region proposals.Non-maximum suppression uses a 0.7 threshold based on proposal objectness scores.
- C. Detection at test time: The detector uses a fixed number of top-ranked proposals after non-maximum suppression.The ranking is determined by proposal objectness scores.
V. EXPERIMENTS AND EVALUATION
The evaluation uses randomized ten-fold splits of 99 ATLAS Dione videos, keeping frames from each video exclusively in training or testing. The multimodal RPN-based detector achieves the strongest reported accuracy with fast per-frame detection.
- Experimental setup: 99 videos were evaluated with ten randomized folds, using 90 videos for training and 9 for testing in each fold.Frames from the same video were never included in both training and testing.
- Results: 0.103 seconds was required for detection per test frame by the multimodal RPN+Fast R-CNN method, excluding already computed optical-flow images.Training required 7.22 hours for 70k iterations.
- Results: 91% (90.65%) Average Precision was achieved by the multimodal RPN+Fast R-CNN detector.The dataset contains one object class: the robotic tool, alongside background.
- Results: 20% Average Precision was obtained by EdgeBoxes+Fast R-CNN, requiring 0.134 seconds for detection plus 2 seconds for region proposals.Selective Search was considered but did not produce drastically superior results compared with EdgeBoxes on this dataset.
- Results: 76% Average Precision was reported for DPM, increasing to 83% with bounding-box regression, with 2.3 seconds detection time.
VI. CONCLUSION
The paper presents an end-to-end deep-learning approach for fast robotic-tool detection and localization in RAS videos, combining region proposals with image and temporal-motion cues. It also introduces ATLAS Dione and reports 91% AP with approximately 0.1 seconds per test frame.
- Conclusion: The architecture jointly predicts objectness and localization using an RPN and multimodal convolutional network fused from image and temporal-motion cues.
- Conclusion: ATLAS Dione contains videos of ten subjects performing six surgical tasks on the daVinci Surgical System with per-frame robotic-tool annotations.
- Conclusion: 91% Average Precision and 0.1 seconds mean computation time per test frame were reported for the proposed approach.
- Conclusion: The multimodal architecture repeatedly and consistently scored higher accuracy than the compared single-modality architecture across experiment sets, although the improvement was small.
- Conclusion: The authors state that the study and ATLAS Dione dataset may form a benchmark for future RAS video-understanding studies.