Source-linked AI summary
DeepPhase: Surgical Phase Recognition in CATARACTS Videos
Odysseas Zisimopoulos, Evangello Flouty, Imanol Luengo, Petros Giataganas, Jean Nehme, Andre Chow, Danail Stoyanov
TL;DR
Automated video-based workflow recognition could support surgical phase segmentation and related intraoperative tasks, while prior hand-crafted visual features limited robustness and transfer across procedures. DeepPhase detects cataract-surgery tools with a residual network and feeds tool information to recurrent phase classifiers, achieving 97.69% AUC on the CATARACTS test set and 78.28% phase accuracy.
Problem
Automated workflow recognition is sought for phase and step segmentation, but hand-crafted visual features limited robustness and transfer to other procedures remained limited.
Source: §1 · pp. 1–2Method
A ResNet recognizes tools in cataract video frames, and recurrent networks classify phases from sequences of tool presence or learned tool features.
Source: §1 · p. 2, §2.3 · p. 4Results
97.69% AUC on the CATARACTS test set and 78.28% phase accuracy were reported for the tool-recognition and phase-recognition models, respectively.
Source: §4 · p. 7Takeaways & Limitations
The results show potential for automating workflow recognition in cataract videos.
Source: §4 · p. 7Scarcity of some phase classes prevented the model from learning all surgical phases equally well.
Source: §4 · p. 7
Abstract
from arXiv · showhide
Automated surgical workflow analysis and understanding can assist surgeons to standardize procedures and enhance post-surgical assessment and indexing, as well as, interventional monitoring. Computer-assisted interventional (CAI) systems based on video can perform workflow estimation through surgical instruments' recognition while linking them to an ontology of procedural phases. In this work, we adopt a deep learning paradigm to detect surgical instruments in cataract surgery videos which in turn feed a surgical phase inference recurrent network that encodes temporal aspects of phase steps within the phase classification. Our models present comparable to state-of-the-art results for surgical tool detection and phase recognition with accuracies of 99 and 78% respectively.
1 Introduction
The introduction motivates automated surgical workflow recognition as a way to support operating-room teamwork and downstream workflow tasks. It positions the work as a deep-learning system for cataract surgery that recognizes tools and uses their features for phase classification.
1 Introduction
- Automated workflow analysis could support teamwork, phase/step segmentation, next-step preparation, early warnings, reporting, scheduling, and video indexing without adding substantial clinician time.The passage also connects these applications to reducing surgical errors and improving resource usage.Source: §1 · p. 1
- Hand-crafted visual features were considered less robust, while transfer and resilience across methods and procedures remained limited in prior workflow systems.Earlier approaches used hands or tool presence, random forests, conditional random fields, and explicit visual features; hand-crafted features limited robustness.Source: §1 · pp. 1–2
- The authors propose a cataract-surgery workflow system in which ResNet tool recognition produces image features for an RNN that classifies phase sequences.Phase annotations were created by hand-labeling the CATARACTS dataset, and the authors report results near state of the art for tool and phase recognition.Source: §1 · p. 2
2 Materials and Methods
The methods use CATARACTS videos and frame-level tool and phase annotations, then connect CNN-based tool recognition to recurrent phase classification. The tool classifier recognizes 21 classes, while the phase models use temporal sequences of binary tool outputs or learned tool features.
2.1 Augmented CATARACT Dataset
- Annotations cover 14 phases across all 50 CATARACTS videos, with 25 additional videos reserved to test phase-recognition generalization.The dataset also includes 25 training videos; tool-presence labels indicate which tools touch the eyeball, and the training videos were split into train, validation, and hold-out test sets.Source: §2.1 · p. 3
2.2 Tool recognition with CNNs
- ResNet-152 performs multi-label recognition of 21 tool classes using a fully connected sigmoid output and shared weights across class-specific tasks.Its 50 residual blocks each contain three convolutional layers, batch normalization, and ReLU activation; the CNN produces tool features for the downstream RNN.Source: §2.2 · pp. 3–4, Figure 2 · p. 4
2.3 Phase recognition with RNNs
- The phase-recognition models use recurrent networks on sequences of binary tool-presence outputs or CNN tool features to capture temporal information.The study trains an LSTM and a two-layer GRU for 14-class prediction; the GRU has 128 nodes per layer, and the LSTM has a 256-node hidden layer.Source: §2.3 · pp. 4–5
3 Experimental results
Tool recognition achieved a 97.69% AUC on the CATARACTS challenge test set, while phase recognition performance varied by recurrent model and input type. Evaluation used tool- and phase-specific metrics, and phase results were qualified by class imbalance.
3.1 Evaluation metrics
- The tool-presence evaluation used ROC AUC, subset accuracy, and hamming accuracy, while phase recognition used per-frame accuracy, mean class precision and recall, and phase-class F1-score.Subset accuracy counts only instances whose binary predictions exactly match ground truth; hamming accuracy is defined over ground-truth and prediction vectors, with N samples and C classes.Source: §3.1 · p. 5
3.2 Tool recognition
- 97.69% AUC on the CATARACTS challenge test set was close to the winning AUC of 99.71%.On the hold-out test set, ResNet-152 achieved 99.07% hamming accuracy and 99.59% AUC; video quality and tool location relative to the eyeball remained challenges.Source: §3.2 · p. 5
3.3 Phase recognition
- The phase experiments compared LSTM and GRU models using binary or tool-feature inputs, with 100-frame sequences processed in sliding windows.Table 1 reports accuracy and average class F1-score across train, validation, and hold-out test sets from 25 training videos, plus a separate test on 25 videos.Source: §3.3 · p. 6, Table 1 · p. 7
- Phase-class imbalance limited learning: phases 3 and 4 appeared in only two videos and were not adequately learned.The authors note these phases appeared in validation but not the hold-out test set, affecting the validation/test performance discrepancy.Source: §3.3 · p. 6
- 78.28% accuracy was achieved by the LSTM with tool-feature inputs on the CATARACTS test set, compared with 68.75% for binary inputs.The LSTM trained on tool features was the best phase-recognition model; those features improved LSTM results across sets but did not improve GRU test performance.Source: §3.3 · pp. 6–7
4 Discussion and Conclusion
The proposed framework combines tool presence from video frames with recurrent models for cataract-surgery phase recognition. It reached 97.69% AUC on the CATARACTS test set for tool recognition and 78.28% phase accuracy, while scarcity of some phase classes limited balanced learning.
4 Discussion and Conclusion
- 97.69% AUC on the CATARACTS test set and 78.28% phase accuracy were reported for tool and phase recognition, respectively.The authors describe tool recognition as state-of-the-art performance and phase recognition as showing potential for automating workflow recognition.Source: §4 · p. 7
- The scarcity of some phase classes prohibited learning all surgical phases equally well.The authors suggest data augmentation, weighted loss functions, or stratified sampling as future approaches.Source: §4 · p. 7
- An end-to-end approach using bidirectional recurrent networks or temporal convolutional networks is proposed for future work.These architectures are proposed as alternatives for further experimentation, not reported results.Source: §4 · p. 7