Source-linked AI summary
TeCNO: Surgical Phase Recognition with Multi-Stage Temporal Convolutional Networks
Tobias Czempiel, Magdalini Paschali, Matthias Keicher, Walter Simson, Hubertus Feussner, Seong Tae Kim, Nassir Navab
TL;DR
Surgical phase recognition is challenging but important for patient safety and intra-operative decision support, while existing recurrent approaches have limited temporal coverage and sequential inference. The paper proposes TeCNO, a causal, dilated multi-stage TCN that refines predictions online, and evaluates it on two laparoscopic video datasets. TeCNO outperforms multiple LSTM-based approaches and provides fast online inference with consistent predictions across phases and ambiguous transitions.
Problem
Surgical phase recognition from intra-operative videos is challenging because of variability in patient anatomy and surgeon style, limited video material, and difficulty capturing long-term temporal patterns.
Method
TeCNO combines frame-wise visual features with a two-stage causal, dilated temporal convolutional network for hierarchical prediction refinement using only current and preceding frames.
Results
TeCNO outperforms various LSTM-based approaches across two datasets and achieves online, fast inference on whole video sequences.
Takeaways & Limitations
Causal multi-stage temporal refinement improves prediction consistency within phases and during ambiguous interphase transitions while supporting online surgical phase recognition.
Takeaways & Limitations
Three refinement stages may overfit the training set when only a limited amount of data is available.
Abstract
from arXiv · showhide
Automatic surgical phase recognition is a challenging and crucial task with the potential to improve patient safety and become an integral part of intra-operative decision-support systems. In this paper, we propose, for the first time in workflow analysis, a Multi-Stage Temporal Convolutional Network (MS-TCN) that performs hierarchical prediction refinement for surgical phase recognition. Causal, dilated convolutions allow for a large receptive field and online inference with smooth predictions even during ambiguous transitions. Our method is thoroughly evaluated on two datasets of laparoscopic cholecystectomy videos with and without the use of additional surgical tool information. Outperforming various state-of-the-art LSTM approaches, we verify the suitability of the proposed causal MS-TCN for surgical phase recognition.
1 Introduction
Surgical phase recognition remains challenging because videos vary across patient anatomy and surgeon style, while existing temporal approaches struggle with long-term patterns. TeCNO addresses this with causal, dilated TCNs designed for accurate, fast online recognition and evaluates the approach on two laparoscopic video datasets.
- Surgical workflow analysis can increase patient safety, reduce surgical errors, and optimize operating-room communication.
- Phase recognition from intra-operative videos remains challenging because patient anatomy and surgeon style vary, while video material is limited in availability and quality.
- Sliding-window RNN approaches have difficulty capturing long-term temporal patterns in surgical videos.
- TeCNO uses dilated Temporal Convolutional Networks to capture full temporal resolution with a large receptive field, fewer parameters, and faster training and inference.
- The paper introduces causal, dilated MS-TCNs for refined online surgical phase recognition and evaluates them on two laparoscopic video datasets.
2 Methodology
TeCNO combines frame-wise visual features with causal, dilated multi-stage temporal convolutions that expand temporal context and progressively refine surgical phase predictions.
- 2.1 Feature Extraction Backbone: TeCNO uses ResNet50 frame features followed by a 2-stage causal TCN that reasons over preceding frames.The feature extractor can operate with phase labels alone or jointly with tool labels when available.
- 2.2 Temporal Convolutional Networks: The temporal model uses only convolutional layers, avoiding pooling that reduces temporal resolution and fully connected layers requiring fixed input dimensions.
- 2.2 Temporal Convolutional Networks: Causal convolutions ensure each prediction uses the current and previous frames, enabling online intra-operative deployment.Unlike acausal convolutions, they do not depend on future frames.
- 2.2 Temporal Convolutional Networks: Exponentially increasing dilation expands the temporal receptive field without pooling, reducing computational cost relative to larger kernels or deeper networks.Three consecutive dilated layers provide an 8-time-step receptive field.
- 2.2 Temporal Convolutional Networks: Additional MS-TCN stages receive preceding-stage outputs and use independent losses to progressively refine the final predictions.Weighted cross-entropy with median-frequency class balancing is applied after each stage.
3 Experimental Setup
The experiments evaluate TeCNO on two laparoscopic cholecystectomy datasets using repeated runs, surgical-phase metrics, ablations, and comparisons with LSTM-based baselines.
- Datasets: The evaluation uses Cholec80 and Cholec51, both containing seven surgical phases, with tool annotations available only in Cholec80.Cholec80 contributes approximately 92,000 frames after 5-fps subsampling; Cholec51 contains 51 videos split into 25 training, 8 validation, and 18 test videos.
- Model Training: Experiments were repeated 5 times with random initialization, and test results were taken from models performing best on validation data.
- Evaluation Metrics: Performance is measured using Accuracy, Precision, and Recall, with Accuracy computed across the video and Precision and Recall evaluated per phase.
- Ablative Testing: Ablations vary the CNN feature extractor between AlexNet and ResNet50 and vary the number of TCN refinement stages.
- Baseline Comparison: Baseline comparisons include PhaseLSTM, EndoLSTM, MTRCNet, ResNetLSTM, and TeCNO, while EndoLSTM and MTRCNet cannot be evaluated on Cholec51 because they require tool labels.
4 Results
Results show that temporal refinement improves recognition across feature extractors, while TeCNO outperforms LSTM-based alternatives and produces consistent predictions during difficult transitions. Three refinement stages may overfit the limited training data.
- Effect of Feature Extractor Architecture: ResNet50 outperforms AlexNet across the board, with accuracy improvements ranging from 2% to 8%.ResNet50 also improves precision by up to 7% and recall by up to 6%.
- Effect of TCN and Number of Stages: Adding one TCN stage increases accuracy by 10% for AlexNet and 6% for ResNet50.The result indicates that temporal refinement improves both CNN feature extractors.
- Effect of TCN and Number of Stages: Stage 2 outperforms Stage 3 by 1% in accuracy for AlexNet and 2% for ResNet50.The authors suggest that three refinement stages may overfit the training set because of the limited data.
- Comparative Methods: TeCNO improves accuracy over ResNetLSTM by 1–2% and precision and recall by 6%–10% across both datasets.TeCNO also outperforms MTRCNet by 6% in accuracy.
- Phase Recognition Consistency: TeCNO produces smooth predictions across phases and ambiguous transitions, including accurate recognition of shorter phases such as P5 and P7.Its performance does not deteriorate when videos are missing phase P1.
5 Conclusion
TeCNO uses causal, dilated multi-stage temporal convolutions to improve surgical phase recognition across two datasets. The method supports fast online inference and more consistent predictions within phases and during ambiguous transitions.
- 5 Conclusion: TeCNO increases performance over LSTM-based approaches across two datasets of laparoscopic videos.The method uses full temporal resolution and a large receptive field for recognition.
- 5 Conclusion: Causal, dilated convolutions enable online and fast inference on whole video sequences.The multi-stage refinement process improves consistency within phases and across ambiguous interphase transitions.