Source-linked AI summary
OperA: Attention-Regularized Transformers for Surgical Phase Recognition
Tobias Czempiel, Magdalini Paschali, Daniel Ostler, Seong Tae Kim, Benjamin Busam, Nassir Navab
TL;DR
Surgical phase recognition is challenging amid variation in anatomy and surgeon style and limited training data. OperA combines a transformer with attention regularization to focus on reliable frame features and identify characteristic frames. It is reported to outperform baselines on two laparoscopic cholecystectomy datasets.
Problem
Anatomy and surgeon-style variability and limited training-data availability and quality challenge automatic surgical phase recognition.
Source: §1 · p. 1Method
OperA applies attention regularization to a transformer so it focuses on reliable CNN features and uses attention weights to identify characteristic frames.
Source: §1.1 · p. 2, §2 · p. 3, §2.2 · p. 4Results
OperA outperformed a variety of baselines on two laparoscopic cholecystectomy video datasets and extracted characteristic high-attention frames.
Source: §5 · p. 8Takeaways & Limitations
The extracted high-attention frames are presented as potentially useful for surgical video summarization.
Source: Highest and Lowest Attention Frames · p. 8
Abstract
from arXiv · showhide
In this paper we introduce OperA, a transformer-based model that accurately predicts surgical phases from long video sequences. A novel attention regularization loss encourages the model to focus on high-quality frames during training. Moreover, the attention weights are utilized to identify characteristic high attention frames for each surgical phase, which could further be used for surgery summarization. OperA is thoroughly evaluated on two datasets of laparoscopic cholecystectomy videos, outperforming various state-of-the-art temporal refinement approaches.
1 Introduction
Surgical phase recognition matters for workflow analysis, but anatomy and surgeon variability and limited training data challenge automated systems. OperA introduces transformer-based online phase prediction and uses attention regularization and attention-derived frames as contributions.
1 Introduction
- Anatomy and surgeon-style variability, together with limited availability and quality of training data, challenge automatic surgical phase recognition.The paper frames phase recognition as an essential component of surgical workflow analysis, with potential relevance to assistance, cognitive guidance, and training.Source: §1 · p. 1
- Self-attention models temporal relationships across long sequences without the forgetting described for LSTMs, and its weights can be visualized for potential model insights.The paper notes that the interpretation afforded by attention remains debated and may depend on the task's notion of explanation.Source: §1 · p. 2, §1.1 · p. 2
1.1 Related Work
- Earlier surgical phase systems used CNN features with LSTM or TCN refinement, while transformers had been explored in surgical data sciences for tool classification.The related-work discussion cites the tool-classification work as.Source: §1.1 · p. 2
- OperA introduces transformer-based online surgical phase prediction for laparoscopic operations, with attention regularization and characteristic-frame extraction among its contributions.The authors evaluate the approach on two challenging surgical video datasets.Source: §1.1 · p. 2
2 Methodology
OperA applies a causal transformer to CNN frame features and regularizes its attention using the backbone's prediction error, aiming to emphasize reliable features. Normalized frame-wise attention also supports selecting frames for inspection.
2 Methodology
- A ResNet-50 CNN extracts per-frame features and class probabilities, and multiple self-attention layers process those features.The backbone is trained frame-wise for phase recognition and, when available, surgical tool detection; it is not sequentially modeled.Source: §2 · p. 3
2.1 Sequential Transformer Network
- The 11-layer transformer uses causal masking to prevent future-frame information from entering current predictions, enabling real-time phase prediction.Each layer includes linear and scaled dot-product attention, layer normalization, and residual connections; a final linear layer and softmax produce frame-wise class probabilities.Source: §2.1 · p. 4
2.2 Normalized Frame-Wise Attention
- Normalized frame-wise attention divides each frame's total attention by how often it is considered, compensating for the causal mask's position-dependent contributions.Without this normalization, the first frame can contribute T times while the last is considered once.Source: §2.2 · p. 4
2.3 Attention Regularization
- The regularizer weights each frame's CNN cross-entropy error by normalized attention, penalizing high attention on low-confidence features while keeping the backbone frozen.It is applied to the first attention layer, and training minimizes L = Lc + λ · Lreg; summed layer-wise attention is also used to extract highest- and lowest-attention frames.Source: §2.3 · pp. 4–5
3 Experimental Setup
OperA is evaluated for surgical phase recognition on Cholec80 and MitiSW, two laparoscopic cholecystectomy datasets. The setup uses video-level accuracy and F1 across five splits, with ablations and baseline comparisons.
Datasets
- Cholec80 contains 80 videos annotated at 1 fps for seven phases and seven tool labels; 20 videos are test cases, with 60 split into training and validation.The videos were subsampled to 1 fps for this work.Source: Datasets · p. 5
- MitiSW contains 85 videos with the seven Cholec80 phases plus Pre-preparation; 20 videos are used for testing and the remaining 65 for training and validation.MitiSW has no tool-presence annotations, and experiments use five-fold cross-validation.Source: Datasets · p. 5
Model Training
- OperA is trained with Adam for 30 epochs, and the reported test model is selected by validation performance for each fold.The initial learning rate is 1e-5, and batch size equals the length of each video.Source: Model Training · p. 5
Evaluation Metrics and Baselines
- Evaluation reports video-level accuracy and F1 averaged over five splits, with ablations of attention-layer count and regularization alongside baseline comparisons.Table 1 describes ablation results for 6 and 11 transformer layers, with and without attention regularization, reporting average metrics and standard deviations.Source: Evaluation Metrics and Baselines · p. 5, Table 1 · p. 6
4 Results and Discussion
Attention regularization slightly improves OperA’s performance across both datasets and layer settings, while its attention maps and selected frames align with more reliable and phase-descriptive video content.
Effect of Layers and Regularization
- Attention regularization improves Accuracy and F1-score by approximately 1% for both datasets and both tested layer counts.The 11-layer model also slightly improves on the 6-layer model; 11 layers was the maximum that fit in VRAM.Source: Effect of Layers and Regularization · p. 6
Predictions and Attention Values
- For video 66, OperA predictions are smoother than CNN predictions, and high-attention frames usually coincide with correct predictions while low-attention frames occur where CNN predictions are wrong.In the frame-wise attention display, yellow indicates higher attention and blue lower attention; triangles mark high-attention frames and inverted triangles mark low-attention frames.Source: Predictions and Attention Values · p. 7, Figure 2 · p. 7
Highest and Lowest Attention Frames
- Low-attention frames are generally less phase-descriptive, and without regularization, characteristic tool-containing frames can receive minimum attention.This contrast is shown for the models with and without attention regularization.Source: Predictions and Attention Values · p. 7, Figure 3 · p. 8
- With regularization, high-attention frames are more diverse and representative of their phase, with tools appearing in every phase except Preparation despite no tool-information training.Without regularization, some high-attention frames are not phase-descriptive and are similar across Cleaning Coagulation and Gallbladder Retraction; the authors note potential for surgery video summarization.Source: Highest and Lowest Attention Frames · p. 8
5 Conclusion
OperA predicts cholecystectomy phases and extracts characteristic high-attention frames; it outperforms baselines on two laparoscopic video datasets.
5 Conclusion
- OperA accurately predicts cholecystectomy phases, outperforms baselines on two laparoscopic video datasets, and extracts characteristic high-attention frames.Future work will apply the method to other laparoscopic procedures and explore surgical video summarization.Source: §5 · p. 8
6 Supplementary
The supplementary material covers layer and regularization ablations, baseline comparisons, and additional attention-frame visualizations.
6 Supplementary
- Ablation results report average metrics and standard deviations over five folds for 6- and 11-layer models with and without attention regularization.The listed metrics are Accuracy, F1, Precision, and Recall.Source: §6 · p. 11
- Baseline comparisons cover Cholec80 and MitiSW, with MTRCNet-CL unavailable for MitiSW because it requires tool information.The listed metrics are Accuracy, F1, Precision, and Recall.Source: §6 · p. 11
- Two additional Cholec80 videos visualize maximum- and minimum-attention frames by phase for models with and without regularization.Blue and red boxes mark frames receiving low attention without regularization despite being descriptive of their phase.Source: Figure 4 · p. 12