Source-linked AI summary
Trans-SVNet: Accurate Phase Recognition from Surgical Videos via Hybrid Embedding Aggregation Transformer
Xiaojie Gao, Yueming Jin, Yonghao Long, Qi Dou, Pheng-Ann Heng
TL;DR
Surgical phase recognition is difficult online because future information is unavailable and surgical videos contain ambiguous, blurred, high-dimensional visual data. Trans-SVNet combines spatial and temporal embeddings through hybrid Transformer aggregation, using spatial information to query temporal sequences. The method consistently outperforms compared state-of-the-art models while reaching real-time processing speed, including 91 fps.
Problem
Online surgical phase recognition must operate without future information despite ambiguous appearances, scene blur, and time-consuming high-dimensional video processing.
Method
Trans-SVNet extracts spatial and temporal embeddings and uses Transformer layers to fuse them by letting spatial embeddings query temporal embedding sequences.
Results
Trans-SVNet consistently outperforms compared state-of-the-art models and achieves a real-time processing speed of 91 fps.
Takeaways & Limitations
Hybrid embedding aggregation retrieves missing critical information with rarely additional cost while maintaining high processing speed for surgical phase recognition.
Abstract
from arXiv · showhide
Real-time surgical phase recognition is a fundamental task in modern operating rooms. Previous works tackle this task relying on architectures arranged in spatio-temporal order, however, the supportive benefits of intermediate spatial features are not considered. In this paper, we introduce, for the first time in surgical workflow analysis, Transformer to reconsider the ignored complementary effects of spatial and temporal features for accurate surgical phase recognition. Our hybrid embedding aggregation Transformer fuses cleverly designed spatial and temporal embeddings by allowing for active queries based on spatial information from temporal embedding sequences. More importantly, our framework processes the hybrid embeddings in parallel to achieve a high inference speed. Our method is thoroughly validated on two large surgical video datasets, i.e., Cholec80 and M2CAI16 Challenge datasets, and outperforms the state-of-the-art approaches at a processing speed of 91 fps.
1 Introduction
Surgical phase recognition supports operating-room monitoring and decision support but remains difficult online because videos contain ambiguous appearances, blur, and high-dimensional data. Trans-SVNet addresses this by reusing spatial features alongside temporal features within a Transformer-based aggregation framework.
- Online surgical phase recognition is challenging because future information cannot assist current decisions, while high-dimensional video processing remains time-consuming.
- Temporal relations are important for surgical video analysis, but predefined dependencies in statistical models have limited representation capacity.
- Transformers can relate sequence entries at different positions concurrently, helping preserve long-term clues while enabling parallel training and inference.
- Trans-SVNet reuses spatial features with temporal features and aggregates them through Transformer attention for surgical phase recognition.
- 91 fps processing speed accompanies Trans-SVNet's reported performance gains over compared methods on two surgical video datasets.
2 Method
Trans-SVNet represents surgical frames with spatial and temporal embeddings, then uses two Transformer layers to aggregate them for refined phase predictions. Spatial embeddings query temporally aggregated sequences, allowing complementary information to be fused without future frames.
- Video Embedding Extraction: The embedding model extracts spatial embeddings with ResNet50 and temporal embeddings with TCN from spatial representations.
- Video Embedding Extraction: ResNet50 is trained with phase labels to produce 2048-dimensional spatial embeddings from average-pooling outputs.
- Video Embedding Extraction: TCN processes reduced spatial embeddings without future information and uses dilated convolutions to obtain temporal embeddings with a receptive field spanning several minutes.
- Hybrid Embedding Aggregation: The aggregation model uses two Transformer layers to fuse hybrid embeddings and predicts phase probabilities with Softmax.
- Hybrid Embedding Aggregation: The first Transformer layer self-aggregates each entry in the temporal embedding sequence, while the second lets spatial embeddings query temporal features as keys and values.
3 Experiments
Trans-SVNet is evaluated on two surgical video datasets using standard phase-recognition metrics and comparisons with existing methods. Experiments show strong performance, robustness to temporal-sequence length, and benefits from its hybrid embedding architecture.
- Datasets and Metrics: Trans-SVNet is evaluated on Cholec80 and M2CAI16, using accuracy, precision, recall, and Jaccard index for phase recognition.Cholec80 contains 80 videos with 7 phases, while M2CAI16 contains 41 videos with 8 phases.
- Comparison with State-of-the-arts: Trans-SVNet outperforms seven compared methods, improving precision and Jaccard by 4% over TeCNO on Cholec80 with approximately 30k additional parameters.It also improves Jaccard by 6%-12% relative to multi-step learning methods with a simpler training procedure.
- Qualitative Comparison: Qualitative ribbons show that ResNet produces frequently jumped predictions, whereas Trans-SVNet yields more consistent and robust phase predictions by aggregating spatial and temporal embeddings.TeCNO smooths predictions but remains vulnerable to excessive reflection affecting spatial features.
- Ablation Study: Increasing the temporal embedding length improves all metrics, with similar performance for n ∈[20, 40]; the model therefore uses n = 30.Over-long sequences introduce too much noise, slowing further gains.
- Ablation Study: Using spatial embeddings as queries over sequential temporal embeddings produces the best architectural outcomes with a clear margin and P-values substantially below 0.05.The result supports the proposed direction of information retrieval from temporal embeddings using spatial queries.
4 Conclusion
The paper presents a Transformer-based framework that fuses spatial and temporal embeddings for accurate real-time surgical phase recognition. Experiments show consistent state-of-the-art performance while maintaining high processing speed and parameter efficiency.
- Conclusion: Trans-SVNet fuses different embeddings with Transformer to retrieve missing critical information for real-time surgical phase recognition at rarely additional cost.Its aggregation design is intended to combine complementary spatial and temporal information.
- Conclusion: Extensive experiments show that Trans-SVNet consistently outperforms state-of-the-art models while maintaining a high processing speed.The paper presents this combination of performance and efficiency as supporting applications in operating rooms.