Source-linked AI summary
LoViT: Long Video Transformer for Surgical Phase Recognition
Yang Liu, Maxence Boels, Luis C. Garcia-Peraza-Herrera, Tom Vercauteren, Prokar Dasgupta, Alejandro Granados, Sebastien Ourselin
TL;DR
Frame-level supervision can confuse similar scenes across phases, while long surgical videos require temporal context. LoViT trains a temporally-rich spatial extractor with video clips and fuses local and global temporal features using phase transition-aware supervision; it reports state-of-the-art performance on Cholec80 and AutoLaparo.
Problem
Frame-level supervision can confuse similar scenes assigned to different phases, and surgical phase recognition must capture temporal relationships across long videos.
Source: §1 · p. 1, §4.5 · p. 10Method
LoViT trains a temporally-rich spatial feature extractor with video-clip-level supervision, then aggregates local and global temporal information with phase transition-aware supervision.
Source: §2.2 · p. 3, §5 · p. 10Results
LoViT achieves state-of-the-art performance on Cholec80 and AutoLaparo and outperforms TeCNO and Trans-SVNet in the reported comparisons.
Source: §5 · p. 10, §4.5 · p. 10Takeaways & Limitations
The phase transition map improves performance and is described as an easy operation that adds no burden to the model.
Source: §4.5 · p. 10LoViT struggles with some phases in unusual operation sequences, and inference speed deteriorates with surgery duration when all previous spatial features are used.
Source: §5 · pp. 10–11
Abstract
from arXiv · showhide
Online surgical phase recognition plays a significant role towards building contextual tools that could quantify performance and oversee the execution of surgical workflows. Current approaches are limited since they train spatial feature extractors using frame-level supervision that could lead to incorrect predictions due to similar frames appearing at different phases, and poorly fuse local and global features due to computational constraints which can affect the analysis of long videos commonly encountered in surgical interventions. In this paper, we present a two-stage method, called Long Video Transformer (LoViT) for fusing short- and long-term temporal information that combines a temporally-rich spatial feature extractor and a multi-scale temporal aggregator consisting of two cascaded L-Trans modules based on self-attention, followed by a G-Informer module based on ProbSparse self-attention for processing global temporal information. The multi-scale temporal head then combines local and global features and classifies surgical phases using phase transition-aware supervision. Our approach outperforms state-of-the-art methods on the Cholec80 and AutoLaparo datasets consistently. Compared to Trans-SVNet, LoViT achieves a 2.4 pp (percentage point) improvement in video-level accuracy on Cholec80 and a 3.1 pp improvement on AutoLaparo. Moreover, it achieves a 5.3 pp improvement in phase-level Jaccard on AutoLaparo and a 1.55 pp improvement on Cholec80. Our results demonstrate the effectiveness of our approach in achieving state-of-the-art performance of surgical phase recognition on two datasets of different surgical procedures and temporal sequencing characteristics whilst introducing mechanisms that cope with long videos.
1. Intruduction
LoViT targets online surgical phase recognition in long videos by combining temporally informed spatial learning, local and global temporal aggregation, and phase transition-aware supervision.
1. Intruduction
- Online phase recognition classifies each frame into a high-level surgical stage and requires long temporal context because each phase contains several actions.Unlike offline recognition, online recognition identifies current activity without future information.Source: §1 · p. 1
- Earlier approaches may require extra instrument or equipment data, while long-video models face limited recurrent memory, coarser TCN features, or quadratic Transformer complexity.Source: §1 · p. 2
- LoViT is proposed to interpret long surgical videos efficiently without impacting performance.The paper presents it as a method for surgical phase recognition.Source: §1 · p. 2
- Its temporally-rich spatial feature extractor leverages temporal information to improve the precision of spatial feature learning.Source: §1 · p. 2
- Its multiscale temporal aggregation uses vanilla self-attention for local information and ProbSparse self-attention for global relationships.Source: §1 · p. 3
- Phase transition-aware supervision highlights critical temporal information at phase transitions.Source: §1 · p. 3
2. Methods
LoViT maps an online video stream to surgical phases by combining temporally rich spatial features, local and global temporal aggregation, and phase transition-aware supervision.
2. Methods
- Online surgical phase recognition maps the video stream up to frame t to the phase label for that current frame.Source: §2 · p. 3
2.1. Overview of LoViT architecture
- LoViT combines a temporally-rich spatial feature extractor, multi-scale temporal aggregation, and phase transition-aware supervision to classify surgical phases.Source: §2.1 · p. 3
2.2. Temporally-rich spatial feature extractor
- Training the spatial feature extractor on frame sets rather than isolated images addresses confusion from similar actions and screens appearing in different phases.The extractor is jointly trained with a temporal aggregator, then retained with fixed weights for spatial feature extraction.Source: §2.2 · p. 3
2.2.1 Local temporal feature aggregator
- Two cascaded L-Trans modules use different local window sizes to extract small- and large-granularity temporal features.Their fusion modules use self-attention and cross-attention to process auxiliary and current features.Source: §2.2.1 · pp. 3–5, Figure 4 · p. 5
2.2.2 Global temporal feature aggregator
- G-Informer processes long local-feature sequences with ProbSparse self-attention to capture long-range dependencies more efficiently.ProbSparse reduces time and memory usage to O(L ln(L)).Source: §2.2.2 · pp. 5–6
2.3. Multi-scale temporal feature fusion head.
- A multi-scale fusion head first merges small- and large-local features, then combines the fused local features with global features.Source: §2.3 · p. 6
2.4. Phase transition-aware supervision.
- Phase transition-aware supervision projects transition-adjacent frames onto an asymmetric Gaussian map and combines map loss with phase-class loss.The map loss is L1 and the phase-class loss is cross-entropy.Source: §2.4 · p. 6
3. Experimental Design
The evaluation uses Cholec80 and AutoLaparo with established dataset splits, then assesses video- and phase-level recognition. Implementation details specify the feature extractor, training procedure, and LoViT configuration.
3.1. Datasets.
- Cholec80 uses 40 videos for training and 40 for testing, while AutoLaparo uses 10 for training, 4 for validation, and 7 for testing.Both datasets were sampled at 1 fps and resized to 250×250 pixels; Cholec80 has 80 videos averaging 39 minutes, and AutoLaparo has 21 averaging 66 minutes.Source: §3.1 · p. 6
3.2. Implementation details.
- LoViT uses an ImageNet-1K-pretrained ViT-B/16 feature extractor and trains the extractor and temporal aggregator in separate stages.The feature extractor is a 12-layer, 12-head Transformer; the rest of LoViT is trained for 50 epochs, with outputs of 512, 64, and 8 dimensions for Ls-Trans, Ll-Trans, and G-Informer.Source: §3.2 · p. 6
- Gradient computation uses the last 100 frames for Ls-Trans and the last 500 for Ll-Trans; the transition-map settings are σl = 3 and σr = 12.The two local fusion modules each have a 2-layer encoder and decoder; G-Informer and the multi-scale-head fusion modules use 2-layer encoders and 1-layer decoders.Source: §3.2 · pp. 6–7
3.3. Evaluation.
- Evaluation reports video-level accuracy and phase-level precision, recall, and Jaccard because short phases have little impact on video accuracy.The measures are accuracy (AC), precision (PR), recall (RE), and Jaccard (JA); phase-level measures capture performance by surgical phase.Source: §3.3 · p. 7
4. Results
LoViT outperformed state-of-the-art methods on both datasets, and the component studies show that local and global temporal information contribute differently across them. The results also support temporally rich spatial features and phase transition-aware supervision.
4.1. Comparison with state-of-the-art methods
- LoViT exceeded Trans-SVNet by 2.4 pp accuracy on Cholec80 and 3.1 pp on AutoLaparo, with AutoLaparo phase-level gains of 20.9 pp precision, 3.8 pp recall, and 5.2 pp Jaccard.The authors also report that LoViT consistently outperformed in video-level and phase-level metrics.Source: §4.1 · pp. 7–8
4.2. Evaluating LoViT architecture performance
- G-Informer performed better than L-Trans on Cholec80, while L-Trans performed better on AutoLaparo; full LoViT outperformed both variants on AutoLaparo using multi-scale fusion.The authors report that global features are more prone to losing fine-grained information, while fusion of local and global features supports improved accuracy and stability.Source: §4.2 · pp. 8–9, Table 2 · p. 8
4.3. Evaluating temporally-rich spatial feature extractor performance
- Video-clip training improved the temporally rich spatial extractor, including a 1.9 pp accuracy gain on AutoLaparo, and its features better distinguished tools than frame-only training.PCA visualizations compare feature distributions on Video 60, while selected similar-context frames provide further evidence of the extractor’s distinction between tools.Source: §4.3 · p. 9, Table 3 · p. 9, Figure 7 · p. 9
4.4. Evaluating phase transition-aware supervision performance
- Phase transition-aware supervision improved performance on both datasets, with transition areas providing information about phase starts and ends and helping reduce confusion between similar clips.The study represents transitions with a supervised phase transition map and reports that this operation adds no model burden.Source: §4.4 · pp. 9–10, Table 4 · p. 9
4.5. Discussion
- LoViT combines local self-attention for short clips with ProbSparse self-attention for long videos, and outperformed TeCNO and Trans-SVNet in the study’s scope.The authors suggest dilated temporal aggregation can lose fine-grained and continuous information as sequences lengthen.Source: §4.5 · p. 10
- AutoLaparo was more challenging than Cholec80: repeated phases complicate its workflow, and its small sample size was associated with lower performance.The experiments indicated local information was more valuable than global relationships on AutoLaparo.Source: §4.5 · p. 10
5. Conclusions
LoViT combines temporally rich spatial features with local and global temporal information and reports state-of-the-art performance. Recognition remains difficult for unusual phase sequences, and inference speed can deteriorate as surgery duration increases.
5. Conclusions
- LoViT combines video-clip-supervised spatial features with a multi-scale temporal aggregator to combine local fine-grained and global macroscopic information for phase recognition.The method also uses a heatmap to learn phase transitions.Source: §5 · p. 10
- LoViT achieves state-of-the-art performance with great improvement over existing methods.Source: §5 · p. 10
- LoViT and other recognition methods struggle to recognize phases that occur in unusual operation sequences, such as P5 appearing before P4.Source: §5 · pp. 10–11
- Inference speed can deteriorate as surgery duration grows because LoViT feeds all previous spatial features into the temporal model for each current frame.The authors identify avoiding redundant calculations by using previous analysis results as a future direction.Source: §5 · p. 11