Shifted Window Temporal Transformer for YOLO-Guided Phase Recognition in Laparoscopic Cholecystectomy Surgery

• Workflow analysis requires both phase recognition and instrument identification, while similar consecutive frames can make spatial-only phase predictions ambiguous and unstable.Source: §1 · p. 2
• Frame-level CNNs miss long-term temporal relations, while CNN-LSTM and CNN-GRU models are limited by sequential processing.Source: §1 · p. 3
• The identified gaps include separate treatment of phase and instrument tasks, underexplored Swin-temporal integration, class imbalance, and prediction instability.Source: §1 · p. 3
• The framework combines a Swin Transformer and Temporal Transformer with YOLOv11s for concurrent phase recognition and real-time instrument localization.Source: §1 · pp. 3–4
• A causal rolling-queue smoothing strategy targets temporal consistency at runtime, with Cholec80 experiments aimed at stable real-time workflow analysis.Source: §1 · pp. 3–4

2.1 Qualitative Instrument Localization

• Qualitative examples show YOLOv11s locating multiple instruments, including Bipolar, Specimen Bag, Grasper, Clipper, Hook, and Irrigator, across varied surgical scenes.The predicted boxes are described as similar to visible instruments across viewpoints, illumination conditions, occlusions, and surgical contexts.Source: §2.1 · p. 4, Figure 1 · p. 5

2.2 Surgical Phase Recognition

• 0.95 and 0.94 were the highest phase F1-scores for CalotTriangleDissection and GallbladderDissection, while CleaningCoagulation scored 0.73.Lower performance in some phases is associated with class imbalance and visual similarity between neighboring stages.Source: §2.2 · p. 5, Table 1 · p. 6
• 92% overall accuracy, macro F1-score 0.86, and weighted F1-score 0.92 were achieved across 47,625 test samples.The overall performance is summarized in Table 2.Source: §2.2 · p. 5, Table 2 · p. 6
• 92.0% accuracy outperformed CNN-based methods and TeCNO, while remaining competitive with transformer-based approaches on Cholec80.The comparison uses a smaller test subset (Videos 61–80) than prior methods (Videos 41–80); prior methods were not directly re-implemented.Source: §2.2 · p. 6

2.3 Confusion Matrix Analysis

• Strong diagonal dominance appears in the normalized confusion matrices, with most errors between neighboring phases sharing similar anatomy and instrument use.Per-class F1-scores are strongest for major dissection phases and lower for more difficult or less frequent phases.Source: §2.3 · p. 6, Figure 3 · p. 7, Figure 2 · p. 7

2.4 YOLOv11s Instrument Detection

• 0.863 mAP@0.5 and 0.84 F1-score were achieved by the YOLOv11s detector; Clipper had the highest AP at 0.959 and Irrigator the lowest at 0.787.Precision reached 1.00 at confidence threshold 0.977 and recall reached 0.92 at threshold 0.000; validation losses stabilized after approximately 30 of 50 epochs, with mAP@0.5:0.95 approximately 0.50.Source: §2.4 · pp. 7–8, Figure 4 · p. 8

3 Discussion

• The detector performs particularly well for the Clipper, while occlusion, smoke, and fluid interference affect the Irrigator more.A detection threshold of 0.266 yields an F1-score of 0.84.Source: §3 · pp. 8–9
• Causal rolling-window smoothing reduces isolated frame-level phase fluctuations, improving temporal consistency.Source: §3 · p. 9
• 92% phase recognition accuracy, macro F1-score 0.86, and instrument-detection mAP@0.5 of 0.863 demonstrate performance across both tasks on Cholec80.The detector also achieves an F1-score of 0.84.Source: §3 · pp. 8–9
• The framework could support surgical-video indexing and postoperative workflow review, but clinical deployment requires testing across clinics, surgeons, and operating-room conditions.Source: §3 · pp. 9–10
• The comparison reports 92.0% accuracy and a macro-averaged Jaccard index of 0.76, but evaluates Videos 61–80 against prior results from Videos 41–80.Prior methods’ published metrics are used; the authors describe the comparison as conservative.Source: Table 3 · p. 9, §3 · p. 10

4.1 Dataset

• Cholec80 provides 80 videos from 13 surgeons, annotated frame by frame for surgical phase and instrument presence; videos 1–40, 41–60, and 61–80 are split for training, validation, and testing.Uniform downsampling reduces the videos from 25 fps to 1 fps, and the video-level split prevents temporal data leakage.Source: §4.1 · p. 11

4.2 Video Preprocessing and Temporal Clip Construction

• Input frames are resized to 224×224, converted from BGR to RGB, and organized into overlapping 12-frame clips to preserve temporal context.The temporal window stride is 8 frames and the clip stride is 5 frames.Source: §4.2 · pp. 11–12

4.3 Model Architecture

• A shared Swin Transformer and Temporal Transformer Encoder support phase classification and instrument recognition, alongside a parallel YOLO detector for instrument localization.The detector processes full-resolution frames; the classification stream models spatial features and temporal relationships.Source: §4.3 · p. 12

4.4 Loss Function and Class Imbalance

• The model jointly trains seven-class phase classification and seven-label multi-label instrument recognition, using focal loss and inverse-frequency sampling to address class imbalance.A causal FIFO queue of length 10 applies majority voting over recent phase predictions to reduce frame-level fluctuations without future frames.Source: §4.4 · pp. 12–13
• The evaluation is limited to Cholec80; the authors recommend larger multi-center validation, and note that smoke, bleeding, occlusion, lens fogging, and rapid camera motion may reduce performance.They also report that visually similar adjacent phases and rare phases or instruments can remain difficult to recognize.Source: §4.4 · p. 14

Funding

• No funds, grants, or other support were received during manuscript preparation.Source: Funding · p. 15

Competing Interests

• The authors report no competing interests.Source: Competing Interests · p. 15

Data Availability

• Cholec80 is publicly available, with access requests directed to the dataset creators.Source: Data Availability · p. 15

Code Availability

• Code is to be made public upon acceptance, and a surgical-phase-recognition demonstration is also announced.Source: Code Availability · p. 15

Ethics Approval

• Ethics approval is listed as not applicable because the study used a public dataset and involved no human participants or animals.Source: Ethics Approval · p. 15