Source-linked AI summary
Large-Small Model Collaboration for Zero-Shot Surgical Phase Recognition
Yiyi Zhang, Ying Zheng, Wenxin Fan, Yu Zhu, Yuchen Yuan, Litao Zhao, Zheng Li, Pheng-Ann Heng
TL;DR
Lightweight surgical phase models capture temporal dependencies but often struggle to generalize across clinical domains, while foundation models lack explicit temporal modeling. LaST combines foundation-model phase priors with iterative temporal refinement, dynamic quality control, dual-model cross-learning, and cycle replay. On Cholec80, LaST exceeds PeskaVLP by +24.85% and surpasses 32-shot SPA in F1-score on both datasets.
Problem
Lightweight surgical phase models capture temporal dependencies but often struggle to generalize across clinical domains, while foundation models lack explicit temporal modeling.
Source: §1 · p. 2Method
Results
On Cholec80, LaST exceeds PeskaVLP by +24.85% and surpasses 32-shot SPA in F1-score on both datasets.
Source: §4 · p. 7Takeaways & Limitations
The results support large–small collaboration as a practical route to zero-shot surgical phase recognition in annotation-scarce settings.
Source: §5 · p. 9Dynamic quality control assumes that the lower-mean loss component corresponds to clean samples.
Source: Dynamic Quality Control · p. 5
Abstract
from arXiv · showhide
Task-specific lightweight models for surgical phase recognition excel at capturing temporal dynamics but generalize poorly under domain shift. Conversely, surgical foundation models (FMs) offer superior transferability via large-scale pretraining, yet their lack of explicit temporal modeling often yields temporally inconsistent predictions, leading to degraded performance. To exploit the complementary strengths of both paradigms, we propose \textbf{La}rge-\textbf{S}mall \textbf{T}emporal adaptation (\textbf{LaST}), a novel large-small collaborative framework that enables zero-shot adaptation to unseen clinical domains. In LaST, the FM initiates the pipeline by generating frame-level phase priors that serve as initial weak supervision. To effectively utilize these noisy phase priors, we introduce an iterative temporal refinement scheme that integrates dynamic quality control to filter reliable predictions and dual-model cross-learning to mitigate confirmation bias. Simultaneously, the lightweight model leverages its intrinsic temporal modeling ability to progressively correct inconsistent predictions and enhance overall accuracy across iterations. At the end, a cycle replay strategy is employed to close the loop: the refined, more accurate predictions are utilized as upgraded supervision signals for the subsequent iterations, fostering a self-reinforcing evolution of both label quality and model capability. Extensive experiments demonstrate that LaST achieves robust adaptation to unseen domains for zero-shot surgical phase recognition, outperforming the baseline (PeskaVLP) by 24.85\%-43.17\% in accuracy and even surpassing fully supervised linear probing and several state-of-the-art few-shot approaches. Codes will be released at https://github.com/YIYIZH/LaST.
1 Introduction
LaST combines a foundation model’s semantic generalization with lightweight temporal modeling to enable zero-shot surgical phase adaptation to unseen clinical domains. Its iterative refinement and cycle replay use noisy initial predictions to progressively improve supervision.
1 Introduction
- LaST improves FM F1-score by 23.48% on Cholec80 and 37.54% on AutoLaparo, exceeds supervised linear probing by 1.49% and 14.61%, and outperforms SPA with 32 shots.Figure 1 reports gains of 24.8% and 43.2% over PeskaVLP [21] without manual annotation.Source: Figure 1 · p. 2, §1 · p. 3
- Task-specific temporal models capture surgical dynamics but generalize poorly across procedures, while SPA requires few-shot labels and is unsuitable for unseen scenarios without labeled data.Domain shifts can reflect differences in operating-room layouts, surgeon preferences, institutional protocols, and patient anatomies.Source: §1 · p. 2
- Surgical vision-language FMs transfer zero-shot but lack clinical precision; linear probing remains suboptimal because static image-text architectures fail to model surgical-phase dynamics.The introduction attributes these limitations to FM zero-shot reliability and precision gaps and to CLIP-like architectures [26] that underuse temporal dynamics.Source: §1 · p. 2
- Noisy FM predictions make direct distillation risky: unreliable labels can cause overfitting and error accumulation, and FM confidence can be spuriously high for incorrect predictions.LaST instead relies on the small model’s temporal modeling to absorb noisy supervision and progressively improve prediction quality.Source: §1 · p. 3
- LaST initializes with FM pseudo-labels, warms up two temporal models, uses class-wise GMM loss filtering and peer-selected clean subsets for cross-training, then replays refined predictions as supervision.This iterative temporal refinement and cycle replay form a self-reinforcing loop intended to reduce confirmation bias and improve label quality across cycles.Source: §1 · p. 3
2 Method
LaST adapts a frozen surgical foundation model to unannotated target videos through a self-reinforcing collaboration between its semantic priors and lightweight models’ temporal modeling. It initializes phase pseudo-labels, refines them within cycles, and reuses the resulting predictions as supervision across cycles.
2.1 Overview of the LaST Framework
- LaST combines a foundation model’s generalization with lightweight networks’ temporal modeling in a self-reinforcing adaptation loop for unseen domains.The framework predicts phase labels for an unannotated surgical video sequence.Source: §2.1 · pp. 3–4
2.2 Zero-Shot Initialization
- The pretrained surgical vision-language model compares each frame’s visual embedding with text embeddings of phase prompts to generate initial pseudo-labels.These labels provide global semantic priors but lack temporal consistency and contain noise, so they serve as noisy supervision.Source: §2.2 · p. 4
2.3 Iterative Temporal Refinement
- Two temporally modeled networks are warmed up with the initial pseudo-labels, then trained through dynamic filtering and peer-model cross-learning to reduce overfitting to noise.Each model updates using the peer model’s filtered clean set.Source: §2.3 · pp. 4–5
Dynamic Quality Control
- Class-wise two-component GMMs on per-sample losses estimate label cleanliness, and posterior confidence separates samples into labeled and unlabeled sets.The lower-mean component is treated as clean; samples with clean posterior wt > τ enter the labeled set, while wt ≤ τ samples remain unlabeled. The cleanliness premise cites, the GMM, and posterior probability.Source: Dynamic Quality Control · p. 5
Dual-Model Cross-Learning
- DivideMix-inspired cross-learning trains each temporal network on the peer’s partition, combining clean-sample supervision, adjacent-frame smoothing, and class-balance regularization.The smoothing uses stop-gradient, while the balance term constrains predictions toward a uniform class prior using KL divergence.Source: Dual-Model Cross-Learning · pp. 5–6
2.4 Cycle Replay
- After each cycle, the ensemble of both networks’ predictions replaces the prior pseudo-labels as supervision; after three cycles, the final ensemble supplies the phase output.The refined labels are intended to improve model accuracy and let quality control identify more reliable clean samples in subsequent cycles.Source: §2.4 · p. 6
3 Experimental Setup
LaST is evaluated for zero-shot adaptation on Cholec80 and AutoLaparo using only their test videos, without access to human annotations. The setup uses a frozen PeskaVLP visual encoder and dual MS-TCN models, with iterative training across three cycle rounds.
Evaluation Benchmarks:
- Cholec80 and AutoLaparo evaluation uses only test videos for adaptation and final evaluation, with no human annotations accessed.The benchmarks contain 80 cholecystectomy videos and 21 laparoscopic hysterectomy videos, respectively, and each has seven annotated phases; official data splits are followed.Source: Evaluation Benchmarks · p. 6
- LaST uses a frozen PeskaVLP visual encoder for initialization and dual MS-TCN small models, and is implemented on a single RTX 3090.PeskaVLP was pretrained on surgical video lectures from open e-learning platforms; all models use Adam with a learning rate of 5 × 10−4.Source: Evaluation Benchmarks · p. 6
- The quality-control threshold τ = 0.9 achieves the best performance, and training takes 0.71 GPU hours on Cholec80 and 0.22 on AutoLaparo.The mechanism performs more reliably with relatively high thresholds (τ>0.7); training runs for 50, 30, and 20 epochs across r = 3 cycle rounds. Table 1 reports Accuracy (Acc) and average F1-Score (F1) on both datasets.Source: Evaluation Benchmarks · p. 6, Table 1 · p. 7
4 Experimental Results
LaST improves zero-shot surgical phase recognition through temporal refinement and iterative collaboration, outperforming PeskaVLP and several few-shot approaches while remaining below fully supervised upper bounds.
4 Experimental Results
- LaST’s zero-shot approach surpasses the strongest foundation model, PeskaVLP, by +24.85% on Cholec80 and +43.17% on AutoLaparo, with larger gains on challenging AutoLaparo.It also surpasses 32-shot SPA in F1-score on both datasets and outperforms 1-shot SPA by +13.79% and +9.54%.Source: §4 · p. 7
- 61.02% vs. 59.72% and 58.32% vs. 53.80%: dual-model training outperforms single-model training; Round 3 exceeds Round 1 by 9.75% and 7.74%.The dual-model protocol uses cross-supervision and mutual agreement, while cycle replay progressively retrains on refined pseudo-labels.Source: Table 2 · p. 8, §4 · p. 8
- Temporal-window averaging improves over frame-level predictions but plateaus as larger windows blur phase boundaries; adaptive video-level modeling overcomes this limit.The comparison contrasts feature averaging with MS-TCN temporal modeling.Source: §4 · pp. 7–8, Figure 3 · p. 8
- On Cholec80, confidence filtering quickly saturates, while adaptive loss modeling progressively recovers a larger, more accurate clean set over rounds.The Clean Ratio measures the percentage of true clean samples among selected clean pseudo-labels; AutoLaparo shows a similar trend, though its curve is omitted. The confidence-based filtering baseline is cited as.Source: §4 · p. 8, Figure 3 · p. 8
5 Conclusion
LaST combines surgical foundation-model transferability with lightweight temporal modeling for zero-shot surgical phase recognition under domain shift. Its iterative refinement and Cycle Replay strategies self-correct noisy pseudo-labels without human supervision, and experiments show gains over zero-shot baselines and state-of-the-art few-shot methods requiring manual annotations.
5 Conclusion
- LaST substantially outperforms zero-shot baselines and state-of-the-art few-shot methods that require manual annotations.The conclusion presents large-small collaboration as a practical route for deployment in annotation-scarce settings.Source: §5 · p. 9
- LaST combines surgical foundation-model transferability with lightweight models’ temporal modeling to produce accurate predictions under domain shift.The framework targets zero-shot surgical phase recognition.Source: §5 · p. 9
- Iterative Temporal Refinement nests dynamic quality control and dual-model cross-learning, while global Cycle Replay reinforces self-correction of noisy pseudo-labels without human supervision.The system iteratively refines its pseudo-labels.Source: §5 · p. 9