Source-linked AI summary
Large-Small Model Collaboration for Zero-Shot Surgical Phase Recognition
Yiyi Zhang, Ying Zheng, Wenxin Fan, Yu Zhu, Yuchen Yuan, Litao Zhao, Zheng Li, Pheng-Ann Heng
TL;DR
Surgical lightweight models capture temporal dynamics but generalize poorly under domain shift, while foundation models transfer broadly yet produce temporally inconsistent or insufficiently precise predictions. LaST combines their complementary strengths through iterative quality-controlled refinement, cross-learning, and cycle replay, achieving stronger zero-shot adaptation than foundation-model baselines and several supervised or few-shot alternatives.
Problem
Lightweight temporal models struggle under domain shift, whereas surgical foundation models lack reliable temporal modeling, motivating complementary zero-shot adaptation.
Method
LaST uses foundation-model predictions as noisy pseudo-labels, applies class-wise dynamic quality control and dual-model cross-learning, then replays refined labels across iterative cycles.
Results
LaST outperforms PeskaVLP by 24.85% and 43.17% and surpasses fully supervised linear probing and 32-shot SPA in reported comparisons.
Takeaways & Limitations
Large-small collaboration provides a practical route to zero-shot surgical phase recognition under domain shift in annotation-scarce settings.
Takeaways & Limitations
The quality-control procedure assumes that the lower-mean component of the class-specific loss mixture represents clean samples and uses a confidence threshold to select them.
Abstract
from arXiv · showhide
Task-specific lightweight models for surgical phase recognition excel at capturing temporal dynamics but generalize poorly under domain shift. Conversely, surgical foundation models (FMs) offer superior transferability via large-scale pretraining, yet their lack of explicit temporal modeling often yields temporally inconsistent predictions, leading to degraded performance. To exploit the complementary strengths of both paradigms, we propose \textbf{La}rge-\textbf{S}mall \textbf{T}emporal adaptation (\textbf{LaST}), a novel large-small collaborative framework that enables zero-shot adaptation to unseen clinical domains. In LaST, the FM initiates the pipeline by generating frame-level phase priors that serve as initial weak supervision. To effectively utilize these noisy phase priors, we introduce an iterative temporal refinement scheme that integrates dynamic quality control to filter reliable predictions and dual-model cross-learning to mitigate confirmation bias. Simultaneously, the lightweight model leverages its intrinsic temporal modeling ability to progressively correct inconsistent predictions and enhance overall accuracy across iterations. At the end, a cycle replay strategy is employed to close the loop: the refined, more accurate predictions are utilized as upgraded supervision signals for the subsequent iterations, fostering a self-reinforcing evolution of both label quality and model capability. Extensive experiments demonstrate that LaST achieves robust adaptation to unseen domains for zero-shot surgical phase recognition, outperforming the baseline (PeskaVLP) by 24.85\%-43.17\% in accuracy and even surpassing fully supervised linear probing and several state-of-the-art few-shot approaches. Codes will be released at https://github.com/YIYIZH/LaST.
1 Introduction
LaST addresses the complementary weaknesses of surgical foundation models and lightweight temporal models for zero-shot adaptation under clinical domain shift. It combines FM-generated weak supervision with iterative temporal refinement and cycle replay to improve phase recognition without manual annotation.
- Motivation: Task-specific lightweight models capture temporal dependencies but generalize poorly across procedures and clinical environments under domain shift.Differences in operating-room layouts, surgeon preferences, institutional protocols, and patient anatomies can require costly manual re-annotation and retraining.
- Motivation: Surgical vision-language foundation models transfer across domains but produce zero-shot predictions with insufficient reliability and precision for clinical use.Linear-probe adaptation remains insufficient because common CLIP-like architectures lack explicit temporal modeling.
- LaST: LaST frames large-small collaboration as a way to combine foundation-model semantic generalization with lightweight-model procedure-specific temporal dynamics.The framework targets zero-shot adaptation to unseen clinical scenarios in surgical phase recognition.
- LaST: The FM generates initial phase predictions as weak supervision, while two lightweight temporal models are warmed up and iteratively refined using quality-controlled, peer-selected samples.Class-wise loss modeling with a Gaussian Mixture Model separates cleaner from noisier samples, and cross-training reduces confirmation bias.
- LaST: Cycle replay feeds refined predictions back as higher-quality pseudo-labels, progressively improving label quality and enabling temporal models to specialize to the target domain.The loop is designed to yield reliable phase recognition under distribution shift without relying on manual annotation.
2 Method
LaST adapts a frozen foundation model to unseen surgical domains by combining its semantic priors with lightweight temporal models. Its three-stage pipeline refines noisy pseudo-labels through quality control, cross-learning, temporal objectives, and cycle replay.
- Overview: LaST targets annotation-free adaptation to unseen domains by combining foundation-model generalization with lightweight temporal modeling.The framework predicts phase-label sequences for unannotated surgical videos.
- Zero-Shot Initialization: Zero-shot initialization uses a pretrained surgical vision-language foundation model to assign each target-video frame a phase pseudo-label from visual-text similarity.These predictions provide global semantic priors but are noisy and temporally inconsistent.
- Iterative Temporal Refinement: Iterative temporal refinement warms up two temporal models, partitions samples by dynamically estimated label cleanliness, and trains each model using its peer’s filtered data.Class-wise loss distributions are modeled with a two-component GMM, and samples exceeding threshold τ are treated as clean.
- Iterative Temporal Refinement: The refinement objective combines sparse supervision on peer-selected clean samples, temporal smoothing across adjacent frames, and class-balance regularization.The temporal term uses a stop-gradient target, while the class-balance term constrains empirical predictions toward a uniform prior.
- Cycle Replay: Cycle replay ensembles the two temporal models’ predictions and feeds the refined pseudo-labels into the next cycle as upgraded supervision.The loop progressively improves pseudo-label quality and model capability; final-cycle predictions are used as output.
3 Experimental Setup
LaST is evaluated for zero-shot surgical phase recognition on Cholec80 and AutoLaparo without human annotation supervision, using frozen PeskaVLP initialization and MS-TCN small models. Results are reported with Accuracy and average F1-Score against zero-shot and few-shot methods.
- Benchmarks: Evaluation uses Cholec80 and AutoLaparo, with adaptation and final testing performed only on test videos without human annotation supervision.Both benchmarks contain seven annotated surgical phases and follow their official data splits.
- Implementation: LaST uses a frozen PeskaVLP visual encoder as its foundation model for zero-shot initialization and instantiates both small models as MS-TCN.The models are optimized with Adam at a learning rate of 5 × 10^-4.
- Metrics: Table 1 reports Accuracy (Acc) and average F1-Score (F1) on Cholec80 and AutoLaparo, with subscripts denoting performance gaps compared with LaST.The table identifies the best and second-best results.
- Baselines: LaST is compared with zero-shot foundation models and few-shot adapters, including Linear Probe, LP+Text, Tip-Adapter-F, and SPA.The few-shot methods use 1 or 32 shots, while the zero-shot methods use none.
4 Experimental Results
LaST improves zero-shot surgical phase recognition over foundation-model baselines and remains competitive with few-shot methods. Ablations attribute gains to temporal modeling, dynamic quality control, dual-model cross-learning, and cycle replay.
- Overall comparison: 24.85% and 43.17% gains over PeskaVLP are reported on Cholec80 and AutoLaparo, respectively, while LaST also surpasses 32-shot SPA in F1-score on both datasets.LaST additionally improves over 1-shot SPA by +13.79% and +9.54%.
- Temporal modeling: Temporal-window averaging improves the frame-level baseline, but its gains plateau as larger windows introduce ambiguity from neighboring phases.Adaptive video-level temporal modeling overcomes this limitation and outperforms static window-based smoothing.
- Dynamic quality control: Loss-distribution-based dynamic quality control progressively recovers a larger and more accurate clean pseudo-label set than confidence-based filtering on Cholec80.The comparison uses Clean Ratio, defined as the percentage of true clean samples among selected clean pseudo-labels.
- Cross-learning and replay: 61.02% versus 59.72% and 58.32% versus 53.80% show that dual-model cross-learning outperforms single-model training.Progressive retraining with cycle replay further improves performance, with Round 3 exceeding Round 1 by 9.75% and 7.74%.
5 Conclusion
LaST combines foundation-model transferability with lightweight-model temporal modeling for zero-shot surgical phase recognition under domain shift. Its iterative refinement and cycle replay improve pseudo-labels without human supervision, supporting annotation-scarce deployment.
- Conclusion: LaST combines surgical foundation-model transferability with lightweight temporal modeling to produce predictions under domain shift.The framework targets zero-shot surgical phase recognition.
- Conclusion: Dynamic quality control and dual-model cross-learning operate within iterative temporal refinement, while cycle replay reinforces self-correction of noisy pseudo-labels.The process is described as operating without human supervision.
- Conclusion: LaST substantially outperforms zero-shot baselines and state-of-the-art few-shot methods requiring manual annotations.The conclusion presents large-small collaboration as a practical route for annotation-scarce settings.