Source-linked AI summary
A Surgical Foundation Model Reveals Task-Dependent Label Efficiency
Florian Philipp Stilz, Lorenzo Arboit, Vinkle Srivastav, CAMMA International Surgical Partners, Jacques Marescaux, Sergio Alfieri, Pietro Mascagni, Nassir Navab, Nicolas Padoy
TL;DR
Expert annotations are costly in surgical AI, and how label efficiency varies across surgical tasks remains largely unexplored. The paper introduces SURGE, a self-supervised surgical foundation model, and evaluates supervision scaling across perception and reasoning tasks. It finds that perception tasks typically saturate with minimal supervision, while reasoning tasks continue to benefit from substantially more.
Problem
How label efficiency differs across surgical perception and reasoning tasks remains largely unexplored, despite the high cost and scarcity of expert annotation.
Source: §1 · pp. 1, 3Method
Results
Surgical perception tasks typically reach high performance with minimal supervision and rapidly saturate, whereas clinically grounded and relational reasoning tasks continue to benefit from substantially more supervision.
Source: §3 · p. 7Takeaways & Limitations
Task-specific label-efficiency patterns persist even with stronger supervision and higher-parameter adaptation, so representation quality alone does not determine learning efficiency.
Source: §1 · p. 3, §3 · p. 7Persistent data requirements for CVS and action-triplet recognition suggest that distillation-based pretext tasks may not fully capture surgery’s causal and logical dependencies.
Source: §3.4 · p. 8
Abstract
from arXiv · showhide
Developing label-efficient models is a central challenge in surgical AI due to the high cost and scarcity of expert annotation. While self-supervised foundation models adapt well to new tasks with minimal data, how label efficiency varies across different surgical tasks remains largely unexplored. Here, we introduce SURGE, a surgical foundation model trained on SurgSpectrum-30M+, the largest pretraining dataset comprising over 30 million frames, with checkpoints released to enable further research. We systematically evaluate label efficiency across 5 task categories and 15 benchmarks. These range from temporal and spatial scene understanding to fine-grained reasoning tied to instrument-anatomy interactions and safety-critical maneuvers. SURGE outperforms prior state-of-the-art on all benchmarks, even surpassing task-specific models on complex reasoning tasks. Crucially, we reveal a task-dependent scaling behavior: while scene understanding tasks saturate with minimal supervision, fine-grained reasoning tasks continue improving with substantially larger annotation budgets, providing a blueprint for allocating expert effort in complex domains. Code: https://github.com/CAMMA-public/SURGE
1. Introduction
Surgical AI depends on expert labels that are costly to produce, and its varied perception and reasoning tasks may require different amounts of supervision. The paper introduces SURGE and uses it to systematically examine label efficiency across these tasks.
1. Introduction
- Expert annotation is costly and time-consuming, making efficient learning and understanding task-specific annotation needs important in high-stakes AI settings.The introduction frames these as complementary goals: achieving strong performance with fewer annotations and allocating expert effort more effectively.Source: §1 · p. 1
- Surgical video analysis spans temporal and spatial perception as well as reasoning about instrument–anatomy interactions and clinically grounded decisions.These tasks differ substantially in complexity, motivating the question of whether they also differ in supervision needs.Source: §1 · p. 1
- Although self-supervised and foundation-model approaches have improved surgical representations and data efficiency, differences in label efficiency across surgical task categories remain largely unexplored.The paper addresses this gap by introducing SURGE, trained on more than 30 million surgical video frames, and systematically characterizing performance scaling with task-specific labeled data.Source: §1 · pp. 1, 3
2. Results
SURGE provides a unified representation for studying how surgical-task performance scales with supervision, using self-supervised ViT-Base and ViT-Huge+ models pretrained on over 30 million surgical video frames. Across tasks, stronger representations improve performance, but label efficiency depends on task structure: perception often saturates quickly, whereas reasoning benefits from substantially more annotation.
2.1. SURGE outperforms other foundation models across surgical AI tasks
- SURGE achieves the best performance across all 15 surgical benchmarks, with gains spanning temporal workflow understanding and fine-grained instrument-anatomy reasoning.For temporal workflow understanding, SURGE-B averages 69.40 F1 versus 63.63 for the strongest competing foundation model; for action triplet recognition, it reaches 38.72 mAP versus 32.87 for the strongest per-dataset baselines.Source: §2.1 · p. 3
2.2. Consistently superior label efficiency across all annotation regimes
- SURGE-H achieves an nAUC of 0.914 versus 0.769 for the strongest baselines, and a low-label nAUC of 0.757 versus 0.624.Larger SURGE models improve performance in 14 of 15 benchmarks, and the overall label-efficiency advantage holds across five surgical procedures.Source: §2.2 · pp. 3–4, Figure 2 · p. 4
2.3. Domain adaptation bridges general and surgical representation strengths
- SURGE combines surgical models’ workflow strengths with general-domain models’ spatial and interaction strengths, reaching 82.95 mIoU on GraSP and 44.20 mAP on CholecT50.Those scores compare with 76.09 for DINOv3-H on GraSP and 37.13 for DINOv3-H on CholecT50.Source: §2.3 · pp. 4–5
2.4. Perception tasks exhibit high label efficiency with task structure dependent variations
- Perception tasks often saturate with little supervision, but finer temporal granularity and more complex label spaces require more data.On StrasBypass70, step recognition improves 17.3% F1 from 1 to 40 videos, compared with 8.5% for phase recognition; Endoscapes-Seg50 also improves more gradually than GraSP because its label space is larger and finer-grained.Source: §2.4 · p. 5
2.5. Reasoning tasks exhibit persistent data requirements
- Reasoning tasks remain unsaturated as supervision grows: action triplet recognition scales persistently, while CVS shows little learning signal with 10 or fewer videos.For CVS on Endoscapes, SURGE-H reaches 53.26 mAP at 120 videos; its low-label nAUC is 0.520.Source: §2.5 · pp. 5–6
2.6. Label instances as an indicator of supervision requirements
- Label-instance counts provide a standardized macro-level proxy for annotation effort, and clinically grounded reasoning tasks have higher counts and continue benefiting from more labels.Temporal and spatial perception tasks use comparatively fewer annotations; semantic segmentation has the most labels per video among them but still shows comparatively efficient learning dynamics.Source: §2.6 · p. 6
2.7. Higher parameter adaptation surpasses supervised SOTA without altering label efficiency
- Partial backbone adaptation raises absolute performance enough for data-intensive reasoning tasks to surpass task-specific supervised baselines, without changing label-efficiency patterns.It yields state-of-the-art gains of 6.4% mAP on Endoscapes and 15.5% mAP on CholecT50 action triplet recognition.Source: §2.7 · p. 7
3. Discussion
Perception tasks reach high performance with little supervision, while clinically grounded and relational reasoning continues to benefit from more. SURGE’s discussion uses this task-dependent pattern to motivate annotation priorities and hybrid approaches, while noting limits of the evaluation protocol and current reasoning performance.
3. Discussion
- Perception tasks saturate rapidly, whereas clinically grounded and relational reasoning tasks continue benefiting from substantially more supervision.Increasing adapted parameters raises absolute performance but does not fundamentally change scaling with supervision.Source: §3 · p. 7
3.1. The saturation of perception tasks
- Spatial segmentation and phase recognition become task-ready with limited supervision as pretraining captures instruments, tissue textures, and broad procedural states.The authors describe the remaining training video as mapping existing representations to class names rather than learning features.Source: §3.1 · p. 7
- For standard procedures and instrument-centric tasks, larger datasets may yield diminishing returns; perception components can support mapping complex surgical logic.Source: §3.1 · p. 7
3.2. The persistent reasoning gap and the need for hybrid modeling
- Action triplet recognition and Critical View of Safety assessment require compositional logic and clinical judgment, and their scaling curves keep growing.The authors suggest these tasks involve learning underlying surgical rules frame by frame, beyond relying on pretrained visual archetypes.Source: §3.2 · p. 7
- Crossing the reasoning gap may require substantially more annotations or hybrid methods combining pretrained models with task-specific modeling, clinical inputs, or causal and relational objectives.Source: §3.2 · p. 7
3.3. A strategic roadmap to jumpstart Surgical AI
- Prioritize scarce expert annotation for relational instrument-anatomy understanding and clinical assessment, while perception tasks can use few-shot or weak supervision.The discussion also proposes task-aware modeling, temporal reasoning, and clinically relevant context for bridging perception and higher-level understanding.Source: §3.3 · pp. 7–8
3.4. Limitations & Future Work
- Linear probing and partial fine-tuning provide a controlled representation analysis but underestimate peak performance from task-specific fine-tuning.The authors release SURGE for further adaptation through specialized pipelines, including architectural adapters or proxy losses.Source: §3.4 · p. 8
- Persistent log-linear data requirements for CVS and action triplet recognition suggest distillation-based pretext tasks may not capture surgery’s causal and logical dependencies.Proposed future pretraining targets deeper clinical reasoning and vision-text alignment, such as surgical reports.Source: §3.4 · p. 8
4. Methods
The methods build SURGE from a diverse surgical-video corpus, train it with image- and patch-level self-supervision, and evaluate representation quality and label efficiency across varied tasks.
4.1. SurgSpectrum-30M+: pretraining dataset generation
- SurgSpectrum-30M+ combines public and private videos across 132 procedures and 47 anatomical targets, yielding 30,381,943 frames from 38,753 videos after filtering.Videos are sampled at 1 FPS, and frames associated with public-source validation or test sets are removed.Source: §4.1 · p. 8
4.2. Model architecture
- SURGE uses ViT-Base and ViT-Huge+ models with transformer layers that process patch embeddings into global image representations and localized patch features.The variants have 12 and 32 layers, respectively.Source: §4.2 · p. 8
4.3. Pretraining objectives
- Pretraining combines DINO image-level supervision across views with iBOT patch-level masked-image modeling and regularization.The final objective combines both losses with a 0.1-weighted regularization term.Source: §4.3 · pp. 8, 10
4.4. Post-Training refinement
- High-resolution adaptation and Gram anchoring restore patch-level consistency and enhance dense-task performance while preserving strong global representations.Gram anchoring aligns patch-feature correlation matrices between student and teacher models.Source: §4.4 · p. 10
4.5. Downstream task evaluation
- Evaluation spans five surgical task categories and 15 benchmarks, with label efficiency summarized by normalized area under performance-versus-annotation-budget curves.Low-label nAUC focuses on performance at one and three training videos; subset experiments use three random video splits.Source: §4.5 · pp. 10–11
4.6. Implementation details
- Linear probing freezes the pretrained backbone and trains task-specific heads, while higher-parameter evaluation fine-tunes selected final layers of SURGE-H.The linear-probing protocol uses a consistent learning-rate search across evaluated methods; the higher-capacity setting selects the number of layers and learning rate on validation data.Source: §4.6 · p. 11
Data availability
SurgSpectrum-30M+ aggregates public and private surgical video sources; its private institutional data cannot be released under privacy regulations and data-sharing agreements.
- Public corpus components are available through their original repositories, but private institutional videos cannot be released under privacy regulations and data-sharing agreements.Downstream evaluation datasets are publicly available from their original providers.Source: Data availability · p. 11
Code availability
Pretrained SURGE-B and SURGE-H weights, along with loading, inference, and evaluation scripts, are available in the project repository.
- SURGE-B and SURGE-H weights and scripts for loading, inference, and downstream evaluation are available at the SURGE GitHub repository.Source: Code availability · pp. 11–12
Competing interests
P.M. and N.P. report financial interests in Scialytics; the remaining authors declare no competing interests.
- P.M. and N.P. are Scialytics co-founders and shareholders; the remaining authors declare no competing interests.Source: Competing interests · p. 12