PaperlayerView exact revision on arXiv

Source-linked AI summary

A Comparative Study in Surgical AI: Potential and Limitations of Data, Compute, and Scaling

Kirill Skobelev, Eric Fithian, Yegor Baranovski, Jack Cook, Sandeep Angara, Shauna Otto, Zhuang-Fang Yi, John Zhu, Neeraj Mainkar, Margaux Masson-Forsythe, Daniel A. Donoho, X. Y. Han

arXiv:2603.27341v4cs.AIcs.CVcs.LG

TL;DR

Surgical benchmarks remain underrepresented, leaving open whether scaling generally capable AI can support surgical competence. This paper evaluates zero-shot and fine-tuned VLMs, scaling variants, and a specialized detector across four surgical datasets. Across datasets, zero-shot VLMs trail simple baselines, while a small specialized detector and fine-tuned VLMs lead; the conclusions remain bounded by the evaluated task, models, and settings.

  • Problem

    Surgical visual recognition is underrepresented in prominent medical benchmarks, leaving the contribution of model scaling to surgical AI uncertain.

  • Method

    The paper evaluates zero-shot and fine-tuned VLMs, LoRA-rank scaling, and a specialized YOLOv12-m detector across four surgical datasets.

  • Results

    Across all four datasets, zero-shot open-weight VLMs fail to surpass simple baselines, while fine-tuned models and a small specialized detector lead frontier VLMs.

  • Takeaways & Limitations

    The results suggest that advancing surgical AI may require task-specialized approaches rather than relying directly on increased computation or architecture size.

  • Takeaways & Limitations

    The study evaluates only surgical tool detection and focuses its VLM analysis on open-weight models with one prompting and decoding setup.

Abstract

from arXiv · show

Recent Artificial Intelligence (AI) models have matched or exceeded human experts in several benchmarks of biomedical task performance, but surgical benchmarks in particular are often missing from prominent medical benchmark suites. Since surgery requires integrating disparate tasks, generally-capable AI models could be particularly attractive as a collaborative tool if performance could be improved. On the one hand, the canonical approach of scaling architecture size and training data is attractive, especially since there are millions of hours of surgical video data generated per year. On the other hand, preparing surgical data for AI training requires significantly higher levels of professional expertise, and training on that data requires expensive computational resources. These trade-offs paint an uncertain picture of whether and to-what-extent modern AI could aid surgical practice. In this paper, we explore this question through a case study of surgical tool detection using state-of-the-art AI methods available in 2026. We demonstrate that even with multi-billion parameter models and extensive training, current Vision Language Models fall short in the seemingly simple task of tool detection in neurosurgery. Additionally, we show scaling experiments indicating that increasing model size and training time only leads to diminishing improvements in relevant performance metrics. Thus, our experiments suggest that current models could still face significant obstacles in surgical use cases. Moreover, some obstacles cannot be simply ``scaled away'' with additional compute and persist across diverse model architectures, raising the question of whether data and label availability are the only limiting factors. We discuss the main contributors to these constraints and advance potential solutions.

1 Introduction

Surgical tool detection is a necessary but underrepresented test of generally capable medical AI. Across multiple experiments, the paper finds that fine-tuning and specialization outperform simply increasing VLM scale, motivating task-specialized surgical AI.

  • Research gap: Surgical tool recognition is necessary, though not sufficient, for surgical competence and is absent from prominent medical benchmark suites.The paper evaluates this gap using 67,634 annotated frames from neurosurgical videos.
  • Zero-shot evaluation: 20 zero-shot open-weight VLMs spanning 2B–235B parameters remain at or near the 13.4% majority-class baseline on SDSC-EEA.This result persists despite nearly three years of model development and gains on general benchmarks.
  • Fine-tuning: 47.6% exact match accuracy follows LoRA fine-tuning with structured JSON generation, improving from 9.8% but leaving a substantial train–validation gap.The result shows that in-domain supervision helps while held-out-procedure generalization remains limited.
  • Fine-tuning: 51.1% validation accuracy is achieved with a classification head, while training accuracy reaches 89.5%, indicating poor generalization to held-out procedures.Changing the output formulation improves validation performance but does not eliminate the train–validation discrepancy.
  • Scaling: 98.6% training accuracy versus below 40% validation accuracy shows that nearly three orders of magnitude of LoRA-rank scaling does not resolve the generalization gap.Increasing adapter capacity alone therefore provides diminishing practical benefit in this setting.
  • Specialization and replication: A 26M-parameter YOLOv12-m reaches 54.7% exact match and outperforms every VLM-based approach with 1,000× fewer parameters than the best VLM.The same broad pattern replicates across CholecT50, PitVis-2023, and SurgVU, where specialized or fine-tuned models lead frontier VLMs.

2 Methods

The study evaluates zero-shot, fine-tuned, scaled, and specialized models on procedure-separated surgical datasets using strict multi-label detection metrics. Its methods combine VLM prompting and LoRA variants with a YOLO baseline and cross-dataset replication.

  • Dataset: SDSC-EEA contains 67,634 annotated frames from 66 endoscopic endonasal neurosurgical procedures.The dataset covers minimally invasive skull-base procedures recorded across 10 surgeons and seven institutions.
  • Dataset: Each frame has multi-label annotations for 31 instrument classes, with an average of 1.72 tools and substantial class imbalance.Suction appears in 63.3% of frames, making it the dominant instrument class.
  • Data split: Fine-tuning uses procedure-level separation: 47,618 frames from 53 procedures train the models and 20,016 frames from 13 procedures form validation.Frames from any procedure appear exclusively in either training or validation to prevent leakage.
  • Zero-shot VLMs: Zero-shot evaluation prompts 20 open-weight VLMs, ranging from 2B to 235B parameters, to return all visible tools as validated JSON.Malformed outputs, schema violations, and hallucinated tool names are treated as empty predictions.
  • Fine-tuning: LoRA experiments compare structured JSON generation, a 31-logit classification head, and adapter ranks from r = 2 to r = 1024.The rank sweep varies trainable parameters from 4.7M to 2.4B while retaining the same procedure-level split.
  • Specialized baseline: YOLOv12-m provides a 26M-parameter specialized object-detection baseline trained on per-frame tool bounding boxes.For external presence-only datasets, synthetic full-frame boxes are used for present tools.
  • Metrics: Evaluation reports exact match, mean Jaccard similarity, per-tool precision, recall, and F1, with ROC-AUC and AUPRC additionally used for continuous classification scores.Exact match requires the predicted tool set to equal the ground-truth set exactly.

3 Results

Across zero-shot, fine-tuned, scaling, and cross-dataset experiments, surgical tool detection improves with task-specific training but remains constrained by generalization and distribution shift. Specialized vision models outperform much larger VLMs, while scaling model or adapter capacity yields limited held-out gains.

  • 3.1 Zero-shot accuracy of open-weight models does not surpass the majority class baseline: 20 open-weight VLMs spanning 2B–235B parameters remain at or near the 13.4% majority-class baseline on SDSC-EEA.Only Qwen3-VL-235B marginally exceeds the baseline; the highest MMBench scorer achieves 10.05% validation accuracy.
  • 3.2 LoRA fine-tuning improves tool detection modestly but remains below human-level: 47.63% exact-match accuracy and 57.34% Jaccard follow LoRA fine-tuning with structured JSON, substantially exceeding zero-shot and baseline performance.The fine-tuned model reaches these values after 10 epochs on the SDSC-EEA validation set.
  • 3.3 LoRA with classification head learns in-sample but fails to generalize out-of-sample: 51.08% exact-match accuracy and 61.33% Jaccard make the classification head the strongest VLM-based method, outperforming JSON generation at the same LoRA rank.The classification approach also achieves 80.5% macro-averaged ROC-AUC and 37.6% macro-averaged AUPRC across 23 tool classes.
  • 3.4 Scaling LoRA adapter rank does not meaningfully improve out-of-sample accuracy: 98.6% training accuracy at r = 1024 contrasts with a validation peak of 39.6%, showing that increasing adapter capacity does not overcome procedure-level distribution shift.Training accuracy rises from 35.9% at r = 2, whereas validation accuracy remains below 40% across ranks.
  • 3.5 Specialized vision models outperform VLMs at lower cost: 54.73% exact-match accuracy from YOLOv12-m, using 26M parameters, exceeds VLM-based approaches requiring a 27B-parameter model.The specialized model also achieves 64.00% Jaccard similarity on SDSC-EEA.
  • 3.6 Replication across independent surgical datasets: The broad pattern reproduces across CholecT50 and PitVis-2023: zero-shot performance is poor, fine-tuning is necessary, and specialized or fine-tuned models outperform zero-shot frontier approaches.On CholecT50, fine-tuned Gemma reaches 83.02% exact match; on PitVis-2023, it reaches 84.77%.

4 Discussion

The discussion finds that surgical tool detection remains difficult for generalist VLMs despite scaling, while smaller specialized models perform better and more efficiently. It argues for specialized data, hybrid architectures, and community-driven infrastructure rather than relying on model scale alone.

  • Scaling limitations: 51.08% exact match accuracy from Gemma 3 27B with a classification head still leaves a train–validation gap that adapter-capacity scaling does not resolve.The rank sweep confirms a persistent generalization bottleneck.
  • Specialization versus generalism: A 26M-parameter YOLOv12-m outperforms VLM-based approaches across SDSC-EEA and three independent public surgical datasets.The repeated pattern includes comparisons with nine frontier VLMs on CholecT50, PitVis-2023, and SurgVU.
  • Specialization versus generalism: Specialized models match or outperform VLMs at one thousandth of the parameter cost, linking task specialization to lower compute and latency.The discussion presents this efficiency and superiority as evidence against assuming that larger end-to-end models are the best path.
  • Hybrid architectures: Hybrid systems could combine YOLO’s recall and F1 strengths with Gemma’s ROC-AUC leadership for 8 of 15 tools.The proposed architecture uses a generalist model to delegate high-precision subtasks to specialized perception modules.
  • Community-driven progress: Progress depends on clinically relevant, standardized data capturing variability across procedures, institutions, and patient populations.The paper connects this need to multi-institutional collaboration, shared data protocols, consensus labeling, and open-access tools.

5 Exploratory Next Steps: Natural Language vs. Operating Room

An exploratory comparison suggests that language-based questions about pituitary tumor surgery may be answered nearly correctly even when the same models fail at surgical tool detection. The paper relates this contrast to the practice-intensive training of neurosurgeons.

  • Natural language versus operating room: LLMs appear nearly entirely correct on pituitary tumor surgery questions but fail at the simpler task of tool detection.The paper notes that neurosurgeons train primarily through practice, offering context for the contrast.

6 Limitations

The study’s conclusions are constrained by the narrow task, model and evaluation choices, and uncertain generalizability across surgical settings.

  • Scope: The evaluation covers surgical tool detection but not higher-order capabilities such as phase recognition, decision support, or anomaly detection.The authors note that VLMs might offer greater advantages on more abstract tasks where language-mediated reasoning matters more.
  • Model and evaluation choices: The VLM study is limited to open-weight models using one prompting and decoding setup.Closed-source models, alternative prompting, or more extensive instruction tuning could produce different results.
  • Generalizability: Generalization to other specialties, institutions, and recording conditions remains uncertain.Similar takeaways across CholecT50, PitVis-2023, and SurgVU provide evidence across four surgical domains, but do not resolve this open question.
  • Dataset limitations: SurgVU label limitations constitute an additional boundary on the study’s conclusions.The supplied passage identifies this limitation but is truncated before specifying its details.

7 Conclusion

The study tests whether scaling recent vision-language models improves surgical tool detection, using neurosurgery as a case study and other domains as robustness checks. Zero-shot VLMs remain near trivial baselines, while fine-tuning helps but does not eliminate generalization gaps; small specialized detection models are substantially more efficient.

  • Conclusion: Across 20 open-weight VLMs with 2B–235B parameters, zero-shot SDSC-EEA validation accuracy remains near the 13.4% majority-class baseline.This persists despite large gains on general benchmarks such as MMBench.
  • Conclusion: 51.08% exact match accuracy is reached with LoRA fine-tuning and a classification head, but a train–validation gap persists under procedure-level distribution shift.The gap indicates limited generalization to held-out procedures.
  • Conclusion: Scaling LoRA rank by nearly three orders of magnitude saturates training accuracy near 99%, while validation accuracy remains below 40%.The result argues against insufficient adapter capacity as the explanation for the validation gap.
  • Conclusion: A 26M-parameter YOLOv12-m outperforms VLM-based approaches at a fraction of their training time and inference cost.The specialized detector is over 1,000× smaller than the evaluated VLM.
  • Conclusion: The findings suggest reliable Surgical AI is more constrained by specialized data than by architecture or training-resource scale.The authors identify pooling and labeling surgical data across institutions as crucial for improving capabilities.

A Per-Dataset Exact-Match Accuracy with Confidence Intervals

The appendix documents per-dataset evaluation settings, exact-match benchmarks, data distributions, prompting, output validation, and fine-tuning procedures.

  • Per-Dataset Exact-Match Accuracy with Confidence Intervals: SDSC-EEA exact-match results use 20,016 validation frames from 13 procedures with 95% bootstrap confidence intervals.The majority-class baseline predicts the most common tool set, and closed-weight frontier VLMs were excluded because private data could not be sent to third-party APIs.
  • Per-Dataset Exact-Match Accuracy with Confidence Intervals: CholecT50, PitVis-2023, and SurgVU tables report exact-match accuracy with 95% bootstrap confidence intervals and majority-class baselines.Their validation sets contain 19,923, 30,896, and 18,919 frames, respectively; output validation failures count as incorrect predictions.
  • Data distribution: Procedure-level splitting creates highly uneven per-tool distributions between training and validation sets.For example, Aspirating dissector has 88 training versus 2,319 validation instances, while Sonopet pineapple tip has 1,991 training instances and none in validation.
  • Evaluation protocol: Zero-shot evaluation requires an exact JSON schema, ontology-matching tool names, and valid output parsing.Invalid outputs are converted to empty predictions, so formatting, schema, and ontology failures affect downstream metrics.
  • Fine-tuning protocol: Fine-tuning uses LoRA adapters, while the rank sweep spans r ∈{2, 4, 8, 16, 32, 64, 128, 256, 512, 1024}.Trainable parameters range from 4.7M at r = 2 to 2.4B at r = 1024, and the sweep requires approximately 62 wall-clock hours.

I Per-Tool Metrics for LoRA Fine-Tuning with Classification Head

Per-tool metrics show that fine-tuned Gemma 3 27B performs strongly on well-represented instruments but less reliably on tools with limited or split-skewed training data; zero-shot outputs also suffer ontology failures.

  • Per-tool performance: ROC-AUC exceeds 0.9 for tools well represented in training, including Suction Coagulator, Drill, and Straight Forceps.Other high-ROC-AUC tools include Curette, Surgical Knife, Cotton Patty, Rongeur, Rhoton Dissector, Backbiting rongeur, and Surgicel.
  • Per-tool performance: Tools with limited training data or predominant appearance in validation procedures have lower ROC-AUC values.This pattern links per-tool performance to representation across the training and validation distributions.
  • Output validation: Qwen2-VL-2B-Instruct has a 41.7% output validation failure rate, often hallucinating tool names absent from the provided ontology.Such failures include syntactically valid JSON containing nonexistent names and are treated as empty predictions downstream.

K YOLOv12-m Training Configuration

YOLOv12-m was trained for 300 epochs using default hyperparameters, without hyperparameter search, with the run selected by best exact match set accuracy.

  • Default hyperparameters: no hyperparameter search was performed for the YOLOv12-m training configuration.
  • 300 epochs: YOLOv12-m used a 300-epoch training run selected by best exact match set accuracy.

L Per-Tool Metrics for CholecT50 Evaluation

On CholecT50, zero-shot Gemma performed poorly, while fine-tuning substantially improved tool classification; YOLOv12-m was competitive on selected tools.

  • 0.966 macro ROC-AUC and 0.883 macro AUPRC: fine-tuned Gemma achieved strong aggregate ranking metrics on CholecT50.
  • Hook achieved the highest fine-tuned F1 of 0.974, while every tool exceeded F1 of 0.7.
  • Bipolar improved from F1 0.120 to 0.831 and irrigator from 0.089 to 0.800 after fine-tuning.
  • YOLOv12-m exceeded Gemma on irrigator F1 (0.862 vs. 0.800), whereas Gemma exceeded YOLOv12-m on grasper F1 (0.913 vs. 0.904).

M Per-Tool Metrics for PitVis-2023 Evaluation

On PitVis-2023, zero-shot Gemma produced almost no reliable tool detections, while fine-tuning improved aggregate metrics but left substantial task difficulty.

  • Per-tool evaluation covered zero-shot and fine-tuned Gemma, alongside YOLOv12-m, using classification and ranking metrics.The validation set contains 30,896 frames.
  • The reported tables organize PitVis-2023 results by per-tool classification metrics and by ROC-AUC or AUPRC.
  • For 11 of 18 tools, zero-shot Gemma produced zero correct detections, with severe over-prediction for bipolar_forceps and suction.Only suction achieved an F1 above 0.1.
  • 0.966 macro-averaged ROC-AUC and 0.691 macro AUPRC: fine-tuned Gemma improved aggregate PitVis-2023 performance.

N Per-Tool Metrics for SurgVU Evaluation

On SurgVU, zero-shot Gemma showed low tool-detection quality, while fine-tuning yielded moderate aggregate ranking performance across the represented tool classes.

  • SurgVU evaluation compared zero-shot Gemma, fine-tuned Gemma with LoRA and a classification head, and YOLOv12-m on 18,919 validation frames.Tools with zero ground-truth instances were omitted from per-tool tables but contributed to macro averages where applicable.
  • The reported tables provide per-tool classification metrics and ROC-AUC results for SurgVU.
  • No zero-shot tool exceeded F1 of 0.5, and cadiere forceps was detected only 0.7% of the time despite appearing in 42% of validation frames.Several tools also produced thousands of false positives.
  • 0.740 macro-averaged ROC-AUC and 0.502 macro AUPRC: fine-tuned Gemma’s performance across the 14 represented SurgVU tool classes.

O Robustness Check: CNN without Bounding Box Supervision

A ResNet-50 tests whether bounding-box supervision is necessary for tool-presence classification. Its set-level-label performance is below models trained with localization supervision and the strongest fine-tuned VLM.

  • 39.6% exact match accuracy was achieved on the validation set using only set-level labels and no bounding box supervision.The model also reached 52.6% Jaccard similarity, 70.3% top-1 accuracy, and 0.673 micro F1.
  • Figure 10 plots training loss on the left and validation exact match accuracy by epoch on the right, with a 13.4% majority-class baseline.
  • The ResNet-50 matched the 3-epoch LoRA rank sweep at r = 1024 but remained below the best fine-tuned VLM’s 51.08% exact match accuracy.
  • The ResNet-50 also trailed YOLOv12-m at 54.7%, supporting a possible advantage from bounding box supervision.The comparison evaluates tool presence rather than spatial localization.
  • Gemma 3 27B’s zero-shot accuracy stayed near 10% from temperatures 0 to 0.7 and declined at higher temperatures, with output failures reaching 38 at T = 2.0.The shaded region denotes a 95% Wilson binomial confidence interval, and all intervals overlap.

Q Per-Tool Metrics for Zero-Shot VLM Evaluation

The appendix presents per-tool precision and recall tables for zero-shot VLMs, while extracted model responses illustrate uncertainty and inconsistent tool naming during detection. The tables organize results by model and sort displayed tools by F1 score.

  • Per-tool tables report precision and recall for each evaluated zero-shot VLM, showing only tools with at least one true or false prediction.Tools are sorted by descending F1 score.
  • The tables cover Qwen, Gemma, MedGemma, Llama, and LLaVA models ranging from 2B to 235B parameters.
  • Model responses vary between predicting Suction with Grasper and predicting Suction with Suction Coagulator for a frame containing two instruments.
  • Some responses include prolonged self-correction over whether the second instrument is Suction, Suction Coagulator, or Grasper.
  • One response explicitly outputs the structured JSON list ["Suction", "Suction Coagulator"].

S.1 Model Configuration

The supplied material pairs a Gemma 3 27B generation-configuration table with general descriptions of endoscopic pituitary surgery and comparisons between microscopic and endoscopic approaches. The procedural overview emphasizes that techniques vary with patient and surgeon factors.

  • Table 56 is identified as the Gemma 3 27B generation configuration table.
  • The endoscopic endonasal transsphenoidal approach is described as a general procedure for removing pituitary adenomas, with specifics varying by tumor size, location, and surgeon preference.
  • The procedural overview includes anesthesia, positioning, nasal access, sellar-floor opening, tumor debulking, reconstruction, and postoperative care.
  • Microscopic and endoscopic approaches differ in visualization, precision, tumor suitability, complications, nasal morbidity, and recovery time.
  • Both approaches are described as effective for microadenomas, while surgical preference for larger or aggressive tumors depends partly on tumor complexity and surgeon experience.
Loading 2603.27341v4…