PaperlayerView exact revision on arXiv

Source-linked AI summary

SurgBench: A Unified Large-Scale Benchmark for Surgical Video Analysis

Jianhui Wei, Zikai Xiao, Danyu Sun, Luqi Gong, Zongxin Yang, Zuozhu Liu, Jian Wu

arXiv:2506.07603v2cs.CVcs.AI

TL;DR

Surgical video foundation model development is constrained by limited dataset diversity and task comprehensiveness, despite the clinical relevance of analyzing surgical videos. SurgBench combines a broad pretraining dataset and evaluation benchmark; experiments report gains over Kinetics-based pretraining and stronger performance than natural-video-pretrained models.

  • Problem

    Surgical video analysis has clinical applications, but existing datasets have limited disease, procedure, specialty, and task diversity.

    Source: §1 · pp. 1–2

  • Method

    SurgBench combines a 53-million-frame pretraining dataset spanning 22 procedures and 11 specialties with an evaluation benchmark covering 72 fine-grained tasks across six categories.

    Source: §1 · p. 2

  • Results

    SurgMAE pretrained on SurgBench yields 7.9% performance gains on downstream tasks relative to models trained on Kinetics, and SurgBench-pretrained models outperform natural-video-pretrained models.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    SurgBench provides a unified platform for surgical video analysis, with the stated aim of accelerating research progress in the domain.

    Source: §6 · p. 9

    Long-tail class distribution challenges model training, and further research is needed to improve minority-class performance.

    Source: §7 · p. 9

Abstract

from arXiv · show

Surgical video understanding is pivotal for enabling automated intraoperative decision-making, skill assessment, and postoperative quality improvement. However, progress in developing surgical video foundation models (FMs) remains hindered by the scarcity of large-scale, diverse datasets for pretraining and systematic evaluation. In this paper, we introduce \textbf{SurgBench}, a unified surgical video benchmarking framework comprising a pretraining dataset, \textbf{SurgBench-P}, and an evaluation benchmark, \textbf{SurgBench-E}. SurgBench offers extensive coverage of diverse surgical scenarios, with SurgBench-P encompassing 53 million frames across 22 surgical procedures and 11 specialties, and SurgBench-E providing robust evaluation across six categories (phase classification, camera motion, tool recognition, disease diagnosis, action classification, and organ detection) spanning 72 fine-grained tasks. Extensive experiments reveal that existing video FMs struggle to generalize across varied surgical video analysis tasks, whereas pretraining on SurgBench-P yields substantial performance improvements and superior cross-domain generalization to unseen procedures and modalities. Our dataset and code are available upon request.

1 Introduction

Surgical video analysis has clinical applications, but limited dataset diversity and task coverage constrain surgical video foundation model development. SurgBench combines a large-scale pretraining dataset with a broad evaluation benchmark.

1 Introduction

  • Surgical video analysis can support intraoperative decision-making, skill evaluation, and postoperative quality improvement, but existing datasets and benchmarks have limited diversity and task comprehensiveness.The paper identifies narrow disease, procedure, specialty, and task coverage as gaps.Source: §1 · pp. 1–2
  • SurgBench-P comprises 53 million frames across 22 procedures and 11 specialties, while SurgBench-E organizes evaluation into 72 fine-grained tasks across six categories.The framework is presented as addressing procedural and disease-type homogeneity and task fragmentation.Source: §1 · p. 2
  • 7.9% performance gains on downstream tasks were reported for SurgMAE pretrained on SurgBench relative to models trained on Kinetics.This is the initial empirical validation reported in the introduction.Source: §1 · p. 2

2 Related Work

Early surgical video analytics often centered on narrow tasks and datasets, while newer surgical foundation models use self-supervised learning and increasingly incorporate vision-language and embodied AI approaches.

2 Related Work

  • Early surgical video analytics focused on single diseases or limited tasks, often using small single-center datasets with limited generalization.Examples include phase recognition, instrument detection, and polyp detection.Source: §2 · pp. 2–3
  • High annotation costs made many early efforts heavily supervised, complicating scale-up to broader clinical environments and multi-task learning.Source: §2 · p. 3
  • Surgical foundation models use self-supervised learning on diverse datasets, and newer vision-language and embodied AI approaches target semantic understanding and interpretability.The cited models are described as demonstrating robustness and cross-domain transferability.Source: §2 · p. 3

3 Dataset Construction Pipeline

SurgBench combines datasets into an unlabeled pretraining set and a labeled evaluation benchmark, standardizing videos and organizing evaluation across a broad surgical task taxonomy. The pipeline also applies split and label-balancing rules before reporting benchmark statistics.

3 Dataset Construction Pipeline

  • The construction pipeline organizes the collected videos into SurgBench-P for pretraining and SurgBench-E for fine-tuning and evaluation.The figure describes SurgBench-P as unlabeled and SurgBench-E as labeled, with 11 specialties, 22 procedures, and 72 evaluation tasks.Source: §3 · p. 3, Figure 1 · p. 3

3.1 Dataset Source

  • SurgBench combines 15 public datasets and one private dataset to cover diverse procedures, specialties, and tasks.Some source datasets have licenses that prohibit modification and redistribution, so they are excluded from the released SurgBench-E.Source: §3.1 · pp. 3–4

3.2 Dataset Preprocessing

  • Videos are standardized to H.264, a minimum dimension of 320 pixels with aspect ratio preserved, and 20–30 FPS.The network input resolution is 224x224.Source: §3.2 · p. 4

3.3 Dataset Formulation

  • Evaluation clips are split 5:5 by video, and the imbalance factor is kept within 10 through clip-duration control, downsampling, and removal of scarce labels.Multi-label samples are separated into single-label samples and evaluated with top-k accuracy.Source: §3.3 · pp. 4–5

3.4 Benchmark statistics:

  • 59,742,875 frames across 225,250 clips and 575 hours make up SurgBench, spanning 22 procedures and 11 specialties.SurgBench-E contains 23,004 clips and 72 fine-grained labels organized into 6 categories and 10 sub-categories.Source: §3.4 · p. 5
  • Training and test labels have a long-tailed distribution, and sample size is positively correlated with the number of labels in each category.The benchmark defines a 6-category, 10-sub-category, 72-task taxonomy.Source: §3.4 · pp. 5–6, Table 3 · p. 6

4 Methodology

The method continually pre-trains VideoMAE on SurgBench-P, then fine-tunes and tests the resulting models on SurgBench-E. It uses masked spatiotemporal reconstruction for pretraining and classifier training for downstream evaluation.

Continual pre-training

  • Continual pre-training refines 225,250 clips through filtering and sampling stages, producing 39,807 clips for the final phase.The process starts from VideoMAE models pre-trained on Kinetics-400 and uses SurgBench-P data.Source: §4 · p. 6
  • VideoMAE reconstructs randomly masked spatiotemporal tubes using a 0.9 masking ratio, with Standard and Large encoder-decoder variants.Both variants use 16 frames per clip sampled at a temporal rate of 4.Source: §4 · p. 6

Evaluation

  • SurgBench-E comprises six categories and 72 tasks, while the results table reports top-1 and top-3 accuracy across tasks T1–T10.The table caption identifies standard-sized models except where marked Large, but does not supply performance values.Source: Figure 5 · p. 6, Table 4 · p. 7

Fine-tuning

  • SurgMAE is produced by fine-tuning SurgMAE-CPT on SurgBench-E, training the classifier to assess backbone features and training dynamics.Standard and Large variants use different learning rates and batch sizes.Source: §4 · p. 6

Training cost

  • Continual pre-training ran for 54 epochs for VideoMAE-Base and 38 epochs for VideoMAE-Large, taking one week on eight NVIDIA A100 GPUs.Fine-tuning both model sizes took 100 epochs and 14 hours on one host with eight RTX3090 GPUs.Source: §4 · p. 7

5 Experiments and Results

Experiments evaluate VideoMAE pretraining on SurgBench-P followed by fine-tuning on SurgBench-E, examining performance, training dynamics, model size, out-of-distribution transfer, and data mixing.

5.1 Main Experiment Results

  • Top-1 accuracy increased by 7% and top-3 accuracy by 7.9% with continual surgical-video pretraining on SurgBench-E.Most subcategories improved, while T4 and T8 remained nearly unchanged.Source: §5.1 · p. 7

5.2 Training Dynamics

  • Across four progressive pretraining stages, loss decreased smoothly within stages, while stage transitions were nonsmooth but did not harm task-test performance.Even when final-stage loss decreased slowly, it still contributed to stabilizing and improving fine-tuning performance.Source: §5.2 · p. 8, Figure 6 · p. 8

5.3 Impact of Model Size on Performance

  • SurgMAE and SurgMAE(L) achieved a 3.9% increase in top-1 accuracy and a 5.7% increase in top-3 accuracy.The paper also reports consistent growth across task dimensions.Source: §5.3 · p. 8

5.4 Can Surg-FM generalize to unseen case?

  • SurgBench pretraining improved LapGyn event-recognition performance by 2.69% on a distribution dissimilar to the training data.The LapGyn4 evaluation covers gynecologic laparoscopic surgery.Source: Figure 7 · p. 8, §5.4 · p. 9

5.5 Data mix strategy between Pre-training and Fine-tuning

  • Using JIGSAWS and CholecT50 for continued pretraining and fine-tuning yielded faster convergence and better final performance than the comparison strategy.The authors describe transitioning from large-scale unrestricted data to smaller-scale distribution-aligned data.Source: §5.5 · p. 9, Figure 8 · p. 9

6 Conclusions

The conclusion presents SurgBench as a unified surgical-video benchmark, combining diverse pretraining data with fine-grained evaluation and reporting gains from self-supervised surgical-video pretraining.

  • SurgBench-P comprises 53 million frames across 22 procedures and 11 specialties, while SurgBench-E covers 6 categories, 10 subcategories, and 72 fine-grained tasks.Source: §6 · p. 9
  • Experiments report significant gains from self-supervised surgical-video learning and better performance than pretraining on natural-video datasets.Source: §6 · p. 9
  • The authors describe SurgBench as a unified platform intended to accelerate surgical-video analysis research.Source: §6 · p. 9

7 Limitations

The paper identifies long-tail class imbalance, unexplored language supervision, and unresolved architectural choices as limitations for SurgBench and surgical video foundation models.

  • Long-tail class distribution challenges training, and improving minority-class performance requires further research.Source: §7 · p. 9
  • Integration of language supervision remains unexplored, although the paper describes it as potentially beneficial.Source: §7 · p. 9
  • Optimal architectures for surgical video foundation models require further investigation to maximize cross-task performance.Source: §7 · p. 9

A Dataset License

SurgBench reviews source-dataset licenses, and its directly usable benchmarking and fine-tuning components primarily use datasets permitting academic or non-commercial research.

  • Core benchmarking and fine-tuning datasets primarily permit academic or non-commercial research use.Examples include Cholec80, Kvasir-Capsule, and SUN-SEG, each with its own stated license terms.Source: A Dataset License · p. 12
  • AVOS, SimSurgSkill2021, and AIxSuture were used in broader pretraining but are excluded from the redistributed fine-tuning suite because of licensing limitations.Source: A Dataset License · p. 12
  • Users are encouraged to consult original dataset sources for complete licensing details.Source: A Dataset License · p. 12

B Pie chart of other three categories

The passage identifies pie charts for camera motion, organ detection, and tool recognition in Figure 9.

C Pretraining Data Composition

The pretraining-data composition is presented in Table 6, whose caption identifies specialties, procedures, and source IDs.

D Label to task mapping

Tables 7 and 8 provide the label-to-task mapping, with their captions identifying the label descriptions as two parts.

E Details of processing steps in each dataset

This section describes source datasets and their processing for surgical video tasks, including video segmentation, annotation, and clip preparation. The datasets vary in scope, task labels, and available data.

Private Data

AutoLaparo

  • AutoLaparo covers workflow recognition, laparoscope-motion prediction, and instrument and anatomy segmentation for laparoscopic hysterectomy.The dataset provides 21 workflow videos, 300 motion clips, and 1,800 annotated images.Source: E Details of processing steps in each… · p. 16

Cholec80

  • Cholec80 provides 80 laparoscopic cholecystectomy videos with seven frame-level phase labels and seven tool-presence labels sampled at 1 fps.Videos are divided into approximately 300-frame clips; the most frequent phase and maximum tool occurrence determine clip labels.Source: E Details of processing steps in each… · p. 16

Hyper-Kvasir

  • Hyper-Kvasir comprises 374 videos totaling 11.62 hours and more than one million frames, plus 10,662 annotated images across 23 lesion types.Long videos are segmented into five-second clips based on categorization.Source: E Details of processing steps in each… · p. 17

Kvasir-Capsule

CholecT50

  • CholecT50 supplies fine-grained laparoscopic cholecystectomy labels for phase, instrument, verb, and target classification.Clips are segmented into five-second videos at 10 fps, with labels standardized to 0–31.Source: E Details of processing steps in each… · p. 18

LDPolypVideo

  • LDPolypVideo contains 160 colonoscopy videos totaling 40,266 frames, including motion blur, bowel folds, and peristalsis.Training videos with polyps are split into two to four segments and compressed to ten seconds each.Source: E Details of processing steps in each… · p. 18

AVOS

Loading 2506.07603v2…