PaperlayerView exact revision on arXiv

Source-linked AI summary

SurgLaVi: Large-Scale Hierarchical Dataset for Surgical Vision-Language Representation Learning

Alejandra Perez, Chinedu Nwoye, Ramtin Raji Kermani, Omid Mohareri, Muhammad Abdullah Jamal

arXiv:2509.10555v2cs.CV

TL;DR

Surgical VLP datasets remain constrained in scale, diversity, semantic quality, and hierarchical workflow structure. SurgLaVi combines automated hierarchical dataset construction with SurgCLIP, which improves recognition across surgical tasks, including zero-shot phase recognition.

  • Problem

    Surgical VLP datasets are limited in scale, procedural diversity, semantic quality, and hierarchical workflow structure, while current models lag task-specific supervised approaches.

    Source: §1 · pp. 1–2

  • Method

    SurgLaVi uses an automated pipeline to create hierarchically segmented, filtered, context-enriched surgical clip–caption pairs, and evaluates them with the dual-encoder contrastive model SurgCLIP.

    Source: §1 · p. 2

  • Results

    SurgCLIP improves phase, step, action, and tool recognition; its zero-shot phase F1 gain on AutoLaparo is +28%.

    Source: §1 · p. 2, §7 · p. 15

  • Takeaways & Limitations

    The results show that hierarchically structured surgical video–language data can support strong downstream recognition with a lightweight base model.

    Source: §1 · p. 2, §7 · p. 15

    The pipeline depends on narrated surgical videos, which are not uniformly available, and automated curation without expert verification may introduce noise or bias.

    Source: §6 · p. 15

Abstract

from arXiv · show

Vision-language pre-training (VLP) offers unique advantages for surgery by aligning language with surgical videos, enabling workflow understanding and transfer across tasks without relying on expert-labeled datasets. However, progress in surgical VLP remains constrained by the limited scale, procedural diversity, semantic quality, and hierarchical structure of existing datasets. In this work, we present SurgLaVi, the largest and most diverse surgical vision-language dataset to date, comprising nearly 240k clip-caption pairs from more than 200 procedures, and featuring hierarchical levels at coarse-, mid-, and fine-level. At the core of SurgLaVi lies a fully automated pipeline that systematically generates fine-grained transcriptions of surgical videos and segments them into coherent procedural units. To ensure high-quality annotations, it applies dual-modality filtering to remove irrelevant and noisy samples. Within this framework, the resulting captions are enriched with contextual detail, producing annotations that are both semantically rich and easy to interpret. To ensure accessibility, we release SurgLaVi-$\b{eta}$, an open-source derivative of 113k clip-caption pairs constructed entirely from public data, which is over four times larger than existing surgical VLP datasets. To demonstrate the value of the SurgLaVi datasets, we introduce SurgCLIP, a CLIP-style video-text contrastive framework with dual encoders, as a representative base model. SurgCLIP achieves consistent improvements across phase, step, action, and tool recognition, surpassing prior state-of-the-art methods, often by large margins. These results validate that large-scale, semantically rich, and hierarchically structured datasets directly translate into stronger and more generalizable representations, establishing SurgLaVi as a key resource for developing surgical foundation models.

1. Introduction

Surgical vision-language pre-training is held back by datasets with limited scale, diversity, semantic quality, and workflow structure. SurgLaVi pairs a large hierarchical dataset and automated curation pipeline with SurgCLIP, which improves performance across surgical recognition tasks.

1. Introduction

  • Existing surgical VLP datasets are limited by fragmented clips, noisy pairs, coarse workflow labels, restricted scale and diversity, and lack of open access.The paper argues that these dataset constraints, rather than model design alone, are a fundamental limitation for surgical VLP.Source: §1 · pp. 1–2
  • The fully automated pipeline generates temporally precise, semantically rich clip–caption pairs through transcription, hierarchical segmentation, dual-modality filtering, and contextual enrichment.The pipeline is designed to represent real surgical workflows at multiple levels of granularity.Source: §1 · p. 2
  • SurgLaVi contains nearly 240k clip–caption pairs spanning over 200 procedure types, with coarse-, mid-, and fine-level structure.Its public-data derivative, SurgLaVi−β, has 113k pairs and is more than four times larger than existing surgical VLP datasets.Source: Figure 1 · p. 1, §1 · p. 2
  • SurgCLIP pretrained on SurgLaVi improves zero-shot phase recognition F1 across benchmarks and advances step, action, and tool recognition.The reported zero-shot phase F1 gains include +28% on AutoLaparo; the paper also reports improvements in few- and full-shot linear probing.Source: Figure 1 · p. 1, §1 · p. 2

2. Related Work

Prior surgical VLP datasets have progressively added workflow structure, semantic enrichment, or broader coverage, but remain constrained in scale, alignment, diversity, or semantic quality. Surgical VLP models likewise grew more elaborate to address limited supervision, with added complexity and computational trade-offs.

2.1. Surgical Vision-Language Pre-Training Datasets

  • Surgical VLP datasets evolved from SVL’s narrated laparoscopic clips to workflow-enriched variants and broader robotic coverage, but retained limitations in scale, alignment, diversity, or semantic coherence.GenSurg+ broadened coverage to 28 robotic and laparoscopic procedures, while OphVL exceeded 375k pairs but remained limited to ophthalmological surgery.Source: §2.1 · pp. 2–3

2.2. Vision-Language Pre-training Models in Surgery

  • Surgical VLP models added hierarchical contrastive learning, caption augmentation, temporal regularization, or multimodal fusion, while increasing architectural or computational complexity.The cited approaches include HecVL, PeskaVLP, VidLPRO, and OphCLIP; the passage notes that each innovation brings methodological or computational trade-offs.Source: §2.2 · pp. 3–4

3. SurgLaVi Dataset and Processing Pipeline

SurgLaVi is built through an automated four-stage pipeline that turns surgical videos into hierarchically structured, filtered video–language annotations. It uses timestamped speech transcriptions, semantically coherent segments at three granularities, and clip-level checks to retain relevant pairs.

3. SurgLaVi Dataset and Processing Pipeline

  • The four-stage pipeline uses foundation models for transcription, temporal segmentation, and filtering to generate hierarchical surgical workflow annotations.The pipeline is designed to produce a scalable video-language dataset from surgical videos.Source: §3 · p. 4

3.1. Stage 1: Speech-to-text conversion

  • WhisperX produces sentence-level timestamped transcriptions and millisecond-accurate word-level timestamps; videos with empty transcriptions are discarded.The word-level alignment supports flexible video-text pairing in later stages.Source: §3.1 · p. 4

3.2. Stage 2: Hierarchical Semantic Clip–Caption Pair Generation

  • GPT-4o restructures sentence transcripts into coarse, mid, and fine segments that represent progressively more granular procedural goals.The hierarchy captures high-level context through localized surgical tasks and increases clip-caption quantity without additional data collection.Source: §3.2 · p. 4
  • At each granularity, segment boundaries are aligned to word timestamps to form captions and corresponding video clips.The result is three multi-granular datasets of clip-caption pairs.Source: §3.2 · p. 4

3.3. Stage 3: Dual-Model Filtering.

  • Dual-model filtering retains clip-caption pairs that are visually surgical and textually descriptive, rather than discarding whole videos.Visual filtering classifies sampled fine-level frames with zero-shot SigLIP, uses a greater-than-50% surgical-frame threshold, and propagates labels to higher levels, assigning mixed-label clips the majority label.Source: §3.3 · pp. 4–5

Final filtering criterion: A clip–caption pair (xg

The pipeline retains surgical, descriptive clip–caption pairs and enriches their captions with procedural context. SurgLaVi organizes the resulting annotations at multiple temporal scales, with a larger full dataset and an open-source subset.

Final filtering criterion: A clip–caption pair (xg

3.4. Stage 4: Contextual Caption Enrichment

  • Stage 4 enriches captions using preceding captions, the video title, and procedure type, producing visually descriptive captions with contextual and procedural information.The preceding-caption context uses a fixed window at the same granularity level.Source: §3.4 · p. 5

3.5. SurgLaVi Dataset Overview

  • SurgLaVi contains 239,776 clip–caption pairs across 5,317 videos and 843 hours; its open-source subset, SurgLaVi-β, contains 112,757 pairs across 2,464 videos and 465 hours.Source: §3.5 · p. 6
  • Coarse-to-fine levels move from longer, context-rich clips to shorter, action-focused clips with higher temporal resolution, explicitly encoding multiple temporal scales.Figure 3 illustrates the levels; the dataset design is described as enabling multi-scale temporal reasoning over surgical activities.Source: Figure 3 · p. 5, §3.5 · p. 6
  • In both datasets, finer levels have shorter clips, more clips per video, and more total clip–caption pairs; duration and clip-count distributions remain consistent between datasets.Source: §3.5 · p. 6, Figure 5 · p. 6, Figure 4 · p. 6
  • Compared with SVL and GenSurg+, SurgLaVi offers substantially greater scale, variable-length hierarchical segmentation, and an open-source subset with captions.Source: §3.5 · p. 7, Table 1 · p. 7
  • SurgLaVi covers 10 specialties, 39 subjects, and 217 procedure types; SurgLaVi-β covers 9, 34, and 170, respectively, while GenSurg+ includes 28 procedure types under a different taxonomy.Source: Figure 6 · p. 6, §3.5 · p. 7

4. SurgCLIP

SurgCLIP uses video and text encoders trained with a symmetric contrastive objective. Mixed-level training batches and duration-adaptive temporal sampling expose it to procedural semantics at multiple temporal scales.

4. SurgCLIP

  • SurgCLIP uses a TimeSFormer ViT-B/16 video encoder and a BERT-base text encoder, projecting their features into a shared embedding space.Source: §4 · p. 7
  • Randomly mixing coarse-, mid-, and fine-level pairs within batches makes the InfoNCE objective align representations across temporal granularities.The model computes video-to-text and text-to-video losses and averages the two directions.Source: §4 · p. 7
  • Dynamic temporal sampling varies frame stride by clip duration: fine clips use higher sampling rates, while coarse clips use lower rates to capture broader procedural context.The number of frames per clip remains fixed.Source: §4 · p. 7

5. Experimental Validation

The experiments evaluate SurgCLIP across zero-shot transfer, frozen-feature and prompt adaptation, temporal modeling, comparisons with task-specific models, and pretraining ablations. Across these settings, they examine how performance relates to benchmark scope, temporal context, and dataset construction.

5.1. Zero-Shot Classification

  • Up to +10, +28, and +25 F1 over strongest prior methods on Cholec80, AutoLaparo, and GraSP, respectively; SurgCLIP(β) beats baselines on most benchmarks, while action recognition remains challenging.The reported gains are for phase recognition; the approach also delivers consistent tool-recognition gains.Source: Table 2 · p. 8, §5.1 · pp. 8–9

5.2.1. Linear Probing

  • The few-shot evaluations distinguish labeled procedure videos (10%, 50%, or 100%) from labeled frames (1, 2, 4, 8, or 16 per class), using 32-frame windows and matched random subsets.The frame-level setup uses unlabeled neighboring frames to construct each temporal window while retaining supervision at the selected frame.Source: §5.2.1 · pp. 9–10, Figure 9 · p. 10

5.2.2. Context Optimization (CoOp)

  • SurgCLIP tops baselines across CoOp, CLS few-shot, and video few-/full-shot settings, except GraSP step recognition under CoOp.CoOp generally improves on zero-shot inference with more than two labeled samples but remains less effective than linear probing on visual features. Qualitative phase predictions are more temporally consistent, with more accurate boundaries than existing models.Source: Figure 8 · p. 9, Figure 9 · p. 10, §5.2.2 · pp. 10–11
  • Performance generally improves with longer temporal windows, except GraSP step recognition, which peaks at 8 frames.For 32-frame embeddings, classifier-depth gains plateau after two layers (three for AutoLaparo); single-frame embeddings generally benefit from added layers.Source: Figure 11 · p. 11, §5.2.2 · p. 11, Figure 10 · p. 11

5.3. Temporal probing with MS-TCN

  • SurgCLIP leads frame-level F1 and Accuracy on Cholec80 and AutoLaparo, while SurgCLIP(β) leads GraSP; both models often more than double the next-best baseline’s F1@10.On GraSP phase recognition, temporal probing does not surpass linear probing; the authors cite long idle intervals and few full-length videos.Source: Table 3 · p. 12, §5.3 · p. 12

5.4. Upper-Bound Results: Comparison with Task-Specific Models

  • With frozen features and one linear layer, SurgCLIP is competitive with task-specific models; specialized architectures retain a 7.4% average F1 advantage despite greater complexity and training requirements.On GraSP, the linear probe matches CNN-based baselines with 16 samples per class and is 2.34% F1 below TAPIS with full training data.Source: Table 4 · p. 13, §5.4 · p. 13, Table 5 · p. 14

5.5. Ablation studies

  • Joint fine-, mid-, and coarse-level pretraining gives the best overall zero-shot performance; fine or mid data alone outperform coarse data alone.The authors describe fine- and mid-level captions as shorter and more focused, while coarse-level descriptions contribute richer procedural information.Source: Table 6 · p. 14, §5.5 · p. 14
  • The complete curation pipeline performs best across benchmarks, with gains of +14.64% on Cholec80, +10.27% on AutoLaparo, +10.35% on GraSP phases, and +3.18% on GraSP steps.Individual components have inconsistent effects; DMF with MLSC improves Cholec80 and GraSP phases, while the full combination exceeds the best pairwise combination by +8.17% on Cholec80.Source: Table 7 · p. 14, §5.5 · pp. 14–15

6. Limitations

The pipeline depends on narrated surgical videos, which are not uniformly available across procedures or clinical settings. Narrations can also be temporally misaligned with the video, and fully automated curation may introduce noise or bias.

6. Limitations

  • The pipeline is limited to narrated surgical videos, which are not uniformly available across procedure types or clinical settings.Narrations may describe recent actions or upcoming steps rather than the moment shown.Source: §6 · p. 15
  • Without expert verification, foundation-model-based automated curation may introduce noise or bias.Narrations may also be temporally misaligned with visual content.Source: §6 · p. 15

7. Conclusion

SurgLaVi provides hierarchical video–language data from narrated surgical videos, and SurgCLIP demonstrates strong performance across downstream recognition tasks. The conclusion argues that dataset quality and structure can be more decisive than model complexity for capturing procedural workflows.

7. Conclusion

  • SurgCLIP demonstrates state-of-the-art performance across phase, step, tool, and action recognition in zero-shot and few-shot probing.The paper presents SurgCLIP as a lightweight CLIP-style baseline paired with SurgLaVi.Source: §7 · p. 15
  • SurgLaVi captures procedural knowledge at coarse, mid, and fine levels from narrated surgical videos.The conclusion describes the dataset as the most hierarchically structured video–language dataset in the surgical domain.Source: §7 · p. 15
  • The conclusion argues that dataset quality and structure can be more decisive than model complexity for capturing complex procedural workflows.It also describes opportunities for multimodal reasoning, video–text understanding, and self-supervised procedural modeling.Source: §7 · p. 15

1. Appendix

The appendix details the SurgLaVi processing pipeline, SurgCLIP training, and evaluation protocols, alongside prompts and benchmark table descriptions.

1.1. SurgLaVi Data Processing Pipeline Details

  • The pipeline transcribes narration with WhisperX, segments word-timed text into phases, steps, and tasks, filters clips visually and captions textually, then enriches captions using five prior captions, video title, and procedure type.Visual filtering uses 24-frame majority voting at a greater-than-50% threshold, with 96.6 F1, 94.29 precision, and 99.0 recall; textual filtering reports 93.95 precision, 94.59 recall, and 94.27 F1.Source: §1.1 · p. 18

1.2. Pre-training Implementation Details

  • SurgCLIP uses a ViT-B/16 vision backbone, BERT-base text encoder, shared 256-dimensional space, and a TimeSformer block over 16 frames.Training uses an InfoNCE video–text contrastive objective and runs for 50 epochs on one NVIDIA H200 GPU.Source: §1.2 · pp. 18–19

1.3.1. Linear Probing Details

  • Frozen visual features concatenate 768-dimensional backbone and 256-dimensional projection outputs into 1024 dimensions; hyperparameters are tuned on validation data using single-frame inference, while final probes use 32-frame windows averaged to represent the center labeled frame.Source: §1.3.1 · p. 19

1.3.2. CoOp Probing Details

  • CoOp keeps both encoders frozen and optimizes only learnable context tokens, using fixed SGD settings and early stopping for up to 500 epochs.The appendix describes comparisons across phase and step recognition, reporting F1, accuracy, standard deviations, and video-wise accuracy.Source: §1.3.2 · p. 19, Table 5 · p. 21

1.4. Temporal Probing Details

  • Temporal transfer is evaluated with an MS-TCN trained on frozen frame-level features using identical three-stage, eight-layer-per-stage, 64-feature-map settings across models and datasets.The protocol uses full-video batches and AdamW for 100 epochs.Source: §1.4 · p. 19

1.5. Zero-shot Prompt Design

  • The appendix supplies benchmark-specific zero-shot prompts and a surgical taxonomy visualization whose wedge sizes correspond to procedure counts.For BernBypass70 and StrasBypass70 phase evaluation, the “other” class is excluded following prior studies [8].Source: §1.5 · p. 19, Figure 6 · p. 22, Figure 18 · p. 24
Loading 2509.10555v2…