Source-linked AI summary
Foundation Model for Endoscopy Video Analysis via Large-scale Self-supervised Pre-train
Zhao Wang, Chang Liu, Shaoting Zhang, Qi Dou
TL;DR
Endoscopic video analysis lacks a dedicated foundation model, despite endoscopy’s routine role in clinical tasks. Endo-FM combines a video transformer and self-supervised view matching, and outperforms reported state-of-the-art methods across classification, segmentation, and detection.
Problem
A foundation model for endoscopic video analysis is lacking, while endoscopy is used in gastrointestinal diagnosis, minimally invasive procedures, and robotic surgery.
Source: §1 · pp. 1–2Method
Endo-FM uses a video transformer and teacher-student self-supervised matching across spatial-temporal views of endoscopic video.
Source: §1 · p. 2Results
Endo-FM surpasses SOTA self-supervised and adapter methods, with VCL margins of 3.1% F1, 4.8% Dice, and 5.5% F1 across classification, segmentation, and detection.
Source: §1 · p. 3Takeaways & Limitations
The authors argue that Endo-FM or similar domain-specific foundation models may help medical applications despite the existence of general-purpose models.
Source: §4 · p. 9
Abstract
from arXiv · showhide
Foundation models have exhibited remarkable success in various applications, such as disease diagnosis and text report generation. To date, a foundation model for endoscopic video analysis is still lacking. In this paper, we propose Endo-FM, a foundation model specifically developed using massive endoscopic video data. First, we build a video transformer, which captures both local and global long-range dependencies across spatial and temporal dimensions. Second, we pre-train our transformer model using global and local views via a self-supervised manner, aiming to make it robust to spatial-temporal variations and discriminative across different scenes. To develop the foundation model, we construct a large-scale endoscopy video dataset by combining 9 publicly available datasets and a privately collected dataset from Baoshan Branch of Renji Hospital in Shanghai, China. Our dataset overall consists of over 33K video clips with up to 5 million frames, encompassing various protocols, target organs, and disease types. Our pre-trained Endo-FM can be easily adopted for a given downstream task via fine-tuning by serving as the backbone. With experiments on 3 different types of downstream tasks, including classification, segmentation, and detection, our Endo-FM surpasses the current state-of-the-art (SOTA) self-supervised pre-training and adapter-based transfer learning methods by a significant margin, such as VCL (3.1% F1, 4.8% Dice, and 5.5% F1 for classification, segmentation, and detection) and ST-Adapter (5.9% F1, 9.6% Dice, and 9.9% F1 for classification, segmentation, and detection). Code, datasets, and models are released at https://github.com/med-air/Endo-FM.
1 Introduction
Endo-FM addresses the lack of a foundation model for endoscopic video analysis with a video transformer and self-supervised spatial-temporal matching, and is evaluated across three downstream task types. Its dataset combines public and privately collected endoscopy videos.
1 Introduction
- The paper focuses on endoscopic video, a routine imaging modality studied for gastrointestinal disease diagnosis, minimally invasive procedures, and robotic surgery.The introduction argues that a foundation model trained on a specific data type is useful at present.Source: §1 · pp. 1–2
- The model combines a ViT-based video transformer with teacher-student matching across spatial-temporal views, and uses a dataset of over 33K clips with up to 5 million frames.The dataset combines nine public datasets with a private collection from Baoshan Branch of Renji Hospital.Source: §1 · p. 2
- The transformer targets long-range spatial and temporal dependencies, while matching views of the same video is intended to learn spatial-temporal information.The paper describes pure image-based pre-training as more feasible because endoscopy clinical routines typically do not involve text data.Source: §1 · p. 2
- Endo-FM surpasses SOTA self-supervised and adapter methods, with VCL margins of 3.1% F1, 4.8% Dice, and 5.5% F1 across classification, segmentation, and detection.The reported ST-Adapter margins are 5.9% F1, 9.6% Dice, and 9.9% F1 for those tasks, respectively.Source: §1 · p. 3
2 Method
Endo-FM encodes video with a space-time transformer, matches teacher and student representations across spatial-temporal views, and combines matching losses with training updates and augmentations.
2 Method
- The method combines a video transformer with self-supervised spatial-temporal matching, with an overview provided in Figure 1.The matching approach uses teacher and student models to predict between views of an input video.Source: §2 · p. 3, §2.2 · p. 4
2.1 Video Transformer for Spatial-Temporal Encoding
- A 12-block video transformer applies spatial and temporal attention to patch tokens, while positional vectors support interpolation for smaller spatial sizes and lower frame rates.A projected class token from the final encoder block provides the pre-training feature.Source: §2.1 · pp. 3–4
2.2 Self-supervised Pre-train via Spatial-Temporal Matching
- Cross-view matching predicts teacher outputs for global views from student outputs for local views, addressing differences in lesion proportion and presence across video views.Local views are sampled from randomly cropped video regions at varied frame rates.Source: §2.2 · p. 4
- Dynamic motion matching predicts a teacher-processed global view from another student-processed global view to learn motion-related context across frame rates.The paper describes predicting nuanced tissue and lesion differences between high- and low-frame-rate views.Source: §2.2 · p. 5
2.3 Overall Optimization Objective and Pre-training Specifics
- The pre-training objective sums cross-view and dynamic-motion losses, and the teacher weights are updated by exponential moving average of student weights.Temporally consistent spatial augmentations are also applied to frames within each view.Source: §2.3 · p. 5
- Each input uses two global and eight local views, with global views at 224×224 and local views at 96×96; pre-training runs for 30 epochs.The dataset table covers pre-training and downstream datasets; Fig. 2 shows example frames from the 10 datasets.Source: §2.3 · pp. 5–6, Table 1 · p. 6, Figure 2 · p. 6
3 Experiment
Experiments evaluate Endo-FM on disease diagnosis, polyp segmentation, and detection, then examine its components and view-construction choices. Endo-FM outperforms the compared methods across all three tasks, while the ablations assess how spatial-temporal matching contributes to diagnosis performance.
3.1 Datasets and Downstream Setup
- The evaluation uses public endoscopy datasets and a new hospital dataset, with downstream tests for diagnosis, polyp segmentation, and detection.Fine-tuning uses a linear layer for PolypDiag, TransUNet for CVC-12k, and STFT for KUMC; reported metrics are F1, Dice, and F1, respectively.Source: §3.1 · pp. 6–7
3.2 Comparison with State-of-the-art Methods
- +3.1% F1, +4.8% Dice, and +5.5% F1 over the second-best methods for classification, segmentation, and detection, respectively.The comparison covers PolypDiag, CVC-12k, and KUMC.Source: §3.2 · p. 7, Table 2 · p. 7
3.3 Analytical Studies
- On PolypDiag, combining spatial and temporal local-view sampling yields a +4.3% F1 improvement, and dynamic matching raises performance to 89.7%.Adding video augmentations further raises performance from 89.7% to 90.7%; joint prediction of global views from local and global views performs best among the tested matching combinations.Source: §3.3 · p. 8, Figure 3 · p. 8
- Increasing the numbers of global and local views and the length variation of local views improves PolypDiag performance.The tested view counts were G ∈ [1, 2] and L ∈ [1, 2, 4, 6, 8].Source: §3.3 · p. 8
4 Conclusion and Discussion
The paper presents Endo-FM as a foundation model designed specifically for endoscopy video analysis, supported by a dataset of over 33K clips. Tests across three downstream tasks show effectiveness and outperformance of other state-of-the-art video pre-training methods.
4 Conclusion and Discussion
- Endo-FM is presented as the first foundation model specifically designed for analyzing endoscopy videos.It uses a video transformer to capture spatial-temporal information and is pre-trained for robustness to spatial-temporal variation.Source: §4 · p. 8
- The constructed endoscopy video dataset contains over 33K video clips.Source: §4 · p. 9
- On downstream segmentation for CVC-12k, SAM achieved performance comparable to Endo-FM when both used the same fine-tuning scheme, although SAM was trained with 10x samples.Source: §4 · p. 9