Source-linked AI summary
Exploiting the potential of unlabeled endoscopic video data with self-supervised learning
Tobias Ross, David Zimmerer, Anant Vemuri, Fabian Isensee, Manuel Wiesenfarth, Sebastian Bodenstedt, Fabian Both, Philip Kessler, Martin Wagner, Beat Müller, Hannes Kenngott, Stefanie Speidel, Annette Kopp-Schneider, Klaus Maier-Hein, Lena Maier-Hein
TL;DR
Labor-intensive annotation and weak cross-site generalization constrain supervised learning for surgical video. The paper pre-trains CNNs using an auxiliary task on unlabeled target-domain endoscopic images, then evaluates them on instrument segmentation. PAT improves on no pre-training and non-medical pre-training, while PAT-Ext achieves the best results among the compared pre-training methods.
Problem
Manual curation of sufficient training data is labor-intensive, and supervised models can lose more than 50% of mean performance when transferred to endoscopic video from another site.
Source: §1 · p. 2Method
The method pre-trains a CNN with an adversarial re-colorization task on unlabeled target-domain images, then refines it with labeled data for the target task.
Source: §2.1 · p. 3Results
PAT outperformed the no-pre-training baseline and non-medical labeled pre-training, while PAT-Ext achieved the best results and medical labeled pre-training generally outperformed PAT alone.
Source: §3 · p. 10, §4 · p. 11Takeaways & Limitations
The approach makes use of available public or non-public, labeled or unlabeled data and may apply to a wide range of target tasks.
Source: §4 · p. 12Absolute performance was much worse on the authors’ own dataset, which they attribute to its challenging images and comparatively low variability in the MICCAI images.
Source: §4 · p. 12
Abstract
from arXiv · showhide
Surgical data science is a new research field that aims to observe all aspects of the patient treatment process in order to provide the right assistance at the right time. Due to the breakthrough successes of deep learning-based solutions for automatic image annotation, the availability of reference annotations for algorithm training is becoming a major bottleneck in the field. The purpose of this paper was to investigate the concept of self-supervised learning to address this issue. Our approach is guided by the hypothesis that unlabeled video data can be used to learn a representation of the target domain that boosts the performance of state-of-the-art machine learning algorithms when used for pre-training. Core of the method is an auxiliary task based on raw endoscopic video data of the target domain that is used to initialize the convolutional neural network (CNN) for the target task. In this paper, we propose the re-colorization of medical images with a generative adversarial network (GAN)-based architecture as auxiliary task. A variant of the method involves a second pre-training step based on labeled data for the target task from a related domain. We validate both variants using medical instrument segmentation as target task. The proposed approach can be used to radically reduce the manual annotation effort involved in training CNNs. Compared to the baseline approach of generating annotated data from scratch, our method decreases exploratively the number of labeled images by up to 75% without sacrificing performance. Our method also outperforms alternative methods for CNN pre-training, such as pre-training on publicly available non-medical or medical data using the target task (in this instance: segmentation). As it makes efficient use of available (non-)public and (un-)labeled data, the approach has the potential to become a valuable tool for CNN (pre-)training.
1 Introduction
Surgical data science uses treatment-process data to support care, but supervised models can generalize poorly across hospitals while manual annotation remains labor-intensive. This section motivates testing whether unlabeled endoscopic video can help pre-train models for medical instrument segmentation.
1 Introduction
- Mean segmentation performance drops by more than 50% when a model trained at one site is applied to endoscopic video from another site.The figure contrasts training and testing at the same hospital with testing at a different site.Source: §1 · p. 2, Figure 1 · p. 3
- Curation of sufficient training data is extremely labor-intensive and is hindering progress in surgical data science.The field includes computer-vision tasks such as device detection, segmentation, and tracking in endoscopic video.Source: §1 · p. 2
- The paper tests whether unlabeled video can teach a target-domain representation that boosts machine-learning performance when used for pre-training.Medical instrument segmentation is used as the example target task.Source: §1 · p. 2
2 Methods
The method pretrains a CNN on unlabeled target-domain endoscopic images using re-colorization, then fine-tunes it for instrument segmentation. A second variant adds labeled segmentation data from a related domain.
2.1 Concept overview
- PAT uses an auxiliary task on unlabeled target-domain images to initialize a CNN, then refines it with labeled data for the target task.Here, the target task is instrument segmentation.Source: §2.1 · p. 3
2.2 Prototype implementation
- The prototype uses a U-Net for segmentation and zero-initialized dummy a/b input channels during luminance-only pretraining, which are filled during target-task training.Source: §2.2 · p. 4, §2.2.1 · p. 5
2.2.1 Auxiliary task: re-colorization
- Re-colorization maps LAB luminance to predicted a/b channels with a U-Net generator and discriminator, combining GAN loss with color-frequency weighting and a counterbalancing term.The color-balancing terms address skewed color frequencies while discouraging invalid outputs dominated by rare colors.Source: §2.2.1 · pp. 4–5
2.2.2 Target task: Instrument segmentation
- PAT fine-tunes the re-colorization-pretrained model with a new segmentation output layer; PAT-Ext first adds segmentation training on labeled data from a similar domain.Source: §2.2.2 · p. 6
2.3 Experiments
- Experiments vary labeled-set size, data augmentation, and unlabeled-data domain, compare other pre-training methods, and test on the held-out DS HD L dataset.The comparison includes labeled-data segmentation pretraining on non-medical and similar-domain medical datasets.Source: §2.3 · pp. 6–8
3 Results
PAT improved segmentation performance over training without pre-training, while combining it with labeled medical-domain pre-training produced the best results. The reported gains depended on training-data size and pre-training domain, and the authors also evaluated annotation-effort reductions and data augmentation.
Performance of re-colorization
- GAN-based re-colorization produced realistic-looking images when trained on medical data, whereas natural-image pre-training data produced reconstructions unlike endoscopic images encountered in practice.The authors assessed re-colorization quantitatively only indirectly, through its effects on segmentation.Source: §3 · p. 8
Effect of training data size
- For fractions 1/20 through 1/3, PAT and PAT-Ext were significantly better than baseline at α = 0.05, with p-values < 0.001 except PAT at 1/20 (p = 0.01).Source: §3 · p. 9
- The authors reported decreases in manual annotation effort of over 60%, more than 50%, and around 25%; the supplied passage does not clearly specify which data fractions correspond to the latter two reductions.Source: §3 · p. 9
Effect of data augmentation
- PAT-Ext with data augmentation during fine-tuning achieved the best performance, and PAT-Ext surpassed augmentation alone especially below 1/6 of the training data.Data augmentation increased performance similarly for the baseline and proposed methods.Source: §3 · p. 9
Effect of pre-training domain
- Target-domain pre-training achieved the best results regardless of fine-tuning image count; non-medical pre-training reduced accuracy but still beat the no-pre-training baseline.For small training datasets, medical images yielded better results than non-medical images.Source: §3 · p. 9, Figure 6 · p. 10
Comparison and combination with other pre-training methods
- Combining PAT with state-of-the-art pre-training on medical data (PAT-Ext) achieved the best results; medical-data pre-training alone generally outperformed PAT.PAT also outperformed state-of-the-art pre-training from a non-medical domain, and the methods' results became more similar as fine-tuning data increased.Source: §3 · p. 10
4 Discussion
The discussion presents PAT as a way to reduce labeling needs through target-domain self-supervision, particularly with small labeled sets. It also notes that results vary across datasets and that auxiliary tasks should be optimized for individual applications.
4 Discussion
- PAT outperformed the no-pretraining baseline and labeled-data state-of-the-art pre-training, particularly with small labeled sets.Table 1 reports DSC and IoU comparisons across labeled-image fractions, but its supplied caption does not include the cell values.Source: §4 · p. 11, Table 1 · p. 11
- The authors position the work as an early investigation of self-supervised learning to reduce manual labeling in medical image segmentation.They note that related self-supervised pre-training work was recent and had largely come from computer vision.Source: §4 · pp. 10–11
- With up to 1/6 of the training data, PAT-Ext's benefits exceeded those of data augmentation.The discussion identifies this as a setting with little training data.Source: §4 · pp. 11–12
- PAT achieved state-of-the-art performance on the MICCAI dataset without augmentation, but absolute performance was much worse on the authors' own dataset.The authors attribute the difference to lower image variability in MICCAI and the challenging nature of their own images.Source: §4 · p. 12
- The authors identify finding optimal auxiliary tasks for each target application as a focus for future work.They also plan to test the method on further target tasks.Source: §4 · p. 12
- The authors conclude that their pre-training approach makes use of public and non-public, labeled and unlabeled data, with potential across a wide range of target tasks.They describe its potential impact on research and possible clinical applications as high.Source: §4 · p. 12