Source-linked AI summary
Anomaly Detection-Inspired Few-Shot Medical Image Segmentation Through Self-Supervision With Supervoxels
Stine Hansen, Srishti Gautam, Robert Jenssen, Michael Kampffmeyer
TL;DR
Few-shot medical segmentation is challenged by prototype-based loss of local information, especially for heterogeneous backgrounds, and by limited dense annotations. The paper models only a foreground prototype, thresholds anomaly scores, and uses 3D supervoxel self-supervision; it reports improved performance and robustness, particularly without weak query labels.
Problem
Prototype-based few-shot segmentation loses local information, while heterogeneous backgrounds are difficult to represent from limited support slices.
Method
The model uses one foreground prototype to compute query-pixel anomaly scores, thresholds them with a learned threshold, and trains with 3D supervoxel self-supervision.
Results
The model improves segmentation performance and robustness on cardiac and abdominal-organ datasets, especially when query images have no weak labels.
Takeaways & Limitations
The approach supports practical few-shot medical segmentation without weak query labels and indicates potential for one-step 3D whole-volume segmentation.
Takeaways & Limitations
The single-foreground-prototype assumption may be insufficient when one foreground class contains multiple distinct regions with strong edges.
Abstract
from arXiv · showhide
Recent work has shown that label-efficient few-shot learning through self-supervision can achieve promising medical image segmentation results. However, few-shot segmentation models typically rely on prototype representations of the semantic classes, resulting in a loss of local information that can degrade performance. This is particularly problematic for the typically large and highly heterogeneous background class in medical image segmentation problems. Previous works have attempted to address this issue by learning additional prototypes for each class, but since the prototypes are based on a limited number of slices, we argue that this ad-hoc solution is insufficient to capture the background properties. Motivated by this, and the observation that the foreground class (e.g., one organ) is relatively homogeneous, we propose a novel anomaly detection-inspired approach to few-shot medical image segmentation in which we refrain from modeling the background explicitly. Instead, we rely solely on a single foreground prototype to compute anomaly scores for all query pixels. The segmentation is then performed by thresholding these anomaly scores using a learned threshold. Assisted by a novel self-supervision task that exploits the 3D structure of medical images through supervoxels, our proposed anomaly detection-inspired few-shot medical image segmentation model outperforms previous state-of-the-art approaches on two representative MRI datasets for the tasks of abdominal organ segmentation and cardiac segmentation.
1. Introduction
Medical segmentation needs label-efficient adaptation because dense annotations are often unavailable and supervised models generalize poorly to unseen classes. The paper addresses prototype-based few-shot segmentation’s loss of local information by modeling only the foreground, adding 3D supervoxel self-supervision, and evaluating without weak query labels.
- Dense annotations are often unavailable, while supervised segmentation models generalize poorly to unseen classes and require retraining for new tasks.
- Prototype-based few-shot segmentation loses local information through average pooling, especially for heterogeneous background classes.
- The proposed method models only a relatively homogeneous foreground with one prototype and computes anomaly scores for query pixels against it.
- Segmentation thresholds anomaly scores with a learned threshold, while 3D supervoxel self-supervision exploits volumetric information within a 2D setup.
- The evaluation protocol segments the entire query image without weak labels indicating which slices contain the target class.
- The contributions include an anomaly-inspired FSS approach, 3D structural self-supervision, and a practical weak-label-free evaluation protocol.
2. Related Work
Prior few-shot semantic segmentation extends metric-learning classification to pixel-level prediction, often using prototypes and self-supervised auxiliary tasks. This work instead avoids explicit background modeling and uses supervoxels to exploit 3D medical-image structure.
- Few-shot semantic segmentation predicts pixel-level masks for new classes from a few densely labeled samples.
- Metric-learning approaches represent support classes with prototypes, but local-information loss motivates additional part-aware or grid-based prototypes.
- Medical FSS includes supervised architectures such as SE-Net, whereas other approaches use self-supervision to remove labeled training data requirements.
- This paper avoids explicitly modeling the complex background and relies on a single foreground prototype.
- Supervoxels group locally similar voxels, with boundaries that tend to follow image structures and provide natural sub-regions.
- During training, support and query slices share an image volume, while inference segments the query volume slice by slice, one class at a time.
3. Problem Definition
Few-shot semantic segmentation learns to adapt from a small labeled support set to new classes and predicts masks for an unlabeled query image. Training and testing proceed episodically over sampled classes.
- FSS learns adaptation from training classes to new test classes using only a few labeled samples.
- Each episode samples N classes and provides annotated support images together with an unlabeled query image.
- In an N-way k-shot setting, the support set contains k image slices with annotations for each of N classes.
- The query set contains an image with one or more of the sampled classes, and the model outputs its predicted mask.
4. Methods
ADNet performs few-shot segmentation by modeling only the foreground with a single prototype, converting query-feature dissimilarity into a learned-threshold anomaly segmentation. Self-supervision uses 3D supervoxels to exploit volumetric structure during episodic training.
- 4.1. Anomaly Detection-Inspired Few-Shot Segmentation: ADNet uses a shared feature extractor but explicitly models only the foreground class, avoiding prototypes for the heterogeneous background.The foreground prototype is obtained from masked average pooling of support features.
- 4.1. Anomaly Detection-Inspired Few-Shot Segmentation: For each query feature, ADNet computes an anomaly score as the negative cosine similarity to the foreground prototype.Identical and oppositely directed feature vectors receive minimum and maximum anomaly scores, respectively, after scaling by α = 20.
- 4.1. Anomaly Detection-Inspired Few-Shot Segmentation: The model predicts the foreground by softly thresholding anomaly scores with a learned parameter T, then derives the background mask as its complement.A shifted Sigmoid makes thresholding differentiable; scores below T produce foreground probabilities above 0.5.
- 4.1. Anomaly Detection-Inspired Few-Shot Segmentation: Training combines binary cross-entropy segmentation loss, a threshold regularizer that encourages compact foreground embeddings, and prototype-alignment regularization.The threshold term minimizes LT = T/α, while prototype alignment reverses support and query roles.
- 4.2. Supervoxel-Based Self-Supervision: The self-supervision task samples a supervoxel as a pseudo-foreground class and selects support and query slices containing it.This cross-slice construction increases information extracted from one unlabeled volume during episodic training.
- 4.2. Supervoxel-Based Self-Supervision: Supervoxels are generated offline with a 3D graph-based segmentation algorithm that accounts for anisotropic voxel resolution.The method uses distances to 26 neighboring voxels and reweights z-direction distances according to spatial ratios; ρ controls minimum supervoxel size.
5.1. Setup
The study uses two public 3D MRI datasets, five-fold cross-validation, and two inference protocols that differ in whether weak query-location labels are required.
- Datasets: The MS-CMRSeg cardiac dataset contains 35 3D scans, while CHAOS contains 20 3D abdominal MRI scans.MS-CMRSeg scans average 13 slices and CHAOS scans average 36 slices.
- Cross-validation: All models are trained self-supervised and evaluated with five-fold cross-validation, repeated three times per fold.Support images are sampled from one patient and the remaining patients serve as query images.
- Evaluation metric: The reported evaluation uses mean Dice score, where 100% denotes a perfect segmentation match.The paper reports mean and standard deviation over three runs per split for EP1 and EP2.
- Evaluation protocols: EP1 divides support and query foreground volumes into three sub-chunks and requires weak labels identifying foreground locations in query volumes.The middle support slice in each sub-chunk segments the corresponding query sub-chunk.
- Evaluation protocols: EP2 uses one labeled middle slice from the support foreground volume to segment the entire query volume without weak query labels.This protocol does not depend on the target volume's position.
5.2. Comparison to state-of-the-art
The proposed method is compared with modern few-shot segmentation baselines under supervised-label-efficient settings and two evaluation protocols. It performs similarly to state of the art under EP1 and outperforms it under the more realistic EP2, particularly on abdominal data.
- Compared models: The comparison includes PANet, ALPNet, PPNet, and CANet, with PPNet using five prototypes per class.CANet adds a dense comparison decoder to the backbone network.
- EP1 results: Under EP1, the proposed model performs similarly to state of the art on both datasets while using significantly fewer prototypes.Models using only a few background prototypes perform poorly on both datasets, while CANet obtains the lowest overall cardiac Dice score.
- Qualitative comparisons: Qualitative abdominal examples show less over-segmentation, while cardiac examples show better capture of left-ventricle myocardium and blood pool.The reported comparisons emphasize robustness to background outside the support slice.
- EP2 results: Under EP2, the proposed approach outperforms state-of-the-art models, with significant differences from pSSL-ALPNet on both datasets at p < 0.05.The comparison uses one-sided Wilcoxon signed-rank tests across mean Dice scores from all runs.
- EP2 results: The abdominal results improve by more than 20 percentage points over pSSL-ALPNet under EP2.The larger gain is associated with evaluating all query slices, which creates a larger and more diverse background.
5.3. Model analysis
Model analyses examine threshold selection, loss components, supervoxel size, sigmoid steepness, and the contribution of volumetric self-supervision. The results support a threshold that transfers to test data, robustness over moderate supervoxel settings, and gains from vSSL over pSSL.
- Threshold analysis: The learned threshold is close to the ideal test-data threshold, leaving little opportunity for increased Dice through threshold adjustment.A line search evaluates thresholds from -20 to -15 on held-out data.
- Loss ablation: Including LT and LPAR improves Dice across all cardiac classes, while LT reduces over-segmentation of the left-ventricle myocardium.LT encourages a more compact foreground embedding.
- Supervoxel sensitivity: Segmentation performance is relatively robust for supervoxel minimum sizes ρ = 1000 to ρ = 2000, but declines at ρ = 500 and ρ = 5000.Larger ρ produces larger and fewer supervoxels.
- Threshold steepness: The model is robust to sigmoid steepness values from κ = 0.1 to κ = 1.0, with more than two percentage points gained by decreasing the steepness.Higher steepness makes assignments harder near the threshold.
- vSSL versus pSSL: Supervoxel self-supervision yields better or comparable results overall and significantly improves ADNet over pSSL on both datasets.The gains are most prominent for the abdominal dataset, which has more slices and more volumetric information.
- vSSL versus pSSL: Extending self-supervision to supervoxels enables training 3D CNNs for direct volume segmentation.This implication follows from using the volumetric structure of the data in the self-supervision task.
5.4. Extension to one-step volume segmentation
The proposed framework extends few-shot segmentation from slice-wise processing to one-step volume segmentation using a 3D CNN backbone and supervoxel-based self-supervision. Results suggest greater benefits from 3D convolutions on abdominal data than cardiac data, while class separation improves for the left kidney and spleen.
- 3D segmentation framework: The study evaluates one-step 3D volume segmentation by replacing the 2D backbone with a 3D ResNeXt-101 within the proposed framework.The 3D encoder processes entire volumes in one step and is pretrained on Kinetics-600.
- 3D segmentation framework: The modified 3D ResNeXt-101 preserves embedding resolution by removing z-direction max pooling and changing convolutional strides.The classifier is replaced with 1 × 1 × 1 convolutions that reduce the feature dimension from 2048 to 256.
- Results: Abdominal data show greater potential for 3D convolutions, whereas cardiac results exhibit relatively small differences between 2D and 3D backbones.This pattern mirrors the stronger abdominal benefit observed when self-supervision was extended from superpixels to supervoxels.
- Results: The 3D CNN reduces right-kidney performance when k = 1, highlighting the importance of strong support features for robust 3D-backbone results.The comparison is reported for the vSSL-ADNet experiments summarized in Table 8.
6. Limitations and Outlook
The method assumes that the foreground is relatively homogeneous, allowing the model to avoid explicitly modeling the heterogeneous background. Its limitations include insufficient single-prototype representation for multi-region foregrounds and class merging caused by weak boundaries in supervoxel self-supervision.
- Assumption: The approach models the relatively homogeneous foreground with one prototype instead of modeling the large, heterogeneous background.The authors identify background modeling as a central challenge in prototypical few-shot medical image segmentation.
- Limitation: A single foreground prototype may be insufficient when one foreground class combines distinct regions with strong edges.The authors give the combined left-ventricle blood pool and myocardium as an example and suggest multiple foreground prototypes as one possible remedy.
- Limitation: Superpixel- and supervoxel-based self-supervision can merge different classes when their boundaries are weak, making them difficult to separate during inference.In the supervoxel case, a weak boundary in one slice can cause leakage across the volume.
- Limitation: Weak boundaries between the left kidney and spleen produce supervoxel confusion associated with lower Dice scores and high standard deviations.The authors identify noisy supervoxel pseudo-labels as a direction for future research.
7. Conclusion
The paper presents an end-to-end anomaly detection-inspired few-shot segmentation network that avoids explicit background modeling and uses supervoxels to capture 3D structure. On cardiac and abdominal segmentation datasets, the approach improves reported performance and robustness and shows potential for one-step 3D volume segmentation.
- Contribution: The paper proposes a novel end-to-end trainable anomaly detection-inspired few-shot segmentation network for medical images.
- Contribution: The anomaly-detection formulation eliminates explicit modeling of the large and heterogeneous background class.
- Contribution: A supervoxel-based self-supervision task captures the 3D nature of medical image data for unsupervised training.
- Results: Evaluation on cardiac and abdominal organ segmentation datasets shows improved segmentation performance and robustness, especially without weak query-image labels.
- Outlook: The proposed model and self-supervision task show potential for one-step 3D segmentation of entire image volumes.