Source-linked AI summary
Cluster-to-Conquer: A Framework for End-to-End Multi-Instance Learning for Whole Slide Image Classification
Yash Sharma, Aman Shrivastava, Lubaina Ehsan, Christopher A. Moskaluk, Sana Syed, Donald E. Brown
TL;DR
Whole-slide images are too large for direct training, and conventional MIL decouples patch encoding from slide aggregation while relying on limited slide-level supervision. C2C jointly clusters and samples patches, applies adaptive attention, and regularizes intra-cluster attention with KL divergence. The framework reports strong classification performance and attention patterns aligned with medically relevant tissue regions.
Problem
WSIs are computationally infeasible to process directly, while two-stage MIL decouples patch encoding from aggregation and may learn sub-optimal representations under weak supervision.
Method
C2C locally clusters WSI patch embeddings, samples patches from each cluster, jointly trains attention-based slide and patch classifiers, and adds KL-divergence regularization.
Results
C2C achieved comparable performance to fully supervised methods and produced high-attention clusters overlapping pathologist-annotated tumor areas or matching medically important celiac-disease patterns.
Takeaways & Limitations
End-to-end MIL with clustering and attention can support WSI classification using slide-level supervision while highlighting patches relevant to diagnosis.
Takeaways & Limitations
The MIL formulation assumes positive bags contain at least one diseased patch and negative bags contain only healthy patches.
Abstract
from arXiv · showhide
In recent years, the availability of digitized Whole Slide Images (WSIs) has enabled the use of deep learning-based computer vision techniques for automated disease diagnosis. However, WSIs present unique computational and algorithmic challenges. WSIs are gigapixel-sized ($\sim$100K pixels), making them infeasible to be used directly for training deep neural networks. Also, often only slide-level labels are available for training as detailed annotations are tedious and can be time-consuming for experts. Approaches using multiple-instance learning (MIL) frameworks have been shown to overcome these challenges. Current state-of-the-art approaches divide the learning framework into two decoupled parts: a convolutional neural network (CNN) for encoding the patches followed by an independent aggregation approach for slide-level prediction. In this approach, the aggregation step has no bearing on the representations learned by the CNN encoder. We have proposed an end-to-end framework that clusters the patches from a WSI into ${k}$-groups, samples ${k}'$ patches from each group for training, and uses an adaptive attention mechanism for slide level prediction; Cluster-to-Conquer (C2C). We have demonstrated that dividing a WSI into clusters can improve the model training by exposing it to diverse discriminative features extracted from the patches. We regularized the clustering mechanism by introducing a KL-divergence loss between the attention weights of patches in a cluster and the uniform distribution. The framework is optimized end-to-end on slide-level cross-entropy, patch-level cross-entropy, and KL-divergence loss (Implementation: https://github.com/YashSharma/C2C).
1. Introduction
WSIs are too large for direct deep-learning training, while slide-level labels leave patch-level information weakly specified. C2C addresses these challenges by coupling cluster-based sampling, attention aggregation, and KL-divergence regularization in an end-to-end framework.
- Motivation: WSIs reach approximately 100k × 100k pixels, making direct deep-learning training computationally infeasible and downsampling prone to losing diagnostic detail.MIL therefore uses patches and slide-level labels to handle whole-slide classification.
- MIL setting: MIL training uses slide-level labels despite uncertainty about which individual patches contain disease-specific information.Positive slides may contain non-diseased patches, making the common assumption that all positive-slide patches are positive unreliable.
- Limitations of prior approaches: Two-stage MIL methods independently encode patches and aggregate them, so the aggregation stage cannot guide the learned patch representations.This decoupling can produce sub-optimal solutions for classification.
- Proposed framework: C2C introduces cluster-based sampling to expose the model to diverse patches, attention-based aggregation for slide prediction, and KL-divergence regularization within clusters.These are the framework’s three stated features.
2. Related Works
Prior WSI classification methods generally separate patch representation learning from slide-level aggregation, using supervised, unsupervised, or end-to-end variants. Existing methods differ in patch selection, representation learning, and aggregation mechanisms.
- Two-stage methods: Two-stage approaches learn patch representations and then aggregate patch predictions, with supervised and unsupervised strategies forming the main categories.Supervised methods require patch labels or noisy assumptions, whereas unsupervised methods do not guarantee discriminative disease features.
- Supervised approaches: Supervised methods include top-k selection, expectation-maximization, multi-scale features, mean pooling, and spatially context-aware patch selection.These approaches combine patch classifiers with different patient-level aggregation procedures.
- Unsupervised approaches: Unsupervised and self-supervised methods learn patch features with compression or contrastive objectives before applying spatial or MIL-based aggregation.Examples use neural image compression, contrastive predictive coding, or self-supervised contrastive learning.
- End-to-end approaches: End-to-end alternatives jointly train instance- and bag-level components using center loss or global-clustering-based part sampling.These methods aim to avoid the strict separation between patch encoding and slide aggregation.
3. Methods
C2C treats a WSI as a bag of patches, locally clusters encoded patches, samples from each cluster, and combines adaptive attention with weakly supervised instance learning. Its losses jointly train slide prediction, patch prediction, and intra-cluster attention regularization.
- Problem formulation: A WSI is represented as a bag of patches, with positive bags containing at least one diseased patch and negative bags containing only healthy patches.This is the MIL assumption used for weakly supervised classification.
- Framework: C2C samples N′ patches from N WSI patches using local clustering, attention aggregation, and KL-divergence regularization of within-cluster attention.The design is intended to expose training to diverse patches while constraining attention variation among similar patches.
- Cluster-based sampling: The encoder maps patches to l-dimensional embeddings, which are clustered independently within each WSI using K-means before sampling from the resulting buckets.Encoder parameters are frozen during clustering, and clustering is recomputed from encoder representations at the start of each epoch.
- Cluster-based sampling: Equal k′ patches are sampled from each of k clusters, with the total sampled patches capped at 64 for computational reasons.The method hypothesizes that richer representations make these samples approximate the full WSI representation.
- Attention aggregation: A two-layer attention module weights sampled patch representations into a WSI representation, while instance representations also receive weakly supervised patch classification.The aggregated representation feeds the WSI classifier and patch representations feed the instance classifier.
- Optimization: The total objective combines WSI cross-entropy, patch cross-entropy, and KL-divergence loss, with KL regularizing attention toward a uniform distribution within each cluster.The reported base model uses ResNet-18, k = 8, and all patches during inference for attention aggregation.
4. Experiment and Results
C2C was evaluated on gastrointestinal and CAMELYON data, with end-to-end aggregation and encoding, KL-divergence regularization, and expert-reviewed attention patterns examined across experiments.
- Evaluation setup: C2C was compared with two-stage MIL and mean-pooling approaches on gastrointestinal data for celiac-versus-normal classification.The comparison included Campanella-MIL variants and Two-Stage Mean Pooling.
- Gastrointestinal results: End-to-end coupling of the encoder and aggregation module enabled C2C to outperform comparison approaches despite using a relatively weaker ResNet backbone.The authors attribute this to synergy between aggregation and encoding during joint training.
- Attention interpretation: Medical review found high-attention patches containing damaged surface epithelium and intraepithelial lymphocytes, features relevant to celiac disease diagnosis.Patches were reviewed together with their cluster assignments and attention weights.
- CAMELYON16 results: 0.9112 ROC-AUC was achieved on CAMELYON16 using only slide-level labels, compared with 0.9223 for the challenge's best model.The reported score would have ranked second on the challenge classification portion and seventh on the open leaderboard.
- CAMELYON16 results: C2C assigned higher attention weights to patches containing tumor regions in CAMELYON16.The authors report that high-attention patches accurately identified tumor regions.
- KL-divergence analysis: KL-divergence made attention weights more uniform across positive MNIST-bag instance classes 8 and 9, reducing variability observed without that loss.Without KL-divergence, the attention module could favor one positive class inconsistently across bags.
5. Conclusion
The paper concludes that C2C combines clustering-based sampling, adaptive attention, and KL-divergence loss for end-to-end WSI classification. It reports strong performance and attention maps aligned with expert-identified diagnostic regions, while identifying multi-class and subtype classification as future work.
- C2C combines clustering-based sampling, adaptive attention, and KL-divergence loss in an end-to-end WSI classification framework.
- C2C achieved strong performance for celiac disease and breast cancer classification, comparable to fully supervised methods using detailed pathologist annotations.
- High-attention breast-cancer clusters overlapped pathologist-annotated tumor areas, while top celiac-disease clusters matched patterns considered important by medical experts.
- The authors identify multi-class and subtype classification as future applications for C2C.
Appendix A. Example and UMAP Plot of WSI
The appendix presents qualitative WSI examples: gastrointestinal patches are ordered by attention across clusters, and CAMELYON patches with high attention are compared with pathologist-annotated tumor regions.
- Gastrointestinal examples show patches sampled from WSI clusters in decreasing order of attention importance for detecting celiac disease.
- For CAMELYON, pathologist-annotated tumor regions are shown alongside patches assigned high attention by the model.
- A separate appendix figure represents Celiac and Normal biopsies through WSI embeddings in the test dataset.
Appendix B. MNIST Bag Set-Up
The MNIST bag experiments evaluate how clustering, sampling, KL-divergence regularization, and pooling choices affect model performance and inference cost.
- MNIST bag set-up: The MNIST bag dataset contains 28 × 28 grayscale digit images, with positive bags containing an 8 or 9.Bag sizes are Gaussian-distributed and rounded to the nearest integer; all experiments use LeNet5.
- Experimental comparisons: The appendix compares performance across different numbers of clusters, sampling strategies, KL-divergence weights, and pooling strategies.Each training configuration samples at most 64 patches per WSI.
- Inference cost: Inference time is reported per WSI in the test dataset.
Appendix D. Medical Expert Qualitative Review
Medical experts qualitatively assessed patch clusters and identified medically relevant histopathologic features among the highest-ranked clusters.
- Expert assessment: Medical experts assessed the qualitative importance of patch clusters, ranking Cluster 1 highest and Cluster 8 lowest.
- Expert assessment: Top clusters contained celiac-disease-relevant features, including intraepithelial lymphocytes, inflammation severity indicators, and damaged surface epithelium.Other identified features included Brunner glands and reduced columnar epithelium.
- Interpretation: The findings show that the method clustered patches using medically relevant histopathologic features.
Appendix E. CAMELYON16 Model and Examples
The CAMELYON16 model uses a configured CNN-based training setup, while example figures compare pathologist tumor annotations with the model’s assigned attention distribution.
- Model configuration: The CAMELYON16 model uses ResNet 18 as its backbone encoder and trains for 30 epochs with Adam at a learning rate of 1e-4.The training uses an 85%-15% training-validation split.
- Model configuration: The CAMELYON16 loss weights are α = 0.01 for WSI loss, β = 1 for patch loss, and γ = 0.01 for KL-divergence loss.
- Qualitative examples: Figure 5 places pathologist-annotated tumor areas beside deep-learning-assigned attention distributions for cancerous CAMELYON16 WSIs.
Appendix F. Limitations
The approach is limited by unstable training with one-WSI batches and by the time-intensive clustering step.
- Training stability: Using a batch size of one WSI can produce unstable training peaks depending on the normalization strategy.The framework processes 64 patches from each WSI per batch and uses momentum tuning plus reinitialized normalization statistics for stabilization.
- Computational cost: Clustering is time-intensive and can slow training, motivating future experiments with sampling strategies to optimize C2C training.The authors propose investigating approaches that reduce the training burden without affecting performance.