Source-linked AI summary
Dual-stream Multiple Instance Learning Network for Whole Slide Image Classification with Self-supervised Contrastive Learning
Bin Li, Yin Li, Kevin W. Eliceiri
TL;DR
WSI classification must handle gigapixel slides with many patches and usually lacks localized annotations, making weakly supervised MIL important. DSMIL combines dual-stream trainable-distance aggregation, self-supervised contrastive representations, and pyramidal multiscale fusion. It outperforms previous MIL methods and remains within 2% accuracy of fully supervised methods across evaluated WSI datasets.
Problem
WSIs have extremely high resolution, contain many patches, and usually lack scalable localized annotations, motivating slide-level MIL classification.
Method
DSMIL jointly learns patch and bag classifiers with dual-stream trainable-distance aggregation, self-supervised contrastive features, and pyramidal multiscale fusion.
Results
DSMIL outperforms previous MIL models and has less than 2% classification-accuracy gap relative to fully supervised methods across evaluated WSI datasets.
Takeaways & Limitations
The proposed aggregator performs strongly on representative WSI datasets and standard MIL benchmarks, supporting MIL-based analysis with slide-level supervision.
Takeaways & Limitations
Future work should adapt self-supervised learning to histopathological data and model spatial relations to capture macroscale WSI structure.
Abstract
from arXiv · showhide
We address the challenging problem of whole slide image (WSI) classification. WSIs have very high resolutions and usually lack localized annotations. WSI classification can be cast as a multiple instance learning (MIL) problem when only slide-level labels are available. We propose a MIL-based method for WSI classification and tumor detection that does not require localized annotations. Our method has three major components. First, we introduce a novel MIL aggregator that models the relations of the instances in a dual-stream architecture with trainable distance measurement. Second, since WSIs can produce large or unbalanced bags that hinder the training of MIL models, we propose to use self-supervised contrastive learning to extract good representations for MIL and alleviate the issue of prohibitive memory cost for large bags. Third, we adopt a pyramidal fusion mechanism for multiscale WSI features, and further improve the accuracy of classification and localization. Our model is evaluated on two representative WSI datasets. The classification accuracy of our model compares favorably to fully-supervised methods, with less than 2% accuracy gap across datasets. Our results also outperform all previous MIL-based methods. Additional benchmark results on standard MIL datasets further demonstrate the superior performance of our MIL aggregator on general MIL problems. GitHub repository: https://github.com/binli123/dsmil-wsi
1. Introduction
WSI classification is difficult because gigapixel slides contain many weakly labeled, highly imbalanced patches, motivating DSMIL’s dual-stream aggregation, contrastive representations, and multiscale fusion. DSMIL outperforms recent MIL models and approaches fully supervised accuracy while improving localization.
- Motivation: WSIs can reach 40,000 × 40,000 pixels, so classification typically divides them into thousands of patches.Patch-level expert labeling is time-consuming and difficult to scale.
- Motivation: Weakly supervised WSI classification treats each slide as a bag of patches and predicts slide labels from aggregated patch features or scores.A bag is positive when any patch is positive under the MIL formulation.
- Challenges: Highly unbalanced positive bags make max-pooling prone to misclassifying positive instances and shifting the decision boundary relative to fully supervised training.Weak supervision can also hinder rich feature learning and promote overfitting.
- Method: DSMIL jointly learns patch and bag classifiers in two streams, selecting a critical instance and weighting other instances by trainable distance to it.Soft instance selection produces a decision boundary intended to better delineate positive-bag instances.
- Method: DSMIL combines self-supervised contrastive feature learning with multiscale feature fusion spanning millimeter-scale structures and cellular-scale tissue context.The method uses these components alongside its dual-stream aggregator for weakly supervised WSI classification.
- Results: At least 2.3% classification-accuracy improvement over recent MIL models was reported, while the gap to fully supervised methods remained below 2%.DSMIL also achieved superior localization accuracy and state-of-the-art performance on general MIL problems.
2. Related Work
Prior deep MIL work uses learned pooling, attention, or contextual instance modeling, while WSI applications also require efficient representation learning and multiscale analysis. DSMIL contributes trainable critical-instance relations, self-supervised contrastive features, and pyramidal multiscale fusion.
- Deep MIL Models: Deep MIL models have used mean or max pooling, neural aggregation, attention-based aggregation, and contextual mechanisms such as graphs or capsule networks.These approaches differ in how they parameterize aggregation and model dependencies among instances.
- Deep MIL Models: DSMIL models instance-to-instance and instance-to-bag relations with a non-local operation whose weights are functions of the two nodes.Its trainable distance is computed between a critical instance and other instances rather than across every pair.
- Representation Learning: WSI MIL representation learning is difficult because large bags impose prohibitive memory requirements and slide-level supervision is noisy.These constraints complicate training CNN feature extractors for MIL.
- Representation Learning: Unlike prior semi-supervised approaches, DSMIL uses self-supervised contrastive learning for MIL feature extraction, a strategy described as underexplored in WSI analysis.Contrastive learning is used to learn visual representations without localized labels.
- Multiscale Analysis: WSI analysis commonly uses multiscale features because pathologists assess slides across scales and prior MIL work benefits from combining magnifications.DSMIL uses a pyramidal strategy to combine multiscale embeddings and preserve local attention constraints.
3. Method
DSMIL formulates WSI classification as MIL and combines a dual-stream aggregator with self-supervised features and pyramidal multiscale fusion. The model jointly uses instance scores, relations to a critical instance, and multiscale patch context to produce slide-level predictions.
- MIL formulation: MIL treats each WSI as a bag of patches with an unknown instance-label distribution, requiring permutation-invariant aggregation for slide-level prediction.The embedding-based formulation aggregates instance embeddings into a bag embedding, while the instance-based formulation aggregates instance scores.
- DSMIL aggregator: The two stream scores are averaged, and the aggregation remains permutation-invariant because the critical-instance selection and distance measurement are order-independent.The resulting bag embedding has constant shape regardless of bag size.
- DSMIL aggregator: DSMIL jointly learns an instance classifier, a bag classifier, and a bag embedding in a dual-stream architecture.This hybrid design combines instance-based and embedding-based MIL approaches.
- DSMIL aggregator: The max-pooling stream selects the highest-scoring patch as the critical instance, while the aggregation stream weights instance information by trainable distances to that instance.The weighted information vectors form a bag embedding that is scored by a bag classifier.
- Multiclass extension: DSMIL extends to multiclass MIL by computing class-specific attention weights and producing a bag-embedding matrix with one output channel per class.The information vectors support intra-instance selection, while distance measurement performs interinstance selection relative to the critical instance.
- Feature learning and multiscale fusion: Self-supervised contrastive learning supplies feature representations for MIL, while pyramidal concatenation preserves lower-magnification context alongside higher-magnification patch features.Duplicating lower-magnification features across corresponding higher-magnification patches supports local attention consistency while retaining scale-specific information.
4. Experiments and Results
Experiments on Camelyon16 and TCGA lung cancer evaluate DSMIL for classification and localization, then ablate contrastive features and multiscale fusion. DSMIL improves over MIL baselines and approaches fully supervised performance.
- Clinical WSI datasets: DSMIL is evaluated on Camelyon16 and TCGA lung cancer using classification and localization experiments.The experiments also include ablations of the aggregator, feature learning, and multiscale integration.
- Localization: DSMIL achieves superior localization accuracy compared with previous MIL models on Camelyon16.Localization is evaluated with pixel-level annotations that are ignored during training and used only for testing.
- Feature ablation: Self-supervised contrastive learning yields at least 16% higher classification accuracy than max-pooling features for unbalanced bags.For balanced bags, it remains more than 14% better than end-to-end max-pooling features and alleviates large-bag memory requirements.
- Multiscale fusion: Multiscale attention improves Camelyon16 accuracy by 3% over the single-scale approach and by at least 1.5% over other multiscale approaches.Using 5×+20× features outperforms using 1.25×+5×+20× by 1.6% in accuracy and 1.3% in AUC.
- General MIL benchmark: The dual-stream MIL aggregator outperforms previous best models, NL, and ANL by an average of 3% across five classical MIL datasets.Experiments use five runs with 10-fold cross-validation.
5. Conclusion and Future Work
The paper presents an MIL-based approach that combines a novel aggregator, self-supervised contrastive learning, and multiscale features for weakly supervised WSI classification. Future work targets histopathology-adapted self-supervision and spatial modeling of macroscale WSI structure.
- Conclusion: The method integrates a novel MIL aggregator, self-supervised contrastive learning, and multiscale features for weakly supervised WSI classification.The authors report improvement over previous methods on representative WSI datasets and benchmark datasets.
- Conclusion: Casting tumor detection as an MIL problem may support clinical applications using large amounts of unannotated slides.
- Future Work: Future work includes self-supervised learning strategies adapted to histopathological data and spatial-relation modeling for macroscale WSI features.The proposed spatial modeling is intended to capture spatially structured features and could potentially improve performance.