DeepSMILE: Contrastive self-supervised pre-training benefits MSI and HRD classification directly from H&E whole-slide images in colorectal and breast cancer

1. Introduction

• HRD and MSI can indicate therapy sensitization in breast and colorectal cancer, respectively.The introduction frames identifying these markers as support for personalized medicine.Source: §1 · p. 1
• Sequencing and targeted DNA methods for HRD or MSI can be expensive, time-consuming, and inaccessible, while IHC still requires additional laboratory testing.The passage also notes that molecular features indicating functional homologous recombination deficiency are inconclusive and debated.Source: §1 · p. 1
• H&E whole-slide images are accessible and inexpensive, yet DDR-related morphology is undescribed and the images are not used routinely to detect DDR defects.The authors identify potential use in early patient stratification for additional tests or therapy decisions.Source: §1 · p. 1
• Without spatial annotations, tile-supervision assigns each tile the WSI label, creating noisy supervision because the DDR signal may not occur in every tile.The cited work associates noisy supervision with large data requirements; existing SSL methods and ImageNet-pretrained features also have limitations for histopathology.Source: §1 · p. 2
• The paper evaluates SimCLR-pretrained features against ImageNet initialization for HRD and MSI, including noisy tile-supervision and attention-based DeepMIL.It tests feature-extractor fine-tuning with tile-supervision and frozen features with DeepMIL.Source: §1 · p. 2

1.1. Contributions

• DeepSMILE combines self-supervised pre-training and heterogeneity-aware multiple instance learning without pixel- or tile-level annotations, outperforming tile-supervision baselines for MSI and HRD classification.The authors also report that MSI improvement depends on tumor annotation, while the method improves HRD classification without manual annotations.Source: §1.1 · p. 2

2.1. Two-stage: tile-supervised WSI-label methods

• Tile-supervised WSI-label methods assign the WSI label to each tile and commonly predict the patient label from the fraction of positively classified tiles.These methods have been used for mutation, MSI, and broader genomic, transcriptomic, and survival-label classification.Source: §2.1 · p. 2

2.2. Two-stage: fully supervised methods

• Fully supervised two-stage methods replace each tile with a pretrained feature representation before supervised learning on the compressed WSI.The literature includes both self-supervised and multi-task supervised feature-extractor pre-training.Source: §2.2 · pp. 2–3

2.3. Two-stage: weakly supervised methods

• Weakly supervised MIL treats a WSI as a labeled bag whose unlabeled tiles may contain the label signal, then aggregates tile features for WSI-level classification.Prior approaches include attention-weighted means, top/bottom scoring, and self-supervised pre-training paired with MIL.Source: §2.3 · p. 3

2.4. End-to-end methods: contextless

• The contextless end-to-end method described clusters tile features into representative parts and concatenates them, but does not model tile interactions for higher-level spatial context.The reported prostate cancer results were similar to the MIL-RNN method.Source: §2.4 · p. 3

2.5. End-to-end methods: context-aware

• Context-aware streaming CNNs address WSI memory constraints, but are approximately 16× slower than MIL methods at low resolution, with runtime scaling non-linearly as resolution increases.The cited work also describes GPU memory optimizations for gigapixel WSIs.Source: §2.5 · p. 3
• DeepSMILE is a two-stage weakly supervised method for HRD and MSI that adds SimCLR-pretrained histopathology features and a DeepMIL extension modeling tumor heterogeneity.The paper positions it among two-stage approaches rather than end-to-end methods.Source: §2 · p. 2, §2.5 · p. 3

3.1. Data

• The study classifies binary genomic labels from gigapixel H&E slides in breast and colorectal cancer, using patient-level splits.The genomic labels are derived from DNA sequencing results.Source: §3.1 · p. 3

3.1.1. Colorectal cancer tissue dataset: TCGA-CRCk

• TCGA-CRCk contains 360 patients and 192,314 tumor-annotated, color-normalized tiles, with a supplied train-test split and five-fold patient-level train-validation splits.The training and test sets include 39 and 26 MSI patients, respectively.Source: §3.1.1 · p. 3

3.1.2. Breast cancer tissue dataset: TCGA-BC

• TCGA-BC HRD labels use a summed HRD score, but its relationship to functional HR and morphology is unknown; the study tests median and tertile splits.The tertile strategy discards patients near the median to provide a better supervisory signal.Source: §3.1.2 · pp. 3–4
• Breast slides are tiled after background segmentation, yielding 2,889,870 tissue tiles from 1,127 WSIs and 1,041 patients.The TCGA-BC splits are patient-level and stratified for median-split HRD score.Source: §3.1.2 · p. 4

3.2. Evaluation

• Performance is assessed with slide-level AUROC and F1 across five test folds, using validation-set patient-level AUC to select the model for final test evaluation.The study compares methods using bootstrap samples and a two-sided t-test, with p < 0.05 as its significance threshold.Source: §3.2 · p. 4

4. Model

• DeepSMILE combines self-supervised pre-training with feature variability-aware DeepMIL to model tumor heterogeneity in a two-stage weakly supervised WSI method.The paper identifies Figure 1 as a model overview and Table 1 as a comparison with existing methods.Source: §4 · p. 4
• The model has two parts: a SimCLR-trained in-domain feature extractor and VarMIL, the second-stage multiple instance learning model.The text introduces these as the feature-extraction and tile-aggregation stages.Source: §4 · p. 4
• The experimental pipeline pre-trains on training-set tiles, extracts and saves features, then loads all WSI tile features as samples with patient-level labels.The pipeline uses patient-level train, validation, and test splits.Source: §4 · p. 5

• Each fold iterates through batches from the data loader.Source: §11 · p. 5
• The model with the highest validation AUC is selected within each fold.Source: §11 · p. 5
• The selected model is evaluated on the test set.Source: §11 · p. 5

27 end

• DeepSMILE pairs a variance-modeling MIL approach with a tile-supervised WSI-label baseline using an ImageNet-pretrained feature extractor.Source: §27 · p. 5

4.1.1. Self-supervised pre-training

• SimCLR pre-trains a histopathology feature extractor without pixel-level annotations or WSI-level labels, using contrastive learning on tile representations.The pre-trained CNN maps each tile to a lower-dimensional feature vector.Source: §4.1.1 · p. 5

4.2. Classification

• VarMIL extends DeepMIL for whole-slide genomic-label prediction by incorporating intra-WSI, inter-tile feature variance, with cross-entropy loss.Source: §4.2 · p. 5

4.2.1. Baseline: tile supervised WSI-label learning

• Tile supervision assigns each tumor-bed tile its patient's label and averages tile predictions for the WSI-level prediction.Source: §4.2.1 · p. 6

4.2.2. Weak label classification: DeepMIL

• DeepMIL represents a slide with an attention-weighted average of tile features, then classifies that representation.Source: §4.2.2 · p. 6

4.2.3. Our method: Weak label classification with VarMIL

• VarMIL adds a learned attention-weighted variance to DeepMIL's mean representation to model feature variability across tiles.The resulting WSI-level representation is passed to a linear classification layer.Source: §4.2.3 · p. 7

5.1. Experiment 1: Self-supervised learning improves MSI classification on TCGA-CRCk for MIL and tile supervision

• 0.87 ± 0.01 AUC with SimCLR tile supervision versus 0.77 ± 0.06 with ImageNet tile supervision on TCGA-CRCk MSI; F1 rose from 0.46 ± 0.18 to 0.61 ± 0.11.SimCLR-VarMIL reached 0.86 ± 0.02 AUC and 0.47 ± 0.10 F1, significantly below SimCLR tile supervision, especially on F1.Source: §5.1 · p. 7, Table 2 · p. 8

5.2. Experiment 2: Self-supervised learning and VarMIL improves HRD Score classification in TCGA-BC compared to tile supervision

• 0.75 ± 0.03 AUC for mHRD and 0.81 ± 0.04 AUC for tHRD with SimCLR-VarMIL, versus tile-supervision AUCs of 0.71 for mHRD and 0.78 for tHRD.The only significant SimCLR-VarMIL versus SimCLR-DeepMIL difference was in mHRD AUC (p < 0.05).Source: §5.2 · p. 7, Table 2 · p. 8
• Pathologist review found SimCLR-VarMIL attended to features expected to indicate DDR deficiency in TCGA-BC and to previously described MSI-predictive features in TCGA-CRCk.The TCGA-BC ROC and PR curves also showed SimCLR-pretrained MIL methods performing better across the curve than ImageNet MIL or tile-supervision methods.Source: §5.1 · p. 7, §5.2 · p. 8

5.3. Experiment 3: Self-supervised learning reduces label requirement by over 50%

• With 40% of labeled data, DeepSMILE achieved higher mean AUC than ImageNet tile supervision trained on 100% of the data.AUC improved with more labeled data for every method; TCGA-BC mHRD curves were relatively flat, likely reflecting the weak median-split HRD supervisory signal, while TCGA-BC tHRD and TCGA-CRCk MSI showed no clear exhaustion point.Source: §5.3 · p. 8

6. Discussion and conclusions

• DeepSMILE pairs a histopathology-specific self-supervised feature extractor with VarMIL, which aggregates tile features while modeling intratumor heterogeneity.The conclusion notes that VarMIL handles weak labels and learns an aggregation function over tiles.Source: §6 · p. 8
• For TCGA-BC HRD, DeepMIL increased AUC by up to 0.04 and F1 by up to 0.05 over tile supervision; on TCGA-CRCk MSI, self-supervised tile supervision gained 0.10 AUC and 0.15 F1 over ImageNet-initialized tile supervision.On TCGA-CRCk MSI, the self-supervised tile-supervision result outperformed SimCLR DeepMIL.Source: §6 · pp. 8–9
• SimCLR VarMIL increased performance over SimCLR DeepMIL by 0.02 AUC and 0.01 F1 for TCGA-BC tHRD, and by 0.01 AUC and 0.11 F1 for TCGA-CRCk MSI.The authors report that these MSI results are slightly below Chowder's 0.92 AUC, while Bilal et al. (2021) achieved 0.9 AUC on the same dataset.Source: §6 · p. 9
• With frozen feature extractors, SimCLR DeepMIL improved over ImageNet DeepMIL by mHRD AUROC +7% and F1 +19%, tHRD +10% and +2%, and MSI +20% and +10%.For tile supervision, the SSL benefit depended on supervisory-signal strength, not dataset size; TCGA-CRCk gains were AUROC +10% and F1 +15%, while TCGA-BC mHRD gains were +0% for both.Source: §6 · pp. 9–10

6.2. Conclusions

• The methods reached ImageNet-initialized tile-supervision baseline performance with 40% of labeled data; Figure 3 charts test AUROC and F1 across training-data fractions.The reported result covers HRD and MSI classification on the evaluated datasets.Source: Figure 3 · p. 10, §6.2 · p. 11

6.1. Limitations

• Preprocessing effects were not isolated, and external-dataset generalizability and robustness of SimCLR features were not evaluated.The authors say their current datasets cannot confirm whether tumor annotations explain the difference in tile-supervision benefit between TCGA-CRCk and TCGA-BC.Source: §6.1 · pp. 10–11
• VarMIL does not model feature interactions, multiple resolutions, or tile spatial context; the authors also caution that detected morphology may correlate with, rather than cause, HRD or MSI.They note that current DeepMIL and VarMIL feature extractors are not fine-tuned for the downstream task.Source: §6.1 · p. 11

6.2. Conclusions

• The authors suggest these methods may reduce reliance on expensive sequencing and support quicker therapy decisions, including in centers without sequencing resources.They also identify tasks with limited labels, such as immune-therapy response prediction from H&E WSIs, as a potential use for DeepSMILE.Source: §6 · p. 9, §6.2 · p. 11

• E.G. is a shareholder of Ellogon.ai.Source: Conflicts of interest · p. 11
• H.M. Horlings received a consultation fee paid to the institute from Roche Diagnostics, outside the scope of this work.Source: Conflicts of interest · p. 11
• The other authors report no conflicts of interest.Source: Conflicts of interest · p. 11

• Visualizations use SimCLR-VarMIL predictions from the first-fold test sets for TCGA-BC and TCGA-CRCk, without cherry-picking.They include positive and negative labels, top six slides, and the first and fifth highest- and lowest-attention tiles.Source: Supplementary material for “DeepSMILE:… · p. 15
• For MSI, high-attention tiles contain colorectal tumor tissue, while low-attention tiles mostly contain normal epithelium and fibro-adipose connective tissue.MSI-H high-attention tiles also show necrosis and mucinous differentiation; many low-attention tiles contain immune infiltrate.Source: Supplementary material for “DeepSMILE:… · p. 15
• For HRD, high-attention tiles contain tumor tissue; HRD-high tiles often show nuclear pleomorphism, unorganized heterogeneous tumor cells, and extravasated blood cells.Low-attention tiles contain blood vessels and connective tissue, while HRD-low high-attention tiles appear mucinous or invasive lobular carcinoma.Source: Supplementary material for “DeepSMILE:… · p. 15
• The visual interpretation may be biased by tile selection and reliance on a single pathologist.The authors suggest a blinded study involving several pathologists to investigate associations between attended morphology and genomic labels.Source: Supplementary material for “DeepSMILE:… · p. 15

• The MSI-H high-prediction visualization is labeled with low- and high-attention tiles.Source: TCGA-CRCk · p. 16
• The MSS low-prediction visualization is labeled with low- and high-attention tiles.Source: TCGA-CRCk · p. 17
• The TCGA-BC tHRD=1 high-prediction visualization is labeled with low- and high-attention tiles.Source: TCGA-CRCk · p. 18
• The TCGA-BC tHRD=0 low-prediction visualization is labeled with high- and low-attention tiles.Source: TCGA-CRCk · p. 19