Source-linked AI summary
Morphology signal in whole slide image foundation models can automatically triage slides
Ayushi Sinha, Shashank Yadav, Benjamin Holmes, Pravat Das, Aaron W. Bogan, James S. Lewis, Santiago Romero-Brufau, Andrew Y. K. Foong, Scott H. Kaufmann, Kathryn M. Van Abel, David M. Routman, Michael R. Lucas
TL;DR
Patients often generate many WSIs, while only a few contain tumor or other biomarkers needed for downstream modeling, making manual slide triage costly and potentially diluting signal when all slides are used. The paper uses zero-shot classification with pathology foundation models to rank slides by tumor content and introduces a ranked evaluation for triage. Across datasets, the models identify relevant tumor slides near the top of the rankings, including top-k guarantees that depend on dataset and model.
Problem
Patients generate many WSIs, but only a few contain biomarkers relevant to downstream prediction, making manual triage costly and risking diluted signal when all slides are used.
Method
The pipeline uses zero-shot classification from image–text embedding cosine similarities to rank each patient’s slides, with a ranked evaluation accounting for pool size, annotated-set size, and hit rank.
Results
The models accurately identify slides with the most tumor; on OV, primary and secondary tumor blocks reached 100% recall at k = 7 for CONCH, k = 9 for mean-CONCH, and k = 6 for TITAN.
Takeaways & Limitations
Publicly available pathology foundation models contain sufficient morphology signal to support automatic slide triage without manual curation.
Takeaways & Limitations
Further evaluation on additional models and datasets is needed to study model differences and generalizability across cancer types.
Abstract
from arXiv · showhide
Patient exams in the cancer diagnosis and staging process typically generate several whole slide images (WSIs). One of the initial steps in training models on WSI data is identifying one or a few slides containing tumor or other diagnostic biomarkers necessary for downstream prediction tasks such as estimating recurrence risk or progression-free survival. This step requires tedious manual curation by experienced pathologists. Many published datasets make the artificial assumption of 1 slide per patient. Alternatively, all slides per patient may be used for model training, which may dilute the signal from the few slides containing tumor or other relevant information. In this paper, we present a pipeline to overcome these challenges using publicly available WSI foundation models (FMs). Our evaluations show that ranking WSIs based on predictions from zero-shot classification using WSI FMs accurately identifies slides with the most tumor, indicating that WSI FMs contain sufficient morphology signal to automatically triage slides. We also present a formulation for ranked evaluation to benchmark FM performance in slide triage. We show, on multiple datasets, that tumor slides are identified in the top-2 ranked slides for patients with up to 43 slides.
1 Introduction
Cancer exams generate many WSIs, but only a few typically contain tumor or other biomarkers relevant to downstream prediction. The paper proposes using pathology foundation models to rank slides automatically, reducing manual triage and avoiding diluted training signal.
- Approach: The pipeline segments slides, extracts tissue patches, generates patch- and slide-level embeddings, and classifies images using class-specific text embeddings.
- Motivation: Only a few WSIs per patient typically contain tumor or other diagnostic biomarkers useful for downstream prediction tasks.Pathologists currently identify these slides by visual review, taking about 30−90s per slide.
- Motivation: Using all slides per patient can dilute tumor-related signal because each slide may produce thousands of patches.
- Approach: Zero-shot classification ranks slides by tumor content, using class probabilities or the percentage of patches assigned to a class.
- Contribution: The paper introduces ranked evaluation incorporating patient slide-pool size, annotated tumor-slide count, and correctly identified slide rank.
2 Methods
The study evaluates publicly available pathology foundation models on annotated WSI datasets using zero-shot classification and slide ranking. Models classify tumor, lymphocyte, and other content from image–text embedding similarities, then select top-k slides.
- Datasets: Two WSI datasets with pathology-report annotations were used to evaluate slide triage.The HN dataset provides coarse part-level annotations, whereas the HN-node subset provides more granular lymph-node annotations and OV was manually curated for tumor-containing blocks.
- Preprocessing: All WSIs were background-segmented, and 512 × 512 patches were extracted from tissue at 20× magnification.
- Foundation models: CONCH generates patch-level embeddings, while mean-CONCH and TITAN generate slide-level embeddings.
- Classification: Zero-shot classification uses ensemble text prompts for tumor, lymphocyte, and other classes, with prompts adapted to cancer type.
- Slide ranking: Cosine similarities between image and text embeddings are converted to class probabilities, which rank slides directly or after patch-level aggregation.Slides with zero class probability or zero assigned patches are excluded from the top-k selection.
3 Evaluation
Evaluation measures whether annotated tumor-associated block codes appear among the top-k ranked slides and adds a score that weights task difficulty by pool size, annotation-set size, and rank. Results show high recall on HN and increasing recall with larger k on OV.
- Recall@k: Recall@k is the fraction of patients for whom at least one annotated block code appears among the top-k ranked slides.
- Model agreement: Model agreement is measured by intersecting the top-ranked block-code sets at each k and normalizing by the ideal intersection size.
- Ranked evaluation: The ranked evaluation weights larger slide pools more heavily because selecting relevant slides becomes harder as pool size increases.
- Ranked evaluation: Smaller annotated sets receive higher weight because fine-grained annotations create a more difficult evaluation task.
- Evaluation scope: Patients with a single slide are excluded because pool_size = 1 does not constitute a selection task.
4 Results
Across datasets, foundation-model rankings identify tumor-containing slides effectively, with CONCH and mean-CONCH strongest on ovarian cancer and robust performance despite large slide pools.
- 4.1 All models achieve high recall: Recall reached 100% at k = 1 for all models on the fine-grained HN-node subset.This subset contained annotated sets averaging 3.6 ± 3.4 block codes.
- 4.1 All models achieve high recall: 100% recall was achieved on the OV dataset at k = 7, 9, and 6 for CONCH, mean-CONCH, and TITAN, respectively.These results used primary and secondary tumor blocks.
- 4.2 CONCH and mean-CONCH show highest agreement: Agreement between CONCH and mean-CONCH rankings was highest across both HN and OV datasets.The two methods use different aggregations of the same embeddings.
- 4.3 Ranked evaluation score highest for CONCH and mean-CONCH: On OV, CONCH and mean-CONCH outperformed TITAN, while all models showed performance declines at the largest pool sizes.Ranked evaluation compared performance across quartiles of patient slide-pool size.
- 4.3 Ranked evaluation score highest for CONCH and mean-CONCH: The tumor slide was identified within the top-2 ranked slides for patients with up to 43 slides.This outcome is reported for the ranked evaluation shown in Figure 3.
- 4.3 Ranked evaluation score highest for CONCH and mean-CONCH: Average hit rank for CONCH and mean-CONCH increased from 1.4 and 1.3 in smaller pools to 2.4 and 2.6 in the largest pool.TITAN’s average hit rank increased from 1.5 to 3.1 as pool sizes grew.
5 Discussion
The pipeline supports automatic slide triage and dataset curation using publicly available pathology foundation models. The authors attribute stronger performance by CONCH to finer-grained patch-level classification, while broader validation remains future work.
- 5 Discussion: Automatic slide triage could streamline pathologist evaluation and curation of datasets for model training.The paper reports successful triage without manual curation using publicly available pathology image-text foundation models.
- 5 Discussion: Patch-level CONCH performed better than slide-level TITAN, potentially because patch-level classification preserves finer-grained tumor morphology.The authors suggest slide-level aggregation may suppress features present in individual patches.
- 5 Discussion: Future work includes testing additional models, studying learned representations, and evaluating generalizability across cancer types.The authors also report preliminary improvement in survival prediction using the top-1 CONCH-ranked slide, with top-k evaluation ongoing.
6 Appendix
The appendix documents the prompts and prompt templates used for zero-shot classification. It includes class-specific prompt tables and templates for generating prompt ensembles.
- The appendix contains details about the prompts used for zero-shot classification.
- Table A1 lists prompts for the tumor class in the HN and OV datasets.
- Table A2 lists prompts for the lymphocyte class in the HN and OV datasets.
- Table A3 lists prompts for the other class in the HN and OV datasets.
- Table A4 provides templates for generating prompt ensembles from the class prompts.Templates replace ‘CLASS-NAME’ with class prompts from Tables A1, A2, and A3.