PRISM: A Multi-Modal Generative Foundation Model for Slide-Level Histopathology

1 Introduction

• Clinical pathology applications span cancer detection, sub-typing, microenvironment characterization, biomarker detection, treatment response, and survival prediction.Computational pathology tools are already available for clinical practice.Source: §1 · p. 1
• Most ground-truth labels are linked to a WSI or specimen, but tile-level foundation models require task-specific aggregators that can overfit, especially on small datasets.Clinical reports offer a natural-language supervisory signal for slide-level pre-training.Source: §1 · p. 2
• PRISM is a multi-modal slide-level foundation model that uses clinical reports as WSI-level supervision and can generate diagnostic reports without further training.Its slide embeddings also support linear classification or fine-tuning for downstream tasks.Source: §1 · pp. 2–3
• Zero-shot cancer detection and sub-typing approach or surpass supervised baselines, while limited-data fine-tuning often outperforms supervised training on the full dataset.The evaluation also covers tissue sub-typing and biomarker prediction.Source: §1 · pp. 2–3

Tile-level foundation models

• Tile-level vision foundation models have scaled to proprietary datasets of 100,000–1.5 million WSIs, but their downstream slide-level use still requires a separately trained aggregator.Virchow shows improved performance from scaling model and dataset size, yet low-data biomarker tasks remain difficult for aggregator training.Source: §2 · p. 3

Language-supervised tile models

• Tile-level language methods use captioning or image-text contrastive learning; PRISM instead trains a specimen-level aggregator to predict summarized clinical reports.PRISM uses a Perceiver to aggregate hundreds of thousands of tiles.Source: §2 · p. 3

Slide-level foundation models

• HIPT and LongViT aggregate tiles at slide level, but use self-supervised objectives rather than clinical-report supervision.HIPT summarizes local regions, whereas LongViT uses dilated attention for full-WSI context.Source: §2 · p. 3

Pathology report generation

• Earlier WSI report-generation systems used CNN tile encoders with RNN decoders, and later methods improved spatial context with HIPT.These approaches subsampled tiles and used memory-efficient RNNs that are slower to train and harder to parallelize than transformers.Source: §2 · p. 3

3 Slide-Level Foundation Model

• PRISM pairs specimen WSIs with GPT-4-rewritten clinical reports and trains a Perceiver slide encoder and BioGPT decoder using alignment and report-generation objectives.The framework yields a slide embedding and can generate reports describing histological features.Source: §3 · pp. 3–4
• Clinical reports are extracted from diagnosis sections, optionally augmented with IHC or molecular results, then rewritten into concise summaries and sampled from five rewrites per specimen.The rewrites increase diagnostic-information density and are sampled to discourage memorizing specimen-to-report associations.Source: §3 · p. 4
• WSIs are tiled at 224×224 pixels and 20× magnification; tiles with at least 25% tissue are encoded by Virchow into 2560-dimensional vectors.Virchow is a 632M-parameter ViT-H/14 trained on over 2B tiles from 1.5 million H&E WSIs.Source: §3 · p. 4
• A memory-efficient Perceiver compresses long sequences of specimen tile embeddings into a fixed-length set of latent features through asymmetric cross-attention.Cross-attention and latent-transformer weights are shared across blocks, with cached context key-value pairs reducing memory use.Source: §3 · p. 5
• PRISM uses 513 learned latents: 512 provide language-decoder context, while the remaining latent is the global slide embedding for contrastive learning and downstream tasks.The extra slide latent produced more accurate reports than mean-pooling.Source: §3 · p. 5
• BioGPT’s first 12 layers produce the report embedding, while layers 13–24 cross-attend to 512 Perceiver latents as context for report generation.The class token attends to all language tokens and represents the clinical report for contrastive alignment.Source: Figure 2 · p. 4, §3 · p. 5
• Contrastive training aligns paired slide and report projections while reducing similarity for unmatched pairs, using a learned temperature and symmetric cross-entropy.The projections have 5120 dimensions and are L2-normalized.Source: §3 · p. 5
• Report generation minimizes token-level negative log-likelihood with teacher forcing, and the final objective weights the contrastive and generative losses.The captioning objective may also improve instance-level representations compared with contrastive training alone.Source: §3 · pp. 5–6

4.1 Training protocols

• 587,196 whole slide images from 195,344 specimens, each with one or more slides and a clinical report, formed the training dataset.Source: §4.1 · p. 6
• Specimens with more than 100,000 tile embeddings were excluded to increase batch size and training speed.The authors state this does not limit clinical applicability because inference is usually performed on individual slides, which had fewer than 100,000 foreground tiles.Source: §4.1 · p. 6

4.2 Evaluation protocols

• Linear probing uses logistic regression on fixed PRISM slide embeddings, while fine-tuning updates the pre-trained slide encoder with a randomly initialized classifier; Virchow remains frozen.Source: §4.2 · pp. 6–7
• PRISM can autoregressively generate slide- or specimen-level reports describing tumor characteristics and relevant tissue features, or observed benign tissue.Decoding proceeds by selecting the highest-probability token until the end-of-sentence token is produced.Source: §4.2 · p. 7
• Zero-shot classification compares normalized slide and prompt embeddings using cosine similarity and a training-learned temperature, marginalizing scores across prompts within each class.Source: §4.2 · p. 7

4.3 Evaluation tasks and data

• Evaluation covers cancer detection, three binary cancer-subtyping tasks, and nine biomarker tasks; all methods use pre-trained Virchow tile embeddings.Cancer detection includes MSKCC and external-site cases, while subtyping includes TCGA and internal MSKCC data; most evaluation datasets contain out-of-distribution data relative to training.Source: §4.3 · pp. 7–8, Table 1 · p. 8

5.1 PRISM demonstrates diagnostic capability

• Pre-trained PRISM outperformed training from scratch across cancer detection and subtyping, with linear probing surpassing supervised baselines and zero-shot prediction in all settings.Fine-tuning was best on MSKCC cancer detection, while linear probing slightly exceeded fine-tuning there; fine-tuning approached linear probing on subtyping.Source: §5.1 · pp. 8–9
• 0.952 AUC versus 0.949 for ILC vs IDC and 0.975 AUC versus 0.980 for LUSC vs LUAD: zero-shot results were comparable to supervised baselines on TCGA subtyping.Each subtype used a single prompt without tuning or ensembling.Source: §5.1 · p. 8
• 0.868 AUC vs 0.925 on rare cancers and 0.906 AUC vs 0.947 on all cancers: zero-shot cancer detection was below the supervised baseline.The authors note that selected prompts may not cover all evaluation histologies, possibly contributing to reduced performance.Source: §5.1 · pp. 8–9

5.2 PRISM allows label-efficient training

• In 6 of 9 biomarker tasks, pre-training raised AUC by up to 5% for Breast-CDH1, and fine-tuning was more label-efficient in all six.Breast-CDH1 and bladder-FGFR needed 10% of training data to meet or exceed the maximum without pre-training; four other tasks required 20% to 50%.Source: §5.2 · p. 10

5.3 PRISM predictions are interpretable

• In four reviewed specimens, generated reports corresponded to histological features in attended tiles, but site-of-origin uncertainty could lead to incorrect subtype assessment.The pathologist confirmed IDC using a larger region when the selected tiles alone were insufficient; one reported origin was the rectum rather than ascending colon, which the authors say is histologically indistinguishable from rectum.Source: §5.3 · p. 10

6 Discussion

• Clinical reports can teach nuanced feature-prediction relationships, while generated reports support training-free zero-shot classification and interpretability.These are presented as motivations for PRISM’s use of natural-language reports.Source: §6 · p. 10
• WSI-level reports summarized by GPT-4 [1] address missing high-level descriptions, while a memory-efficient Perceiver addresses computational bottlenecks.Source: §6 · p. 10
• In one case, PRISM found sparse malignant cells but predicted serous carcinoma for a pleura biopsy reported as likely lung non-small-cell carcinoma.The report did not confirm the model’s metastasis suggestion.Source: §6 · p. 12
• Figure 6 pairs the top 16 attention-weighted tiles with generated reports; the WSI region at right is contextual and was not model input.Source: Figure 6
• Report alignment boosts in-domain transfer and benefits transfer to tasks not covered by reports (Fig. 5).The discussion attributes this to slide embeddings being encouraged to represent diverse features rather than clustering around a categorical target.Source: §6 · p. 13
• The approach depends on clinical reports covering population variation and rare cases, and reports may omit details or introduce noise, including linguistic-prior errors.Source: §6 · p. 13
• Further work could expand data and model scale, enrich specimen information, refine GPT-4 [1] rewriting, and explore larger, diverse contrastive batches.Source: §6 · p. 13

A.1 Zero-shot prompts

• All zero-shot tasks are binary; pan-cancer detection sums probabilities across multiple prompts per class to produce each class score.Source: Table A.1.1 · p. 16

A.2 Hyperparameters

• PRISM hyperparameters are specified for training on 16 NVIDIA V100 32GB GPUs.Source: Table A.2.1 · p. 17

A.3 Pancancer AUCs

• Table A.3.1 reports AUC by tumor-origin site for the MSKCC cancer detection task.Source: Table A.3.1 · p. 18

• TCGA BRCA contains 1,002 readable diagnostic slides selected for IDC or ILC: 798 IDC (80%) and 204 ILC (20%).Source: §A.4 · p. 19
• TCGA NSCLC contains 1,043 diagnostic slides: 531 LUAD (51%) and 512 LUSC (49%), after unreadable slides were discarded.Source: §A.4 · p. 19
• MSKCC DCIS/IDC contains 896 specimens: 346 DCIS (39%) and 550 IDC (61%), including rare invasive sub-types.Source: §A.4 · p. 19

• CDH1 positivity requires biallelic loss through loss-of-heterozygosity or a second somatic loss-of-function mutation; monoallelic variants are excluded.Source: §A.5 · p. 19
• Colon-MSI uses IHC and MSK-IMPACT sequencing for dMMR or MSI-H, prioritizing IHC when both results are available.Source: §A.5 · p. 20
• Bladder-FGFR labels specified FGFR3 mutations and FGFR3-TACC3 fusions or FGFR2 mutations using MSK-IMPACT data.Source: §A.5 · p. 20
• PTEN and EGFR labels use oncogenic mutation status from MSK-IMPACT and OncoKB, counting oncogenic-effect variants as positive and excluding unknown-status variants.Source: §A.5 · p. 20
• Gastric-HER2 labels amplification as HER2 IHC 2+/FISH+ or IHC 3+; the biomarker datasets' case, slide, and positive-label statistics are presented in a referenced table.Source: §A.5 · pp. 19–20, Table A.5.1 · p. 19
• Skin-BRAF uses oncogenic mutation and V600E status, prostate-AR uses copy-number fold change greater than 2, and ovarian-FGA uses a fraction of genome altered of at least 30%.Source: §A.5 · p. 20

A.6 Biomarker results

• Table A.6.1 reports top AUCs for biomarker tasks with and without PRISM pre-training as means and standard deviations across three runs, alongside training-set fractions.Source: Table A.6.1 · p. 20

A.7 GPT-4 prompt for clinical report rewriting

• The GPT-4 prompt requests five consistent, concise JSON summaries of diagnosis and tissue, including positive IHC or molecular findings while omitting negative results and measurements.Source: §A.7 · p. 21

A.8 Examples of generated reports

• Generated-report examples include major diagnostic discordances, such as invented endometrial carcinoma or lymphoma diagnoses, as well as mismatches in specimen details.Source: §A.8 · pp. 22–24

A.9 Perceiver implementation details

• The Perceiver uses Virchow tile embeddings and learned latents with cross-attention and a six-layer latent transformer; its slide embedding is returned from the latent features.Source: §A.9 · p. 26

A.10 Acronyms

• The appendix glossary expands abbreviations spanning biomarkers, cancer types, assays, and model terminology, including AUC, IHC, WSI, and ViT.Source: §A.10 · p. 27