PaperlayerView exact revision on arXiv

Source-linked AI summary

Multimodal Whole Slide Foundation Model for Pathology

Tong Ding, Sophia J. Wagner, Andrew H. Song, Richard J. Chen, Ming Y. Lu, Andrew Zhang, Anurag J. Vaidya, Guillaume Jaume, Muhammad Shaban, Ahrong Kim, Drew F. K. Williamson, Bowen Chen, Cristina Almagro-Perez, Paul Doucet, Sharifa Sahai, Chengkuan Chen, Daisuke Komura, Akihiro Kawabe, Shumpei Ishikawa, Georg Gerber, Tingying Peng, Long Phi Le, Faisal Mahmood

arXiv:2411.19666v1eess.IVcs.AIcs.CVcs.LGstat.AP

TL;DR

Patient- and slide-level clinical challenges remain difficult for patch-based pathology foundation models. TITAN is a multimodal whole-slide model that combines visual self-supervision with vision-language alignment, and it consistently outperforms other slide encoders across diverse downstream tasks, including data-constrained settings.

  • Problem

    Patch-based foundation models still face complex patient- and slide-level clinical challenges.

  • Method

    TITAN combines vision-only histology knowledge distillation with contrastive vision-language alignment to build a multimodal whole-slide foundation model.

  • Results

    TITAN consistently outperforms other state-of-the-art slide encoders across downstream tasks, including rare-disease settings and without task-specific finetuning.

  • Takeaways & Limitations

    Off-the-shelf multimodal slide embeddings show potential for assisting clinical retrieval and analysis across diverse pathology tasks.

  • Takeaways & Limitations

    The authors note a scope limitation from pretraining on 8K×8K region crops and extrapolating to entire whole-slide images.

Abstract

from arXiv · show

The field of computational pathology has been transformed with recent advances in foundation models that encode histopathology region-of-interests (ROIs) into versatile and transferable feature representations via self-supervised learning (SSL). However, translating these advancements to address complex clinical challenges at the patient and slide level remains constrained by limited clinical data in disease-specific cohorts, especially for rare clinical conditions. We propose TITAN, a multimodal whole slide foundation model pretrained using 335,645 WSIs via visual self-supervised learning and vision-language alignment with corresponding pathology reports and 423,122 synthetic captions generated from a multimodal generative AI copilot for pathology. Without any finetuning or requiring clinical labels, TITAN can extract general-purpose slide representations and generate pathology reports that generalize to resource-limited clinical scenarios such as rare disease retrieval and cancer prognosis. We evaluate TITAN on diverse clinical tasks and find that TITAN outperforms both ROI and slide foundation models across machine learning settings such as linear probing, few-shot and zero-shot classification, rare cancer retrieval and cross-modal retrieval, and pathology report generation.

Introduction

Computational pathology foundation models have advanced patch-level representation learning, but whole-slide clinical applications remain limited by WSI scale, scarce disease-specific cohorts, weak multimodal capabilities, and restricted evaluation. TITAN addresses these gaps with large-scale vision-only and vision-language pretraining for general-purpose slide representations and clinical tasks, including rare cancer retrieval.

  • Motivation: Patch-based foundation models capture histologic morphology, yet patient- and slide-level applications remain difficult because gigapixel WSIs and small disease-specific cohorts complicate clinical model development.Rare diseases are especially challenging because slide-level encoders often still require training from scratch, while slide and report retrieval require specialized bridging algorithms.
  • Limitations: Existing whole-slide foundation models are limited by vision-only pretraining, fewer training samples, insufficiently transferable representations, and limited evaluation in few-shot and retrieval settings.Vision-only approaches discard pathology-report supervision and multimodal capabilities, while current multimodal models still require end-to-end training or finetuning.
  • TITAN: TITAN is a Transformer-based multimodal whole-slide vision-language model that uses millions of high-resolution ROIs for resolution-agnostic pretraining and scalable WSI encoding.Its vision-only model was trained on 336K WSIs spanning 20 organ types and produces slide representations for subtyping, biomarker prediction, prognosis, and retrieval.
  • Contributions and results: Across diverse clinical tasks, TITAN outperforms supervised baselines and existing multimodal slide foundation models, particularly in low-data regimes, language-guided zero-shot classification, and rare cancer retrieval.The evaluation includes the first rare cancer retrieval application across 43 rare cancer types and demonstrates the utility and scaling potential of synthetic fine-grained morphological descriptions.
  • Vision-language pretraining: TITAN’s language extension uses 423K synthetic fine-grained ROI captions and 183K slide-level pathology reports to support report generation, zero-shot classification, and cross-modal retrieval.The synthetic captions were generated with PathChat, a multimodal generative AI copilot for pathology.

Results

TITAN and TITANV outperform other slide encoders across diverse pathology tasks, with particularly strong gains in morphological subtyping and whole-slide-context evaluations. Results also show that scaling pretraining data, ALiBi positional encoding, and pretrained initialization improve downstream performance.

  • Scaling analysis: Using all Mass-340K pretraining data improves TITANV performance across all four tasks, with average gains of 3.65%, 3.21%, and 1.21% over 12.5%, 25%, and 50% subsets.This establishes a scaling trend as more pretraining images are used.
  • Benchmark results: TITAN and TITANV outperform other slide encoders overall, with TITAN achieving +8.4% and TITANV +6.7% over PRISM on morphological subtyping tasks.The gains are averaged across multi-class balanced accuracy and binary AUROC, spanning fine-grained cancer classification and non-cancerous tasks such as allograft rejection.
  • Benchmark results: TITANV benefits from long-context extrapolation, exceeding PRISM by +7% and +9.5% on TCGA-OT, +10% and +16% on OT108, and +9% and +9.1% on EBRAINS.TITANV and TITAN also outperform PRISM on TCGA-UT-8K by +6% and +7.5%, respectively.
  • Evaluation settings: Pretrained TITAN weights outperform random initialization by 3.63% on average, while TITAN’s linear probe surpasses supervised ABMIL by encoding richer contextual and semantic slide details.Task-specific fine-tuning generally improves over linear probing, but pretrained initialization remains advantageous for limited-patient cohorts.

Discussion

TITAN extends self-supervised learning to multimodal whole-slide pathology representations, combining vision-only distillation with ROI-caption and slide-report alignment. It performs broadly without task-specific finetuning, while remaining limited by dataset scale, context coverage, and report preprocessing.

  • Contribution: TITAN combines slide-level self-supervised learning, histology knowledge distillation, and vision-language alignment between ROIs, synthetic captions, whole-slide images, and pathology reports.It uses a Vision Transformer with ALiBi positional encoding and was pretrained on 336K WSIs, 423K ROI captions, and 183K pathology reports.
  • Results: TITAN consistently outperforms state-of-the-art slide encoders, including PRISM, GigaPath, and CHIEF, from cancer subtyping through molecular classification and rare disease retrieval.These results hold in data-constrained settings such as rare disease classification and histology slide retrieval.
  • Multimodal capabilities: Multimodal alignment enables zero-shot diagnosis, slide-report retrieval, and report generation, while combining ROI captions with pathology reports captures fine-grained and coarse morphological information.Generated ROI captions made this multiscale alignment insight possible for the first time in the described framework, and TITAN outperforms PRISM on all cross-modal tasks.
  • Ablations: Ablations show that unimodal pretraining captures patch-feature context efficiently, while vision-language alignment further improves slide representations, especially for untuned embedding-space methods.TITAN improves over TITANV on average for slide-level tasks and outperforms all baselines, including TITANV, on slide retrieval and few-shot tasks.
  • Clinical potential: Off-the-shelf TITAN retrieves diagnostically similar slides and reports across more than 100 cancer types, including rare cancers, without dedicated algorithms for each task.The authors argue this demonstrates generalizable slide embeddings that simplify slide-level clinical tasks and could support routine diagnostic workflows.
  • Limitations and future directions: Limitations include fewer pretraining slides than some competing datasets, incomplete context from 8K×8K crops, and difficult manual tuning of morphology-focused report preprocessing.The paper proposes expanding Mass-340K, using larger contexts or alternative positional encodings, restructuring reports, and scaling data and architecture.

Online Methods · Pretraining dataset

TITAN’s pretraining used Mass-340K, a diverse 335,645-WSI collection spanning 20 organs and H&E/IHC stains, supplemented by synthetic ROI captions and curated slide-report pairs. These datasets were processed to preserve representative morphology, align text with individual slides, remove irrelevant sensitive information, and diversify language.

  • Pretraining dataset: Mass-340K comprises 335,645 WSIs across 20 organs, with 90% H&E and 10% IHC slides sourced in-house and from GTEx consortium114.
  • Pretraining dataset: Three additional Mass-340K partitions contained 12.5%, 25%, and 50% of the original dataset while preserving source ratios and organ distribution.
  • Synthetic caption generation using PathChat: For Stage 2 vision-language alignment, PathChat79 generated detailed morphological captions for 423,122 diverse 8,192×8,192 ROIs sampled from Mass-340K.
  • Synthetic caption generation using PathChat: Because PathChat79 could not process 8,192×8,192 inputs directly, each ROI was divided into 64 1,024×1,024 patches.
  • Synthetic caption generation using PathChat: K-means clustering with K = 16 selected one patch per cluster, producing 16 morphologically representative patches for PathChat79.
  • Synthetic caption generation using PathChat: Qwen2-7B-Instruct87 rewrote generated captions to increase language diversity, with prompts and examples provided in Extended Table 124-125.
  • Curation of slide-report dataset: For Stage 3 alignment, 182,862 slide-report pairs combined in-house clinical reports with pathology notes from GTEx consortium114.
  • Curation of slide-report dataset: To resolve patient-level report noise and multi-slide ambiguity, Qwen2-7B-Instruct87 extracted slide-specific descriptions, removed unrelated sensitive information, and diversified report text.Removed content included gross descriptions, hospital and doctor names, and patient clinical history.

Unimodal visual pretraining

TITAN’s unimodal visual pretraining pipeline segments WSIs into tissue patches, extracts robust region features with CONCHv1.5, and trains a slide encoder with iBOT on spatially organized tissue groups. It uses multi-view region sampling and 2D positional encoding to support whole-slide inference from region-crop training.

  • Preprocessing and feature extraction: WSIs were tissue-segmented, tiled into non-overlapping 512×512-pixel patches at 20× magnification, and encoded with CONCHv1.5 features.CONCHv1.5 was trained on 1.26 million image-caption pairs using the CoCa objective and covers diverse stains and tissue types.
  • Preprocessing and feature extraction: Tissue contours were spatially grouped to improve ROI sampling, excluding groups with fewer than 16 patches and producing 345,782 tissue groups from Mass-340K.Grouping addresses dispersed biopsy fragments and slides containing multiple tissue slices.
  • iBOT pretraining: TITANV was trained with iBOT, combining student–teacher knowledge distillation and masked image modeling in the patch-embedding space.Training uses randomly sampled 8,192×8,192-pixel region crops represented as 16×16 feature grids, with two global and ten local views.
  • iBOT pretraining: Random region sampling increases training diversity by exposing the model to different parts of the same WSI across epochs.Embedding-space frozen feature augmentations avoid the computational cost of generating many pixel-space or generative feature-space augmentations.

Vision-language continual pretraining

TITANV uses two-stage continual vision-language pretraining to address scarce slide-report pairs, brief clinical reports, and computationally costly WSI contrastive learning. It progressively aligns synthetic ROI captions and real WSI reports to build fine-grained, diagnostic, and global-context representations for WSI understanding and generation.

  • Vision-language continual pretraining: The approach addresses scarce paired slide-report data, brief clinical reports, and large-batch requirements that make contrastive learning computationally infeasible for WSIs.Educational ROI captions provide more detailed morphological descriptions than typical real-world clinical reports.
  • Vision-language continual pretraining: Two-stage continual pretraining first aligns synthetic captions for 8,192×8,192 ROIs, then aligns real clinical reports for whole-slide images.The stages progressively increase context from local ROI morphology to full-scale WSI context.
  • Vision-language continual pretraining: The first stage learns fine-grained pathological concepts from detailed morphological captions, while the next targets diagnostic terminology, reasoning, and zero-shot downstream understanding.The second stage also acts as high-resolution fine-tuning from ROI-local contexts to global WSI contexts.
  • Vision-language continual pretraining: CoCa combines TITANV’s image encoder with attentional poolers for a global WSI representation supporting contrastive evaluation and 128 image tokens supporting multimodal caption generation.The model uses pretrained CONCHv1.5 text encoders and multimodal decoders.

Evaluation setting · Baselines · Linear and K-nearest neighbors probe evaluation

The evaluation tests slide-encoder transfer and representation quality with linear and k-nearest-neighbor probes, comparing TITAN with unsupervised, supervised, and cross-modal baselines. Baselines include recent slide encoders, mean-pooled patch features, attention-based MIL, and pretrained-encoder finetuning.

  • Baselines: TITAN is compared with unsupervised slide encoders Prov-GigaPath (GigaPath)58, PRISM62, CHIEF74, and mean-pooled patch-feature baselines.Mean pooling is primarily evaluated with CONCHv1.5 features and additionally with each slide encoder’s corresponding patch encoder on a subset of tasks.
  • Baselines: Cross-modal zero-shot retrieval and clinical report generation use TITAN alongside the cross-modal baseline PRISM62.The supplied passages identify these comparison tasks but do not provide their results.
  • Baselines: GigaPath uses LongNet with dilated attention, was trained on 171,189 in-house WSIs by masked autoencoding, and uses ViT-G/14 DINOv2 patch features.Only its unimodal model is assessed because the multimodal model is not publicly available; Transformer layer 11 supplies the slide representation.
  • Baselines: PRISM62 uses a Perceiver slide encoder with CoCa-based vision-language alignment on 195,344 specimen-report pairs totaling 587,196 WSIs.Its patch encoder is Virchow 11, a DINOv2-pretrained ViT-H/14 trained on an in-house dataset.
  • Baselines: CHIEF74 aggregates features with attention and was trained through slide-level contrastive learning using anatomic-site information and CTransPath4 patch features.CTransPath4 is a self-supervised SwinTransformer121 trained on 15 million patches.
  • Baselines: The evaluation also includes supervised attention-based MIL (ABMIL)45,67 and finetuning of pretrained slide encoders.For validation-equipped tasks, the best model is selected by validation loss; publicly available finetuning code is used for GigaPath and CHIEF.
  • Linear and K-nearest neighbors probe evaluation: Transfer capability and fixed-embedding quality are assessed using linear logistic-regression probes and non-parametric k-nearest-neighbor probing.Linear probing minimizes cross-entropy with scikit-learn L-BFGS and validation-selected ℓ2 regularization; k-NN uses either SimpleShot class prototypes or k = 20 with Euclidean distance on centered, normalized embeddings.

Slide retrieval … Survival analysis

The paper evaluates slide representations through uni-modal and cross-modal retrieval, few-shot classification, and survival analysis. These evaluations use centered and normalized embeddings, task-specific similarity or predictive models, and cohort-aware validation procedures.

  • Slide retrieval: Slide retrieval uses training and validation WSIs as candidate keys and test WSIs as query slides for content-based retrieval.Embeddings are centered by subtracting their Euclidean centroid and then ℓ2-normalized before retrieval.
  • Slide retrieval: Retrieval ranks candidates by ℓ2 distance, with performance measured by Acc@K for K ∈{1, 3, 5} and MVAcc@5 using class labels.Acc@K tests whether any top-K slide shares the query label, whereas MVAcc@5 uses the majority label among the top five.
  • Cross-modal retrieval: Cross-modal retrieval embeds slides and reports into a shared vision-language space, then evaluates both slide-to-report and report-to-slide retrieval.Both modalities are ℓ2-normalized, and pairwise cosine similarity determines retrieval rankings.
  • Cross-modal retrieval: Class-based cross-modal retrieval counts a result as successful when it belongs to the query’s diagnostic class and reports Recall@K for K ∈{1, 3, 5, 10}.Mean recall averages the four Recall@K values.
  • Few-shot slide classification: Few-shot classification varies the number of shots per class across k in {1, 2, 4, 8, 16, 32}, using all available samples when a class has fewer than k.The evaluation follows the SimpleShot 100 framework for self-supervised models.
  • Few-shot slide classification: SimpleShot averages few-shot samples into class prototypes and classifies test embeddings by prototype distance after centering and normalization.Linear probing is additionally assessed for comparability with supervised baselines such as ABMIL.
  • Survival analysis: Survival analysis fits a linear Cox proportional hazards model to disease-specific survival using a single slide embedding rather than a bag of patch embeddings.Patients can be batched, unlike typical memory-constrained single-patient MIL survival prediction.
  • Survival analysis: To reduce batch effects, survival modeling uses five-fold site-preserved stratification, with four folds for training and the remaining fold for evaluation without a validation fold.The regularization coefficient C is selected from 25 logarithmically spaced α values between 10^1 and 10^5 for each encoder and cancer type.

Zero-shot slide classification … Statistical analysis

The section specifies TITAN’s zero-shot slide-classification procedure, report-generation evaluation, task-specific metrics, and statistical-analysis protocols. Zero-shot classification uses cosine similarity between slide and class-text embeddings, while report generation is evaluated against ground-truth pathology reports and results are summarized with cross-validation or bootstrap statistics.

  • Zero-shot slide classification: Zero-shot slide classification predicts each slide’s class from the highest cosine similarity between its slide embedding and class-text embedding.Class text prompts are encoded and compared with ℓ2-normalized slide embeddings using CLIP104-style prediction logits.
  • Zero-shot slide classification: An ensemble of prompts is created for each class to reduce sensitivity to the specific text-prompt choice.The complete prompt ensembles are provided in Extended Data Table 101.
  • Report generation: TITAN’s zero-shot captioning is evaluated on 10,108 TCGA slide-report pairs against PRISM 62.The experiment uses CoCa’s generative objective to produce concise pathology reports and applies beam-search decoding with five beams and one beam group.
  • Evaluation metrics: Classification tasks report balanced accuracy and weighted F1-score for multiclass tasks, quadratic weighted Cohen’s κ for ordinal tasks, and balanced accuracy with AUROC for binary tasks.Survival tasks use the concordance index to measure agreement between predicted risks and actual survival times.
  • Evaluation metrics: Slide retrieval is evaluated with Acc@K for K ∈1, 3, 5 and MVAcc@5, measuring top-K class recovery and majority-vote correctness.Acc@K requires at least one matching-class slide among the retrieved results, whereas MVAcc@5 evaluates the majority class of the top five.
  • Evaluation metrics: Cross-modal retrieval reports Recall@K for K ∈1, 3, 5, 10 and mean recall, the average of the four Recall@K values.Recall@K measures the proportion of queries whose correct result appears among the top-K retrieved items.
  • Evaluation metrics: Generated pathology reports are compared with ground-truth reports using METEOR106, ROUGE107, and BLEU108, including ROUGE-1 and BLEU-1.METEOR considers unigram precision, recall, synonyms, stemming, and word order; ROUGE emphasizes overlap recall, while BLEU-1 evaluates word-level precision.
  • Statistical analysis: Five-fold datasets report mean performance and standard deviations across folds, whereas single-split datasets use 1,000-sample non-parametric bootstrapping to calculate mean and standard deviation.The statistical protocol differs according to whether datasets have five-fold or single splits.

Downstream evaluation datasets

The downstream evaluation spans newly introduced TCGA, slide-report, and rare-cancer datasets alongside existing pathology benchmarks, using slide-level TITAN inputs and patient embeddings averaged across slides. These datasets cover pan-cancer subtyping, report alignment, rare-cancer retrieval, and diverse clinical classification settings with site- or patient-preserving splits.

  • Evaluation setup: TITAN uses each whole-slide image as one input, while patient-level embeddings average the TITAN slide embeddings for that patient.Pre-extracted CONCHv1.5 patch features are rearranged into tissue-centered feature grids, with background features masked.
  • Introduced datasets: TCGA-Slide-Reports contains 10,108 slide-level WSI-report pairs, created by matching block diagnoses to slide-level diagnoses with GPT4o-mini.The original patient reports often describe multiple tissue blocks, creating a one-to-one alignment challenge for report generation and cross-modal retrieval.
  • Rare-cancer evaluation: Rare-Cancer-Public contains 1,982 WSIs spanning 28 rare cancer types, while the in-house Rare-Cancer extension contains 3,039 WSIs across 43 types and 186 OncoTree codes.Rare-cancer retrieval uses patient-level folds, with one fold queried against remaining rare and common cancer slides as support.

Computing Software and Hardware

The study used PyTorch-based training with modified iBOT and CoCa implementations, distributed across NVIDIA GPUs, while downstream experiments and whole-slide processing relied on dedicated software and hardware.

  • Training infrastructure: TITANV and TITAN training modified public iBOT and CoCa implementations and used four and eight × 80GB NVIDIA A100 GPUs, respectively, with distributed data-parallel training.PyTorch 2.0.1 with CUDA 11.8 supported deep-learning training and inference.
  • Downstream processing: Downstream experiments ran on single 24GB NVIDIA 3090 GPUs, and whole-slide image processing used OpenSlide, openslide-python, and CLAM.The study used Python 3.9.16 for all experiments and analyses.
  • Evaluation software: Scikit-learn implemented K-Nearest Neighbors, logistic regression, and SimpleShot, while scikit-survival supported survival tasks.Weakly supervised ABMIL training adapted the CLAM training scaffold.
  • Visualization and benchmarks: Matplotlib and Seaborn created plots and figures, while benchmarked slide-encoder implementations included GigaPath, PRISM, and CHIEF.Other miscellaneous Python libraries were listed in the Reporting Summary.

Data availability

The study’s pretraining and benchmark datasets are available through their respective public portals, with TITAN providing TCGA-UniformTumor-8K coordinates and labels and Extended Data Table 13 collecting dataset links.

  • Data availability: GTEx pretraining data are accessible through the GTEx portal, while TCGA and CPTAC benchmark data are available through the NIH Genomic Data Commons and Proteomics Data Commons, respectively.
  • Data availability: Coordinates and labels for the TCGA-UniformTumor-8K dataset are publicly available in the TITAN GitHub repository.
  • Data availability: Other publicly available benchmark datasets can be accessed through their respective data portals, including EBRAINS, with links compiled in Extended Data Table 13.

Code availability

TITAN and TITANV code and model weights are available for academic research, while the retrospective study used IRB-approved, anonymized archival pathology data with waived informed consent.

  • Code availability: Code and model weights for loading TITAN and TITANV are available for academic research.
  • Ethics Statement: The retrospective analysis of internal pathology images and reports received Mass General Brigham institutional review board approval.
  • Ethics Statement: All internal digital data were anonymized, and informed consent was waived because the study used archival slides without direct patient participation or recruitment.

AUROC

TITAN reports higher AUROC values than PRISM across all three reported comparisons, while evaluations span molecular classification, rare-cancer retrieval, and generated pathology reports.

  • AUROC: Linear-probe evaluation covers binary molecular-status prediction, five-fold TCGA splits with external CPTAC and EBRAINS datasets, and six-level ER and PR prediction from MGH IHC slides.Extended Data Figure 2 defines the molecular classification evaluation scope.
  • AUROC: TITAN and TITANV slide embeddings are visualized with slide-encoder baselines across organs in the TCGA-OT cohort.The UMAP is intended for visual decluttering rather than reporting numerical AUROC comparisons.
  • AUROC: Ablations compare mean pooling, ABMIL, linear probing, and fine-tuning from pretrained or random initialization, including TITANL without vision pretraining.The experiments report changes in balanced accuracy relative to the linear probe across four subtyping tasks.
  • AUROC: Generated reports from TITAN describe renal clear cell carcinoma, prostate adenocarcinoma with invasion, and hepatocellular carcinoma with organ-specific findings.The examples compare TITAN-generated reports with corresponding clinical reports and PRISM-generated reports.
  • AUROC: Rare-cancer retrieval examples show highly similar retrieved slides sharing either the query’s diagnostic label or organ.Extended Data Figure 6 identifies cosine similarity as the retrieval score.
  • AUROC: 0.761±0.0063, 0.798±0.0043, and 0.989±0.0006 are TITAN’s reported values, exceeding PRISM’s 0.536±0.0061, 0.599±0.0055, and 0.966±0.0010.The passage lists three corresponding values for PRISM and TITAN, but does not label the individual metrics or tasks.
Loading 2411.19666v1…