PaperlayerView exact revision on arXiv

Source-linked AI summary

Molecular-driven Foundation Model for Oncologic Pathology

Anurag Vaidya, Andrew Zhang, Guillaume Jaume, Andrew H. Song, Tong Ding, Sophia J. Wagner, Ming Y. Lu, Paul Doucet, Harry Robertson, Cristina Almagro-Perez, Richard J. Chen, Dina ElHarouni, Georges Ayoub, Connor Bossi, Keith L. Ligon, Georg Gerber, Long Phi Le, Faisal Mahmood

arXiv:2501.16652v1cs.CVcs.AI

TL;DR

Pathology foundation models often encode only small regions of whole-slide images, limiting their use for diverse oncology tasks. THREADS uses molecularly guided multimodal pretraining to generate slide-level representations and achieved state-of-the-art performance across 54 oncology tasks.

  • Problem

    Existing pathology foundation models generally encode small regions rather than complete whole-slide images, while downstream oncology tasks often have scarce labels.

    Source: Introduction · p. 2

  • Method

    THREADS uses multimodal contrastive pretraining on over 47,000 whole-slide images paired with genomic or transcriptomic profiles to generate slide-level embeddings.

    Source: Introduction · p. 2, Whole-slide image classification with… · p. 4

  • Results

    THREADS achieved state-of-the-art performance across 54 oncology tasks and significantly outperformed whole-slide encoders and attention-based multiple-instance-learning baselines.

    Source: Introduction · p. 2, Discussion · p. 13

  • Takeaways & Limitations

    THREADS supports off-the-shelf transfer learning for data-scarce oncology applications, including rare-disease and specialized prediction tasks.

    Source: Discussion · p. 14

    Despite training on over 47,000 histomolecular pairs, THREADS cannot encompass the full spectrum of molecular and morphological heterogeneity.

    Source: Discussion · p. 14

Abstract

from arXiv · show

Foundation models are reshaping computational pathology by enabling transfer learning, where models pre-trained on vast datasets can be adapted for downstream diagnostic, prognostic, and therapeutic response tasks. Despite these advances, foundation models are still limited in their ability to encode the entire gigapixel whole-slide images without additional training and often lack complementary multimodal data. Here, we introduce Threads, a slide-level foundation model capable of generating universal representations of whole-slide images of any size. Threads was pre-trained using a multimodal learning approach on a diverse cohort of 47,171 hematoxylin and eosin (H&E)-stained tissue sections, paired with corresponding genomic and transcriptomic profiles - the largest such paired dataset to be used for foundation model development to date. This unique training paradigm enables Threads to capture the tissue's underlying molecular composition, yielding powerful representations applicable to a wide array of downstream tasks. In extensive benchmarking across 54 oncology tasks, including clinical subtyping, grading, mutation prediction, immunohistochemistry status determination, treatment response prediction, and survival prediction, Threads outperformed all baselines while demonstrating remarkable generalizability and label efficiency. It is particularly well suited for predicting rare events, further emphasizing its clinical utility. We intend to make the model publicly available for the broader community.

Introduction

THREADS is a multimodal pathology foundation model that generates whole-slide image embeddings guided by molecular profiles, addressing data scarcity and the scale limitations of patch-based models. It was evaluated across 54 oncology tasks and significantly outperformed existing whole-slide encoders and multiple-instance learning baselines.

  • Pathology prediction is difficult because oncology cohorts are often small while whole-slide images can occupy several gigabytes, creating a high-dimensional, low-sample training regime.Many prognostication and treatment-response studies involve fewer than 100 patients.Source: Introduction · p. 2
  • Existing pathology foundation models mainly encode small image patches, while slide-level models remain constrained by organ- or disease-specific training data and limited predictive capability.Additional aggregation models can be computationally expensive and label-intensive to train.Source: Introduction · p. 2
  • THREADS generates whole-slide image embeddings through multimodal contrastive pretraining guided by next-generation sequencing molecular profiles.The authors posit that molecular data provides a holistic view of tissue morphology and clinically relevant composition.Source: Introduction · p. 2
  • MBTG-47K contains 47,171 paired whole-slide images and molecular profiles from 39 organs across MGH, BWH, TCGA, and GTEx.The dataset includes 6,899 MGH, 20,556 BWH, 10,209 TCGA, and 9,507 GTEx samples.Source: Introduction · p. 2
  • THREADS significantly outperformed PRISM, GIGAPATH, CHIEF, and attention-based multiple-instance learning baselines across 54 oncology tasks (all P-value<0.001).It also significantly improved fine-tuning over training from scratch (P-value<0.001).Source: Introduction · p. 2, Figure 1 · p. 3

Results

THREADS achieved superior performance across broad whole-slide imaging benchmarks, including data-scarce prediction, fine-tuning, retrieval, molecular prompting, and scaling analyses. Its multimodal molecular alignment produced transferable and clinically relevant slide representations.

Data and label efficiency of THREADS

  • THREADS outperformed baselines in all seven treatment-response tasks and five of six survival tasks, with significant overall advantages and strong performance in label-scarce settings.Few-shot experiments showed THREADS was best for most training-sample sizes and, for rare brain tumors, k=4 outperformed PRISM at k=16.Source: Data and label efficiency of THREADS · p. 6

Whole-slide image classification with THREADS

  • THREADS significantly outperformed PRISM, GIGAPATH, and CHIEF in linear probing across 54 tasks, with absolute gains of 6.3%, 9.9%, and 6.7%, respectively.It won in 44/54 tasks against PRISM, 49/54 against CHIEF, and 54/54 against GIGAPATH; Figure 2 summarizes the benchmark comparison.Source: Whole-slide image classification with… · p. 5, Figure 2 · p. 7

Retrieval capabilities of THREADS

  • THREADS produced the best retrieval performance across all k values for rare brain tumors and outperformed all baselines on CPTAC retrieval at k=1, 5, and 10.Retrieval used mean Average Precision at k on 2,319 EBRAINS cases and 2,115 CPTAC slides, demonstrating clinically relevant case similarity without additional training.Source: Retrieval capabilities of THREADS · p. 8

THREADS fine-tuning

  • THREADS fine-tuning exceeded CHIEF and GIGAPATH by 17.9% and 7.3%, respectively, and surpassed attention-based MIL trained from scratch by 4.3% across 54 tasks.The largest gains occurred in mutation prediction, treatment response and survival, and IHC status, while clinical subtyping and grading remained comparable to competing fine-tuned models.Source: THREADS fine-tuning · p. 8, Figure 3 · p. 9

Molecular prompting with THREADS

  • Molecular prompting achieved competitive zero-shot performance, including 0.960 AUC for IDH status and 0.687 C-index for renal-cell-carcinoma prognosis.The method forms class-wise molecular prototypes and assigns query slides to the nearest prototype without task-specific model development.Source: Molecular prompting with THREADS · p. 10
  • THREADS improved with more training data by 3.9% overall and was parameter-efficient, while two attention heads performed best and six heads reduced performance by 1.0%.Its embedding gains over mean pooling reached 6.36%, and clustering exceeded baselines, including ARI 0.654 versus 0.354 for CHIEF.Source: Molecular prompting with THREADS · pp. 10, 13

Discussion

THREADS uses molecular-guided multimodal pretraining to generate biologically and clinically relevant whole-slide representations, achieving state-of-the-art performance across 54 oncology tasks and diverse evaluation settings. Its generalizability, label efficiency, and clinical potential are balanced by unresolved scaling questions and limitations in molecular and architectural coverage.

  • THREADS achieved state-of-the-art performance across 54 oncology tasks, including subtyping, grading, mutation, immunohistochemistry, treatment response, survival, generalization, few-shot learning, and retrieval.Simple linear models built on THREADS embeddings reached clinical-grade performance for subtyping and grading, while the model also showed potential for outcome and treatment-response prediction.Source: Discussion · p. 13
  • THREADS’s molecular-profile alignment provides an objective pretraining signal that captures morphological features reflecting tissue molecular composition, unlike pathology-report matching or weaker tissue-site and masked-autoencoding signals.The discussion contrasts THREADS with PRISM 13, CHIEF 11, and GIGAPATH 6, while noting that the optimal model and dataset scaling point remains unresolved.Source: Discussion · p. 13
  • Larger and more diverse pretraining datasets mainly benefit challenging treatment-response and molecular prediction tasks, while THREADS’s diversity across 39 tissue sites may support broad generalizability despite its smaller model size.THREADS is 7.5 × smaller than GigaPath and 4.0 × smaller than PRISM, suggesting model size alone may not determine general-purpose slide representation quality.Source: Discussion · pp. 13–14
  • THREADS cannot capture the full molecular and morphological spectrum, and future scaling may require larger cohorts, additional assays, and architectural changes suited to slide-level context.The current architecture treats patch embeddings independently; Vision Transformer replacements underperformed simpler models, while 512 × 512-pixel patches may reduce the need for explicit context modeling.Source: Discussion · p. 14
  • THREADS enables low-cost, data-efficient prototyping and transfer learning for rare diseases, rare molecular alterations, treatment response, resistance prediction, and clinical retrieval of rare conditions.Future work will scale the pretraining dataset and increase biological richness by incorporating additional modalities.Source: Discussion · p. 14

Online Methods

The study was a retrospective analysis of de-identified pathology slides, corresponding NGS assays, and reports approved by the MGB institutional review board, with informed consent waived.

Ethics and study design

  • MGB’s institutional review board approved the retrospective analysis of pathology slides, NGS assays, and corresponding reports; informed consent was waived and slides were de-identified before digitization.Participants were not directly involved or recruited, and sample sizes were determined by data availability.Source: Online Methods · p. 14

Pretraining dataset curation

MBTG-47K is a 40.7-TB multimodal pretraining dataset comprising 47,171 FFPE H&E whole-slide images paired with bulk RNA expression and DNA variant data across 39 organs. It integrates curated molecular and histological data from MGH, BWH, TCGA, and GTEx, with source-specific processing and gene-selection strategies.

Dataset overview

  • MBTG-47K contains 47,171 WSIs from 39 major organs, including 26,615 RNA-paired and 20,556 DNA-variant-paired slides, totaling 125,148,770 tiles and 40.7 TB.The dataset combines FFPE H&E WSIs with bulk RNA expression and SNV, indel, and CNV data from MGH, BWH, GTEx, and TCGA.Source: Pretraining dataset curation · p. 15

MGH

  • MGH contributed 6,899 FFPE H&E WSI–bulk RNA pairs across 25 tissue sites using FusionPlex assay data and log2-TPM normalization across 54 genes.RNA abundance was quantified with Kallisto 0.50.1, summed across isoforms, and not subjected to additional batch-effect normalization; the dataset totaled 11.0 TB.Source: Pretraining dataset curation · p. 15

BWH

  • BWH contributed 20,556 FFPE H&E slides across 32 tissue sites with OncoPanel SNV, indel, and CNV data encoded across 239 genes into 1,673-element vectors.OncoPanel testing followed routine clinical workflows with expert molecular pathologist review, and the final BWH dataset totaled 12.0 TB. [32]Source: Pretraining dataset curation · p. 15

TCGA

  • TCGA contributed 10,209 FFPE H&E WSIs from 32 cancer types paired with bulk RNA expression, selecting 4,848 encoder-vocabulary genes from 4,917 cancer-related genes. [33]TCGA RNA measurements were log2-transformed TPM values without additional batch-effect normalization, and the final dataset totaled 10.5 TB.Source: Pretraining dataset curation · pp. 15–16

GTEx

  • GTEx contributed 9,507 FFPE H&E WSI–bulk RNA pairs from 29 tissue sites, selecting 4,932 scGPT-vocabulary genes from the 5,000 most variable genes. [12, 18]The non-cancerous dataset totaled 7.2 TB and used log2-TPM normalization without additional batch-effect normalization.Source: Pretraining dataset curation · p. 16

Downstream tasks and datasets

The benchmark spans 54 tasks across 23 datasets and nine organs, covering morphology, grading, IHC, mutation, treatment-response, and survival prediction. It combines internal and public cohorts with external validation and standardized all-data and few-shot splits.

Downstream tasks and datasets

  • The benchmark includes 54 tasks from 23 datasets across nine major organs, spanning six task types: morphology, grading, IHC, mutation, treatment response, and survival.Tasks are documented in Extended Data Tables 2–7.Source: Downstream tasks and datasets · p. 16
  • Cohorts cover breast, lung, renal, colorectal, prostate, brain, ovarian, glioblastoma, pancreatic, and other cancers using internal and public datasets.Examples include MGB-Breast and MGB-Lung, BCNB, MUT-HET-RCC, IMP, PANDA, BRACS, EBRAINS, and CPTAC.Source: Downstream tasks and datasets · pp. 16–17
  • Treatment-response and survival evaluations use clinically defined outcomes across ovarian, prostate, glioblastoma, breast, colorectal, metastatic breast, and ovarian cancer cohorts.Response labels include bevacizumab response, residual cancer burden after NADT, post-treatment survival, lymphovascular invasion, RECIST1.1, and progression-free survival.Source: Downstream tasks and datasets · pp. 17–18
  • TCGA cohorts provide external generalizability tests for IDH, breast and lung subtyping, breast IHC, and survival in LUAD and PDAC.External cohorts include EBRAINS, MGB-Breast, BCNB, MGB-Lung, CPTAC-LUAD, and CPTAC-PDAC.Source: Downstream tasks and datasets · pp. 18–19
  • Evaluation uses all-sample and few-shot splits, with 1, 2, 4, 8, 16, or 32 examples per class when sufficient labels exist.Splits use official folds, 80:20 cross-validation, or bootstrapping while keeping each task’s test set fixed across splits.Source: Data splits · p. 19

Model design and development

THREADS combines tissue-aware patch processing, multimodal molecular encoders, and attention-based slide encoding to produce 1024-dimensional whole-slide representations. Pretraining uses rank-based model selection, while downstream fine-tuning uses a fixed five-epoch protocol.

Model design

  • THREADS processes each whole-slide image through tissue detection and patching, patch-level feature extraction, and slide encoding.Only tissue-overlapping regions are processed, using non-overlapping 512×512-pixel patches at 20× magnification.Source: Model design and development · p. 19

Patch encoder

  • The patch encoder is CONCHV1.5, initialized from UNI and fully fine-tuned on 1.17 million pathology image-caption pairs before processing resized and ImageNet-normalized slide patches.UNI was trained on 100 million pathology image patches, while CONCHV1.5 uses 448×448-pixel inputs.Source: Model design and development · p. 19

Slide encoder

  • THREADS uses gated attention-based multiple instance learning to aggregate patch embeddings into 1024-dimensional slide representations, with configurable single- or multi-headed attention.Multi-headed attention concatenates head-specific features before a linear projection to the final 1024-dimensional embedding.Source: Model design and development · pp. 19–20

Molecular encoders

  • The molecular branch encodes bulk RNA expression with a modified scGPT initialized from a pancancer checkpoint and encodes SNVs and CNVs with a multi-hot multilayer perceptron.The scGPT pathway fine-tunes all encoder layers, while the mutation encoder maps a 1673-dimensional hidden representation to 1024 dimensions.Source: Model design and development · p. 20

Pretraining and model selection

  • Pretraining runs for up to 101 epochs on four 80GB A100 GPUs with rank-based checkpointing, selecting models when the smooth rank of training-set slide embeddings increases.Smooth rank is used to monitor embedding diversity and avoid collapse without repeatedly optimizing downstream test performance.Source: Model design and development · pp. 20–21

Finetuning

  • Downstream fine-tuning uses weighted cross-entropy for five epochs with batch size 1 and 2,048 sampled patches per batch, without early stopping or weight decay.Fine-tuning uses AdamW at a base learning rate of 0.000025 on a 24GB NVIDIA 3090Ti with bf16 precision.Source: Model design and development · p. 21

Baselines

Threads was evaluated against four baseline families: GIGAPATH, PRISM, CHIEF, and attention-based multiple instance learning (ABMIL). These baselines span pretrained slide encoders, mean-pooled patch embeddings, and an attention-based aggregation model.

Baselines

  • Threads was compared with GIGAPATH, PRISM, CHIEF, and attention-based multiple instance learning (ABMIL).GIGAPATH, PRISM, and CHIEF provide pretrained slide-level encoders, while mean-pooling variants were also defined for their patch embeddings.Source: Baselines · p. 21, GIGAPATH · pp. 21–22

GIGAPATH

  • GIGAPATH used a ViT patch encoder pretrained on 171,000+ pan-cancer WSIs and a LongNet slide encoder, with frozen patch features and task-specific finetuning.Patch features were extracted at 20× magnification, and the official slide-encoder recipe added a randomly initialized classification head.Source: GIGAPATH · pp. 21–22

RESNET50-IN

  • RESNET50-IN formed a baseline by averaging patch embeddings from an ImageNet-trained ResNet50 model into a slide embedding.This baseline used mean pooling rather than a learned slide encoder.Source: RESNET50-IN · p. 23

Attention-based multiple instance learning

  • ABMIL assigned single-headed, non-gated attention scores to patches, weighted and summed their embeddings, and used the resulting slide representation for classification.Training sampled 2048 patch features per WSI, while testing used all patches; balanced cross-entropy supported classification and adapted negative log-likelihood supported survival prediction.Source: Attention-based multiple instance… · p. 23

Evaluation

Evaluation combines standardized linear probing, retrieval, and prompting protocols with task-specific metrics and statistical procedures for within-dataset, transferability, and few-shot analyses.

Linear probing

  • Linear probing uses fixed scikit-learn classifiers and CoxNet settings consistently across pooled-feature methods, with separate transferability experiments evaluated across datasets using 100 test-set bootstraps.Classification uses cost 0.5, lbfgs, 10,000 iterations, and balanced class weights; survival uses α=0.07 for overall survival and α=0.01 for progression-free survival, with noted convergence exceptions.Source: Linear probing · p. 23

Retrieval

  • Retrieval compares raw slide embeddings using L2 distance, evaluates up to ten nearest slides, and reports cancer-type and EBRAINS subtype mAP@k for k∈{1,5,10}.Cancer-type retrieval uses ten CPTAC cohorts and assesses whether retrieved labels match each query slide’s class.Source: Retrieval · p. 24

Prompting

  • Prompting constructs class prompts from averaged RNA expression profiles and predicts by nearest L2 distance, while survival prompts use the top and bottom 25% of uncensored patients.A CONCHV1.5 mean-pooling baseline instead constructs prompts from averaged WSI embeddings, and survival risk is the distance to the high-risk prompt minus the low-risk prompt.Source: Prompting · p. 24

Metrics

  • Metrics are matched to task type: macro-AUC for binary classification, balanced accuracy for multi-class subtyping, quadratic weighted kappa for grading, and c-index for survival.These metrics respectively address threshold-free discrimination, class imbalance, ordinal agreement, and risk-order concordance.Source: Metrics · p. 24

Statistical analysis

  • Statistical analysis reports fold means and standard errors or 95% bootstrap confidence intervals, and uses ANOVA, Tukey HSD, log-rank tests, and mixed-effects comparisons across datasets and task families.Few-shot results are shown with box plots spanning quartiles and whiskers to points within 1.5× the interquartile range.Source: Statistical analysis · p. 25

Computing hardware and software

The study used a Python 3.10.12 and PyTorch 2.3.0 software stack, with distributed A100-GPU pretraining and single-GPU downstream computations supported by open-source pathology and machine-learning libraries.

Hardware and training

  • THREADS was trained with a modified CLIP implementation on 4× 80GB NVIDIA A100 GPUs using distributed data-parallel training, while downstream experiments used single 24GB NVIDIA 3090 GPUs.WSI processing used OpenSlide, openslide-python, and Trident with Pillow, segmentation-models-pytorch, and opencv-python.Source: Computing hardware and software · p. 25

Software stack

  • Experiments used Python 3.10.12, PyTorch 2.3.0 with CUDA 12.3, scGPT, Scikit-learn, Scikit-survival, and faiss for implementation and downstream modeling.Downstream models included logistic regression and Cox proportional hazards regression.Source: Computing hardware and software · p. 25

Benchmarking and visualization

  • Benchmark slide encoders included GigaPath, PRISM, and CHIEF, while Matplotlib and Seaborn were used to create plots and figures.GigaPath additionally required fairscale, flash-attn, and ninja.Source: Computing hardware and software · p. 25

Code availability

Preprocessing and benchmarking code, curated public-cohort labels, and study data splits are publicly accessible, while THREADS weights and slide-embedding extraction code will be released upon publication.

Code availability

  • Preprocessing and benchmark code, curated public-cohort labels, and study data splits are available through the listed repositories and dataset.The preprocessing code covers tissue segmentation, whole-slide image patching, and patch embedding extraction for CONCHV1.5, CTRANSPATH, GIGAPATH, and VIRCHOW.Source: Code availability · p. 26
  • THREADS model weights and slide-embedding extraction code will be released upon publication.Source: Code availability · p. 26

Data availability

Public TCGA and GTEx imaging and transcriptomics data are accessible through designated portals, whereas proprietary pretraining and in-house benchmark cohorts are unavailable. Benchmark download links and curated labels are provided through Extended Data Table 8 and the THREADS-Benchmarking GitHub repository.

Data availability

  • TCGA and GTEx imaging and transcriptomics data are publicly accessible through the NIH Genomic Data Commons, Xena Hub, and GTEx portal, while BWH and MGH pretraining data are proprietary.The proprietary pretraining data cannot be made publicly available.Source: Data availability · p. 26
  • Benchmark cohort download links are listed in Extended Data Table 8, and curated labels are available through the THREADS-Benchmarking GitHub repository.In-house benchmark cohorts cannot be made publicly available.Source: Data availability · p. 26

Related papers

Loading 2501.16652v1…