PaperlayerView exact revision on arXiv

Source-linked AI summary

Molecular-driven Foundation Model for Oncologic Pathology

Anurag Vaidya, Andrew Zhang, Guillaume Jaume, Andrew H. Song, Tong Ding, Sophia J. Wagner, Ming Y. Lu, Paul Doucet, Harry Robertson, Cristina Almagro-Perez, Richard J. Chen, Dina ElHarouni, Georges Ayoub, Connor Bossi, Keith L. Ligon, Georg Gerber, Long Phi Le, Faisal Mahmood

arXiv:2501.16652v1cs.CVcs.AI

TL;DR

Pathology foundation models remain limited in encoding entire gigapixel whole-slide images and in using complementary multimodal data, especially for data-scarce oncology tasks. THREADS uses molecularly guided multimodal pretraining on 47,171 paired tissue sections and molecular profiles, then outperforms baselines across 54 oncology tasks. Its representations support broad downstream use and transfer learning, although the training cohort cannot capture all molecular and morphological heterogeneity.

  • Problem

    Foundation models often encode only small whole-slide image regions and lack complementary multimodal data, while oncology tasks frequently involve scarce labels.

  • Method

    THREADS combines an attention-based slide encoder with multimodal pretraining guided by transcriptomic and genomic profiles from paired H&E whole-slide images.

  • Results

    THREADS achieved state-of-the-art performance and significantly outperformed whole-slide encoders and attention-based multiple-instance-learning baselines across 54 oncology tasks.

  • Takeaways & Limitations

    THREADS provides biologically and clinically relevant whole-slide representations that can support data-scarce tasks and initialize task-specific fine-tuning.

  • Takeaways & Limitations

    Although pretrained on more than 47,000 histomolecular pairs, THREADS cannot encompass the full spectrum of molecular and morphological heterogeneity.

Abstract

from arXiv · show

Foundation models are reshaping computational pathology by enabling transfer learning, where models pre-trained on vast datasets can be adapted for downstream diagnostic, prognostic, and therapeutic response tasks. Despite these advances, foundation models are still limited in their ability to encode the entire gigapixel whole-slide images without additional training and often lack complementary multimodal data. Here, we introduce Threads, a slide-level foundation model capable of generating universal representations of whole-slide images of any size. Threads was pre-trained using a multimodal learning approach on a diverse cohort of 47,171 hematoxylin and eosin (H&E)-stained tissue sections, paired with corresponding genomic and transcriptomic profiles - the largest such paired dataset to be used for foundation model development to date. This unique training paradigm enables Threads to capture the tissue's underlying molecular composition, yielding powerful representations applicable to a wide array of downstream tasks. In extensive benchmarking across 54 oncology tasks, including clinical subtyping, grading, mutation prediction, immunohistochemistry status determination, treatment response prediction, and survival prediction, Threads outperformed all baselines while demonstrating remarkable generalizability and label efficiency. It is particularly well suited for predicting rare events, further emphasizing its clinical utility. We intend to make the model publicly available for the broader community.

Introduction

Computational pathology models face scarce labels and gigabyte-scale whole-slide images, while existing foundation models often encode only small regions. THREADS addresses these challenges through molecularly guided multimodal pretraining and broad oncology evaluation.

  • Motivation: Small oncology cohorts and gigabyte-scale whole-slide images challenge predictive modeling.Many prognostic and treatment-response studies involve fewer than 100 patients.
  • Motivation: Existing pathology foundation models are often patch encoders restricted to regions far smaller than clinical whole-slide images.Addressing whole-slide encoding may otherwise require an additional, computationally expensive model and many downstream labels.
  • Approach: THREADS uses multimodal contrastive learning guided by molecular profiles to generate whole-slide image embeddings.Its pretraining dataset, MBTG-47K, contains more than 47,000 paired whole-slide images and molecular profiles from four sources.
  • Evaluation: 54 downstream tasks cover clinical subtyping and grading, gene mutation prediction, immunohistochemistry status prediction, and treatment response and survival prediction.The evaluation is organized into four task families.
  • Results: THREADS significantly outperformed PRISM, GIGAPATH, CHIEF, and attention-based multiple-instance-learning baselines across the benchmark.Each reported comparison has P-value<0.001, and fine-tuning from THREADS initialization significantly improved over training from scratch.
  • Pretraining data: 47,171 whole-slide images span 39 organs and four institutions in the THREADS pretraining dataset.The dataset distribution and embedding space are summarized by tissue site and primary organ.

Results

THREADS combines molecularly guided multimodal pretraining with attention-based whole-slide aggregation to produce transferable slide representations. Across oncology benchmarks, it delivers strong performance, especially for data-scarce treatment, survival, retrieval, and molecular-prompting tasks.

  • Model design: THREADS uses transcriptomic and genomic profiles to guide slide representations, with an ROI encoder and attention-based slide encoder producing whole-slide embeddings.The model was pretrained using multimodal contrastive learning on next-generation sequencing data.
  • Linear probing: 6.3%, 9.9%, and 6.7% absolute performance gains over PRISM, GIGAPATH, and CHIEF, respectively, were achieved in linear probing across 54 tasks.THREADS significantly outperformed all three baselines, and led on 44/54, 49/54, and 54/54 tasks against PRISM, CHIEF, and GIGAPATH.
  • Diagnostic tasks: 98.3% AUC and 98.2% AUC were reached for breast and lung cancer subtyping, respectively, using linear probing.Performance was competitive with attention-based specialist MIL models trained on the same data.
  • Data-scarce clinical tasks: THREADS outperformed baselines in all seven treatment-response tasks and achieved the best performance in five of six survival tasks.These evaluations included cohorts ranging from 36 to 183 patients for treatment-response prediction and multiple cancer types for survival prediction.
  • Data and label efficiency: In few-shot learning, THREADS was best for most sample sizes, with k=4 brain-tumor subtyping performance exceeding PRISM at k=16.Here, k denotes the number of training samples per class.
  • Fine-tuning: THREADS fine-tuning improved performance over CHIEF and GIGAPATH by 17.9% and 7.3%, respectively, across the 54-task evaluation.Fine-tuning also yielded a 4.3% gain over an attention-based MIL model trained from scratch, with larger gains in mutation, treatment-response, survival, and IHC tasks.

Discussion

THREADS combines molecularly guided multimodal pretraining with whole-slide representations and shows strong performance across diverse oncology tasks, including data-scarce settings. The study also identifies remaining limits in molecular coverage, architecture, and scaling.

  • Discussion: THREADS provides biologically and clinically relevant H&E whole-slide representations and achieves state-of-the-art performance across subtyping, mutation, IHC, treatment-response, and survival tasks.The model was evaluated under in-domain and out-of-domain generalization, few-shot learning, and case-retrieval scenarios.
  • Discussion: Molecular-profile alignment distinguishes THREADS from models using pathology reports, tissue-site alignment, or masked autoencoding.The authors hypothesize that molecular profiles provide an objective view of cellular and tissue states and may capture subtle morphology relevant to clinical tasks.
  • Discussion: Over 47,000 histomolecular pairs do not capture the full spectrum of molecular and morphological heterogeneity.The authors propose scaling the cohort and adding assays such as immunohistochemistry and special stains.
  • Discussion: THREADS can support off-the-shelf prototyping, clinical-grade systems for rare diseases, task-specific fine-tuning, and retrieval of rare conditions.The discussion connects reduced training-data requirements with specialized tasks such as rare molecular alteration and treatment-response prediction.

Online Methods

The study received institutional review board approval for retrospective analysis of pathology slides, sequencing assays, and associated reports without direct participant recruitment.

  • Online Methods: The MGB institutional review board approved retrospective analysis of pathology slides, NGS assays, and corresponding reports.Participants were not directly involved or recruited for the study.

Pretraining dataset curation

MBTG-47K combines paired H&E whole-slide images with bulk RNA expression and genomic variation data from multiple institutions and consortia. The component cohorts differ in assay type, tissue coverage, and scale.

  • Pretraining dataset curation: MBTG-47K contains paired FFPE H&E WSIs, bulk RNA expression, and DNA variant data including SNVs, indels, and CNVs.The data were sourced from MGH, BWH, TCGA, and GTEx.
  • Pretraining dataset curation: 6,899 MGH FFPE H&E WSI–bulk RNA expression pairs from 25 tissue sites were generated using a clinically validated RNA assay and 40× scanning.Expression was summarized as log2-normalized TPM across 54 genes without additional batch-effect normalization.
  • Pretraining dataset curation: BWH contributed 20,556 FFPE H&E slides paired with SNV, CNV, and indel data from routine clinical OncoPanel testing.The assay included expert molecular-pathologist review.
  • Pretraining dataset curation: TCGA contributed 10,209 FFPE H&E WSIs from 32 cancer types paired with bulk RNA expression.A cancer-related gene set was reduced to 4,848 genes represented in the transcriptomic encoder vocabulary.
  • Pretraining dataset curation: GTEx contributed 9,507 FFPE H&E WSI–bulk RNA expression pairs from 29 tissue sites, including non-cancerous samples.GTEx WSIs were scanned at 40×, and its RNA data came from whole-transcriptome sequencing.

Downstream tasks and datasets

The benchmark spans 54 tasks across tumor morphology, grading, mutation prediction, IHC status, treatment response, and survival, using internal and public cohorts across multiple cancers.

  • Downstream tasks and datasets: The benchmark includes 54 tasks from 23 datasets across nine major organs and six task types.The task types are morphological subtyping, grading, IHC status prediction, gene-level mutation prediction, treatment response, and survival prediction.
  • Downstream tasks and datasets: MGB-Breast contains 1,264 WSIs used for morphological subtyping and ER, PR, and HER2 status prediction.The cohort includes both biopsies and resections from Brigham and Women’s Hospital.
  • Downstream tasks and datasets: MGB-Lung contains 1,939 WSIs used for lung morphological subtyping and six IHC status-prediction tasks.The slides were scanned at Brigham and Women’s Hospital, with one WSI per patient.
  • Downstream tasks and datasets: BCNB contributes 1,058 WSIs for ER, PR, and HER2 prediction, while MUT-HET-RCC contributes 1,291 WSIs for BAP1, PBRM1, and SETD2 mutation prediction.Both are public datasets used for targeted prediction tasks.
  • Downstream tasks and datasets: IMP provides 5,333 WSIs for three-class colorectal lesion grading, and PANDA provides 9,555 labeled prostate biopsy slides for six-class ISUP grading.PANDA excludes slides with equivocal labels.
  • Downstream tasks and datasets: CPTAC cohorts support mutation prediction for genes including PIK3CA, TP53, BAP1, PBRM1, KRAS, and EGFR across several cancer types.The included cohorts cover breast, renal, colorectal, glioblastoma, and head-and-neck cancers.
  • Downstream tasks and datasets: The benchmark also includes BRACS, EBRAINS, OV-Bevacizumab, NADT-Prostate, GBM-Treatment, Post-NAT-BRCA, SURGEN, MBC, and BOEHMK cohorts.These cohorts cover breast and brain subtyping, treatment response, lymphovascular invasion, mutation status, and survival-related outcomes.
  • Downstream tasks and datasets: SURGEN, MBC, and BOEHMK provide cohorts for prognostic and treatment-related analyses, including overall survival, progression-free survival, and response outcomes.MBC and BOEHMK involve patients receiving common treatment regimens, linking survival-related prediction with treatment response.

Model design and development

THREADS combines tissue patch encoding, attention-based slide aggregation, and molecular encoders to produce slide-level representations. Its pretraining and fine-tuning pipeline is designed for whole-slide and patient-level modeling.

  • Model pipeline: Each WSI is processed through tissue detection and patching, patch feature extraction, and THREADS slide encoding.
  • Tissue processing: 512×512-pixel patches are extracted at 20× magnification after excluding background regions through tissue segmentation.
  • Patch encoder: CONCHV1.5 provides patch embeddings after multimodal pretraining on pathology image-caption pairs, with resized and normalized slide patches as input.
  • Slide encoder: THREADS aggregates projected patch features with gated attention, computing attention weights before producing slide-level features through softmax-weighted patch representations.
  • Slide encoder: Multi-headed configurations concatenate independently aggregated head features and project them to a final 1024-dimensional slide embedding.
  • Molecular encoders: Molecular inputs include bulk RNA expression encoded with adapted scGPT and SNV/CNV data encoded with a multi-hot multilayer perceptron.
  • Embedding extraction: During evaluation, all patches from each slide are used for extraction, while patient embeddings combine patch embeddings across multiple WSIs.
  • Fine-tuning: THREADS fine-tuning uses weighted cross-entropy for five epochs with 2048 sampled patches per batch and no early stopping.

Baselines

THREADS is compared with whole-slide and patch-based baselines spanning multiple pretrained architectures and pooling strategies. The comparison includes both foundation-model slide encoders and conventional attention or mean-pooling approaches.

  • Foundation-model baselines: The benchmark compares THREADS with GIGAPATH, PRISM, CHIEF, and attention-based multiple instance learning.
  • GIGAPATH: GIGAPATH combines a ViT patch encoder pretrained on 171,000+ WSIs with a LongNet slide encoder trained using masked autoencoding.
  • GIGAPATH: GIGAPATH fine-tuning freezes its patch encoder, initializes a classification head, and uses gradient accumulation, weight decay, and layer-wise learning-rate decay.
  • Pooling baselines: Additional baselines average patch embeddings from GIGAPATH, Virchow, CTransPath, or ImageNet-trained ResNet50 models into slide embeddings.
  • PRISM: PRISM uses Virchow patch encoding followed by a Perceiver trained contrastively on 587,000 WSI–clinical-report pairs.
  • CHIEF: CHIEF uses CTransPath patch embeddings with an ABMIL slide encoder trained by aligning slide representations with tissue-site text embeddings.
  • ABMIL: ABMIL assigns patch-level importance scores with single-headed non-gated attention and sums weighted patch embeddings for classification.

Evaluation

Evaluation tests embedding quality across in-domain classification, cross-dataset transfer, retrieval, prompting, few-shot settings, and statistical comparisons. Metrics and uncertainty procedures are matched to task type and fold structure.

  • In-domain evaluation: In-domain classification uses fixed-recipe linear probing without validation-based hyperparameter search.
  • Transferability: Out-of-domain transfer trains linear classifiers on one dataset and evaluates them on an independent dataset, with performance estimated over 100 test-set bootstraps.
  • Retrieval: Retrieval evaluates cancer-type and EBRAINS subtype matching using L2 distance between test-slide and training-slide embeddings.
  • Prompting: Prompting averages labeled RNA profiles into class prompts and assigns test slides to the nearest prompt in embedding space.
  • Metrics: Binary classification uses macro-AUC, multi-class subtyping uses balanced accuracy, grading uses quadratic weighted kappa, and survival uses the c-index.
  • Uncertainty: Multi-fold tasks report means and standard errors, whereas single-fold tasks use 95% confidence intervals from 100 non-parametric bootstrap replicates.
  • Statistical testing: Baseline comparisons use two-way ANOVA followed by two-sided Tukey HSD tests when the ANOVA is significant.
  • Cross-task comparison: Full-benchmark comparisons use mixed-effects models accounting for dataset-level random effects and pairwise estimated-marginal-means contrasts.

Computing hardware and software

The study uses GPU-accelerated Python and PyTorch infrastructure with open-source implementations for pretraining, downstream models, and benchmarked encoders.

  • Hardware and software: THREADS pretraining uses four 80GB NVIDIA A100 GPUs and the study’s experiments and analyses use Python 3.10.12 with PyTorch 2.3.0 and CUDA 12.3.
  • Hardware and software: Downstream analyses use scikit-learn, scikit-survival, and faiss, while baseline implementations are sourced from public GIGAPATH, PRISM, and CHIEF repositories or checkpoints.

Code availability

The study provides preprocessing, benchmarking, and curated-label access through public repositories, while identifying the associated code and datasets.

  • Preprocessing code for tissue segmentation, whole-slide image patching, and patch embedding extraction is available through the trident repository.
  • Benchmark execution code is available through the patho-bench repository.
  • Curated labels and study data splits are available through the patho-bench dataset on Hugging Face.

Data availability

Public access is provided for several pretraining and benchmark resources, but proprietary and in-house cohorts cannot be publicly released.

  • TCGA imaging data, TCGA transcriptomics data, and GTEx imaging and transcriptomics data are available through their respective public portals.
  • Pretraining data from BWH and MGH are proprietary patient data and cannot be made publicly available.
  • Publicly available benchmark cohorts can be accessed through links in Extended Data Table 8, while curated labels are provided through the THREADS-Benchmarking repository.
  • In-house benchmark cohorts cannot be made publicly available.
Loading 2501.16652v1…