Source-linked AI summary
HEST-1k: A Dataset for Spatial Transcriptomics and Histology Image Analysis
Guillaume Jaume, Paul Doucet, Andrew H. Song, Ming Y. Lu, Cristina Almagro-Pérez, Sophia J. Wagner, Anurag J. Vaidya, Richard J. Chen, Drew F. K. Williamson, Ahrong Kim, Faisal Mahmood
TL;DR
Spatial transcriptomics studies are limited by cost, rapid technological change, small cohorts, and inconsistent formats, while linked tissue morphology is often underused. HEST-1k unifies paired ST profiles, H&E whole-slide images, and metadata with HEST-Library support, then demonstrates benchmarking, biomarker exploration, and multimodal learning. The resource spans 1,229 samples and enables analyses including expression–morphology pairing and pathology foundation-model evaluation, while remaining subject to noise and batch effects.
Problem
High costs, evolving technologies, and missing standards have constrained ST computation to narrow tasks and small cohorts, while H&E morphology is often overlooked.
Method
The authors curate paired ST profiles, H&E whole-slide images, and metadata into HEST-1k and provide HEST-Library for data interaction and assembly.
Results
HEST-1k contains 1,229 samples, 2.1 million expression–morphology pairs, and over 76 million nuclei, supporting benchmark, biomarker, and multimodal use cases.
Takeaways & Limitations
HEST-1k provides a broad resource for evaluating pathology foundation models and exploring relationships between tissue morphology and molecular expression.
Takeaways & Limitations
Staining, compression, acquisition variation, and significant imaging and transcriptomic batch effects can reduce HEST-1k quality.
Abstract
from arXiv · showhide
Spatial transcriptomics enables interrogating the molecular composition of tissue with ever-increasing resolution and sensitivity. However, costs, rapidly evolving technology, and lack of standards have constrained computational methods in ST to narrow tasks and small cohorts. In addition, the underlying tissue morphology, as reflected by H&E-stained whole slide images (WSIs), encodes rich information often overlooked in ST studies. Here, we introduce HEST-1k, a collection of 1,229 spatial transcriptomic profiles, each linked to a WSI and extensive metadata. HEST-1k was assembled from 153 public and internal cohorts encompassing 26 organs, two species (Homo Sapiens and Mus Musculus), and 367 cancer samples from 25 cancer types. HEST-1k processing enabled the identification of 2.1 million expression--morphology pairs and over 76 million nuclei. To support its development, we additionally introduce the HEST-Library, a Python package designed to perform a range of actions with HEST samples. We test HEST-1k and Library on three use cases: (1) benchmarking foundation models for pathology (HEST-Benchmark), (2) biomarker exploration, and (3) multimodal representation learning. HEST-1k, HEST-Library, and HEST-Benchmark can be freely accessed at https://github.com/mahmoodlab/hest.
1 Introduction
Spatial transcriptomics has advanced in resolution and sensitivity, but high costs, rapid technological change, and limited standardization have restricted computational studies. HEST-1k addresses these constraints by linking large-scale ST profiles with H&E morphology and metadata for benchmarking, biomarker exploration, and multimodal learning.
- High costs and rapidly evolving technologies have constrained ST computational methods to narrow tasks and small patient cohorts.
- Standardized resources and unified ST formats remain limited, hindering large-scale deep learning development.
- HEST-1k contains 1,229 paired ST profiles, H&E-stained WSIs, and metadata assembled from 153 cohorts.
- HEST-1k processing produced 2.1 million expression–morphology pairs and 76 million detected nuclei.
- The dataset supports foundation-model benchmarking, biomarker exploration, and multimodal representation learning through the HEST-Benchmark and HEST-Library.
2 Related work
Prior work provides libraries for ST analysis, molecular prediction from histology, pathology foundation models, and task-specific benchmarks. HEST-Benchmark extends this landscape with diverse, challenging gene-expression prediction tasks from histology.
- Scanpy and Seurat anchor libraries for processing, visualization, and analysis of spatial transcriptomics data.
- Molecular profiling from H&E has been studied at both slide level and patch level for mutations, microsatellite instability, and gene-expression changes.
- Pathology foundation models commonly learn general-purpose patch embeddings through self-supervised learning for downstream diagnosis or prognosis tasks.
- Existing histopathology benchmarks have often focused on specific cancer types or region-of-interest classification tasks.
- HEST-Benchmark assesses foundation models on diverse, challenging tasks for predicting gene expression from histology.
3 HEST-1k Dataset
HEST-1k unifies heterogeneous spatial transcriptomics cohorts with aligned H&E images, comprehensive metadata, standardized expression objects, tissue processing, and nuclear annotations. The resulting resource supports integrated molecular–morphological analysis across broad biological and technical variation.
- HEST-1k aggregates publicly available and internal cohorts providing paired ST and H&E whole-slide images.
- The dataset standardizes heterogeneous formats and resolutions while providing generic, histology, and expression-related metadata.
- Cancer samples are harmonized using OncoTree codes, while metadata records species, disease category, organ, genes, spots, resolution, and sequencing statistics.
- CellViT identified 76.4 million nuclei across samples and classified them into five nuclear categories.
- Expression data are unified in ANNDATA objects containing raw gene counts by spot and associated sample metadata.
- All samples were re-aligned between WSIs and corresponding expression data to address low-resolution, nonstandardized, or low-quality public alignments.
4 HEST-Library
HEST-Library provides a unified interface for assembling, querying, visualizing, and processing HEST-1k data. Its utilities cover format conversion, alignment, patching, downloading, and batch-effect analysis.
- HEST-Library enables assembling and querying HEST-1k, visualizing and mitigating batch effects, and running HEST-Benchmark.
- The library converts common WSI formats into pyramidal generic TIFF files compatible with OpenSlide and QuPath.
- Automatic utilities detect Visium fiducials, infer image resolution, and support Xenium affine alignment information.
- ST files from formats including CSV, MEX, TXT, and H5 are converted into ANNDATA objects containing counts and sample metadata.
- The library segments tissue and tessellates it into fixed-size patches around spatial spots at predefined resolutions.
- Users can download selected dataset subsets and visualize or mitigate batch effects with ComBat, Harmony, and mutual-nearest-neighbor methods.
5 HEST-Benchmark
HEST-Benchmark provides nine diverse human-cancer tasks for predicting gene expression from histology and evaluating pathology foundation models. Across the benchmark, larger vision encoders correlate strongly with performance, while more pretraining patches show a weaker association and task performance varies substantially.
- Benchmark design: HEST-Benchmark comprises nine gene-expression prediction tasks spanning eight human cancers and nine organs, including one metastatic dataset.The tasks use diverse and challenging human-cancer settings to assess foundation models for histology.
- Benchmark design: Each task predicts the top 50 highly variable genes from 112×112 µm H&E regions using patient-stratified cross-validation to prevent patient-level leakage.ccRCC uses k/2-fold cross-validation because it contains many patients.
- Evaluation: Eleven pathology foundation models are evaluated with regression from model-specific patch embeddings to log1p-normalized gene expression, using Pearson correlation as the metric.Main results use PCA-reduced embeddings with Ridge regression or XGBoost; PCA reduces dimensionality differences across models.
- Scaling laws: Model size correlates strongly with average performance (R=0.81, P-value<0.01), whereas pretraining patch count shows a weaker, non-significant correlation (R=0.48, P-value=0.13).Performance increases logarithmically with model size, while the patch-count analysis may overlook WSI counts and morphological diversity.
- Results: Performance remains low for some tasks, including READ and ccRCC, potentially reflecting weaker morphology–expression relationships or noisier cohorts.The paper cites batch effects, low sensitivity, dropout, and spillover as possible sources of cohort noise.
6 HEST for biomarker exploration
HEST-1k links spatial transcriptomics with H&E morphology to examine molecular and morphological variation in invasive ductal carcinoma. In Xenium IDC samples, nuclear size-related features show the strongest associations with gene expression, illustrating the dataset’s biomarker-exploration use case.
- Analysis setup: HEST-1k analyzes invasive ductal carcinoma Xenium samples by pairing local gene expression with H&E morphology and CellViT-derived neoplastic-nucleus measurements.The analysis examines both morphological correlates of expression and tumor heterogeneity across molecular and morphological views.
- Morphology–expression associations: GATA3 expression and neoplastic nuclear area show a moderate correlation of R=0.47 (P-value<10^-4).The correlation is measured specifically among neoplastic cells.
- Morphology–expression associations: Across 12 human-interpretable features, size-related morphology has the highest association with gene expression, while topology, shape, and cell-distribution features have lower correlations (R<0.2).The pattern is also observed in another IDC sample.
- Implication: These analyses demonstrate that HEST-1k can identify fine-grained morphological correlates of expression and support larger-scale characterization of tumor heterogeneity.The supported use case connects paired morphology and expression measurements rather than establishing causal relationships.
7 HEST for multimodal representation learning
HEST-1k supports multimodal representation learning by pairing spatial expression profiles with histology patches and fine-tuning CONCH on breast cancer data. CONCH-FT was evaluated for molecular-status prediction from whole-slide images and generally outperformed the unfine-tuned CONCH model.
- Data and training: CONCH-FT was fine-tuned with multimodal contrastive alignment on five Xenium invasive breast cancer cases and tested on an independent cohort.The training cases included four ductal and one lobular breast cancer case.
- Data and training: 47,051 paired 224×224-pixel patches and expression profiles were extracted from Xenium samples using 112×112-µm regions at 20× magnification.The expression profiles contained 238 common genes and were log1p normalized.
- Evaluation: WSI-level ER, PR, and HER2 status was predicted by mean-pooling patch embeddings and applying logistic regression on the BCNB dataset of 1,058 WSIs.The evaluation used binary molecular-status prediction.
- Results: CONCH-FT outperformed CONCH on most evaluated metrics, while its embedding space also showed a larger rank.The authors interpret the result as evidence that tissue-specific fine-tuning can improve patch-embedding expressivity, while noting the training set contained only five paired WSIs.
8 Discussion
The discussion presents HEST-1k, HEST-Library, and HEST-Benchmark as resources for biomarker exploration, multimodal learning, and histology-based gene-expression prediction. It also identifies data quality, batch effects, and incomplete format coverage as limitations while describing plans to expand the dataset.
- Discussion: HEST-1k contains paired spatial transcriptomics, H&E-stained whole-slide images, and metadata, while HEST-Benchmark provides nine tasks across eight cancer types and nine organs.The benchmark also revealed data and model scaling laws across 11 pathology foundation models.
- Discussion: 2.1 million spots and over 76 million cells were included in HEST-1k, supporting biomarker exploration and multimodal representation learning.These figures summarize the processed dataset scale reported by the authors.
- Limitations: Staining and compression artifacts, varying acquisition protocols, and significant imaging and transcriptomic batch effects can negatively affect HEST-1k quality.The study does not quantify or mitigate batch effects, although HEST-Library provides helpers for exploring them.
- Limitations and future work: HEST-Library is intended as a versatile blueprint rather than a complete solution covering every existing spatial-transcriptomics format.The authors plan to keep updating HEST-1k as new datasets become available.
Checklist
The paper connects computational pathology with spatial transcriptomics and describes the technologies, data resources, and library infrastructure underlying HEST-1k. Its checklist materials document scope, intended use, data handling, and access conditions.
- Access and use: HEST-1k data are distributed through Hugging Face, with full-dataset or metadata-filtered downloads under the CC BY-NC-SA 4.0 license.The provided checklist also states that samples include licenses permitting sharing and redistribution.
- Scope: Computational pathology uses H&E whole-slide images for clinical outcomes, whereas spatial transcriptomics maps gene activity to tissue locations for biomarker research.The study frames HEST-1k as a connection between these two fields.
- Technologies: Spatial transcriptomics samples in HEST-1k use sequencing-based ST, Visium, or Visium HD and imaging-based Xenium paradigms.Sequencing-based methods support large transcript panels, whereas Xenium provides transcript locations but has a panel-size limitation.
- Infrastructure: HEST-Library transforms heterogeneous spatial-transcriptomics and histology data into unified formats for computational pipelines.The dataset and library are assembled from public and internal cohorts and include comprehensive metadata.
C.3 HEST-Benchmark
HEST-Benchmark evaluates histology patch encoders for gene-expression prediction across nine cancer and organ tasks. It standardizes gene selection, patient-level cross-validation, regression baselines, and comparison across foundation models, while also supporting alignment and biomarker analyses.
- Prediction models: XGBoost and Ridge regression models map patch embeddings from each encoder to the selected gene-expression profiles.The models are trained on the same 50-gene target panel for each task.
- Data processing: Automatic Visium alignment detects corner fiducials and derives spot coordinates when at least three of four fiducials are found.The pipeline uses a fine-tuned YOLOv8 model and estimates pixel resolution from the detected slide geometry.
- Benchmark design: Nine benchmark tasks cover expression prediction across eight human cancers and nine organs using H&E regions.The tasks include IDC, PRAD, PAAD, SKCM, COAD, READ, ccRCC, LUAD, and axillary lymph nodes from IDC patients.
- Representation analysis: The smooth rank measure quantifies embedding quality as the entropy of L1-normalized singular values of the patch-embedding matrix.The matrix has N patches and feature dimension d, with d < N.
F.1 Motivation for dataset creation
HEST-1k was created to address fragmented, inconsistently formatted spatial transcriptomics resources and to support large-scale computational use. Its intended applications include multimodal learning, biomarker exploration, and pathology foundation-model benchmarking.
- HEST-1k targets the lack of unified, user-friendly formatting for bringing spatial transcriptomics into deep learning.
- The dataset was designed for multimodal histology–transcriptomics representation learning, biomarker exploration, and pathology foundation-model benchmarking.
- Each instance links a whole-slide image to a unique gene-expression profile with comprehensive metadata.
- The dataset release includes all data, original-data links, and publication-specific licensing information.
- HEST-Benchmark provides gene-expression prediction tasks from histology images with patient-stratified evaluation splits.
F.3 Data collection process
HEST-1k was manually inspected and curated from published cohorts, with standardized preprocessing and tools intended to reduce missing information. The dataset preserves unique expression–image pairs while acknowledging variable image quality, noisy expression data, and limited public access to raw data.
- The authors manually inspected and curated the collected data, including all unique expression–whole-slide-image pairs from the underlying studies.
- Original publications span 2016 to 2024, with newer publications potentially added as the dataset grows.
- Missing alignment files and metadata were addressed with computational tools intended to approach near-complete metadata.
- Whole-slide images vary substantially in quality because of staining, compression artifacts, and resolution, while gene-expression data are inherently noisy.
- Images were converted to pyramidal TIFF objects, alignments were manually inspected, and expression data were standardized as Scanpy objects.
- Raw data are not publicly shared, although public samples can be re-downloaded using release metadata.
- Preprocessing source code is publicly available through the HEST-Library.
F.5 Dataset distribution
HEST-1k is publicly distributed with metadata, licensing information, benchmark materials, and source-specific dataset records. The release is maintained by the authors and may expand as additional samples become available.
- HEST-1k is distributed through HuggingFace Datasets and is publicly accessible there.
- The dataset uses the CC BY-NC-SA 4.0 license, with no access or export restrictions unless the license terms are violated.
- The authors maintain the dataset, which may evolve as additional samples become publicly available and will use versioning for updates.
- The GitHub README is designated for communicating obsolescence, while no repository currently tracks papers using HEST-1k.
- Users may contact the authors to contribute additional data meeting dataset standards, although no dedicated contribution-communication system exists.
- Human and animal ethical details are delegated to original publications, while internal cohorts underwent dedicated institutional review.
- HEST-Benchmark tasks predict the 50 most variable genes from 112×112 µm H&E image patches centered on spatial-transcriptomics spots.