PaperlayerView exact revision on arXiv

Source-linked AI summary

Linking spatial biology and clinical histology via Haiku

Yan Cui, Jacob S. Leiby, Wenhui Lei, Dokyoon Kim, Yanxiang Deng, Aaron T. Mayer, Zhenqin Wu, Alexandro E. Trevino, Zhi Huang

arXiv:2605.00925v1cs.LGcs.CVq-bio.QM

TL;DR

Jointly modeling molecular, morphological, and clinical data remains limited despite its importance for biomedical research. Haiku aligns these modalities in a shared embedding space and outperforms competing approaches across retrieval, prediction, and biomarker-inference tasks.

  • Problem

    Systematic frameworks for jointly modeling molecular, morphological, and clinical data remain limited despite the complementary information these modalities provide.

  • Method

    Haiku uses tri-modal contrastive learning to jointly align H&E histology, spatial proteomics, and clinical text in a shared embedding space.

  • Results

    Across evaluations, Haiku outperformed competing and unimodal approaches in retrieval, classification, clinical prediction, and biomarker inference, reaching Recall@50 of 0.611 for H&E-to-mIF.

  • Takeaways & Limitations

    Haiku provides a unified representation that captures shared biological and pathological semantics across molecular, morphological, and clinical modalities.

  • Takeaways & Limitations

    Counterfactual analyses are single-patient proof-of-concept studies whose exploratory findings require further validation.

Abstract

from arXiv · show

Integrating molecular, morphological, and clinical data is essential for basic and translational biomedical research, yet systematic frameworks for jointly modeling these modalities remain limited. Here we present Haiku, a tri-modal contrastive learning model trained on multiplexed immunofluorescence (mIF). It comprises 26.7 million spatial proteomics patches from 3,218 tissue sections across 1,606 patients spanning 11 organ types, with matched hematoxylin and eosin (H&E) histology and clinical metadata aligned in a shared embedding space. Haiku enables three-way cross-modal retrieval, improves downstream classification and clinical prediction tasks over unimodal baselines, and supports zero-shot biomarker inference through fusion retrieval conditioned on clinical metadata-only text descriptions. Across tasks, Haiku outperforms competing approaches, achieving cross-modal retrieval (Recall@50 up to 0.611 versus near-zero baseline), survival prediction (C-index 0.737, +7.91% relative improvement), and zero-shot biomarker inference (mean Pearson correlation 0.718 across 52 biomarkers). Furthermore, we introduce a counterfactual prediction framework in which modifying only clinical metadata while fixing tissue morphology surfaces niche-specific molecular shifts associated with breast cancer stage progression and lung cancer survival outcomes. In a lung adenocarcinoma case study, the counterfactual analysis recovers niche-specific shifts characterized by increased CD8 and granzyme B, reduced PD-L1, and decreased Ki67, broadly consistent with patterns reported for favorable outcomes. We present these counterfactual results as exploratory, hypothesis-generating signals rather than mechanistic claims. These capabilities demonstrate that tri-modal alignment via Haiku enables integrative analysis of spatial biology, bridging molecular measurements with clinical context for biological exploration.

1 Introduction

Integrating molecular, morphological, and clinical data is increasingly important but computationally challenging because multimodal relationships may reveal insights unavailable from any single data type. Haiku addresses this need by unifying spatial biology, H&E morphology, and clinical or semantic information in a shared representation for cross-modal analysis and hypothesis generation.

  • Motivation: Multimodal biomedical datasets provide complementary views of tissue biology, but learning their multi-directional relationships remains a fundamental computational challenge.These relationships may encode biological and clinical insights inaccessible through single-modality analysis.
  • Motivation: H&E morphology, spatial protein expression, and clinical or semantic context represent complementary views of the same underlying tissue biology.Prior work has separately predicted spatial proteomics from H&E, aligned histopathology with text or clinical metadata, and used semantic descriptors to connect molecular data with interpretation.
  • Haiku framework: Haiku unifies molecular, morphological, and clinical information within a shared representation to enable coherent cross-modal alignment, retrieval, and integration.The framework is designed to address the limitations of models that do not fully exploit multi-directional relationships across biomedical modalities.
  • Haiku framework: Haiku supports improved predictive and retrieval performance while enabling systematic multimodal exploration for knowledge discovery and biological hypothesis generation.Its embedding space supports robust retrieval across modalities, including Recall@50 of 0.604 for mIF-to-H.

2 Results

Haiku jointly aligns H&E histology, mIF spatial proteomics, and clinical text through tri-modal contrastive learning. The resulting shared embedding space supports cross-modal retrieval and clinically relevant prediction tasks, outperforming unimodal baselines.

  • Haiku framework: Haiku jointly models H&E images, spatial proteomics, and textual information using contrastive alignment in a shared embedding space.Registered H&E and mIF images are partitioned into 256 × 256 pixel patches, while clinical metadata are converted into structured textual descriptions.
  • Cross-modal retrieval: Haiku evaluates cross-modality retrieval on 336 held-out paired tissue slices across H&E, mIF, and text queries.Qualitative examples include text-to-mIF retrieval and H&E-to-mIF retrieval, with the corresponding mIF patch retrieved at rank 1 in the latter example.
  • Zero-shot evaluation: Haiku’s embeddings support zero-shot annotation across modalities for organ type, tumor grade, and tissue type using 1-nearest-neighbor classification.The model also performs CLIP-style zero-shot classification for organ type across 10 categories and disease across 11 categories.
  • Clinical prediction: Haiku(Fusion) achieves macro F1 values of 0.942 for N stage, 0.961 for T stage, 0.942 for tumor grade, 0.999 for organ types, and 0.998 for tissue types.These results outperform Haiku unimodal embeddings and unimodal baselines, including MUSK for H&E and VirTues for mIF.
  • Clinical prediction: 0.737 is the mean C-index achieved by Haiku(mIF) for CRC survival prediction, compared with 0.683 for the VirTues baseline.The comparison uses attention-based MIL Cox regression with five-fold cross-validation, yielding an improvement of approximately 0.054.

AUROC AUPRC 0.0

Haiku’s tri-modal embedding supports zero-shot biomarker inference and counterfactual perturbation by combining morphology, molecular references, and clinical-text edits. Counterfactual analyses reveal compartment- and niche-specific biomarker shifts, including immune infiltration, checkpoint reduction, and altered antigen presentation.

  • Zero-shot biomarker inference: Haiku estimates 52 biomarker abundances by similarity-weighted retrieval from H&E patches and metadata-only clinical text against an mIF reference atlas.Predictions are evaluated using Pearson correlation across 336 held-out tissue regions.
  • Counterfactual perturbation: Tri-modal alignment enables text edits to shift retrieval in a shared latent space while morphology remains fixed for counterfactual analysis.The diverse reference atlas helps retrieved neighbors reflect biological variation rather than nearest-available artifacts.
  • Lung adenocarcinoma counterfactuals: Across four lung adenocarcinoma niches, the counterfactual Alive state shows immune infiltration in the tumor core, checkpoint relief in another niche, lymphocyte trafficking, and barrier reduction.The niches were defined by K-means clustering of 154 H&E-embedding patches into four manually annotated spatial compartments.
  • Lung adenocarcinoma counterfactuals: In the epithelial-dominant tumor core, Alive-state shifts include CD8 +50.6%, granzyme B +38.0%, CD45RO +36.8%, and PD-L1 −61.7%, all with P < 0.001.These coordinated changes indicate increased cytotoxic and memory T-cell markers alongside reduced immune-checkpoint expression.

3 Discussion

Haiku addresses the lack of a shared representation connecting text with spatial omics and imaging by jointly aligning spatial proteomics, H&E histology, and textual metadata. Its evaluations and applications support multimodal retrieval, downstream prediction, biomarker inference, and exploratory counterfactual analysis, while paired-data training and encoder quality remain limitations.

  • Contribution: Haiku is presented as the first tri-modal foundation model jointly aligning spatial proteomics, H&E histology, and textual metadata within a shared embedding space.It addresses the absence of a model connecting human-interpretable text with structured spatial omics and imaging modalities.
  • Evaluation: Large-scale held-out evaluations demonstrate patch-level retrieval across H&E, mIF, and text, while representations support tissue, tumor-status, and tumor-stage classification beyond unimodal baselines.The retrieval framework is described as ready to use, and the classification results consistently outperform unimodal baselines.
  • Downstream applications: Tri-modal biomarker inference combines H&E embeddings with metadata-only text descriptions to improve prediction accuracy over H&E-only retrieval by transferring complementary semantic knowledge.The finding provides direct evidence that metadata contributes information useful for molecular prediction.
  • Downstream applications: Counterfactual perturbations modify clinical metadata while fixing H&E morphology, revealing niche-specific mIF biomarker shifts during breast-cancer staging and lung-cancer survival analysis.These staging-based and survival-based analyses are presented as exploratory rather than mechanistic findings.
  • Limitations: Haiku is limited by training exclusively on paired datasets and by dependence on modality-specific encoder quality, although future advances could improve performance.Mixtures of paired and unpaired data may improve scalability and use of large unimodal corpora.

4 Methods · 4.1 Dataset introduction

The dataset comprises 7,600 multiplexed immunofluorescence tissue slices from a multi-center, multi-disease cohort, with paired and mIF-only subsets divided into held-out evaluation and training sets.

  • 4.1 Dataset introduction: 7,600 mIF tissue slices comprise the multi-center, multi-disease cohort.The cohort was curated by Enable Medicine.
  • 4.1 Dataset introduction: 3,554 slices are paired with co-registered H&E and clinical metadata, while 4,046 are mIF-only.The paired and mIF-only subsets together account for all 7,600 slices.
  • 4.1 Dataset introduction: 534 held-out test slices include 336 paired slices and 198 mIF-only slices selected randomly.The held-out evaluation set contains slices from both dataset subsets.
  • 4.1 Dataset introduction: 7,066 remaining slices comprise 3,218 paired and 3,848 unpaired slices used for training.The passage identifies these remaining slices as the training set.

4.2 Data preprocessing

Haiku preprocessing aligns H&E, mIF, and text at the tissue-patch level, while normalizing mIF channels, filtering tissue-rich patches, and synthesizing biomarker-spatial-clinical descriptions. Spatial statistics and interpretable rules characterize biomarker organization for text generation.

  • Tri-modal patch alignment: H&E and mIF images are pre-registered in a common coordinate system so each tissue patch has aligned image modalities and a corresponding text description.Coordinates extracted from mIF patches are reused to extract paired H&E patches.
  • mIF preprocessing: A DAPI-based pretrained ResNet50 segmentation network produces tissue masks, and only patches with tissue coverage greater than 90% are retained.The mask defines foreground tissue as M(r) = 1 and background as M(r) = 0.
  • mIF preprocessing: Each mIF biomarker channel is independently normalized by adaptive clipping, linear rescaling to [0, 1], and quantization to 8-bit resolution.Bounds use median background intensity and the 99th percentile of foreground intensity scaled by 1.1, with histogram-based refinement and fallback provisional bounds.
  • Patch-level feature extraction: For each patch, biomarker expression and spatial organization are summarized using foreground intensity, region-normalized statistics, variability, clustering, texture, and coverage features.Texture features include GLCM homogeneity and contrast, while clustering is defined from inverse mean foreground gradient magnitude.
  • Text synthesis: Fixed rules map continuous coverage, variability, and clustering metrics to interpretable labels such as sparse, uniform, clustered, or heterogeneous for text synthesis.Examples include sparse labeling at Cov < 0.1 and high-coverage uniform labeling when CV < 0.5 and homogeneity > 0.6.

4.3 Overview of the tri-modal representation learning framework

Haiku jointly embeds H&E histology, multiplexed protein images, and text descriptions in a shared latent space through modality-specific encoders, projection heads, and contrastive pretraining. Its mIF encoder also uses protein-specific ESM-3 embeddings to accommodate irregular biomarker channels and unseen biomarkers with available embeddings.

  • Framework overview: Haiku jointly embeds H&E images, mIF images, and text descriptions in a shared latent space using dedicated encoders, modality-specific projection heads, and contrastive alignment.The three modalities correspond to the same spatial tissue location.
  • Modality-specific encoders: The framework uses MUSK for H&E, a VirTues encoder pretrained from scratch for mIF, and BiomedBERT for text.VirTues is pretrained on private mIF data containing mIF-only slices and paired training slices.
  • Biomarker encoding: Each mIF biomarker is represented by its corresponding ESM-3 protein embedding, enabling biomarkers absent from pretraining when an ESM embedding exists.DAPI and other markers without protein embeddings instead use a learnable embedding.
  • Projection heads: Each encoder output is mapped through a modality-specific projection head into the shared latent space.The projection heads share an architecture but have independent parameters across modalities.
  • Projection heads: The projection heads are two-layer multilayer perceptrons with output dimension d = 512.All modalities use the same projection-head architecture.

W2 ReLU(W1h)

Haiku trains ℓ2-normalized projected embeddings with cosine-similarity contrastive losses across all cross-modal pairs. Tri-modal alignment uses selective encoder fine-tuning, AdamW, modality-specific learning rates, and a two-stage schedule over 25 epochs.

  • Contrastive objective: All projected embeddings are ℓ2-normalized before contrastive learning.
  • Contrastive objective: Contrastive losses use cosine similarity in a shared latent space and are summed across all cross-modal pairs with symmetric counterparts.The temperature parameter is fixed to τ = 0.07.
  • Optimization and training details: AdamW optimization fine-tunes the H&E and text encoders in their final two transformer blocks while keeping the mIF encoder frozen.Learning rates are 1 × 10−5 for H&E, 2 × 10−5 for text, and 1 × 10−4 for projection heads.
  • Optimization and training details: Training warms learning rates for 5,000 optimization steps, then applies cosine annealing for the remainder of 25 epochs.

4.4 Downstream evaluation … 4.4.5 Patch-level linear probing classification

Haiku’s downstream evaluation tests frozen, tri-modal representations across large-scale cross-modality retrieval, neighborhood and zero-shot classification, and supervised patch-level linear probing. These evaluations use held-out or slice-level cross-validation protocols, explicit unimodal and random or majority-vote baselines, and macro-averaged metrics to assess alignment, label consistency, and representation quality.

  • 4.4 Downstream evaluation: Haiku representations are evaluated across patch-level retrieval, classification, biomarker inference, and slice-level clinical prediction using frozen pretrained encoders unless otherwise specified.Backbone parameters are not fine-tuned during these downstream evaluations.
  • 4.4 Downstream evaluation: MUSK and VirTues provide unimodal baselines from identically fine-tuned H&E and mIF backbones, evaluated under the same splits and downstream protocols as Haiku.The baselines use backbone outputs before Haiku projection heads to ensure a fair comparison.
  • 4.4.1 Cross-modality patch-level retrieval: Cross-modality retrieval tests whether semantically corresponding patches from different modalities align in a shared latent space through similarity-based target-modality retrieval.Queries and galleries span all patches in held-out tissue sections, with spatial correspondence defining the primary ground-truth match.
  • 4.4.2 Evaluation metrics: Retrieval is evaluated with top-K accuracy on held-out tissue sections using global query and gallery sets spanning the entire held-out dataset.The reported top-K score is macro-averaged across queries, creating a large-scale cross-sample retrieval setting.
  • 4.4.3 K-nearest-neighbor (KNN) classification evaluation: KNN classification evaluates label consistency in shared embedding neighborhoods by aggregating labels from the top-K nearest target-modality gallery patches rather than retrieving a specific paired patch.Predictions use cosine similarity, similarity-weighted voting, and macro-averaged F1 score, primarily with K = 1.
  • 4.4.4 Zero-shot cross-modality classification: Zero-shot classification assigns held-out mIF patches to semantic class descriptions by nearest normalized text prototype without training an additional classifier.Five prompt templates per task are used, with metrics summarized by mean and standard deviation and compared with a repeated uniform random-guess baseline.
  • 4.4.5 Patch-level linear probing classification: Patch-level linear probing trains multinomial logistic regression on fixed Haiku(H&E), Haiku(mIF), or concatenated Haiku(Fusion) embeddings for organ, tissue, tumor stage, and grade labels.Five-fold tissue-slice cross-validation prevents spatial leakage; MUSK, VirTues, and majority-vote baselines use matched protocols, and performance is reported with macro-averaged F1 mean and standard deviation.

4.4.6 Slice-level prediction using multiple-instance learning … 4.4.9 Zero-shot fusion retrieval–based biomarker inference

The paper uses attention-based multiple-instance learning to aggregate mIF patch embeddings for slice-level classification and survival prediction, with patient-level cross-validation to prevent leakage. It also formulates biomarker inference as metadata-conditioned zero-shot fusion retrieval, using similarity-weighted mIF aggregation and Pearson correlation across 52 validated biomarkers.

  • 4.4.6 Slice-level prediction using multiple-instance learning: Each evaluation slice is represented as a bag of mIF patch embeddings, which are aggregated into a slice-level representation for task-specific prediction heads.Variable-length bags are padded and masked, and instance embeddings are ℓ2-normalized before modeling.
  • 4.4.6 Slice-level prediction using multiple-instance learning: Attention-based MIL computes instance weights while excluding padded instances, then forms the slice representation as an attention-weighted sum.An instance encoder independently transforms each instance before attention pooling.
  • 4.4.7 MIL classification for treatment response and clinical endpoints: Binary clinical endpoints use a sigmoid classification head trained with positive-class weighted binary cross-entropy to mitigate class imbalance.Five-fold cross-validation is performed at the patient level, keeping all acquisitions and patches from each patient within one fold.
  • 4.4.8 MIL survival analysis with Cox proportional hazards: Survival modeling maps each pooled slice representation to a Cox risk score and optimizes the normalized negative partial log-likelihood over risk sets.Evaluation uses patient-level five-fold cross-validation and the concordance index, while median within-fold predicted risk defines high- and low-risk groups.
  • 4.4.9 Zero-shot fusion retrieval–based biomarker inference: Biomarker inference is formulated as zero-shot fusion retrieval, querying held-out mIF patches with both H&E and metadata-only text representations.Metadata-only descriptions remove biomarker-specific molecular content so text contributes clinical and contextual information.
  • 4.4.9 Zero-shot fusion retrieval–based biomarker inference: Score-level fusion is equivalent to retrieving mIF candidates with a single tri-modal fused query embedding controlled by fusion weight α.The same formulation is used for counterfactual retrieval, with α = 1 and α = 0 representing H&E-only and Text-only baselines.
  • 4.4.9 Zero-shot fusion retrieval–based biomarker inference: Retrieved biomarker abundances are estimated by similarity-score-weighted aggregation over top-K mIF patches and evaluated using per-region Pearson correlations of spatial expression patterns.The aggregate mean PCC is reported across a validated set of 52 biomarkers retained from channels present in at least 80% of evaluation regions.

4.5 Counterfactual retrieval analysis and microenvironment stratification

This section defines metadata-only counterfactual retrieval that changes clinical context while fixing H&E morphology, then quantifies retrieval composition and biomarker shifts within morphologically stratified microenvironments. It further characterizes heterogeneity in these shifts using PCA and relates dominant change axes to baseline biomarker states.

  • Counterfactual retrieval framework: Counterfactual retrieval compares metadata-only control and modified clinical descriptions while keeping each H&E query patch fixed.Descriptions exclude explicit biomarker profiles and encode clinical context such as staging, diagnosis, tissue type, and survival status.
  • Counterfactual retrieval framework: Clinical composition is assessed by parsing TNM categories for retrieved regions and comparing paired control–counterfactual proportions with multiple-testing correction.Analyses are performed independently for T and N stages using paired two-sided tests and Benjamini–Hochberg FDR control.
  • Microenvironment stratification: H&E embeddings are clustered with K-means into K_clust = 4 morphological groups, which are manually assigned compartment labels using representative prototype patches.For each cluster, up to m = 3 nearest-to-centroid patches are selected for visual annotation.
  • Microenvironment stratification: Within each H&E-derived cluster, similarity-weighted biomarker abundances are compared between original and counterfactual retrievals to isolate metadata-induced molecular shifts under fixed morphology.The framework uses paired query-level testing with Wilcoxon signed-rank tests and Benjamini–Hochberg FDR adjustment.
  • Microenvironment stratification: PCA summarizes dominant variation in per-patch counterfactual biomarker shifts, while biomarker-by-PC Pearson correlations relate these axes to baseline biomarker states.Incomplete shift vectors are excluded after removing channels missing across all patches.

Conflict of Interests

The authors disclose that Aaron T. Mayer, Zhenqin Wu, and Alexandro E. Trevino are employees of Enable Medicine, Inc.

  • Aaron T. Mayer, Zhenqin Wu, and Alexandro E. Trevino are employees of Enable Medicine, Inc.

Paired Test Dataset Category proportion Distrbution

The paired held-out test set contains 336 slices, with category proportions reported across tissue type, organ type, and disease axes. Primary tumor and lung are the largest reported tissue- and organ-type categories, comprising 88.9% and 38.9% of slices, respectively.

  • Dataset composition: The paired held-out test set comprises 336 slices, with proportions summarized across tissue type, organ type, and disease axes.These are three parallel dataset-level category axes computed across all 336 slices.
  • Tissue Type: 88.9% of slices are primary tumor, followed by 8.3% normal, 2.1% benign/precancerous, and 0.7% metastatic tumor.
Loading 2605.00925v1…