Cross-Modal Contrastive Learning for the Retrieval of Immunotherapy-Associated Molecular Signatures from Histopathology

• The “Inflamed/Non-Inflamed” phenotype predicts ICI response, but identifying it requires a costly 10-gene RNA signature that limits routine clinical use.Source: §1 · p. 1
• Paired histology-genomic data are scarce, and converting continuous gene expression into binary labels can discard patients’ positions on the biological spectrum.Source: §1 · p. 2
• CCMIL learns a shared histology-transcriptomics embedding for cross-modal retrieval.Source: §1 · p. 2
• The learned representation captures biologically coherent patient relationships, supporting RNA signature imputation and downstream classification from H&E alone.Source: §1 · p. 2

• Contrastive learning pulls positive pairs together and pushes negative pairs apart; ConVIRT and CLIP applied it to vision-language retrieval, with medical adaptations including MedCLIP.Source: §2 · p. 2
• Unimodal MIL models such as ABMIL, CLAM, and TransMIL may not capture the full transcriptomic complexity of heterogeneous tumors.Source: §2 · p. 2
• Although multimodal methods such as MCAT fuse histology and genomics, they require both modalities at inference; CCMIL instead uses an aligned manifold to retrieve molecular signatures from histology alone.Source: §2 · p. 2

3.1 Cross-modal contrastive architecture

• Parallel histological and genomic encoders map both modalities into a shared joint embedding space.Source: §3.1 · p. 2, Figure 1 · p. 3
• The histology stream uses CONCH features, spatial encodings, and THCA attention conditioned on learned histology-type embeddings to form slide-level representations.Source: §3.1 · p. 3
• The genomic stream encodes the 10-gene signature by combining gene-specific identity vectors with expression-weighted aggregation, then refines the result with an SNN.Source: §3.1 · p. 3
• Table 1 compares framework and baseline performance using mean ± standard deviation across 5-fold cross-validation.Source: Table 1 · p. 4

3.2 Losses

• The supervised contrastive loss groups same-label WSI and RNA embeddings, including both same-patient cross-modal pairs and other embeddings sharing the phenotype label.Source: §3.2 · p. 4
• An unweighted average of BCE losses from the WSI and genomic probes encourages both modalities to retain diagnostic utility and follow the task decision boundary.Source: §3.2 · p. 5

3.3 Inference strategy

• At inference, the genomic stream is detached; H&E embeddings can feed a classification probe or retrieve RNA embeddings for signature imputation.Source: §3.3 · p. 5

4.1 Dataset and preprocessing

• The curated TCGA cohort contains 265 diagnostic slides, with inflammation labels from a 10-gene RNA signature and histology subtypes assigned by expert pathologists.Slides were segmented and patched at 40×, then embedded using CONCH.Source: §4.1 · pp. 5–6
• Figure 2 presents a UMAP of the joint representation and RNA cosine similarity across retrieval ranks, with CSLS + Reciprocal closest to the RNA-space Oracle.The caption identifies retrieval rank k as the horizontal comparison dimension for mean RNA cosine similarity.Source: Figure 2 · p. 6

4.2 Training setup

• Training used five-fold cross-validation with early stopping, Adam, and a fixed learning rate of 5×10−5.Each fold used train, validation, and test splits of 80%, 10%, and 10%.Source: §4.2 · p. 6
• Figure 3 contrasts attention maps with baselines and depicts transcriptomically coherent neighbors along a continuous inflammatory gradient.The caption describes CCMIL attention hotspots as dense and highly localized.Source: Figure 3 · p. 7
• Baselines included three H&E-only MIL models, multimodal MCAT, and a spatially matched retrieval baseline using instance-level CLIP alignment.All models used identical CONCH embeddings and the same five-fold splits.Source: §4.2 · p. 7

5.1 Quantitative Evaluation

• Test →Test retrieval achieved R@5 of 0.92 and MRR of 0.80, while Test →Train weighted k-NN voting yielded Accuracy of 0.72 and F1-score of 0.76.The Test →Test measures are reported relative to the CLIP baseline comparison; Test →Train mimics querying new slides against training cases.Source: §5.1 · p. 8
• CSLS + Reciprocal reranking mitigated high-dimensional hubness and maintained the highest RNA cosine similarity to the RNA-space Oracle upper bound.The strategy was selected using validation-set evaluation of multiple retrieval metrics.Source: §5.1 · pp. 7–8
• CCMIL achieved the highest F1-score (0.76) and Recall (0.92), while ablations showed poorer Accuracy without representation choices and lower retrieval scores without contrastive learning or THCA.Representation learning alone yielded Accuracy 0.54; omitting contrastive learning yielded R@1 0.63; replacing THCA with standard attention yielded R@1 0.60 and F1-score 0.74.Source: §5.1 · p. 8

5.2 Qualitative Evaluation

• CCMIL attention produced dense, localized hotspots aligned with tumor segmentations, unlike baseline maps described as under-activated, saturated, or diffuse.The authors interpret biologically coherent activations as capturing subtle patterns associated with an Inflamed signature that is visually indistinguishable to pathologists.Source: §5.2 · p. 8
• Slides projected along an inflammation axis formed a continuous phenotype gradient, with transcriptomically coherent neighbors retrieved through rank 10.The qualitative analysis selected seven evenly spaced query slides and displayed their retrieved RNA profiles at ranks 1, 2, 3, and 10.Source: §5.2 · p. 8

6. Discussion and Conclusions

• CCMIL integrates a composite loss and THCA module to learn a shared histology-transcriptomics latent space, improving cross-modal alignment relative to evaluated baselines.The framework is presented as an interpretable retrieval method for molecular signatures from standard H&E slides.Source: §6 · pp. 8–9
• The search-by-case engine retrieves transcriptomically coherent neighbors to impute a patient's continuous inflammatory phenotype beyond binary classification.The conclusion frames the representation as supporting downstream classification and immunotherapy pre-screening.Source: §6 · p. 9
• Evaluation was limited to a retrospective TCGA cohort; independent external-cohort validation is future work for assessing generalizability.The conclusion describes the study as a proof of concept.Source: §6 · p. 9