Source-linked AI summary
Cross-Modal Contrastive Learning from Histopathology and CT for Automated Renal Cell Carcinoma Grading
Amit Das, Tanmay Shukla, Naofumi Tomita, Faraz Farhadi, Jessica Sin, Ari Hakimi, Chad Vanderbilt, Jie-Fu Chen, Ritesh Kotecha, Weijie Ma, Bing Ren, Saeed Hassanpour
TL;DR
ccRCC grade assessment relies on tissue sampling that can be limited by biopsy risk and sampling variability, while conventional CT alone does not reliably provide grade. RCC-Align uses paired histopathology and CT during training to improve CT-based grade prediction; the reported conclusion is that this approach may complement tissue diagnosis in selected settings.
Problem
Biopsy can carry procedural risk and a single core may misclassify grade because it misses ccRCC intratumoral heterogeneity, while CT alone does not reliably provide grade information.
Source: INTRODUCTION · p. 4Method
RCC-Align aligns paired histopathology and CT representations during training so grade-discriminative tissue information can support CT-based prediction.
Source: INTRODUCTION · p. 5Results
RCC-Align improved CT-based ccRCC grade prediction compared with CT-only baselines through alignment of radiologic representations with pathology-derived grade-discriminative features.
Source: DISCUSSION · pp. 16, 18Takeaways & Limitations
The authors conclude that pathology-guided learning may complement tissue diagnosis when biopsy is unsafe, infeasible, or limited by intratumoral heterogeneity.
Source: INTRODUCTION · p. 4The paired cohort was relatively small, and validation in larger, multi-institutional cohorts with external test sets and prospective clinical evaluation is needed.
Source: DISCUSSION · pp. 17–18
Abstract
from arXiv · showhide
Background: Clear cell renal cell carcinoma (ccRCC) exhibits substantial clinical heterogeneity, and accurate grade assessment is essential for risk stratification and treatment planning. However, conventional grading requires invasive tissue sampling. We developed RCC-Align, a cross-modal contrastive learning framework that leverages paired histopathology and computed tomography (CT) data during training to improve noninvasive CT-based ccRCC grade prediction. Methods: RCC-Align aligns paired whole-slide histopathology images (WSIs) and CT scans through contrastive cross-modal objectives, transferring grade-discriminative information from microscopic tissue morphology to macroscopic radiologic representations. The framework was trained and evaluated on paired TCGA and CPTAC cohorts using patient-level five-fold cross-validation. Performance for low- versus high-grade ccRCC classification was compared against CT-only baselines (DINOv2-Base and DINOv2-Finetuned) and a WSI-based reference model (GigaPath-Finetuned). Cross-modal alignment was assessed using cosine similarity analysis. Results: RCC-Align achieved an AUC of 0.601 (95% CI, 0.524-0.673) and AUPRC of 0.599 (95% CI, 0.541-0.676), outperforming DINOv2-Finetuned (AUC 0.545; AUPRC 0.543) with significantly improved low-grade prediction (p = 0.004). RCC-Align also demonstrated stronger paired WSI-CT embedding alignment compared with baselines. The WSI-based GigaPath reference achieved an AUC of 0.719. Conclusion: Pathology-guided contrastive learning improves CT-based ccRCC grading while requiring only CT at inference. This approach may complement tissue diagnosis when biopsy is unsafe, infeasible, or limited by intratumoral heterogeneity. Validation in larger, multi-institutional cohorts with external testing is needed before clinical translation.
TAKE HOME MESSAGE
Tissue-based grading can be limited by biopsy risk and sampling variability, while CT alone does not reliably provide tumor grade. RCC-Align uses paired histopathology and CT during training to improve CT-based ccRCC grade prediction, with the intended benefit of noninvasive risk assessment.
INTRODUCTION
- Biopsy-based grading carries procedural risk and may misclassify grade when a single core misses intratumoral heterogeneity.CT is routinely obtained and noninvasive, but conventional CT alone does not reliably provide tumor grade for clinical decisions.Source: INTRODUCTION · p. 4
- More accurate imaging-based grade assessment could support preoperative risk stratification and treatment planning where tissue sampling is limited.The introduction notes that grade assessment informs surgical decisions, surveillance, and systemic or adjuvant treatment selection.Source: INTRODUCTION · p. 4
- RCC-Align uses paired whole-slide histopathology and CT during training to improve CT-based low- versus high-grade ccRCC prediction.The framework aligns tissue and radiologic representations so grade-relevant tissue information can support radiologic prediction.Source: INTRODUCTION · p. 5
TAKE HOME MESSAGE
- The model uses CT alone at inference, retaining a noninvasive setting for grade estimation when biopsy is unsafe, infeasible, or affected by sampling variability.The authors frame this as a potential complement to tissue diagnosis.Source: INTRODUCTION · pp. 4–5
METHODS
RCC-Align predicts ccRCC grade from CT while using paired histopathology during training. The study groups ISUP grades into low- and high-grade classes to target a clinically relevant binary distinction.
Overview
- RCC-Align predicts ccRCC grade from CT features while leveraging complementary information from paired histopathology WSIs during training.Histopathology features are aggregated into slide-level representations, and CT features are derived using a pretrained DINOv2 encoder.Source: Overview · p. 5
- Training jointly uses a pathologist-grade supervised classification loss and a cross-modal contrastive loss that aligns histopathology and CT embeddings.This setup supports CT-based grade prediction at inference while benefiting from multimodal supervision during training.Source: Overview · pp. 5–6
Outcome definition
- Low-grade ccRCC is defined as ISUP grades 1–2 and high-grade as grades 3–4, matching a clinically used distinction between indolent and aggressive tumors.The binary grouping is described as reducing label noise and class imbalance while improving annotation reliability relative to adjacent individual grades.Source: Outcome definition · p. 6
Datasets for Fine-tuning GigaPath Encoder
The study draws on paired TCGA and CPTAC WSI–CT cases and a larger WSI-only cohort for histopathology fine-tuning. Patient-level cross-validation and standardized preprocessing address cohort structure and variation in CT contrast phase.
Datasets for Fine-tuning GigaPath Encoder
- CPTAC and TCGA supplied paired H&E WSIs and CT scans, while MSK contributed a large WSI-only ccRCC cohort for histopathology fine-tuning.The H&E slides came from surgical resections rather than needle core biopsies, providing broader sampling of tumor morphology.Source: Datasets for Fine-tuning GigaPath… · p. 6
Paired WSI and CT Dataset
- The paired CPTAC and TCGA cohort comprised 312 WSIs from 211 patients, with high- and low-grade case counts reported separately for each cohort.TCGA included 101 high-grade and 62 low-grade cases; CPTAC included 59 high-grade and 90 low-grade cases.Source: Paired WSI and CT Dataset · p. 7
- Model development used five-fold cross-validation stratified by patient and balanced by cohort and grade to prevent data leakage.Some patients contributed multiple WSIs, while CT labels were assigned at the patient level.Source: Paired WSI and CT Dataset · p. 7
- All scans were contrast-enhanced, but contrast phase varied and most scans were delayed phase or lacked a specific phase label.The same preprocessing workflow was applied to reduce variability from phase heterogeneity.Source: Paired WSI and CT Dataset · pp. 7–8
WSI-Only Fine-Tuning Cohort
The histopathology encoder is adapted using a large grade-labeled WSI cohort, then slide patches are embedded for model use. CT preprocessing segments kidneys and retains standardized kidney-containing slices for downstream analysis.
WSI-Only Fine-Tuning Cohort
- The WSI-only fine-tuning cohort included 10,589 slides from MSK, TCGA, CPTAC, and Dartmouth-Hitchcock, with 6,328 high-grade and 4,261 low-grade slides.Fine-tuning continued from a metastasis-pretrained checkpoint using RCC grade labels to adapt histopathology representations to grade-specific morphology.Source: WSI-Only Fine-Tuning Cohort · p. 8
Data Preprocessing
- H&E slide patches were extracted and converted into tile-level embeddings, while kidney masks were generated from CT volumes for downstream processing.The preprocessing pipeline prepared both modalities for the framework.Source: Data Preprocessing · p. 8
Patch Extraction & Embedding Generation
- Non-overlapping 224 × 224-pixel WSI patches from regions with at least 20% stained tissue were embedded with the Prov-GigaPath tile encoder.Binary tissue masks were made from downsampled WSIs and upscaled to 20× magnification; embeddings and patch coordinates were stored.Source: Patch Extraction & Embedding Generation · p. 8
Generating Segmentations for CT Images
- TotalSegmentator kidney masks were merged, and axial slices containing kidney tissue were windowed, resized to 512 × 512 pixels, and saved.This produced kidney-centered CT slices for patient-level feature aggregation while reducing background variability; the workflow is shown in Figure 1.Source: Generating Segmentations for CT Images · p. 9
RCC-Align Framework
RCC-Align trains on paired histopathology and CT representations, then supports single-modality prediction at inference. Its objectives align the modalities while retaining information relevant to grade prediction.
RCC-Align Framework
- RCC-Align uses paired WSI and CT data during training, producing slide-level pathology and patient-level CT representations for grade prediction.WSI patch embeddings are aggregated by a slide encoder; CT slice features are pooled into a patient-level representation.Source: RCC-Align Framework · p. 9
- At inference, the trained CT encoder and classification head predict grade from CT alone, benefiting from multimodal supervision during training.Source: RCC-Align Framework · p. 10, Loss Functions · p. 12
Loss Functions
- The inter-modality InfoNCE objective brings same-patient WSI–CT embeddings together while separating unmatched samples within a batch.Source: Loss Functions · p. 11
- The intra-modality consistency loss uses 50% randomly masked WSI patch embeddings and minimizes mean squared error between the resulting slide embeddings.Source: Loss Functions · pp. 11–12
- Modality-specific classification losses preserve grade-discriminative information, and the overall objective sums inter-modality, intra-modality, and classification losses.Source: Loss Functions · pp. 11–12
Evaluation
Evaluation used patient-level five-fold nested cross-validation and compared CT models with a WSI-based reference. The study also assessed paired cross-modal embedding similarity.
Evaluation
- All slides from each patient stayed in the same fold in five-fold nested cross-validation to prevent data leakage.Performance metrics were aggregated across test folds, with 95% confidence intervals estimated by bootstrapping.Source: Evaluation · p. 12
- DINOv2-Finetuned, trained using only classification loss, served as a baseline for assessing the cross-modal and intra-modality objectives.Cosine similarity of paired and unpaired H&E–CT embeddings was also computed for DINOv2-Finetuned and RCC-Align.Source: Evaluation · p. 13
- GigaPath was fine-tuned on 10,589 unpaired WSIs from MSK, TCGA, CPTAC, and DHMC as a histopathology grading reference.Source: Evaluation · p. 13
RESULTS
RCC-Align outperformed the CT-only DINOv2-Finetuned baseline for low-grade tumor prediction, while performing worse than the WSI-based GigaPath-Finetuned reference.
Model Performance
- RCC-Align was trained with paired H&E and CT data, whereas DINOv2-Finetuned used CT classification loss without the paired dataset structure.Source: Model Performance · p. 13
- RCC-Align performed worse than GigaPath-Finetuned, which uses whole-slide images with richer, higher-resolution information.Source: Model Performance · pp. 13–14
- In the CT-based RCC Grade classification evaluation, RCC-Align significantly outperformed DINOv2-Finetuned for low-grade tumor prediction on Wilcoxon signed-rank testing.Table 2 compares DINOv2-Base, DINOv2-Finetuned, RCC-Align, and GigaPath-Finetuned, reporting AUC, weighted F1-score, and AUPRC with 95% confidence intervals.Source: Model Performance · pp. 13–14, Table 2 · p. 14
Cosine Similarity Analysis
RCC-Align showed stronger paired WSI–CT embedding alignment than DINOv2-Finetuned. The discussion frames pathology-guided CT grading as a potential clinical adjunct while noting cohort, imaging, and validation limits.
Cosine Similarity Analysis
- Paired WSI–CT cosine similarity exceeded shuffled-pair similarity for RCC-Align, with a more pronounced difference than for DINOv2-Finetuned.The analysis compared correctly matched and mismatched test-set pairs across folds.Source: Cosine Similarity Analysis · p. 15
- A Wilcoxon signed-rank test found significantly stronger alignment for RCC-Align than DINOv2-Finetuned using per-slide difference scores.The scores compared paired and shuffled similarities.Source: Cosine Similarity Analysis · p. 16
DISCUSSION
- The authors argue that paired histopathology and CT provide a learning signal beyond classification loss alone.They suggest the alignment strategy could extend to other tumor types and imaging settings where tissue features carry prognostic information not fully captured by imaging.Source: DISCUSSION · p. 16
- CT-based grade prediction is proposed as an adjunct when biopsy is unsafe, infeasible, or limited by sampling variability and intratumoral heterogeneity.The discussion cautions that it is not a replacement for tissue diagnosis.Source: DISCUSSION · p. 17
- For selected patients with indolent small renal masses, imaging-based grade estimates could support surveillance of likely low-grade disease and earlier intervention for higher-risk cases.The proposed choices are surveillance, biopsy, or definitive treatment, depending on estimated risk.Source: DISCUSSION · p. 17
- The study’s paired cohort was relatively small, institutional imaging protocols were heterogeneous, and the CT encoder lacked domain-specific pretraining.These are identified limitations of the study setting and model.Source: DISCUSSION · p. 17
- RCC-Align performed below the histopathology-based GigaPath reference, which the authors relate to the information asymmetry between macroscopic CT and microscopic histopathology.The discussion also identifies larger, more diverse cohorts, CT-specific pretraining, and contrast-enhanced phases as possible routes to further gains.Source: DISCUSSION · p. 17
- External testing in larger multi-institutional cohorts and prospective clinical evaluation are needed to define the approach’s clinical utility.MRI is also identified as a future direction for imaging-based grade estimation.Source: DISCUSSION · pp. 17–18
Ethics approval and consent to participate
The retrospective study used existing, publicly available TCGA and CPTAC data under expedited review, with consent documentation and HIPAA authorization waivers.
Ethics approval and consent to participate
- The study followed the Declaration of Helsinki as revised in 2013 and received Dartmouth Health IRB approval under expedited review.Approval was granted December 9, 2025, and the study was classified as no greater than minimal risk.Source: Ethics approval and consent to… · p. 18
- Consent documentation and HIPAA authorization were waived because the study used existing data, records, documents, and specimens retrospectively.Source: Ethics approval and consent to… · p. 18
- The study used publicly available, de-identified data from TCGA and CPTAC under their respective data-access requirements.Source: Ethics approval and consent to… · p. 18
Consent for publication
TCGA and CPTAC data are publicly available, while DHMC and MSK study data may be requested subject to approvals and privacy policies. The paper reports its funding, author roles, and limited use of generative AI for language editing and proofreading.
Availability of data and materials
- TCGA and CPTAC datasets are publicly available; DHMC and MSK datasets may be requested with appropriate approvals and subject to privacy and institutional sharing policies.Source: Availability of data and materials · p. 19
Funding
- Funding included National Library of Medicine and NCI support, with additional support for R.R.K. from a Department of Defense award and a Kidney Cancer Association partnership.The listed grants include R01LM013833, P30CA008748, and W81XWH-21-1-0942.Source: Funding · p. 19
Authors' contributions
- The authors assigned roles spanning conceptualization, methods, software, analysis, investigation, resources, data curation, writing, visualization, supervision, and funding acquisition.All authors reviewed and approved the final manuscript.Source: Authors' contributions · p. 19
Declaration of Generative AI and AI-assisted Technologies in the Writing Process
- ChatGPT was used for language editing and proofreading, not for study design, data collection, analysis, interpretation, or scientific-content generation.The authors reviewed and edited its output and take responsibility for the publication.Source: Declaration of Generative AI and… · pp. 19–20