Source-linked AI summary
Semantic-Guided Multimodal Preprocessing for Vision Transformer-Based Clear Cell Renal Cell Carcinoma Grading
Fatemeh Javadian, Zhu Chen, Zahra Aminparast, Johannes Stegmaier
TL;DR
CCRCC grading needs to connect contextual nuclear-level evidence with patch-level tumor grading, but existing approaches keep these analyses separate. The paper integrates pre-trained nuclei classification maps with RGB images through semantic-guided preprocessing for ViT grading. The best method achieves 0.916 balanced accuracy versus 0.707 for RGB-only processing and 0.427 for max-voting, with robustness assessed under simulated map errors.
Problem
Existing CCRCC methods either grade patches directly from RGB images or classify nuclei and aggregate them, leaving fine-grained and coarse-grained analyses disconnected.
Method
The method fuses pre-trained nuclei classification maps with RGB images through multiplicative modulation and perceptually optimized overlays before ViT input.
Results
0.916 balanced accuracy was achieved, versus 0.707 for the RGB-only baseline and 0.427 for max-voting aggregation.
Takeaways & Limitations
The preprocessing fusion bridges nuclei-level information and tissue context while retaining an advantage over RGB-only processing under simulated perturbations up to 60%.
Takeaways & Limitations
Sensitivity analysis perturbs ground-truth maps with random errors rather than predictions from a real upstream nuclei model, so deployment robustness remains unverified.
Abstract
from arXiv · showhide
Clear cell renal cell carcinoma (CCRCC) grading is essential for treatment planning, yet existing approaches either analyze patch-level images directly or focus solely on nuclei-level classification, without linking to final tumor grading. We propose a semantic-guided multimodal preprocessing method that integrates nuclei classification maps from existing pre-trained models with RGB histopathology images for Vision Transformer (ViT)-based CCRCC grading. Our approach employs classification map channel concatenation and multiplicative modulation, with optimized overlays to leverage nuclei grading information, while preserving RGB textural features. Evaluation of multiple preprocessing strategies demonstrates that semantic-guided enhancement achieves 0.916 balanced accuracy, outperforming RGB-only baseline (0.707) and max-voting aggregation from prior studies (0.427). Sensitivity analysis reveals that this 21 percentage point improvement over baseline persists even under simulated perturbation at rates matching current state-of-the-art nuclei classification model error thresholds, suggesting both effective semantic utilization and practical robustness. These findings show that preprocessing-based multimodal fusion can leverage the diagnostic potential of existing imperfect nuclei classifiers, effectively bridging previously isolated fine-grained nuclear-level analysis with coarse-grained ViT-based patch classification. Per-class recall was consistent across grades (0.93, 0.91, 0.91), indicating that gains are not concentrated in the majority class. Because the sensitivity analysis perturbs ground-truth maps rather than predictions from an actual nuclei model, this result characterizes robustness under simulated error rather than deployment with a real upstream model, which remains for future work.
1 Introduction
CCRCC grading requires contextual assessment of nuclear features, but existing fine-grained nuclei classifiers and coarse-grained patch classifiers remain disconnected. Max-voting can miss clinically significant high-grade nuclei, motivating semantic-guided fusion of nuclei maps with RGB images for ViT grading.
- Clinical grading context: CCRCC grading reflects nucleolar prominence, spatial distribution, and nuclear morphology rather than formal quantitative thresholds.Grades 1–3 form a nucleus-centered continuum, while grade 4 is analyzed separately because its nuclei lack consistent patterns.
- Limits of nuclei aggregation: Max-voting can under-grade patches when sparse high-grade nuclei determine clinical significance despite lower-grade nuclei being more abundant.Pathologists may assign grade 3 based on the presence and spatial distribution of minority high-grade nuclei.
- Limits of nuclei aggregation: Scattered high-grade nuclei alone are insufficient for grade assignment because clustering, density, and morphological context also matter.These relationships are subjective and non-linear, which simple voting cannot capture.
- Disconnected analysis levels: Fine-grained nuclei classification captures heterogeneity, whereas coarse-grained ViT methods grade patches directly from RGB images.The two approaches provide complementary information but remain isolated in existing pipelines.
- Proposed bridge: Preprocessing-based fusion of nuclei maps with RGB images was identified as an unexplored route for linking nuclear-level analysis to patch grading.The proposed method integrates pre-trained nuclei classification maps with RGB inputs for ViT-based CCRCC grading.
2 Methodology
The methodology combines nuclei classification maps with RGB histopathology patches before ViT-based CCRCC grading, using semantic-guided preprocessing rather than architectural changes. It evaluates channel concatenation, multiplicative modulation, parameterized overlays, and robustness to simulated nuclei-map errors.
- Dataset: The patch-level split followed the source benchmark because slide-level grouping could not be verified, leaving possible cross-partition correlations from shared slides and staining characteristics.Targeted flips produced a near-balanced training distribution.
- Vision Transformer baseline: The selected RGB-only baseline was Google ViT Base Patch32-384, fine-tuned for three-class CCRCC classification after resizing patches to 384 × 384 pixels.The model processes 32 × 32 pixel-token sequences and captures local and global tissue context.
- Multimodal preprocessing: Semantic preprocessing combines pre-trained nuclei classification maps with RGB patches before ViT input through Classification Map Channel Concatenation and Multiplicative Modulation.Both inputs originate from the same H&E tissue image, and the ViT architecture remains unchanged.
- Fusion methods: HEC replaces the RGB blue channel with a nuclei map after H&E color deconvolution, while MM preserves RGB and combines intensity modulation, grade weighting, smoothing, and color overlay.The map encodes background, tumor grades 1–3, and non-tumorous cells; HEC does not explicitly emphasize clinically important grades.
- Multiplicative Modulation: MM computes I′(x, y) = I(x, y) · (1 + α · f(C(x, y))), then applies grade-dependent weighting, Gaussian smoothing, and optional overlay to emphasize nuclei semantics while retaining RGB gradients.The sigmoid weighting and optimized ranges prioritize higher tumor grades; tested overlays in Fig. 2 span 0 to 0.5.
- Sensitivity analysis: Robustness was tested by simultaneously injecting random segmentation and adjacent-grade classification errors into ground-truth maps from 0% to 60% during evaluation.The perturbation interval was calibrated against measured error rates of pre-trained nuclei models, but the models were trained only on ground-truth maps.
- Evaluation: Balanced accuracy was used as average per-class recall, giving equal importance to all grades despite class-frequency imbalance.This metric addresses the original dataset imbalance when comparing preprocessing methods.
3 Results and Discussion
Semantic-guided preprocessing improved CCRCC grading over RGB-only and max-voting baselines, with optimized multiplicative modulation providing strong performance and robustness to perturbed nuclei maps.
- All optimized fusion methods outperformed the RGB-only baseline, while multiplicative modulation was superior because it preserved gradient information and directly modulated RGB intensities.
- Color overlay strength of 0.3–0.5 was the most influential parameter, and removing it significantly reduced balanced accuracy.
- 0.916 balanced accuracy and 0.922 F1 were achieved, compared with RGB-only values of 0.707 and 0.761 and max-voting balanced accuracy of 0.427.
- Per-class recall was reported as consistent across grades, indicating that performance was not concentrated in the majority class.
- Performance remained above RGB-only baselines through 60% simultaneous segmentation and classification perturbation, despite degrading as error increased.
4 Conclusion
The study concludes that preprocessing-based integration of nuclei maps with RGB images improves ViT-based CCRCC grading without architectural modification and remains robust under simulated map errors. However, evaluation with predicted maps from an actual nuclei model remains future work.
- Semantic-guided preprocessing combined pre-trained nuclei maps with RGB images through multiplicative modulation and optimized overlays, reaching 0.916 balanced accuracy.
- The method improved over RGB-only processing at 0.707 and max-voting aggregation at 0.427 while allowing ViTs to use nuclei and tissue information.
- Sensitivity analysis kept performance above the RGB-only baseline through 60% combined segmentation and classification perturbation.
- The full pipeline with predicted nuclei maps at both training and evaluation time remains to be evaluated.
5 Compliance with Ethical Standards
The study used publicly available TCGA data with nuclei annotations from Gao et al.; no additional ethics approval was required.
- The study used publicly available TCGA Research Network data with nuclei annotations from Gao et al., requiring no additional ethics approval.