Source-linked AI summary
MedCORE: Criteria-Grounded Clinical Reasoning for Interpretable Medical Image Diagnosis
Asim Khan, Samee Ullah Khan, Dwarikanath Mahapatra
TL;DR
Direct image-to-label models do not explicitly interrogate the clinical criteria that clinicians use to justify diagnoses. MedCORE grounds criterion-specific evidence in image regions and combines it through structured reasoning; experiments across three imaging benchmarks report improved diagnostic performance and explanatory structure.
Problem
Deep visual models often map image features directly to labels without exposing the clinical criteria, their locations, or their reliability, limiting diagnostic transparency.
Source: Introduction · p. 2Method
MedCORE decomposes diagnosis into clinical criteria, localizes and encodes their evidence, models inter-criterion dependencies, and aggregates evidence with uncertainty-calibrated weighting.
Source: Introduction · pp. 2–3, §3.3 · p. 5Results
Experiments on ISIC 2018, BUSI, and IDRiD report stronger diagnostic performance and explanatory structure across dermoscopy, breast ultrasound, and retinal fundus imaging.
Source: Introduction · p. 2, Conclusions and Future Directions · p. 13Takeaways & Limitations
The framework provides evidence-level interpretation across dermoscopy, breast ultrasound, and retinal fundus imaging.
Source: Introduction · p. 3
Abstract
from arXiv · showhide
Clinical diagnosis is inherently a structured reasoning process, yet existing deep learning models often bypass this structure by mapping image features directly to disease labels without explicitly interrogating the morphological and textural criteria that clinicians systematically evaluate. This limits diagnostic transparency and may compromise safe clinical deployment. We present MedCORE (Medical Criteria-Oriented Reasoning and Evidence), a structured diagnostic framework that operationalizes clinical reasoning within a vision-language architecture. For each input image, MedCORE decomposes the diagnostic process into clinically defined criteria, spatially localizes each criterion to diagnostically relevant image regions, encodes evidence through multi-scale representations that capture macro-structural and micro-textural pathological characteristics, and refines criterion representations using a Graph Attention Network that explicitly models inter-criteria dependencies. Criterion representations are further aligned with clinical text descriptors, reinforced through class-wise visual prototypes, and aggregated using uncertainty-calibrated weighting that proportionally discounts low-confidence diagnostic evidence. MedCORE is validated across three clinically heterogeneous imaging modalities, including dermoscopic lesion classification on ISIC 2018, breast ultrasound lesion characterization on BUSI, and diabetic retinopathy grading on IDRiD. Quantitatively, MedCORE achieves 89.2% accuracy, 85.7% macro-F1, and 96.4% AUC on ISIC 2018; 96.1% accuracy, 95.2% macro-F1, and 98.4% AUC on BUSI; and 84.3% accuracy, 80.2% macro-F1, and 92.8% AUC on IDRiD. These results demonstrate consistent improvements over strong CNN, transformer, biomedical vision-language, concept-based, and prototype-based baselines.
Introduction
Direct image-to-label models do not expose the clinical criteria supporting a diagnosis, their locations, or their reliability. MedCORE grounds diagnosis in clinical criteria and evaluates this approach across three imaging modalities.
Introduction
- Prior vision–language approaches often use global similarity, report-level supervision, or weak region–word alignment, while concept-based methods commonly omit spatial grounding, criterion relationships, or criterion-level uncertainty.These gaps motivate MedCORE’s explicit criterion-based reasoning.Source: Introduction · p. 2
- Each criterion is localized to relevant image regions and encoded at multiple scales to capture coarse morphology and fine texture.Learned criterion queries attend to image tokens without spatial supervision.Source: Introduction · pp. 2–3
- MedCORE frames diagnosis as grounded clinical-criteria reasoning, supporting evidence-level interpretation across dermoscopy, breast ultrasound, and retinal fundus imaging.The evaluation spans ISIC 2018, BUSI, and IDRiD.Source: Introduction · pp. 2–3
- Graph attention models dependencies among criteria, while criterion-wise visual prototypes reinforce class-discriminative evidence.The method treats correlated clinical signs as related evidence rather than isolated cues.Source: Introduction · p. 3
- Uncertainty-calibrated aggregation down-weights unreliable or ambiguous criteria in the final diagnostic decision.This makes criterion reliability part of evidence aggregation.Source: Introduction · p. 3
2 Related Work
Prior biomedical vision–language and concept-based methods offer useful medical semantics and interpretable concepts, but often lack a unified treatment of criterion-specific localization, relationships, and uncertainty. MedCORE organizes evidence around clinical criteria and combines spatial grounding, relational refinement, and uncertainty weighting.
Medical vision language learning
- MedCORE uses clinical text anchors to define explicit diagnostic criteria, then spatially grounds, relationally refines, and uncertainty-weights visual evidence before classification.Figure 1 depicts the framework’s criterion-query pipeline and its diagnostic, reliability, uncertainty, and heatmap outputs.Source: §2 · p. 3, Figure 1 · p. 4
Concept-based interpretability and medical concept learning
- Concept-based medical methods demonstrate the value of clinical concepts, but most do not jointly address criterion-specific localization, inter-criterion relationships, and criterion-level uncertainty.The cited approaches include Explicd [9], MICA [2], AdaCBM [6], and Med-MICN [10].Source: §2 · p. 3
Spatial grounding, relational reasoning and uncertainty
- MedCORE learns criterion-specific attention without pixel-level criterion annotations, uses class-wise prototypes as criterion-level memory, and assigns uncertainty to each criterion.The related-work discussion contrasts these choices with generic masks or image-level saliency and image-level confidence.Source: §2 · p. 4
3 Methodology
MedCORE formulates diagnosis as evidence aggregation over dataset-specific clinical criteria rather than a direct image-to-label mapping. A frozen biomedical visual backbone supplies dense features while only task-specific modules are trained.
3.1 Problem Formulation
- MedCORE decomposes diagnosis into dataset-specific clinical criteria, evaluates each independently, models their dependencies, and aggregates evidence weighted by per-criterion confidence.The criteria correspond to clinically evaluated visual properties, such as lesion asymmetry or ultrasound shape and margin.Source: §3.1 · pp. 4–5
3.2 Visual Backbone and Feature Extraction
- A shared visual backbone produces dense image features, and the backbone remains frozen while task-specific modules are trained.This preserves visual priors acquired during pre-training.Source: §3.2 · p. 5
3.3 Criterion-Aware Spatial Localisation
MedCORE uses criterion queries to produce spatially grounded region representations without spatial supervision, then encodes criterion evidence at complementary structural and textural scales.
3.3 Criterion-Aware Spatial Localisation
- Learned criterion queries generate attention maps over image tokens and produce one criterion-specific region descriptor per criterion without spatial supervision.The query-key mechanism localizes evidence to diagnostically relevant image regions.Source: §3.3 · p. 5
3.4 Multi-Scale Criterion Encoding
- Multi-scale encoding fuses global pooled evidence with a local patch representation around the peak attention location to capture complementary criterion evidence.The local representation uses a depthwise convolutional encoder; the fused vector encodes both scales.Source: §3.4 · p. 5
3.5 Graph Attention Network for Inter-Criteria Reasoning
MedCORE models diagnostic criteria as interdependent evidence rather than independent findings. A fully connected criterion graph and Graph Attention Network contextualize each criterion using the others.
3.5 Graph Attention Network for Inter-Criteria Reasoning
- Clinical findings can reinforce one another, such as malignant asymmetry strengthening irregular-border evidence or co-occurring hard exudates and microaneurysms elevating DR severity.Source: §3.5 · p. 5
- MedCORE constructs a fully connected graph whose nodes are criterion representations and whose edges carry learnable relational weights.Source: §3.5 · p. 5
- A single Graph Attention Network layer attends over all criteria to refine each representation in the context of the others, using four concatenated attention heads.Source: §3.5 · p. 6
3.6 Vision–Language Criterion Alignment
MedCORE pairs each clinical criterion with a text descriptor and aligns its visual representation with that description. A symmetric contrastive loss draws matching visual and text representations together while separating representations of other criteria.
3.6 Vision–Language Criterion Alignment
- Each criterion is paired with a short clinical text descriptor, which a frozen BiomedCLIP text encoder maps to an embedding.Source: §3.6 · p. 6
- A symmetric InfoNCE loss aligns criterion visual representations with their corresponding clinical descriptions across a batch.Source: §3.6 · p. 6
- The loss pulls each criterion’s visual representation toward its matching description and away from descriptions of other criteria, grounding features in clinical language.Source: §3.6 · p. 6
3.7 Class-Wise Visual Prototype Reinforcement
MedCORE reinforces criterion-level class discrimination with class-wise visual prototypes. Prototype predictions use cosine similarity, and a prototype cross-entropy loss reinforces class-discriminative structure.
3.7 Class-Wise Visual Prototype Reinforcement
- MedCORE maintains class-wise visual prototypes for each criterion, updated online as exponential moving averages of criterion features from each class.Source: §3.7 · p. 6
- For each criterion, prototype predictions are computed from cosine similarities between its features and the class prototypes.Source: §3.7 · p. 6
- Prototype cross-entropy reinforces class-discriminative structure within each criterion’s feature space.Source: §3.7 · p. 6
3.8 Uncertainty-Calibrated Criterion Aggregation
MedCORE estimates uncertainty for each criterion and uses it to weight criterion contributions to the final class logits. High-uncertainty evidence is suppressed, while confident evidence on decisive features can dominate.
3.8 Uncertainty-Calibrated Criterion Aggregation
- For each criterion, a two-layer MLP predicts a scalar uncertainty estimate from its criterion representation.Source: §3.8 · p. 7
- Criterion contributions to final class logits are weighted inversely by uncertainty through a softmax-normalized scheme.Source: §3.8 · p. 7
- High uncertainty suppresses a criterion’s vote, while a criterion that fires confidently on diagnostically decisive features dominates aggregation.Source: §3.8 · p. 7
3.9 Training Objective
MedCORE is trained end-to-end with a three-term objective combining classification, language alignment, and prototype discriminability losses. The uncertainty head learns implicitly from classification gradients, without explicit uncertainty supervision.
3.9 Training Objective
- MedCORE is trained end-to-end with a three-term objective.The objective combines cross-entropy on aggregated logits with language-alignment and prototype-discriminability losses.Source: §3.9 · p. 7
- Cross-entropy on aggregated logits, language alignment, and prototype discriminability form the three loss terms.Source: §3.9 · p. 7
- The uncertainty head learns implicitly from classification gradients, assigning higher uncertainty to criteria that consistently mislead classification without explicit uncertainty supervision.Source: §3.9 · p. 7
4 Experimental Analysis
Experiments span three distinct medical-image diagnostic settings, and MedCORE achieves the best performance across all three datasets, with an average Macro-F1 of 87.0%.
4 Experimental Analysis
- The evaluation covers ISIC 2018 dermoscopy, BUSI breast ultrasound, and IDRiD retinal imaging, with criterion sets tailored to each diagnostic task.Criteria include asymmetry and border irregularity for dermoscopy, lesion shape and margins for ultrasound, and microaneurysms and haemorrhages for retinal images.Source: §4 · pp. 7–8, Table 1 · p. 8
4.1 Evaluation Metrics
- Accuracy, Macro-F1, and AUC assess classification, with Macro-F1 primary because the datasets are class-imbalanced; reliability diagrams assess calibration.AUC uses one-vs-rest classification for multiclass tasks, averaged across classes.Source: §4.1 · pp. 8–9
4.2 Comparison with Baseline Methods
- MedCORE achieves the best performance across three datasets, averaging 87.0% Macro-F1 and exceeding BiomedCLIP fine-tuning by 3.3, 2.4, and 3.7 percentage points on ISIC 2018, BUSI, and IDRiD, respectively.Table 3 reports the complete classification results.Source: §4.2 · p. 9, Table 3 · p. 10
4.3 Ablation Study
- Replacing criterion-level reasoning with a global feature representation reduces average Macro-F1 by 3.4 percentage points; removing criterion localization costs 2.9 points.Removing GAT reasoning reduces Macro-F1 by 2.0 points, while the multi-scale encoder and uncertainty weighting each contribute 1.7 points; vision–language alignment and prototypes provide smaller, consistent gains.Source: Figure 2 · p. 10, Table 4 · p. 10, §4.3 · p. 11
4.4 Sensitivity to the Number of Clinical Criteria
- Average Macro-F1 rises from 83.6% at K = 1 to 87.0% at K = 5, then declines at K = 7 or 9; K = 5 matches the manually defined clinical criterion set.Larger criterion sets may introduce redundant, weakly defined, or noisy criteria, destabilizing localization and graph reasoning.Source: Figure 3 · p. 11, Table 5 · p. 11, §4.4 · pp. 11–12
4.5 Calibration Analysis
- At 0.95 confidence, empirical accuracy is 0.83 for BiomedCLIP-FT versus 0.93 for MedCORE; equal-weight mean calibration gaps are approximately 8.2% versus 1.2%.Across confidence bins, MedCORE follows the perfect-calibration line more closely.Source: Figure 4 · p. 12, Table 6 · p. 12, §4.5 · p. 12
Conclusions and Future Directions
MedCORE replaces direct image-to-label prediction with criteria-grounded reasoning, combining localized evidence, graph-based criterion refinement, prototype reinforcement, and uncertainty-calibrated aggregation. Across three imaging benchmarks, it reports stronger performance than several baseline families and identifies larger multicentre evaluation, expert-validated annotations, and clinical decision-support studies as future work.
Conclusions and Future Directions
- MedCORE combines frozen biomedical vision–language representations with criterion-level localization, graph attention, visual prototypes, and uncertainty-calibrated weighting across three imaging benchmarks.The paper reports stronger accuracy, macro-F1, and AUC than CNN, Transformer, vision–language, concept bottleneck, and prototype-based baselines.Source: Conclusions and Future Directions · p. 13
- Criterion-level evidence, reliability scores, uncertainty estimates, and spatial heatmaps give MedCORE predictions explanatory structure.The paper describes these outputs as making predictions more transparent and clinically interpretable.Source: Conclusions and Future Directions · p. 13
- Future work will extend evaluation to larger multicentre datasets, incorporate expert-validated criterion annotations, and study calibration and usefulness in clinical decision support.Source: Conclusions and Future Directions · p. 13