PaperlayerView exact revision on arXiv

Source-linked AI summary

MVC-Bench: Benchmarking Calibration of Medical Vision-Language Models

Ashshak Sharifdeen, Shihab Aaqil Ahamed, Ufaq Khan, Muhammad Akhtar Munir Sujair Ibrahim, Mohamed Rafeek Mareer Ahamed, Yutong Xie, Imran Razzak, Muhammad Haris Khan

arXiv:2608.27004v1cs.CV

TL;DR

Medical-VLM calibration remains underexplored even though clinical use requires confidence that reflects correctness. MVC-Bench evaluates calibration across modalities, backbones, domain shifts, calibration and prompt-tuning strategies, and variation in templates and seeds, while MCM regularization addresses a reported source of underconfidence. The benchmark finds that calibration is sensitive to setting and that MCM improves in-domain calibration while remaining competitive under domain shift.

  • Problem

    Medical-VLM calibration is underexplored because existing efforts mainly optimize accuracy, despite clinical reliance on confidence that reflects correctness.

  • Method

    MVC-Bench systematically evaluates calibration across three axes using eight backbones, three medical modalities, domain shifts, calibration strategies, prompt-tuning methods, templates, and random seeds.

  • Results

    Calibration is sensitive to domain shift, modality, backbone, prompt initialization, and random seeds, while MCM reduces in-domain calibration error and remains competitive under domain shift.

  • Takeaways & Limitations

    Calibration should be treated as a first-class requirement for safe medical VLM deployment and evaluated for robustness, adaptation effectiveness, and stability.

  • Takeaways & Limitations

    The study covers medical image classification, and MCM is not universal because its improvements are less consistent in histopathology where overconfidence is more prominent.

Abstract

from arXiv · show

Reliable evaluation of vision-language models (VLMs) and medical vision-language models (Medical-VLMs) requires calibrated confidence, particularly under realistic clinical conditions. However, existing efforts mainly focused on improving accuracy, leaving calibration in the medical domain underexplored. To this end, we propose MVC-Bench, a calibration-centric benchmark for medical image classification with VLMs and Medical-VLMs. MVC-Bench assesses the calibration across three axes: (i) robustness to modality, backbone, and domain shift (ii) effectiveness of calibration strategies and prompt-tuning methods (iii) stability under prompt-template and random-seed variations. The benchmark covers eight different backbones, three medical modalities, including fundus imaging, histopathology, and chest X-ray under in-domain and domain shift settings. It compares post-hoc calibration, train-time calibration, and zero-shot inference methods, together with six prompt-tuning methods. Across more than 1638 controlled experiments, we report accuracy and Expected Calibration Error (ECE) as primary metrics, and further report results with complementary calibration measures, including Maximum Calibration Error (MCE) and Adaptive Calibration Error (ACE). We further investigate the underlying causes of miscalibration in VLMs and Medical-VLMs and propose a simple train-time calibration method, Multi-Class Margin (MCM) regularization, which achieves lowest ECE on 10 out of 12 settings in in-domain and remains competitive under domain shifts. Collectively, MVC-Bench provides a structured evaluation framework and actionable guidance for improving calibration in safety-critical medical workflows.

1 Introduction

MVC-Bench addresses underexplored confidence calibration in prompt-tuned VLMs and Medical-VLMs through a systematic evaluation across clinically relevant modalities, backbones, shifts, methods, prompts, and seeds. It also analyzes miscalibration and introduces MCM regularization to improve calibration.

  • Motivation: Medical-VLM calibration remains underexplored because prior prompt-tuning work mainly optimizes accuracy despite clinical reliance on reliable confidence scores.Miscalibration can produce overconfident incorrect predictions or unreliable underconfident correct predictions, affecting review, triage, and trust.
  • Benchmark scope: MVC-Bench evaluates calibration along robustness, effectiveness, and stability axes across modality, backbone, domain shift, calibration methods, prompt tuning, templates, and random seeds.The benchmark covers fundus, histopathology, and chest X-ray datasets and more than 1638 controlled experiments.
  • Benchmark scope: The benchmark compares classical and VLM-aware calibration techniques across eight backbones and six prompt-tuning methods.CoOp serves as a baseline, alongside KgCoOp, MaPLe, PromptSRC, HiCroPL, and ProGrad.
  • Benchmark scope: MVC-Bench investigates calibration in fundus imaging, histopathology, and chest X-ray under in-domain and domain-shift settings.Its design targets heterogeneous medical datasets where annotations are scarce and costly and acquisition conditions vary.
  • MCM regularization: MCM regularization increases true-versus-rest logit margins while controlling margin variance to reduce underconfidence and spurious confidence spikes.The method is motivated by weak inter-class separation and achieves the lowest ECE for most backbones while remaining competitive under domain shifts.

3 MVC-Bench

MVC-Bench stress-tests medical VLM calibration across robustness, effectiveness, and stability, spanning modalities, backbones, domain shifts, calibration strategies, prompt tuning, templates, and seeds. Results show calibration is method-, backbone-, modality-, and shift-dependent, with accuracy alone insufficient to indicate reliable confidence.

  • Benchmark design: MVC-Bench evaluates calibration across modality, backbone, and domain shifts, alongside calibration strategies, prompt-tuning methods, prompt templates, and random seeds.The protocol covers fundus, histopathology, and chest X-ray tasks and treats calibration as a stress test rather than a single in-domain leaderboard.
  • Robustness to domain shift: Domain-shift rank transfer is weak and metric-dependent across modalities and backbones.Complementary ACE and MCE analyses show that ranking stability under some closer shifts is not universal.
  • Backbone and modality effects: Domain-specific backbones generally achieve lower mean ECE than generic CLIP variants, but medical pretraining alone does not guarantee calibrated confidence.BioMedCLIP-X-ray outperforms MedCLIP-X-ray, while PLIP and QuiltNet calibrate strongly in aligned modalities and MedCLIP-X-ray is a counterexample.
  • Accuracy–calibration relationship: Higher accuracy does not generally imply better calibration because the accuracy–ECE relationship depends strongly on calibration method and backbone.ACE and MCE analyses support the same method- and backbone-dependent pattern.
  • Calibration-method effectiveness: MBLS, Base, and MDCA are most consistent in-domain, but no existing method is uniformly reliable across in-domain and domain-shift settings.Under domain shift, ECE is mostly degraded; only ECCV_ZS and ECCV_Penalty improve in subsets of cases.
  • Prompt-tuning effectiveness: PromptSRC reaches approximately 72% accuracy with approximately 11.5% ECE, whereas CoOp, HiCroPL, and KgCoOp balance approximately 68–75% accuracy with approximately 3.5–4.8% ECE.ProGrad performs worst in both accuracy and ECE, while MaPLe is intermediate.
  • Prompt-template stability: Prompt-template wording affects calibration stability: templates P1–P4 are low and stable, while noisy P7 has high variance.Among the evaluated templates, “a photo of a” gives the most stable calibration.
  • Random-seed stability: Seed stability varies across backbones, with BioMedCLIP-DR-B/32, PLIP-DR-B/32, and CLIP-DR-ViT-B/32 showing low seed variability.Stability is measured as the standard deviation of ECE across three seeds.

5 Multi-Class Margin Regularization (MCM)

MCM addresses underconfidence caused by weak inter-class separation by expanding true-vs-rest margins while controlling their dispersion. It achieves the lowest ECE in 10/12 in-domain settings and remains competitive under domain shift, though effectiveness varies by modality.

  • Weak inter-class separation produces underconfidence when the correct-class logit only slightly exceeds competing logits.
  • MCM increases true-vs-rest margins and controls margin variance to reduce underconfidence and spurious confidence spikes.The margin term is weighted by α, while the dispersion term is weighted by β.
  • 10/12 ID settings achieved the lowest ECE with MCM, compared with 4/12 settings under DS.Other methods, particularly ECCV_ZS, remained competitive in individual DS configurations.
  • MCM is not uniformly effective: it increases ECE for PLIP-Histo. and QuiltNet-Histo., where overconfidence is more prominent than low-margin underconfidence.
  • 3.43% ECE and 69.05% accuracy resulted from combining margin expansion and margin control, whereas all-pairs variance alone increased ECE to 5.45% and 6.37%.

6 Conclusion

MVC-Bench evaluates calibration in VLMs and Medical-VLMs under domain, modality, backbone, prompt, and seed variation. Its findings show that calibration is not universally aligned with accuracy, while MCM reduces in-domain error and remains competitive under domain shift.

  • MVC-Bench shows calibration sensitivity to domain shift, modality, backbone choice, hard-prompt initialization, and random seeds.
  • No calibration or prompt-tuning method is consistently optimal across all evaluated settings.
  • Accuracy and calibration are not universally aligned in the benchmark.
  • MCM reduces in-domain calibration error and remains competitive under domain shift.

7 Limitations

MVC-Bench is limited to medical image classification and does not establish whether its calibration findings extend to other medical VLM tasks. MCM is also not universal, particularly where overconfidence dominates.

  • The benchmark covers medical image classification but not segmentation, detection, report generation, retrieval, or medical VQA.Those tasks require different uncertainty definitions and evaluation protocols.
  • MCM targets low-margin underconfidence and is not a universal solution to all calibration failures.
  • MCM improvements are less consistent in histopathology, particularly for PLIP-Histo. and QuiltNet-Histo., where overconfidence is more prominent.
  • Further work is needed to understand modality-specific margin behavior and calibration objectives under broader clinical shifts.

Appendices

The appendices provide supplementary evaluation details, calibration baselines, loss-component analyses, prompt-learning comparisons, and prompt-template specifications.

  • The supplementary material includes evaluation metrics and protocol details.
  • It reports the evaluated calibration baselines.
  • It analyzes the components of the MCM loss.
  • It provides prompt-learning comparisons and prompt-template details.

6. Reliability diagram for other backbones (Sec. F)

This section covers the loss-components analysis and reliability-diagram/ECE analysis presented in the appendix.

  • Loss components analysis is presented in Appendix Section G.
  • Reliability diagrams and ECE analysis for other backbones are presented in Appendix Section H.

A Evaluation metrics and protocol

MVC-Bench evaluates predictive accuracy and calibration using complementary metrics, calibration baselines, prompt-learning methods, and controlled hyperparameter experiments. The analysis shows that combining MCM loss components balances overconfidence and underconfidence, while calibration reduces ECE with little accuracy change.

  • Evaluation metrics: Accuracy, ECE, MCE, ACE, and reliability diagrams evaluate predictive performance and calibration.ECE measures weighted binwise confidence–accuracy gaps; MCE captures the worst binwise discrepancy, while ACE uses adaptive equal-sized bins.
  • Calibration baselines: MDCA, label smoothing, MBLS, ECCV_ZS, ECCV_penalty, and temperature scaling provide train-time or post-hoc calibration baselines.
  • MCM analysis: Mean-term expansion drives overconfidence, variance restriction produces underconfidence, and their combination in MCM yields a stable margin field and lower ECE.
  • Prompt-learning evaluation: Five prompt-learning methods are compared on diabetic retinopathy, histopathology, and chest X-ray benchmarks with and without calibration.
  • Prompt-learning evaluation: 12.44 to 10.02: MaPLe’s average ECE decreases after calibration while accuracy remains effectively unchanged.The reported calibrated variants also reduce average ECE for KgCoOp, PromptSRC, ProGrad, and HiCroPL.
  • Hyperparameter selection: 69.05 accuracy and 3.43 ECE: the selected MCM setting is (α, β) = (0.1, 0.01).At fixed β = 0.01, increasing α raises accuracy but worsens ECE; increasing β generally worsens ECE and reduces accuracy.

G.1 Leave One Dataset Out Hyperparameter Transfer

The LODO analysis excludes each target dataset during hyperparameter selection and finds a stable MCM configuration across diabetic-retinopathy datasets. The broader paired comparison shows consistent in-domain ECE gains but only competitive performance under domain shift.

  • LODO protocol: Every LODO fold selects (α, β) = (0.1, 0.01) using the other three diabetic-retinopathy validation splits.The held-out target dataset is excluded from hyperparameter selection, with no final test labels used for choosing the setting.
  • LODO results: 3.43 versus 4.44: LODO-MCM reproduces default-MCM ECE and improves on Base across held-out diabetic-retinopathy datasets.The same setting is fixed for remaining backbones, modalities, and domain-shift experiments.
  • Domain-shift rank transfer: −0.0598 mean Kendall correlation across 30 tests indicates weak preservation of calibration-method rankings from in-domain to shifted domains.Seventeen tests produce negative correlations, 13 produce positive correlations, and none show significantly positive rank preservation at p < 0.05.
  • Paired MCM comparison: 10 of 12 in-domain configurations favor MCM against most baselines, while 6–9 of 12 shifted-domain configurations favor MCM without statistically significant paired differences.The in-domain improvements are significant against ECCV_ZS, ECCV_Penalty, and temperature scaling; under domain shift, MCM remains competitive rather than uniformly dominant.

J.2 Domain-Shift Evaluation

The domain-shift evaluation counts controlled calibration-transfer runs across fundus, chest X-ray, and histopathology settings. Across the core benchmark, 1,638 seed-level runs generate 4,914 binning-based calibration metric evaluations, alongside accuracy measurements.

  • Aggregate scale: 1,638 seed-level controlled runs comprise the core MVC-Bench calibration benchmark.The benchmark excludes the zero-shot calibration setting for domain-shift evaluation and uses seven calibration methods there.
  • Fundus domain shift: 504 seed-level runs cover fundus domain shift across eight backbones, three shifted datasets, three seeds, and seven calibration methods.
  • Chest X-ray domain shift: 42 seed-level runs cover chest X-ray domain shift using two backbones, one shifted dataset, three seeds, and seven calibration methods.
  • Histopathology domain shift: 84 seed-level runs cover histopathology domain shift using two backbones, two shifted datasets, three seeds, and seven calibration methods.
  • Aggregate scale: 4,914 binning-based calibration metric evaluations result from applying ECE, MCE, and ACE to every controlled run.Accuracy is also computed for each run to examine predictive performance alongside calibration reliability.

J.4 Summary of Experimental Scale

MVC-Bench’s core scale is organized around in-domain and domain-shift calibration across modalities, backbones, methods, and seeds, with ACE and MCE supplementing ECE. The appendix analyses also examine whether calibration rankings transfer across shifts and how accuracy relates to calibration.

  • Experimental scale: 1,638 seed-level controlled runs form the conservative core benchmark count, excluding supplementary diagnostic analyses.The core count covers in-domain and domain-shift calibration across backbones, datasets, methods, and seeds.
  • Domain-shift analysis: ACE and MCE domain-shift analyses show metric-dependent transfer, with some rank preservation under closer shifts but instability across modalities and backbones.ACE and MCE identify positive transfer in some retinopathy and histopathology settings, but negative transfer for BioMedCLIP under X-ray shift.
  • Experimental scale: Three medical modalities and seven calibration methods structure the core evaluation, with zero-shot calibration excluded from domain-shift counting.The benchmark includes fundus, histopathology, and chest-X-ray configurations under ID and DS settings.
  • Backbone analysis: Backbone choice strongly affects calibration, with several domain-specific backbones below approximately 3% mean MCE and MedCLIP-X-ray reaching 20.84%.MedCLIP-X-ray also has the highest reported ACE at 21.31%, while generic CLIP variants generally show higher ACE.
  • Accuracy and calibration: Accuracy–calibration relationships depend on both calibration method and backbone rather than following a universal pattern.ACE and MCE broadly reproduce ECE-based method-dependent correlations, while differing in whether broader or worst-bin errors dominate.

K.4 Effectiveness: Which calibration methods show consistent calibration performance

Calibration effectiveness varies across methods, backbones, modalities, and evaluation metrics. The proposed method is most consistent in-domain, while prompt initialization and modality affect calibration stability and complementary metrics reveal modality-specific trade-offs.

  • Calibration methods: Ours achieves the highest ACE Nbest count in-domain at 8 and remains among the leaders under domain shift.Under domain shift, performance deteriorates across methods, but Ours and ECCV_ZS have the highest Nbest counts.
  • Calibration methods: Ours leads MCE consistency in-domain with Nbest = 8 and under domain shift with Nbest = 5.The results describe superior in-domain calibration consistency and competitive domain-shift performance across metrics.
  • Prompt tuning: HiCroPL has the lowest prompt-tuning errors, with ACE 3.53% and MCE 1.06%, followed by CoOp and KgCoOp.These methods maintain relatively low average and worst-case calibration errors.
  • Prompt stability: Hard-prompt initialization affects both average and worst-case calibration: P1 and P2 are most stable, whereas later templates become more variable.For MCE, P1 and P2 have standard deviations of 0.80 and 1.02 and mean values of 1.45 and 1.87, respectively.
  • Complementary metrics: Brier scores are mostly preserved or slightly improved over vanilla CE, with strongest benefits in fundus and less consistent effects in histopathology.The proposed loss slightly increases Brier scores for PLIP-Histo and QuiltNet-Histo.
  • Class-wise calibration: The proposed loss improves class-wise calibration across fundus backbones and modestly for X-ray, but increases WSECE for PLIP-Histo and QuiltNet-Histo.For BioMedCLIP-X-ray, WSECE decreases from 14.31 to 13.19.

N Results

The results evaluate calibration across three medical modalities and multiple backbone configurations in in-domain and domain-shift settings. MCM improves calibration consistently for diabetic retinopathy and chest X-ray, but gains are not uniform in histopathology or under every shift.

  • Evaluation scope: Three modalities—diabetic retinopathy, histopathology, and chest X-ray—are evaluated across 12 modality-specific configurations.The configurations include generic CLIP, MedCLIP, BioMedCLIP, PLIP, and QuiltNet variants.
  • In-domain results: MCM improves calibration in diabetic retinopathy CLIP backbones and in Medical-VLM examples without compromising accuracy.The supplied results specifically report calibration improvements for DR and Medical-VLM configurations.
  • In-domain results: MedCLIP-X-ray improves accuracy from 65.77 to 65.83 and ECE from 23.97 to 20.03, with MCE and ACE also decreasing.These values are averaged over two X-ray datasets.
  • In-domain results: Histopathology gains are inconsistent: PLIP-Histo ECE rises from 7.00 to 9.92, while QuiltNet-Histo ECE rises from 6.21 to 8.55.PLIP-Histo accuracy decreases slightly, whereas QuiltNet-Histo accuracy increases slightly.
  • Domain-shift results: Under domain shift, improvements occur for CLIP-DR-ViT-B/32, MedCLIP-DR-B/32, and BioMedCLIP-X-ray, while PLIP and QuiltNet DR variants degrade.For example, QuiltNet-DR-B/32 accuracy falls from 40.08 to 27.92 and ECE rises from 15.42 to 23.72.
Loading 2608.27004v1…