Source-linked AI summary
Learning to Adapt and Calibrate: Score Distribution Alignment for Few-Shot Uncertainty Prediction in Medical VLMs
Xuan Cuong Ngo, Ngan Le
TL;DR
Few-shot supervised fine-tuning can shift nonconformity scores and break the exchangeability required for reliable conformal coverage, while transductive alternatives underuse labeled support data. AlignCP aligns reweighted calibration and unlabeled target score distributions after adaptation, and experiments report improved adaptation performance with reliable coverage, smaller prediction sets, and reduced class-conditioned coverage disparities.
Problem
Supervised fine-tuning in few-shot transfer shifts nonconformity scores and breaks calibration–test exchangeability, making standard conformal coverage unreliable while transductive alternatives underuse labeled support data.
Method
AlignCP fine-tunes the medical VLM on labeled support data and learns a reweighted calibration score distribution aligned with unlabeled target scores to reduce the induced coverage gap.
Results
AlignCP improves adaptation performance, maintains reliable coverage, produces smaller prediction sets, and reduces class-conditioned coverage disparities across diverse medical tasks and modalities.
Takeaways & Limitations
Score-distribution alignment provides a way to combine supervised few-shot adaptation with conformal reliability under target shift.
Takeaways & Limitations
The setup uses labeled calibration data from a source distribution and unlabeled target queries from the target marginal distribution.
Abstract
from arXiv · showhide
Uncertainty estimation for medical vision--language models (VLMs) using conformal prediction has gained increasing attention due to its distribution-free coverage guarantees. However, standard conformal prediction relies on exchangeability between calibration and test data and typically requires a sufficiently large calibration set to obtain reliable coverage. These assumptions are difficult to satisfy in few-shot transfer settings, where only a small labeled support set is available to adapt a pretrained VLM to a new medical task, while an unlabeled query set is used for evaluation. Supervised fine-tuning on the support set changes the model parameters and consequently shifts the nonconformity score distribution, breaking exchangeability between calibration and query samples and leading to unreliable coverage under distribution shift. Existing transductive conformal adaptation methods often preserve validity by avoiding supervised updates. While this helps maintain conformal assumptions, it underutilizes the scarce labeled support data and limits task adaptation, which is the primary objective in few-shot learning. In this setting, conformal prediction should serve as an uncertainty estimation layer that supports the adapted model, rather than preventing adaptation itself. To this end, we propose AlignCP, a framework that reconciles supervised few-shot adaptation with conformal uncertainty estimation under non-exchangeability. AlignCP learns a reweighted calibration distribution that reduces the score-level discrepancy between the labeled support set and the unlabeled query set. By aligning the one-dimensional nonconformity score distributions, AlignCP aims to close the coverage gap induced by adaptation without requiring query labels.
1. Introduction
Few-shot supervised adaptation improves task alignment but shifts nonconformity scores, undermining conformal coverage. AlignCP addresses this mismatch by reweighting calibration scores to better match unlabeled test scores while preserving adaptation benefits.
- Standard conformal prediction loses its coverage guarantee when supervised fine-tuning breaks exchangeability between calibration and test samples.
- Transductive methods preserve coverage by avoiding supervised updates, but this underuses scarce labeled support data and limits few-shot task adaptation.
- The framework treats conformal prediction as an auxiliary uncertainty layer for the adapted model rather than a constraint preventing supervised learning.
- AlignCP fine-tunes the medical VLM on labeled support data, then learns a reweighted calibration score distribution that better matches unlabeled test scores.
- Experiments across diverse medical domains show stronger predictive performance, reliable empirical coverage, and competitive prediction-set efficiency under distribution shift.
2. Related Work
Related work establishes conformal prediction as a coverage-guaranteed uncertainty framework whose validity depends on matching calibration and test score distributions. Domain shift therefore motivates methods extending conformal prediction beyond standard exchangeability.
- Classification conformal prediction constructs finite-sample marginally covered prediction sets under exchangeability.
- Common classification nonconformity scores include LAC, APS, and RAPS, which calibrate prediction sets using held-out data.
- Under domain shift, coverage guarantees can fail when calibration and test samples do not follow the same score distribution.
3. Methodology
The methodology formalizes how supervised adaptation shifts nonconformity scores between source calibration and target queries, then uses score-distribution reweighting to reduce the resulting coverage gap without target labels.
- Coverage-gap formulation: The coverage gap Δ_P,Q(α) measures the change in marginal coverage when the test-score distribution changes from P_s to Q_s.Reducing score-level discrepancy is therefore linked to improving target reliability.
- Distribution-shift setup: After supervised adaptation, source calibration scores follow P_s while target test scores follow Q_s, so standard split conformal validity does not directly transfer when P_s ≠ Q_s.The same source-calibrated threshold is evaluated under different test-score distributions.
- Reweighted calibration: AlignCP assigns nonnegative weights to labeled calibration scores so their weighted empirical distribution better matches the target score distribution.Uniform weights recover the standard unweighted empirical calibration distribution.
- Unlabeled target approximation: Because target labels are unavailable, lower and upper candidate-label scores for each unlabeled query bracket the unknown target score and induce computable surrogate distributions.Their empirical CDFs satisfy F_Q↑(t) ≤ F_Qs(t) ≤ F_Q↓(t).
- Computable alignment objective: AlignCP replaces the inaccessible target-distribution discrepancy with a computable surrogate based on the lower and upper query-induced CDFs.The surrogate is obtained by applying the bound to both distributions and averaging the resulting inequalities.
- Coverage control: Minimizing the empirical objective reduces an upper bound on Δ_P,Q, which tightens the lower bound on target coverage and helps recover the desired coverage level.The weighted threshold and weighted prediction set are then constructed from the reweighted calibration distribution.
4. Experiments
Across nine medical datasets and multiple adaptation strategies, AlignCP maintains coverage near target levels while preserving supervised adaptation gains and improving prediction-set efficiency. Its alignment remains stable across optimization settings and extends to natural-image benchmarks.
- Experiment Settings: Experiments span nine datasets across histology, ophthalmology, and chest X-ray, using modality-specific pretrained foundation models and supervised linear-probe, LoRA, and adapter adaptation.All datasets are multi-class classification problems with an average of eight classes per dataset.
- Overall Performance: AlignCP preserves supervised adaptation accuracy while maintaining coverage close to target levels at α = 0.10 and α = 0.05.Adapt+SCP matches the linear-probe AlignCP ACA of 63.9 but consistently undercovers, whereas AlignCP also achieves smaller prediction sets than SCP and SCAT among coverage-valid methods.
- Set Efficiency: Across all shot regimes, AlignCP consistently produces smaller prediction sets than SCP and SCAT.Its CCV is slightly worse than SCAT in the extremely low-shot regime, then becomes the lowest as more calibration samples become available.
- Coverage Distribution Analysis: Compared with Adapt+SCP under the same supervised predictor, AlignCP shifts coverage distributions toward the target across nine datasets and reduces undercoverage.The distributions are centered close to target on Gleason, MESSIDOR, MMAC, and CheXpert5x200; Skin, FIVES, and NIH are slightly conservative, while COVID retains a larger shift.
- Natural-Image Generalization: On ImageNet and CIFAR-10, AlignCP preserves supervised-adaptation accuracy gains while maintaining target-level coverage and competitive prediction-set size and CCV.These results indicate that the score-distribution alignment also applies beyond medical imaging.
- Sensitivity Analysis: Coverage, set size, and CCV vary only marginally across initialization, learning-rate, and iteration settings, indicating stable optimization behavior.Table 4 averages each hyperparameter comparison over the remaining settings under LAC at α = 0.10.
- Statistical Significance: AlignCP improves ACA over SCP and SCAT by 13.815 and 8.729, respectively, across 100 matched runs.Both comparisons have p = 3.9 × 10−18, with narrow paired-bootstrap 95% confidence intervals remaining above zero.
5. Conclusion
AlignCP combines supervised few-shot adaptation with conformal uncertainty estimation by learning a reweighted calibration score distribution that better matches target-domain behavior. Across diverse medical tasks and modalities, it improves adaptation performance, maintains reliable coverage, produces smaller prediction sets, and reduces class-conditioned coverage disparities, while its bounds can loosen when surrogate score distributions poorly approximate the test distribution.
- AlignCP learns a reweighted calibration score distribution to address the score-level mismatch introduced by supervised adaptation.
- Experiments across diverse medical tasks and modalities show improved adaptation performance, reliable coverage, smaller prediction sets, and reduced class-conditioned coverage disparities.
- The coverage gap bound can loosen when surrogate score distributions poorly approximate the test distribution.
- In extremely low-shot settings, limited calibration data may make the bounds less stable and increase CCV.