Source-linked AI summary
Prompt-Anchored Residual Adaptation for Biomedical Vision-Language Models
Jingxuan Kang, Qianying Yue, Che Liu, Chen Qin
TL;DR
Biomedical few-shot classification must capture subtle class differences from limited labeled supports, whose composition can shape predictions. PARA anchors a support-learned visual prediction to frozen prompt predictions, and repeated-support evaluation separates sampling from optimization variation. Across biomedical datasets, it reports the strongest average and worst-20% results while maintaining base-to-novel generalization.
Problem
Pretrained biomedical representations may miss subtle target-class visual differences, while few-shot predictions can be shaped by support examples that capture only part of class variation.
Source: Introduction · pp. 1–2Method
PARA retains the frozen prompt prediction as an anchor and adds a support-learned visual prediction through a residual computed from frozen support geometry, alongside repeated-support evaluation.
Source: Introduction · p. 2Results
Across 11 biomedical datasets, PARA achieves the strongest average and worst-20% results while maintaining base-to-novel generalization.
Source: Conclusion · p. 7Takeaways & Limitations
The repeated-support protocol separates support-selection variation from optimization randomness and characterizes both average and lower-tail performance.
Source: Introduction · p. 2
Abstract
from arXiv · showhide
Pretrained biomedical vision-language models achieve strong zero-shot performance in biomedical image classification. However, downstream biomedical classification often depends on subtle visual differences between classes that may not be fully captured by pretrained representations. Few-shot adaptation addresses this mismatch by optimizing a task-specific predictor on a small labeled support set. Because the selected examples capture only part of the visual variation within the target classes, the adapted predictions can depend strongly on their composition. We propose Prompt-Anchored Residual Adaptation (PARA), which retains the frozen prompt prediction as a support-invariant semantic anchor and incorporates a visual prediction learned from the support set through an anchor-relative residual. The residual step is computed in a closed form from frozen support embeddings using anchor discrepancy and support agreement. Support-set dependence also limits evaluation: comparisons are fair within a shared draw but remain conditional on its composition. To obtain more reliable comparisons, we introduce a repeated-support protocol that separates support-selection variation from optimization randomness and reports both average and worst-20% performance. PARA achieves state-of-the-art performance in both few-shot classification and base-to-novel generalization.
Introduction
Biomedical VLMs provide transferable zero-shot representations, but subtle task-specific visual differences and scarce expert-labeled data motivate few-shot adaptation. Because sampled supports capture only part of class variation, PARA anchors adaptation to frozen prompt predictions and the paper evaluates performance across repeated shared support draws.
Introduction
- Biomedical VLMs enable zero-shot classification, but pretrained representations may miss subtle target-class differences, while sufficient expert-labeled data are costly to collect.Few-shot adaptation uses a small labeled support set to address this constraint.Source: Introduction · p. 1
- On COVID-19, Accuracy spans 33.9%–74.4% for CoOp across 20 support draws, while BiomedCoOp and vMFCoOp span 17.3% and 30.5%.These ranges illustrate how predictions can depend on which examples compose the support set.Source: Introduction · p. 1, Figure 1 · p. 1
- PARA retains the frozen prompt prediction as an anchor and adds a support-learned visual prediction through a residual determined by frozen support geometry.The residual uses anchor discrepancy and support agreement; the same global bound is used across datasets without per-dataset tuning or a validation set.Source: Introduction · p. 2
- The protocol evaluates 20 shared support sets per dataset and shot level, separates support-selection variation from optimization randomness, and reports mean and CVaR20 for Accuracy and Macro-F1.CVaR20 is average performance over the worst 20% of support draws.Source: Introduction · p. 2
- Across 11 biomedical datasets and five shot levels, PARA achieves the highest mean and CVaR20 for Accuracy and Macro-F1, with the strongest base-to-novel generalization.The evaluation uses repeated shared support draws and reports average and lower-tail performance.Source: Introduction · p. 2
Related Work
Biomedical adaptation methods modify prompts, visual representations, or classifiers, while retaining pretrained text knowledge is another design direction. PARA keeps the frozen prompt prediction as a separate branch, and repeated-support evaluation measures both average and lower-tail performance.
Biomedical Vision-Language Adaptation
- Biomedical adaptation methods learn prompts or alter visual representations and classifiers, but their task-specific signal remains conditioned on the sampled support examples.Examples include biomedical prompt methods, feature adapters, cache classifiers, visual prompt tuning, and low-rank updates.Source: Biomedical Vision-Language Adaptation · p. 2
Anchoring Adaptation to Frozen Text Knowledge
- PARA preserves the frozen prompt prediction as a separate branch and sets an anchor-relative residual step from frozen support geometry.Prior approaches incorporate frozen text knowledge into an adapted classifier or training objective; Figure 2 depicts PARA's separate-branch prediction and geometry-based step.Source: Anchoring Adaptation to Frozen Text… · p. 2, Figure 2 · p. 3
Evaluation under Support-Set Sampling
- The protocol uses 20 shared support draws, aggregates repeated optimization within each draw, and reports mean performance and CVaR20 over the worst 20% of draws.This separates support-selection variation from optimization randomness, which three-draw mean-and-standard-deviation reporting does not.Source: Evaluation under Support-Set Sampling · p. 3
Method
PARA combines a frozen prompt prediction with a support-trained visual prediction, using frozen support geometry to set an anchor-relative residual. It also transports base-class visual–text displacements to semantically related novel classes.
Method
- PARA separates learning task-specific visual evidence from deciding how far the prediction should move toward it.The prediction retains the frozen prompt branch and adds the support-learned visual prediction as an anchor-relative residual.Source: Method · p. 3, Anchor-Relative Residual Prediction · p. 4
Prompt-Anchored Visual Adaptation
- LoRA parameters and a visual logit scale are optimized on support examples, while adapted class prototypes combine the fixed text anchor with adapted support representations.For training, each support image is excluded from its own positive-class prototype; in one-shot classes, that prototype reduces to the fixed text anchor.Source: Prompt-Anchored Visual Adaptation · pp. 3–4
Frozen Support Geometry
- Anchor discrepancy measures departure from the text anchor, while support agreement reflects support count and directional alignment.Agreement is one in the one-shot case and reaches the number of support examples when all support representations are aligned.Source: Frozen Support Geometry · p. 4
Anchor-Relative Residual Prediction
- The residual step uses discrepancy and agreement jointly, so a class contributes strongly only when its support direction departs from the anchor and is consistently supported.The step is computed from frozen support representations and is bounded above by λ; Table 2 reports Accuracy CVaR20 over the worst 20% of support draws, averaged across 11 datasets.Source: Anchor-Relative Residual Prediction · p. 5, Table 2 · p. 5
Base-to-Novel Residual Transport
- For novel classes without support, PARA transfers base-class visual–text displacements according to semantic similarity between frozen text anchors.Semantically closer base classes receive larger transfer weights, and the adjusted novel anchor is scored using the frozen visual representation.Source: Base-to-Novel Residual Transport · p. 5
Experiments
PARA leads the reported few-shot and base-to-novel comparisons, while repeated shared support draws separate support-selection variation from optimization randomness. The experiments also examine performance consistency and the effects of removing individual design components.
Experimental Setup
- For each dataset, evaluation uses one fixed class-stratified split, with 30% of images reserved as support pool and 70% as test set; all methods share the split and support draws.Source: Repeated-Support Evaluation Protocol · p. 5
Repeated-Support Evaluation Protocol
- Twenty shared, class-balanced support draws per dataset and shot level are paired across methods; three optimization seeds per draw for stochastic methods separate support-selection variation from optimization randomness.Mean summarizes typical performance, while CVaR20 averages the four lowest support-level scores.Source: Repeated-Support Evaluation Protocol · p. 5
Main Results
- 80.85% Accuracy HM and 74.46% Macro-F1 HM exceed the best competing results of 75.18% and 71.07%, respectively, while PARA preserves novel-class performance.The same highest-HM pattern holds for CVaR20; Figure 4 reports mean Base and Novel Accuracy averaged over 10 datasets and 20 support selections.Source: Table 3 · p. 6, Main Results · p. 7, Figure 4 · p. 7
Few-shot classification.
- Few-shot classification reports Accuracy and Macro-F1 at K = 1, 2, 4, 8, and 16, with two-sided Wilcoxon tests and Holm correction over baseline comparisons.Correction is applied separately for mean and CVaR20 within each metric.Source: Few-shot classification · p. 6
Base-to-novel generalization.
- For base-to-novel evaluation, models adapt from 16-shot base-class support and are evaluated separately on base and novel classes; BUSI is excluded because its novel split has one class.The evaluation reports Base, Novel, and harmonic-mean results for Accuracy and Macro-F1, using the BiomedCoOp class splits.Source: Base-to-novel generalization · p. 6
Main Results
- PARA attains the lowest error on 8 of 11 datasets, stays within a factor of 1.15 on all 11, and achieves the highest average Accuracy CVaR20 at 62.41%, outperforming the strongest competing result by 2.52%.Paired Wilcoxon tests show significant gains in both metrics over all tested baselines except BiomedCoOp, for which only Macro-F1 is significant.Source: Main Results · p. 7
- 66.95% Accuracy and 61.34% Macro-F1 averaged across shot levels improve over the strongest competing results by 2.18% and 2.38%, respectively.PARA achieves the highest Accuracy and Macro-F1 at every shot level.Source: Main Results · p. 7
Ablation Study
- Removing the prompt-anchored residual lowers Accuracy CVaR20 to 57.75% and raises SD from 3.06 to 4.84; a fixed step lowers mean Accuracy and Accuracy CVaR20 to 64.78% and 60.17%.The ablation also reports modest drops after removing prototype anchoring, from 66.95% and 62.41% to 66.66% and 61.70% for mean Accuracy and Accuracy CVaR20.Source: Ablation Study · p. 7
Conclusion
PARA anchors support-set adaptation to frozen prompt predictions, and its repeated-support protocol separates support-selection variation from optimization randomness. Across 11 biomedical datasets, it reports the strongest results on average and in the worst 20% while maintaining base-to-novel generalization.
Conclusion
- PARA combines a frozen prompt anchor with an adapted prediction as an anchor-relative residual, with the step computed from anchor discrepancy and support agreement in frozen space.The protocol reports average and worst-20% performance across 11 biomedical datasets; PARA achieves the strongest results on both measures while maintaining base-to-novel generalization.Source: Conclusion · p. 7
A.1 Dataset Overview
The evaluation uses 11 biomedical datasets and shared, class-balanced support draws, with repeated optimization seeds to separate support-selection variation from optimizer randomness. Base-to-novel evaluation uses 16-shot adaptation on base classes and reports results on base and novel classes.
A.1 Dataset Overview
- Each of 11 datasets uses a class-stratified split with 30% of images in a support pool and 70% in a disjoint fixed test set shared by all methods.Images are converted to RGB, resized and cropped to 224 × 224, and normalized using BiomedCLIP ViT-B/16 preprocessing.Source: §A.1 · p. 10
A.2 Repeated Support Few-Shot Protocol
- For each dataset and shot level K ∈{1, 2, 4, 8, 16}, the protocol uses 20 shared class-balanced draws, with support-pool example reuse balanced across draws.Stochastic methods use three optimizer seeds per draw; Accuracy and Macro-F1 are computed on the common fixed test set, with support draws as the statistical units.Source: §A.2 · p. 10
A.3 Base-to-Novel Protocol
- Base-to-novel evaluation adapts with K = 16 support from base classes, then reports Base, Novel, and harmonic mean results for Accuracy and Macro-F1 on ten datasets.BUSI is excluded because its novel split contains only one class.Source: §A.3 · p. 10
B Implementation Details
Implementation uses a frozen BiomedCLIP ViT-B/16 checkpoint and a fixed text anchor built from released class descriptions. The description bank contains 50 GPT-4-generated descriptions per medical class.
B Implementation Details
- The implementation keeps all pretrained image and text parameters frozen and uses the public BiomedCLIP ViT-B/16 checkpoint.It uses 50 descriptions generated by GPT-4 for each medical class, following BiomedCoOp's released descriptions.Source: B Implementation Details · p. 11
- The released descriptions were generated by requesting visual discriminative features for distinct medical cases of each class in its modality.The instruction asks for N textual descriptions for [CLASS] found in [MODALITY].Source: B Implementation Details · p. 11
- The fixed text anchor for each class is the normalized mean of the normalized embeddings of its 50 released descriptions.The resulting anchor is denoted t_c.Source: B Implementation Details · p. 11
C.1 Results by Dataset
Results compare PARA with adaptation baselines across datasets and shot levels, while paired statistical tests assess differences across datasets. PARA's reported gains are significant against most baselines for Accuracy and all baselines for Macro-F1.
C.1 Results by Dataset
- Figures A1 and A2 show mean Accuracy and Macro-F1 by dataset and shot level, with average panels across datasets; Table A3 gives the complete baseline comparison averaged over five shot levels.The figures display PARA, TaskRes, LP++, TAMP, CLAP, and BiomedCoOp; BiomedCLIP appears at K = 0 as the support-independent prediction.Source: §C.1 · p. 11
C.2 Statistical Comparisons
- Paired differences are tested across 11 datasets after averaging optimizer seeds within support draws and shot levels, with Holm-adjusted p-values over 12 baselines per metric and summary.The analysis uses exact two-sided Wilcoxon signed-rank tests.Source: §C.2 · p. 11
- Both mean and CVaR20 Macro-F1 gains are significant over all 12 baselines; mean Accuracy improves significantly over 11, and Accuracy CVaR20 over 10.The exceptions are BiomedCoOp for mean Accuracy (pH = 0.1230), and Tip-Adapter (pH = 0.0645) plus BiomedCoOp (pH = 0.1748) for Accuracy CVaR20.Source: §C.2 · p. 11
C.3 Failure Analysis
Failure analysis identifies a metric trade-off on imbalanced DermaMNIST and a weak frozen anchor on CHMNIST, where combining the anchor with a stronger visual branch lowers PARA's performance. Figures A1 and A2 show mean results over support draws across datasets and shot levels.
C.3 Failure Analysis
- On DermaMNIST, BiomedCoOp has higher mean/CVaR20 Accuracy (59.23/54.90%) than PARA (53.21/43.24%), while PARA has higher Macro-F1 (33.71/30.03% versus 27.05/25.30%).The melanocytic nevus class accounts for 4,693/7,008 test images (66.97%), while the two smallest classes have 80 and 99 images; the reported gap suggests majority class bias in BiomedCoOp and more balanced PARA performance.Source: §C.3 · p. 11
- On CHMNIST, the frozen anchor has 30.43% Accuracy and 28.61% Macro-F1, while the visual branch averages 75.28%/74.84% versus PARA's 66.96%/65.45%.The text anchor is weak relative to the visual branch, so mixing them pulls PARA away from the stronger visual prediction, especially at low shot counts.Source: §C.3 · p. 12
- Figures A1 and A2 plot mean Accuracy and Macro-F1 by dataset and shot level; their Average panels are means across 11 datasets.Adaptation points average 20 support draws after averaging optimizer seeds within each draw; BiomedCLIP at K = 0 is independent of sampled support.Source: Figure A1 · p. 13, Figure A2 · p. 14
C.4 Complete Result Tables
Table A3 provides few-shot results across datasets and methods, with separate panels for frozen and CLIP-based adaptation methods and prompt-learning methods. Results are averaged over five shot levels and reported as Mean and CVaR20.
C.4 Complete Result Tables
- Table A3 covers few-shot results across datasets for the main method comparison, averaging each value over five shot levels.Mean and CVaR20 are reported in separate blocks.Source: Table A3 · p. 15
- The comparison separates frozen and CLIP-based adaptation methods from prompt-learning methods, with PARA repeated in the latter panel.Source: Table A3 · pp. 15–16
- The listed methods are BiomedCLIP, Tip-Adapter, CLIP-LoRA, TaskRes, CLAP, LP++, SS-Text+, TAMP, and PARA.Source: §C.4 · p. 15
D Complete Base-to-Novel Results
Tables A4 and A5 report base-to-novel results by dataset for Accuracy and Macro-F1, respectively. Each table presents Mean and CVaR20 blocks and computes HM from the displayed Base and Novel values.
D Complete Base-to-Novel Results
- Table A4 reports base-to-novel Accuracy for each dataset, with Mean and CVaR20 shown separately.HM is computed from the displayed Base and Novel values.Source: Table A4 · p. 17
- In both tables, bold marks the best result in each row.Source: Table A4 · p. 17, Table A5 · p. 18
- Table A5 reports base-to-novel Macro-F1 for each dataset, with Mean and CVaR20 shown separately.HM is computed from the displayed Base and Novel values.Source: Table A5 · p. 18