PaperlayerView exact revision on arXiv

Source-linked AI summary

How Medical VLMs Underutilize Their Vision Encoders: A Dermatology Perspective

Janet Wang, Yunbei Zhang, Xiao Wang, Jihun Hamm

arXiv:2609.36557v1cs.CVcs.AI

TL;DR

The paper investigates why MedGemma’s capable vision encoder does not yield equally strong end-to-end zero-shot dermatology diagnosis. It tests explanations for this gap and evaluates frozen-VLM prompting and encoder-assisted reranking, reporting improvements across five VLM backbones in the evaluated dermatology settings.

  • Problem

    The paper asks why a strong vision encoder’s discriminative ability does not translate into MedGemma’s end-to-end zero-shot diagnostic accuracy, and whether inference-time strategies can narrow the gap with the VLM frozen.

    Source: §1 · p. 2

  • Method

    The study examines distribution mismatch, vision-branch under-reliance, and objective misalignment, then combines clinical descriptions, describe-then-decide prompting, and low-label encoder-assisted reranking without VLM fine-tuning.

    Source: §1 · p. 2, §6 · p. 9

  • Results

    The combined interventions improve performance across MedGemma, SkinVL, InternVL3, InternVL3.5, and biomed-Qwen in the evaluated dermatology settings.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The paper’s evaluated approach pairs frozen VLM reasoning with visual-encoder candidate selection, while additional medical modalities support the study of frozen visual representations.

    Source: §1 · p. 2

    The detailed analysis centers on MedGemma-4B and dermatology; the complete pipeline still needs evaluation across medical specialties and prospective validation of clinical utility.

    Source: §6 · p. 9

Abstract

from arXiv · show

Medical Vision-Language Models (VLMs) show significant promise for clinical image understanding, offering accurate diagnosis with interpretable reasoning. However, a critical performance gap exists between their strong vision encoders and the full multimodal model: in dermatology, the MedSigLIP encoder outperforms MedGemma by an average of 10.26 percentage points even when both use zero target-task labels; few-shot linear probing provides further evidence of strong visual representations. This gap motivates an investigation of how visual information is used in end-to-end diagnosis and why plausible-sounding predictions can lack grounding in image evidence. Using dermatology as our primary testbed, we systematically investigate three hypotheses for this phenomenon. We further provide a mechanistic analysis of the model's internal attention patterns, showing that a simple describe-then-decide prompting strategy increases vision attention by 30-40% during generation. Task-specific fine-tuning improves dermatology classification but reduces cross-domain medical question-answering performance in our evaluation. To address these challenges, we combine label-free prompting with low-label encoder-assisted reranking while keeping the VLM frozen. We validate the interventions across five VLM backbones in dermatology and provide supporting representation and attention analyses across additional medical modalities.

1 INTRODUCTION

Dermatology diagnosis is difficult and clinically useful models must ground interpretable decisions in evidence. The paper investigates why MedGemma’s strong vision encoder does not translate into comparable zero-shot diagnosis, and explores frozen-model strategies to narrow the gap.

1 INTRODUCTION

  • MedSigLIP shows strong discriminative power across five skin-disease datasets, yet that strength does not translate into MedGemma’s zero-shot diagnostic accuracy.It is comparable to or sometimes better than PanDerm, while LoRA fine-tuning still underperforms a linear probe; the Fitzpatrick17k-adapted checkpoint loses 5.04 and 15.00 percentage points on closed- and open-ended VQA-RAD accuracy.Source: §1 · p. 2
  • The study asks what drives the encoder–VLM gap and whether inference-time strategies can narrow it while keeping the VLM frozen.It investigates training-distribution mismatch, vision-branch under-reliance, and vision–language objective misalignment.Source: §1 · p. 2
  • The paper combines clinical descriptions, describe-then-decide reasoning, and low-label candidate reranking, reporting improvements across five VLM backbones in evaluated dermatology settings.The approach avoids fine-tuning the VLM; additional medical modalities provide supporting evidence about frozen visual representations.Source: §1 · p. 2
  • The authors frame the analysis as relevant beyond dermatology, linking it to broader concerns about language priors and modality misalignment in vision-language research.The introduction describes this broader relevance as insight into improving reliability and visual grounding in safety-critical applications.Source: §1 · p. 2

2 RELATED WORK

Prior dermatology VLM work combines specialized image–text data and multimodal architectures, while related research examines visual underuse and adaptation with frozen models. This paper situates its clinical-description, describe-then-decide, and encoder-assisted reranking strategies among those approaches.

Foundation Medical Vision-Language Models for Dermatology

  • DermLIP, MONET, SkinVL, and SkinGPT-4 illustrate dermatology VLMs built from specialized image–text collections, CLIP adaptation, or language-model integration.DermLIP uses Derm1M’s one million image–text pairs, while MONET adapts CLIP on over 100K image–caption pairs; SkinVL and SkinGPT-4 integrate language models with dermatology corpora.Source: §2 · pp. 2–3
  • MedGemma provides the study’s testbed, combining a medically tuned MedSigLIP encoder with a Gemma 3 language backbone (Team et al., 2025; Sellergren et al., 2025).The related-work passage describes MedGemma as enabling analysis of visual representation and end-to-end diagnosis.Source: §2 · p. 3

Misalignment of VLMs and Over-reliance on Language Priors

  • Related work reports that VLMs may underuse visual representations, rely on language priors, or lose visual perception after instruction tuning (Xing et al., 2025; Zhai et al., 2023b).Prior studies also connect declining image attention during generation with visually ungrounded responses (Yang et al., 2025).Source: §2 · p. 3

Adaptation with Fixed Foundation Models

  • Frozen-model adaptation includes learned visual prompts, local steering, and multimodal image in-context learning for pathology classification (Ferber et al., 2024).This paper instead pairs textual clinical descriptions and DtD with a low-label encoder probe for reranking while keeping the VLM frozen.Source: §2 · p. 3

3 EVALUATION SETUP AND BASELINE COMPARISONS

The evaluation centers on MedGemma-4B and five dermatology datasets, with additional VLM backbones included. Baseline comparisons show that MedSigLIP’s advantage persists without target-task labels, while image in-context learning and linear probing provide few-shot reference points.

3 EVALUATION SETUP AND BASELINE COMPARISONS

  • MedGemma-4B is selected because the study focuses on visual diagnosis rather than EHR data, unlike the additional medical corpus used to pretrain MedGemma-27B.Both variants use the MedSigLIP image encoder.Source: §3 · p. 3
  • The study evaluates five dermatology datasets spanning broad disease coverage, skin cancer, neglected tropical diseases, varied demographics, and familiar pretraining data.The datasets are SD-260, Fitzpatrick17k, Derm7pt, eSkinHealth, and PAD-UFES-20; additional backbones include InternVL3-4B, InternVL3.5-4B, and biomed-Qwen2.5-VL-3B.Source: §3 · p. 3
  • Clinical descriptions and DtD use no labeled target images, whereas Top-5-to-Top-1 trains a lightweight probe on a small labeled support set before frozen-VLM prediction.Multimodal image ICL instead supplies labeled image–class pairs as demonstrations.Source: §3 · p. 3
  • MedSigLIP exceeds MedGemma by 7.72–13.18 percentage points on all four evaluated zero-label datasets, averaging a 10.26-point advantage.Table 1 reports Top-1 accuracy; its zero-label columns use no fitted classifier.Source: §3 · p. 3, Table 1 · p. 4
  • The 8-shot image-ICL baseline improves over zero-shot MedGemma by 2.67–4.80 points, while 8-shot LP has higher accuracy on all four datasets.Table 1 compares image ICL, linear probing, and LP-assisted Top-5-to-Top-1 prediction in its 8-shot columns.Source: §3 · p. 3, Table 1 · p. 4

4 METHODS

The section tests three explanations for the gap between dermatology vision encoders and end-to-end VLM diagnosis, then evaluates corresponding inference-time interventions. Across the ablation and cross-backbone evaluations, combining strategies performs best, while individual interventions show gains under specified settings.

4.1 HYPOTHESIS 1: TRAIN-TEST DISTRIBUTION MISMATCH

  • Class-aligned descriptions outperform length-matched generic text and descriptions shuffled across labels: 24.01%, 16.68%, and 12.71% on Fitzpatrick17k, and 52.23%, 36.36%, and 15.64% on PAD-UFES-20.The authors note that MedGemma’s exact training-data composition is not public; it can describe both BCC and LE despite zero-shot errors, motivating tests of class-aligned clinical context.Source: §4.1 · p. 4

4.2 HYPOTHESIS 2: UNDER-RELIANCE ON THE VISION BRANCH

  • When the primary lesion was masked, MedGemma failed to abstain in 88% of cases, often defaulting to BCC.Describe-then-decide raises abstention on the same 200 masked images from 12% to 72%; its heatmaps also show more lesion-focused attention qualitatively.Source: §4.2 · p. 5, Figure 3 · p. 6

4.3 HYPOTHESIS 3: MISALIGNMENT BETWEEN THE LANGUAGE AND VISION BRANCHES

  • MedGemma’s structured lesion descriptions score predominantly 4 or 5 out of 5 on self-evaluation, contrasting with hallucinated explanations during standard zero-shot diagnosis.The five criteria are location site, lesion type, shape/border, color, and texture; this contrast motivates encoder-assisted candidate selection.Source: §4.3 · p. 6
  • Top-5 to Top-1 uses an 8-shot vision-encoder linear probe to select five candidates, which the full VLM then reranks to one diagnosis.At 8 shots on Fitzpatrick17k it obtains 37.82% compared with 38.43% for the linear probe, and its benefit diminishes at 16 or more shots; the strategy also reduces options from 20 or more to five while retaining VLM reasoning.Source: §4.3 · p. 6

4.4 ABLATION STUDY AND COMBINED PERFORMANCE

  • The all-combined design consistently achieves the highest performance, substantially outperforming any single strategy across five datasets.Each of the three strategies also independently improves Top-1 accuracy over direct answering; the ablation uses an overlapping-class subset of SD-260 because descriptions for all 260 classes were infeasible.Source: Table 3 · p. 7, §4.4 · p. 7

4.5 GENERALIZATION TO OTHER VLMS FOR DERMATOLOGY

  • On SkinVL, the fine-tuning-free pipeline yields consistent gains over direct answering across multiple datasets, including Fitzpatrick17k, which was in its pre-training.SkinVL is LLaVA-based; its stricter context window led the authors to apply Top-5-to-Top-1 filtering first.Source: Table 4 · p. 7, §4.5 · p. 7
  • On InternVL3.5, Top-5-to-Top-1 improves over direct answering by 11.11 and 12.91 percentage points, respectively.Describe-then-decide improves over direct answering in all six model–dataset pairs, and candidate-assisted inference gives the highest accuracy among the three settings.Source: §4.5 · pp. 7–8, Table 5 · p. 8

5 MECHANISTIC ANALYSIS OF VISION ATTENTION

The analysis measures how much generation-time attention reaches vision tokens and finds that Describe-then-Decide increases this attention across medical modalities, especially at global-attention layers.

Vision Attention Ratio

  • Vision Attention Ratio is the fraction of generation-time attention allocated to vision tokens, averaged over generation steps and test images.The comparison contrasts direct classification with first describing the image and then classifying.Source: §5 · p. 8

DtD Increases Vision Attention Across Modalities

  • Describe-then-Decide raises attention most at global-attention layers 5, 10, 17, 22, 27, and 33 across modalities.The per-layer pattern is consistent across all six datasets.Source: §5 · pp. 8–9
  • Describe-then-Decide produces lower attention entropy than Direct Answer across all six modalities, indicating more focused and selective attention.The authors attribute increased visual attention to the description phase’s need to consult vision tokens, unlike direct classification that can use language priors alone.Source: §5 · p. 8
  • 30–40% higher mean Vision Attention Ratio under Describe-then-Decide than Direct Answer across six medical imaging datasets.On Fitz17k, the ratio rises from 2.85% to 3.98%, a relative gain of 40%.Source: §5 · p. 8, Figure 4 · p. 9

Generality Across Medical Modalities

  • Eight-shot linear probing of frozen MedSigLIP features outperforms zero-shot MedGemma on every additional medical-imaging dataset.The comparison extends the representation-level result beyond dermatology.Source: Table 6 · p. 9

6 CONCLUSION AND DISCUSSION

The paper combines frozen-VLM prompting and encoder-assisted candidate selection to improve dermatology classification and visual grounding, while calling for broader clinical validation.

6 CONCLUSION AND DISCUSSION

  • The combined inference strategies improve classification accuracy, reduce diagnostic bias, and enhance visual grounding while keeping the VLM frozen and using a lightweight probe for candidate selection.Intervention transfer is tested across SkinVL, InternVL3, InternVL3.5, and biomed-Qwen; cross-modality experiments extend representation and attention analyses.Source: §6 · p. 9
  • The detailed analysis centers on MedGemma-4B and dermatology; the complete pipeline still needs evaluation across specialties and prospective validation of clinical utility.Establishing causal visual reliance also requires additional controlled interventions.Source: §6 · p. 9

Appendix

  • The appendix provides additional baselines and prompt controls, other-backbone and VQA-RAD evaluations, and extended confusion-matrix and attention analyses.It also includes qualitative examples, implementation details, shot ablations, dataset statistics, and limitations.Source: Appendix · p. 14

A.1 ZERO-LABEL ENCODER COMPARISON

  • MedSigLIP exceeds MedGemma by an average of 10.26 percentage points across four datasets without labeled target images, parameter updates, or a fitted classifier.The comparison uses image–text similarity with the same candidate class names.Source: Table 7 · p. 14, §A.1 · p. 14

A.2 MULTIMODAL IMAGE IN-CONTEXT LEARNING

  • Eight-shot image in-context learning improves accuracy by 3.61 percentage points on average, but the eight-shot linear probe remains stronger on all four datasets.Reranking has the highest accuracy on Derm7pt, eSkinHealth, and PAD-UFES-20; on Fitzpatrick17k it is 0.61 points below the probe.Source: Table 8 · p. 14, §A.2 · p. 14

A.3 CLINICAL-DESCRIPTION CONTROLS

  • Class-aligned clinical descriptions perform best among four prompt-only settings on both datasets; generic text does not improve over class names alone, and shuffled descriptions reduce accuracy.The comparison supports a role for alignment between clinical information and candidate labels.Source: §A.3 · p. 14, Table 10 · p. 15

B ADDITIONAL BACKBONE EVALUATION

Evaluations on additional backbones test whether the dermatology inference interventions transfer beyond MedGemma.

B ADDITIONAL BACKBONE EVALUATION

  • The evaluation covers InternVL3-4B, InternVL3.5-4B, and biomed-Qwen2.5-VL-3B on Derm7pt and Fitzpatrick17k.InternVL3.5 uses the same settings as InternVL3.Source: B ADDITIONAL BACKBONE EVALUATION · p. 15
  • Describe-then-Decide improves direct-answer accuracy in all six evaluated backbone–dataset comparisons.Table 12 retains all six comparisons.Source: B ADDITIONAL BACKBONE EVALUATION · p. 15
  • Top-5-to-Top-1 produces the strongest result among the evaluated inference settings across the additional backbone comparisons.These results test intervention transfer across backbones.Source: B ADDITIONAL BACKBONE EVALUATION · p. 15

C CROSS-DOMAIN VQA AFTER TASK-SPECIFIC FINE-TUNING

The section reports a cross-domain VQA trade-off after dermatology adaptation and presents complementary confusion-matrix, heatmap, and attention analyses.

C CROSS-DOMAIN VQA AFTER TASK-SPECIFIC FINE-TUNING

E ATTENTION HEATMAP CONSTRUCTION

  • Label-token heatmaps measure image-token attention during diagnosis words, while all-tokens heatmaps measure it across the description and diagnosis.The examples show more lesion-focused attention under Describe-then-Decide; its all-tokens heatmap is broader as the model scans the image to enumerate attributes.Source: E ATTENTION HEATMAP CONSTRUCTION · p. 16

D FULL CONFUSION-MATRIX COMPARISON

F PER-LAYER VAR ANALYSIS ACROSS MODALITIES

  • Describe-then-Decide elevates per-layer Vision Attention Ratio at every layer across all six medical-imaging datasets, with particularly large gains at global-attention layers.The global layers are 5, 10, 17, 22, 27, and 33; Table 15 provides numerical mean ratios and relative improvements.Source: F PER-LAYER VAR ANALYSIS ACROSS… · p. 17

G ATTENTION ENTROPY ANALYSIS

Describe-then-Decide (DtD) produces lower attention entropy than Direct Answer (DA) across six medical modalities, indicating more focused attention. The section also reports increased vision attention, more abstention on masked lesions, and supporting prompt and implementation details.

G ATTENTION ENTROPY ANALYSIS

  • Entropy reductions range from −0.08 on PathMNIST and BloodMNIST to −0.21 on OrganAMNIST under DtD versus DA across six modalities.OrganAMNIST decreases from 4.88 to 4.67, CheXpert from 4.81 to 4.61, and PatchCamelyon from 4.74 to 4.58.Source: G ATTENTION ENTROPY ANALYSIS · p. 17, Table 14 · p. 18
  • DtD increases vision attention across all six datasets and modalities, with the strongest gains at global attention layers.The entropy reductions are distributed across layers and are most pronounced at global-attention layers that also show the largest VAR increases.Source: G ATTENTION ENTROPY ANALYSIS · pp. 17–18, Table 15 · p. 18, Figure 8 · p. 19

H.1 ABSTENTION TEST ON MASKED IMAGES

  • On 200 lesion-masked test images, DA failed to abstain in 88% of cases, while DtD produced a 72% abstention rate.Under DA, the model defaulted to a BCC diagnosis; the figure contrasts this with more frequent abstention under DtD.Source: §H.1 · pp. 19–20, Figure 9 · p. 20

H.2 DESCRIPTION QUALITY EXAMPLE

  • A MedGemma-4B example shows a detailed, clinically accurate description of lesion attributes before diagnosis, unlike the hallucinated reasoning noted for standard zero-shot classification.The example is presented as a description-quality illustration, not a quantitative evaluation.Source: §H.2 · p. 20, Figure 11 · p. 20

H.3 ZERO-SHOT PROMPT TEMPLATES

I.1 LORA FINE-TUNING

  • LoRA fine-tuning used QLoRA with a selected rank of 8, alpha of 16, and learning rate of 1e−4 for 10 epochs.The model was loaded in 4-bit precision, and the selected configuration was chosen based on validation performance.Source: §I.1 · p. 21, Table 17 · p. 21

I.2 LINEAR PROBE

  • The linear probe uses the mean of the vision encoder’s last hidden-state tokens as the global image representation and trains a head for 100 epochs.The head uses AdamW and CrossEntropyLoss with batch size 128.Source: §I.2 · p. 21, Table 19 · p. 22

I.3 VISION ATTENTION RATIO COMPUTATION

  • VAR is computed from attention weights across all 34 MedGemma-4B transformer layers, averaged over heads and generation steps.The analysis samples 100 test images per dataset under both DA and DtD prompting.Source: §I.3 · pp. 22–23

I.4 ABLATION STUDY ON THE NUMBER OF SHOTS

The 8-shot setting is selected for encoder-assisted reranking because the linear probe retrieves the correct diagnosis in its Top-5 without yet being consistently accurate at Top-1. In the low-data regime, the two-stage approach outperforms direct linear-probe Top-1 accuracy.

I.4 ABLATION STUDY ON THE NUMBER OF SHOTS

  • At 8 shots, the encoder captures the correct diagnosis in its Top-5 but is not yet perfect at Top-1, making this the selected setting for VLM reranking.Beyond 8 shots, the encoder becomes self-sufficient and the added value of VLM reasoning diminishes.Source: §I.4 · p. 23
  • In the low-data regime, the Top-5-to-Top-1 strategy consistently outperforms direct linear-probe Top-1 accuracy.The strategy uses a k-shot linear probe to select five candidates, then the VLM makes the final prediction.Source: Table 20 · p. 23

J PERFORMANCE ANALYSIS ON SKINVL

On SkinVL, a dermatology VLM built on LLaVA, the frozen-encoder linear probe outperforms both zero-shot inference and full fine-tuning across three datasets. This pattern extends the reported vision–language performance gap beyond MedGemma.

J PERFORMANCE ANALYSIS ON SKINVL

  • On Fitzpatrick17k, SkinVL’s linear probe reaches 58.88% versus 10.05% for zero-shot, and the linear probe outperforms zero-shot and full fine-tuning on all three datasets.The passage describes the largest gap as occurring on Fitzpatrick17k.Source: J PERFORMANCE ANALYSIS ON SKINVL · p. 23, Table 21 · p. 23

K LIMITATIONS

The study’s main analysis is centered on dermatology, while other specialties extend only the representation and attention analyses. The authors identify broader pipeline evaluation and prospective clinical-utility assessment as remaining work.

K LIMITATIONS

  • The complete pipeline remains unevaluated across radiology, pathology, hematology, and CT; those specialties currently extend only representation and attention analyses.The main analysis covers five dermatology datasets, with additional backbone comparisons on Derm7pt and Fitzpatrick17k.Source: K LIMITATIONS · pp. 23–24
  • Prompt-only interventions use no target labels, but Top-5-to-Top-1 reranking requires a labeled support set and a fitted linear probe.Source: K LIMITATIONS · p. 24
  • Description quality is measured through model self-evaluation, and independently verified description errors remain a target for further analysis.Source: K LIMITATIONS · p. 24
  • Evaluating real-world deployment requires a clinical-utility study with multiple experts and inter-rater reliability analysis.Source: K LIMITATIONS · p. 24

Related papers

Loading 2609.36557v1…