PaperlayerView exact revision on arXiv

Source-linked AI summary

Ranking-Aware Prompt Optimization for Multimodal Clinical Diagnosis

Tian Xia, Minghao Liu, Yiqing Liang, Laixi Shi, Jiayun Wang

arXiv:2609.40361v1cs.LGcs.CLcs.CV

TL;DR

Accuracy-based objectives can leave MLLM clinical diagnosis poorly aligned with ranking under class imbalance. The paper introduces pair-level Pareto prompt evolution, which aligns prompt search with AUROC; it reports gains over accuracy-based prompt evolution across two open-source model families and three MIMIC disease tasks.

  • Problem

    Under heavy clinical class imbalance, accuracy-based adaptation can leave positive-negative ranking quality unassessed, even when accuracy is high.

    Source: §1 · p. 1

  • Method

    Ranking-PE replaces instance-correctness events with positive-negative ordering events and aligns the scores matrix, reflector feedback, and final selection with empirical AUROC.

    Source: §1 · p. 2, §4.2 · p. 4

  • Results

    Ranking-PE beats Accuracy-PE on AUROC across two open-source MLLM families and three MIMIC disease tasks, with gains of +5.8 AUROC pp on Qwen3-VL-8B + SFT (VE-tuned) and +16.2 AUROC pp on MedGemma-4B.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The reported recipe pairs Ranking-PE with a medical-grade visual backbone obtained through vision-encoder-tuned SFT or medical pretraining.

    Source: §6 · p. 9

    The evaluation covers three CheXpert pathologies on one cohort and is not validated for clinical use or other populations, imaging modalities, or care settings.

    Source: ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

Multimodal large language models (MLLMs) are rapidly advancing clinical diagnosis, yet their adaptation pipelines remain anchored to accuracy-based objectives. Clinical data are heavily class-imbalanced: a constant-majority predictor can score above 90% accuracy while being clinically useless. We therefore evaluate and optimize for AUROC, a threshold-free score that ranks positives above negatives and is invariant to class balance. We focus on prompt optimization in MLLMs. Reflective methods such as GEPA use a binary scores matrix with one row per evaluation instance and one column per candidate prompt; cells record per-instance correctness, so the column average is accuracy and drives candidate selection. We introduce pair-level Pareto prompt evolution (Ranking-PE), which replaces each correctness row with a pairwise-ordering row over (positive, negative) instance pairs: the cell is 1 if the candidate scores the positive higher than the paired negative. The column average then equals empirical AUROC (by the Wilcoxon-Mann-Whitney identity). We apply this swap at all three layers the prompt evolution search reads from - the scores matrix that decides Pareto dominance, the per-example feedback to the reflection LM, and final candidate selection - at no extra model calls and with no surrogate loss. Across three diseases on MIMIC, accuracy-based prompt evolution can degrade ranking; Ranking-PE reverses this, beating the accuracy-based recipe by +5.8 AUROC pp on fine-tuned Qwen3-VL-8B and +16.2 pp on MedGemma-4B. Ablations examine each design component and show that a medical-grade visual backbone - via vision-encoder-tuned SFT or medical pretraining - is a prerequisite that prompt search cannot replace - our recipe extends reflective prompt evolution from text-only data to multimodal clinical decision-making.

1 INTRODUCTION

Clinical diagnosis requires ranking-aware rather than accuracy-only optimization because class imbalance can make high accuracy clinically uninformative. The paper introduces Ranking-PE and reports improved AUROC over accuracy-based prompt evolution across two MLLM families.

1 INTRODUCTION

  • Under severe class imbalance, a constant-majority predictor can exceed 90% accuracy without distinguishing positive from negative cases, while AUROC evaluates ranking across thresholds.The paper notes that missed cases and false alarms can have asymmetric clinical costs, which accuracy cannot represent; AUROC itself weighs both error types symmetrically.Source: §1 · pp. 1–2
  • Ranking-PE replaces instance-correctness events with positive-negative ordering events, making the scores-matrix average empirical AUROC and aligning search decisions with ranking.The paper applies this change to the scores matrix, reflector feedback, and final selection.Source: §1 · p. 2
  • +5.8 AUROC pp on Qwen3-VL-8B + SFT (VE-tuned) and +16.2 AUROC pp on MedGemma-4B versus Accuracy-PE, across three MIMIC disease tasks.The study also reports that Ranking-PE exceeds the corresponding base model in Balanced Accuracy.Source: §1 · p. 2
  • The study extends reflective prompt evolution to multimodal clinical diagnosis and identifies visual-domain adaptation as an important foundation for prompt evolution.The evaluation covers three diseases and two open-source MLLM families on MIMIC.Source: §1 · pp. 2–3

2 RELATED WORK

Prior work spans medical MLLM adaptation, direct ranking optimization, and prompt evolution, but their objectives and optimizer requirements differ. This paper positions pairwise ordering as a direct ranking objective compatible with reflective prompt search.

2 RELATED WORK

  • Medical MLLM adaptation commonly relies on downstream accuracy, which leaves positive-negative rank quality unassessed under heavy class imbalance.The paper contrasts this with AUROC optimization and says base SFT alone is limited on image-dependent diseases.Source: §2 · p. 3
  • Gradient-based ranking methods smooth pairwise indicators into surrogates, whereas whether smoothing is needed depends on the optimizer class.The cited examples include RankNet, LambdaRank, and medical-imaging AUC maximization.Source: §2 · p. 3
  • Reflective prompt-evolution methods consume binary per-instance signals directly, and pairwise ordering aligns their scores matrix with an unrelaxed ranking objective.The passage identifies APO, DSPy [Khattab et al., 2023], Prompt-Breeder [Fernando et al., 2023], and GEPA [Agrawal et al., 2025] as related approaches.Source: §2 · p. 3

3 TASK: CLINICAL DIAGNOSIS AS RANKING

The paper formulates binary clinical diagnosis as ranking a positive case above a negative one, using AUROC as the threshold-free target. It derives a continuous score from the model’s Yes/No output and reports Balanced Accuracy as a deploy-time check.

3 TASK: CLINICAL DIAGNOSIS AS RANKING

  • The diagnosis input pairs a medical image with structured EHR context and a target disease, while the model emits a Yes/No label-token distribution.The EHR context can include demographics, imaging view, and laboratory values.Source: §3 · p. 3
  • A log-odds score from the Yes and No token probabilities supplies the continuous value needed for ranking without an additional model rollout.The same score feeds the scores matrix, per-example feedback, and final candidate selection.Source: §3 · p. 3
  • AUROC is the probability that a randomly drawn positive case is ranked above a randomly drawn negative case, avoiding commitment to a deployment threshold.The paper uses AUROC for both evaluation and prompt-evolution optimization, and reports Balanced Accuracy at the default decoder threshold as a deploy-time sanity check.Source: §3 · p. 4

4 METHOD: RANKING-AWARE PROMPT EVOLUTION

The method first strengthens the visual backbone through medical-domain SFT, then refines prompts by aligning Pareto scoring, reflector feedback, and final selection with AUROC. Pairwise-ordering rows make the matrix column mean empirical AUROC, while ranking-shaped feedback adds clinical error, confidence, and cross-example rank signals.

4.1 SFT WITH VISION-ENCODER TUNING

  • Vision-encoder-tuned SFT yields higher post-evolution AUROC than frozen-encoder SFT on all three Qwen diseases; medically pretrained checkpoints can proceed directly to prompt evolution.SFT uses LoRA adapters on the language backbone and full vision-encoder weights.Source: §4.1 · p. 4

4.2 RANKING-AWARE PROMPT EVOLUTION

  • Reflective Pareto search ordinarily records per-instance correctness, making each candidate’s column average accuracy and steering selection toward accuracy.The search also uses the correctness matrix to filter Pareto-dominated candidates.Source: §4.2 · p. 4

4.2.1 PAIR-LEVEL PARETO SCORING

  • Pair-level rows score each positive-negative pair by ordering, with half credit for ties; their column mean equals validation AUROC and defines ranking-based Pareto dominance.This is the Wilcoxon–Mann–Whitney identity, so final selection maximizes validation AUROC.Source: §4.2.1 · p. 5
  • Pair scores require no additional model rollouts: they are table lookups from validation scores, and capped uniform row subsampling remains an unbiased AUROC estimator.The cap changes estimator variance, not the optimized quantity.Source: §4.2.1 · p. 6

4.2.2 RANKING-SHAPED PER-EXAMPLE FEEDBACK

  • Ranking-shaped reflector feedback combines clinical error type, confidence magnitude, and each example’s rank context against reference validation scores.For negative examples, rank context includes position among reference negatives and the fraction of reference positives at or below the current score.Source: §4.2.2 · p. 6

4.2.3 RANKING-PE PIPELINE

  • The full Ranking-PE configuration achieves the highest average AUROC among the ablated configurations; pair-level scoring improves over Accuracy-PE, with ranking-shaped feedback adding further gains.The three cumulative alignments are AUROC-based selection, pair-level Pareto scoring, and ranking-shaped feedback.Source: §4.2 · p. 5, §4.2.3 · p. 6

5 EXPERIMENTS

Across three MIMIC disease tasks, Ranking-PE improves AUROC over accuracy-based prompt evolution on both main backbones. Results also show that visual or medical adaptation provides an important foundation, while component ablations distinguish the contributions of pair-level scoring and ranking-shaped feedback.

5.1 SETUP

  • The evaluation uses MIMIC-CXR images linked to structured MIMIC-IV context for three independent disease tasks, with open-source Qwen3-VL-8B and MedGemma-4B as the main backbones.The task model and reflector receive the same multimodal inputs; Accuracy-PE and Ranking-PE share the SFT base, population size, rollout budget, and reflector.Source: §5.1 · pp. 6–7

5.2 RESULTS

  • Ranking-PE exceeds the three alternative-objective baselines in AUROC for every disease on both backbones, with average margins of 4.2 points on Qwen and 10.7 on MedGemma over the strongest alternatives.The strongest alternative is scalar-AUROC search on Qwen (69.6%) and balanced-accuracy selection on MedGemma (59.0%).Source: §5.2 · p. 7
  • 73.8% average AUROC on Qwen3-VL-8B + SFT (VE-tuned) and 69.7% on MedGemma-4B without SFT; Ranking-PE beats Accuracy-PE by 5.8 and 16.2 percentage points, respectively.Ranking-PE also reaches 68.6% and 60.6% average balanced accuracy on those settings, respectively.Source: §5.2 · p. 7, Table 1 · p. 8
  • Replacing the adapted Qwen self-reflector with GPT-5-nano or GPT-5-mini lowers average AUROC from 73.8% to 66.7% or 67.9%, respectively.Neither alternative matches task-adapted self-reflection on any of the three diseases in this setting.Source: §5.2 · p. 8
  • With Ranking-PE, Qwen average AUROC rises from 68.4% without SFT to 70.3% with VE-frozen SFT and 73.8% with VE-tuned SFT; MedGemma-4B reaches 69.7% versus 51.6% for matched-scale Gemma3-4B.Vision-encoder tuning adds 3.5 points over frozen-encoder SFT; the MedGemma comparison supports medical pretraining as another foundation for prompt evolution.Source: §5.2 · pp. 7–8, Table 2 · p. 8
  • For Qwen3-VL-8B + SFT (VE-tuned), changing final selection from accuracy to AUROC lowers average test AUROC from 68.0% to 61.3%; adding pair-level Pareto raises it to 70.7%, and ranking-shaped feedback reaches 73.8%.Replacing pair-level Pareto with an instance-level matrix while retaining ranking feedback yields 65.8% average AUROC.Source: §5.2 · p. 8, Table 3 · p. 9
  • On Consolidation, full ranking feedback reaches 79.3% AUROC and 68.5% balanced accuracy, versus 77.6% and 64.6% with correctness-only feedback.Clinical-error and confidence feedback configurations yield AUROC values of 78.2% and 79.8%, respectively.Source: §5.2 · p. 9
  • Across pair caps of 300, 1,000, 3,000, and 10,000, Cardiomegaly AUROC ranges from 70.6% to 71.3%, and every reported configuration exceeds Accuracy-PE on both metrics.The AUROC trajectory is non-monotone as the pair cap increases.Source: §5.2 · p. 9

6 CONCLUSION

The conclusion presents Ranking-PE as a way to align reflective prompt search with AUROC, and reports that it outperforms Accuracy-PE across the evaluated MIMIC tasks when paired with a medical-grade visual backbone. The paper’s scope is limited to MIMIC-IV, three diseases, and two open-source MLLM families; the appendix describes implementation and reproducibility materials.

6 CONCLUSION

  • The evaluation covers MIMIC-IV multimodal data, three diseases, and two open-source MLLM families; extensions to other modalities, languages, and label spaces remain open.The pair-level formulation is presented for binary AUROC, while multi-label and ordinal extensions are described as untested.Source: §6 · p. 9
  • The evaluation uses small patient-disjoint test sets with limited representation of the negative minority.Closed-source baseline comparisons use a three-level verbal score when top-k log probabilities are unavailable, and emphasis is placed on open-source models targeted by the recipe.Source: §6 · p. 9

APPENDIX

  • The appendix documents the multimodal pipeline, SFT and prompt-search hyperparameters, cohort construction, feedback strings, and seed and evolved prompts.MIMIC-IV cannot be redistributed under PhysioNet credentialing; code release is planned subject to internal review.Source: REPRODUCIBILITY STATEMENT · p. 10, APPENDIX · p. 13

A IMPLEMENTATION DETAILS

The appendix gathers implementation information deferred from the main methods section, spanning the multimodal interface, SFT, prompt evolution, and cohort construction.

A IMPLEMENTATION DETAILS

  • The appendix organizes implementation details into four parts: the multimodal interface, Stage 1 SFT hyperparameters, Stage 2 prompt-evolution hyperparameters, and cohort and split construction.These parts are identified as Appendix A.1 through A.4.Source: A IMPLEMENTATION DETAILS · p. 13
  • The multimodal interface documentation covers the bridge between the task model and reflection LM.It is presented in Appendix A.1.Source: A IMPLEMENTATION DETAILS · p. 13
  • The implementation appendix also documents SFT settings, prompt-search settings, and cohort and split construction.These are covered in Appendix A.2, A.3, and A.4, respectively.Source: A IMPLEMENTATION DETAILS · p. 13

A.1 MULTIMODALITY DESIGN AND IMPLEMENTATION

The multimodal implementation passes image and structured clinical context to the task model, and routes images into reflection examples so the reflection LM can access the visual evidence.

A.1 MULTIMODALITY DESIGN AND IMPLEMENTATION

  • The task module accepts image, clinical_profile, lab_context, and finding fields and returns an answer and reasoning.The answer is Yes or No; images are passed by file path and resolved inside the model server.Source: §A.1 · p. 13
  • Reflection examples embed images alongside text traces, predictions, and per-example feedback so the reflection LM retains access to visual evidence.Without image-aware routing, the reflection LM would receive only a textual surrogate.Source: §A.1 · p. 13
  • The task-model role is restricted to open-source MLLMs because clinical inputs may be sensitive, the recipe requires fine-tunable weights, and closed endpoints rarely expose the required log odds.The score is defined in the passage as log p(Yes) − log p(No).Source: §A.1 · p. 13
  • The model score uses answer-token log probabilities before the subsequently generated reasoning, with a coarse verbal fallback when required log probabilities are unavailable.The score is log p(Yes) − log p(No); the fallback assigns 1 for Yes, 0 for No, and 0.5 for an unparsable answer.Source: §A.1 · p. 13

A.2 STAGE 1: SFT HYPERPARAMETERS

The implementation uses a shared SFT recipe with vision-tower tuning as the variant, followed by Pareto-based prompt evolution. The appendix specifies search alternatives, ranking feedback, pair sampling, and computational costs.

A.2 STAGE 1: SFT HYPERPARAMETERS

  • The VE-tuned SFT variant unfreezes the vision tower, while the VE-frozen variant keeps it frozen; the best-AUROC validation step supplies the final checkpoint.The shared training recipe uses LoRA adapters and selects the checkpoint by eval_auroc.Source: §A.2 · pp. 13–14

A.3 STAGE 2: PROMPT-EVOLUTION HYPERPARAMETERS

  • Stage 2 uses reflective Pareto search with a candidate population targeted at 18, a reflection minibatch of 3, and validation column-mean argmax for final selection.The selected column mean is accuracy for Accuracy-PE and AUROC for Ranking-PE.Source: §A.3 · p. 14
  • 3,911 and 3,482 estimated task-model evaluations per run are reported on average for Accuracy-PE and Ranking-PE, respectively.These estimates exclude reflection calls and test evaluation; reported mean runtimes range from 39 to 61 minutes, and pair scores reuse validation predictions.Source: §A.3 · p. 14
  • Alternative search baselines vary selection or score weighting, while retaining binary correctness feedback; Scalar-AUROC PE instead assigns each candidate column the same AUROC across rows.BAcc-Select uses validation balanced accuracy for final selection, and Class-Weighted PE weights correctness inversely by class frequency.Source: §A.3 · p. 14
  • Pair-level search reuses validation-pass scores and samples up to 10,000 positive-negative pairs uniformly without replacement when the full set exceeds the cap.The selected pair set is shared across candidates and reused across iterations; Table 4 reports test AUROC and balanced accuracy for Cardiomegaly across pair caps.Source: §A.3 · p. 14, Table 4 · p. 14
  • Ranking-shaped feedback concatenates clinical error type, confidence magnitude, and cross-example rank context in a fixed order.The passage points to an appendix example and per-signal ablation.Source: §A.3 · p. 14

A.4 DATA CONSTRUCTION

The study constructs an admission-level MIMIC cohort linking chest X-rays with EHR context and CheXpert-derived disease labels. It selects three diseases spanning different prevalence levels and radiological cues.

A.4 DATA CONSTRUCTION

  • Each retained admission contributes its earliest AP or PA chest X-ray from 24–72 hours after admission, linked to MIMIC-IV records and CheXpert labels.Uncertain and missing labels are excluded independently for each disease; labels 1 and 0 map to Yes and No.Source: §A.4 · p. 15
  • Atelectasis, Cardiomegaly, and Consolidation were selected for benchmark comparability, distinct prevalence levels, and different dominant visual cues.Their combined-cohort positive prevalences are 96.2%, 78.7%, and 68.9%, respectively.Source: §A.4 · p. 15
  • Splits are patient-disjoint and label-stratified; Stage 2 training uses 500 examples, while validation and test sets remain full.Test-set positive prevalences are 96.6%, 74.2%, and 56.9% for Atelectasis, Cardiomegaly, and Consolidation, respectively.Source: §A.4 · p. 15

B ADDITIONAL RESULTS AND ABLATIONS

The appendix reports variability and expands analyses of SFT staging, reflector choice, prompt transfer, and AUPRC. These analyses contextualize the main results and provide additional per-disease reporting.

B ADDITIONAL RESULTS AND ABLATIONS

  • The appendix provides SFT-stage details, a reflector-substitution probe, and a cross-model prompt-transfer probe alongside the main-text foundation analysis.Table 7 covers SFT-stage detail, Table 8 the reflector probe, and Table 9 prompt transfer.Source: B ADDITIONAL RESULTS AND ABLATIONS · p. 15

B.1 VARIATION ACROSS RUNS

  • Tables 5 and 6 report per-disease AUROC and balanced accuracy as mean ± SD; aggregate averages are unweighted disease means.Table 5 does not report aggregate SD.Source: Table 5 · p. 15, Table 6 · p. 16

B.2 SFT STAGING DETAIL

  • With Ranking-PE, Qwen average AUROC increases at each SFT stage, with vision-encoder tuning providing the largest increment.Table 7 reports disease entries as mean ± SD and the unweighted disease-mean average.Source: Table 7 · p. 16

B.3 REFLECTOR SUBSTITUTION

  • Self-reflection outperforms both GPT reflectors on all three diseases in AUROC and balanced accuracy.Table 8 reports test AUROC and balanced accuracy, with an unweighted average across diseases.Source: Table 8 · p. 16

B.4 CROSS-MODEL PROMPT TRANSFER

  • On average, a prompt transferred from MedGemma-4B to Qwen3-VL-8B + SFT beats Accuracy-PE on the target model without further prompt optimization.The table reports test AUROC and balanced accuracy across the three diseases.Source: Table 9 · p. 16

B.5 AUPRC BREAKDOWN

  • AUPRC is reported as a complement to AUROC, with test-set positive prevalence given for each disease.Table 10 reports test AUPRC by disease and an unweighted average of the disease means.Source: §B.5 · p. 17, Table 10 · p. 17

C RANKING-SHAPED FEEDBACK DETAILS

Ranking-shaped feedback combines error type, confidence magnitude, and cross-example rank context into feedback for the reflector. Its ablation improves Consolidation results, while the prompt examples illustrate different evolved reasoning styles.

C RANKING-SHAPED FEEDBACK DETAILS

  • The error signal labels outcomes TP, TN, FN, or FP; only false negatives receive a “clinically dangerous” annotation, and both error types get tactical nudges.Correct cases do not receive those nudges.Source: C RANKING-SHAPED FEEDBACK DETAILS · p. 17, Table 11 · p. 17
  • Confidence feedback pairs signed log-odds with low, moderate, high, or extremely high magnitude buckets; incorrect predictions with |s| ≥3 trigger an overconfidence flag.The magnitude thresholds are fixed to the observed experimental scale of sΦ.Source: C RANKING-SHAPED FEEDBACK DETAILS · p. 17
  • Rank context adds within-class rank position and cross-class threat fraction from cached validation scores, while feedback remains per-example.For a missed positive, Figure 3 illustrates how low ρ+ and high τ− flag ranking miscalibration.Source: C RANKING-SHAPED FEEDBACK DETAILS · pp. 17–18, Figure 3 · p. 18
  • Full feedback raises Consolidation test AUROC by 1.7 percentage points and balanced accuracy by 3.9 percentage points over Basic.The configurations add feedback signals cumulatively while retaining pair-level Pareto scoring and AUROC final selection.Source: C RANKING-SHAPED FEEDBACK DETAILS · p. 18, Table 12 · p. 18

D PROMPT EVOLUTION EXAMPLE

  • 0.7054 vs 0.6835 validation AUROC: the Ranking-PE-optimized and Accuracy-PE-optimized Cardiomegaly prompts, respectively, come from the same seed and identical infrastructure.Figure 4 describes the Ranking-PE prompt as deliberative, with calibration checks and two-sided error remedies, unlike Accuracy-PE’s hard-threshold flowchart.Source: Figure 4 · p. 19, D PROMPT EVOLUTION EXAMPLE · p. 20, Figure 5 · p. 21

Related papers

Loading 2609.40361v1…