PaperlayerView exact revision on arXiv

Source-linked AI summary

Aligning Medical Vision Language Models at the Capability Boundary with Entropy Adaptive Rubric Grounding

Hongyu Bai, Jiaqi Zeng, Chunzheng Zhu

arXiv:2610.39835v1cs.CVcs.AI

TL;DR

Medical diagnostic reasoning needs verifiable links between visual evidence and clinical knowledge, but current alignment objectives inadequately address this need. MedCBR couples boundary-aware preference alignment with entropy-adaptive rubric-grounded reinforcement learning. Across six benchmarks it achieves the best performance, reaching 61.2% average accuracy.

  • Problem

    Current medical VLM alignment does not adequately ensure verifiable links between visual evidence and pathological knowledge, while uniform SFT and correctness-only rewards inadequately address boundary learning and grounded reasoning.

    Source: §1 · p. 1

  • Method

    MedCBR aligns counterfactual preferences near the capability boundary, then applies entropy-adaptive clinical rubric rewards with KL regularization anchored to the boundary-aligned policy.

    Source: §3.1 · p. 3, §3.2 · p. 3, §3.3 · p. 5

  • Results

    MedCBR reaches 61.2% average accuracy, achieves the best performance on all six benchmarks, and outperforms V-RFT-PRA by +1.3 points.

    Source: §4.2 · p. 5

  • Takeaways & Limitations

    The results suggest clinically reliable multimodal alignment should jointly optimize boundary sensitivity and process-grounded supervision.

    Source: §5 · p. 7

    Compatibility of boundary alignment and clinical fidelity with deployment directions beyond current medical QA evaluation requires separate validation.

    Source: §2 · p. 3

Abstract

from arXiv · show

Medical diagnostic reasoning requires verifiable links between visual evidence and clinical knowledge, yet current medical vision language models are aligned with objectives that reward the wrong behavior. Uniform supervised fine tuning lets easy cases dominate updates while clinically critical boundary cases saturate prematurely. Sparse rewards based only on final-answer correctness cannot assess anatomical grounding or clinical logic, inviting reward hacking through answer shortcuts. We propose MedCBR, a two-stage alignment framework coupling boundary-aware preference optimization with rubric-grounded reinforcement learning. First, we construct counterfactual boundary preferences at the model's capability frontier using empirical pass rates and preference pairs that share reasoning prefixes but diverge at decisive diagnostic branches. Difficulty Weighted Boundary Preference Optimization (DBPO) concentrates learning on this calibrated boundary region. Second, we distill reusable clinical rubrics from boundary trajectories and activate them according to prediction entropy. Entropy Gated Grounded Policy Optimization (EGPO) rewards answer correctness, anatomical grounding, and reasoning fidelity while suppressing rubric noise on reliable samples. A reference policy anchored to the boundary-aligned model couples population-level boundary priors with instance-level uncertainty, making diagnostic reasoning more accurate and clinically grounded. Experiments on six medical benchmarks show consistent improvements over strong general, specialized, and reinforcement learning baselines, reaching 61.2% average accuracy with the largest gains on challenging differential diagnosis tasks. Ablations show super-additive gains: combining the two stages exceeds the sum of their individual improvements.

1 Introduction

Medical diagnostic reasoning requires verifiable links between visual evidence and pathological knowledge, but current alignment objectives do not adequately address this need. MedCBR focuses learning on uncertain capability-boundary cases and adds clinically grounded process feedback.

1 Introduction

  • Medical VLMs have advanced diagnostic assistance and can articulate diagnostic logic, but establishing verifiable causal chains from visual evidence to pathological knowledge remains inadequately addressed.The cited advances include reinforcement learning from verifiable rewards and chain-of-thought reasoning.Source: §1 · p. 1
  • Uniform SFT lets easy samples dominate gradients while loss on clinically critical boundary cases saturates prematurely.Answer-correctness-only rewards cannot assess anatomical grounding or valid differential logic, inviting reward hacking with potentially catastrophic clinical consequences.Source: §1 · p. 1
  • The framework targets uncertain capability-boundary decisions with dense clinical process feedback instead of sparse outcome signals.Its contributions include adaptive rubric selection and coupling population-level boundary priors with instance-level uncertainty.Source: §1 · p. 2
  • MedCBR combines counterfactual boundary preferences and difficulty-weighted preference optimization with entropy-driven rubric-grounded reinforcement learning.It anchors the reinforcement-learning stage to the boundary-aligned policy rather than the SFT policy.Source: §1 · p. 2
  • MedCBR consistently surpasses state-of-the-art methods across six medical VQA benchmarks, with its largest gains on challenging fine-grained differential diagnosis datasets.The benchmarks span radiology, pathology, and general medical knowledge.Source: §1 · p. 2

2 Related Work

Related work spans medical visual evidence representation, reasoning supervision, and structured feedback, alongside broader approaches to reliability and deployment. MedCBR distinguishes itself by aligning diagnostic branch points and adapting clinical rubric rewards to uncertainty.

2 Related Work

  • Medical VLM work combines instruction tuning, pretraining, curated knowledge, retrieval, and varied visual evidence representations; MedCBR instead aligns diagnostic branch points and adaptively evaluates clinical reasoning.Prior grounding interventions include semantic-stage rewards, counterfactual visual confidence, and image-caption consistency.Source: §2 · p. 2
  • Prior alignment and process-feedback work includes medical verifiable rewards, anatomy or rubric criteria, clinical consensus, and reusable memory or skill resources; MedCBR extracts clinical rubrics from boundary trajectories and gates their reward contribution.The cited literature also covers multidimensional evaluation and judge reliability.Source: §2 · p. 2
  • Related reliability approaches address sample selection, curricula, optimizer dynamics, and uncertainty, while MedCBR assumes verified answers and uses population pass rates and instance-level entropy.The passage distinguishes these approaches from MedCBR’s clinical boundary criterion.Source: §2 · p. 2
  • Compatibility of boundary alignment and clinical fidelity with deployment directions beyond the current medical QA evaluation requires separate validation.The related deployment work includes clinician-facing integration, resource-efficient inference, and agent orchestration.Source: §2 · pp. 2–3

3 Method

MedCBR aligns diagnostic reasoning by first targeting samples near the model’s capability boundary, then applying clinically grounded process supervision according to uncertainty. Its second stage uses rubric rewards and a reference policy anchored to the boundary-aligned model.

3.1 Problem Formulation

  • The objective combines diagnostic accuracy with clinically grounded reasoning, using process rewards alongside a KL constraint to a reference policy.The method frames alignment as deciding where to learn—near the capability frontier—and how to refine reasoning there.Source: §3.1 · p. 3

3.2 Stage 1: Counterfactual Boundary Preference Alignment

  • DBPO concentrates preference learning on uncertain boundary samples, pairing correct and incorrect trajectories that share reasoning prefixes and diverge at a critical diagnostic branch.The difficulty weight peaks at pass rate 0.5 and decays toward either extreme, shifting optimization away from saturated easy samples.Source: §3.2 · p. 3

3.3 Stage 2: Meta Rubric Grounded Policy Optimization

  • MRG distills reusable clinical rubrics from stage-one trajectories and gates their activation by prediction entropy, increasing rubric supervision for uncertain samples while limiting noise on reliable ones.The rubric library covers anatomical localization, sign description, differential reasoning, evidence consistency, and uncertainty calibration; its rewards combine with answer verification.Source: §3.3 · p. 4
  • EGPO uses group-normalized advantages and KL regularization anchored to the CBP policy, coupling population-level boundary priors with instance-level uncertainty.The paper describes the two stages as complementary and reports that their integration yields super-additive gains in the ablation.Source: §3.3 · p. 5

4 Experiments

Experiments evaluate MedCBR across six medical benchmarks using a constructed alignment corpus and shared backbone setup. The protocol compares general, medical, and reinforcement-learning baselines using accuracy.

4.1 Experimental Setup

  • Six benchmarks span radiology, pathology, and cross-domain reasoning, with accuracy measured on untouched official test sets.The training corpus combines benchmark training splits with de-identified teaching cases and expert-written differential-diagnosis questions, deduplicated against test images.Source: §4.1 · p. 5
  • Stage 1 boundary mining over ∼210K candidate queries retains the ∼62% whose pass rates fall within [τ_l, τ_u], yielding ∼48K counterfactual preference pairs; Stage 2 draws ∼12K on-policy rollout prompts, oversampling hard differential-diagnosis and multispecialty questions.Source: §4.1 · p. 5
  • The implementation trains CBP and MRG for 18 and 24 GPU hours, respectively, using 8×A100 80GB GPUs.Stage 1 uses N=16 trajectories per query; Stage 2 runs EGPO over approximately 12K rollouts.Source: §4.1 · p. 5
  • Comparisons include general and medical VLMs plus RL baselines, all using the Qwen2.5 VL 7B backbone and LoRA setup.Accuracy is averaged over three seeds, with paired bootstrap testing at p < 0.01.Source: §4.1 · p. 5

4.2 Main Results

MedCBR achieves the strongest reported accuracy across the six benchmarks, with larger gains on the more challenging tasks. The authors report that reasoning alignment is the primary bottleneck at the 7B scale.

4.2 Main Results

  • 61.2% average accuracy: MedCBR ranks best on all six benchmarks and beats V-RFT-PRA by +1.3 points.The reported gains are +1.9 on SLAKE and +1.6 on MMMU-Med.Source: §4.2 · p. 5
  • MedCBR also surpasses domain-specialized models that use large-scale medical pretraining or retrieval augmentation.The authors interpret this result as evidence that reasoning alignment, rather than domain knowledge acquisition, is the primary bottleneck at 7B scale.Source: §4.2 · p. 5
  • The reported larger gains on SLAKE and MMMU-Med are associated with tasks requiring multi-step reasoning and evidence grounding.Source: §4.2 · p. 5

4.3 Ablation study

The ablations report super-additive gains from combining CBP and MRG, with the synergy depending on the boundary-aligned reference policy. Component removals and static rubric application identify distinct contributions and tradeoffs.

4.3 Ablation study

  • The CBP–MRG combination gains +9.7 with Δsyn=+2.8, exceeding the sum of individual gains by 29%.CBP alone yields +6.4 over SFT and MRG alone +5.5.Source: §4.3 · pp. 5–6, Table 2 · p. 6
  • Anchoring KL to πsft instead of πcbp reduces synergy from +2.8 to +1.7, a 39% reduction.Synergy peaks at +4.6 on MMMU-Med and +3.0 on MedXpertQA.Source: §4.3 · p. 6
  • Removing boundary filtering costs 1.6 points, counterfactual mining 1.2, and difficulty weighting 0.5.Static all-rubric application reaches 59.7% and scores 73.2 on VQA-RAD versus 73.9 for V-RFT-PRA.Source: §4.3 · p. 6

4.4 Training data composition.

The two stages use data differently: boundary-filtered Stage 1 is radiology-skewed, while Stage 2 oversamples harder differential-diagnosis cases. Qualitative examples contrast baselines’ lengthy unsupported traces with MedCBR’s concise localization of decisive signs.

4.4 Training data composition.

  • Stage 1 processes about 48K boundary pairs and is radiology-skewed, while Stage 2’s roughly 12K rollouts oversample harder cases from 16.6% to 19.8%.Public benchmarks supply about 68% of Stage 1 and 52% of Stage 2 data; curated data covers differential-diagnosis scenarios underrepresented in public data.Source: §4.4 · p. 6, Figure 3 · p. 6

4.5 Qualitative case study

  • Across CT, MRI, and ultrasound cases, baselines produce 185–238-token traces with irrelevant structures and unsupported evidence, while MedCBR resolves cases in 34–44 tokens.Source: §4.5 · p. 6

4.4 Training data composition.

  • Figure 3 depicts a shift from radiology-dominated Stage 1 pairs toward oversampled differential-diagnosis cases in Stage 2, alongside super-additive combined gains.Its caption says the surplus Δsyn is largest on the hardest benchmarks.Source: Figure 3 · p. 6, §4.4 · p. 6

4.5 Qualitative case study

  • MedCBR names the lesion location and discriminative sign, the differential-reasoning and uncertainty dimensions where the baselines collapse.Source: §4.5 · p. 6

4.6 Training dynamics and rubric selection.

Training signals show a stronger Stage 2 start with CBP, declining entropy and rubric activation across stages, and positive benchmark results for entropy-driven selection.

4.6 Training dynamics and rubric selection.

  • About 0.80 composite reward for full MedCBR versus 0.38 for no-CBP; the ablation saturates and never recovers during Stage 2.The passage attributes the gap to the margins carried into Stage 2 by the CBP policy.Source: §4.6 · p. 6
  • Entropy falls across stages (u≈2.2 → 1.6 → 1.1) as mean activated rubrics decline from 4.2 to 2.8.Source: §4.6 · p. 6
  • Entropy-driven selection is the only policy positive on all six benchmarks, with +2.2 over static on MMMU-Med and 4.3 of 5 rubrics activated versus 2.1 on VQA-RAD.Source: §4.6 · p. 6
  • MedCBR trajectories grow to ∼3.0K tokens early, then compress to ∼2.6K, while the no-CBP ablation stays verbose at ∼3.2K, a reported hallmark of reward hacking.Source: §4.6 · p. 6

4.7 Efficiency and qualitative analysis.

MedCBR combines a modest training-time increase over V RFT PRA with the highest final accuracy, while radiologist ratings improve across all five rubric dimensions.

4.7 Efficiency and qualitative analysis.

  • MedCBR needs 42 total hours (18h CBP + 24h MRG), 5% above V RFT PRA while delivering the highest final accuracy.Source: §4.7 · p. 7
  • MedCBR yields 2.7× accuracy improvement per additional GPU hour versus V RFT PRA’s 2.0×, using 66 GB peak memory with zero inference overhead.Source: §4.7 · p. 7
  • On 200 MMMU-Med cases, three board-certified radiologists rated MedCBR above Qwen2.5 VL 7B on anatomy, sign, differential, evidence, and uncertainty.The largest margins were on differential exclusion and uncertainty calibration; inter-rater agreement was κ=0.78.Source: §4.7 · p. 7

4.8 Cross architecture generalization.

With identical hyperparameters, MedCBR improves results across three diverse VLM backbones, with consistent gains that are larger on the hardest benchmarks.

4.8 Cross architecture generalization.

  • MedCBR gains are 51.5→61.2 (Δ=+9.7) for Qwen2.5 VL 7B, 53.0→62.4 (Δ=+9.4) for InternVL2.5 8B, and 50.8→59.9 (Δ=+9.1) for LLaVA OneVision 7B.Source: §4.8 · p. 7
  • Gain standard deviation is 0.3 across the three backbones, and hardest-benchmark gains are Δhard=+9.9, +10.2, and +9.5, respectively.Source: §4.8 · p. 7
  • The authors attribute architecture independence to boundary estimation from empirical pass rates and rubric evaluation by an external frozen judge, making the recipe plug and play for medical VLMs supporting standard RL fine-tuning.Source: §4.8 · p. 7

5 Conclusion

The conclusion presents MedCBR as a framework coupling capability-boundary preference alignment with entropy-adaptive rubric grounding, reporting improved accuracy and reasoning quality across six benchmarks, especially for harder differential reasoning.

5 Conclusion

  • Across six benchmarks, MedCBR improves diagnostic accuracy and reasoning quality, especially on harder differential reasoning.The conclusion suggests clinically reliable multimodal alignment should jointly optimize boundary sensitivity and process-grounded supervision.Source: §5 · p. 7

Related papers

Loading 2610.39835v1…