Source-linked AI summary
When Does RL Help Medical VLMs? Disentangling Vision, SFT, and RL Gains
Ahmadreza Jeddi, Kimia Shaban, Negin Baghbanzadeh, Natasha Sharan, Abhishek Moturu, Elham Dolatabadi, Babak Taati
TL;DR
The paper asks whether RL improves medical visual reasoning or mainly sharpens behavior already supported by medical VLMs. It disentangles vision, SFT, and RL with controlled MedMNIST evaluations, then proposes and validates a support-first post-training recipe. RL mainly improves Accuracy@1 when baseline support is non-trivial, while SFT expands support and makes RL more effective.
Problem
It remains unclear whether RL improves medical visual reasoning or mainly sharpens behaviors already induced by SFT, especially across modalities.
Method
The study uses MedMNIST-v2 to probe frozen vision towers, compare Accuracy@1 with Pass@K, test RL from base and SFT-initialized models, and derive a boundary-aware recipe.
Results
RL mainly sharpens the output distribution, improving Accuracy@1 when baseline support is already non-trivial, while medical SFT improves perception and support and RL offers limited benefits under larger shifts.
Takeaways & Limitations
Diagnose support first, bridge weak-support regimes with targeted finetuning, and apply RL afterward to improve sampling efficiency.
Takeaways & Limitations
Downstream gains are bounded when a frozen vision encoder cannot linearly separate classes for a dataset.
Abstract
from arXiv · showhide
Reinforcement learning (RL) is increasingly used to post-train medical Vision-Language Models (VLMs), yet it remains unclear whether RL improves medical visual reasoning or mainly sharpens behaviors already induced by supervised fine-tuning (SFT). We present a controlled study that disentangles these effects along three axes: vision, SFT, and RL. Using MedMNIST as a multi-modality testbed, we probe visual perception by benchmarking VLM vision towers against vision-only baselines, quantify reasoning support and sampling efficiency via Accuracy@1 versus Pass@K, and evaluate when RL closes the support gap and how gains transfer across modalities. We find that RL is most effective when the model already has non-trivial support (high Pass@K): it primarily sharpens the output distribution, improving Acc@1 and sampling efficiency, while SFT expands support and makes RL effective. Based on these findings, we propose a boundary-aware recipe and instantiate it by RL post-training an OctoMed-initialized model on a small, balanced subset of PMC multiple-choice VQA, achieving strong average performance across six medical VQA benchmarks.
1 Introduction
Medical VLM post-training can improve clinical visual question answering, but reported gains vary across modalities and tasks. This study controls vision, SFT, and RL to determine what drives those gains and when RL helps.
- Medical VLMs support visual question answering, report assistance, and decision support, but clinical deployment also requires reliability, transparency, and robustness.
- Existing pipelines often apply limited SFT and/or RL to small medical subsets, producing uneven results across modalities and inconsistent cross-modality generalization.
- The study asks how much improvement comes from visual perception, language-side alignment, SFT, and RL, and under what conditions RL is worthwhile.
- The study probes visual perception, compares Accuracy@1 with Pass@K, evaluates RL across modality transfers, and validates a staged support-first recipe on medical VQA benchmarks.
- MedMNIST-v2 is used as a controlled multi-modality testbed to disentangle vision, SFT, and RL in medical VLM post-training.
2 Related Work
Prior medical VLM pipelines commonly combine SFT and RLVR, but differing data, rewards, and modality coverage obscure which stage drives improvement. This work instead diagnoses support boundaries and treats RL as conditional sharpening.
- Medical VLMs typically undergo multimodal pre-training followed by SFT on medical image–text, VQA, or report-style data.
- RLVR/GRPO-style methods are increasingly used to encourage structured reasoning traces and improve answer reliability under limited supervision.
- Differences in data mixtures, reward designs, and modality coverage make it difficult to isolate what drives improvements or predict when RL will help.
- The paper diagnoses RL usefulness with Accuracy@1 versus Pass@K and modality-wise perception probing rather than assuming the standard SFT→RL pipeline is universally effective.
- Its support-first recipe raises Pass@K through targeted bridging when needed, then applies RL as a sharpening stage.
3 Disentangling Vision, SFT, and RL in Medical VLMs
Controlled MedMNIST-v2 evaluations separate visual representation quality, latent answer support, and RL’s effects across transfer regimes. The results indicate that SFT expands support, whereas RL mainly sharpens outputs when support is already adequate.
- 3 Disentangling Vision, SFT, and RL in Medical VLMs: MedMNIST-v2 covers three imaging modalities and twelve tasks at 224×224 resolution, enabling vision probing, VLM multiple-choice evaluation, and efficient RLVR experiments.
- 3 Disentangling Vision, SFT, and RL in Medical VLMs: The comparison uses Qwen2.5-VL-7B-Instruct as MBase, OctoMed as MSFT, and QoQ-Med as MRL.
- 3.1 RQ1: How Strong Are the Visual Representations in Medical VLMs?: Frozen-ViT linear probing tests whether medical VLM failures reflect visual representation bottlenecks, with MedViT-v2 as a vision-only reference.
- 3.1 RQ1: How Strong Are the Visual Representations in Medical VLMs?: The base model has reasonably separable visual features, SFT improves them especially on weaker datasets, and RL does not consistently improve ViT probe accuracy.
- 3.1 RQ1: How Strong Are the Visual Representations in Medical VLMs?: Several datasets remain below the MedViT-v2 reference, indicating perception bottlenecks that cap downstream gains.
- 3.2 RQ2: What Is the Reasoning-Capacity of Medical VLMs?: Accuracy@1 can fall far below Pass@K, revealing correct answers in the model’s support that greedy decoding fails to realize.
- 3.2 RQ2: What Is the Reasoning-Capacity of Medical VLMs?: Medical SFT raises both Accuracy@1 and Pass@K, while the RL-post-trained baseline does not consistently improve Accuracy@1 and often reduces Pass@K.
- 3.3 RQ3: When Does RL Help Medical VLMs?: RL sharpens outputs most effectively with non-trivial support, improving Acc@1 and narrowing the Acc@1–Pass@K gap in-domain and under small within-modality shifts.
4 From Analysis to Practice: A Recipe for RL Post-Training
The paper turns its support-boundary analysis into a staged post-training rule: bridge weak support with targeted finetuning, then use RL to sharpen sampling when support is sufficient. Applied to an OctoMed-initialized model, this recipe improves average performance across six medical VQA benchmarks.
- Support and sharpening: The decision rule estimates latent support and greedy accuracy, using their difference as the support gap.S_K(D) denotes Pass@K, A(D) denotes Acc@1, and G_K(D)=S_K(D)−A(D).
- Boundary-aware post-training: If support is below the target threshold, targeted task- or modality-proximal SFT prioritizes expanding coverage rather than sharpening outputs.The recipe diagnoses S_K(D) against τ and bridges weak-support regimes before applying RL.
- Boundary-aware post-training: Once support reaches the threshold, RL improves sampling efficiency by increasing greedy accuracy while monitoring support and the support gap.The procedure warns against over-sharpening that collapses support.
- Boundary-aware post-training: The recipe matches the observed pattern: bridging increases support, whereas RL mainly converts existing support into higher Accuracy@1.This distinction motivates sequencing SFT or continued training before RL when support is weak.
- Instantiation: RL post-training starts from OctoMed-7B and uses 8,000 balanced PMC-VQA multiple-choice questions spanning common clinical imaging categories.The balanced subset is sampled from PMC-VQA’s training split.
- Evaluation on medical VQA: Across six multimodal medical benchmarks, the resulting model achieves the strongest average performance among listed Qwen2.5-VL-based medical VLM baselines under the same decoding setting.The reported pattern is consistent with RL benefiting from an already-bridged checkpoint.
5 Conclusion
The study separates vision, medical SFT, and RL effects, finding that SFT improves perception and support while RL mainly sharpens output distributions when baseline support is already non-trivial. These findings motivate a boundary-aware sequence of diagnosing support, bridging weak regimes, and then applying RL.
- 5 Conclusion: The study disentangles the effects of vision, medical SFT, and RL on medical VLM post-training.
- 5 Conclusion: Medical SFT reliably improves perception and support, whereas RL mainly sharpens output distributions and improves Accuracy@1 when baseline support is non-trivial.RL offers limited benefits under larger distribution shifts.
- 5 Conclusion: The boundary-aware recipe diagnoses support with Pass@K, bridges weak-support regimes with targeted finetuning, and then applies RL to improve sampling efficiency.