Med-R1: Reinforcement Learning for Generalizable Medical Reasoning in Vision-Language Models

Yuxiang Lai; Jike Zhong; Ming Li; Shitian Zhao; Yuheng Li; Konstantinos Psounis; Xiaofeng Yang

Source-linked AI summary

Med-R1: Reinforcement Learning for Generalizable Medical Reasoning in Vision-Language Models

Yuxiang Lai, Jike Zhong, Ming Li, Shitian Zhao, Yuheng Li, Konstantinos Psounis, Xiaofeng Yang

arXiv:2503.13939v5Mar 18, 2025cs.CV

TL;DR

Medical VLMs need clinically sound, generalizable reasoning despite limited expert annotations and weaknesses in SFT and CoT supervision. Med-R1 uses GRPO-based reinforcement learning and improves cross-modality and cross-task generalization, while No-Think and Think-After analyses show that reasoning quality and domain alignment matter more than reasoning length.

Problem
Medical VLMs must handle clinically sound, multi-step reasoning across diverse imaging modalities and tasks, while SFT and high-quality CoT annotations remain limited.
Method
Med-R1 applies GRPO-based reinforcement learning to post-train a vision-language model for medical reasoning without relying solely on static token-level supervision.
Results
Med-R1 improves average cross-modality accuracy by 29.94% over Qwen2-VL-2B and question-type generalization by 32.06%, surpassing larger VLMs including Qwen2-VL-72B.
Takeaways & Limitations
Medical reasoning performance depends more on reasoning quality, timing, and domain alignment than on producing longer explicit rationales.
Takeaways & Limitations
The frame-level VQA setting simplifies real-world volumetric and dynamic imaging, and limited high-quality medical reasoning data can make explicit reasoning introduce noise.

Abstract

from arXiv · show

Vision-language models (VLMs) have achieved impressive progress in natural image reasoning, yet their potential in medical imaging remains underexplored. Medical vision-language tasks demand precise understanding and clinically coherent answers, which are difficult to achieve due to the complexity of medical data and the scarcity of high-quality expert annotations. These challenges limit the effectiveness of conventional supervised fine-tuning (SFT) and Chain-of-Thought (CoT) strategies that work well in general domains. To address these challenges, we propose Med-R1, a reinforcement learning (RL)-enhanced vision-language model designed to improve generalization and reliability in medical reasoning. Built on the DeepSeek strategy, Med-R1 adopts Group Relative Policy Optimization (GRPO) to encourage reward-guided learning beyond static annotations. We comprehensively evaluate Med-R1 across eight distinct medical imaging modalities. Med-R1 achieves a 29.94% improvement in average accuracy over its base model Qwen2-VL-2B, and even outperforms Qwen2-VL-72B-a model with 36x more parameters. To assess cross-task generalization, we further evaluate Med-R1 on five question types. Med-R1 outperforms Qwen2-VL-2B by 32.06% in question-type generalization, also surpassing Qwen2-VL-72B. We further explore the thinking process in Med-R1, a crucial component for the success of Deepseek-R1. Our results show that omitting intermediate rationales (No-Thinking-Med-R1) not only improves in-domain and cross-domain generalization with less training, but also challenges the assumption that more reasoning always helps. These findings suggest that in medical VQA, it is not reasoning itself, but its quality and domain alignment, that determine effectiveness. Together, these results highlight that RL improves medical reasoning and generalization, enabling efficient and reliable VLMs for real-world deployment.

I. INTRODUCTION

Med-R1 applies reinforcement learning to medical VLMs to improve generalization and interpretability across modalities and clinical tasks. Its results indicate that explicit reasoning is not uniformly beneficial: reasoning quality, timing, and domain alignment matter more than length.

Motivation: Medical VLMs must integrate complex clinical evidence across imaging modalities and task types while producing clinically sound interpretations.Examples include combining lesion localization, morphology, and context for diagnosis.
Limitations of Existing Methods: SFT can encourage task-specific shortcut memorization, while scarce high-quality CoT annotations limit traceable and robust medical reasoning.These limitations contribute to black-box predictions and weak out-of-domain performance.
Med-R1: Med-R1 uses GRPO-based reinforcement learning to explore diverse reasoning strategies through reward signals without requiring explicit CoT annotations.GRPO stabilizes training through rule-based rewards and group-relative comparisons.
Results: 29.94% higher average generalization accuracy across modalities and 32.06% higher question-type generalization accuracy distinguish Med-R1 from Qwen2-VL-2B.Med-R1 also surpasses Qwen2-VL-72B in both evaluation settings.
Reasoning Strategies: No-Think improves cross-modality generalization, whereas Think can produce hallucinated rationales under domain shift.Think After preserves interpretability without sacrificing accuracy, offering a balance between reliability and explainability.
Contribution: Med-R1 studies rule-based RL for medical reasoning across eight imaging modalities and five clinical question types.The contribution targets modality-specific and cross-modality reasoning without token-level supervision.

II. RELATED WORKS

Prior medical VLMs largely rely on supervised adaptation and often generalize poorly beyond narrow datasets or radiology settings. Med-R1 extends rule-based reinforcement learning toward broader modality-agnostic medical reasoning.

General and Medical VLMs: General-purpose VLMs advance natural image-text understanding but struggle with domain-specific medical reasoning.Medical adaptations such as LLaVA-Med and Med-Flamingo use SFT on medical datasets.
General and Medical VLMs: Medical SFT models can perform effectively in-domain while overfitting to narrow corpora and lacking cross-modality or cross-task generalization.This motivates scalable, modality-agnostic adaptation with reinforcement learning.
Related RL Approaches: MedVLM-R1 applies GRPO-based RL in a single radiology setting, training on MRI and testing on CT and X-ray with roughly 600 samples.Its reported generalization is within radiology rather than across the broader evaluation considered here.
Med-R1: Med-R1 applies rule-based RL and GRPO to train a multimodal language model across eight medical imaging modalities.The study systematically assesses RL for medical reasoning and generalization.
Baseline Setup: The SFT baselines use Qwen2-VL-2B and Qwen2.5-VL-3 with one epoch of mixed-precision training and gradient accumulation.These implementation details define the supervised comparison setup.

A. Group Relative Policy Optimization (GRPO)

GRPO replaces value-function-based advantage estimation with group-relative reward comparisons and rule-based feedback. In Med-R1, rewards enforce output format and answer accuracy while KL regularization limits policy drift.

Efficiency: 50% more resource- and computation-efficient than PPO is the reported advantage of GRPO.This comparison is stated for the GRPO algorithm relative to PPO.
Objective: The GRPO objective combines clipped policy updates with a KL-divergence penalty relative to a frozen reference policy.The clipping threshold constrains updates, while KL regularization limits deviation from the reference model.
GRPO Mechanism: GRPO estimates advantages from normalized rewards across groups of responses sampled for the same question.Unlike PPO, it does not use a critic model or value function for advantage estimation.
GRPO Mechanism: GRPO uses fixed rule-based rewards rather than a learned reward model.The objective compares candidate responses within each sampled group.
Reward Design: Format reward gives a score of 1 when <think> and <answer> tags are correctly present in the response.This reward encourages structured output for readability.
Reward Design: Accuracy reward gives a score of 1 when the predicted answer matches the ground truth.For multiple-choice questions, responses beginning with the correct letter option are treated as correct.

B. No-Thinking Med-R1

No-Think RL removes explicit reasoning and format supervision, training Med-R1 to output only answers; the paper contrasts this strategy with prior findings and other post-training variants.

Motivation: Prior evidence that removing reasoning and format supervision can improve performance comes from structured classification tasks with simple output spaces.The paper distinguishes medical VQA because its inputs are multimodal and semantically complex.
No-Think RL: No-Think RL removes explicit reasoning and retains an accuracy reward based on matching extracted answers to ground-truth labels.The format constraint forces the model to generate only the answer.
Evaluation: Table I reports cross-modality accuracy across eight medical imaging modalities, with rows representing training modalities and columns representing test modalities.Darker cells indicate higher accuracy for corresponding training–test pairs in each column; the caption identifies Qwen2.5-VL-3B as the base model.

C. Think-after Med-R1

Think-After makes the answer prediction before generating a post-hoc rationale, separating decision and explanation to balance accuracy with interpretability.

Think-After protocol: Think-After first predicts a single-letter answer and then generates a step-by-step rationale for that decision.The protocol is designed for reasoning that physicians can review and validate.
Evaluation: Table IV compares performance across eight medical modalities against zero-shot, medical-domain, and supervised fine-tuning VLM baselines.The caption states that the GRPO-finetuned model outperforms these baseline groups while maintaining scalability.
Design rationale: Separating answer prediction from rationale generation reduces interference between reasoning and decision processes.The paper describes this as achieving a balanced trade-off between accuracy and interpretability.

A. Setup

The study evaluates medical VQA generalization across modalities and task types using OmniMedVQA, standardized accuracy averaging, and bootstrap confidence intervals.

Dataset: OmniMedVQA contains 82,059 images and 88,996 VQA pairs spanning eight imaging modalities and five VQA question types.The listed modalities include CT, MRI, X-Ray, Ultrasound, Dermoscopy, Fundus, OCT, and Microscopy.
Task setting: Cross-modality generalization trains on one of eight modalities and evaluates on the other seven.Cross-task generalization trains on one task type and evaluates on the other four.
Task setting: The evaluation focuses on VQA as a unified framework combining classification, grounding, and reasoning abilities.The paper presents VQA as a representative precursor to detection or captioning tasks.
Evaluation: Table V evaluates cross-task generalization across five clinical reasoning task types, while Table VI reports the corresponding No-Think results.Both tables use rows for training tasks and columns for test tasks; darker shading indicates stronger generalization.
Metric: Performance is measured with VQA choice accuracy, where the model selects the correct answer from K clinically validated options.Accuracy is defined using an indicator that equals 1 when the predicted answer matches the ground truth and 0 otherwise.
Evaluation protocol: The study computes 95% bootstrap confidence intervals using 10,000 resamplings of binary accuracy labels.Results report the mean together with the confidence-bound half-width.
Evaluation protocol: The reported fine-tuned VLM results use macro-averaged accuracy across all cross-modality training–testing pairs.This identical averaging protocol is used for SFT, GRPO+Think, GRPO+Think-After, and GRPO+No-Think.

B. Cross-Modality Generalization

Med-R1 generalizes across eight medical imaging modalities while outperforming larger and supervised-fine-tuned baselines. Its 2B-parameter model achieves broad cross-modality accuracy with substantial gains over Qwen2-VL-72B and SFT-tuned Qwen2-VL-2B.

Cross-modality results: 69.91% overall accuracy demonstrates Med-R1’s cross-modality performance across eight medical imaging modalities.The evaluation covers CT, MRI, Ultrasound, Dermoscopy, Fundus Photography, OCT, Microscopy, and X-ray.
Cross-modality results: 72.35% overall accuracy for X-ray-trained models is the highest reported modality-specific generalization score, followed by CT at 71.44% and MRI at 71.26%.Fundus Photography and Microscopy training produce lower overall scores of 67.67% and 67.54%.
Baseline comparisons: 1.86% higher overall accuracy than Qwen2-VL-72B shows Med-R1’s parameter efficiency despite a 36× parameter disparity.Med-R1 uses 2B parameters, whereas Qwen2-VL-72B is substantially larger.
Baseline comparisons: Med-R1 improves diagnostic performance over Qwen2-VL-72B in MRI and dermoscopy, reaching 71.67% versus 69.39% and 72.33% versus 65.31%, respectively.These comparisons are reported for the MRI and dermoscopy modalities.
Baseline comparisons: 15.84% accuracy gains over SFT-tuned Qwen2-VL-2B establish an advantage for GRPO fine-tuning.The reported overall accuracies are 69.91% for Med-R1 versus 54.07% for the SFT baseline.

C. Cross-Task Generalization

Med-R1 is evaluated for transfer across five clinical question types and compared with zero-shot and SFT baselines. Disease-diagnosis training transfers best overall, while lesion grading delivers stronger in-task performance but weaker transferability.

Generalization patterns: 81.64% overall accuracy makes disease-diagnosis training the strongest source of cross-task generalization.The paper attributes this transfer to disease diagnosis’s use of anatomical and pathological cues.
Generalization patterns: 86.24% in-task accuracy for lesion grading accompanies relatively lower transferability across tasks.The results indicate that lesion grading captures more specialized features.
No-Thinking comparison: Cross-task gains for No-Thinking-Med-R1 are mixed across training and test task pairs, unlike its consistently stronger cross-modality results.The figure encoding uses training tasks on the y-axis, test tasks on the x-axis, and red or blue cells for improvement or degradation.
Baseline comparisons: 74.64% versus 72.58% shows Med-R1 outperforming Qwen2-VL-72B in zero-shot generalization across the five clinical tasks.The comparison concerns overall generalization performance across the task set.

D. Analysis of No-Think

No-Thinking-Med-R1 strengthens in-domain performance and cross-modality generalization relative to Med-R1, but its cross-task effects vary. The results question whether explicit intermediate rationales are consistently beneficial in medical VQA.

Performance: No-Thinking-Med-R1 achieves stronger in-domain performance than Med-R1 across all reported settings.The result suggests that removing explicit reasoning generation can support more effective task-specific learning.
Generalization: No-Thinking-Med-R1 consistently outperforms Med-R1 in cross-modality settings.Cross-task results are more mixed, with gains in some tasks and declines in others.
Practical motivation: High-quality CoT supervision is described as prohibitively costly and often impractical in clinical domains.The paper presents this constraint as a practical motivation for RL without reasoning supervision.
Interpretation: Unsupervised rationales may become unreliable under domain shift, so reasoning does not necessarily improve medical performance as it does in general domains.The paper frames this as a domain-specific qualification rather than a universal claim about reasoning.

E. Analysis of Think-after

Think-After is designed to retain interpretable reasoning while preserving predictive accuracy. It converges faster than Think and receives higher accuracy rewards, while reader evaluations favor its reasoning–answer consistency.

Design objective: Think-After targets the dual objective of high accuracy and interpretable reasoning in medical VLMs.Its reasoning traces are intended for clinician review and validation.
Training behavior: Think-After converges faster and achieves higher accuracy rewards than Think.The comparison suggests that generating reasoning tokens before the answer may disrupt autoregressive generation and hinder optimization.
Performance trade-off: Think-After remains slightly below No-Think in performance, leaving additional contributors such as residual contextual coupling or reasoning-token noise as possibilities.These mechanisms are presented as possible explanations rather than established causes.
Reader study: Three experienced readers independently assessed 100 VQA samples for factual correctness and reasoning–answer consistency.The study covered diverse imaging modalities and reasoning types.
Reader study: Think-After achieved the highest reader agreement among the evaluated model variants.The supplied passages state this outcome for the reader study’s assessments.

G. Limitations and Future Work.

Med-R1 is evaluated in a simplified frame-level setting, while the authors identify limitations in volumetric and dynamic imaging, reasoning data, and explicit rationale quality. Future work targets richer medical inputs, clinically grounded reasoning supervision, and improved reasoning frameworks.

Scope limitations: Frame-level VQA simplifies real-world medical imaging, which often requires reasoning across CT and MRI slices or ultrasound time sequences.The authors also identify patient context and volumetric or multi-frame inputs as future extensions.
Reasoning supervision: Limited high-quality medical reasoning data may explain why No-Thinking models occasionally outperform reasoning-enabled variants.The paper suggests that scarce clinically faithful CoT annotations can make explicit reasoning introduce noise rather than insight.
Future directions: Future work should scale medically grounded CoT data and align RL rewards with clinically validated reasoning quality.The authors also propose extending Med-R1 to multi-frame or volumetric inputs and incorporating patient context.
Future directions: Med-R1 is presented as an initial step toward RL-enhanced medical VLMs, with reasoning quality and domain alignment emphasized over reasoning quantity.The conclusion frames the approach as a scalable path toward reliable and interpretable medical AI systems.