PaperlayerView exact revision on arXiv

Source-linked AI summary

GMAI-VL-R1: Harnessing Reinforcement Learning for Multimodal Medical Reasoning

Yanzhou Su, Tianbin Li, Jiyao Liu, Chenglong Ma, Junzhi Ning, Cheng Tang, Sibo Ju, Jin Ye, Pengcheng Chen, Ming Hu, Shixiang Tang, Lihao Liu, Bin Fu, Wenqi Shao, Xiaowei Hu, Xiangwen Liao, Yuanfeng Ji, Junjun He

arXiv:2504.01886v1cs.CV

TL;DR

Existing medical multimodal models often lack the reasoning flexibility needed for complex clinical decision-making because they rely heavily on supervised memorization. GMAI-VL-R1 combines reinforcement learning tuning with synthesized Chain-of-Thought data, and it outperforms baseline and SFT systems across medical multimodal benchmarks. The results support reinforcement learning’s role in improving generalization, although medical VQA did not exhibit the clear self-validation “aha moments” seen in some other reasoning tasks.

  • Problem

    Existing medical multimodal models rely heavily on supervised fine-tuning and memorization, limiting reasoning and flexibility in novel or complex clinical scenarios.

  • Method

    GMAI-VL-R1 applies reinforcement learning tuning to a base vision-language model and uses 10,000 curated medical VQA pairs with filtered Chain-of-Thought reasoning data.

  • Results

    GMAI-VL-R1 consistently outperforms baseline and SFT models across medical multimodal tasks, including 57.33% accuracy on MMMU, a 2% improvement over baseline.

  • Takeaways & Limitations

    The experiments highlight reinforcement learning’s role in improving medical multimodal reasoning and generalization, particularly for diagnosis and visual question answering.

  • Takeaways & Limitations

    Medical visual question-answering experiments did not show the self-validation “aha moments” observed in some mathematical or coding reasoning tasks.

Abstract

from arXiv · show

Recent advances in general medical AI have made significant strides, but existing models often lack the reasoning capabilities needed for complex medical decision-making. This paper presents GMAI-VL-R1, a multimodal medical reasoning model enhanced by reinforcement learning (RL) to improve its reasoning abilities. Through iterative training, GMAI-VL-R1 optimizes decision-making, significantly boosting diagnostic accuracy and clinical support. We also develop a reasoning data synthesis method, generating step-by-step reasoning data via rejection sampling, which further enhances the model's generalization. Experimental results show that after RL training, GMAI-VL-R1 excels in tasks such as medical image diagnosis and visual question answering. While the model demonstrates basic memorization with supervised fine-tuning, RL is crucial for true generalization. Our work establishes new evaluation benchmarks and paves the way for future advancements in medical reasoning models. Code, data, and model will be released at \href{https://github.com/uni-medical/GMAI-VL-R1}{this link}.

1. Introduction

Existing medical multimodal models perform well on familiar tasks but often rely on memorization rather than deeper reasoning, limiting flexibility in complex clinical scenarios. GMAI-VL-R1 addresses this gap with reinforcement learning tuning, reasoning data synthesis, and evaluations showing stronger performance and generalization than SFT.

  • Motivation: Existing medical multimodal models often rely on supervised fine-tuning and input-output memorization, limiting reasoning in novel or complex clinical scenarios.Pattern recognition alone is described as inadequate for medical decision-making under complexity and uncertainty.
  • Approach: GMAI-VL-R1 applies reinforcement learning tuning directly to a base vision-language model, using accuracy, format, and redundancy rewards to optimize explicit reasoning and reflection.The procedure samples multiple reasoning responses and updates the policy through policy-gradient optimization.
  • Data: GMAI-Reasoning10K contains 10,000 curated medical VQA pairs from 95 datasets across 12 imaging modalities, with filtered Chain-of-Thought instructions for comparing RLT and SFT.GPT-4o generated reasoning chains that were refined through specialized filtering.
  • Results: Across six large-scale medical multimodal benchmarks, GMAI-VL-R1 consistently excels in question answering, disease diagnosis, and recognition, including familiar and previously unseen scenarios.The authors report stronger generalization than traditional SFT methods.

2. Related Work

Medical multimodal models remain constrained by limited data quality and training methods, especially for complex reasoning and decision-making. Related work motivates reinforcement learning and high-quality reasoning data as ways to improve generalization and flexibility.

  • Medical Vision-Language Models: Existing medical multimodal models commonly depend on specialized image-text datasets, but limited data quality can hinder generalization.Examples include models adapted with large medical image-text collections whose performance remains constrained by data quality or dataset limitations.
  • Reasoning Models: Medical multimodal systems must integrate images and text for decision-making, while current reasoning frameworks still face challenges on complex vision-language tasks.Chain-of-Thought decomposes reasoning into steps, and other approaches use intermediate supervision or supervised fine-tuning.
  • Reasoning Models: Supervised fine-tuning can improve reasoning structure and logic but remains limited on more complex tasks, motivating reinforcement-learning-based reasoning models.DeepSeek-R1 and related models are cited as examples of RL-based refinement for complex reasoning.
  • Reasoning Models: Combining supervised fine-tuning with reinforcement learning is described as strengthening reasoning structure while improving knowledge generalization and flexibility under limited data.The related-work discussion attributes this broader capability to reinforcement learning’s strategy adjustments.

3. Methodology

The methodology combines a curated multimodal medical VQA dataset with reinforcement learning tuning that samples, rewards, and iteratively optimizes reasoning outputs.

  • GMAI-Reasoning10K: GMAI-Reasoning10K contains 10,000 medical VQA pairs spanning 12 imaging modalities, with corresponding reasoning CoT annotations for method comparison.The data were collected from 95 datasets and include modalities such as X-ray, CT, and MRI.
  • GMAI-Reasoning10K: The dataset construction pipeline preprocesses multimodal data through augmentation, standardization, slice extraction for 3D scans, and frame extraction for videos.CT/MRI slices are normalized to 0–255, while video data are sampled at 2 frames per second.
  • GMAI-Reasoning10K: Detailed GPT-4o-generated chain-of-thought explanations are added to each VQA pair to support a fair comparison between reinforcement learning and supervised fine-tuning.The approach differs from methods that use only questions and answers as instructions.
  • Reinforcement Learning Tuning: The RLT pipeline applies GRPO to a base multimodal model, using reference and reinforcement-learning policies to generate and optimize medical reasoning responses.The procedure samples multiple outputs for each input, enabling diverse responses and relative policy evaluation.
  • Reinforcement Learning Tuning: Each sampled response receives accuracy, format, and repetition rewards, after which group-relative advantages guide stable policy updates with KL regularization.Format rewards require designated reasoning and answer tags, while repetition penalties discourage redundant outputs.
  • Reinforcement Learning Tuning: The iterative RLT process progressively updates model parameters until convergence, and a case study illustrates diagnosis selection from medical images using visible symptoms.RLT encourages explicit reasoning before selecting the correct multiple-choice answer.

4. Experiments

Across medical multimodal benchmarks, reinforcement learning tuning improves generalization beyond supervised fine-tuning, with gains strongest on out-of-distribution and reasoning-intensive tasks. Additional analyses show that performance depends on model scale, inference strategy, and training duration, while medical VQA does not exhibit language-model-style “aha moments.”

  • Main Results: SFT improves in-distribution OmniMedVQA from 58.41% to 66.34% but degrades GMAI-MMBench from 40.02% to 39.65% and from 40.59% to 36.12% on validation and test.RLT instead improves both in- and out-of-distribution performance, surpassing SFT on GMAI-MMBench and gaining +5.56% on MMMU-pro.
  • Main Results: 57.33% on MMMU, 34.03% on MMMU-pro, 43.14% and 43.84% on GMAI-MMBench, 23.80% on MedXpertQA-MM, and 61.01% on OmniMedVQA show strong Base+RL performance across benchmarks.The reported gains over the base model are +2.00%, +5.56%, +3.12%, +3.25%, +3.50%, and +2.60%, respectively.
  • Main Results: Using 10K QA pairs, Base+RL reaches 57.33% on MMMU versus HuatuoVision’s 50.30% and 43.14% and 43.84% on GMAI-MMBench versus MedDR’s 41.95% and 43.69%.The comparison uses a 7B base model and substantially fewer data resources than cited state-of-the-art systems.
  • More results: RLT improves disease diagnosis from 47.42% with Base+SFT to 50.64% with Base+RL, while counting drops 7.52% under SFT but rises 17% with RL.Surgical Instrument Recognition shows no significant RLT–SFT difference, indicating weaker benefits for simpler recognition tasks.
  • Main Results: Medical VQA responses lack the explicit self-correction “aha moments” observed in mathematical or coding tasks and instead use noticeably shorter reasoning chains.The authors attribute this difference to medical VQA’s more direct associations between visual cues and clinical concepts.
  • More results: CoT raises MMMU to 57.73% after RLT but lowers GMAI-MMBench to 43.14%, showing that overly long or repetitive reasoning can hurt final-answer accuracy.Direct generation after RLT reaches 54.67% on MMMU and 45.07% on GMAI-MMBench.
  • More results: After RLT, the 7B model improves by 2.40% on MMMU and 3.12% on GMAI-MMBench, whereas the 3B model improves 1.40 points on GMAI-MMBench but decreases 2.33% on MMMU.The results suggest stronger models benefit more from reinforcement signals, and ten RLT steps already increase MMMU and GMAI-MMBench by 0.67% and 1.06%.

5. Conclusion

GMAI-VL-R1 combines reinforcement learning tuning with multiagent reasoning-data synthesis to improve multimodal medical reasoning and generalization. It outperforms current state-of-the-art multimodal medical models on several benchmarks, especially medical image diagnosis and visual question answering.

  • The model uses reinforcement learning tuning to enhance reasoning and reflection, while optimizing decision-making through long-chain reasoning.
  • GMAI-VL-R1 outperforms current state-of-the-art multimodal medical models across several benchmark tasks, particularly medical image diagnosis and visual question answering.
  • Figure 5 compares answer-length distributions for Baseline, +SFT, and +RLT models using total and correct-answer counts.

A. Appendix Title

The appendix identifies the training settings used for GMAI-VL-R1’s supervised fine-tuning and reinforcement-learning stages.

  • Table 5 presents the training settings for GMAI-VL-R1’s Stage I supervised fine-tuning and Stage II reinforcement learning.

A.1. benchmarks

The experiments use benchmarks covering medical reasoning, multimodal integration, and fine-grained clinical understanding.

  • The MMMU Health & Medicine track evaluates reasoning in complex medical scenarios using university exams, quizzes, and textbooks.
  • MMMU-Pro Health & Medicine tests multimodal reasoning with filtered text-only questions, expanded answer options, and vision-only input.
  • OmniMedVQA evaluates recognition of fundamental medical imaging concepts through paired images and text, emphasizing cross-modal reasoning.
  • GMAI-MMBench assesses fine-grained object identification in complex clinical scenarios and long-context tasks.

A.2. system promot

The appendix contains system and prompt templates for constructing medical imaging questions and multiple-choice questions.

  • Table 6 provides the system prompt used in the experiments.
  • Table 7 provides a prompt template for constructing medical imaging multiple-choice questions.
  • Table 8 provides a prompt template for constructing medical imaging questions.
Loading 2504.01886v1…