PaperlayerView exact revision on arXiv

Source-linked AI summary

MedGR$^2$: Breaking the Data Barrier for Medical Reasoning via Generative Reward Learning

Weihai Zhi, Jiayan Guo, Shangyang Li

arXiv:2508.20549v2cs.LGcs.AI

TL;DR

Medical VLMs face scarce expert-annotated data, limiting generalization while making reliable RL rewards expensive to obtain. MedGR2 co-develops a multimodal data generator and reward model to create filtered training data for SFT and GRPO. It achieves state-of-the-art performance and strong parameter efficiency, although RL provides less benefit once sufficiently large amounts of high-quality data are available.

Abstract

from arXiv · show

The application of Vision-Language Models (VLMs) in medicine is critically hampered by the scarcity of high-quality, expert-annotated data. Supervised Fine-Tuning (SFT) on existing datasets often leads to poor generalization on unseen modalities and tasks, while Reinforcement Learning (RL), a promising alternative, is stymied by the lack of reliable reward signals in this data-scarce domain. To break this impasse, we introduce Generative Reward Learning for Medical Reasoning (MedGR$^2$), a novel framework that creates a self-improving virtuous cycle. MedGR$^2$ co-develops a data generator and a reward model, enabling the automated, continuous creation of high-quality, multi-modal medical data that serves as both a superior training source for SFT and RL. Our experiments demonstrate that SFT with MedGR$^2$-produced data already surpasses baselines trained on large-scale, human-curated datasets. Crucially, when leveraging this data for RL via Group Relative Policy Optimization (GRPO), our model achieves state-of-the-art cross-modality and cross-task generalization, significantly outperforming specialized RL-based methods. Furthermore, our compact model, empowered by MedGR$^2$, achieves performance competitive with foundation models possessing over 10 times more parameters. MedGR$^2$ presents a new paradigm for data-efficient learning in high-stakes domains, transforming the problem from data scarcity to data generation and unlocking the full potential of RL for building truly generalizable medical AI.

Introduction

Medical VLMs are constrained by scarce expert-annotated data, limiting generalization across modalities and tasks. MedGR2 addresses this bottleneck by generating and reward-filtering multimodal data for SFT and RL, achieving strong benchmark performance and parameter efficiency.

  • Scarce expert-annotated medical data, constrained by privacy, cost, and specialized expertise, limits coverage and generalization across unseen modalities and tasks.Source: Introduction · p. 1
  • MedGR2 reframes medical AI from data scarcity to data generation through a self-improving cycle coupling multimodal sample generation with reward-based quality filtering.Generated samples support SFT, while the resulting model provides a stronger initialization for downstream RL.Source: Introduction · pp. 1–2
  • The framework autonomously synthesizes high-quality reasoning data and establishes a virtuous cycle for medical understanding.Source: Introduction · p. 2
  • SFT on MedGR2-generated data already surpasses prior fine-tuned methods, while combining the data with RL improves performance further.Source: Introduction · p. 2
  • +19.7% absolute gain over the 72B-parameter Qwen2-VL-72B demonstrates MedGR2’s parameter efficiency.Source: Introduction · p. 2

Data Construction and Curation for Medical VLMs

Medical VLM dataset construction relies on semi-automated methods because manual annotation is expensive and labor-intensive. Existing pipelines reformat biomedical image-text pairs or mine instruction data using language models and heuristics.

Generative Models for Medical Data Synthesis

GANs and diffusion models can synthesize realistic medical images, but they do not produce the semantically coherent instruction-following multimodal pairs needed for medical reasoning. MedGR2 instead generates complete reasoning tasks and applies reward-guided filtering.

  • GANs and diffusion models augment pixel-level tasks such as classification and segmentation but fall short of producing reasoning-oriented multimodal pairs.Source: Generative Models for Medical Data… · p. 2
  • Medical reasoning requires data capturing clinical logic, procedural knowledge, and image-language alignment, which MedGR2 targets through complete reasoning tasks and reward-guided filtering.Source: Generative Models for Medical Data… · p. 2

Reinforcement Learning for VLM Generalization

SFT can encourage rote pattern learning rather than robust generalization, whereas RL has shown data-efficient reasoning benefits in general and medical VLMs. MedGR2 couples scalable instruction generation with reinforcement-based policy optimization in a dynamic generator–reward-model ecosystem.

  • SFT often encourages rote pattern learning over robust generalization, especially problematic for medical models facing novel modalities and diagnostic scenarios.Source: Reinforcement Learning for VLM… · p. 2
  • RL methods, including GRPO, have demonstrated data-efficient learning and improved cross-task or cross-modal performance in general and medical domains.Source: Reinforcement Learning for VLM… · p. 2
  • MedGR2 couples a Multimodal Generator Gθ and Reward Model Rϕ to support reward-filtered SFT followed by reinforcement-based optimization of a Reasoning Policy πψ.Source: Methodology · p. 2

Multimodal Generator Gθ: Engineering the Data Engine

The multimodal generator Gθ creates clinically meaningful, rationale-grounded VQA triplets and improves them through controllable prompting and reward-guided self-updating.

Multimodal Generator Gθ: Engineering the Data Engine

  • Gθ synthesizes diverse, clinically meaningful VQA triplets containing a medical image, instruction, and rationale-grounded answer.Unlike classical augmentation, it creates semantically novel instruction-answer pairs guided by domain priors and prompt structure.Source: Multimodal Generator Gθ: Engineering… · p. 2
  • Meta-cognitive prompting yields the highest-quality data among the Direct, Step-by-Step, and Meta-cognitive strategies.This strategy prompts reflection on question types and expected answer structure, producing more diverse and clinically aligned outputs.Source: Multimodal Generator Gθ: Engineering… · p. 2, Figure 1 · p. 3
  • The generator is periodically fine-tuned on high-reward examples filtered by Rϕ, making generated samples increasingly relevant and higher quality across iterations.Rϕ is a lightweight vision-language model with a regression head that scores candidate triplets on clinically relevant dimensions.Source: Multimodal Generator Gθ: Engineering… · p. 3, Reward Model Rϕ: The Co-Evolving Expert · p. 3
  • Rϕ provides automated multi-grade supervision for candidate quality rather than relying only on pairwise preferences.The reward dataset includes radiologist reports, lightly perturbed reports, incoherent answers, and irrelevant or hallucinated responses with scores from 10 to -6.Source: Reward Model Rϕ: The Co-Evolving Expert · p. 3
  • Continual retraining lets Rϕ become more discerning as the reasoning policy generates increasingly challenging tasks.The reward model is initialized from a small seed dataset and adapts to distribution shift and increasing task complexity.Source: Reward Model Rϕ: The Co-Evolving Expert · p. 3

Two-Stage Policy Optimization

MedGR2 trains the final reasoning policy in two stages: reward-filtered SFT provides a warm start, followed by GRPO reinforcement learning to broaden generalization beyond demonstrations.

Two-Stage Policy Optimization

  • The two-stage policy uses high-quality data filtered by Rϕ first for SFT, then refines πψ with GRPO reinforcement learning.This sequence combines base alignment with subsequent policy optimization for generalization.Source: Two-Stage Policy Optimization · p. 3, Stage 1: Warm Start via… · p. 4, Stage 2: Generalization through… · p. 4
  • Reward-filtered SFT mitigates RL cold start by anchoring πψ in coherent reasoning and correct formatting, stabilizing exploration and accelerating convergence.The policy is trained on Dgen with a next-token prediction objective.Source: Stage 1: Warm Start via… · p. 3
  • GRPO promotes exploration beyond demonstrations by optimizing a composite reward for reasoning quality and answer correctness.The reward combines semantic and clinical plausibility from Rϕ with an indicator enforcing alignment with the correct extracted answer entity.Source: Stage 2: Generalization through… · p. 4
  • The RL stage is intended to improve generalization to out-of-distribution scenarios and unforeseen clinical inputs.It avoids restricting the policy to imitation of a single ground-truth trajectory.Source: Stage 2: Generalization through… · p. 4

The Iterative MedGR2 Loop

MedGR2 operates as a closed, iterative loop in which generation, reward filtering, and policy improvement repeatedly inform one another.

The Iterative MedGR2 Loop

  • MedGR2 is organized as a closed iterative loop connecting candidate generation, reward evaluation, dataset growth, and policy optimization.The loop supports repeated interaction between the generator, reward model, and reasoning policy.Source: The Iterative MedGR2 Loop · p. 4

1. Generate: The generator G(t)

The first loop step generates a new batch of candidate samples, which the reward model evaluates so high-scoring samples can augment the training data.

1. Generate: The generator G(t)

  • The generator produces a new batch of candidate samples for evaluation by the reward model.High-scoring candidates are then added to Dgen for subsequent training.Source: §1 · p. 4

3. Train Policy: π(t)

The policy ψ is updated through a two-stage supervised fine-tuning and reinforcement-learning process.

  • The policy ψ is updated via a two-stage SFT+RL process.Source: §3 · p. 4

4. Evolve: π(t+1)

MedGR2 evaluates its self-improving training framework on OmniMedVQA through standard and cross-domain protocols. The full SFT+GRPO system achieves state-of-the-art accuracy, improves transfer across tasks and modalities, and remains effective with reward-filtered data, although RL benefits diminish when high-quality data is abundant.

RQ1: State-of-the-Art Generalization

RQ3: A Granular Analysis of Generalization and Its Origins

RQ1: State-of-the-Art Generalization

RQ3: A Granular Analysis of Generalization and Its Origins

  • Diagnosis training produces broad positive task transfer, while frequent MRI training transfers strongly to CT and Microscopy.These patterns indicate transfer beyond simple frequency matching.Source: Transferability Across Tasks and… · p. 7
  • MedGR2 corrects more baseline errors than it introduces across modalities, including substantial gains in X-Ray and OCT and improvements in Dermatology.The head-to-head analysis compares corrected errors with new errors.Source: Error Correction Breakdown · p. 7
Loading 2508.20549v2…