PaperlayerView exact revision on arXiv

Source-linked AI summary

GRASP: Group-Adaptive Hybrid RL-SFT Post-Training for Medical VQA without Reasoning Annotations

Wanqi Dong, Wanyue Dong, Jingze Ge, Mehul Motani

arXiv:2610.34561v1cs.CVcs.AI

TL;DR

Medical VQA must adapt models from scarce short answers without reasoning annotations while retaining OOD generalization. GRASP combines RL and gold-answer SFT on the same model-generated reasoning rollouts, and its trained models outperform compared objectives and VLMs on domain and OOD evaluations.

  • Problem

    Medical VQA data are scarce and provide short answers without reasoning annotations, while post-training must fit the training domain without eroding OOD generalization.

    Source: §1 · p. 1

  • Method

    GRASP combines RL and group-adaptive SFT on the same model-generated reasoning rollouts, supervising the gold answer after the model’s reasoning.

    Source: §1 · p. 2, §3.4 · p. 5

  • Results

    GRASP outperforms all baseline training objectives in-domain on three backbones and, on InternVL3-8B and Qwen3-VL-8B, surpasses all 10 compared VLMs on OOD open-ended questions.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    The results suggest that supervising the gold answer after the model’s own reasoning is an effective way to combine SFT and RL.

    Source: §7 · p. 9

    The training set covers four sources, contains few normal cases, and is restricted to answers of at most three words; fine-grained diagnosis remains hard, with every model below 13% judged on VQA-Med 2020.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Adapting general-purpose vision-language models (VLMs) to medical visual question answering (VQA) is valuable for helping clinicians read images and make diagnoses, but annotated data are scarce and provide only short answers, without reasoning annotations. Supervised fine-tuning (SFT) fits these labels but overfits to them and amplifies their biases, eroding out-of-domain (OOD) generalization. Reinforcement learning (RL) better preserves generalization but never uses the gold answer as a training target: GRPO-style algorithms learn from the contrast between correct and incorrect answers within a group of sampled responses, so all-correct or all-wrong groups give no correctness signal. Existing SFT-RL hybrids either apply SFT and RL in separate stages or switch between them per question, or need reasoning traces from experts or stronger models, which medical VQA data lack. We propose GRASP, which combines RL and SFT in every update using only short answers: RL lets the model explore its own reasoning, SFT teaches the gold answer after that reasoning to supply the direct supervision RL lacks, and their balance adapts to how well the model answers each question. Tuning only about 1% of the parameters with LoRA, GRASP outperforms all compared training objectives both in domain and on OOD open-ended questions across Qwen3-VL-8B, InternVL3-8B and Gemma 3-4B. On an OOD benchmark spanning eight imaging modalities, GRASP-trained InternVL3-8B and Qwen3-VL-8B beat all 10 compared general and medical VLMs in open-ended accuracy, including models trained on up to 2,600 times more data or with over 250 times more trainable parameters. We release MedTerse, a dataset of 12.9K short-answer questions, and our code at https://github.com/sea-comet/GRASP-MedVQA.

1 Introduction

Medical VQA needs open-ended adaptation from scarce short-answer data without reasoning annotations, while preserving generalization beyond the training domain. GRASP combines RL and SFT on the same model-generated reasoning rollouts and outperforms compared objectives and models on domain and OOD evaluations.

Motivation

  • Open-ended medical VQA offers no answer options, yet current medical VLMs get most questions wrong; scarce annotations provide only short answers and no reasoning traces.Models must fit the training domain without eroding what they know about images from other hospitals, devices, and modalities.Source: §1 · p. 1
  • SFT can overfit and amplify label biases, whereas RL methods may receive no correctness signal when every sampled answer is wrong.Training all parameters on millions of curated examples is also unaffordable for many medical research groups.Source: §1 · p. 1
  • Existing hybrids separate the signals by stage or question, or depend on expert reasoning traces that medical VQA datasets rarely provide.Bare-answer supervision can reduce accuracy, while gold injection can teach the policy to skip reasoning.Source: §1 · p. 2

Contribution

  • GRASP applies RL and SFT to the same rollouts, supervising the gold answer after the model’s own reasoning without reasoning annotations.Its group-adaptive weight concentrates supervision on questions the model gets wrong and declines as it learns to answer them.Source: §1 · p. 2

Results

  • 27.3% exact match versus at most 21.6%: GRASP-trained InternVL3-8B and Qwen3-VL-8B beat all 10 compared VLMs on OOD open-ended questions.They were tuned on 12.9K questions with about 1% of parameters, against models trained on up to 2,600 times more data or with over 250 times more trainable parameters.Source: §1 · p. 2

2 Related Work

Prior medical VLM work chiefly uses large-scale supervised training or RLVR, while newer hybrids differ in when and how they combine signals. GRASP instead applies both signals to the same rollouts using short answers.

Medical VQA

  • Medical VLMs commonly use large-scale SFT, sometimes followed by RL, while RLVR methods use GRPO-style training with differing reward designs.These approaches can be costly, and in-domain gains do not always transfer to new sources; RL-only methods can reinforce only answers the model already samples.Source: §2 · pp. 2–3

Combining SFT and RL

  • Hybrid approaches include staged SFT-then-RL, per-question switching, and trace-based methods that require full reasoning traces unavailable in most medical VQA data.Without annotations, RFT trains only on the model’s own correct reasoning, while ViSurf scores the gold answer as an extra rollout.Source: §2 · p. 3
  • GRASP applies RL and SFT to the same rollouts in one update, supervising the gold answer after each rollout’s own reasoning with a group-accuracy-dependent weight.Unlike methods that discard all-wrong and all-correct groups, GRASP keeps them, corrects their advantages, and supervises all-wrong groups with the gold answer.Source: §2 · p. 3

3 Method

GRASP trains RL and SFT branches on the same model-generated reason-then-answer rollouts, using short gold answers rather than annotated reasoning. A group-accuracy-dependent weight allocates answer supervision alongside the RL objective.

3 Method

  • GRASP updates RL and SFT branches on the same rollouts, using the model’s reasoning and answers for RL and gold answers for questions RL fails.Its training examples are medical images, questions, and one-to-three-word gold answers; LoRA adapters sit in the vision encoder and language model.Source: §3 · p. 3, §3.1 · p. 3

3.1 Group Sampling and Rewards

  • The reward combines exact match, synonym similarity, image-grounding similarity, and format penalties, with exact match as the main training signal.Groups are classified as all-wrong, all-correct, or mixed by their exact-match accuracy; extreme groups are common early in training.Source: §3.1 · p. 3

3.2 RL Branch: GSPO-EGC

  • GSPO-EGC modifies advantages for all-wrong and all-correct groups, where within-group comparison cannot reinforce correct answers or push down all wrong answers as a whole.Its correction compares the group outcome with running training accuracy; when accuracy is usually high, an all-wrong group is penalized more, and when correct answers are rare, a fully solved group is rewarded more.Source: §3.2 · p. 4

3.3 SFT Branch: Answer Supervision on Own Reasoning

  • The SFT branch keeps each rollout’s own reasoning as a loss-free context and computes cross-entropy only over the teacher-forced gold-answer span.This requires no expert or distilled traces and does not train the model to repeat its own potentially wrong explanation.Source: §3.3 · p. 4

3.4 Group-Adaptive Weight and Joint Objective

  • The SFT weight decreases with group accuracy: all-wrong groups receive the most supervision, while all-correct groups receive none.The RL term acts on every group, and SFT acts concurrently when group accuracy is below one; a fixed-weight KL term keeps the policy near the base model.Source: §3.4 · p. 5

4 Experimental Setup

The experiments use MedTerse short-answer training data, evaluate on held-out in-domain and eight-modality OOD questions, and compare training objectives under shared training conditions and multiple metrics.

Data

  • MedTerse contains 12,893 short-answer questions from four public sources, with questions requiring options rewritten and ambiguous or overly broad questions removed.The questions have one- to three-word answers and no reasoning annotations.Source: §4 · p. 5

Evaluation sets

  • Evaluation includes 6,600 held-out in-domain questions, 11,575 OOD questions across eight OmniMedVQA modalities, and external endoscopy and specific-diagnosis sets.The OOD questions are converted to short-answer form, with gold answers taken from option labels and corrected.Source: §4 · p. 5, Figure 2 · p. 5

Models and training

  • Training uses Qwen3-VL-8B as the main backbone, with InternVL3-8B and Gemma 3-4B also tested, and tunes rank-32 LoRA adapters.For Qwen3-VL-8B, the adapters comprise 102.7M parameters, or 1.17% of the model.Source: §4 · p. 5

Baselines

  • Training-objective baselines share the backbone, data, LoRA, rewards, and evaluation protocol, while 10 released general and medical VLMs provide model-level comparisons.The objective baselines vary how SFT and RL are used; the released models are evaluated with the same prompt, images, decoding, and scorer.Source: §4 · pp. 5–6, Table 2 · p. 6

Metrics

  • The study reports normalized exact match for open-ended, yes/no, and all questions, balanced accuracy for OOD yes/no, and LLM-judged semantic equivalence.Two physicians labeled 200 prediction-gold pairs; on pairs where they agreed, the judge matched them 93.9% of the time (Cohen’s κ=0.87).Source: §4 · p. 6

5 Results

GRASP leads the compared medical VLMs on OOD open-ended questions and outperforms alternative training objectives across in-domain and OOD evaluations. Its gains hold across backbones, and ablations show that supervision placement and the hybrid’s components matter.

5.1 Comparison with Single-Signal Training

  • Across three seeds, GRASP leads SFT and GSPO by 6.4–14.4 points in-domain and 2.4–2.6 points on OOD open-ended questions, with standard deviations at most 0.3.It is best on every in-domain split and on OOD open-ended questions overall and per modality; SFT’s OOD yes/no balanced accuracy falls from 63.6% to 47.1%.Source: §5.1 · p. 6

5.2 Comparison with Hybrid Training

  • The HPT switch is the most balanced of four alternative hybrids but trails GRASP by 3.8 in-domain and 1.3 OOD open-ended points.Its slight OOD overall-accuracy edge comes from OOD yes/no questions; staged training, ViSurf, and RFT retain only parts of SFT’s in-domain fit and RL’s OOD generalization.Source: §5.2 · p. 6

5.3 Comparison with Medical VLMs

  • GRASP on Qwen3-VL-8B and InternVL3-8B beats all 10 compared VLMs on OOD open-ended questions, reaching up to 27.3% EM versus at most 21.6%.The lead also holds on CT, X-ray, MRI, and ultrasound separately; GRASP-Qwen gains 9.6 points over its base model.Source: §5.3 · pp. 6–7, Table 3 · p. 7

5.4 Generalization across Backbones

  • On InternVL3-8B, OOD open-ended EM is 27.3 for GRASP, 25.0 for GSPO-EGC, 24.9 for SFT→GSPO-EGC, and 22.5 for SFT.Across InternVL3 and Gemma 3, GRASP also leads the single-signal and staged controls in-domain and on OOD open-ended questions.Source: Figure 3 · p. 7, §5.4 · p. 7

5.5 Ablations

  • Every ablation change costs 1.9–4.1 in-domain open-ended points; removing answer-after-reasoning costs 4.1 in-domain and 2.4 OOD open-ended points.The exclusive switch loses 3.8 points and constant SFT weighting loses 3.4, while removing extreme-group correction costs 2.7 points.Source: §5.5 · p. 8, Table 4 · p. 8

6 Analysis

GRASP combines answer supervision with RL on the same rollouts, while group-level supervision and advantage correction change as the model learns; its tested hyperparameter variations affect in-domain more than OOD open-ended accuracy.

Supervision dynamics

  • All-correct groups reach about half of groups in epoch two, while average supervision weight falls from 0.21 to 0.17 and remains concentrated on all-wrong groups.The correction shifts from favoring all-correct groups early toward penalizing all-wrong groups as running accuracy rises.Source: §6 · p. 8

Why simultaneous beats the other hybrids

  • In GRASP, failing groups are pushed away from wrong answers and toward the gold answer in the same update; bare-answer injection instead reduces valid reasoning from 74% to under 5% within 70 steps.RFT learns only from questions already solved, but 16 samples never solve nearly three quarters of open-ended training questions.Source: §6 · p. 8

Hyperparameter sweep

  • The default hyperparameters give the highest in-domain accuracy; weaker supervision costs 3.8 in-domain open-ended points and faster fading costs 2.6, while OOD accuracy stays within 0.7 points.OOD open-ended accuracy was insensitive to these hyperparameters in the tested range.Source: §6 · p. 8

7 Conclusion

GRASP applies RL and gold-answer supervision to the same rollouts, supervising the answer after the model’s own reasoning and adapting its weight by group accuracy. It outperforms the reported training objectives and, on two backbones, all 10 compared VLMs on OOD open-ended questions.

7 Conclusion

  • GRASP applies RL and gold-answer supervision simultaneously to the same rollouts, placing answer supervision after the model’s own reasoning and weighting it by group accuracy.GSPO-EGC corrects advantages for all-wrong and all-correct groups.Source: §7 · p. 8
  • GRASP outperforms all baseline training objectives in-domain on three backbones and beats all 10 compared general and medical VLMs on OOD open-ended questions with InternVL3-8B and Qwen3-VL-8B.The OOD comparison uses open-ended questions; training uses 12.9K questions and tunes about 1% of parameters.Source: §7 · p. 9
  • GRASP requires only short answers and no reasoning annotations.The reported training set contains 12.9K questions.Source: §7 · p. 9

Limitations

The authors identify limitations in answer-reasoning consistency, clinical evaluation, and the training data’s coverage and answer length.

Limitations

  • Supervising the gold answer after the model’s reasoning can sometimes teach answers inconsistent with that reasoning, and trained models sometimes report findings on normal OOD images.Normal cases are scarce in the training data; consistency gating and adding normal cases are proposed future directions.Source: Limitations · p. 9
  • Exact match can miss synonyms and abbreviations, and neither exact match nor judged accuracy fully replaces clinical evaluation.The evaluation also reports results under common and model-specific prompts because baseline prompts and training corpora differ.Source: Limitations · p. 9
  • Training data cover four sources, contain few normal cases, and are restricted to answers of at most three words; every model scores below 13% judged on VQA-Med 2020.The authors identify fine-grained diagnosis as remaining difficult.Source: Limitations · p. 9

Ethics Statement

The work uses public research benchmarks with de-identified images and collected no patient data.

Ethics Statement

  • All datasets are public research benchmarks with de-identified images and are used under their licenses.Source: Ethics Statement · p. 9
  • Only question and answer text, never images, was sent to an external LLM service.Source: Ethics Statement · p. 9
  • The models are research artifacts and should not be used for diagnosis or treatment without regulatory validation and expert oversight.Source: Ethics Statement · p. 9

Appendix A Method and Training Details

The appendices detail GRASP’s training losses, rewards, and evaluation protocol, alongside MedTerse’s construction and release. The evaluation covers held-out in-domain tests, an eight-modality OOD set, and two external datasets.

Appendix A Method and Training Details

  • The main run averages policy gradients over rollouts, token-normalizes weighted answer losses, and uses separate RL and SFT loss scales.Batch-level reward normalization avoids stretching groups with nearly identical rewards to unit variance.Source: Appendix A · p. 11
  • Supervised loss masks reasoning tokens; unusable rollout reasoning is truncated or replaced, while 98.1% of supervised rollouts retain their original reasoning.Truncated reasoning accounts for 1.9% and borrowed reasoning for 0.01% in the main run.Source: Appendix A · p. 11
  • For fixed reasoning distributions, answer supervision raises a lower bound on the likelihood of the gold answer, which GRASP combines with verifiable rewards.The lower bound is the expected conditional log-likelihood of the gold answer after sampled reasoning.Source: Appendix A · p. 11
  • Rewards combine format checks, exact match, Bio_ClinicalBERT BERTScore, and BiomedCLIP image–reasoning similarity, with removed terms zeroed in ablations.Exact match gives 2 for a normalized match and 0 otherwise; similarity terms are clipped as specified.Source: Appendix A · p. 12
  • The main configuration, SFT, and GSPO each use three seeds; Table 6 reports mean and standard deviation, while main tables show one run.The three-seed runs keep data, rewards, and optimization settings fixed.Source: Appendix A · p. 12, Table 6 · p. 12

Appendix B Data Construction

  • MedTerse retains short-answer questions from PathVQA, SLAKE, VQA-RAD, and GMAI-MMBench, rewriting option-dependent questions and discarding ambiguous rewrites.The released 12,893 training items include processing type, source, original question and image identifiers, and corrected answers; images are not redistributed.Source: Appendix B · pp. 12–13
  • Evaluation uses 6,600 in-domain questions, 11,575 OOD OmniMedVQA questions across eight modalities, 1,346 Kvasir-VQA questions, and 353 VQA-Med 2020 diagnosis questions.The OOD set is a held-out source with its own question distribution, not an unseen imaging modality; balanced accuracy is reported for its yes/no questions.Source: Appendix B · p. 12

Appendix C Additional Results

Under each model’s own prompt, GRASP remains best on OOD open-ended questions and Kvasir-VQA, while HealthGPT-Pro leads only on in-domain all-question exact match. Additional analyses show modality-level results, training dynamics, a hyperparameter sweep, and illustrative examples.

Appendix C Additional Results

  • 71.9% EM on all questions: GRASP trails HealthGPT-Pro’s 73.3% in-domain score, but remains best on OOD open-ended questions and Kvasir-VQA.HealthGPT-Pro trains over 80 times as many parameters and about 1,000 times as much data, yet trails GRASP on OOD open-ended questions.Source: Appendix C · p. 13
  • GRASP leads on each of the four radiology modalities in OOD open-ended accuracy under both exact match and the LLM judge.Figure 4 presents exact match and LLM-judged results.Source: Appendix C · p. 13, Figure 4 · p. 13
  • Figure 5 tracks rollout-group types, mean supervision weight, and extreme-group correction during the main run.Its panels show group counts by number of correct answers, epoch-averaged supervision weight, and training-adaptive correction.Source: Appendix C · p. 13, Figure 5 · p. 13
  • The hyperparameter analysis varies λmax, γ, and κ one at a time and reports open-ended exact match over two epochs.Non-default Figure 6 points are single runs; the default is a three-seed mean with ±1 standard deviation bands.Source: Appendix C · p. 13, Figure 6 · p. 13

Appendix D Qualitative Examples

  • Three correctly answered examples illustrate an in-domain X-ray, an OOD X-ray, and a held-out endoscopy image, rather than estimating case frequency.The examples were selected because GRASP answered correctly and its reasoning identified relevant evidence.Source: Appendix D · p. 13

Appendix E Baseline Protocols

The appendix describes shared baseline evaluation and several excluded systems, then details hybrid baselines and a physician audit of the LLM judge. The audit finds high agreement on clear cases, with errors concentrated in borderline equivalence judgments.

Appendix E Baseline Protocols

  • Baselines receive the same images and preprocessing, decode greedily, and use a shared scorer; results also use each model’s own prompt to avoid format penalties.Table 7 compares models under their own prompts, with direct-answer models prompted for a word or phrase and reasoning models using their own formats.Source: Appendix E · p. 13, Table 7 · p. 14
  • MedCCO and Med-R1 are excluded because MedCCO lacked a checkpoint and used a different protocol, while Med-R1 trains on OmniMedVQA and reports multiple-choice accuracy.Med-R1 trains separate checkpoints by modality or task, so OmniMedVQA is not held out for its evaluation.Source: Appendix E · p. 14
  • ViSurf injects the gold answer as a ninth group member, but valid-reasoning rollouts fall from 74.5% to under 5% by step 70.Without reasoning annotations, the injected answer has no reasoning; its few tokens dominate token-averaged loss, and the eliminate rule cannot prevent learning to skip it.Source: Appendix E · p. 14
  • The HPT-switch baseline uses each rollout’s own reasoning before gold-answer supervision because the dataset lacks expert reasoning traces.This version switches each question between SFT and RL, unlike simultaneous GRASP training.Source: Appendix E · p. 14
  • RFT trains on correct samples with valid reasoning from 16 sampled answers, but only 26.4% of open-ended questions yield a correct sample, versus 86.5% of yes/no questions.At most four correct samples per question are used to train a rank-32 LoRA for two epochs.Source: Appendix E · p. 14

Appendix F LLM Judge and Human Audit

  • On physician-consensus pairs, the LLM judge matches labels 93.9% of the time, with disagreements concentrated in near-differences; OOD open-ended rankings match under judge and exact match.Agreement falls to 78.3% on deliberately ambiguous near-differences; the audit strata were deliberately sampled, not population-representative.Source: Appendix F · p. 15, Table 9 · p. 15

Related papers

Loading 2610.34561v1…