PaperlayerView exact revision on arXiv

Source-linked AI summary

From Reward Signal to Visual Utility: A Controlled Audit of Medical VLM Post-Training

Wang Jingxin

arXiv:2609.31450v1cs.CVcs.AI

TL;DR

Medical VLM post-training is often assessed by answer accuracy, but accuracy alone does not describe how decisions depend on images. This study compares SFT, GRPO, and a counterfactual evidence objective while tracing paired answers and score changes. SFT substantially redistributes individual visual-benefit events despite uncertain aggregate improvement, and evidence-objective gains on validation remain inconsistent at matched training doses.

  • Problem

    The study examines how post-training changes visual benefit for individual answers and when objective improvements translate into better image-conditioned decisions.

    Source: §1 · p. 1

  • Method

    The study compares SFT scopes, answer-only GRPO, and a counterfactual evidence objective, tracking paired answers and score changes across training and evaluation.

    Source: §7 · p. 15

  • Results

    SFT substantially redistributes individual successes and image dependence despite uncertain aggregate task improvement, while dose-matched validation shows inconsistent incremental benefit from the evidence objective.

    Source: §5 · p. 14, §7 · p. 15

  • Takeaways & Limitations

    Paired answers and decision margins reveal where objective progress translates into useful image-conditioned behavior and where it stops.

    Source: §7 · p. 15

    The study uses one 3B backbone, PMC-VQA, and a limited set of training recipes; evidence-objective validation reuses the set used during development.

    Source: §6 · p. 14

Abstract

from arXiv · show

Medical vision-language model (VLM) post-training is commonly evaluated through answer accuracy. We examine how changes in accuracy and training objectives relate to image-conditioned decisions in a controlled Qwen2.5-VL-3B study on PMC-VQA. We compare supervised fine-tuning (SFT) with low-rank adaptation (LoRA) restricted to the language model, expanded multimodal adaptation scopes, standard answer-only Group Relative Policy Optimization (GRPO), and a counterfactual evidence objective. On 2,000 clean-test questions, language model LoRA SFT changes correct-image accuracy by +1.10 percentage points (95% paired bootstrap CI:-0.85 to +3.05), while visual-benefit events decrease by 2.40 points and image sensitivity decreases by 5.60 points. Paired records reveal 155 acquired and 203 lost visual-benefit events. Broader adaptation yields lower correct-image accuracy than language-model LoRA SFT. Standard GRPO produces mixed-reward groups and parameter updates, with an uncertain clean test accuracy change. A generation audit reveals that canonical option scores can follow a different token path from generated answers. With scores taken along the greedy generation path, the evidence target improves on the training set; its gains over standard GRPO remain inconsistent on validation data at matched training doses. Sample-level analyses trace how evidence scores, decision margins, and generated answers change during post-training. This empirical and measurement audit identifies gaps between optimization activity, target acquisition, and useful held-out visual behavior.

1 INTRODUCTION

Post-training can shift whether medical VLM answers depend on visual evidence even when average accuracy changes little. This study tracks paired questions across training stages and image interventions to connect objective changes with image-conditioned decisions.

1 INTRODUCTION

  • The study asks how post-training changes visual benefit for individual answers and when objective improvements translate into better image-conditioned decisions.It motivates tracking both answers and their dependence on the image because models can also rely on question information or learned answer priors.Source: §1 · p. 1
  • The analysis links aggregate outcomes to individual decisions through paired answers, score decompositions, and matched-dose controls.These analyses distinguish changes in correctness patterns from changes in evidence scores and additional training exposure.Source: §1 · p. 1
  • The study compares SFT parameter scopes and answer-only GRPO on PMC-VQA, with counterfactual evidence-objective analyses on training and validation subsets.A separate SLAKE experiment examines how answer-token learning relates to generation and termination.Source: §1 · p. 2
  • The three main findings are substantial SFT gains and losses in visual-benefit events despite small net accuracy change, lower accuracy with broader adaptation, and uncertain evidence-objective transfer.Standard GRPO leaves accuracy nearly unchanged despite active optimization.Source: §1 · p. 2

2 RELATED WORK

Related work motivates evaluating visual utility beyond accuracy, comparing adaptation scopes and medical reinforcement learning, and checking whether option scores follow generated-answer token paths.

2.1 ACCURACY AND INTERVENTION-DEFINED VISUAL UTILITY

  • Beyond Accuracy uses correct and shuffled images to measure visual reliance, and this study extends it to paired questions across post-training stages and parameter scopes.Tracking questions identifies acquired, retained, and lost visual-benefit events and relates transitions to aggregate changes.Source: §2.1 · p. 2

2.2 PARAMETER-EFFICIENT ADAPTATION AND MEDICAL RL

  • Prior work on connector adaptation motivates comparing language-model LoRA, merger tuning, and vision-LoRA on the same questions.Medical RL studies motivate measuring reward variation, verifying updates, and comparing answer-only GRPO with its SFT initialization.Source: §2.2 · p. 2

2.3 SCORING FAITHFULNESS AND GROUNDING-ORIENTED OBJECTIVES

  • Prior scoring studies motivate comparing canonical option scores with scores recorded along actual multimodal generation trajectories.The counterfactual objective uses gold-answer score differences across correct and altered image conditions, with standard GRPO at matched dose as a control and score decomposition relating effects to the correct-image margin.Source: §2.3 · pp. 2–3

3 EXPERIMENTAL DESIGN AND DEFINITIONS

The experiments compare post-training variants on PMC-VQA using paired image interventions and predefined correctness measures. The design also specifies training controls, uncertainty estimation, and reproducible sample selection.

3.1 STUDY DESIGN, DATA, AND MODEL VARIANTS

  • The clean-test cohort contains 2,000 questions and is compared with 10,000 SFT training questions; the study uses Qwen2.5-VL-3B-Instruct and three SFT trainable-component configurations.Selected checkpoints use validation accuracy under correct-image conditions, while visual-benefit measures are evaluated separately; the evidence-objective validation analyses are exploratory.Source: §3.1 · p. 3

3.3 TRAINING AND INFERENCE CONTROLS

  • Standard GRPO starts from M1-L, while M3 uses canonical option-token scores and M4 uses scores aligned with the actual generation path.Clean-test generation uses a fixed answer format and greedy decoding; outputs that fail the parser count as incorrect, and M0 correct-image accuracy is 51.70%.Source: §3.1 · p. 3, §3.3 · p. 4

3.2 THREE IMAGE INTERVENTIONS

  • Each question is tested with its original image, no image, and a shuffled replacement while its question, choices, and gold label remain fixed.Visual-benefit and visual-harm rates capture opposite Correct-versus-Shuffled correctness patterns; strict vision-dependent success is pattern 100, while success under all conditions is 111.Source: §3.2 · pp. 3–4, Figure 1 · p. 4

3.4 STATISTICAL ANALYSIS AND CASE SELECTION

  • Paired question-level bootstrap intervals use 10,000 replicates and a fixed seed to quantify sampling uncertainty for evaluated checkpoints and cohort.Case selection uses the lowest numerical sample index within each specified behavioral category, requiring valid parsed options for every displayed model and condition.Source: §3.4 · pp. 4–5

4 RESULTS

Across the PMC-VQA results, post-training changes image-conditioned answers in ways that aggregate accuracy alone does not capture. Broader adaptation and evidence-objective results vary by configuration, measurement path, and training dose; the separate SLAKE experiment also distinguishes answer learning from completion.

4.1 SFT REDISTRIBUTES SUCCESS ACROSS QUESTIONS AND IMAGE CONDITIONS

  • +1.10 percentage points (95% CI [-0.85, +3.05]) was the paired Correct accuracy change from M0 to M1-L on the 2,000-question clean test.Correct accuracy rose from 51.70% to 52.80%, but the interval leaves the direction uncertain. VBR fell by 2.40 points and IS by 5.60 points, while pattern111 rose by 3.20 points.Source: §4.1 · p. 5, Table 1 · p. 5
  • 155 VBR gains and 203 losses yielded a net loss of 48/2,000, even as correct-image success was gained on 209 questions and lost on 187.Examples include transitions from 100 to 111, 100 to 000, and 000 to 100, showing distinct ways success was redistributed.Source: §4.1 · pp. 5, 8, Figure 3 · p. 6

4.2 EXPANDING ADAPTATION SCOPE LOWERS ACCURACY IN THE TESTED SFT CONFIGURATIONS

  • M1-ML and M1-VML had Correct accuracy changes of -1.55 and -1.65 points, respectively, relative to M1-L.M1-ML reduced VDS by 1.40 points and M1-VML reduced VBR by 1.65 points; other visual-measure differences had intervals crossing zero.Source: §4.2 · p. 9, Figure 5 · p. 9

4.3 STANDARD GRPO PRODUCES REWARD VARIATION AND PARAMETER UPDATES WITH LITTLE CHANGE IN ACCURACY

  • 71.2%, 68.4%, and 67.2% of GRPO prompt groups had nonzero reward variance across three training segments, and all 288 language-adapter tensors changed.M2-500 Correct accuracy was 52.60% versus 52.80% for M1-L, a paired change of -0.20 points (CI [-0.80, +0.40]).Source: §4.3 · p. 10, Figure 6 · p. 10

4.4 GENERATION-PATH SCORING CHANGES THE INTERPRETATION OF THE EVIDENCE OBJECTIVE

  • 906 of 1,500 M1-L Correct-image VALID canonical-score argmaxes disagreed with generated answers; generation-path scoring erased M3-500’s apparent evidence-alignment improvement over M2-500.The generation-path differences were -0.037234 on VALID pattern100 and -0.015625 on pattern110, in raw score units.Source: §4.4 · pp. 11–12

4.5 THE EVIDENCE OBJECTIVE IMPROVES TRAIN SCORES, WITH UNCERTAIN MATCHED-DOSE TRANSFER

  • +0.213867, +0.366699, and +0.428711 were M4’s TRAIN pattern100 target-score gains over M1-L at 500, 1,000, and 2,000 prompts.Matched-dose VALID differences versus GRPO varied across doses, and Correct, VBR, VHR, VDS, and C-S showed no stable evidence-objective advantage.Source: §4.5 · p. 12, Figure 8 · p. 13
  • Evidence and decision margins could move in opposite directions or together through the correct-image gold score; their change association was 0.2542 on 188 VALID pattern100 questions.The exploratory association after controlling for correct-image gold-score change was -0.0689, with an interval spanning zero.Source: Figure 7 · p. 11, §4.5 · p. 12

4.6 NATIVE-ANSWER TRAINING IMPROVES GOLD-ANSWER LIKELIHOOD BUT DISRUPTS OUTPUT COMPLETION

  • On 4,918 SLAKE TRAIN QA, normalized exact match fell from 56.57% to 2.34% while mean gold-answer teacher-forced NLL/token improved from 4.3688 to 0.3759.R1 outputs contained comma-separated lists on 97.25% of TRAIN questions, while 71.88% had a first segment exactly matching the gold answer; omitted termination supervision was a potential contributor, with causal role unresolved.Source: §4.6 · pp. 13–14

5 DISCUSSION

Post-training changes visual dependence and individual answers even when aggregate accuracy moves little. The discussion links those changes to measurement choices and finds held-out utility inconsistent across tested configurations.

5 DISCUSSION

  • The analysis traces effects from learning signals through generation-aligned traces, training-target acquisition, dose-matched validation, and decision-level changes.Correct-image margins and paired transitions show how training changes relate to individual answers.Source: §5 · p. 14
  • SFT substantially redistributes individual successes and image dependence, while aggregate task improvement remains uncertain and pattern100 prevalence changes little.Paired transitions explain how nearly stable averages coexist with substantial gains and losses on individual questions.Source: §5 · p. 14
  • Cross-condition raw-logit targets change under condition-specific constants even when within-condition probabilities and decisions are preserved.Generation-path alignment measures scores at the relevant response branch but does not remove this shift sensitivity.Source: §5 · p. 14
  • Held-out utility remains inconsistent across tested configurations despite expanded trainable scope, reward variation, and improved training margins.Distinguishing perception, panel binding, option priors, and optimization effects requires measurements beyond the three-bit response pattern.Source: §5 · p. 14

6 LIMITATIONS

The study’s conclusions are bounded by its model, dataset, evaluation choices, and development procedure. Its uncertainty estimates and image-intervention measures have specific limits, and broader matched post-training comparisons remain open.

6 LIMITATIONS

  • The controlled study uses one 3B backbone, PMC-VQA, and a limited set of training recipes; bootstrap intervals condition on trained checkpoints.Repeated images may make intervals understate image-clustering uncertainty, training-seed variation is unmeasured, endpoints are unadjusted for multiplicity, and examples were selected post hoc.Source: §6 · p. 14
  • Evidence objectives were repeatedly developed and examined on the same VALID set, so positive subset effects require confirmation on an independent cohort.M4 held-out evaluation also uses this development validation set.Source: §6 · p. 14
  • PMC shuffled replacements can cross modalities or contain nonclinical content, and image removal can introduce distribution shift.C-N and C-S therefore reflect these specific input changes; their clinical meaning depends on the question and replacement image.Source: §6 · p. 14
  • Multiple-choice performance can depend on option information, native-answer exact match is lexically sensitive, and labels and images lacked independent clinician adjudication.Clinical validity, including generated explanations, requires separate assessment.Source: §6 · p. 15
  • Beyond PMC-VQA, evidence is limited to the SLAKE native-answer pipeline diagnostic and VQA-Med-2019 zero-shot baseline.Matched post-training comparisons across medical datasets remain an open evaluation question.Source: §6 · p. 15

7 CONCLUSION

Across tested Qwen2.5-VL-3B configurations on PMC-VQA, individual answers changed substantially while aggregate task improvement remained uncertain. The conclusion links these outcomes to image-conditioned behavior and the limits of training-objective gains.

7 CONCLUSION

  • Substantial individual-answer changes coexist with uncertain aggregate task improvement across the tested post-training configurations.SFT creates and removes visual-benefit events; broader adaptation lowers correct-image accuracy, and selected-checkpoint accuracy under standard GRPO is nearly unchanged despite active optimization.Source: §7 · p. 15
  • After generation-path alignment, the counterfactual evidence objective improves its training target, but dose-matched validation shows inconsistent incremental benefit.The passage links interpretation of the objective to measurement token path, training amount, and changing score components.Source: §7 · p. 15
  • Tracing decision margins and paired answers reveals where training-objective progress translates into useful image-conditioned behavior and where it stops.Source: §7 · p. 15

A.1 PATTERN COMPOSITION AND ADDITIONAL CASES

The appendix presents aggregate SFT pattern distributions and paired question-level transitions, alongside selected examples of answer changes under expanded adaptation and standard GRPO.

A.1 PATTERN COMPOSITION AND ADDITIONAL CASES

  • Figure S1 reports three-bit pattern distributions across SFT parameter scopes over 2,000 questions from 1,440 images.Figure 3 complements the aggregate distributions with paired question-level transitions.Source: Figure S1 · p. 17
  • Figure S2 includes selected cases of lost correct-panel and Correct-image foot-scar answers, plus a Correct-image gain when NoImage and Shuffled were already correct.Each case is the lowest-index member of its specified selection pool.Source: Figure S2 · p. 17
  • In the abdominal-CT example, GRPO brings the Correct-image answer into agreement with already correct NoImage and Shuffled answers.Source: Figure S2 · p. 18

A.2 NATIVE-ANSWER DIAGNOSTICS AND CROSS-DATASET BASELINES

The appendix reports native-answer diagnostics and cross-dataset baseline results, including separate measures of answer-token acquisition and termination behavior.

A.2 NATIVE-ANSWER DIAGNOSTICS AND CROSS-DATASET BASELINES

  • Figure S3 separates semantic answer-token scores from EOS scores in the SLAKE native-answer experiment, showing improved answer-token likelihood alongside deteriorating termination behavior.Source: Figure S3 · p. 19
  • VQA-Med-2019 baseline C-N is +8.45 points (CI [7.00, 9.95]) and C-S is +3.55 points ([2.05, 5.05]) under the native-answer protocol.The SLAKE shuffled mapping has 2 self-maps among 96 source-image groups, reducing effective image replacements.Source: Table A1 · p. 19

A.3 SCORING AND DATA PROVENANCE

The appendix documents scoring and data provenance, then provides a table organizing experimental questions, principal findings, and supporting analyses.

A.3 SCORING AND DATA PROVENANCE

  • The generation-trace audit found 0 mismatches among 13,500 trajectories between raw parsed answers and saved outputs; 473 strict lexical-argmax differences arose from tie handling.M0 primary clean-test accuracy is 51.70% under the relaxed answer parser.Source: §A.3 · p. 20
  • SLAKE native-answer scoring uses normalized exact match from the first nonempty output line, with first-segment scoring reported separately as an answer-acquisition diagnostic.Source: §A.3 · p. 20
  • Frozen evaluation manifests track case provenance; selected images match recorded SHA256 hashes and split membership follows manifest assignment.Some source identifiers retain the pmc_train_ prefix after assignment to the repackaged VALID split.Source: §A.3 · p. 20

A.4 SUMMARY OF EXPERIMENTAL QUESTIONS AND FINDINGS

  • Table A2 organizes experimental questions, principal findings, and their supporting analyses.Source: Table A2 · p. 20

Related papers

Loading 2609.31450v1…