PaperlayerView exact revision on arXiv

Source-linked AI summary

Separating Capability from Confidence: Grounded Dual-State Calibration for GRPO-Trained Medical Vision-Language Models

Yangyang Xie, Ke Hao, Jiaqi Liu, Yun Gu, Xinglin Zhang

arXiv:2609.06419v1cs.CVcs.LG

TL;DR

Medical VLM confidence should reflect both correctness and image support, but verbalized confidence jointly optimized with answers may interfere with answer learning and become near-binary. DualRead freezes the GRPO-trained actor and estimates reliability from internal states, while CCG-AUC tests confidence under real-image substitutions. Across in- and out-of-distribution medical VQA, DualRead improves confidence reliability and grounding over verbalized confidence while preserving actor answers.

  • Problem

    Medical VLM confidence needs to reflect answer correctness and patient-specific visual support, yet verbalized scores jointly optimized with answers may interfere with correctness learning and become near-binary.

    Source: Introduction · p. 1

  • Method

    DualRead freezes the GRPO-trained actor and combines a pre-answer solvability read with a post-answer assessment of visual support; CCG-AUC evaluates confidence when real hard-negative images change correctness.

    Source: Introduction · pp. 1–2

  • Results

    Across ID and OOD medical VQA, DualRead improves confidence reliability and grounding over verbalized RLVR confidence while preserving the actor’s answers.

    Source: Results · p. 5, Conclusion · p. 7

  • Takeaways & Limitations

    Under same-question hard negatives, DualRead confidence responds to decisive visual evidence while the same actor’s verbalized token does not.

    Source: Conclusion · p. 7

    Evidence is limited to short-answer medical VQA on 2–4B models; larger models and clinical report generation remain to be tested.

    Source: Conclusion · pp. 7–8

Abstract

from arXiv · show

Medical vision-language models (VLMs) require confidence that reflects both answer correctness and patient-specific visual evidence. Recent GRPO-based methods optimize verbalized confidence together with answer generation. However, this joint optimization may interfere with answer learning and drive confidence toward near-binary values. Verbalized confidence also provides no explicit assessment of visual support. We therefore separate capability learning from confidence estimation and propose \textbf{DualRead}. DualRead builds on the insight that reliability can be read from the actor's internal states at critical moments in the answering process. It freezes the GRPO-trained actor and combines pre-answer solvability with a post-answer assessment of the generated answer and its visual support. To further assess whether confidence reflects visual grounding, we introduce \textbf{Counterfactual Confidence Grounding AUC} (CCG-AUC). It measures whether confidence decreases when real-image substitution changes the actor from correct to incorrect. Across two VLM backbones and both in- and out-of-distribution medical VQA benchmarks, DualRead improves correctness discrimination and calibration over verbalized confidence while preserving answer accuracy. CCG-AUC reveals whether confidence responds to answer-relevant visual evidence rather than primarily to non-visual cues.

Introduction

Medical VLM confidence must reflect both answer correctness and patient-specific visual support, but verbalized confidence optimized jointly with answers may interfere with capability learning. DualRead instead freezes the trained actor and estimates reliability from internal states, while CCG-AUC tests whether confidence responds to decisive visual evidence.

Introduction

  • Clinical selective workflows need confidence aligned with actual correctness so high-confidence cases can be handled automatically and uncertain cases deferred to clinicians.Overconfidence remains in medical VQA across model families and scales.Source: Introduction · p. 1
  • Medical VLM confidence should indicate whether an answer is supported by patient-specific image evidence, rather than relying primarily on learned linguistic priors.Source: Introduction · p. 1
  • Jointly optimizing verbalized confidence with answers may interfere with correctness learning, while direct score rewards can polarize confidence toward near-binary values.A verbalized score is an autoregressive action affected by prompt wording, output format, and reward design.Source: Introduction · p. 1
  • DualRead freezes the GRPO-trained actor, then combines pre-answer solvability and post-answer visual-support reads to estimate confidence without changing its answer policy.PSR reads the state before response generation; PER assesses the completed answer’s support from the image, and their logits are calibrated separately before fusion.Source: Introduction · pp. 1–2
  • CCG-AUC tests whether confidence decreases when replacing an image with a real hard negative changes the actor’s answer from correct to incorrect.It fixes the question, actor, and decoding rule, keeping counterfactual inputs in distribution while isolating visual evidence.Source: Introduction · p. 2

Related Work

Prior medical VQA reliability work mainly refines self-reported confidence, while calibration-aware policy methods still express confidence through generated tokens. DualRead and CCG-AUC address gaps in decoupling confidence from answer optimization and evaluating whether confidence itself responds to visual evidence.

Related Work

  • Medical VQA reliability methods have mainly refined self-reported confidence through prompting, answer-set strategies, or semantic perturbation.Verbalized uncertainty in medical VQA remains poorly calibrated.Source: Related Work · p. 2
  • Calibration-aware GRPO variants add confidence objectives, but still use the jointly optimized policy’s generated token stream, which is sensitive to prompts and format.Examples include proper-scoring-rule, gradient-decoupling, ranking-aware, and multimodal calibration approaches.Source: Related Work · p. 2
  • Existing approaches leave a gap for confidence estimation decoupled from answer-policy optimization that reads principled generation states and accounts for visual grounding.The passage contrasts this gap with methods that predict correctness through verbalized confidence or hidden-state probes.Source: Related Work · p. 2
  • Prior grounding protocols target answers or use absent or corrupted images, whereas CCG-AUC evaluates confidence with real in-distribution hard negatives when image changes determine correctness.Unlike actor retraining with a hard-negative contrastive objective, DualRead leaves the actor and its answers unchanged and adjusts confidence estimation.Source: Related Work · p. 2

DualRead Confidence Calibration

DualRead separates answer-capability training from confidence estimation: it freezes a GRPO-trained actor and reads its internal states before and after answering. The two reads are calibrated separately and fused without changing the actor or its answer.

DualRead Confidence Calibration

  • DualRead trains the actor for answer capability with GRPO, then freezes it while reliability supervision updates only a separate confidence proxy.The actor and its answer policy remain unchanged during confidence fitting.Source: DualRead Confidence Calibration · pp. 2–3

Natural-response Replay

  • Natural-response replay extracts final-layer hidden states immediately before response generation and after the answer is complete.The actor first generates its natural response, which is then replayed under teacher forcing.Source: Natural-response Replay · p. 3

Dual-state Confidence Proxies

  • PSR maps the pre-answer state, which is independent of response wording, length, and fluency, to a correctness logit.Causal masking limits this state to the input image and question.Source: Dual-state Confidence Proxies · p. 3

Post-Answer Evidence Read (PER).

  • PER combines a post-answer state with answer-conditioned pooling of image-token states to assess the generated answer against visual evidence.P-State is fitted first and frozen while P-Visual learns a residual correction; cosine-similarity pooling selects image states aligned with the answer.Source: Post-Answer Evidence Read (PER) · pp. 3–4

Calibration and Dual-State Fusion

  • Separately temperature-scaled PSR and PER confidences are fused by an equal-weight average in log-odds space, with no learned fusion parameters.The actor and its generated answer remain unchanged.Source: Calibration and Dual-State Fusion · p. 4

Counterfactual Confidence Grounding Metric

CCG-AUC evaluates whether confidence responds to answer-relevant visual evidence by comparing original images with real, same-question hard negatives. Its strict variant holds the canonical answer fixed, and the metric does not assess pixel-level localization or rationale faithfulness.

Counterfactual construction

  • CCG-AUC compares confidence on an original image and a real hard negative paired with the same question but a different reference answer.Using real images keeps the counterfactual in distribution.Source: Counterfactual Confidence Grounding… · p. 4

Decisive cases

  • The decisive set contains cases where image replacement changes the actor’s answer from correct to incorrect, testing whether confidence decreases accordingly.The same frozen actor and decoding rule generate answers for both images.Source: Counterfactual Confidence Grounding… · p. 4

Scoring

  • CCG-AUC scores 1 when confidence is higher for the original image, 0 when higher for the hard negative, and 0.5 for a tie.A score of 0.5 indicates no directional response; 1.0 indicates confidence always decreases after decisive image replacement.Source: Counterfactual Confidence Grounding… · p. 4

Scope

Natural and strict variants

  • Strict CCG-AUC evaluates confidence response to image evidence while holding the canonical answer fixed.Natural CCG-AUC instead uses naturally generated responses, which may change after image replacement.Source: Counterfactual Confidence Grounding… · p. 4

Experiments and Results

Across in- and out-of-distribution medical VQA, DualRead improves confidence reliability and visual grounding over verbalized confidence while preserving the frozen actor’s answer accuracy. The experiments examine its benchmark performance, component contributions, confidence distributions, counterfactual behavior, and inference costs.

Datasets and Evaluation Metrics

  • 1,061 SLAKE-test questions form the in-distribution benchmark; six OOD datasets contribute 4,584 examples, macro-averaged equally by domain.No OOD examples are used to fit actors, confidence readers, or calibration temperatures.Source: Datasets and Evaluation Metrics · p. 5

Implementation Details

  • The capability actors use Qwen3-VL-2B-Instruct and MedGemma1.5-4B, post-trained with GRPO; reader fitting and temperature scaling use image-disjoint SLAKE-train partitions without target-domain adaptation.All methods except pretrained Base models are evaluated over three random seeds; calibrated branches are fused by an equal-weight mean in log-odds space.Source: Implementation Details · p. 5

Ablation study..

  • DualRead combines PSR and PER; adding the visual residual improves discrimination, calibration, and grounding, while the two branches capture complementary confidence information.The combined method has the highest AUROC and lowest Brier score on both backbones, and best CCG-AUC on Qwen3VL-2B; PER slightly exceeds its CCG-AUC on MedGemma1.5-4B.Source: Ablation study · p. 5, Table 2 · p. 6

Results

  • DualRead achieves the best ID AUROC and Natural CCG-AUC on both backbones, and improves OOD AUROC and both CCG metrics over verbalized RLVR while preserving actor accuracy.It achieves the best Strict CCG-AUC on Qwen3VL-2B and strongest OOD grounding on MedGemma1.5-4B, while some individual OOD metrics favor SAPLMA or BICR.Source: Results · p. 5, Table 1 · p. 6

Policy optimization compresses verbalized confidence..

DualRead

  • Reading confidence from frozen internal states recovers useful resolution without modifying the actor or its answers.DualRead reads states before response generation and after the completed answer, with the post-answer read incorporating visual evidence conditioned on the answer.Source: DualRead · p. 6

DualRead responds to visual evidence..

  • After same-question image replacement, DualRead assigns lower confidence to a hard-negative image even when the actor retains its answer and becomes incorrect.The examples illustrate Strict CCG-AUC behavior; they are qualitative and should be read alongside aggregate CCG-AUC results.Source: DualRead responds to visual evidence · p. 7, Figure 4 · p. 7

DualRead provides an efficient compute–reliability trade-off..

  • DualRead offers strong reliability with substantially lower additional inference under distribution shift than repeated sampling, while repeated sampling remains competitive on some metrics.Table 3 compares the compute–reliability trade-off across methods.Source: DualRead provides an efficient… · p. 7, Table 3 · p. 7

Conclusion

DualRead reads confidence from a frozen GRPO-trained actor’s internal states, improving confidence assessment without changing its answers. The evidence covers short-answer medical VQA on 2–4B models.

Conclusion

  • DualRead fuses pre-answer solvability and post-answer visual-support reads to improve discrimination, calibration, and selective prediction without changing the actor’s answers.Under same-question hard negatives, its confidence responds to decisive visual evidence while the same actor’s verbalized token does not, across both backbones.Source: Conclusion · p. 7
  • The evidence is limited to short-answer medical VQA on 2–4B models; larger models, report generation, and non-uniform open/closed behavior remain future directions.Future work should test state-based training signals and extend counterfactual evaluation to entity-level claims in clinical reports.Source: Conclusion · pp. 7–8
  • The conclusion presents hidden-state confidence as an alternative to confidence generated by a GRPO-trained medical VLM.It identifies confidence estimation, rather than answer generation, as the part separated from the actor’s generated output.Source: Conclusion · p. 7

Technical Supplement

The supplement documents the frozen experimental protocol, GRPO actor-training formulation, and hyperparameter configurations.

Technical Supplement

  • The supplement records the frozen protocol, exact configurations, and extended measurements underlying the main paper.Source: Technical Supplement · p. 10

A. GRPO Formulation

  • GRPO samples response groups, uses correctness and formatting rewards to form group-relative advantages, and applies a clipped policy objective to train the actor.The objective increases the likelihood of responses that outperform other samples for the same input.Source: A. GRPO Formulation · p. 10

B. Hyperparameters

  • Table 4 lists the frozen configurations for GRPO actor training, DualRead head fitting, and evaluation.Source: B. Hyperparameters · p. 10

C. Dataset Details

The dataset protocol uses frozen manifests and specifies the splits, sampling rules, and answer interfaces for SLAKE and the evaluation datasets.

C. Dataset Details

  • The experiments use frozen manifests for SLAKE, VQA-RAD, PathVQA, PMC-VQA, OmniMedVQA, ReXVQA, and VQA-Med 2019.Each manifest has one row per question and an image identifier used for disjoint fitting, subgroup analysis, and clustered resampling.Source: C. Dataset Details · p. 10
  • SLAKE records answer type and content family, with validation reserved for model selection and threshold selection rather than test metrics.Subgroup metrics are recomputed from item-level predictions.Source: C. Dataset Details · p. 10
  • VQA-RAD uses a frozen 600-question, 270-image manifest as a zero-touch transport test under SLAKE’s short free-form answer contract.Source: C. Dataset Details · p. 10
  • PathVQA evaluation samples 500 open and 500 closed questions, with at most two questions per image, from its 6,719-question test manifest.The resulting set contains 706 unique images, and its selection rule and IDs are fixed across methods and seeds.Source: C. Dataset Details · p. 10
  • PMC-VQA, OmniMedVQA, and ReXVQA use four-option semantic-answer inputs and contribute to the six-domain OOD average.Their frozen factual manifests contain 1,000, 920, and 1,000 questions, respectively.Source: C. Dataset Details · p. 10
  • VQA-Med 2019 contributes a frozen 64-question closed subset selected by question identity after removing four images duplicated in VQA-RAD.Source: C. Dataset Details · p. 10

D. Evaluation Details

Evaluation uses a fixed correctness grader and answer-confidence parsing contract, with confidence metrics and uncertainty procedures defined for the reported measurements.

D. Evaluation Details

  • A deterministic non-LLM grader scores an answer correct only when its non-empty canonical form exactly matches the canonical gold answer.Laterality is preserved, and no synonym additions are made after test or OOD inspection.Source: D. Evaluation Details · p. 10
  • Confidence parsing requires one integer from 0 to 100 in the specified confidence block; invalid values are retained and assigned 0.5 for evaluation.State readers always return a finite scalar and have parse rate one.Source: D. Evaluation Details · p. 10
  • AUROC measures correct-versus-incorrect confidence ranking, while ECE-15 uses 15 fixed confidence intervals and skips empty bins.Source: D. Evaluation Details · p. 10
  • AURC ranks confidence from high to low and treats equal-confidence items as indivisible blocks with uniform random ordering.HCE@0.9 measures error mass across the full evaluation set, not conditional error among accepted cases.Source: D. Evaluation Details · pp. 10–11
  • Reader weights and standardization exclude calibration, validation, test, and OOD images; scalar temperatures are fitted on image-disjoint SLAKE-train calibration data and frozen.No temperature is refit on the six OOD datasets.Source: D. Evaluation Details · p. 11
  • Reported uncertainty is the standard deviation across actor seeds, and paired claims resample source images while retaining all questions from each sampled image.Grounding uses 10,000 percentile-bootstrap replicates.Source: D. Evaluation Details · p. 11

E. Baseline Implementation Details

This appendix specifies the policy and estimator baselines and the prompts, parsing rules, and inference costs used to compare them.

E. Baseline Implementation Details

  • The implementation distinguishes GRPO policy baselines from estimators, with Base, RLVR, RLCR, DCPO, and VL-Calibration generating their own answers.RLVR is the capability actor; RLCR adds a pointwise proper-scoring term, while DCPO and VL-Calibration retain their specified confidence and policy objectives.Source: E. Baseline Implementation Details · pp. 11–12
  • P(True), self-probing, and prompt ensemble assess the frozen actor’s existing answer; none changes it, and the ensemble averages valid scores from four prompts.P(True) uses normalized first-token Yes/No probabilities; self-probing takes the last valid integer, and invalid scores are handled as specified by the parsing rules.Source: E. Baseline Implementation Details · p. 12, Exact Prompt Templates · p. 13
  • Hidden-state readers use teacher-forced replay, while sequence likelihood uses answer-token probabilities and sampling baselines require stochastic generations.SAPLMA uses a fixed mid-layer activation and classifier; BICR contrasts real- and black-image states, and semantic uncertainty groups sampled answers before calibration.Source: E. Baseline Implementation Details · p. 12

Exact Prompt Templates

  • Most methods share the same natural-answer prompt, with greedy evaluation using it unchanged; SC and semantic uncertainty change decoding rather than prompt wording.The model-visible instructions are shared across Qwen and MedGemma, while each processor applies its native chat template.Source: Exact Prompt Templates · pp. 12–13
  • VL-Calibration alone uses its native five-part prompt, including separate vision and reasoning confidence values combined by a frozen harmonic aggregation.Its native output contract requires 512 completion tokens, compared with 256 for the shared short format; DualRead and several hidden-state baselines use one teacher-forced replay.Source: E. Baseline Implementation Details · p. 12, Exact Prompt Templates · pp. 13–14

F. Properties and Scope of CCG-AUC

This section describes CCG-AUC’s counterfactual construction, invariance properties, evaluation context, and constraints on interpretation and use.

F. Properties and Scope of CCG-AUC

  • CCG-AUC compares confidence on an original image and a real hard-negative image for cases where substitution changes the actor’s answer from correct to incorrect.Donors share the normalized question but have a different canonical gold answer and image; strict CCG-AUC additionally requires equal, non-empty generated answers across the pair.Source: F. Properties and Scope of CCG-AUC · p. 14
  • CCG-AUC is parameter-free, scale-free, and model-agnostic, consuming per-item confidence and correctness pairs without fitting or cross-estimator normalization.Its rule depends on the sign of the within-item confidence difference and is invariant to strictly increasing reparameterizations of confidence.Source: F. Properties and Scope of CCG-AUC · p. 14
  • Construction requires repeated-question groups, gold answers, a shared grader, and at least 30 independent source and donor images for an eligible cell.The metric can be underpowered or unconstructible for near-unique questions or datasets without gold labels; it measures answer-decisive evidence, not localization or clinical causality.Source: F. Properties and Scope of CCG-AUC · pp. 14–15

G. Paired Bootstrap Contrasts

  • Paired image-cluster bootstrap contrasts compare P-Visual with P-State on SLAKE validation and DualRead with RLCR on ID and OOD splits.The passage identifies the comparisons but supplies no contrast values.Source: G. Paired Bootstrap Contrasts · p. 15

Calibration metrics corresponding to the main table

  • Tables 8 and 9 report Acc./AUROC with ECE-15 and Brier, averaging six OOD domains with equal domain weight after computing each metric separately.SLAKE values are from the ID test set; the OOD summary averages seed-level domain values before the across-seed summary.Source: Calibration metrics corresponding to… · p. 15

Per-domain OOD reliability diagrams

  • The OOD reliability diagrams use 15 fixed equal-width bins and common axes, with pooled seed predictions shown for eight confidence interfaces across six domains.Their printed panel ECE is pooled across concatenated seed arrays, whereas the tables summarize separately computed seed-level ECE values.Source: Per-domain OOD reliability diagrams · p. 15
  • The confidence-support audit finds DualRead emits a distinct non-endpoint score per example, unlike near-constant endpoint scores from RLVR.Qwen RLVR assigns confidence one to 1060/1061 examples and MedGemma RLVR to all 1061; the audit measures score resolution separately from correctness ranking.Source: Per-domain OOD reliability diagrams · p. 15

I. Grounding Details

  • Grounding comparisons use frozen confidence readers and temperatures on evaluation sets, with CCG-AUC eligibility conditional on the actor-defined decisive set.The protocol also reports paired SLAKE actor/confidence audits and eligibility alongside OOD grounding.Source: §I · p. 15

J. Reproducibility and Scope

The reproducibility appendix records the evaluation sequence and identifies the aggregate records and visual summaries associated with the reported analyses.

J. Reproducibility and Scope

  • Each seed follows greedy generation, deterministic grading, one teacher-forced replay, reader fitting, disjoint temperature fitting, and frozen ID/OOD evaluation.Readers are actor-specific, OOD temperatures remain frozen, and CCG-AUC is conditional on an actor-defined decisive set.Source: J. Reproducibility and Scope · p. 15
  • Figures 5–10 provide per-domain OOD reliability diagrams for VQA-RAD, PathVQA, PMC-VQA, OmniMedVQA, ReXVQA, and closed VQA-Med 2019.Their captions specify sample sizes of 600, 1000, 1000, 920, 1000, and 64, respectively.Source: Figure 5 · p. 16, Figure 6 · p. 17, Figure 7 · p. 18
  • The appendix lists paired bootstrap contrasts, reliability tables for SLAKE and five OOD domains plus OOD6, and open/closed reliability and grounding tables.Table 11 covers OOD CCG-AUC, a paired SLAKE audit, and eligibility; the supplied captions identify scope, not table cell outcomes.Source: Table 7 · p. 22, Table 8 · p. 22, Table 9 · p. 23
Loading 2609.06419v1…