Source-linked AI summary
Which Medical Questions Deserve Rationales? Perturbation-Sensitive Selection for Robust QA
Yuexin Wu, Dayou Yu, Vasile Rus
TL;DR
Medical QA has abundant answer labels but scarce, costly, and noisy rationale supervision, motivating selection of already-labeled questions under a fixed rationale-token budget. The paper proposes RMS-RSP, which measures rationale-local perturbation of the gold–distractor margin and evaluates it across five medical QA datasets. RMS-RSP does not establish universal accuracy gains, but consistently improves robustness and semantic consistency under answer-option reorderings while using far fewer tokens than full supervision.
Problem
Medical QA datasets commonly provide answer labels at greater scale than carefully checked rationales, making budgeted selection of rationale supervision an important acquisition problem.
Method
RMS-RSP perturbs hidden states only at rationale tokens and ranks candidates by the root-mean-square shift in the gold-versus-best-distractor margin.
Results
Across five datasets, RMS-RSP consistently improves option-order robustness and semantic consistency, while its standard-accuracy advantage is heterogeneous and not universal.
Takeaways & Limitations
Rationale-local representation sensitivity can serve as a mechanism-aligned signal for selecting supervision when stable medical decisions matter.
Takeaways & Limitations
The evidence is limited to one 4B medical model, three training seeds, relatively small pools, and a narrower option-permutation robustness test.
Abstract
from arXiv · showhide
Medical question-answering datasets often contain answer labels, whereas high-quality rationales remain scarce, noisy, or costly to validate. This changes the acquisition question: rather than asking which questions should be labeled, we ask which already-labeled questions should receive rationale supervision under a fixed token budget. We study an offline version of this problem in which candidate rationales are visible to the selector but withheld from downstream training unless selected. We propose root-mean-square Robustness-based Sample Prioritization (RMS-RSP), which perturbs hidden states only at rationale tokens and measures the resulting shift in the gold-versus-best-distractor margin. Across five medical QA datasets, MedGemma-4B-IT, three training seeds, ten budgeted non-RSP selectors, and an unbudgeted full-supervision reference, RMS-RSP provides a deliberately qualified result. Its locked-budget accuracy is 60.61% on average versus 60.08% for Random, with a statistically resolved gain only on AfriMed-QA (+1.44 points). Its full-budget accuracy area is not better than Random. However, after three answer-option reorderings, RMS-RSP improves robust accuracy and semantic consistency by 1.91 and 2.85 points on average, respectively, with the same direction on all five datasets. Training on every pool rationale raises macro accuracy to 63.74%, but consumes 29--254 times more rationale tokens and does not uniformly improve robustness. These findings do not establish universal accuracy gains; they instead suggest that rationale-local boundary sensitivity can identify supervision that improves invariance to semantically equivalent formatting changes.
1 Introduction
The paper frames rationale acquisition as selecting already-labeled medical questions for additional supervision under a fixed token budget. RMS-RSP prioritizes rationale-local sensitivity to the gold–distractor boundary, with consistent robustness improvements but no universal accuracy advantage.
- Medical QA offers answer labels at scale, while carefully checked rationales are costly, noisy, and potentially irrelevant even when answers are correct.
- RMS-RSP injects normalized Gaussian noise into rationale-token hidden states and scores the root-mean-square shift in the gold-versus-best-distractor margin.
- The study evaluates budgeted rationale selection across five medical QA datasets against Random, uncertainty-based methods, rationale length, and recent reasoning-data selectors.
- The paper separates answer supervision from rationale supervision and formulates selecting already-labeled examples whose rationales should be acquired under budget.
- RMS-RSP is presented as a rationale-local, relative-scale margin-shift score, including a signed ablation.
- Across datasets, standard accuracy is heterogeneous, whereas option-order robustness and semantic consistency improve consistently, with an all-rationales reference for comparison.
2 Related work
Prior work studies rationale selection using uncertainty, complexity, likelihood, and training dynamics, while medical QA adds concerns about explanation faithfulness and option-order sensitivity. This paper positions its evaluation around clinical decision-boundary support and controlled invariance testing.
- Medical rationale supervision can improve multi-step reasoning, but generated explanations may not faithfully describe the computation behind an answer.
- Medical benchmarks differ in explanation provenance, including short, physician-written, paired, and literature-linked explanations.
- Reasoning-data selectors use answer entropy, token rank, surprisal, local step likelihood, or perturbation-based losses to prioritize traces.
- Medical multiple-choice QA requires evaluating more than original-order accuracy because option reordering can change predictions despite unchanged semantic content.
- The paper treats option permutations as a controlled invariance test aligned with decision-boundary motivation, not as a simulation of every clinical distribution shift.
3 Budgeted rationale selection
The method trains an answer-only model, scores visible candidate rationales under a token budget, and unlocks only selected rationales for downstream training. RMS-RSP perturbs rationale positions and prioritizes candidates whose local effects make the answer boundary sensitive.
- The candidate pool contains questions, option sets, known correct answers, and candidate rationales, while the initial model is trained only on answer labels.
- Each rationale has a token cost, and the selector chooses a subset whose total cost does not exceed the budget.
- Candidate rationales are visible to rationale-aware scoring functions but unlocked for rationale training only when selected, defining an offline selection setting.
- For each candidate, the model scores answer options after appending the rationale and a final-answer suffix, using the gold-versus-best-distractor margin.
- Perturbations apply only to rationale-token hidden states, isolating dependence of the answer boundary on the candidate explanation rather than on the question generally.
- RMS-RSP selects larger perturbation scores first, while its direction-agnostic RMS magnitude treats either margin increases or decreases as local sensitivity.
- Selected examples contribute rationale-training records alongside matched answer-only replay from unselected items, reducing gains attributable solely to differing answer supervision.
4 Experimental design
The evaluation uses frozen protocols across five datasets, a 4B medical model, multiple seeds, budgeted selectors, and an unbudgeted all-rationales reference. It measures accuracy, calibration-related metrics, token-budget curves, and invariance to answer-option permutations.
- Datasets and splits: The datasets use official or deterministic derived splits, with English MedExpQA, expert AfriMed-QA, exact deduplication, and a protocol-aligned MedMCQA evaluation.
- Training setup: All runs use MedGemma-4B-IT with LoRA adapters, three training seeds, and two-epoch rationale branches initialized from answer-only adapters.
- Budget and selection: RMS-RSP uses late layers −2, −4, and −8, four perturbations per layer, scale α = 0.10, and candidate budgets of 256, 512, and 1,024 rationale tokens.
- Uncertainty: Random varies acquisition and training seeds, whereas deterministic selectors use three training seeds with the same selected set across downstream runs.
- Reference condition: The all-rationales reference consumes 29,399–129,935 scored rationale tokens and is reported as unbudgeted because it is neither token- nor update-matched.
- Selectors: The selector comparison includes Random, answer entropy and margin, rationale length, and five adapted reasoning-data selectors.
- Metrics: The primary metric is multiple-choice accuracy, with macro-F1 secondary and Brier score plus expected calibration error released in result artifacts.
- Metrics: Token-AUBC is normalized trapezoidal area under accuracy at 0, 256, 512, and 1,024 acquired rationale tokens.
5 Results
Under the locked rationale-token budget, RMS-RSP improves average accuracy only slightly and does not consistently outperform Random. Its stronger result is robustness: option-order metrics improve across datasets, while the all-rationales reference shows higher average performance but substantial heterogeneity.
- Standard accuracy: 60.61% versus 60.08%: RMS-RSP’s five-dataset macro accuracy only narrowly exceeds Random.AfriMed-QA is the sole dataset with a statistically resolved accuracy difference, at +1.44 points.
- Standard accuracy: 55.71 versus 55.68: RMS-RSP essentially matches Random on macro-F1.Signed-RSP and answer margin perform better on macro-F1, so canonical RMS-RSP does not provide general discrimination gains.
- Option-order robustness: 1.91 and 2.85 percentage points: RMS-RSP improves macro REACC and RECON over Random, with robustness gains directionally positive on all five datasets.The reported intervals resolve both differences for AfriMed-QA, while the remaining intervals include zero.
- Interpretation: Canonical RMS-RSP measures the magnitude of rationale-local gold–distractor boundary shifts, so strong coupling can support invariance without guaranteeing improved clean accuracy.This is presented as an empirical interpretation rather than causal proof; signed aggregation instead performs better on locked-budget accuracy and macro-F1.
- Full supervision: 63.74% macro accuracy and 59.84 macro-F1: the all-rationales reference exceeds RMS-RSP by 3.12 and 4.13 points, respectively.Its aggregate robustness gains conceal heterogeneity, including lower PubMedQA robustness despite a higher accuracy point estimate.
6 Discussion and limitations
The evidence supports a narrow efficiency claim: rationale-local sensitivity can improve robustness under a fixed budget, but accuracy gains are dataset- and budget-dependent. The study’s offline setting and experimental design limit broader conclusions about annotation cost, clinical explanation quality, and deployment robustness.
- Evidence scope: Only AfriMed-QA has a clearly nonzero locked-budget accuracy gain over Random, so RMS-RSP does not consistently improve medical QA accuracy.Accuracy gains are dataset- and budget-dependent.
- Practical boundary: The experiment selects visible candidate rationales for downstream training and therefore does not measure clinician annotation time or expert rewriting quality.A prospective study should compare total generation, review, and correction cost.
- Efficiency under scarcity: RMS-RSP often approaches the high-resource all-rationales reference with far fewer rationale tokens, rather than universally outperforming full supervision.The full-supervision reference uses 29–254 times the locked token budget and is strongest on average but not uniformly more robust.
- Practical boundary: Candidate-rationale visibility makes this offline curation rather than classic pre-annotation active learning.A future proxy would score questions without candidate rationales or use cheap drafts before expert review.
- Experimental limits: The evidence is limited to one 4B medical model, three training seeds, relatively small pools, and benchmark perturbations narrower than clinical deployment shifts.The all-rationales reference is neither token- nor update-matched, and dataset rationales were not newly audited by clinicians.
7 Conclusion
The paper formulates budgeted rationale selection for answer-labeled medical QA and evaluates representation sensitivity as a mechanism-aligned signal. Its supported conclusion is a limited role for RMS-RSP in improving robustness and semantic consistency under option reorderings, not universal accuracy gains.
- Conclusion: The study evaluates rationale-local perturbation selection against recent reasoning-data selectors across five medical QA datasets.The formulation separates answer supervision from rationale supervision under a token budget.
- Conclusion: RMS-RSP does not dominate standard accuracy or Token-AUBC, while all-rationale training costs 29–254 times more tokens and is not uniformly more robust.These comparisons frame the method as a budgeted alternative rather than a universal winner.
- Conclusion: RMS-RSP’s reproducible low-budget advantage is increased robustness and semantic consistency under answer-option permutations.Representation sensitivity is therefore suggested as a mechanism-aligned signal when stable medical decisions matter.
A Additional implementation details
The implementation applies rationale-local perturbations through deterministic segmentation, bounded token scoring, repeated noisy passes, and shared sequence and optimization settings.
- Scoring procedure: Rationale steps are segmented by newline boundaries followed by a sentence heuristic, with at most 384 rationale tokens scored.RSP uses one clean pass and 12 noisy passes per candidate across three layers and four perturbations per layer.
- Scoring procedure: The pipeline requests the final answer with a fixed suffix and scores only single-token option letters.Answer-only and rationale branches share a maximum sequence length of 1,024.
- Optimization: Both branches use AdamW, cosine learning-rate schedules, and 3% warmup.These shared settings constrain differences between answer-only and rationale training branches.
B Complete macro-F1 results
Table 5 reports test macro-F1 across downstream seeds, comparing budgeted selectors with an unbudgeted all-rationales reference.
- Table 5: Test macro-F1 is reported as a percentage mean across three downstream seeds, while Random averages nine runs.The table distinguishes budgeted methods from the unbudgeted all-rationales condition.
- Table 5: Bold marks the best budgeted method, and All rationales is reported as an unbudgeted reference.The table’s emphasis therefore separates the budgeted comparison from the high-resource condition.
C Dataset-level Token-AUBC and robustness
The section reports dataset-level comparisons using accuracy Token-AUBC and option-order robustness metrics. It also presents paired differences from Random and notes the intended release of artifacts and manifests.
- Token-AUBC: Accuracy Token-AUBC is reported over 0, 256, 512, and 1,024 rationale tokens.
- Robustness: Option-order robust accuracy is reported using REACC.
- Robustness: Semantic consistency across answer-option orders is reported using RECON.
- Comparisons: Paired differences from Random are expressed in percentage points.
- Artifacts: Code and processed split manifests should be released with a non-identifying archive at submission time.