PaperlayerView exact revision on arXiv

Source-linked AI summary

Perturbation-Based Data Selection for Medical Question Answering

Yuexin Wu, Dayou Yu, Vasile Rus

arXiv:2610.33440v1cs.CL

TL;DR

Compact medical-QA adaptation requires choosing a fixed-budget subset from existing question banks, while common uncertainty and diversity signals do not directly measure local representation effects on answer boundaries. This paper compares two question-and-options-only perturbation selectors, then finds positive dataset-averaged differences over pool-matched random selection at the evaluated budgets, with performance varying across settings.

  • Problem

    The paper examines fixed-budget selection from existing medical-QA banks and whether local representation changes provide a useful selection signal beyond uncertainty or diversity criteria.

    Source: §I · p. 1

  • Method

    After an entropy prefilter, RSP combines prediction-change frequency and non-flipping margin contraction using automatic RMS scaling, while RSP-DROP ranks by contraction alone.

    Source: §I · p. 1, C. RSP: Automatic RMS Scale Balancing · p. 3

  • Results

    At B = 128, RSP and RSP-Drop achieved dataset-averaged Macro-F1 scores of 47.55 and 48.28, respectively, versus 45.11 for pool-matched random selection; relative performance varied across datasets and metrics.

    Source: §VII · p. 9

  • Takeaways & Limitations

    The selected-set case study and full-pool ablation identify component choice, candidate-pool restriction, and retained budget as distinct aspects, without establishing a universal selector-choice rule.

    Source: §VII · p. 9

    The retrospective study uses one backbone, one LoRA recipe, three seeds, and three budgets, and does not validate a rule for choosing between selectors on unseen datasets.

    Source: §VI · p. 8

Abstract

from arXiv · show

Selecting a compact training subset is an important step in adapting medical language models from existing question-answering banks. We investigate perturbation-based data selection using two signals derived from Gaussian perturbations of the final-block representation: prediction-change frequency and non-flipping margin contraction. After a shared entropy prefilter, Representation-Sensitivity Prioritization (RSP) combines the signals using automatic root-mean-square scaling, while RSP-Drop ranks candidates by contraction alone. Both selectors use only questions and answer options during selection. We evaluate them with MedGemma-4B on five medical QA datasets at retained-set budgets of 64, 128, and 256 examples. At the primary budget of 128, RSP and RSP-Drop obtain dataset-averaged Macro-F1 scores of 47.55 and 48.28, respectively, compared with 45.11 for entropy-pool-matched random selection. Their relative performance varies across datasets and budgets. A selected-set case study identifies contrasting clean-margin profiles among the examples exchanged by the two rules, while a full-pool ablation shows that entropy prefiltering is not uniformly beneficial. These retrospective results characterize the empirical utility and limitations of perturbation-based selection under a fixed adaptation protocol. Evaluation uses one backbone and benchmarks with documented training-family overlap; it does not establish clinical effectiveness.

I. INTRODUCTION

The paper studies question-and-options-only selection of a fixed-size medical QA training subset, using perturbation responses to examine how local representation changes affect answer boundaries. It compares two sensitivity rules and situates them among uncertainty, diversity, and explanation-aware approaches.

I. INTRODUCTION

  • Selection uses only questions and options before auxiliary text is introduced, while the downstream adaptation recipe is held fixed for comparisons.The study evaluates offline curation of an existing bank, not patient-level decision making or clinical deployment.Source: §I · pp. 1–2
  • Unlike entropy, margin, BALD-style uncertainty, and diversity criteria, the proposed signals examine how local changes to question representations affect the answer boundary.The paper treats these rules as operational ways to characterize when perturbation sensitivity helps or fails, not as a universally optimal selector.Source: §I · p. 1
  • RSP combines prediction changes with non-flipping margin contraction, while RSP-DROP ranks by contraction alone after the same entropy prefilter.The comparison tests whether adding Flip-prioritized examples improves adaptation relative to continuous-drop ranking.Source: §I · p. 1
  • The study compares the selectors across five medical QA datasets and budgets of 64, 128, and 256 examples, alongside uncertainty, diversity, and auxiliary-text-aware comparators.It also analyzes selected-set differences, score support, and entropy-prefilter ablations.Source: §I · p. 1
  • This study replaces the prior workshop study’s labeled-development coefficient search with automatic RMS scale balancing and adds RSP-DROP and broader retrospective analyses.Both studies use the same five benchmark families and MedGemma backbone, so cross-paper numerical differences do not isolate the effect of automatic scaling.Source: §I · pp. 1–2

II. RELATED WORK

  • The related-work comparison includes explanation-aware selectors that score auxiliary text, unlike the proposed question-and-options-only rules.The cited explanation-aware methods are RSR-, ASLEC-DROP-, and TEMP-inspired implementations [11], not exact reproductions.Source: §II · p. 2

III. RSP AND RSP-DROP

The selectors choose examples from a fixed-size bank using question-and-options model behavior before labels or explanatory text enter selection. They derive sensitivity signals from Gaussian perturbations and changes in the clean answer prediction or margin.

A. Problem Setup

  • A selector retains B examples using question-and-options input alone; labels and auxiliary text are withheld until selection is frozen.Retained examples later include auxiliary text during training, with loss only on final-answer tokens; evaluation again uses questions and options without auxiliary text.Source: A. Problem Setup · p. 2

B. Shared Perturbation Signals

  • Each candidate’s final decoder-block output tensor receives independent absolute Gaussian noise across sequence positions and hidden dimensions.The scoring setup starts from the unadapted MedGemma checkpoint without a warm-up adapter.Source: B. Shared Perturbation Signals · p. 2
  • Because later normalization and the language-model head are token-wise, only perturbation at the final prompt position affects the restricted answer probabilities.The score uses next-token logits for legal option letters, rather than probabilities for full option-text sequences.Source: B. Shared Perturbation Signals · p. 2
  • A Flip occurs when a perturbed prediction differs from the clean prediction; the other signal measures contraction of the clean winner’s margin without a prediction flip.The clean prediction is the highest-probability legal answer option.Source: B. Shared Perturbation Signals · p. 2

C. RSP: Automatic RMS Scale Balancing

RSP balances prediction-change frequency and non-flipping margin contraction by scaling their empirical magnitudes within an entropy-prefiltered candidate pool. The coefficient is computed automatically from unlabeled scores before top-B selection.

C. RSP: Automatic RMS Scale Balancing

  • Flip is a frequency in increments of 1/K, while Drop is a probability-margin contraction, so their empirical scales differ.The method first retains the min(4B, N) highest-entropy examples as its candidate pool and computes each component’s root-mean-square scale there.Source: C. RSP: Automatic RMS Scale Balancing · p. 3
  • When the Drop RMS magnitude exceeds ε, λ0 scales Drop to approximately match Flip’s RMS magnitude, without guaranteeing equal influence on the ranking.The coefficient is computed separately for each dataset–budget–acquisition-seed pool after prefiltering, then frozen before top-B selection; it uses neither labels nor auxiliary text and requires no adapter fitting or development evaluation.Source: C. RSP: Automatic RMS Scale Balancing · p. 3
  • For a zero coefficient, candidates are ranked directly by the Drop component, with exact score ties resolved by original-pool index.The implementation tests whether aF is at most 10^-8 to identify this case in the reported prefiltered settings.Source: C. RSP: Automatic RMS Scale Balancing · p. 3
  • All 45 dataset–budget–seed records had positive coefficients and no all-candidate score collapse, with unique priority counts ranging from 89 to 402.The positive-coefficient expression used by the retained historical implementation gives the same scores on these runs.Source: C. RSP: Automatic RMS Scale Balancing · p. 3

D. RSP-Drop: Direct Contraction Ranking

RSP-DROP ranks candidates by non-flipping margin contraction within the shared entropy-restricted pool, without a mixing coefficient or additional scoring model.

D. RSP-Drop: Direct Contraction Ranking

  • RSP-DROP ranks candidates by contraction alone within the same entropy-restricted pool used by RSP.It retains the top B candidates, breaking exact ties by original pool index.Source: D. RSP-Drop: Direct Contraction Ranking · p. 3
  • A candidate that flips on every perturbation pass receives a contraction score of zero because the score divides by all K passes.The score therefore ranks non-flipping contractions rather than rewarding prediction changes.Source: D. RSP-Drop: Direct Contraction Ranking · p. 3
  • Table I lists the datasets and fixed study partitions, with the MedExpQA test count excluding four pool-overlap questions.Source: Table I · p. 3

E. Frozen Acquisition Protocol

The acquisition protocol fixes scoring and ranking choices before selection, while dataset partitions and overlap handling define the retrospective evaluation scope.

E. Frozen Acquisition Protocol

  • The frozen acquisition protocol uses K = 8 perturbations, the final decoder block, σ = 0.02, and an entropy-prefilter multiplier of four.It evaluates budgets B = 64, 128, and 256, with B = 128 primary; membership is ranked by score, while training restores pool-index order.Source: E. Frozen Acquisition Protocol · p. 3

A. Datasets and Auxiliary-Text Provenance

  • All normalized dataset records have nonempty questions, options, labels, and auxiliary text, and their UID sets are disjoint.UID disjointness does not establish independence from backbone training or compliance with original benchmark splits.Source: A. Datasets and Auxiliary-Text… · p. 3
  • Auxiliary-text sources differ by dataset, including answer rationales, human-validated explanations, abstract conclusions, and expert explanations.The specified fields are AfriMed-QA answer_rationale, English MedExpQA full_answer_no_ref, MedExQA’s first human-validated explanation, PubMedQA long_answer, and MedMCQA exp.Source: A. Datasets and Auxiliary-Text… · p. 3
  • Partition construction varies across datasets, including reshuffled PubMedQA records and a MedExQA study test set that is not an untouched official test.AfriMed-QA, MedExQA, and PubMedQA deduplicate normalized questions before internal splits; MedExpQA maps official splits before filtering.Source: A. Datasets and Auxiliary-Text… · p. 3
  • MedExpQA’s four overlapping test questions were removed post hoc, but partitions were not rebuilt and models were not retrained.The other four datasets had no recovered exact normalized-question cross-split intersection.Source: A. Datasets and Auxiliary-Text… · p. 4
  • MedGemma’s documented post-training includes four benchmark families, so these experiments assess continued adaptation rather than learning in wholly unseen domains.Exact example overlap cannot be established without the unreleased item manifest.Source: A. Datasets and Auxiliary-Text… · p. 4

B. Compared Selectors

The comparison distinguishes question/options-only selectors from auxiliary-text-aware comparators and holds the downstream training recipe fixed across methods.

B. Compared Selectors

  • Random@4B samples uniformly from the same top-min(4B, N) entropy pool used by the proposed methods, whereas Entropy retains its most entropic items.The matched random control is not unrestricted full-bank random when N > 4B.Source: B. Compared Selectors · p. 4
  • RSR-, ASLEC-DROP-, and TEMP-inspired comparators read auxiliary text during selection, so they are not information-matched controls for question-only scoring.A shared retained-set budget and training recipe does not equalize acquisition cost or information access.Source: B. Compared Selectors · p. 4

C. Training and Evaluation

  • Training uses two epochs and answer-letter targets, while evaluation selects among legal next-letter logits and reports Accuracy and Macro-F1 across three branch seeds.For stochastic selectors, each branch seed determines both acquisition randomness and downstream training; original prediction files lack test UIDs, so positional alignment relies on frozen split order.Source: C. Training and Evaluation · p. 5
  • Scoring time is unavailable, and requested GPU classes do not establish actual devices or precision.Both selectors require entropy scoring, and cache reuse is not zero cost.Source: C. Training and Evaluation · p. 5

D. Computational Resource Inventory

The results table covers the B = 128 benchmark comparison, while the resource inventory records selected per-run measurements and reconstructed token counts.

D. Computational Resource Inventory

  • Scoring time is unavailable, and recorded resource data are descriptive rather than controlled timing or end-to-end efficiency comparisons.Entropy-cache construction, perturbation scoring, and sorting or sampling times cannot be separated.Source: D. Computational Resource Inventory · p. 5

V. RESULTS

  • Table III reports test Accuracy and Macro-F1 at B = 128, with a dataset-macro average equally weighting five dataset means.Bold indicates the largest mean separately for each metric and column, not statistical significance.Source: Table III · p. 6
  • The reported benchmark comparison includes study-specific selector implementations, with some methods using auxiliary text and others restricted to questions and options.MedExpQA excludes four pool-overlap test questions, leaving 121 items.Source: §V · p. 6

A. RQ1: Overall Performance at the Primary Budget

At B = 128, RSP-DROP has the highest reported dataset-averaged Accuracy and Macro-F1 among the selectors, while results vary by dataset. Both sensitivity selectors exceed entropy-pool-matched random selection in dataset-averaged Macro-F1, but the comparison is within the entropy-restricted pool.

A. RQ1: Overall Performance at the Primary Budget

  • 53.60/48.28 (Accuracy/Macro-F1) for RSP-DROP and 53.00/47.55 for RSP are the highest reported dataset averages at B = 128.The averages weight the five evaluated benchmarks equally.Source: A. RQ1: Overall Performance at the… · p. 6
  • Dataset-level leaders differ: RSP-DROP leads on MedExpQA and MedExQA, while other methods lead on AfriMed-QA, PubMedQA, and MedMCQA.RSP has the highest AfriMed-QA accuracy, but SIFT-inspired has slightly higher Macro-F1; BatchBALD-inspired and RSR-inspired lead on PubMedQA and MedMCQA, respectively.Source: A. RQ1: Overall Performance at the… · p. 6

B. RQ2: Comparison with Pool-Matched Random Selection

  • At B = 128, dataset-averaged Macro-F1 is 45.11 for Random@4B, 47.55 for RSP, and 48.28 for RSP-DROP.These are descriptive improvements of 2.44 and 3.17 percentage points over the pool-matched control; the comparison tests sensitivity reranking within the entropy-restricted pool.Source: B. RQ2: Comparison with Pool-Matched… · p. 6
  • The pool-matched comparison does not test unrestricted random selection on the three datasets whose entropy pools exceed the retained budget.At B = 128, MedExpQA and MedMCQA have pools no larger than 4B, so their matched control is full-pool uniform random.Source: B. RQ2: Comparison with Pool-Matched… · pp. 6–7

C. RQ3: Component Utility and Selection Behavior

RSP adds prediction-change frequency to contraction-based ranking, and the audited selected sets show that this changes which candidates are included. Across budgets, RSP-DROP has higher mean Macro-F1, but differences vary by dataset and seed and do not establish a universal ordering.

C. RQ3: Component Utility and Selection Behavior

  • Adding Flip changes selected-set membership by introducing a distinct candidate group, rather than merely refining contraction ranking.Across 15 audited B = 128 comparisons, RSP-exclusive samples have Fi = 1 and Di = 0, while RSP-Drop-exclusive samples have Fi = 0 and Di > 0.Source: C. RQ3: Component Utility and… · p. 7
  • In the MedMCQA B = 128, seed-13 case, the two selectors share 102 of 128 examples, while their exclusive groups have clean-margin means of 0.098 and 0.707.The contrast describes selected examples, not the cause of downstream performance differences or a rule for choosing a selector.Source: C. RQ3: Component Utility and… · p. 7
  • No selection-cutoff ties occur at B = 64 or 128; at B = 256, both rules have zero-score ties in all datasets except AfriMed-QA.RMS balancing matches empirical component scales, but does not calibrate downstream utility.Source: C. RQ3: Component Utility and… · p. 7

D. RQ4: Sensitivity to the Retained-Set Budget

  • 42.03, 48.28, and 50.43 versus 41.41, 47.55, and 50.37 are the dataset-averaged Macro-F1 means for RSP-DROP versus RSP at B = 64, 128, and 256.Both sensitivity rules exceed pool-matched random on averaged Accuracy and Macro-F1 at each budget, but the B = 256 means are especially close and are not an established or universal ordering.Source: D. RQ4: Sensitivity to the… · p. 7
  • At B = 128, paired RSP-DROP-minus-RSP Macro-F1 differences are +1.82, +0.44, and −0.08 percentage points across the three seeds.At B = 256, the mean difference is +0.06 pp with SD 1.67 pp; dataset-level directions also vary across budgets.Source: D. RQ4: Sensitivity to the… · p. 7
  • Cross-budget comparisons do not isolate selection effects because fixed two-epoch training increases optimizer updates and token exposure as B grows.Within-budget comparisons hold retained-set size and recipe fixed.Source: D. RQ4: Sensitivity to the… · p. 7

E. RQ5: Effect of Entropy Prefiltering

The full-pool ablation finds that entropy prefiltering is not uniformly helpful or harmful. Its effects vary by method, metric, dataset, and budget, and the comparison concerns implemented procedures rather than statistical significance.

E. RQ5: Effect of Entropy Prefiltering

  • The ablation compares only nine dataset–budget settings where the top-4B prefilter removes candidates.When min(4B, N) = N, selectors produce the same selected UID order, so separate training runs cannot identify a prefiltering effect.Source: E. RQ5: Effect of Entropy Prefiltering · pp. 7–8
  • Prefiltering increases Accuracy in five of nine active settings and Macro-F1 in six; at B = 128, mean Macro-F1 increases for both methods on all three active datasets.At B = 256, the only active comparison is AfriMed-QA, where both methods favor the full pool on both metrics.Source: E. RQ5: Effect of Entropy Prefiltering · p. 8
  • The largest reported gains and losses differ by method, dataset, metric, and budget, so entropy restriction is neither uniformly helpful nor uniformly harmful.For example, RSP-Drop's largest increases are on MedExQA at B = 128, while its largest decreases are on PubMedQA at B = 64.Source: E. RQ5: Effect of Entropy Prefiltering · p. 8

VI. LIMITATIONS AND SCOPE

Across the evaluated budgets, both selectors retain positive dataset-averaged differences over pool-matched random selection, but performance varies and the study does not establish a universal choice rule. The evidence is retrospective and bounded to one backbone and the evaluated benchmarks.

VI. LIMITATIONS AND SCOPE

  • The selectors and the single no-update reference do not isolate explanation conditioning because training includes auxiliary explanations while evaluation omits them, without a matched answer-only control.This train–test context shift is shared by the compared adaptation protocols.Source: §VI · p. 8
  • Automatic λ0 balances component scales but does not establish downstream usefulness; finite-pass Flip frequencies are coarse and σ is not normalized to hidden-state magnitude.The study uses one backbone, one LoRA recipe, three seeds, and three budgets; retrospective analyses on viewed tests are not confirmatory.Source: §VI · p. 8
  • The five-dataset analysis mixes backbone-overlap conditions, and MedExpQA’s post-hoc pool-overlap filter is not a full pool/dev/test reconstruction or retraining.One within-test duplicate remains, and exact item overlap with benchmark training families in MedGemma is unresolved.Source: §VI · p. 8
  • The study does not demonstrate transfer across model revisions, other backbones, larger budgets, languages, institutions, or newly collected data.The five-dataset analysis mixes backbone-overlap conditions.Source: §VI · pp. 8–9
  • The benchmarks do not measure patient outcomes, harmful clinical errors, calibration, fairness, or deployment readiness, and neither selector assesses explanation faithfulness or factuality.Future clinical use would require appropriate governance and independent clinical validation.Source: §VI · p. 9

VII. CONCLUSION

  • At B = 128, RSP and RSP-DROP score 47.55 and 48.28 dataset-averaged Macro-F1, respectively, versus 45.11 for pool-matched random selection.Both rules retain positive dataset-averaged differences over that control across the three evaluated budgets, while relative performance varies across datasets and metrics.Source: §VII · p. 9
  • Adding Flip exchanges contraction-selected examples for a group with a different clean-margin profile, while entropy restriction can help or hurt depending on setting.These findings treat component choice, candidate-pool restriction, and retained budget as distinct aspects; they do not establish a universal selector-choice rule.Source: §VII · p. 9
  • Assessing transfer beyond this retrospective study requires evaluation on independent data and additional backbones, with fully tracked inference and training configurations.The conclusion identifies these as needed for evaluating transfer.Source: §VII · p. 9
Loading 2610.33440v1…