Source-linked AI summary
Dissociating Answerability from Extraction in LLM Question Answering over Radiology Reports
Priyanshu Sinha, Kelsey McDonald
TL;DR
RadQA’s overall F1 combines the decision to answer with the quality of the extracted span. The paper evaluates these separately and finds that an unanswerable demonstration mainly shifts answer decisions, while similar-F1 models can differ substantially in false-answer rates.
Problem
RadQA reports overall F1, which combines whether a report contains an answer with how well the model extracts its span.
Source: §6 · p. 8Method
The authors report FAR and OAR beside span F1 and decompose F1 changes into extraction and answerability terms, including a test of which answers new abstentions remove.
Source: §1 · p. 2Results
The unanswerable demonstration raised over-abstention in 12 of 13 models, and three-quarters of withdrawn answers had F1 ≥0.5, while extraction was flat under token and RadGraph F1.
Source: §1 · p. 2Takeaways & Limitations
For extractive clinical QA, the authors recommend reporting FAR and OAR beside span F1 and checking prompt or model changes with the decomposition and selectivity test.
Source: §5 · p. 8, §6 · p. 8FAR and OAR measure agreement with physician annotators’ answerability labels, not whether an answer would mislead a clinician.
Source: Limitations · p. 8
Abstract
from arXiv · showhide
Answering questions from radiology reports involves two decisions: whether the report contains an answer and which span gives it. A false answer presents an unsupported claim as fact; an over-abstention hides what the report says. Yet RadQA results are reported as a single F1 that merges both. We propose an evaluation method that separates them, reporting false-answer and over-abstention rates beside F1 and splitting F1 changes into extraction and answerability terms. We apply it to 13 LLMs on the RadQA test set, zero-shot and with three cumulative demonstration sets, the last adding an unanswerable example. Models with similar F1 differed more than twofold in false-answer rate. The unanswerable example changed when models answered, not how well they extracted: 12 of 13 models abstained more often, and three-quarters of the answers they withdrew had F1 $\geq$ 0.5. Within models, false answers showed no clear association with extraction quality, but acting on the decision itself helped: keeping an answer only when enough other models also answered lowered the zero-shot false-answer rate by 4 points without a detectable F1 change. Because the two decisions can change independently, we suggest radiology QA evaluations report them separately, not through F1 alone.
1 Introduction
Radiology QA requires models to decide both whether a report answers a question and which span supplies the answer, but overall F1 merges these decisions. The paper separates answerability from extraction and finds that prompting can shift answer decisions without improving extraction.
1 Introduction
- The proposed evaluation reports FAR and OAR beside span F1 and decomposes F1 changes into extraction and answerability terms, with a test of which answers new abstentions remove.The study applies the method to 13 LLMs across zero-shot and three cumulative demonstration sets.Source: §1 · p. 2
- The unanswerable demonstration raised over-abstention in 12 of 13 models; its −0.0126 F1 change was almost entirely answerability shift (−0.0127), with extraction flat under both token and RadGraph F1.For answerable demonstrations, the decomposition depended on the span metric.Source: §1 · p. 2
- Three-quarters of 272 withdrawn answers had F1 ≥0.5, while models gave a net 55 fewer false answers, most from one model.Source: §1 · p. 2
- Eleven models within 0.07 zero-shot F1 differed more than twofold in false-answer rate, and within-model analyses found no association between extraction quality and false answers.The cross-model F1–FAR correlation is largely built into the metric.Source: Figure 1 · p. 2, §1 · p. 2
- A consensus gate lowered zero-shot FAR by 4 points with no detectable F1 change.Source: §1 · p. 2
2 Related Work
Prior clinical and radiology QA work established extractive datasets and models, while related evaluation research addressed abstention and demonstration sensitivity. This paper distinguishes answer decisions from extraction in that setting.
2 Related Work
- Clinical QA progressed from domain-adapted encoders to radiology-specific models, while RadQA evaluations reported span F1 and EM rather than separating answer decisions from extraction.RadQA questions were written from clinical referrals without seeing report text, so unanswerable questions arise naturally.Source: §2 · pp. 2–3
- SQuAD 2.0 reports answerable and unanswerable breakdowns, but HasAns F1 still mixes span quality with abstention, unlike this paper’s extraction term.Source: §2 · p. 3
- Prior work finds in-context learning sensitive to demonstration choice, order, and label balance; here, the unanswerable example changes abstention frequency, not extraction quality.Source: §2 · p. 3
- The study complements token F1 with RadGraph F1 and uses report-clustered inference for evaluation reliability.RadGraph represents entities and relations in radiology reports.Source: §2 · p. 3
3 Methods
The study evaluates 13 LLMs on RadQA using fixed cumulative demonstrations and separates answer-decision outcomes from span extraction. Its analyses include a cross-fitted consensus gate and report-level statistical inference.
3.1 Data
- The RadQA test split contains 614 questions over 104 reports: 154 are unanswerable, while 460 answerable questions come from Findings or Impression sections.The analysis scores one prediction per question.Source: §3.1 · p. 3
3.2 Models and Inference
- The evaluation covers 13 models from six families and asks each to return an exact, minimal report span or mark the question unanswerable.Calls still failing after three attempts are scored as abstentions; this applied to 3 of 31,928 predictions.Source: §3.2 · p. 3
3.3 Demonstration Conditions
- Four cumulative conditions add an answerable Findings example, an answerable Impression example, then an unanswerable example; effects are attributed to the set, not solely its size.The same fixed examples are used across models and test questions; the unanswerable example is placed last.Source: §3.3 · pp. 3–4
3.4 Scoring and Decomposition
- Scoring follows SQuAD 2.0: answerability disagreement scores 0, mutual unanswerability scores 1, and otherwise span overlap determines F1 or EM; FAR and OAR measure answer and abstention rates against annotator labels.The study also uses answerable and extraction F1, and RadGraph F1 as an entity-level complement to token F1.Source: §3.4 · p. 4
- The F1 decomposition separates span changes when answer decisions remain fixed from score changes when decisions shift, and tests whether new abstentions remove weak or valid answers.Within-model analyses relate report-level extraction quality and answering propensity to answers on unanswerable questions, controlling for report difficulty.Source: §3.4 · p. 4
3.5 Consensus Gate
- The consensus gate keeps a model’s span only when enough other models also answer, replacing other answers with abstentions; thresholds are cross-fitted across report halves.Thresholds are selected per model or pooled across models, with 200 repeated splits.Source: §3.5 · p. 5
3.6 Statistical Analysis
- Inference is clustered by report, with cluster-bootstrap confidence intervals and paired report-level tests; the analyses also use GEE and descriptive Spearman correlations.The report-level bootstrap uses 10,000 replicates, or 2,000 for decomposition terms.Source: §3.6 · p. 5
4 Results
Across 13 models, similar zero-shot F1 coexisted with substantially different false-answer rates. Demonstrations changed answer decisions and extraction in different ways, while consensus gating reduced false answers without a detectable F1 change.
4.1 Zero-Shot Performance
- Zero-shot FAR ranged from 0.117 to 0.260 even though 11 of 13 models had F1 between 0.669 and 0.738.Overall FAR differences were significant (Cochran’s Q = 69.2, df = 12, p < 10−5).Source: §4.1 · p. 5
- No model reached RadQA-DPO’s F1 of 0.775 in any condition, and every model scored below inter-annotator agreement under all three metrics.At zero-shot, models reached 81% of human agreement under token F1 and 73% under RadGraph F1; 84.9% of EM-mismatches shared clinical entities with the correct answer.Source: §4.1 · p. 5
4.2 What Demonstrations Change
- Adding the unanswerable demonstration lowered F1 in 12 of 13 models, raised mean OAR from 0.105 to 0.142, and lowered mean FAR from 0.171 to 0.143.Answerable demonstrations raised F1 in 10 of 13 models (Wilcoxon over model means, p = 0.048).Source: §4.2 · pp. 5–6
- A −0.0126 token-F1 drop after adding the unanswerable example was almost entirely answerability shift (−0.0127), with extraction flat (+0.0001).The same pattern held under RadGraph F1: −0.0093 of the −0.0101 drop was answerability shift; extraction was flat.Source: §4.2 · p. 6, Figure 2 · p. 6
- For answerable demonstrations, 64% of the +0.0136 token-F1 gain was answerability shift, while 73% of the +0.0235 RadGraph F1 gain was extraction.Without radiologist adjudication, the authors cannot say which metric better reflects clinical correctness.Source: §4.2 · p. 6
4.3 Which Answers Do New Abstentions Remove?
- Three-quarters of the 272 newly declined answers had prior F1 ≥0.5, and the selectivity difference was not statistically distinguishable from zero.The confidence interval included zero, though modest selectivity could not be excluded.Source: §4.3 · p. 7, Table 2 · p. 7
4.4 How False Answers Relate to Accuracy
- Across models, F1–FAR correlation was not distinguishable from zero with demonstrations; within models, extraction quality was not associated with false answers.The within-model estimates were OR 0.94–1.04, but noisy measurement means a modest effect could not be excluded.Source: §4.4 · p. 7
4.5 Consensus Gating of the Answer Decision
- The consensus gate lowered held-out mean FAR by 0.040 [0.026, 0.057] with no detectable F1 change (+0.005 [−0.000, +0.009]).It raised OAR by 0.010; the effect was concentrated in high-FAR models and was smallest after adding the unanswerable demonstration.Source: §4.5 · p. 7
5 Discussion
The discussion argues that span F1 alone misses clinically distinct false-answer and over-abstention trade-offs, and that answerability can shift independently of extraction. It recommends measuring both decisions and checking prompt changes, while noting the evidence is limited to one unanswerable demonstration.
5 Discussion
- False answers and over-abstentions have different consequences, and FAR and OAR carry information that F1 rankings do not.The relative cost depends on the clinical use, such as structured-field pre-filling versus answering a clinician’s patient-specific question.Source: §5 · p. 7
- Across models, the unanswerable demonstration withdrew about three good answers (F1 ≥0.5) for each false answer removed; LLaMA 4 Scout accounted for 34 of the net 55 fewer false answers.FAR fell in only 7 of 13 models, while OAR rose in 12 of 13.Source: §5 · pp. 7–8
- The observed threshold shift is evidence about one unanswerable demonstration; multiple example types and count-matched answerable controls are needed to test generality.The authors propose testing clean absences and near misses with their own rationales.Source: §5 · p. 8
- Within models, extraction quality showed no association with answering unanswerable reports, while an external consensus gate lowered FAR without detectable F1 change.The per-report measure is noisy, so modest effects cannot be excluded; the gate is not a deployment recipe and requires separate calibration and local recalibration.Source: §5 · p. 8
- Demonstrations changed model-mean F1 by at most 0.014, while the remaining gap to RadQA-DPO and annotators lay in extracted spans and entities.The authors present this as a hypothesis about span-level supervision; where exact text matters, these models do not yet substitute for a fine-tuned extractor or human review.Source: §5 · p. 8
- The authors recommend reporting FAR and OAR beside span F1, using multiple span metrics, and matching the approving metric to the output’s clinical use.Neither token nor RadGraph F1 has been checked against radiologists on short spans, so local clinical review remains necessary.Source: §5 · p. 8
6 Conclusion
The conclusion separates answerability from extraction in RadQA evaluation. The tested unanswerable demonstration primarily shifted answer decisions, while similar zero-shot F1 scores concealed substantial variation in false-answer rates.
6 Conclusion
- The unanswerable demonstration raised over-abstention in 12 of 13 LLMs, and three-quarters of withdrawn answers had F1 ≥0.5.At similar zero-shot F1, models differed more than twofold in false-answer rate; within models, extraction quality showed no association with false answers.Source: §6 · p. 8
Limitations
The evaluation is limited to one dataset and one fixed demonstration set, and its automated agreement measures were not clinically adjudicated. Additional limits affect cross-model correlations, consensus-gate validation, external comparisons, and possible training-data contamination.
Limitations
- FAR and OAR reflect agreement with physician answerability labels, not whether answers mislead clinicians; no radiologist reviewed correctness or clinical consequences.The answerability labels are imperfect, and token and RadGraph F1 have not been validated against radiologists on short spans.Source: Limitations · pp. 8–9
- All results use RadQA from one institution and report style, with one fixed demonstration set and a single near-miss unanswerable example.Because each condition adds a new demonstration type, the effects of a third demonstration cannot be separated from those of an unanswerable example.Source: Limitations · p. 9
- Cross-model correlations rely on 13 models, and several are sensitive to individual models.The consensus gate is also built from and evaluated on the same 13 models, with its threshold cross-fitted within the test set rather than set on a separate calibration split.Source: Limitations · p. 9
- Human, BERT-MIMIC, and RadQA-DPO comparisons use results from other papers and are descriptive.The LLM scoring is slightly more lenient by crediting the best of several predicted spans, and Soni et al. do not specify the reference used for each annotator.Source: Limitations · p. 9
- The authors did not test for contamination, and training data are undisclosed for most evaluated models.Comparisons between conditions are less affected because they share the same test items.Source: Limitations · p. 9
Ethical Considerations
The study used credentialed, de-identified RadQA data under PhysioNet’s license and routed report text only through configured services with zero data retention. The authors do not release report-derived text or outputs and state that the evaluated systems are not validated for clinical use.
Ethical Considerations
- RadQA derives from de-identified MIMIC-III records accessed through PhysioNet credentialed access under its data license and data use agreement.Source: Ethical Considerations · p. 9
- Report text was sent to AWS Bedrock and Google Vertex AI with zero data retention confirmed, and was not stored, used for training, or subject to human review.No consumer chat interface was used.Source: Ethical Considerations · p. 9
- The released code contains no report text or reconstructive derived data; reproducing demonstrations requires credentialed PhysioNet access.Demonstrations are drawn at run time from the training split using a fixed seed and question identifiers.Source: Ethical Considerations · p. 9
- An AI assistant helped run experiments and write the manuscript; the authors verified the code, results, and text and take responsibility for the content.Source: Ethical Considerations · p. 9
- The paper proposes an evaluation method, not a clinical system, and none of the evaluated configurations has been validated for clinical decision-making.The authors state that deployment would require local validation and clinical adjudication; passing this evaluation alone is not evidence of clinical safety.Source: Ethical Considerations · p. 9
A Models and Inference Details
The evaluation used provider-specific APIs for some models and fixed, cumulative demonstrations drawn from the RadQA training split. Each demonstration was paired with a rationale written for its question type.
A Models and Inference Details
- Bedrock models were called through the Bedrock Converse API, while Gemini models were called through Vertex AI.Source: A Models and Inference Details · p. 12
- Demonstrations were drawn from the RadQA training split after excluding reports longer than 4,000 characters and shuffling with a fixed seed.The same selected Findings, Impression, and unanswerable examples were used in every prompt for their corresponding conditions.Source: A Models and Inference Details · p. 12
- The three demonstrations were an answerable Findings question, an answerable Impression question, and an unanswerable question, added in that order.Each was paired with a fixed rationale for its question type, rather than a model-generated rationale.Source: A Models and Inference Details · p. 12
B Statistical Details
The analyses use report-clustered uncertainty and tests suited to paired conditions, model differences, and question-level predictors. They also check whether the reported no-overlap pattern depends on chosen thresholds.
B Statistical Details
- Paired report-mean condition effects use Wilcoxon signed-rank tests with Benjamini–Hochberg correction, while model-mean effects use Wilcoxon tests across 13 models.Model-level confidence intervals use report-level cluster bootstrap resampling.Source: B Statistical Details · p. 12
- FAR differences across models are tested with Cochran’s Q, then each model is compared with the lowest-FAR model on the same 154 unanswerable questions.The pairwise comparisons use exact McNemar and report-level signflip permutation tests with within-condition correction.Source: B Statistical Details · p. 12
- Within-model false-answer predictors are estimated with logistic GEE, including extraction quality, answering propensity, model effects, and report difficulty.Selectivity uses Gaussian GEE; pooled fits include model fixed effects and are checked by leaving out one model at a time.Source: B Statistical Details · p. 12
- Cross-model correlations use Spearman ρ with bootstrap resampling over models and leave-one-model-out p-values.Source: B Statistical Details · p. 12
- Table 3 lists evaluated models and developer announcement dates; Table 7 reports mean input and output tokens by condition.Source: Table 3 · p. 13
- Table 4 reports F1, FAR, and OAR across cumulative demonstration conditions, adding Findings, Impression, and then an unanswerable example.Source: Table 4 · p. 13
- The no-overlap rate declines from k=0 to +F+I at F1 thresholds of 0.05, 0.10, and 0.20; the reported rate at 0.10 is 1.88%.Changing the EM-mismatch threshold from 0.4 to 0.6 does not affect the decline.Source: B Statistical Details · p. 13
C Additional Results
The additional-results tables describe score decomposition, model-level FAR comparisons, consensus-gate effects, and analyses of cross-model and within-model patterns. Token counts have stated comparability and reporting caveats.
C Additional Results
- Table 5 decomposes mean score changes into extraction and answerability terms, which sum to the total, and defines newly abstained and newly false-answered questions.Its estimates pool 13 models and include report-cluster bootstrap intervals.Source: Table 5 · p. 14
- Table 6 reports FAR by model and condition; all Cochran’s Q tests have p ≤0.0003, and stars mark FAR above the lowest-FAR reference under both tests.Source: Table 6 · p. 14
- Table 7 token means are comparable across conditions within a model but not across model families; LLaMA 4 input counts are approximate.The caption attributes inflated means to retried calls, while noting that answers are unaffected.Source: Table 7 · p. 14
- Table 8 reports Spearman correlations between FAR and other measures across the 13 models; its caption defines answerable F1 and extraction F1.Source: Table 8 · p. 15
- Table 9 models within-model false answers using extraction, answering propensity, and report difficulty, with model effects and report clustering.The table reports odds ratios per standard deviation and uses 1,596–1,616 model-question instances per condition across 52 reports.Source: Table 9 · p. 15
- Table 10 reports held-out model-mean changes from a cross-fitted consensus gate relative to ungated models, with intervals over 200 report splits.The pooled-t analysis uses one threshold across models, with modal t = 4.Source: Table 10 · p. 15