Source-linked AI summary
Can Language Models Learn When to Abstain? Policy Learning and Calibrated Answer Selection
Juan Terven, Diana-Margarita Córdova-Esparza, Ruben Gomez, Carlos Miranda Rocha, Julio-Alejandro Romero-González, Edgar Chavez-Urbiola
TL;DR
Language models can produce plausible but incorrect factual answers, so useful abstention requires choosing when to answer rather than refusing everything. The paper compares policy training with a learned answer selector on Qwen3-14B. The selector improves answer selection on TriviaQA, while results on NQ-Open and the tested RL changes leave reliable cross-dataset abstention unresolved.
Problem
The challenge is selective abstention: determine whether training the answering model or a separate selector improves on calibrated confidence filtering and transfers across thresholds and datasets.
Source: §1 · p. 1Method
The study compares reinforcement-learning interventions and a learned answer selector on Qwen3-14B across three seeds, two held-out thresholds, and TriviaQA and NQ-Open.
Source: §1 · p. 1Results
The selector improves mean coverage and reduces pooled exact-match selective risk on TriviaQA, but does not establish positive utility on NQ; tested RL comparisons remain inconclusive.
Source: §6 · p. 10Takeaways & Limitations
The findings support answer selection under the tested distribution while leaving reliable transfer across datasets an open empirical problem.
Source: §6 · p. 10The study covers one model family, short English QA, and three seeds, and its remaining responses have not been reviewed semantically.
Source: §5 · p. 10
Abstract
from arXiv · showhide
A useful language model should know when to answer and when to say "I don't know." Can we improve that decision by training the model itself or by training a separate component to check its answers? We study both approaches on Qwen3-14B for factual questions, using a score that rewards correct answers, penalizes mistakes, and allows abstention. Across three training seeds and two held-out confidence thresholds, a learned answer selector improves threshold-averaged utility over the original model's calibrated confidence filter. On TriviaQA, mean coverage increases from 20.2% to 30.9%, while the pooled error rate among scored answers decreases from 23.0% to 14.5% under exact-match grading. The tested reinforcement-learning changes do not establish consistent gains. On NQ-Open, the selector improves threshold-averaged utility over the filter but still performs worse than always abstaining. A partial human-reviewed audit preserves the TriviaQA finding. Our results support learning to select answers in this setting while showing that reliable abstention across datasets remains unresolved.
1 Introduction
The introduction frames abstention as useful only when it is selective: penalizing errors can make admitting uncertainty preferable to guessing. It compares training the answering model with training a separate answer selector, asking whether either improves on calibrated confidence filtering and transfers across thresholds and datasets.
Motivation
Research questions
- Selective abstention must balance answering with refusing: refusing every question avoids mistakes but provides no answers.The introduction contrasts training the answering model to abstain with keeping it fixed and using a separate selector, noting that existing evidence does not settle which suits a given model and compute budget.Source: §1 · p. 1
Study and contribution
- The study compares reinforcement-learning interventions with a learned answer selector on Qwen3-14B, using three seeds, two held-out thresholds, and TriviaQA and NQ-Open tests.Its clearest result is improved mean coverage and reduced pooled selective risk on TriviaQA; RL comparisons remain inconclusive, and the selector does not establish positive utility on NQ.Source: §1 · p. 1
2 Related Work
The related work connects factual reliability, uncertainty estimation, selective prediction, and calibration to the paper’s comparison of policy learning and answer selection. It motivates evaluating these approaches empirically while distinguishing the evidence and guarantees they provide.
2.1 Hallucinations and factual reliability
- Exact-match failures may reflect valid answers missing from reference aliases [6], while agreement-based detectors can miss consistent systematic errors.The study concerns short factual answers without retrieval; these grading and detection limits complicate interpreting errors as hallucinations or identifying their causes.Source: §2.1 · p. 2
2.2 Confidence estimation and selective prediction
- Confidence elicitation can be overconfident and uneven across tasks [14], while domain shifts challenge error prediction by calibrators [17].Conformal abstention [18] offers marginal error control under exchangeability assumptions, unlike the paper’s empirical logistic calibration after dataset change.Source: §2.2 · p. 2
2.3 Learning uncertainty and abstention
- Prior work trains models to express uncertainty [2] [19] or rewards answer-or-abstain decisions [3] [20], but differing targets and settings limit direct comparison.The paper’s policy experiments score the answer-or-IDK decision directly, and the cited approaches do not determine which intervention helps a particular model within this study’s training budget.Source: §2.3 · p. 2
2.4 Calibration and evaluation metrics
- Calibration, score learning, and release thresholds are distinct operations, while coverage, selective risk, and utility measure different aspects of answer selection.The paper compares established policy-training and selection choices with held-out thresholds, shared candidates, cross-dataset evaluation, and a measured local training budget; lexical grading can reject semantically valid answers [6].Source: §2.4 · p. 3
3 Methodology
The methodology compares policy updates with post-generation selection for factual question answering, using held-out thresholds and a partial audit to assess decision quality and grading sensitivity.
3.1 Task and decision rule
- The task scores correct answers, mistakes, and abstentions differently, with a threshold making mistakes more costly and answering preferable when correctness probability exceeds it.Correct answers earn 1 − τ, incorrect answers −τ, and abstentions 0.Source: §3.1 · p. 4
3.2 Model, data, and training budget
- Tests use 2,000 TriviaQA questions and 500 NQ-Open questions, with three seeds and primary thresholds 0.65 and 0.85 held out from policy training.Training thresholds are 0.60, 0.75, and 0.90; the selector uses the full 2,000-question training pool, while RL uses a reduced 504-question panel.Source: §3.2 · p. 4
3.3 Training the answering model
- Four RL arms vary reward-group size, threshold exposure schedule, and reward conditioning while holding questions, ordering, updates, and total generations identical.All arms have three training exposures and eight sampled responses per exposure; the group-size contrast changes the reward baseline, not the number of generated responses.Source: §3.3 · pp. 4–5
3.4 Selecting answers from a fixed generator
- The selector leaves generation fixed and releases a candidate only when its calibrated correctness probability exceeds the requested threshold.A separate LoRA selector learns exact-match correctness from question–candidate pairs, while a TRAIN logistic control transforms the original scalar score and cannot learn new features.Source: §3.4 · p. 5
3.5 Evaluation and uncertainty
- Utility, coverage, and selective risk measure decision performance, while ROC AUC and Brier score describe candidate-level score quality before release decisions.Utility is averaged across thresholds; pooled errors are divided by total scored answers, and Brier reductions alone do not establish better calibration.Source: §3.5 · pp. 5–6
3.6 Audit of automatic grading
- A stratified audit reviews 150 response units with AI-assisted labeling and one annotator reviewing every label, then applies reviewed labels to identical units across methods.Its bootstrap intervals exclude annotation and audit-sampling uncertainty, and the partial analysis does not estimate semantic accuracy over the whole test set.Source: §3.6 · p. 6
4 Results
Under exact-match grading, the learned selector increases TriviaQA coverage and lowers pooled error relative to the calibrated filter, but its NQ utility remains negative and the tested RL changes show no consistent gains. A partial human review preserves the TriviaQA result, with the audit limited to reviewed labels and sampled units.
4.1 Answer selection on TriviaQA and NQ-Open
- TriviaQA mean coverage rises from 20.2% for the original calibrated filter to 30.9% for the selector, while pooled error falls from 23.0% to 14.5%.Table 2 reports primary exact-match results averaged across thresholds and, for trained methods, three seeds.Source: §4.1 · p. 7, Table 2 · p. 7
- TriviaQA mean utility rises from 0.02418 to 0.04248; the selector beats TRAIN logistic after calibration by 0.01834, with higher utility at both held-out thresholds.The selector's utility improvement over the original filter is positive for each seed; most of the coverage increase occurs at τ = 0.85, where the filter returns only two correct answers.Source: §4.1 · p. 7
- On NQ-Open, selector mean utility improves from −0.03075 to −0.01197 but remains below always abstaining's zero; at τ = 0.85, it trails the filter.At τ = 0.85, selector utility is −0.00817 versus −0.00170 for the filter, based on one incorrect answer from the filter.Source: §4.1 · p. 7
- TriviaQA ROC AUC rises from 0.779 to selector mean 0.852, while calibrated Brier score falls from 0.189 to 0.154; NQ diagnostics improve too, without positive utility.These are descriptive comparisons on the same valid, non-abstaining candidates before answer thresholds are applied.Source: §4.1 · p. 7, Table 3 · p. 8
4.2 Reinforcement learning and computational cost
- All three TriviaQA RL comparisons have adjusted intervals including zero and seed-varying directions, so the results establish neither consistent gains nor equivalence.Figure 2 shows wide RL intervals; its selector comparisons favor the learned selector on average, although utility remains negative on NQ.Source: §4.2 · p. 7, Figure 2 · p. 8
- Raising the requested threshold from 0.65 to 0.85 reduces TriviaQA coverage by 2.03 points in C and 2.23 points in fixed-cost control D, even without reward conditioning.This does not show that conditioning the reward adds further control over abstention.Source: §4.2 · pp. 7, 9
- Recorded experiments took 35.44 serial task hours, including 23.68 hours for twelve RL runs and 1.88 hours for three selector runs.These are elapsed experimental-job times, not GPU-utilization or deployment-latency measurements; training budgets differed.Source: §4.2 · p. 9
4.3 Effect of the partial human review
- Among 150 reviewed response units, 40 exact-match errors were judged semantically correct; corrected labels preserve the TriviaQA selector advantage, but the audit remains conditional on reviewed samples.The analysis does not account for grading errors in unreviewed responses; on NQ, the review does not establish positive utility.Source: §4.3 · p. 9, Table 4 · p. 9
5 Discussion
The learned selector gives the strongest evidence for improving which answers to show, while the tested reinforcement-learning results remain inconclusive. Transfer, grading, and the study’s limited scope constrain the conclusions.
5 Discussion
- Across the two primary thresholds, the learned selector improves mean utility and coverage while reducing pooled selective risk on the same TriviaQA candidates.The comparison concerns practical outcomes; matched RL contrasts and shared-candidate filter comparisons support narrower inferences.Source: §5 · p. 9
- Unlike answer selection, the RL interventions change both generated answers and abstention decisions, and their training data and costs also differ from the selector’s.These differences limit what practical outcome comparisons can establish about the mechanism.Source: §5 · p. 9
- On NQ, improved confidence ranking can coexist with negative observed utility when calibration is learned on another dataset.The audit correction preserves the TriviaQA selector advantage, but remaining responses have not been reviewed semantically; the review was not independently replicated.Source: §5 · p. 10
- The study covers one model family, short English QA, and three seeds, while intervals do not resample shared training candidates or calibrators.The protocol was frozen locally rather than externally preregistered, and any future recipe selected after these results needs a new test set for independent confirmation.Source: §5 · p. 10
6 Conclusions
The selector most clearly improves useful abstention on TriviaQA, while the tested reinforcement-learning interventions remain inconclusive. The findings support answer selection under the tested distribution but leave reliable transfer unresolved.
6 Conclusions
- On TriviaQA, the selector increases mean coverage and reduces pooled exact-match selective risk relative to the original calibrated filter across the two primary thresholds.Its utility advantage remains after the observed audit corrections.Source: §6 · p. 10
- The tested reinforcement-learning interventions remain inconclusive, and neither better confidence scores nor partial corrections establish an advantage over always abstaining on NQ.Source: §6 · p. 10
- The findings support answer selection under the tested distribution, while reliable transfer remains an open empirical problem.Source: §6 · p. 10
A Earlier phases and evaluator evidence
The appendix distinguishes historical evaluations and audits from the present follow-up, rather than pooling their results. The historical conditioned model did not demonstrate utility superiority over the fixed-cost training or calibrated-filter alternatives.
A Earlier phases and evaluator evidence
- A separate evaluation reused 24 final adapters, three original models, and shared warmup on 2,000 new questions per dataset; those questions are excluded from the present tests.Source: A Earlier phases and evaluator evidence · p. 10
- The historical binary-reward control gave correct answers reward 1 and incorrect answers and abstentions reward 0; the conditioned control used the requested threshold, while evaluation used Uτ for every method.Source: A Earlier phases and evaluator evidence · p. 10
- Table 5 reports cost-sensitive evaluation utility at thresholds 0.60 and 0.80 for a different test population and training recipe, retaining those values as antecedents rather than pooling them with the follow-up.Source: Table 5 · p. 10
- The historical conditioned Qwen3-14B did not demonstrate utility superiority over fixed-cost training or the calibrated original filter.Its mean assigned training threshold was 0.716575, compared with the fixed reward parameter of 0.75, a confound addressed by the symmetric follow-up threshold set.Source: A Earlier phases and evaluator evidence · p. 10
- The historical audit documented 150 records from 136 questions, including eleven human-correct answers rejected by matching, and its corrections left the initial conditioned–binary conclusion positive under partial sensitivity analysis.The diagnostics neither replace human annotation nor establish population-wide evaluator accuracy.Source: A Earlier phases and evaluator evidence · pp. 10–11
B Per-seed contrasts
The per-seed intervals retain pairing across questions and thresholds, but uncertainty is conditional on a shared training bank and fixed calibration partition. The small seed count is particularly consequential for the RL contrasts.
B Per-seed contrasts
- Table 6 reports all three prespecified training seeds, with pairing preserved across questions and thresholds.Source: Table 6 · p. 11
- The five TriviaQA intervals form the nominal 95% family, while NQ intervals are descriptive despite using the same percentile level.Source: Table 6 · p. 11
- The selector’s uncertainty is conditional on the one shared TRAIN bank and fixed calibration partition, and the small number of seeds is particularly consequential for RL contrasts.All three prespecified seeds are retained; no model is chosen after test inspection.Source: B Per-seed contrasts · p. 11
C Compute and reproduction
The compute ledger reports projected durations for reinforcement-learning runs against an 80-hour planning target and records what artifacts are retained for reproduction.
C Compute and reproduction
- Table 7 sums outer job durations, including loading, without adding nested ledgers again.Rounded components may differ from the rounded total.Source: Table 7 · p. 11
- 89.36 hours were projected for 1,008 RL questions and 59.64 hours for 504, against an 80-hour planning target.The uniform reduction was locked before test inference; earlier phases add 40.50 recorded task-hours.Source: C Compute and reproduction · p. 11
- The code repository retains configurations, source locks, model and data manifests, aggregate results, and reproduction instructions.Large local weights, response-level data, audit annotations, and the manuscript are excluded; hash manifests identify retained artifacts.Source: C Compute and reproduction · p. 11
D Optimization details
The optimization setup uses BF16 LoRA for reinforcement learning and answer selection, with group-relative reward advantages for RL updates.
D Optimization details
- RL uses BF16 LoRA with rank 16 and alpha 32, targeting all linear modules, at learning rate 10^-5 and KL coefficient 0.04.Temperature is 0.8; prompts and responses are capped at 512 and 32 tokens, respectively.Source: D Optimization details · p. 11
- Each update uses two questions and 48 responses, while selector training uses microbatch four with accumulation four.RL uses microbatch four with accumulation twelve; evaluation is deterministic.Source: D Optimization details · p. 12
- Each response’s advantage is its reward minus the mean reward of the other responses in its comparison group.Reward standard-deviation scaling is disabled, and the dr_grpo token-loss reduction is used.Source: D Optimization details · p. 12
- Mixed answer/abstention groups are needed for the centered reward to retain a direct abstention-cost signal.When every response is an answer, changing τ subtracts a common reward constant and leaves the advantages unchanged; larger groups alone do not guarantee better optimization.Source: D Optimization details · p. 12
E.1 Primary evaluation populations
The primary evaluation compares selector and calibrated-filter decisions on full test populations, while diagnostic panels, prompt templates, and calibration controls specify the evaluation and fitting procedures.
E.1 Primary evaluation populations
- The primary results use full populations of 2,000 TriviaQA and 500 NQ-Open questions, averaging selector utility and coverage across three seeds and pooling selective risk.The original deterministic filter is evaluated once; exact-match results precede partial audit corrections.Source: §E.1 · p. 12, Table 8 · p. 12
- On TriviaQA, the selector has higher point-estimate utility than the calibrated filter at both primary thresholds; on NQ, it is higher at τ = 0.65 but lower at τ = 0.85.The NQ positive average contrast therefore does not hold at both thresholds; these comparisons are descriptive and add no significance tests.Source: §E.1 · p. 12
E.2 Common diagnostic question panels
- Diagnostic Tables 9 and 10 use the same 200 questions per dataset at each threshold, with utility and coverage averaged over seeds and risk pooled across scored answers.They cover training thresholds, the two held-out primary thresholds, and 0.95; estimates are descriptive without additional confidence intervals or multiplicity-adjusted tests.Source: §E.2 · p. 12, Table 9 · p. 13, Table 10 · p. 14
- On TriviaQA, the calibrated selector has positive point-estimate utility at all six thresholds, while its coverage decreases from 45.7% at τ = 0.60 to 1.3% at τ = 0.95.All four RL arms have negative utility from τ = 0.75 onward.Source: §E.2 · p. 12
F Prompt templates and calibration
- The prompt templates use chat messages with extended thinking disabled, and reference answers appear only during label construction, not in model inputs.Braces mark substitutions for the question, candidate answer, and formatted threshold.Source: F Prompt templates and calibration · p. 15
F.1 Threshold-conditioned answer generation
- Threshold-conditioned generation uses the same two demonstrations for every threshold and asks for an answer only above the requested confidence threshold; otherwise it requests exactly IDK.The final question and requested threshold are the changing parts of that template.Source: §F.1 · p. 15
F.2 Candidate generation for the filters
- Filter candidates are generated by one greedy and three sampled candidates, while greedy calibration and test candidates use the same template.That template asks for the best short answer even when uncertain and prohibits abstention.Source: §F.2 · p. 15
F.3 Candidate correctness scoring
- Candidate correctness scoring reads the next-token logits for True and False from a question–candidate prompt, rather than generating verbal confidence.The scorer is used for the original scorer, TRAIN logistic input scores, and learned selectors.Source: §F.3 · p. 15