Source-linked AI summary
The Missing "I Don't Know": Why Three Reasoning-Reliability Findings Converge on Calibrated Abstention
Srijith Ravikumar
TL;DR
The paper connects three independent reliability findings to calibrated abstention, while distinguishing their underlying gaps. It proposes evaluation reforms to select for abstention and argues these are necessary, but not sufficient, for principled closure; its proposal is scoped to discrete-answer settings.
Problem
The paper asks how to address reliability failures when systems commit despite unavailable tools, safety constraints, or limits on consistent reasoning, while leaderboard benchmarks do not select for abstention.
Source: §1 · pp. 1–2Method
The paper synthesizes two empirical findings and a theorem, then proposes triple-scoring, abstention-rate reporting, capability-stratified evaluation, and calibration metrics.
Source: §1 · pp. 1–2Results
The paper argues that the three findings converge on calibrated abstention and that benchmark reform is necessary, though not sufficient, for closing the gap identified by the theorem.
Source: §1 · pp. 1–2Takeaways & Limitations
Evaluation should distinguish correct, incorrect, and abstained outcomes and report abstention and calibration, while treating benchmark reform as one part of closure.
Source: §1 · p. 2The proposal does not generalize beyond discrete-answer settings; operationalizing calibrated abstention and triple-scoring for long-form, multi-step, and interactive settings remains open.
Source: Limitations · p. 10
Abstract
from arXiv · showhide
Three recent results describe what look like unrelated LLM reliability problems. Yin et al. (2026) show reasoning RL collapses tool-reliability representations. Suleymanov et al. (2026) show that under safety-constrained generation, large models rewrite flagged spans while small models truncate. Bastounis et al. (2024) prove any consistent-reasoning system without an implicit "I don't know" function must hallucinate infinitely often on broad problem classes. We argue these findings converge on a single intervention: calibrated abstention is what each independently identifies as the missing capability, even though the unavailability they document, a capability gap, a policy gap, and a recursion-theoretic gap, has a different source in each case. Honesty post-training has narrowed the gap in deployed models, but principled closure of the class Bastounis identifies requires a calibrated abstention function whose training signal at the leaderboard level is absent: dominant benchmarks assign zero reward to decline, so the leaderboard gradient that would select for the function does not exist. We propose four changes to evaluation: triple-scoring, abstention-rate reporting, capability-stratified evaluation, and mandatory calibration metrics. Benchmark reform is necessary, not sufficient, for closing the gap the theorem identifies.
1 Introduction
Three independent findings point to calibrated abstention as a shared intervention, despite arising from capability, policy, and recursion-theoretic gaps. The paper argues that benchmark incentives do not select for abstention and proposes evaluation reforms, while noting that reform is necessary but not sufficient for principled closure.
1 Introduction
- Yin et al. (2026), Suleymanov et al. (2026), and Bastounis et al. (2024) describe failures involving unavailable tools, safety-constrained rewrites, and consistent reasoning without an implicit “I don’t know” function.The findings concern independent empirical settings and a theorem, not instances of one structural phenomenon.Source: §1 · p. 1
- The paper argues calibrated abstention addresses the shared failure of committing when commitment is unwarranted, although each finding’s unavailability has a different source.Yin’s is a capability gap, Suleymanov’s a policy gap, and Bastounis’s a recursion-theoretic gap.Source: §1 · p. 1
- Honesty post-training has narrowed the gap, but the theorem requires an implicit “I don’t know” function that current training pipelines do not compute structurally.The paper proposes triple-scoring, abstention-rate reporting, capability-stratified evaluation, and ECE and Brier calibration metrics; benchmark reform is necessary but not sufficient for closure.Source: §1 · pp. 1–2
- Abstention-specific mechanisms exist, but dominant leaderboard benchmarks have not been reformed, and their selection signal for practical abstention is backwards.The paper says leaderboards select model releases and dominant benchmarks assign zero reward to decline.Source: §1 · p. 2
2 The Three Findings
The section presents two empirical reliability findings and a formal result, each concerning cases where a system should not simply commit. The formal result identifies an implicit refusal function as an escape, while cautioning against equating practical abstention methods with that theoretical object.
2.1 Reasoning RL collapses tool-reliability representations
- Reasoning-targeted RL raises tool hallucination alongside reward-signal gains, even after tool-free GSM8K training, while destabilizing tool-reliability pathways and late-layer residual streams.The effects also appear under GRPO-style RL and inference-time prompt switching; prompt instructions offer marginal gains, while DPO reduces hallucination only by sacrificing utility.Source: §2.1 · p. 2
2.2 Safety-constrained generation produces scale-dependent failure modes
- Across 13 models, Suleymanov et al. (2026) report that GPT-5 rewrites flagged spans and preserves utility at 8.58/10, while smaller and midsized models truncate text by 50–73%.Both strategies reduce leakage, but the authors argue the Editor role requires a minimum reasoning threshold to avoid refinement collapsing into refusal.Source: §2.2 · p. 2
- On Qwen3-8B, reasoning raises leakage from 74% to 84% without defense, while SemSIEdit-active results are consistent with a reversal, at 42% versus 47%.These are reported single-run point estimates; the paper says reasoning can connect benign facts into sensitive inferences without defense and power rewrites under constraint.Source: §2.2 · p. 2
2.3 Consistent reasoning without abstention is provably unreliable
- Bastounis et al. (2024) prove that consistent-reasoning systems that always answer hallucinate infinitely often on certain problem classes, including basic arithmetic.Their formal consistency property propagates correctness across computably enumerable paraphrases; an implicit partial-computable “I don’t know” function is the identified escape.Source: §2.3 · pp. 2–3
- The paper does not claim that practical calibrated-abstention methods instantiate the theorem’s recursion-theoretic function or that current models satisfy its consistency assumption.It adopts the function as a target for benchmark selection rather than applying the theorem directly to LLM inference.Source: §2.3 · p. 3
3 Convergent Evidence
The three findings converge on calibrated abstention despite arising from capability, policy, and recursion-theoretic gaps. The paper treats this as convergent evidence, not a shared formal mechanism, and distinguishes which failures evaluation reform can address.
3 Convergent Evidence
- The three findings support calibrated abstention as a shared remedy, although their gaps differ and Yin’s and Suleymanov’s settings are not instances of Bastounis’s theorem.The paper calls the convergence defeasible: a mechanism-level intervention implementing no abstention or selective prediction that closed all three failure modes would refute it.Source: §3 · p. 3
- Bastounis’s theorem cannot be empirically closed, so the paper targets its measurable manifestation, including reasoning-induced abstention degradation.The stated falsification test is whether a mechanism-level intervention implementing no abstention or selective prediction closes all three empirical failure modes.Source: §3 · p. 3
- Reasoning fine-tuning degrades abstention by 24% on average, while outcome-reward RL selects against honest decline and can generalize numeric-output behavior to tool-hallucination settings.The 24% figure is Kirichenko et al.’s average across two reasoning and non-reasoning model pairs; the proposed connection does not claim that RL achieves the theorem’s recursion-theoretic consistency.Source: §3 · p. 3
- The proposal scores only one of four distinct abstention-related constructs; policy gaps remain with safety benchmarks, while the reform is scoped to discrete answers.The paper distinguishes the theorem’s formal object, the scored construct, safety evaluation, and the discrete-answer scope.Source: Table 1 · p. 4, §3 · p. 4
- Reasoning’s reliability effect depends on the surrounding loop: extra compute worsens abstention in Kirichenko’s setting but enables constructive rewriting in Suleymanov’s.The paper attributes the difference to the loops’ incentives, not to different reasoning capabilities.Source: §3 · p. 4
- Suleymanov et al. (2026) report GPT-5 answer expansion of 110 characters, interpreted here as abstention-with-context rather than technically calibrated abstention.The account contrasts this constructive rewrite with smaller models’ truncation and distinguishes policy-prescriptive refusal from epistemic abstention.Source: §3 · p. 4
4 Why Current Benchmarks Make This Worse
Dominant benchmarks reward attempts at least as much as decline, leaving no leaderboard incentive to abstain. Their discrete-answer format and error audits also make appropriate abstention indistinguishable from failure.
4 Why Current Benchmarks Make This Worse
- GSM8K outcome rewards are zero for nonnumeric matches, so RL receives no decline signal; leaderboard reform targets model-release selection, not deployment decisions.The paper distinguishes training-time gradients, evaluation-time selection, and deployment-time product decisions; honesty post-training addresses the latter.Source: §4 · pp. 4–5
- Attempting weakly exceeds declining in expected reward across all four dominant benchmarks; guessing strictly dominates abstention on multiple choice, while open-generation depends on signal above noise.Table 2 presents the expected-score argument for MMLU, HellaSwag, HumanEval, and GSM8K.Source: §4 · p. 4, Table 2 · p. 5
- 6.49% mean MMLU error and 57% Virology error coexist with scores that treat abstaining on ambiguous or mislabeled items like label failure.The passage notes that the Virology subset’s small denominator widens the Wilson-interval bound considerably.Source: §4 · p. 5
- The four benchmarks all use discrete answers, the format where the paper locates the selection signal its proposal targets.The cited passage frames discrete-answer scope as the site of the selection signal, not merely a convenience.Source: §4 · p. 5
5 A Concrete Proposal
The proposal combines four benchmark reforms—triple-scoring, abstention-rate reporting, capability stratification, and calibration metrics—because none alone supplies the required leaderboard signal. It applies a cost-sensitive abstention threshold and treats the reforms as complementary, not proven sufficient.
5 A Concrete Proposal
- Triple-scoring makes correct, incorrect, and abstained outcomes distinct, with positive abstention reward outside reliable capability and zero reward elsewhere.The abstention threshold is a leaderboard-design choice; under current scoring, guessing dominates decline when attempting has positive expected correctness.Source: §5 · p. 6
- Abstention rates should accompany accuracy by problem stratum: equal accuracy can reflect different coverage, which current leaderboards cannot distinguish.A model with 0.85 accuracy and 5% abstention is doing something different from one with 0.85 accuracy and 40% abstention.Source: §5 · p. 7
- Environment-level unanswerability is scorable now, while model-level capability boundaries remain an open benchmark-design problem.The near-term substitute for model-level boundaries is observable difficulty or subject bins, with rollout estimates and intervals where rankings may change.Source: §5 · p. 7
- GPT-4o ranks second on accuracy but sixth at p*=0.5, below Claude-3-haiku, which attempts a quarter of the items.At p*≈0.67, the two most abstention-heavy models rank first and second; the null policy ranks above every model at both points.Source: Table 4 · p. 8
- Calibration metrics should accompany accuracy, but ECE depends on estimator choices and proper scores require coverage to be reported alongside them.Calibration figures are comparable only at matched coverage; on open-ended generation, confidence must be derived from quantities such as semantic-equivalence clusters or sampling estimates.Source: §5 · p. 8
- The four reforms are complementary, and weaker bundles have not produced the leaderboard-level gradient the proposal requires.Prior reforms address adjacent dimensions, but the proposal's contribution is composing them into one requirement; it does not claim the bundle is minimal or sufficient.Source: §5 · pp. 5, 6, 8
- Triple-scoring requires subgroup abstention audits because pooled rates can hide disparities unrelated to item difficulty.Reporting accuracy-given-attempted on matched strata helps distinguish abstention tracking capability limits from systematic differences across groups.Source: §5 · p. 9
- A transition path starts with side metrics, but requiring structured abstention can advantage proprietary releases unless benchmarks supply a prompt-template policy.The stated costs include annotating environment-level unanswerability, fixing the operating point, and preserving comparability through parallel reporting.Source: §5 · p. 9
6 Counterexamples and Boundary Conditions
Honesty-oriented post-training improves abstention behavior, but the authors argue that it does not establish closure of the theorem-identified class. They distinguish that proposed structural requirement from reliability gains on inputs where models recognize uncertainty.
6 Counterexamples and Boundary Conditions
- Kirichenko et al. (2025) report 24% mean abstention degradation across 20 datasets under reasoning fine-tuning, while Yin et al. (2026) find DPO reduces hallucination only by sacrificing utility.The authors say honesty post-training narrows the gap on inputs where the model recognizes uncertainty, but conjecture it does not address the complementary slice.Source: §6 · p. 9
- The authors would withdraw the structural-necessity reading if post-training without scoring reform achieved sub-10% residual error on a calibrated, environment-stratified abstention benchmark.Environment-level unanswerability is a proxy for the conjectured model-level slice, which remains an open benchmark-design problem.Source: §6 · p. 9
- Benchmark reform is necessary, not sufficient, for closing the theorem-identified class; post-training is necessary, not sufficient, for the class users most often encounter.The authors say the interventions address different gradients and that pursuing only one will plateau at the boundary of the other.Source: §6 · p. 9
Limitations
The authors qualify both their convergence reading and their evaluation proposal: the interventions remain distinct, and the proposal has defined scope. They also identify conditions that could challenge the reading or limit its application.
Limitations
- The proposal is not the only possible response: mechanism-level interventions could retrofit existing benchmarks, though the authors argue scoring reform is needed to select for them.Examples include calibration heads, refusal tokens, and conformal layers.Source: Limitations · p. 10
- The authors do not claim their reading is the only one; other accounts may fit the three papers equally well.Source: Limitations · p. 10
- The three findings have different sources of unavailability, and the proposal evaluates capability gaps while leaving policy gaps to safety benchmarks.Capability gaps can be probed with ground-truth unanswerable tasks; policy gaps require benchmarks specifying the ruling-out rule.Source: Limitations · p. 10
- The proposal is limited to discrete-answer settings; operationalizing triple-scoring for long-form, multi-step, or interactive settings remains open.In those settings, commitment is distributed across turns and has no clean discrete analogue at the document or dialogue level.Source: Limitations · p. 10
Use of AI Assistants
AI assistants supported drafting, editing, structural revision, LATEX formatting, and bibliography verification. The author retained responsibility for the central argument, substantive claims, and factual and theoretical assertions.
Use of AI Assistants
- AI assistants supported manuscript drafting and editing, while the author conceived the argument, selected and read cited works, and verified citations.Assistants were not used to generate citations or fabricate evidence.Source: Use of AI Assistants · p. 10