Hallucinations Undermine Trust; Metacognition is a Way Forward

• In factoid question-answering without external tools, factuality gains have mostly expanded model knowledge, while limited discriminative power may make awareness of knowledge boundaries fundamentally difficult.The paper argues that this asymmetry creates a discrimination gap between correct and incorrect answers.Source: §1 · pp. 2, 4
• Because calibration does not ensure discrimination, guaranteeing zero errors through abstention can suppress valid answers, creating a utility cost that encourages models to answer despite uncertainty.The introduction cites weak generalization of truthfulness probes, confident hallucinations, failed confession training, and reasoning-related increases in hallucinations as evidence of a discrimination gap.Source: §1 · p. 2
• Reframing hallucinations as confident errors makes appropriately expressed uncertainty a third path beyond answering or abstaining, allowing models to preserve utility while communicating uncertainty honestly.An appropriately hedged error is presented as a hypothesis rather than a hallucination.Source: §1 · pp. 2, 4
• Faithful uncertainty requires each answer’s linguistic uncertainty to reflect its intrinsic uncertainty, rather than relying on uniform hedging or aggregate calibration.It is an instance-level property: each hedge should reflect the internal state for that specific answer.Source: §1 · pp. 2, 4
• Metacognition means recognizing and acting on uncertainty: communicating it honestly in direct interaction and using it to govern tool invocation and the weighing of retrieved information.The paper argues that this capability can prevent both inefficient tool overuse and dangerous underuse; knowledge expansion remains complementary for confident errors.Source: §1 · p. 3
• The paper recommends prioritizing discriminative measures over calibration, quantifying interventions’ utility costs, and pursuing open problems in faithful uncertainty and metacognition.It also notes promising research directions in prompting, fine-tuning, model internals, reasoning models, and intrinsic signals as reinforcement-learning rewards.Source: §1 · pp. 3–4

2.1. Problem Scope

• The paper focuses on extrinsic hallucinations—factually incorrect generations—in parametric LLMs, while distinguishing them from tool-augmented systems, intrinsic hallucinations, and reasoning errors.Implications for tool use are discussed later; the study targets long-tail facts about very rare entities rather than common misconceptions or head knowledge.Source: §2.1 · p. 4

2.2. Measuring Reliability

• Reliable evaluation must track both overall accuracy and attempted accuracy, since refusing uncertain questions can eliminate hallucinations while making a model practically useless.The passage identifies F1 and Omniscience Index as summary metrics for maximizing both measures.Source: §2.2 · p. 5
• Calibration aligns confidence with empirical accuracy, whereas discrimination separates correct from incorrect answers; calibration alone does not ensure discrimination.A model assigning every answer confidence 0.6 can be perfectly calibrated at 60% accuracy yet have zero discriminative power.Source: §2.2 · p. 5

2.3. Existing Mitigation Strategies

• Existing mitigation strategies intervene during training through data filtering, regularization, factuality penalties, collaborative-game rewards, and linguistic calibration.These approaches aim to reduce non-factual outputs or overconfidence.Source: §2.3 · p. 5
• Inference-time strategies steer factual generations without changing model weights, using custom decoding, internal signals, or self-verification.Source: §2.3 · p. 5

3.1. The Theoretical Ceiling

• Prior theoretical results identify limits on universal truth verification and learning, hallucination for facts not inferable from others, and a consistency–breadth trade-off that can force mode collapse.The cited work argues that reducing hallucinations below a critical threshold requires drastically reducing output diversity.Source: §3.1 · p. 5

3.2. The Discriminative Gap

• Confidence calibration estimates average error rates, but weak discrimination between correct and incorrect answers leaves realistic factual QA models unable to avoid a substantial utility tax.Reported AUROC values are 0.70–0.85 across realistic factual QA tasks; Farquhar et al. (2024) report 0.79 across 30 model×task combinations, Savage et al. (2025) report GPT-4 at 0.79 in medical QA, and Kang et al. (2025) report GPT-4o-mini at 0.68–0.72 on biography generation. At AUROC 0.71, reducing errors from 25% to 5% requires discarding 52% of valid answers; even at 0.85, the tax is ∼28%, while it falls below 5% only at AUROC ≥0.95.Source: §3.2 · p. 6, Figure 2 · p. 6

3.3. Corroborating Anomalies

• Truthfulness probes generalize poorly, confident hallucinations reveal missing error signals, confession training does not transfer from safety violations, and extended reasoning can worsen hallucination and abstention.The section argues that models cannot be aligned to report errors they cannot internally represent; prioritizing reasoning-path completion over abstention can rationalize incorrect answers to satisfy utility objectives.Source: §3.3 · p. 7

3.4. The Empirical Tradeoff

• Frontier models either preserve coverage alongside high hallucination rates or abstain more to improve factuality, leaving the ideal high-coverage, high-factuality region empty.The paper interprets that gap as evidence that current models lack the intrinsic ability to distinguish their hallucinations from their knowledge.Source: Figure 3 · p. 7, §3.4 · p. 8

4. Faithful Uncertainty

• Models should expand their knowledge where possible while faithfully expressing uncertainty where knowledge falls short and discrimination remains difficult.This complements, rather than abandons, the factuality objective of knowledge expansion.Source: §4 · p. 8

4.1. Defining the Objective

• Faithful uncertainty aligns linguistic confidence expressed in words with intrinsic confidence in an assertion’s meaning; it requires matching the model’s internal state, not the external world.Intrinsic uncertainty concerns the likelihood of generating conflicting answers; linguistic uncertainty includes hedges such as “I might be mistaken.”Source: §4.1 · p. 8

4.2. The Feasibility Argument

• Because a model’s confidence signal is computable from its weights, mapping that signal to verbal uncertainty is a closed-loop problem that may be improved through architecture, data, and training.The model need not determine whether a confidence value corresponds to real-world truth; it must recognize and verbalize that confidence.Source: §4.2 · p. 8

4.3. Reliable Utility

• Faithful uncertainty preserves answers that blanket abstention would discard: at 60% intrinsic confidence, abstaining to avoid errors also loses 60% correct answers, while appropriate hedging retains utility.Reliable utility means maximizing provided information without compromising trust by aligning a claim’s decisiveness with intrinsic confidence; confident errors remain hallucinations, while appropriately qualified errors can still be useful.Source: §4.3 · p. 9

4.4. The Research Opportunity: Tractable Headroom

• Current state-of-the-art models remain far from faithful uncertainty, while prompting, supervised fine-tuning, and internal-representation steering are being explored as promising approaches.The passage characterizes current models as expressing high linguistic confidence even when intrinsic uncertainty is low; faithful uncertainty alongside knowledge expansion is proposed as a path to more knowledgeable, trustworthy models.Source: §4.4 · p. 9

• Tools amplify the need for faithful uncertainty: without it, agents may overuse or underuse tools and mishandle conflicts between retrieved evidence and internal beliefs.Source: §5 · pp. 9–10

Tool-Use Masks the Reliability Problem

• Final-output-accuracy benchmarks can reward retrieval without testing why search was needed, concealing metacognitive weakness and fragility when tools fail or return unexpected results.Source: §5 · p. 9
• Recent work (Lin et al., 2025; Qian et al., 2025; Xu et al., 2026; Yan et al., 2026) finds modern search agents lack self-awareness and systematically overuse tools.Source: §5 · p. 10

Storage vs Control

• Tools solve storage but create a control problem: agents must judge when to delegate retrieval and weigh conflicting or low-quality results against internal priors.Faithful uncertainty governs these decisions, acting as the control layer between the LLM and harness, as Figure 4 illustrates.Source: §5 · p. 10, Figure 4 · p. 10

Towards Metacognitive LLMs

• Metacognitive agents need introspection to assess uncertainty and regulation to adjust behavior, dynamically deciding when information suffices, when to verify, and when to halt.The section presents metacognition as a prerequisite for reliable autonomous agents in open-ended environments.Source: §5 · p. 10

6.1. Challenges for Metacognitive LLMs

• Static supervised fine-tuning labels can misrepresent a model’s changing knowledge, motivating dynamic datasets or bootstrapping methods that avoid stale-boundary overfitting.Models rarely express doubt naturally, so teaching hedging through supervised fine-tuning risks inducing hallucinated uncertainty or confidence when labels no longer match what the model knows.Source: §6.1 · p. 11
• Post-training can degrade pre-trained uncertainty representations and increase overconfidence, motivating alignment methods that preserve uncertainty while supporting safety and instruction following.The passage cites evidence of calibrated pre-trained uncertainty being degraded during post-training (Achiam et al., 2023; Kadavath et al., 2022; Tian et al., 2023b; Zhu et al., 2023), and of alignment increasing overconfidence (He et al., 2025; Song et al., 2025b).Source: §6.1 · pp. 11–12
• Faithful linguistic uncertainty requires identifying its source—not just assigning a scalar confidence—and mapping that source to an appropriate hedge (Delacroix et al., 2025).Sources can include prompt ambiguity, lack of knowledge, or uncertainty about alignment behavior; disentangling them can produce more informative and actionable signals.Source: §6.1 · p. 12
• Causal evaluations must distinguish genuine awareness of internal uncertainty from heuristic hedging, using approaches such as concept injection, cross-model evaluations, or strategic games.Strategic games can test whether models benefit from assessing and using their confidence (Ackerman, 2026); the passage also cites concept injection (Lindsey, 2025) and cross-model evaluations (Binder et al.; Kapoor et al., 2024; Li et al., 2025a; Song et al., 2025a).Source: §6.1 · p. 12
• Agent evaluations should assess process-based control, penalizing needless searches for known facts and trust in conflicting sources even when the final answer is correct (Yona et al., 2025).This shifts evaluation from end-to-end correctness toward control-centric, model-dependent measures, addressing agents that “luck into” correct answers despite poor reasoning.Source: §6.1 · p. 12

6.2. Better Hallucination Mitigation Evaluations

• Hallucination-mitigation evaluations should visualize the full Utility-Error Curve, which shows the utility sacrificed to reach a target error rate.Calibration-based metrics such as ECE and summary discrimination metrics such as AUROC can obscure operating costs, including prohibitive abstention rates at high reliability.Source: §6.2 · p. 12
• To demonstrate frontier improvement, compare utility at a fixed error rate rather than relying on a single operating point or shifting along the existing trade-off curve.Increasing a refusal threshold may reduce hallucinations without improving underlying capability; the recommended comparison is higher utility than the baseline for the same error rate.Source: §6.2 · p. 12
• Measure interventions’ spillover costs across general capabilities, including head knowledge, reasoning, coding, and creative tasks, not just recall on the target set.Refusal tuning for long-tail queries can also make a model evasive on head knowledge or less helpful on other tasks.Source: §6.2 · p. 13

7.1. We Should Not Be Deprioritizing Factuality

• Faithful uncertainty complements continued knowledge expansion, especially at competence boundaries; the combination of knowledge and metacognition is better than either alone.Emerging areas such as multimodal understanding still have room for basic factuality gains (Cheng et al., 2025).Source: §7.1 · p. 13

7.2. Users Prefer Confidence Over Uncertainty

• Faithful uncertainty targets neither creative domains nor intrusive hedging; localized flags for specific code or dates can add value without blocking users.The objection is that users may prefer decisive answers and perceive constant hedging as friction or incompetence.Source: §7.2 · p. 13

7.3. Latent Truth Exists, We Just Need Better Probes

• Latent-truth research remains valuable, but assumes universal truth representations, while accessible confidence signals offer metacognitive headroom today.Mechanistic interpretability has demonstrated distilling self-awareness and confidence (Stolfo et al., 2024), and reasoning models express confidence better even while hallucinating more (Podolak and Verma, 2025; Yoon et al., 2025). Intrinsic signals are also used as reinforcement-learning rewards to encourage diversity and improve reasoning.Source: §7.3 · pp. 13–14

8. Discussion

• Because a discrimination gap makes fully eliminating hallucinations fundamentally challenging, the paper proposes faithful uncertainty as a complementary objective.Source: §8 · p. 14
• In agentic systems, metacognitive awareness of uncertainty serves as a control layer for robust tool use.Source: §8 · p. 14
• Faithful uncertainty is a form of honesty that can support human oversight by prompting verification, additional sources, or independent judgment when models express doubt.Realizing this vision requires shifting model development beyond benchmarks focused on factual accuracy and changing user expectations so people expect and appropriately interpret uncertainty.Source: §8 · p. 14

A. Additional Details

• The simulation used 25,000 samples with a 25% base hallucination rate, drawing correct-answer scores from Beta(1.8, 1.0) and incorrect-answer scores from Beta(1.0, 1.3).The distributions model overlapping confidence profiles typical of modern LLMs and were chosen to reproduce profiles reported by Nakkiran et al. (2025).Source: A. Additional Details · p. 21
• Isotonic regression calibrated the scores to near-perfect calibration (smECE ≈ 0.014), isolating confidence-distribution overlap as the source of utility–error trade-offs.The procedure was intended to ensure the trade-offs stemmed from limited discriminative power rather than miscalibrated probabilities.Source: A. Additional Details · p. 21
• The utility–error curve swept rejection threshold τ from 0 to 1, measuring the proportion of all samples answered correctly as Utility and incorrectly as Error Rate.The curve was computed at each threshold using the calibrated scores.Source: A. Additional Details · p. 21

B. Faithful Uncertainty: Definitions and Measurement

• Intrinsic confidence is high when repeated samples generate answers that do not contradict the candidate assertion.Intrinsic uncertainty is quantified by the likelihood of conflicting answers under repeated sampling.Source: B. Faithful Uncertainty: Definitions… · p. 21
• Linguistic uncertainty is measured as decisiveness: the probability readers assign to an assertion’s truth based solely on the response language, using an LLM judge (Liu et al. (2025); Yona et al. (2024)).Source: B. Faithful Uncertainty: Definitions… · pp. 21–22
• A response is faithful when decisiveness tracks intrinsic confidence assertion by assertion; a score of 1 indicates perfect alignment, while lower scores reflect systematic over- or under-hedging.Source: B. Faithful Uncertainty: Definitions… · p. 22
• Current state-of-the-art models score cMFG 0.5–0.7, indicating weak alignment; 0.5 means decisiveness is independent of actual confidence.cMFG is expected faithfulness uniformly averaged across confidence levels, implemented with equal-width confidence bins.Source: B. Faithful Uncertainty: Definitions… · p. 22