PaperlayerView exact revision on arXiv

Source-linked AI summary

Hallucinations Undermine Trust; Metacognition is a Way Forward

Gal Yona, Mor Geva, Yossi Matias

arXiv:2605.01428v1cs.CL

TL;DR

Hallucinations remain a trust problem because models can deliver incorrect information confidently. This paper proposes faithful uncertainty as a metacognitive objective, concluding that honest uncertainty communication can preserve utility and guide agentic tool use.

  • Problem

    LLMs continue to produce confident factual errors, while expanding knowledge has outpaced reliable awareness of what they do not know.

  • Method

    The paper reframes hallucinations as confident errors and proposes faithful uncertainty, aligning linguistic uncertainty with intrinsic uncertainty.

  • Results

    The paper concludes that metacognitive uncertainty can communicate uncertainty honestly and serve as a control layer for agentic retrieval and tool use.

  • Takeaways & Limitations

    Trustworthy and capable LLMs require honest representation of epistemic uncertainty, enabling users to verify information and exercise judgment when models express doubt.

  • Takeaways & Limitations

    Models may imitate faithful-uncertainty style through heuristics without actually sensing their internal state, making rigorous evaluation necessary.

Abstract

from arXiv · show

Despite significant strides in factual reliability, errors -- often termed hallucinations -- remain a major concern for generative AI, especially as LLMs are increasingly expected to be helpful in more complex or nuanced setups. Yet even in the simplest setting -- factoid question-answering with clear ground truth-frontier models without external tools continue to hallucinate. We argue that most factuality gains in this domain have come from expanding the model's knowledge boundary (encoding more facts) rather than improving awareness of that boundary (distinguishing known from unknown). We conjecture that the latter is inherently difficult: models may lack the discriminative power to perfectly separate truths from errors, creating an unavoidable tradeoff between eliminating hallucinations and preserving utility. This tradeoff dissolves under a different framing. If we understand hallucinations as confident errors -- incorrect information delivered without appropriate qualification -- a third path emerges beyond the answer-or-abstain dichotomy: expressing uncertainty. We propose faithful uncertainty: aligning linguistic uncertainty with intrinsic uncertainty. This is one facet of metacognition -- the ability to be aware of one's own uncertainty and to act on it. For direct interaction, acting on uncertainty means communicating it honestly; for agentic systems, it becomes the control layer governing when to search and what to trust. Metacognition is thus essential for LLMs to be both trustworthy and capable; we conclude by highlighting open problems for progress towards this objective.

Faithful Uncertainty

Reframing hallucinations as confident errors reveals a third path beyond abstaining or predicting: faithful expression of uncertainty. Aligning linguistic output with intrinsic confidence preserves valid information while making errors less harmful.

  • Faithful Uncertainty: Models may lack the discriminative power to perfectly separate truths from errors, creating a choice between suppressing valid information and risking confident errors.Abstention incurs a utility tax, while prediction can erode trust.
  • Faithful Uncertainty: Faithful uncertainty aligns linguistic output with intrinsic confidence, allowing models to provide valid information as appropriately hedged estimates.This reframing avoids treating every error as a hallucination.
  • Faithful Uncertainty: Expressing uncertainty renders errors less harmful while retaining useful information.The approach offers a third path beyond abstaining or predicting.

1. Introduction

The paper argues that factuality gains have mainly expanded models’ knowledge boundaries, while reliably recognizing their limits may be fundamentally difficult. It proposes faithful uncertainty—metacognitively aligning expressed uncertainty with intrinsic uncertainty—as a way to mitigate confident errors while preserving utility and guiding tool use.

  • Faithful uncertainty: Faithful uncertainty aligns linguistic hedging with the model’s intrinsic, instance-level uncertainty rather than merely calibrating aggregate error rates.Uniform hedging can match error rates while remaining uninformative for individual answers.
  • Metacognition: Metacognition means recognizing uncertainty and acting on it: communicating uncertainty in direct interaction and using it to guide tool use in agentic systems.Faithful uncertainty provides an honest signal of the model’s internal state, including whether it would likely repeat or conflict with an answer.
  • Problem: In factoid question-answering with clear ground truth, most factuality gains have expanded models’ knowledge boundaries rather than improved awareness of those boundaries.The authors conjecture that models may lack the discriminative power to perfectly separate truths from errors.
  • Reframing hallucinations: Treating hallucinations as confident errors reveals a third path beyond answering or abstaining: communicating uncertainty to preserve both trust and utility.An appropriately hedged error is framed as a hypothesis rather than a hallucination.
  • Complementarity and recommendations: Faithful uncertainty complements knowledge expansion: it communicates the remaining boundary, while continued expansion addresses honest mistakes where models are confidently wrong.The framework also motivates evaluating discriminative measures and the utility costs of hallucination-mitigation interventions.

2. Background

The background distinguishes parametric from tool-augmented LLMs, focuses on hallucinations involving sparse, long-tail knowledge, and frames evaluation around the utility–factuality trade-off. It also separates calibration from discrimination, emphasizing that eliminating hallucinations requires distinguishing correct from incorrect answers by confidence.

  • Extrinsic Hallucinations in Parametric Models: Parametric LLMs rely on their parameters, whereas tool-augmented LLMs retrieve information from external sources during inference.The paper primarily focuses on parametric models and discusses tool use later.
  • The Challenge of Tail Knowledge: Common hallucination evaluations may understate failures by emphasizing misconceptions or head knowledge rather than sparse-data, long-tail facts.The paper therefore uses benchmarks with explicit questions about very rare entities.
  • The Utility-Factuality Trade-off: Zero hallucinations can be achieved by refusing uncertain questions, but this makes the model practically useless; evaluation should track accuracy and attempted accuracy.Attempted accuracy measures correctness only among questions for which the model answered.
  • Calibration vs. Discrimination: Calibration aligns confidence with empirical accuracy, while discrimination measures whether confidence distinguishes correct from incorrect answers.A static confidence of 0.6 can be perfectly calibrated while having zero discriminative power.
  • Calibration vs. Discrimination: Eliminating hallucinations in practice requires good discrimination, not calibration alone.Calibration does not imply that a model can separate correct answers from incorrect ones.

3. Why Hallucinations Persist

Fully eliminating hallucinations faces fundamental theoretical and practical limits. In particular, models can be calibrated on average while lacking the discriminative power to identify specific errors, forcing a trade-off between factuality and utility.

  • Theoretical limits: Theoretical results argue that universal truth verification and learning all ground-truth functions are impossible for computable models, making some hallucinations structurally unavoidable.Calibrated models are also bound to hallucinate when a fact’s truth cannot be inferred from other facts.
  • Calibration vs. discrimination: Calibration measures average error rates, whereas discrimination identifies which specific answers are errors; the two can be weakly correlated.This conjectured lack of discriminative power is presented as a reason standard hallucination-mitigation techniques fail.
  • Empirical discrimination gap: AUROC clusters at 0.70–0.85 for realistic factual QA, while GPT-4o-mini reaches only 0.68–0.72 on biography generation.AUROC of 1.0 is perfect discrimination and 0.5 is random discrimination.
  • Utility–error trade-off: At AUROC = 0.71, reducing errors from 25% to 5% requires discarding 52% of valid answers; even AUROC 0.85 incurs a ∼28% utility tax.The tax becomes negligible (<5%) only at AUROC ≥0.95, above reported levels for knowledge-intensive tasks.
  • Utility–error trade-off: Current models lack the internal separability needed to maximize factuality without destroying utility, so higher factuality requires aggressive abstention and reduced coverage.Frontier models generally optimize coverage with high hallucination rates, while factuality-oriented models move toward abstention rather than maintaining utility.
  • Empirical evidence: Poor generalization of truthfulness probes and the existence of confident hallucinations provide empirical evidence for the discrimination gap.Confident hallucinations are factual errors delivered with high intrinsic confidence.

4. Faithful Uncertainty

Faithful uncertainty complements knowledge expansion by requiring models to express residual uncertainty in alignment with their intrinsic confidence. This framing aims to preserve reliable utility while reducing confident errors, though current models remain far from achieving it.

  • Faithful Uncertainty: The proposed objective combines maximizing knowledge with faithfully expressing whatever uncertainty remains when knowledge falls short.This complements, rather than abandons, factuality-oriented knowledge expansion.
  • Faithful Uncertainty: Faithful uncertainty aligns intrinsic uncertainty—the model’s statistical confidence in an assertion—with linguistic uncertainty expressed through words.High intrinsic uncertainty implies a high probability of generating conflicting answers.
  • Faithful Uncertainty: Because confidence is computable from model weights, aligning internal confidence with output strings is framed as a fully observable, closed-loop problem.This avoids requiring a universal truth direction in activation space for perfect discrimination.
  • Faithful Uncertainty: 60% intrinsic confidence means 60% correct answers when calibrated, whereas eliminating hallucinations by abstaining on the entire set discards those correct answers.Faithful uncertainty instead targets reliable utility by matching decisiveness to intrinsic confidence.
  • Faithful Uncertainty: Current state-of-the-art models remain far from faithful uncertainty, often expressing high linguistic confidence despite low intrinsic uncertainty.Promising approaches include meta-cognitive prompting, supervised fine-tuning, and steering based on internal representations.

5. Metacognition in the Age of Agents

External tools do not eliminate the need for faithful uncertainty: agents must recognize when internal knowledge is insufficient and regulate retrieval, verification, and communication accordingly. Metacognition provides this control through introspection and behavior regulation, while current evaluations and search agents often conceal or lack such self-awareness.

  • Tool-Use Masks the Reliability Problem: Final-output accuracy evaluations can reward successful retrieval without testing whether models understood why they needed to search.Such evaluations may mistake retrieval quality for metacognitive competence, leaving systems factually correct but unaware of their limitations.
  • Towards Metacognitive LLMs: A metacognitive agent can retrieve only when confidence is low and express doubt when retrieved evidence conflicts with internal priors.This dynamic behavior can improve efficiency and reliability compared with external routing based on query-type heuristics.
  • Tool-Use Masks the Reliability Problem: Modern search agents lack self-awareness, leading to systematic overuse of tools.The passage connects this lack of metacognitive awareness to current search-agent behavior.
  • Storage vs Control: Tools solve the storage problem but introduce a control problem: agents must decide when internal knowledge suffices and when to delegate to external retrieval.This control includes governing retrieval, verification, and orchestration through the agent harness.
  • Towards Metacognitive LLMs: Metacognition comprises introspection—assessing uncertainty—and regulation—adjusting behavior based on that assessment.Future agents need dynamic control over when information suffices, when to verify, and when to halt.

6. Call to Action

The paper calls for research addressing methodological obstacles to faithful uncertainty and metacognitive evaluation, alongside stronger practices for measuring hallucination-mitigation trade-offs. It emphasizes that eliminating hallucinations entails unavoidable utility costs that current evaluations do not fully capture.

  • Methodological hurdles: Faithful uncertainty research faces a bootstrapping paradox because static SFT labels can become incorrect as the model’s knowledge state changes.Base models rarely express doubt naturally, so supervised fine-tuning is needed to teach hedging, but uncertainty labels are dynamic relative to the model.
  • Methodological hurdles: Post-training can degrade pretrained uncertainty representations by inducing mode-seeking and overconfident behavior, creating tension between alignment and fidelity to the model’s knowledge boundary.The paper identifies the base model as often best capturing that boundary, while safety and instruction-following alignment may erase subtle uncertainty signals.
  • Methodological hurdles: Faithful linguistic uncertainty requires attributing confidence to distinct sources, including prompt ambiguity, missing knowledge, and uncertainty about alignment behavior.A single scalar confidence score cannot adequately represent these different causes.
  • Methodological hurdles: Metacognition requires rigorous causal evaluation to distinguish genuine internal-state sensing from learned uncertainty style or heuristics.The paper warns that models might hedge based on superficial cues, such as rare entities, rather than their actual internal state.
  • Evaluation practices: For hallucination mitigation, researchers should visualize the full Utility-Error Curve, demonstrate frontier improvements at fixed error rates, and measure helpfulness spillovers across general capabilities.The paper argues that utility costs are unavoidable, single-point accuracy comparisons are insufficient, and refusal tuning can damage performance on head knowledge, reasoning, coding, or creative tasks.

7. Alternative Viewpoints

Faithful uncertainty is presented as a complement to continued knowledge expansion, with its use scoped to factual claims rather than creative generation. The paper argues that accessible confidence signals provide practical headroom despite unresolved questions about latent truth representations.

  • Knowledge expansion: Faithful uncertainty complements rather than replaces knowledge expansion, while emerging domains retain room for basic factuality improvements.The authors argue that factuality work should continue because different domains have different headroom before discrimination becomes limiting.
  • Product concerns: Faithful uncertainty does not target creative domains where hallucination is desirable, and localized uncertainty can add value without blocking users.Examples include flagging a specific line of code or date during long-form generation.
  • Latent truth: Searching for latent truth representations may ease the tradeoff, but it assumes a universal representation exists across the long tail of facts.The authors remain skeptical of that assumption while treating latent-truth research as valuable.
  • Metacognitive signals: Reasoning models express confidence better despite hallucinating more, suggesting metacognitive signals are distinct from factual signals.The passage also notes that intrinsic signals already serve as reinforcement-learning rewards for diversity and reasoning.

8. Discussion

The discussion argues that fully eliminating hallucinations faces fundamental challenges from a discrimination gap, motivating faithful uncertainty as a complementary objective. This metacognitive awareness supports robust tool use in agentic systems and honest communication that enables appropriate human oversight.

  • Fully eliminating hallucinations faces fundamental challenges due to a discrimination gap.
  • Faithful uncertainty is proposed as a complementary objective and a metacognitive control layer for robust tool use in agentic systems.
  • Faithful uncertainty is a form of honesty requiring models to represent their epistemic state accurately rather than project false confidence.
  • Communicating uncertainty enables human oversight by inviting users to verify, seek additional sources, or exercise their own judgment.

A. Additional Details

The study used a synthetic dataset to model overlapping confidence profiles and examine utility–error trade-offs under controlled conditions. Isotonic regression supplied near-perfect calibration, isolating discriminative power as the limiting factor.

  • Simulation Methodology: The simulation used a synthetic dataset of N=25,000 with a fixed base hallucination rate of 25%.The dataset was designed to reproduce empirical confidence profiles reported by Nakkiran et al. (2025).
  • Simulation Methodology: Correct-answer scores followed Beta(1.8, 1.0), while incorrect-answer scores followed Beta(1.0, 1.3), producing overlapping confidence distributions.Correct scores were skewed toward high confidence and incorrect scores toward low confidence, reflecting profiles typical of modern LLMs.
  • Simulation Methodology: Isotonic Regression enforced near-perfect calibration with smECE ≈0.014, so observed utility-error trade-offs reflected overlapping distributions rather than probability miscalibration.This setup was intended to isolate discriminative power as the limiting factor.

B. Faithful Uncertainty: Definitions and Measurement

The section defines faithful uncertainty by comparing a model’s intrinsic confidence, inferred from repeated-sampling consistency, with linguistic decisiveness, inferred from how confidently responses communicate assertions. It introduces cMFG to correct raw faithfulness scores for confidence-distribution effects, with state-of-the-art models typically showing only weak alignment.

  • Definitions: Intrinsic uncertainty is quantified by the likelihood that repeated samples generate conflicting answers to a fact-seeking query.Consistent assertions indicate high intrinsic confidence in the candidate assertion.
  • Definitions: Linguistic uncertainty is quantified through decisiveness: the probability a reader assigns to an assertion being true from the response’s language alone.Decisiveness is implemented using an LLM-as-a-judge and reflects hedges, qualifiers, and epistemic markers.
  • Definitions: Faithful uncertainty requires decisiveness to track intrinsic confidence assertion-by-assertion.A score of 1 indicates perfect alignment, whereas lower scores indicate systematic over- or under-hedging relative to internal confidence.
  • Measurement: cMFG averages expected faithfulness uniformly across confidence levels to reduce confounding from a model’s confidence distribution.It is implemented using equal-width confidence bins.
  • Measurement: 0.5–0.7 is the typical cMFG range for current state-of-the-art models, indicating only weak alignment between expressed uncertainty and intrinsic confidence.A cMFG of 0.5 corresponds to decisiveness being independent of actual confidence.
Loading 2605.01428v1…