PaperlayerView exact revision on arXiv

Source-linked AI summary

Do LLMs Really Know What They Don't Know? Internal States Mainly Reflect Knowledge Recall Rather Than Truthfulness

Chi Seng Cheang, Hou Pong Chan, Wenxuan Zhang, Yang Deng

arXiv:2510.09033v3cs.CL

TL;DR

The paper asks whether internal signals reveal truthfulness or instead reflect knowledge recall, and proposes a taxonomy separating associated from unassociated hallucinations. Its analyses find that AHs overlap with factual outputs in internal representations, while UHs are more separable and more reliably detected.

  • Problem

    It remains unclear whether internal signals associated with factual correctness reflect truthfulness or other factors, since LLMs are not explicitly trained to represent truthfulness.

    Source: §1 · p. 1

  • Method

    The paper classifies false outputs as AHs or UHs according to reliance on learned subject associations and analyzes their internal processes and representations against factual associations.

    Source: §1 · p. 2, §3.2 · p. 3

  • Results

    For LLaMA, detection AUROC is ≈0.48–0.69 for AHs versus ≈0.86–0.93 for UHs, consistent with hidden states reflecting knowledge recall rather than truthfulness.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    The paper recommends reporting detection separately for AHs and UHs and combining hidden states with external feedback such as fact-checking or retrieval-based verification.

    Source: §7 · p. 9

    The analysis focuses on factual completion prompts and does not extend to long-form or open-ended text generation.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Recent work suggests that LLMs "know what they don't know", positing that hallucinated and factually correct outputs arise from distinct internal processes and can therefore be distinguished using internal signals. However, hallucinations have multifaceted causes: beyond simple knowledge gaps, they can emerge from training incentives that encourage models to exploit statistical shortcuts or spurious associations learned during pretraining. In this paper, we argue that when LLMs rely on such learned associations to produce hallucinations, their internal processes are mechanistically similar to those of factual recall, as both stem from strong statistical correlations encoded in the model's parameters. To verify this, we propose a novel taxonomy categorizing hallucinations into Unassociated Hallucinations (UHs), where outputs lack parametric grounding, and Associated Hallucinations (AHs), which are driven by spurious associations. Through mechanistic analysis, we compare their computational processes and hidden-state geometries with factually correct outputs. Our results show that hidden states primarily reflect whether the model is recalling parametric knowledge rather than the truthfulness of the output itself. Consequently, AHs exhibit hidden-state geometries that largely overlap with factual outputs, rendering standard detection methods ineffective. In contrast, UHs exhibit distinctive, clustered representations that facilitate reliable detection.

1 Introduction

The introduction asks whether internal signals detect truthfulness or instead reflect how models recall parametric knowledge. It distinguishes hallucinations that lack subject-linked associations from those driven by spurious associations, then shows why that distinction matters for detection and refusal tuning.

1 Introduction

  • The open question is whether internal signals reflect truthfulness or confounding factors, since LLMs are not explicitly trained to represent truthfulness.Prior signals include residual streams, attention weights, and output-token logits.Source: §1 · p. 1
  • Because a model can use the same learned association to produce a correct or incorrect answer, internal representations may fail to distinguish AHs from factual outputs.The example contrasts a correct association between Barack Obama and Chicago with a false claim about his birthplace using that association.Source: §1 · p. 2
  • The paper classifies false outputs as UHs when they lack strong learned associations with the subject and as AHs when strong but spurious subject associations drive them.The proposed taxonomy distinguishes outputs by their relationship to the subject entity, not factual correctness alone.Source: §1 · p. 2
  • AHs and FAs share similar internal processes and hidden-state geometries, while UHs rely on a different process and remain more separable.Both AHs and FAs are driven by learned associations with the subject; UHs lack that reliance.Source: §1 · p. 2
  • For LLaMA, detection AUROC is ≈0.48–0.69 for AHs versus ≈0.86–0.93 for UHs, reflecting their different representational separability.The introduction attributes weaker AH detection to internal states reflecting knowledge recall rather than truthfulness.Source: §1 · p. 2
  • Refusal tuning achieves an 82% refusal rate on UHs but only 33% on AHs while preserving original factual responses, consistent with UHs being more separable from FAs.The passage attributes the difference to substantial representational overlap between AHs and FAs.Source: §1 · p. 2

2 Related Work

Related work distinguishes representation-based detectors from confidence-based detectors, while noting that confidence is often miscalibrated. The paper addresses the remaining explanatory gap by tracing internal computations involved in knowledge recall.

2 Related Work

  • Representation-based detectors train classifiers on hidden states to predict hallucinations, whereas confidence-based detectors use uncertainty signals such as token probabilities, verbalized confidence, or consistency across sampled outputs.Confidence-based methods typically flag responses below a predetermined confidence threshold.Source: §2 · pp. 2–3
  • Poor calibration means expressed confidence often fails to align with factual accuracy, raising questions about whether LLMs know their knowledge boundaries.Prior work had not provided a mechanistic explanation for this problem.Source: §2 · p. 3
  • The paper uses mechanistic interpretability to trace internal computations underlying knowledge recall and address that explanatory gap.Source: §2 · p. 3

3 Dataset Construction

The dataset construction moves beyond correctness labels by using causal interventions to distinguish hallucinations according to reliance on learned subject associations. It builds verifiable factual queries, categorizes outputs, and labels hallucinations using shifts in output distributions when subject information is blocked.

3 Dataset Construction

  • The dataset construction labels hallucinations by estimating whether they rely on learned associations related to the subject, beyond assigning factual correctness alone.Each example includes an input query, ground-truth answer, and model response.Source: §3 · p. 3

3.1 Data Collection

  • Queries are cloze-style prompts built from Wikidata subject–relation pairs, restricted to relations with objectively verifiable answers; LLaMA-3 and Mistral-v0.3 responses are generated with greedy decoding.The corresponding triple object is the ground-truth answer.Source: §3.1 · p. 3

3.2 Categorization of Knowledge

  • Outputs are categorized by factual correctness and reliance on subject representations: FAs are correct recalled knowledge, AHs are false content using input-triggered parametric associations, and UHs are false content without such reliance.Source: §3.2 · p. 3

3.3 Labeling Procedure

  • Factual recall is described as enriching subject representations in early layers, propagating subject information to the prediction position through middle-layer attention, and decoding the relevant attribute in upper layers.Source: §3.3 · p. 4
  • The labeling procedure blocks attention from subject tokens and measures the output-distribution shift with JS divergence; hallucinations above the average FA divergence are AHs, and those at or below it are UHs.Figure 2 maps interventions across LLaMA-3-8B layers; darker heatmap regions indicate greater causal influence on predictions.Source: §3.3 · p. 4, Figure 2 · p. 4

4 Analysis of Internal States in LLMs

The analysis finds that factual associations and associated hallucinations use similar subject-linked computational pathways, while unassociated hallucinations rely less on subject information and have more separable representations. Across subject representations, attention flow, and last-token geometry, internal states primarily reflect knowledge use rather than truthfulness.

4.1 Visualization of Information Flow

  • Fine-grained interventions find that factual associations and associated hallucinations share highly similar information flows, while unassociated hallucinations show substantially weaker flows.Subject-token and last-token interventions independently support the shared mechanism beyond attention-flow interventions used in labeling.Source: §4.1 · p. 5

4.2.1 Norm of Subject Representations

  • AH subject-representation norm ratios stay near 1 relative to FAs, while UH ratios fall to approximately 0.95 in middle layers.The comparison uses each category’s average L2 norm across layers, with FAs as the baseline.Source: Figure 3 · p. 5, §4.2.1 · p. 5

4.2.2 Correlation with Subject Popularity

  • FAs and AHs rise with popularity (low/high bins: 5%/52% and 1%/14%), while UHs fall from 94% in the lowest bin to 34% in the highest.Popularity is measured using average monthly Wikipedia page views as a proxy for subject frequency during pre-training.Source: Figure 4 · p. 6, §4.2.2 · p. 6
  • Because AHs mainly involve popular subjects, they may form a substantial share of deployment hallucinations and resist popularity-based distinction from FAs.This implication is conditional on high-popularity subjects accounting for a substantial portion of real-world LLM usage.Source: §4.2.2 · p. 6

4.3 Visualization of Attention Flow

  • AHs and FAs show large subject-to-last attention-contribution norms in mid-layers, whereas UHs show consistently lower norms.The pattern challenges the view that strong subject-token attention flow signals factuality, since it also appears for AHs.Source: Figure 5 · p. 6, §4.3 · p. 6

4.4 Analysis of Last Token Representations

  • By layer 25, last-token cosine similarity falls to ≈0.2 for FAs and AHs but only to ≈0.5 for UHs.All categories begin near ≈0.9 similarity in early layers.Source: Figure 6 · p. 7, §4.4 · p. 7
  • At layer 25, UHs form a distinct last-token representation cluster, while AHs substantially overlap with FAs.Silhouette and Davies-Bouldin metrics also show higher separability for UHs than AHs.Source: Figure 7 · p. 7, §4.4 · p. 7
  • Strong subject-specific updates yield low-entropy outputs for FAs and AHs, whereas weaker updates yield broader, higher-entropy output distributions.The broader distributions can assign probability to multiple possible answers.Source: Figure 8 · p. 7, §4.4 · p. 7

5 Revisiting Hallucination Detection

Existing hallucination detectors distinguish unassociated hallucinations from factual associations more reliably than associated hallucinations. The evaluation finds that both internal probes and black-box features track parametric-knowledge use more than output truthfulness.

Experimental Setups

  • The evaluation tests white-box probes of subject, attention-flow, and last-token representations alongside black-box answer-token probability and subject popularity features.It uses AH Only, UH Only, and Full settings, with five random data splits per setting.Source: §5 · pp. 7–8

Experimental Results

  • White-box AUROC is ≈0.91–0.93 for UH Only but ≈0.58–0.69 for AH Only; black-box AUROC is ≈0.86–0.87 versus ≈0.48–0.49.The table covers AH Only and UH Only detection settings.Source: Table 2 · p. 8, §5 · p. 8
  • In the Full setting, adding AHs to training dilutes UH detection, reducing AUROC from ≈0.9 in UH Only to ≈0.8.Figure 9 shows detection is consistently stronger on UH samples than AH samples.Source: Figure 9 · p. 8, §5 · p. 8
  • Because AHs and FAs arise through similar recall processes, detection signals risk misclassifying associated falsehoods as correct and fostering overconfidence.Sensitivity-specificity trade-offs are also more difficult for AH detection than UH detection.Source: §5 · p. 8

6 Challenges of Refusal Tuning

Refusal tuning generalizes better across Unassociated Hallucinations (UHs) than Associated Hallucinations (AHs), whose diverse representations overlap with factual answers. This limits the model’s ability to learn consistent refusal patterns for AHs.

6 Challenges of Refusal Tuning

  • Refusal tuning requires a shared feature pattern across hallucinated outputs to generalize, but AHs do not provide that consistent pattern.The authors contrast this with UHs, whose distinct representations should make refusal generalization easier.Source: §6 · pp. 8–9
  • The tuning comparison trains on 1,000 hallucination samples paired with refusal templates while preserving 1,000 factual answers, then evaluates 200 samples per category.A response matching any refusal template counts as a refusal, including refusals of factual answers.Source: §6 · p. 9
  • UH-only refusal tuning yields 82% refusal on UHs but 28% on AHs; AH-only tuning yields 33% on AHs and 23.5% on UHs, while UH-trained tuning refuses 29.5% of factual answers.The reported evaluation also finds that AH-trained tuning generalizes poorly across categories.Source: §6 · p. 9
  • UH representations form a consistent cluster that supports refusal generalization, whereas AH representations are too geometrically diverse for a consistent refusal pattern.The authors attribute the difficulty of generalizing AH refusals to their subject-specific hidden states.Source: §6 · p. 9

7 Conclusions and Future Work

The paper concludes that model hidden states primarily reflect reliance on parametric knowledge rather than truthfulness, limiting detection when hallucinations arise through knowledge recall. It recommends category-specific evaluation, complementary factuality feedback, and greater attention to AH detection.

7 Conclusions and Future Work

  • Hidden states encode reliance on parametric knowledge more than truthfulness, so detection works best for outputs detached from input knowledge and struggles with hallucinations generated through the same recall process as correct answers.The conclusion identifies this as the central constraint on representation-based detection.Source: §7 · p. 9
  • The authors recommend reporting AH and UH detection separately, supplementing hidden states with fact-checking or retrieval-based verifiers, and prioritizing AH detection because popular-topic errors threaten user trust and reliability.They note that AHs occur more frequently on widely known or highly popular topics.Source: §7 · p. 9

Limitations

The study’s conclusions are bounded by its factual-completion setting, detector coverage, and access to open-source models.

Limitations

  • The analysis is limited to factual-completion prompts and does not cover long-form or open-ended generation.The authors call for extending the investigation to those tasks for more comprehensive conclusions.Source: Limitations · p. 9
  • The detector analysis covers white-box internal-state methods and two black-box external-feature approaches, but not verbalization-based strategies such as prompting models to report confidence.The paper treats verbalization-based strategies as a separate line of approach.Source: Limitations · pp. 9–10
  • The study analyzes open-source LLMs; access restrictions prevent mechanistic analysis of commercial black-box models.The authors suggest alternative evaluation protocols or collaborations as possible future routes.Source: Limitations · p. 10

Ethical Considerations

The work analyzes models using Wikidata data from public sources without private or sensitive information, and describes filtering choices for the factual-query dataset. Its relation selection aims to reduce ambiguity in subject–relation–object examples.

Ethical Considerations

  • The work uses Wikidata data released under CC0 1.0, derived from public resources, and reports no private or sensitive personal information; LLM tools were used for language polishing.The passage describes these points under Ethical Considerations.Source: Ethical Considerations · p. 10

A.1 Selected Relations and Prompt Templates

  • Relation filtering excludes queries with variable answer granularity and relations whose answers can be guessed from shallow surface patterns.The examples given are location answers at different levels of detail and citizenship guessed from lexical patterns.Source: §A.1 · p. 13
  • The dataset selects objectively verifiable relations and filters for person subjects to reduce subject-level ambiguity, using father, mother, spouse, and date of birth relations.The four relations are Wikidata properties P22, P25, P26, and P569.Source: §A.1 · p. 13

A.2 Implementation Details

The appendix specifies the models, labeling procedure, and analysis settings used in the experiments. It also reports that Mistral exhibits patterns similar to LLaMA in internal computations, hidden states, detection, and refusal tuning.

A.2 Implementation Details

  • Experiments use LLaMA-3-8B and Mistral-7B-v0.3 checkpoints, greedy decoding at temperature 0 in BF16, and one-epoch QLoRA refusal tuning with rank 8 and α 8.QLoRA adapters are applied to attention and MLP modules; experiments ran on four NVIDIA L40S GPUs.Source: §A.2 · pp. 13–14

A.3 Labeling Factual Correctness

  • Factual correctness labels come first from spaCy named-entity matching and otherwise from a Qwen2.5-14B-Instruct judge, including when extraction fails.The judge compares the predicted answer with the ground truth using the prompt shown in Figure 11.Source: §A.3 · p. 14, §A.2 · p. 14, Figure 11 · p. 14

B Calculation of Subject Representation Norms

  • The norm analysis averages subject-token hidden-state L2 norms by layer, aggregates samples by knowledge category, and reports hallucination ratios against factual associations.Categories include FAs, UHs, and AHs.Source: B Calculation of Subject… · pp. 14–15

C Parallel Experiments on Mistral

  • Mistral exhibits patterns similar to LLaMA in internal computations, hidden-state behavior, hallucination detection, and refusal tuning.The parallel experiments used the same settings as those described in the main text.Source: C Parallel Experiments on Mistral · p. 16

D.1 Analysis of Parametric Subspace Overlap for Subject Representation

The analyses connect early-layer subject representations to alignment with parametric MLP subspaces and compare how hallucination categories cluster in hidden-state space. They find that UHs separate from factual associations, whereas AHs overlap with them; subject-token states show weaker separation than last-token states.

D.1 Analysis of Parametric Subspace Overlap for Subject Representation

  • UHs have significantly lower early-layer subspace-overlap ratios than AHs in both LLaMA and Mistral, consistent with stronger alignment of FA and AH subject representations to parametric weights.The authors suggest the models learned representations for FA and AH subjects during pretraining, but not for UH subjects.Source: §D.1 · p. 16, Figure 22 · p. 17

D.2 Quantitative Analysis of Cluster Separability

  • Silhouette values of 0.33–0.40 and DBI values of 0.93–1.13 for UH–FA comparisons contrast with AH–FA Silhouette values of 0.0022 and −0.0011 and DBI values of 14.22 and 25.25, indicating separation versus overlap.The metrics quantitatively confirm the t-SNE observations for both models.Source: §D.2 · p. 17, Table 4 · p. 17

D.3 More Visualization on Hidden States

  • Subject-token states show moderate UH separation from factual and associated samples, but less separation than last-token states, matching the stronger detection performance using last-token representations.Source: §D.3 · p. 17

E.1 Statistical Significance Analysis

Detection methods distinguish UHs from factual associations more effectively than AHs, a pattern also seen with semantic consistency scores. Threshold analysis finds robust UH detection but threshold-sensitive AH performance.

E.1 Statistical Significance Analysis

  • Across settings relevant to the claims, AUROC for UHs is significantly higher than for AHs (all p < 0.01), based on results across five random seeds.Source: §E.1 · p. 17

E.2 Semantic Consistency-Based Detection

  • Semantic consistency yields AUROC ≈0.84 for distinguishing UHs from FAs but ≈0.62 for AHs from FAs, aligning with black-box and white-box detection patterns.AHs produce similarly consistent outputs across runs, while UHs exhibit higher variance, which the authors link to weaker reliance on subject representations.Source: §E.2 · p. 18, Table 6 · p. 18

E.3 Sensitivity-Specificity Trade-offs

  • For UH-only detection, sensitivity stays above 0.83 as the threshold rises from 0.1 to 0.9, while specificity exceeds 0.96 at the highest setting; AH-only performance is highly threshold-sensitive.The reported trade-offs use detectors trained on LLaMA last-token representations.Source: §E.3 · p. 18, Table 7 · p. 18

Related papers

Loading 2510.09033v3…