LLM Internal States Reveal Hallucination Risk Faced With a Query

1 Introduction

• LLM hallucinations can undermine trust and real-world applications because models may answer with plausible-sounding but unfaithful or nonsensical content.Unlike humans who may hesitate when they recognize they do not know, LLM assistants lack cognitive uncertainty estimation.Source: §1 · p. 1
• Prior work explored internal-state representations and text-level hallucination detection, but did not exclusively analyze texts produced by the same LLMs whose states were examined.The paper frames this as a need to investigate how a model’s own internal states correlate with its uncertainty and hallucination occurrence.Source: §1 · pp. 1–2
• 84.32% average estimation accuracy across 15 NLG tasks shows query-associated internal states can indicate whether an LLM is likely to hallucinate.The probing estimator also identifies whether a query appeared in training data with 80.28% accuracy.Source: §1 · p. 2

2 Hallucination and Training Data

• Hallucinations can arise from data-related causes, including unseen knowledge and noisy training data, or from modeling factors such as architecture and training objectives.Task-specific divergence is also identified as a data-related source.Source: §2 · p. 2
• To assess whether LLMs recognize queries seen during training, the authors collect news from periods before and after the analyzed model’s release, representing them as unseen and seen data, respectively.Source: §2 · p. 2
• Because training data cannot be definitively inspected and hallucinations may arise from either seen or unseen queries, the study also labels generated responses across NLG tasks by hallucination level.The labels assess the risk associated with a query through the response it elicits.Source: §2 · p. 2

3 Methodology

• The study analyzes whether internal states signal query familiarity or hallucination risk, visualizes selected neurons, and applies a probing classifier to the last-token state.The methodology proceeds through problem formulation, dataset construction, preliminary neuron analysis, and estimator design.Source: §3 · p. 2

3.1 Problem Formulation

• Hallucination risk h is estimated from internal states associated with a query, with labels based on training-data presence or hallucination degree in its generated response.The estimator is defined over model-specific and query-only attributes.Source: §3.1 · pp. 2–3

3.2 Data Construction

• The study uses 15 NLG task categories and over 700 datasets, evaluating generated responses with Rouge-L, NLI, and reference-dependent QuestEval.For the seen/unseen analysis, 2020 and 2024 BBC news queries are selected to represent likely-seen and unseen examples; response labels combine multiple evaluation signals.Source: §3.2 · pp. 3–4

3.3 Preliminary Analysis: Neurons for Hallucination Perception from Internal States

• Mutual-information feature selection identifies relevant dimensions, and eight neurons from the last activation layer distinguish uncertainty levels across dialogue, QA, and translation tasks.The visualized neurons are reported as sensitive to uncertainty and able to distinguish known from unknown queries associated with hallucinations.Source: Figure 2 · p. 3, §3.3 · p. 4

3.4 Internal State-based Estimator

• The estimator takes the activation for the query’s last token from a specified LLM layer and applies a Llama-adapted MLP with gating and SiLU.The architecture uses linear down-projection, up-projection, and gate layers.Source: §3.4 · p. 4

4.1 LLM

• The experiments primarily use Llama2-7B and also explore the impact of different internal states in Mistral-7B.Source: §4.1 · pp. 4–5

4.2 Baselines

• Baselines include zero-shot and in-context yes/no prompts, plus a perplexity threshold selected for maximum training-set accuracy and applied to test queries.Source: §4.2 · p. 5

4.3 Estimator Evaluation Protocols

• Classification performance is measured with F1 and Accuracy, and Figure 4 reports F1 scores for internal states from different layers.Source: §4.3 · p. 5, Figure 4 · p. 5
• Table 1 covers zero-shot automatic evaluation results within the same task and across different tasks.Source: Table 1 · p. 5

5.1 Results for Internal State-based Estimator

• The internal-state estimator distinguishes seen from unseen questions with 80.28% F1 and 80.24% accuracy.The authors interpret these results as evidence that internal states can identify queries likely absent from training data.Source: §5.1 · p. 5

(2) Hallucination Risk faced with the Query

• Across tasks, the internal-state estimator outperforms baselines on F1 and ACC and remains stable, with translation at F1 and ACC 76.90% and Number Conversion at F1 94.04%, ACC 96.00%.Zero-shot and ICL are similar, with ICL slightly better; both prompt methods have 99% recall while tending to overpredict accurate responses. PPL varies by task: translation F1 33.73%, ACC 50.36%; data-to-text F1 88.28%, ACC 92.08%.Source: (2) Hallucination Risk faced with the… · pp. 5–6

5.2 Analysis

• Deeper layers perform best at hallucination-risk estimation, while high-impact query tokens include unfamiliar named entities and critical information.The token analysis uses average input-embedding gradients; the authors suggest fluent response generation despite knowledge gaps may contribute to these token effects.Source: §5.2 · pp. 6–7, Figure 7 · p. 7
• Internal-state features generalize acceptably within individual tasks but transfer weakly across tasks, and a model’s own states remain its strongest predictor.Using Mistral-7B states to assess Llama2 lowers results on four NLG tasks compared with Llama2’s own states.Source: §5.2 · p. 6
• On the English samples of the out-of-domain ANAH generative-QA dataset, the QA-trained estimator reaches 78.56% F1 and 78.83% accuracy.The authors conclude that internal-state features transfer to out-of-domain data within the same task, but not across tasks.Source: §5.2 · p. 6
• The internal-state estimator takes 0.05 seconds per sample, compared with 1.36 seconds for the likelihood-based method.Generation takes 3.37 seconds at maximum token length 50 and batch size 1; QuestEval takes 10.25 seconds total.Source: §5.2 · pp. 6–7, Figure 5 · p. 6
• Hallucination rates vary across NLG tasks: Title Generation is highest and Number Conversion lowest.The authors attribute the contrast to Title Generation’s divergent nature and lack of a unique standard answer, versus Number Conversion’s relative ease and fixed answer.Source: §5.2 · p. 7, Figure 6 · p. 7
• The estimator still makes errors, including predicting a correct answer where the model hallucinates and predicting a hallucination where the reply is correct.The examples include a model answering “ligaments” instead of “cartilage,” and an estimator flagging a query that receives a non-hallucinated reply.Source: §5.2 · p. 7

6 Related Work

• Knowledge-boundary research often focuses on specialized tasks, while this method addresses multiple NLG tasks without LLM fine-tuning.The cited work distinguishes universal, model-agnostic unanswerability from model-specific predictions.Source: §6 · pp. 7–8
• Earlier hallucination approaches include response truthfulness, self-consistency, hallucinated-continuation detection, and query-only QA methods.The related-work discussion also connects uncertainty estimation with hallucination detection.Source: §6 · p. 8
• Unlike approaches measuring text that may come from another model, this work examines query-level self-awareness across NLG tasks.Source: §6 · p. 8

7 Conclusion

• 84.32% average estimation accuracy across tested datasets; internal-state assessment outperforms PPL-based and prompt-based baselines.Seen-query recognition reached 80.28% accuracy.Source: §7 · p. 8
• The study examines 15 NLG tasks spanning over 700 datasets and explores neurons, activation layers, and tokens involved in uncertainty perception.Source: §7 · p. 8
• Deeper activation layers correlate with higher predictive accuracy, while model-specific estimation remains optimal despite potential zero-shot transfer.Source: §7 · p. 8
• Cross-task generalization remains challenging despite promising generalization within the same NLG tasks; future work targets broader task and LLM coverage.Source: §7 · pp. 8–9

8 Limitation

• The study primarily investigates Llama2 and does not encompass other LLMs.The authors plan to extend the model scope.Source: §8 · p. 9
• The data-labeling process did not include human evaluation because of the time and financial costs of judging hallucinations at this dataset scale.Source: §8 · p. 9
• Because the method estimates risk from the query before generation, it may trade performance against approaches using both query and response.Source: §8 · p. 9

9 Ethical Considerations

• The study used public or synthetic datasets, excluded personally identifiable information, and anonymized data before analysis.The authors say the data were curated to uphold privacy and confidentiality.Source: §9 · p. 9
• The authors note that hallucinations may disproportionately affect demographic groups because of training-data biases and commit to identifying and rectifying such biases.Source: §9 · p. 9

A Dataset

• The benchmark contains 1,616 datasets across 76 task types; the study selects 15 NLG task types.Source: A Dataset · p. 13

• The three regression targets are absolute metric values, min-max-normalized values, and relative rankings; higher golden scores indicate fewer hallucinations.Source: B Results and Analysis · p. 13
• RMSE measures the average difference between predicted and actual continuous scores.Source: B Results and Analysis · p. 13
• RMSE is smallest when predicting absolute metric values and highest when predicting relative rankings, making precise ordering more challenging for hidden states.Source: B Results and Analysis · p. 13
• The Llama MLP outperforms the standard MLP backbone.Source: B Results and Analysis · p. 13

• The classifier input dimension is 4096 and its hidden dimension is 11008, aligned with Llama2-7B.Source: C Implementation Details · p. 13
• Training uses 10 epochs, batch size 128, learning rate 1e-5, AdamW with a linear scheduler, and one NVIDIA A800 GPU.Source: C Implementation Details · p. 13
• Figure A1 reports RMSE scores for Rouge-L, NLI, and QuestEval; Table A1 is titled as classifier-backbone results.Source: Figure A1 · p. 13, Table A1 · p. 13

• Other listed categories include 4 code-to-text tasks, 9 data-to-text tasks, 13 dialogue-generation tasks, and 6 explanation tasks.Source: D AI Assistants Using · p. 14
• The inventory also lists 2 grammar-error-correction tasks, 2 number-conversion tasks, 2 overlap-extraction tasks, and 12 paraphrasing tasks.Source: D AI Assistants Using · p. 14
• Program execution is listed with 90 tasks.Source: D AI Assistants Using · p. 14
• The inventory lists 207 question-answering tasks and 394 translation tasks.Source: D AI Assistants Using · pp. 15–16