PaperlayerView exact revision on arXiv

Source-linked AI summary

LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinations

Hadas Orgad, Michael Toker, Zorik Gekhman, Roi Reichart, Idan Szpektor, Hadas Kotek, Yonatan Belinkov

arXiv:2410.02707v4cs.CLcs.AI

TL;DR

LLM error research has established that internal states can encode output truthfulness, but the scope and structure of that information remain incompletely understood. This paper probes internal representations to characterize truthfulness signals, error types, and their relation to generated answers, finding token-localized and multifaceted information alongside internal–external discrepancies. The work also identifies practical scope boundaries for applying these methods.

  • Problem

    Behavioral analyses do not adequately explain how diverse LLM errors are encoded internally, and the generality of internal truthfulness signals remains uncertain.

  • Method

    The paper trains probing classifiers on LLM internal representations and compares them with logit-, probability-, and prompting-based error-detection methods across varied tasks and datasets.

  • Results

    Internal representations encode token-localized truthfulness signals, predict error types, and can distinguish correct from incorrect answers even when the model repeatedly generates the incorrect answer.

  • Takeaways & Limitations

    Truthfulness information in LLMs is multifaceted rather than universal, while internal knowledge may offer information for error analysis, response selection, and targeted mitigation.

  • Takeaways & Limitations

    The approach requires access to internal representations and focuses on QA tasks with clear gold labels, leaving open-ended tasks for future research.

Abstract

from arXiv · show

Large language models (LLMs) often produce errors, including factual inaccuracies, biases, and reasoning failures, collectively referred to as "hallucinations". Recent studies have demonstrated that LLMs' internal states encode information regarding the truthfulness of their outputs, and that this information can be utilized to detect errors. In this work, we show that the internal representations of LLMs encode much more information about truthfulness than previously recognized. We first discover that the truthfulness information is concentrated in specific tokens, and leveraging this property significantly enhances error detection performance. Yet, we show that such error detectors fail to generalize across datasets, implying that -- contrary to prior claims -- truthfulness encoding is not universal but rather multifaceted. Next, we show that internal representations can also be used for predicting the types of errors the model is likely to make, facilitating the development of tailored mitigation strategies. Lastly, we reveal a discrepancy between LLMs' internal encoding and external behavior: they may encode the correct answer, yet consistently generate an incorrect one. Taken together, these insights deepen our understanding of LLM errors from the model's internal perspective, which can guide future research on enhancing error analysis and mitigation.

1 INTRODUCTION

This work studies hallucinations through LLM internal representations, finding that they encode detailed truthfulness information. It identifies token-localized signals, task-specific error detection, predictable error types, and cases where internal knowledge diverges from generated behavior.

  • The study examines factual inaccuracies, biases, common-sense failures, and other LLM errors through internal representations rather than only behavioral analysis.
  • Truthfulness information is concentrated in exact answer tokens, and using these tokens substantially improves error detection.For example, the relevant token can be the answer itself, such as “Hartford” in a factual response.
  • Probing classifiers generalize within similar-skill tasks but not reliably across different tasks, challenging the idea of a universal truthfulness encoding.The findings instead indicate multiple, skill-specific notions of truth and motivate caution in deploying trainable error detectors.
  • Internal representations also predict error types, including whether an error is repeatedly generated or occurs occasionally among different errors.This supports more targeted error mitigation strategies.
  • LLMs can internally distinguish correct from incorrect answers yet repeatedly generate the incorrect response, revealing a discrepancy between internal encoding and external behavior.The discrepancy may support strategies that harness information already present in the model.

2 BACKGROUND

The background reviews competing definitions of hallucinations and existing approaches to detecting LLM errors. It motivates methods that use the model’s own computations and internal representations across broad error types.

  • “Hallucination” lacks a consensus definition, with different subfields and frameworks interpreting the term differently.One cited analysis identified 31 distinct frameworks for conceptualizing hallucinations.
  • This study adopts a broad definition of hallucinations as any LLM-generated error, including factual inaccuracies, biases, and common-sense reasoning failures.
  • Existing error-detection research includes task-specific methods, but general-purpose LLMs motivate approaches capable of addressing varied error types.
  • Some detection methods use external knowledge sources or external LLM judges, whereas this work relies on logits, probabilities, and hidden states computed by the LLM.
  • Uncertainty estimation commonly uses logits, agreement across sampled answers, verbalized probabilities, or direct prompting to derive confidence signals.
  • Probing classifiers train small classifiers on model representations to discover and use truthfulness features, often from answer tokens.

3 BETTER ERROR DETECTION

The section defines white-box error detection and evaluates methods across diverse models and datasets. It finds that truthfulness signals are localized in exact answer tokens, where probing classifiers perform especially well.

  • 3.1 TASK DEFINITION: Error detection predicts whether an LLM-generated response is correct using internal states without external resources.Correctness labels are obtained by comparing generated responses with ground-truth answers; refusals are excluded.
  • 3.2 EXPERIMENTAL SETUP: Experiments cover four LLMs and ten datasets spanning question answering, reasoning, classification, sentiment, and movie-role tasks.Responses are generated unrestrictedly with greedy decoding.
  • 3.2 EXPERIMENTAL SETUP: Exact answer tokens are generated tokens whose modification changes answer correctness, excluding subsequent content.The study compares this location with commonly used prompt-final or response-final token choices.
  • 3.3 RESULTS: Truthfulness signals appear immediately after the prompt, weaken during generation, and peak again at exact answer tokens, especially in middle-to-late layers.These patterns are reported across nearly all datasets and models.
  • 3.3 RESULTS: Using exact answer tokens improves most error-detection methods across datasets, while probing consistently outperforms the reported baselines.The comparison uses AUC across models and datasets; the last exact answer token performs best among tested exact-answer positions.

4 GENERALIZATION BETWEEN TASKS

Probe-based truthfulness signals show limited cross-dataset generalization beyond what output logits already provide. Some transfer remains between tasks requiring similar skills, indicating multifaceted rather than universal truthfulness representations.

  • Method: The broader ten-dataset evaluation tests whether truthfulness probes trained with each dataset’s optimal token-layer combination transfer to other datasets.Generalization is evaluated across realistic task settings using absolute AUC, which accounts for reversed learned signals.
  • Results: Most apparent cross-dataset probe generalization rarely exceeds performance from the strongest logit-based baseline after adjustment.The adjusted heatmap subtracts Logit-min-exact performance, which uses output logits alone.
  • Results: Successful transfer appears mainly among tasks requiring similar skills, including factual retrieval and common-sense reasoning.Examples include TriviaQA, HotpotQA, and Movies for factual retrieval, plus Winobias, Winogrande, and NLI for common-sense reasoning.
  • Conclusion: These findings support multiple skill-specific truthfulness mechanisms rather than a single universal encoding.The paper characterizes truthfulness representations as multifaceted, with distinct mechanisms corresponding to different notions of truth.

5 INVESTIGATING ERROR TYPES

The paper analyzes repeated generations to classify behavioral error patterns and tests whether intermediate representations encode those fine-grained types. Error types are predictable from these representations, supporting targeted analysis and interventions.

  • 5.1 TAXONOMY OF ERRORS: K = 30 responses sampled at T = 14 expose distributions distinguishing consistent errors, occasional mistakes, and diffuse incorrect answers.The analysis examines repeated responses for each TriviaQA example.
  • 5.1 TAXONOMY OF ERRORS: The taxonomy records answer diversity, correct-answer frequency, and most-common-incorrect-answer frequency to define recurring error patterns.Categories include refusal, consistently correct, consistently incorrect, and more varied answer patterns.
  • 5.1 TAXONOMY OF ERRORS: The taxonomy covers 96% of TriviaQA errors for Mistral-7b-instruct, despite some overlap between categories.It is explicitly presented as an extrinsic, behavior-based analysis of error patterns.
  • 5.2 PREDICTING ERROR TYPES: A one-to-many probe trained on greedy-decoding answer representations predicts specific error types from intermediate activations.The setup tests each error type against all others.
  • 5.2 PREDICTING ERROR TYPES: Error-type predictability indicates that representations encode fine-grained information beyond binary output correctness.Performance varies across types, but types C and D remain above random.

6 DETECTING THE CORRECT ANSWER

The paper uses an error-detection probe to select among 30 generated answers and compares it with standard decoding, random selection, and majority vote. Probe selection improves accuracy most for error types lacking an external preference for the correct answer, revealing a gap between internal encoding and generation behavior.

  • Experimental Setup: The probe selects the highest-correctness-probability answer from K = 30 candidates across TriviaQA, Winobias, and Math.Comparisons use greedy decoding, random selection, and majority vote baselines.
  • Results: 30–40 points of improvement occur in some TriviaQA categories, especially mostly incorrect, two competing answers, and many answers.The mostly correct category shows minimal gain.
  • Results: Probe selection improves accuracy across all examined tasks, but gains vary by error type.The probe is most effective when the model lacks an external preference for the correct answer during generation.
  • Scope: The probe is presented as a diagnostic tool rather than as an error-mitigation strategy.The authors describe mitigation as a direction for future research.

7 DISCUSSION AND CONCLUSIONS

The study finds that internal representations support error detection and error-type prediction, but truthfulness features generalize poorly across tasks and datasets. It also identifies cases where models encode correct answers while repeatedly generating incorrect responses, within a scope limited to open-source models and closed-label QA tasks.

  • Scope: The study tested 10 datasets across 4 model architectures but focused on open-source models and QA tasks with clear gold labels.Open-ended tasks remain for future research.
  • Limitations: Truthfulness features showed poor generalization across tasks and datasets, requiring caution when applying trained error detectors in varied settings.The authors suggest exploring layer-token combinations, diverse training datasets, and task-specific feature overlaps.
  • Implications: Task-specific probes can detect errors, predict error types, and guide response selection from resampled outputs in reliability-critical fields.The discussion specifically identifies medicine and law as potential application areas.
  • Conclusions: Models may repeatedly output incorrect responses despite internally encoding the correct answer, revealing a discrepancy between internal states and external behavior.The authors suggest likelihood-driven mechanisms may override truthfulness-promoting mechanisms.

8 REPRODUCIBILITY STATEMENT

The reproducibility statement provides source code, instructions, seeds, and execution guidance for the experiments. It also situates the implementation within broad error detection across diverse LLM tasks.

  • Reproducibility: The repository includes scripts, command examples, seeds, and documentation for setup, dataset preparation, and experiment execution.The materials cover answer generation, probing, resampling, and error-type analysis.
  • Scope: The study targets general error detection across factual inaccuracies, biases, arithmetic mistakes, and other LLM error categories.The dataset suite was designed to cover varied tasks and model limitations.

A.2 PROBING: IMPLEMENTATION DETAILS

The implementation probes intermediate representations at extracted exact-answer tokens and constructs correctness-labeled datasets across diverse QA and classification tasks. It also compares probability aggregation baselines and documents extraction, labeling, splitting, and dataset procedures.

  • Probing: The analysis uses intermediate representations from the final MLP output at exact answer tokens.Preliminary comparisons found no significant differences between MLP output, residual stream, and attention heads.
  • Probing: The probing classifier uses logistic regression with default L2 and LBFGS settings, an 80-20 train-validation split, and bootstrap-sampled tests.These implementation choices were applied for each random seed.
  • Labeling: Correctness labels are obtained by comparing generated responses with gold answers using heuristics or an instruct-tuned LLM.Natural Questions With Context receives additional validation because the heuristic produced false negatives.
  • Answer extraction: Exact answers are extracted from long responses, then four boundary tokens around the answer span are identified for probing.For multi-token answers, the first and last answer tokens are probed separately.
  • Answer extraction: Questions are retained only when a valid exact answer is successfully extracted, reducing correlation between invalid answers and incorrectness labels.The extraction process prompts an instruct LLM up to five times and verifies substring validity.
  • Datasets: The dataset suite spans varied skills and error types, using free-text generation and generally 10K training and 10K test samples per dataset.Math uses approximately 2,000 training and 650 test samples, while IMDB uses one-shot prompting.
  • Baselines: Probability baselines aggregate log-probabilities or raw probabilities over generated tokens, with additional strategies restricted to exact answer tokens.The study also evaluates minimum and maximum aggregation alongside the mean.

B FULL ERROR DETECTION RESULTS

Full error-detection results report AUC comparisons across four model variants and datasets, while layer-token analyses show that probe performance concentrates at exact answer tokens. Generalization results broadly match the main-paper pattern but differ across models, suggesting distinct truthfulness-encoding mechanisms.

  • Token-layer analysis: Layer-token AUC patterns are similar across datasets and models.The reported figures focus on Mistral-7b-instruct and note comparable patterns for other models.
  • Token-layer analysis: Detection performance spikes at the exact answer tokens across layers and tokens for Mistral-7b-instruct.The figure reports AUC values for a probe error detector.
  • Error detection: AUC comparisons are provided across Mistral-7B, Mistral-7B-Instruct, Llama-8b, and Llama-8b-Instruct.Tables 4–7 contain the full baseline and dataset results.
  • Generalization: Generalization results for other models show similar high-level patterns to Mistral-7b-instruct but also notable differences.These differences suggest that models may use different mechanisms to encode truthfulness.

D TAXONOMY OF ERRORS

The paper develops a nuanced, non-orthogonal taxonomy of LLM error types and shows that these types can be predicted from internal representations. It also documents evaluation across Winobias, Math, TriviaQA, and resampling-based analyses.

  • Resampling: For TriviaQA with Mistral-7b-instruct, resampling shows that some answers missed by greedy decoding become correct in at least one sample, with the plot plateauing around 30 resamples.Figure 10 reports the percentage of answers for which at least one generated answer was correct.
  • Error taxonomy: Error types can overlap, such as answers that are consistently incorrect while occasionally producing the correct answer and otherwise generating many variants.The taxonomy permits simultaneous category membership rather than forcing each error into one class.
  • Error taxonomy: The taxonomy is intended to capture nuanced error patterns and support general insights across different error types.The stated goal is to provide actionable information for questions exhibiting overlapping error patterns.
  • Error taxonomy: One-to-many probes support the non-orthogonal taxonomy by allowing a single case to receive multiple error-type classifications.This design is presented as enabling precise error analysis and tailored solutions.
  • Evaluation: Error-type classification results are reported for Winobias and Math, with AUC tables documenting the evaluations.Table 8 covers Winobias, while Table 9 covers Math.
  • Evaluation: Qualitative examples of the error types are provided for TriviaQA and Math.The examples are presented in Tables 10 and 11.

E DETECTING THE CORRECT ANSWER FULL RESULTS

The paper describes how internal representations and error-type predictions can guide answer selection and targeted interventions. Its examples include probe-selected correct answers, alternative decoding strategies, and task-specific integration guidance.

  • Correct-answer detection: For consistently incorrect cases that occasionally yield the correct answer, the probe successfully identifies the correct answer in sampled examples.Table 12 reports Mistral-7b-instruct questions sampled 30 times and selected by the probe.
  • Correct-answer detection: The paper reports similar conclusions across datasets and models, with significant improvement for error types where the LLM shows no preference for the correct answer.Tables 13 and 14 compare answer-choice strategies for non-instruct and instruct models.
  • Integration guidance: Exact answer-token representations can support task-specific probes that provide confidence scores for warning users or intervening on unreliable outputs.The paper gives math calculations as an example of a task-specific deployment.
  • Integration guidance: Predicted error types can inform customized mitigation strategies, including RAG, resampling, probe-ranked answer selection, or weight updates.The proposed intervention depends on the predicted error category.
  • Integration guidance: Universal generalization across unrelated tasks should be approached cautiously because probing classifiers are mainly useful for task-specific error detection.This guidance follows the reported cross-task generalization boundary.
Loading 2410.02707v4…