PaperlayerView exact revision on arXiv

Source-linked AI summary

Do I Know This Entity? Knowledge Awareness and Hallucinations in Language Models

Javier Ferrando, Oscar Obeso, Senthooran Rajamanoharan, Neel Nanda

arXiv:2411.14257v2cs.CLcs.AIcs.LG

TL;DR

Language models can hallucinate, but how their internal mechanisms relate to hallucination or refusal on a given prompt remains poorly understood. This paper uses sparse autoencoders to identify entity-recognition directions, finding that they can steer chat-model refusal and are linked to the factual-recall mechanism.

  • Problem

    Mechanistic understanding of whether a language model will hallucinate on a given prompt remains limited, despite prior interpretability work on factual recall.

    Source: §1 · p. 1

  • Method

    The paper uses sparse autoencoders to identify representation-space directions that detect whether a model recognizes an entity and can recall facts about it.

    Source: §1 · pp. 1–2

  • Results

    Entity-recognition directions causally affect chat-model refusal, and unknown-entity directions disrupt factual recall by suppressing attention from attribute-extraction heads.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Entity recognition is one mechanism relevant to knowledge refusal and factual-recall behavior.

    Source: §1 · p. 2, §9 · p. 10

    This form of knowledge awareness does not establish other forms of self-knowledge and may be specific to factual recall.

    Source: §1 · p. 1

Abstract

from arXiv · show

Hallucinations in large language models are a widespread problem, yet the mechanisms behind whether models will hallucinate are poorly understood, limiting our ability to solve this problem. Using sparse autoencoders as an interpretability tool, we discover that a key part of these mechanisms is entity recognition, where the model detects if an entity is one it can recall facts about. Sparse autoencoders uncover meaningful directions in the representation space, these detect whether the model recognizes an entity, e.g. detecting it doesn't know about an athlete or a movie. This suggests that models can have self-knowledge: internal representations about their own capabilities. These directions are causally relevant: capable of steering the model to refuse to answer questions about known entities, or to hallucinate attributes of unknown entities when it would otherwise refuse. We demonstrate that despite the sparse autoencoders being trained on the base model, these directions have a causal effect on the chat model's refusal behavior, suggesting that chat finetuning has repurposed this existing mechanism. Furthermore, we provide an initial exploration into the mechanistic role of these directions in the model, finding that they disrupt the attention of downstream heads that typically move entity attributes to the final token.

1 INTRODUCTION

The paper investigates why language models hallucinate or refuse when asked about entities, a gap left by prior work focused mainly on recalling known facts. Using sparse autoencoders, it identifies entity-recognition directions that suggest knowledge awareness and affect refusal, hallucination, and factual-recall mechanisms.

1 INTRODUCTION

  • The authors use sparse autoencoders to find interpretable directions in representations, with Gemma Scope for Gemma 2 and LlamaScope for Llama 3.1 8B.The paper reports reproducing its findings on Llama 3.1 8B.Source: §1 · pp. 1–2
  • SAE-derived directions on an entity’s final token detect whether the model can recall facts about it, suggesting a form of knowledge awareness.The paper limits this claim to entity knowledge and notes it may be specific to factual recall.Source: §1 · pp. 1–2
  • Entity-recognition directions generalize across players, films, songs, and cities.The paper also describes directions that appear to represent uncertainty and predict incorrect answers.Source: §1 · p. 2
  • Base-model SAE directions causally steer chat-model knowledge refusal: unknown-entity steering promotes refusal, while known-entity steering can induce hallucination.The effect transfers to the chat model despite the SAEs being trained on the base model.Source: §1 · p. 2
  • Unknown-entity recognition directions disrupt factual recall by suppressing attention from attribute-extraction heads.Prior work identifies these heads as a key part of the factual-recall mechanism.Source: §1 · p. 2

2 SPARSE AUTOENCODERS

Sparse autoencoders represent model states using sparse combinations of learned directions. The section explains how thresholded latent activations are defined and how changing them provides a way to steer model representations.

2 SPARSE AUTOENCODERS

  • SAEs project model representations into a larger latent space and approximately reconstruct them as linear combinations of decoder latents, with a sparsity penalty encouraging sparse reconstructions.The reconstruction includes a decoder bias.Source: §2 · pp. 2–3
  • Figure 1 identifies final-token SAE latents that activate almost exclusively on known or unknown entities; increasing the known-entity latent can increase hallucination tendency.The example concerns a question about a made-up athlete.Source: Figure 1 · p. 3
  • The JumpReLU activation is zero below a learned threshold and equals its input above it; latent activation refers to a component of a(x), while latent direction refers to a decoder-dictionary row.The encoder and decoder each have weight matrices and biases.Source: §2 · p. 3
  • Changing an SAE latent’s activation is equivalent to steering the residual stream with its decoder direction.The representation is reconstructed as a linear combination of decoder latents plus a bias.Source: §2 · p. 3

3 METHODOLOGY

The study measures whether sparse-autoencoder latents distinguish entities the model can recall facts about from entities it cannot. It builds entity prompts, labels entities by recall performance, and compares latent activations across known and unknown sets.

3 METHODOLOGY

  • The dataset covers basketball players, movies, cities, and songs, with Wikidata attributes used to form prediction prompts.Prompts provide entity type, relation, and entity name; the model predicts the associated attribute.Source: §3 · p. 3
  • Figure 2 tracks the top five latents by known- and unknown-entity separation across layers, with middle layers performing best.The error bars show maximum and minimum scores, while MaxMin tracks the best latent’s minimum separation across entities.Source: Figure 2 · p. 4
  • Entities are labeled known when at least two attributes are correct and unknown when all are wrong; intermediate cases are discarded.The authors note that guessing or failure to recall selected attributes can introduce labeling inaccuracies; the split is 50% train, 10% validation, and 40% test.Source: §3 · p. 4
  • For each layer, the method compares SAE latent activity on the final entity token across known and unknown prompts to obtain separation scores.A latent is active when its activation is greater than zero; the score is the difference between known- and unknown-entity activation frequencies.Source: §3 · p. 4

4 SPARSE AUTOENCODERS UNCOVER ENTITY RECOGNITION DIRECTIONS

SAE latents distinguish known from unknown entities, with the strongest and most general distinctions appearing in intermediate layers. The authors also check activation patterns on song titles released after the models’ knowledge cutoff.

4 SPARSE AUTOENCODERS UNCOVER ENTITY RECOGNITION DIRECTIONS

  • Separation scores rise through the model, peak around layer 9, and then plateau.The pattern is reported across entity types.Source: §4 · p. 4
  • Latents with stronger minimum separation across entity types concentrate in intermediate layers, indicating more generalized known- and unknown-entity features there.The authors describe a progression from more specialized, lower-quality early-layer latents to more generalized features in middle layers.Source: §4 · pp. 4–5
  • The analysis excludes latents active on more than 2% of randomly sampled Pile tokens and focuses on those with the highest minimum separation across entity types.The minimum is taken across players, songs, cities, and movies.Source: §4 · pp. 4–5
  • On 283 post-cutoff song titles, unknown-entity latents activate more often and known-entity latents less often, though overlap with pretraining data is possible.The authors report consistent activation patterns across multiple models.Source: §4 · p. 5

5 ENTITY RECOGNITION DIRECTIONS CAUSALLY AFFECT KNOWLEDGE REFUSAL

Steering entity-recognition latents changes knowledge-refusal behavior in the chat model, including refusals about known entities and hallucinations about unknown ones. Projecting out the unknown-entity direction reduces refusal responses.

5 ENTITY RECOGNITION DIRECTIONS CAUSALLY AFFECT KNOWLEDGE REFUSAL

  • Knowledge refusals are detected in a 100-question test set using string matching for common refusal completions in the instruction-tuned chat model.The refusal definition excludes safety-based refusals and concerns answers declined for lack of information or database access.Source: §5 · p. 5
  • Unknown-entity latent steering raises Gemma 2 2B refusals on 100 unknown-entity queries to almost 100%, while known-entity latent steering slightly reduces refusals.Figure 3 compares the original model, steering with known and unknown latents, orthogonalization, and random-latent steering.Source: Figure 3 · p. 6, §5 · p. 6
  • Projecting the unknown-entity direction out of model weights substantially reduces refusal responses, consistent with that direction contributing to knowledge refusal.The same configuration with 10 random latents provides a comparison baseline.Source: §5 · p. 6, Figure 3 · p. 6
  • Steering with the known-entity latent makes the model hallucinate about the nonexistent player Wilson Brown, whom it otherwise refuses to discuss.Conversely, unknown-entity latent steering induces refusal about a well-known basketball player.Source: §5 · p. 6

6 MECHANISTIC ANALYSIS

The mechanistic analysis links entity-recognition directions to attention patterns used in factual recall. Steering these directions also shifts the model’s explicit Yes/No expression of whether it knows an entity.

6 MECHANISTIC ANALYSIS

  • Activation patching supports a recall circuit in which early heads merge entity names into the entity’s final token and downstream heads move attributes to the prompt’s final position.The patching analysis replaces unknown-entity representations with clean representations from known-entity prompts and measures recovered logit difference.Source: §6 · pp. 6–7, Figure 4 · p. 7
  • Unknown-entity latent steering reduces attention to entity tokens in downstream attribute-extraction heads, while known-entity latent steering increases it.Attention to entity tokens is already higher for known entities; the steering effects are most pronounced in attribute-extraction heads.Source: Figure 4 · p. 7, §6 · p. 7
  • On explicit knowledge questions, unknown-latent steering lowers the Yes-minus-No score for known entities, while known-latent steering shifts that score positively for unknown entities.For unknown entities, the original model already has a positive Yes-minus-No bias, which the authors interpret as difficulty expressing uncertainty accurately.Source: Figure 5 · p. 8, §6 · p. 8

7 UNCERTAINTY DIRECTIONS

The section searches chat-model representations for directions associated with uncertainty and errors, using SAE activations at the model token. A selected latent separates correct from incorrect test responses and is associated with uncertainty-related concepts.

7 UNCERTAINTY DIRECTIONS

  • The analysis excludes refusals and examines prompts eliciting correct predictions or errors, using pre-trained SAEs and train, validation, and test splits.The Gemma 2B IT SAE is at layer 13; Gemma 2 9B IT SAEs are at layers 10, 21, and 32.Source: §7 · pp. 8–9
  • The study reads residual-stream activations at the model token, hypothesized to aggregate information about the question, and compares correct and incorrect responses.The t-statistic measures the difference between activation sample means while accounting for within-sample variability.Source: §7 · pp. 8–9
  • The selected Gemma 2B IT latent separates correct from incorrect test responses, achieving 73.2 AUROC and an F1 score of 72 after threshold calibration.Similar latent separations are observed in Gemma 2 9B IT.Source: §7 · p. 9
  • The highest-scoring Gemma 2B IT latent activates on corpus text about uncertainty or undisclosed information, and its influence raises logits for tokens associated with unknownness.Figure 6 presents the latent's activations on correct and incorrect responses and the top tokens whose logits increase under its influence.Source: §7 · p. 9, Figure 6 · p. 9

8 RELATED WORK

Prior work has identified components of factual recall and mechanisms proposed to underlie hallucinations, but the mechanisms behind attribute-extraction failures remain incompletely understood. Related research also links internal signals to answer accuracy and examines predicting truthfulness and applying sparse autoencoders.

8 RELATED WORK

9 CONCLUSIONS

The paper uses sparse autoencoders to identify representation-space directions associated with entity knowledge awareness and uncertainty. Its conclusion reports that these directions relate to refusal, hallucination, attention to entity tokens, and answer correctness.

9 CONCLUSIONS

  • SAE-derived directions suggest encoded entity knowledge awareness, are causally relevant to knowledge refusal in chat models, and can be manipulated to control refusal or hallucination tendencies.The conclusion also reports directions representing query-specific uncertainty that discriminate between correct and mistaken answers.Source: §9 · p. 10

A ENTITY DIVISION INTO KNOWN AND UNKNOWN

  • The entity-division pipeline evaluates model responses to entity attributes and classifies known versus unknown using an accuracy threshold of τ = 1.Table 3 is captioned as listing entity types and attributes extracted from Wikidata.Source: Figure 7 · p. 19, Table 3 · p. 19

D GEMMA 2 9B LAYERWISE EVOLUTION OF THE TOP 5 LATENTS

  • The Gemma 2 9B layerwise comparison reports that middle layers provide the best-performing known- and unknown-entity latents.Figure 9 shows the top five latents and their separation scores, with error bars spanning maximum and minimum scores.Source: Figure 9 · p. 22

H ACTIVATION PATCHING

  • The appendix includes residual-stream activation-patching figures for movies, players, and cities in Gemma 2 2B, and songs, movies, players, and cities in Gemma 2 9B.Figure 12 is described as activation patching over the residual stream.Source: Figure 12 · p. 25, Figure 13 · p. 26, Figure 14 · p. 27

M CHANGE OF ATTENTION SCORES TO ENTITIES AFTER STEERING

Steering with known-entity latents generally increases attention to entity tokens, while steering with unknown-entity latents generally decreases it relative to random-latent steering. The analysis also includes self-knowledge-reflection and chat-model unknown-latent figures.

M CHANGE OF ATTENTION SCORES TO ENTITIES AFTER STEERING

  • Known-latent steering significantly increased entity-token attention versus random latents in 10/10 Gemma 2 2B, 10/10 Gemma 2 9B, and 7/10 Llama 3.1 8B tests.The tested comparison uses steering coefficients α = 100 for Gemma models and α = 20 for Llama 3.1 8B.Source: Figure 16 · p. 29, M CHANGE OF ATTENTION SCORES TO… · p. 29, Figure 18 · p. 31
  • The significance tests compare entity-recognition-latent steering with random SAE latents of the same layer and steering coefficient, using 100 prompts and 10 random latents.The stated alternative is increased attention for known latents and decreased attention for unknown latents.Source: M CHANGE OF ATTENTION SCORES TO… · p. 29
  • Unknown-latent steering significantly lowered attention in 9/10 Gemma 2 2B, 1/10 Gemma 2 9B, and 10/10 Llama 3.1 8B tests; Gemma 2 9B's second unknown latent differed significantly in 9/10 tests.The Gemma 2 9B top unknown latent did not show strong reductions.Source: Figure 17 · p. 30, M CHANGE OF ATTENTION SCORES TO… · p. 30, Figure 19 · p. 31

N GEMMA 2 9B SELF KNOWLEDGE REFLECTION

  • Figure 22 measures the logit difference between “Yes” and “No” answers to a self-knowledge question after steering with unknown or known entity-recognition latents.The caption specifies Gemma 2 9B.Source: Figure 22 · p. 33

O GEMMA 2 9B IT TOP ‘UNKNOWN’ LATENTS

  • Figure 23 presents the top two Gemma 2 9B IT unknown latents ranked by t-statistic score.Figure 24 presents the top two Gemma 2B IT unknown latents with errors separated by known and unknown entity types.Source: Figure 23 · p. 33, Figure 24 · p. 33

Q LLAMA 3.1 8B REPLICATION

The Llama 3.1 8B replication broadly reproduces Gemma’s entity-latent findings: the most effective and generalizable latents concentrate in intermediate layers, while steering and attention effects vary by latent.

Q LLAMA 3.1 8B REPLICATION

  • The most effective and generalizable entity latents concentrate in intermediate layers, consistent with earlier findings.Figure 25 shows layerwise separation scores for known and unknown latents.Source: Figure 25 · p. 34, Q LLAMA 3.1 8B REPLICATION · p. 34
  • Steering with the top unknown-entity latent increased refusals, while orthogonalizing it decreased refusals; the top known-entity latent did not decrease refusals.The figure compares refusal counts across 100 queries about unknown and known entities.Source: Figure 26 · p. 34, Q LLAMA 3.1 8B REPLICATION · p. 34
  • The replication of key Gemma findings in Llama 3.1 8B strengthens confidence in the approach and broader applicability across substantially different model architectures and SAEs.Source: Q LLAMA 3.1 8B REPLICATION · p. 34
  • Steering with the top known-entity latent increased attention to entity tokens, whereas the unknown-entity latent diminished it.Figure 27 reports aggregated attention scores to entity tokens.Source: Q LLAMA 3.1 8B REPLICATION · p. 34, Figure 27 · p. 35

R ACTIVATION FREQUENCY ON SONGS DATA AFTER KNOWLEDGE CUTOFF

  • Table 8 reports activation frequencies of top known and unknown entity latents for songs released after the August 2024 knowledge cutoff.Source: Table 8 · p. 35

S TOKEN LIKELIHOOD HYPOTHESIS

The authors test whether entity-recognition latents merely track token predictability by comparing activations with next-token probabilities and related measures. The observed associations are negligible overall, supporting a richer knowledge-awareness interpretation.

S TOKEN LIKELIHOOD HYPOTHESIS

  • In Gemma 2 2B, known-entity latents activate on 0.6% of tokens and unknown-entity latents on 0.5%.Table 9 covers activation frequency and correlation with conditional next-token probability for top latents in each model.Source: S TOKEN LIKELIHOOD HYPOTHESIS · p. 35, Table 9 · p. 36
  • Perplexity of surrounding tokens and next-token prediction entropy also showed no substantial correlations with the latents.Source: S TOKEN LIKELIHOOD HYPOTHESIS · p. 35
  • Unknown-entity latent activations coincide with modestly lower prediction probabilities, but sparse activation makes token predictability alone insufficient to explain the latents.The authors interpret these findings as supporting a more sophisticated form of knowledge awareness.Source: S TOKEN LIKELIHOOD HYPOTHESIS · pp. 35–36
  • Correlations between latent activations and ground-truth next-token probabilities are negligible across all tested models.Figure 28 compares probability distributions across all tokens and tokens activating known- or unknown-entity latents.Source: S TOKEN LIKELIHOOD HYPOTHESIS · p. 35, Figure 28 · p. 36
Loading 2411.14257v2…