PaperlayerView exact revision on arXiv

Source-linked AI summary

Distributional Semantics Tracing: A Framework for Explaining Hallucinations in Large Language Models

Gagan Bhatia, Somayajulu G Sripada, Kevin Allan, Jacobo Azcona

arXiv:2510.06107v3cs.CLcs.AIcs.CE

TL;DR

LLMs can generate fluent continuations unsupported by prompts, while existing interpretability approaches do not provide a compact layer-wise semantic trace. DST builds such maps and uses them to examine representational drift; in the reported Racing Thoughts evaluation, it achieves the highest mean faithfulness across models.

  • Problem

    LLMs can produce fluent outputs unsupported by prompts, while attribution, probing, and intervention methods have limitations in exposing compact layer-wise latent semantics for a single generation.

    Source: §1 · p. 1

  • Method

    DST builds layer-wise semantic maps by decoding answer-position residual states into concepts and estimating directed concept support with minimal-corruption causal tracing.

    Source: §1 · pp. 1–2, §3 · p. 3, §3.1 · p. 4

  • Results

    DST achieves the highest mean faithfulness across models in the Racing Thoughts evaluation setting.

    Source: §3.3 · p. 6

  • Takeaways & Limitations

    DST maps and CAS traces make representational drift observable, with operational layer markers indicating when semantic failures become detectable during the forward pass.

    Source: §4 · p. 8

    DST’s discrete top-K retrieval and detokenization can miss concepts or blur multi-token semantics, while tracing adds forward passes that may limit scalability.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Hallucinations in large language models (LLMs) produce fluent continuations that are not supported by the prompt, especially under minimal contextual cues and ambiguity. We introduce Distributional Semantics Tracing (DST), a model-native method that builds layer-wise semantic maps at the answer position by decoding residual-stream states through the unembedding, selecting a compact top-$K$ concept set, and estimating directed concept-to-concept support via lightweight causal tracing. Using these traces, we test a representation-level hypothesis: hallucinations arise from correlation-driven representational drift across depth, where the residual stream is pulled toward a locally coherent but context-inconsistent concept neighborhood reinforced by training co-occurrences. On Racing Thoughts dataset, DST yields more faithful explanations than attribution, probing, and intervention baselines under an LLM-judge protocol, and the resulting Contextual Alignment Score (CAS) strongly predicts failures, supporting this drift hypothesis.

1 Introduction

The paper addresses how to expose the semantic content of intermediate representations and trace when errors emerge, since existing explanation methods do not provide a compact layer-wise account. It introduces DST and proposes correlation-driven representational drift as a hypothesis for hallucinations.

1 Introduction

  • LLMs can produce fluent outputs unsupported by prompts, including factual hallucinations, misdisambiguation under minimal cues, and failures to follow counterfactual premises.Such behavior is reported in both standard generation and retrieval-augmented settings.Source: §1 · p. 1
  • Attribution, probing, and intervention methods respectively lack direct access to layer-wise semantics, yield token lists, or require multiple runs and additional experimental design.These limitations motivate lightweight diagnostics for a single generation.Source: §1 · p. 1
  • DST constructs layer-wise semantic maps by retrieving top-K concepts from answer-position residual states and tracing directed causal support between them.The maps summarize which concept neighborhood is assembled and how it supports competing continuations.Source: Figure 1 · p. 2, §1 · p. 2
  • DST defines Contextual Alignment Score (CAS) and operational layer markers to quantify representational drift and identify when semantic failures become detectable during the forward pass.The supplied contribution description does not specify the markers’ names or CAS’s formula.Source: §1 · p. 2
  • The paper proposes that hallucinations arise when correlation-driven drift pulls representations toward locally coherent but context-inconsistent neighborhoods reinforced by training co-occurrences.The authors state that DST maps and CAS traces make this drift observable and predictive of error.Source: §1 · p. 2

2 Related Work

Related work moves from characterizing hallucinations and mitigating their symptoms toward mechanistic interpretability of internal representations. The paper situates DST among methods that inspect model states and causally analyze components.

Causes of Hallucination

  • Early hallucination research developed error taxonomies and evaluation suites, while identifying training-data noise, brittle factual memorization, and shortcut learning as possible causes.Mitigation approaches include retrieval grounding and prompting for self-correction, but largely leave internal mechanisms unaddressed.Source: §2 · p. 3

Mechanisms of Interpretability

  • Mechanistic interpretability differs from classical XAI by causally analyzing model components, using tools such as causal tracing, sparse autoencoders, patching, and the Logit Lens.Prior work has used these tools to study factual knowledge, multi-hop reasoning, and multilingual processing.Source: §2 · p. 3
  • Mechanistic interpretability has also been applied to diagnose hallucination and to control behavior through representation engineering.The related-work discussion connects these efforts to research on architectural trade-offs in scaling.Source: §2 · p. 3

3 Tracing Semantic Failures

DST traces answer-position semantics across layers to examine when incorrect continuations emerge and how representational drift relates to hallucinations. The section reports higher explanation faithfulness for DST on Racing Thoughts, a strong relationship between CAS and failure, and examples of shifts toward context-inconsistent concepts.

3.1 Distributional Semantics Tracing (DST)

  • DST builds layer-wise semantic maps by reading answer-position residual states through the unembedding and selecting compatible concepts as graph nodes.The section defines the answer-position residual state as the representation being traced; DST’s maps summarize local compatibility and concept relationships rather than decomposing the full forward pass.Source: §3 · p. 3, §3.1 · p. 4
  • Directed edge weights estimate how much a concept’s supporting prompt evidence contributes to another concept by measuring probability drops after minimal token corruption.The procedure selects an influential prompt position for a node and re-runs the model after replacing that token.Source: §3.1 · p. 4

3.2 Evaluating Explanation Faithfulness

  • The evaluation compares DST with attribution, probing, and intervention methods by having an LLM judge explanations for identifying the controlling cue and its influence on the model’s prediction.The benchmark uses Racing Thoughts minimal pairs; explanations are normalized to cite evidence at each method’s output granularity.Source: §3.2 · pp. 5–6

3.3 Results: DST Produces More Faithful Traces

  • DST achieves the highest average faithfulness score on Racing Thoughts across four compact models, with an average score of 0.71.Explanations were assessed using an LLM-as-judge protocol.Source: Table 1 · p. 6, §3.3 · p. 6

3.4 When do hallucinations start?

  • CAS tracks alignment with context-consistent concepts, while onset, inversion, and commitment mark sustained decline, threshold crossing, and persistently low alignment.The semantic inversion threshold is 0.8; commitment marks low CAS persisting through the remaining depth.Source: §3.4 · pp. 6–7
  • In the bank ambiguity example, the map shifts from river-related concepts at Layer 15 toward financial concepts by Layers 23 and 27, alongside a wrong financial continuation.The river cue specifies the intended sense; the trace marks prediction onset, semantic inversion, and commitment across depth.Source: Figure 2 · p. 5, §3.4 · p. 7

3.5 Why do hallucinations occur?

  • The proposed account is that training co-occurrences can pull representations from multiple early neighborhoods toward a locally coherent but context-inconsistent correlated neighborhood.In an entity example, a neighborhood around the frequent entity Musk dominates despite the prompt specifying Elon Ganor, and next-token probability favors Elon Musk.Source: §3.5 · pp. 7–8
  • Final-layer CAS is negatively associated with hallucination rate across evaluation examples (Pearson r = −0.863; R2 = 0.746).Hallucination rate is reported as error rate over 750 evaluation examples per model.Source: Figure 3 · p. 7, §3.5 · p. 8

4 Conclusion

The conclusion presents DST as a model-native way to map layer-wise semantics and CAS with operational markers for tracking when semantic failures become detectable. It proposes testing whether similar dynamics extend to stereotypes and broader model misalignment.

4 Conclusion

  • DST produces layer-wise semantic maps for a single prompt by combining unembedding-based concept retrieval with lightweight causal tracing.Source: §4 · p. 8
  • CAS and onset, inversion, and commitment markers indicate when semantic failures become detectable during the forward pass.Source: §4 · p. 8
  • Future work will test whether similar layer-wise dynamics underlie stereotypes, model bias, and broader forms of model misalignment.Source: §4 · p. 8

Limitations

The stated limitations concern concept retrieval and edge estimation, computational cost, evaluation scope, and generalization to other architectures and retrieval pipelines.

Limitations

  • Top-K retrieval and detokenization can miss relevant concepts, blur compositional or multi-token semantics, and become noisy as K increases.Source: Limitations · p. 9
  • Edge magnitudes, and occasionally directions, may vary with corruption schemes, distributed evidence across spans, or unnatural local token replacements.Source: Limitations · p. 9
  • Step 3 requires additional forward passes per retrieved concept, potentially across layers, which may limit scalability for large models or long contexts.Source: Limitations · p. 9
  • Evaluation uses controlled ambiguity minimal pairs and an LLM-as-judge protocol, whose coverage of long-form, tool-augmented, or human-judged cases may be incomplete.Generalization across substantially different architectures and retrieval pipelines remains an open empirical question.Source: Limitations · p. 9

Ethical Considerations

DST raises dual-use, interpretation, bias, and privacy concerns that call for safeguards and careful handling of sensitive inputs.

Ethical Considerations

  • DST’s causal sensitivities could support auditing and drift localization, but also more effective steering or exploitation strategies.The authors call for use guidelines and safeguards for large-scale intervention sweeps.Source: Ethical Considerations · p. 9
  • Semantic maps show model-supported concept neighborhoods and counterfactual effects under a particular intervention design, not ground-truth reasoning.The authors recommend treating them as diagnostic evidence requiring corroboration.Source: Ethical Considerations · p. 9
  • DST may reveal biased training-data associations, while traces of sensitive prompts could expose private or proprietary content through stored readouts.The authors recommend fairness audits and careful access controls.Source: Ethical Considerations · p. 9
  • Data handling should match input sensitivity, including redaction and limited retention.Source: Ethical Considerations · p. 9

Broader Impact

DST is presented as a way to make hallucination debugging, evaluation, and auditing more informative, while its scope and risks call for careful communication and broader evaluation.

Broader Impact

  • Layer-wise semantic objects and a contextual-alignment trace could shorten debugging and support earlier, more targeted hallucination mitigation.The proposed trace identifies the onset of representational drift before final commitment.Source: Broader Impact · p. 9
  • DST could provide more informative evaluation than aggregate accuracy and connect black-box hallucination taxonomies with testable mechanistic accounts.The authors suggest this may inform architectural and training-objective choices that better preserve contextual grounding.Source: Broader Impact · p. 9
  • DST-style evidence could improve failure documentation and auditing, but risks adversarial steering and over-trust in visually coherent maps.The authors call for conservative scope communication and evaluation across diverse tasks and populations.Source: Broader Impact · p. 9

A Metric Validation.

The authors validate their faithfulness metric through comparison with LLM judges, model verbalisations, and targeted sensitivity tests.

Metric validation

  • The validation comprised an LLM-as-Judge evaluation, correlation analysis against model verbalisations, and sensitivity analysis.Source: A Metric Validation · p. 14
  • r = 0.9245 (p < 0.0001) between average LLM Judge Scores and the metric supports agreement with the five-judge panel’s faithfulness ratings.The panel scored 100 explanations, and annotation reliability was Fleiss’ Kappa = 0.85.Source: A Metric Validation · p. 14
  • r = 0.8942 (p < 0.001) indicates a strong positive semantic correlation between method explanations and the model’s Chain-of-Thought reasoning.Source: A Metric Validation · p. 14
  • Perturbations designed to degrade individual metric facets lowered the score, supporting sensitivity to components of explanation faithfulness.The authors conclude that the correlations and sensitivity analysis provide evidence of metric validity and reliability.Source: A Metric Validation · p. 14

B LLM-as-Judge Faithfulness Prompt

The appendix prompt asks an evaluator to judge explanation faithfulness for a specific prompt, output, and explanation using a defined rubric and score format.

LLM-as-Judge Faithfulness Prompt

  • Table 2 compares the faithfulness metric with LLM-judge examples and includes a small-sample Human Score for supplementary comparison.Source: Table 2 · p. 15
  • The evaluator rates whether an explanation identifies what most likely caused the model’s output for the specific prompt.Source: B LLM-as-Judge Faithfulness Prompt · p. 15
  • The requested response is a single 0–10 score and a 1–3 sentence justification referencing the prompt and explanation.Source: B LLM-as-Judge Faithfulness Prompt · p. 15
  • Faithfulness means identifying likely prompt evidence and a plausible mechanism for the output, even when the output is wrong.The rubric distinguishes matching model behavior from telling a persuasive story.Source: B LLM-as-Judge Faithfulness Prompt · p. 15
  • The rubric scores evidence grounding from 0–4, mechanistic plausibility from 0–4, and specificity and non-contradiction from 0–2.Source: B LLM-as-Judge Faithfulness Prompt · p. 15
  • Score bands run from 9–10 for strongly faithful explanations to 0 for completely unfaithful ones.Intermediate bands describe mostly faithful, mixed, and weak explanations.Source: B LLM-as-Judge Faithfulness Prompt · p. 15

C Results on Halogen Dataset

The Halogen evaluation reports explanation-technique faithfulness across five task domains and two Gemma2 model sizes.

C Results on Halogen Dataset

  • Table 3 covers five Halogen task domains and Gemma2 models sized 2B and 9B.It reports faithfulness results for explanation techniques; the passage does not provide comparative values.Source: Table 3 · p. 16

Related papers

Loading 2510.06107v3…