PaperlayerView exact revision on arXiv

Source-linked AI summary

ReDeEP: Detecting Hallucination in Retrieval-Augmented Generation via Mechanistic Interpretability

Zhongxiang Sun, Xiaoxue Zang, Kai Zheng, Yang Song, Jun Xu, Xiao Zhang, Weijie Yu, Yang Song, Han Li

arXiv:2410.11414v2cs.CL

TL;DR

RAG systems can still hallucinate when accurate retrieved information conflicts with parametric knowledge, and existing detection approaches do not always disentangle these sources. The paper uses mechanistic interpretability to study their roles and proposes ReDeEP, which significantly improves detection performance; it also introduces AARF to mitigate hallucinations.

  • Problem

    RAG models may contradict or fail to support accurate retrieved context, while existing detection approaches can mix external and parametric knowledge without separating their contributions.

    Source: §1 · p. 2

  • Method

    The paper uses mechanistic interpretability to measure external-context and parametric-knowledge use, then treats these as covariates in ReDeEP for hallucination detection.

    Source: §1 · pp. 2–3

  • Results

    ReDeEP significantly improves performance across the LLaMA family and RAG hallucination benchmarks, outperforming existing detection methods.

    Source: §1 · p. 3, §6 · p. 11

  • Takeaways & Limitations

    The identified mechanisms guide interventions that reduce hallucinations without updating model parameters.

    Source: §6 · p. 11

    The proposed context and knowledge metrics are intuitive proxies and may not be exact given LLM complexity.

    Source: §3.1 · p. 5

Abstract

from arXiv · show

Retrieval-Augmented Generation (RAG) models are designed to incorporate external knowledge, reducing hallucinations caused by insufficient parametric (internal) knowledge. However, even with accurate and relevant retrieved content, RAG models can still produce hallucinations by generating outputs that conflict with the retrieved information. Detecting such hallucinations requires disentangling how Large Language Models (LLMs) utilize external and parametric knowledge. Current detection methods often focus on one of these mechanisms or without decoupling their intertwined effects, making accurate detection difficult. In this paper, we investigate the internal mechanisms behind hallucinations in RAG scenarios. We discover hallucinations occur when the Knowledge FFNs in LLMs overemphasize parametric knowledge in the residual stream, while Copying Heads fail to effectively retain or integrate external knowledge from retrieved content. Based on these findings, we propose ReDeEP, a novel method that detects hallucinations by decoupling LLM's utilization of external context and parametric knowledge. Our experiments show that ReDeEP significantly improves RAG hallucination detection accuracy. Additionally, we introduce AARF, which mitigates hallucinations by modulating the contributions of Knowledge FFNs and Copying Heads.

1 INTRODUCTION

ReDeEP detects RAG hallucinations by separating the roles of external context and parametric knowledge, while AARF modulates model components to mitigate hallucinations. The paper reports that accurate retrieved context can still conflict with parametric knowledge and that ReDeEP outperforms existing detection methods.

1 INTRODUCTION

  • Figure 1 contrasts a correct response grounded in external knowledge with a hallucination despite accurate external context.Both examples involve correct retrieved documents that conflict with parametric knowledge.Source: Figure 1 · p. 1
  • RAG hallucinations can occur despite accurate, relevant retrieved context, so the paper distinguishes them from knowledge conflicts, which do not always cause hallucinations.The study focuses on detection when the retrieved external context is accurate and relevant.Source: §1 · p. 2
  • Existing detection approaches focus on parametric knowledge, external context, or both, but the mixed approach does not decouple their individual contributions.The paper frames these approaches as three causal frameworks in which one knowledge source confounds the other or both are mixed.Source: §1 · p. 2
  • Hallucinations typically involve Knowledge FFNs over-adding parametric knowledge while Copying Heads neglect or fail to retain retrieved information.The paper reports that the model can lose information attended to by Copying Heads during generation.Source: §1 · pp. 2–3
  • ReDeEP significantly outperforms existing detection methods, while AARF improves the truthfulness of LLaMA models.ReDeEP treats parametric knowledge and external context as covariates; AARF modulates Knowledge FFNs and Copying Heads in the residual stream.Source: §1 · p. 3

2 BACKGROUND AND RELATED WORKS

The background explains how attention heads and FFNs contribute information to transformer residual streams. The related work motivates mechanistic interpretability for analyzing RAG hallucinations and contrasts detection approaches that do not disentangle external and parametric knowledge.

2.1 BACKGROUND

  • Decoder-only Transformers pass a residual stream through layers, with attention heads and FFNs adding information that contributes to the final prediction.The paper studies this architecture through mechanistic interpretability.Source: §2.1 · p. 3
  • Copying Heads transfer information between tokens through their OV circuits, helping preserve attended tokens for external-context use.They are identified by analyzing positive eigenvalues of the OV matrix.Source: §2.1 · p. 3
  • FFNs store parametric knowledge as key-value pairs and retrieve and integrate that information into predictions.Keys encode knowledge, while values represent its output.Source: §2.1 · p. 3
  • LogitLens decodes hidden states into vocabulary distributions using LayerNorm and the unembedding matrix for interpretability.Figure 3(a) depicts the calculation of the External Context Score and Parametric Knowledge Score, while panels (b) and (c) show interventions on attention heads and FFNs.Source: §2.1 · p. 3, Figure 3 · p. 4

2.2 RELATED WORK

  • The paper applies mechanistic interpretability to study RAG hallucination mechanisms through LLM use of external and parametric knowledge.It situates this work among research on FFNs, attention heads, residual streams, and the logit lens.Source: §2.2 · pp. 3–4

3 EMPIRICAL STUDY

The study measures how LLMs use retrieved context and parametric knowledge, then examines how those mechanisms relate to RAG hallucinations. Its experiments associate hallucinations with weaker use or retention of external information and greater parametric-knowledge contributions from later-layer FFNs.

3 EMPIRICAL STUDY

  • The empirical study uses Llama2-7B-chat and the RAGTruth training set, whose examples include a query, retrieved context, response, and hallucination label.The label is 0 for truth and 1 for hallucination.Source: §3 · p. 4

3.1 METRICS FOR LLMS’ UTILIZATION OF EXTERNAL CONTEXT AND PARAMETRIC KNOWLEDGE

  • The External Context Score assesses attention to relevant retrieved context and whether that information is retained and used during generation.The Parametric Knowledge Score uses LogitLens and Jensen-Shannon divergence to quantify the parametric knowledge added by FFNs to the residual stream; the authors describe both metrics as intuitive proxies rather than exact measures.Source: §3.1 · pp. 4–5

3.2 EXPERIMENTS

  • 1006 of 1024 Llama2-7B attention heads have higher external-context scores on truthful than hallucinated responses.Most heads also show a negative correlation between external-context scores and hallucination labels; further analysis attributes the main issue to losing attended information during generation rather than failing to attend to it.Source: §3.2 · p. 6
  • Later-layer FFNs have significantly higher parametric-knowledge scores for hallucinated responses, with higher scores on average across all layers.These later-layer FFNs are designated Knowledge FFNs; their scores are positively correlated with the hallucination label, and the paper concludes that excessive parametric-knowledge addition leads to hallucinations.Source: §3.2 · pp. 6–7
  • Intervening on Copying Heads and Knowledge FFNs has a significantly greater impact on negative log-likelihood loss than intervening on control heads and FFNs.The study summarizes the two causally related factors as losing information retrieved by Copying Heads during generation and excessive parametric-knowledge injection by Knowledge FFNs.Source: §3.2 · p. 7
  • When the model knows the truthful answer, Copying Heads more accurately capture and use external knowledge, while Knowledge FFNs add less parametric knowledge than in hallucination scenarios.The comparison uses an LLM-known dataset and a hallucination dataset.Source: §3.2 · p. 7

4 METHODS

ReDeEP detects RAG hallucinations by modeling external-context use and parametric-knowledge use separately. The section also presents chunk-level scoring and AARF, which intervenes on attention heads and FFNs without updating model parameters.

4.1 TOKEN-LEVEL HALLUCINATION DETECTION — REDEEP (TOKEN)

  • ReDeEP regresses decoupled External Context and Parametric Knowledge Scores as covariates to predict hallucinations and address confounding.The study links hallucinations to insufficient external-context use by Copying Heads and excessive parametric-knowledge reliance by Knowledge FFNs.Source: §4.1 · p. 8, §4 · p. 8

4.2 CHUNK-LEVEL HALLUCINATION DETECTION — REDEEP (CHUNK)

  • Chunk-level ReDeEP segments retrieved context and responses, pairing high-attention chunks and comparing their embeddings for an external-context score.It sums token-level parametric-knowledge scores within each chunk to form the chunk-level score.Source: §4.2 · p. 8

4.3 TRUTHFUL RAG GENERATION — AARF

  • AARF reduces RAG hallucinations by intervening on attention heads and FFNs without updating model parameters.It first detects hallucinations token by token, then reweights attention-head and FFN contributions to the residual stream.Source: §4.3 · p. 8
  • When a token’s hallucination score exceeds a threshold, AARF amplifies Copying Head contributions and reduces Knowledge FFN contributions.Below or at the threshold, it uses normal output computation.Source: §4.3 · p. 9

5 EXPERIMENTS

Across two datasets, multiple backbones, and different metrics, ReDeEP consistently improves RAG-hallucination detection performance. The experiments also evaluate AARF’s effect on truthfulness using GPT-4-o pairwise judgments.

5.1 SETTINGS

  • Evaluation uses RAGTruth and Dolly (AC), covering QA, data-to-text writing, news summarization, text summarization, closed-QA, and information extraction.RAGTruth is described as a manually annotated RAG-hallucination dataset, while Dolly (AC) has accurate context.Source: §5.1 · p. 9
  • Experiments use three LLaMA variants and assess detection with AUC, PCC, accuracy, recall, and F1.The baselines span Parametric Confounded by External, External Confounded by Parametric, and Mixed Parametric and External methods.Source: §5.1 · p. 9

5.2 EXPERIMENTS

  • ReDeEP consistently improves detection performance across two datasets, various backbone methods, and different metrics.The comparison is against baselines in Table 1.Source: §5.2 · p. 9, Table 1 · p. 10
  • GPT-4-o pairwise judgments on RAGTruth and Dolly (AC) indicate that AARF reduces hallucinations to a certain extent versus the baseline.The evaluation uses automatic assessment of truthfulness.Source: §5.2 · p. 9

6 CONCLUSION

The conclusion attributes RAG hallucinations to insufficient external-context use by Copying Heads and over-reliance on parametric knowledge by Knowledge FFNs. It reports that ReDeEP outperforms existing detection methods across the LLaMA family and RAG-hallucination benchmarks.

6 CONCLUSION

  • The study reports that ReDeEP improves performance across the LLaMA family and RAG-hallucination benchmarks, outperforming existing detection methods.It detects hallucinations by analyzing LLM use of parametric knowledge and external context.Source: §6 · pp. 10–11

A FULL BACKGROUND ON ATTENTION HEADS, FFNS, AND LOGIT LENS

  • The background frames the residual stream as accumulating information from attention heads and FFNs across transformer layers.The final residual-stream state is projected into vocabulary space to produce a probability distribution for token sampling.Source: A FULL BACKGROUND ON ATTENTION HEADS,… · p. 15

B CALCULATION OF COPYING HEADS SCORE

  • The proposed Copying Head Score estimates a head’s copying tendency using trace-based ranking refined with the Gershgorin Circle Theorem.This avoids directly calculating eigenvalues of large matrices, which the traditional identification method requires.Source: B CALCULATION OF COPYING HEADS SCORE · p. 16

C DIVE INTO THE RATIONALE BEHIND THE EXTERNAL CONTEXT SCORE

The external context score distinguishes whether attention heads focus on the correct retrieved information from whether the model retains and uses it during generation. The analysis finds low scores mostly reflect loss of attended information, and the intervention procedures probe the roles of Copying Heads and Knowledge FFNs.

C DIVE INTO THE RATIONALE BEHIND THE EXTERNAL CONTEXT SCORE

  • The validation experiment tests whether the attended span supports a conflict in a hallucinated response, separating correct attention from effective retention and use.The attended span is selected by mean-pooling attention scores over segmented retrieved-document spans; GPT-4-o evaluates whether it supports the conflict.Source: C DIVE INTO THE RATIONALE BEHIND THE… · p. 17
  • Low external context scores mostly reflect the model losing information that attention heads attended to, although the heads usually focus on the appropriate context.Table 2 reports the proportion of cases where Llama2-7B attention heads attend to correct information.Source: C DIVE INTO THE RATIONALE BEHIND THE… · p. 17, Table 2 · p. 17
  • AARF aims to reduce hallucinations by increasing Copying Heads’ output in the residual stream.Source: C DIVE INTO THE RATIONALE BEHIND THE… · p. 18

D DETAILED INTERVENTION PROCEDURES

  • Attention-score noise simulates removing meaningful attention patterns to test how Copying Heads’ focus on external context affects hallucinations.The intervention samples noise from a standard normal distribution.Source: D DETAILED INTERVENTION PROCEDURES · p. 18
  • The Knowledge FFN intervention increases FFN contributions to the residual stream tenfold (k = 10) to examine parametric knowledge’s influence on generation.Source: D DETAILED INTERVENTION PROCEDURES · p. 18
  • Causal matching pairs the top 32 Copying Heads and top 5 hallucination-related Knowledge FFNs with nearby controls to reduce layer- or head-position bias.Copying Heads are matched within the same layer; Knowledge FFNs are matched in adjacent layers.Source: D DETAILED INTERVENTION PROCEDURES · p. 18

E DETAILED ANALYSIS OF RQ3: HALLUCINATION BEHAVIOR ANALYSIS

The analysis considers whether the model’s parametric knowledge contains the truthful answer and examines how Copying Heads and Knowledge FFNs behave in each case. When the answer is known, Copying Heads more accurately capture and retain external knowledge, while Knowledge FFNs add less parametric knowledge than in hallucination data.

E DETAILED ANALYSIS OF RQ3: HALLUCINATION BEHAVIOR ANALYSIS

  • When parametric knowledge lacks the truthful answer, hallucinations may arise as Knowledge FFNs over-add parametric information while Copying Heads fail to attend to or retain retrieved information.Source: E DETAILED ANALYSIS OF RQ3:… · p. 18
  • The known-answer dataset is formed by testing responses on the truthful dataset without retrieved documents and retaining responses GPT-4-o judges to match the expected answers.Source: E DETAILED ANALYSIS OF RQ3:… · pp. 18–19
  • When parametric knowledge contains the truthful answer, responses are typically truthful, Copying Heads use external knowledge more accurately and stably, and Knowledge FFNs add less parametric information than in hallucination data.Source: E DETAILED ANALYSIS OF RQ3:… · pp. 18–19

F PROMPT FOR EVALUATING PARAMETRIC KNOWLEDGE

G DETAILS ABOUT RAG HALLUCINATION DATASETS

The paper describes manual hallucination annotation for RAGTruth and Dolly (AC), then groups detection baselines by how they use parametric knowledge and external context. The dataset notes also identify a model-specific annotation gap for RAGTruth.

G DETAILS ABOUT RAG HALLUCINATION DATASETS

  • RAGTruth covers QA, Data-to-Text Writing, and News Summarization, but lacks Llama3-8B hallucination labels, so three graduate-level annotators assessed its RAG responses.Annotations assess accuracy and relevance of retrieved and generated content.Source: G DETAILS ABOUT RAG HALLUCINATION… · p. 19
  • Three qualified annotators manually assessed Dolly (AC) RAG responses for hallucinations, checking whether generated content aligns with external context.Source: G DETAILS ABOUT RAG HALLUCINATION… · p. 19

H DETAILS ABOUT BASELINE MODELS

I IMPLEMENTATION DETAILS.

Experiments use greedy decoding and default model parameters, with validation and hyperparameter tuning procedures varying by dataset. ReDeEP and AARF configurations differ across model-dataset combinations.

  • Experiments ran on V100 GPUs and 52-core Intel Xeon Gold 6230R CPUs using Hugging Face Transformers, greedy decoding, and otherwise default model settings.Source: I IMPLEMENTATION DETAILS · p. 21
  • RAGTruth uses validation-set hyperparameter selection, while Dolly (AC) uses twofold validation; baseline tuning follows ranges from the original works.Source: I IMPLEMENTATION DETAILS · p. 21
  • ReDeEP configurations vary by model, dataset, and token-or-chunk variant; chunk processing uses LangChain RecursiveCharacterTextSplitter and BGE embeddings.AARF settings also vary by dataset and model.Source: I IMPLEMENTATION DETAILS · p. 21

J ABLATION STUDY

The ablation study reports that using both PKS and ECS outperforms either score alone, which the authors attribute to avoiding confounding through multivariate regression.

J ABLATION STUDY

  • Full ReDeEP outperforms using only PKS or only ECS for RAG hallucination detection.Table 3 is the ablation study; the supplied prose gives no exact values.Source: J ABLATION STUDY · p. 21, Table 3 · p. 22
  • Using PKS and ECS together in multivariate regression addresses the confounding issues attributed to either score alone.The authors link the lower Only PKS and Only ECS results to those confounding issues.Source: J ABLATION STUDY · p. 21

K EFFICIENCY ANALYSIS

The section compares detection efficiency across methods and describes the prompt used to evaluate truthfulness in AARF and baseline generations.

K EFFICIENCY ANALYSIS

  • ReDeEP (chunk) is more efficient than ReDeEP (token), and methods without external models are more efficient than methods using them.The authors place ReDeEP among high-efficiency methods and say its token-level runtime supports AARF’s efficiency.Source: K EFFICIENCY ANALYSIS · p. 22

L PROMPT FOR TRUTHFULNESS EVALUATION

  • The truthfulness-evaluation prompt compares AARF and baseline answers given a query and external context, returning a winning model and reason.The prompt permits a tie when no distinction can be made.Source: L PROMPT FOR TRUTHFULNESS EVALUATION · p. 23
Loading 2410.11414v2…