PaperlayerView exact revision on arXiv

Source-linked AI summary

Lookback Lens: Detecting and Mitigating Contextual Hallucinations in Large Language Models Using Only Attention Maps

Yung-Sung Chuang, Linlu Qiu, Cheng-Yu Hsieh, Ranjay Krishna, Yoon Kim, James Glass

arXiv:2407.07071v2cs.CLcs.AIcs.LG

TL;DR

Contextual hallucinations occur when LLMs produce inaccurate outputs despite receiving the relevant facts in their input context. This paper introduces Lookback Lens, an attention-based detector and guided-decoding method, and finds that it transfers across tasks and models while reducing hallucinations in XSum.

  • Problem

    LLMs can produce inaccurate summaries and document-based answers even when the correct facts are present in the input context.

  • Method

    Lookback Lens trains a linear classifier on per-head ratios of attention weights assigned to context tokens versus newly generated tokens, then integrates it into guided decoding.

  • Results

    The detector transfers across tasks and models, while guided decoding reduces hallucinations by 9.6% for LLaMA-2-7B-Chat on XSum and 3.2% when transferred to LLaMA2-13B-Chat.

  • Takeaways & Limitations

    Attention-map features can support contextual-hallucination detection and mitigation without retraining the detector for a larger model.

  • Takeaways & Limitations

    Guided decoding cannot correct errors when the LLM fails to sample the correct chunk among the eight candidates, and sampling multiple candidates increases total inference time.

Abstract

from arXiv · show

When asked to summarize articles or answer questions given a passage, large language models (LLMs) can hallucinate details and respond with unsubstantiated answers that are inaccurate with respect to the input context. This paper describes a simple approach for detecting such contextual hallucinations. We hypothesize that contextual hallucinations are related to the extent to which an LLM attends to information in the provided context versus its own generations. Based on this intuition, we propose a simple hallucination detection model whose input features are given by the ratio of attention weights on the context versus newly generated tokens (for each attention head). We find that a linear classifier based on these lookback ratio features is as effective as a richer detector that utilizes the entire hidden states of an LLM or a text-based entailment model. The lookback ratio-based detector -- Lookback Lens -- is found to transfer across tasks and even models, allowing a detector that is trained on a 7B model to be applied (without retraining) to a larger 13B model. We further apply this detector to mitigate contextual hallucinations, and find that a simple classifier-guided decoding approach is able to reduce the amount of hallucination, for example by 9.6% in the XSum summarization task.

1 Introduction

The paper targets contextual hallucinations, where LLMs generate inaccurate outputs despite receiving correct information, and introduces Lookback Lens to detect and mitigate them using attention-based features. The approach matches or can surpass more complex detectors and transfers across tasks and models.

  • Motivation: Contextual hallucinations occur when LLMs produce inaccurate outputs despite correct facts in the input context, affecting summarization and document-based question answering.The paper highlights risks even when correct documents are retrieved for retrieval-augmented generation.
  • Method: Lookback Lens computes each attention head’s ratio of attention weights on provided context versus newly generated tokens and feeds concatenated features to a linear classifier.The classifier predicts generation truthfulness from lookback ratios across layers and heads.
  • Results: The detector performs on par with or sometimes surpasses detectors using LLM hidden states or text-based entailment models trained on extensively annotated datasets.This comparison concerns feature-based hallucination detectors.
  • Results: 9.6% reduction in contextual hallucinations was achieved by Lookback Lens Guided Decoding for LLaMA-2-7B-Chat on XSum.The detector is integrated during decoding to guide generation.
  • Results: 3.2% reduction in XSum hallucinations was achieved when a detector trained on LLaMA-2-7B-Chat transferred to LLaMA2-13B-Chat without retraining.The paper attributes this transfer to using higher-level attention-map features.

2 Contextual Hallucinations Detection

Lookback Lens detects contextual hallucinations from attention-map ratios comparing attention to provided context with attention to newly generated tokens. It averages these features over spans and uses a linear classifier, outperforming entailment baselines and generalizing better than hidden-state classifiers across tasks.

  • Lookback ratio: The lookback ratio measures, for each attention head and time step, attention assigned to context tokens versus newly generated tokens.These ratios are concatenated across heads and layers into a feature vector.
  • Classifier: Lookback Lens averages lookback-ratio vectors over a text span and uses logistic regression to classify the span as factual or hallucinated.The classifier predicts factual spans with label 1 and hallucinated spans with label 0.
  • Span definition: The detector supports predefined spans and sliding-window spans, with sliding windows labeling a chunk hallucinated when any hallucinated content appears within it.Sliding-window labeling is intended for decoding without predefined span boundaries but is more challenging because chunks can mix factual and hallucinated content.
  • Evaluation: The evaluation compares Lookback Lens with text-based entailment and hidden-state classifiers across QA and summarization, using predefined spans and size-8 sliding windows.Table 2 reports AUROC for source-task validation and out-of-domain transfer.
  • Results: Lookback Lens slightly outperforms the hidden-state classifier and significantly outperforms NLI models, with a larger advantage in sliding-window settings.The hidden-state classifier tends to overfit source training sets and drops substantially on out-of-domain tasks, whereas Lookback Lens consistently performs better after transfer.
  • Guided decoding: Lookback Lens Guided Decoding samples multiple candidate chunks, scores their lookback-ratio features, and selects the candidate less likely to hallucinate.The method uses attention-map features during decoding to choose among candidate chunks.

3 Contextual Hallucinations Mitigation

Lookback Lens Guided Decoding uses a hallucination detector to select among multi-token candidate chunks, improving contextual faithfulness across summarization, question answering, and conversation settings. The method reduces hallucinations while preserving broader generation quality in the evaluated conversation setting.

  • Method: Lookback Lens Guided Decoding incorporates the Lookback Lens into generation to guide outputs toward greater contextual accuracy.Because the classifier evaluates multi-token chunks rather than individual next-token choices, decoding samples candidate chunks and selects the one predicted best by the detector.
  • Experimental Setup: The evaluation covers summarization on XSum, question answering on NQ, and multi-turn conversations on MT-Bench.The Lookback Lens is trained on CNN/DM summarization data; XSum is same-task transfer, while NQ and MT-Bench are cross-task transfer settings.
  • Main Results: 9.6% improvement on XSum and 3% on NQ were achieved with Lookback Lens Guided Decoding.On XSum, hallucinated examples fell from 510 to 414, an 18.8% reduction, matching the performance of the cited state-of-the-art NLI-guided approach despite much less training data.
  • Main Results: Lookback Lens Guided Decoding boosts MT-Bench performance on the hallucination-focused setting while maintaining the original setting’s performance.The original setting evaluates broader factors including helpfulness and relevance, whereas the hallucination setting focuses on faithfulness to chat history.

4 Cross-model Transfer

The Lookback Lens can transfer across LLaMA-2 model sizes without retraining by mapping attention-head features between models. Cross-model transfer remains effective, though simultaneous cross-model and cross-task transfer is more difficult.

  • Transfer Approach: A Lookback Lens trained on LLaMA-2-7B-Chat can be applied to LLaMA-2-13B-Chat without retraining.The transfer relies on higher-level lookback-ratio patterns and is motivated by potential correlation between the source and target models’ attention patterns.
  • Transfer Approach: 1024 heads in the 7B model and 1600 heads in the 13B model are mapped using linear regression before applying the 7B classifiers.The models are evaluated on the same generated content by decoding from 7B and running 13B on the 7B outputs.
  • Detection Results: Cross-model transfer produces non-trivially high AUROC scores, although it is slightly worse than same-model transfer.The comparison combines cross-model and cross-task transfer, which the authors describe as a difficult setting.
  • Mitigation Results: Cross-model same-task transfer results are closer to same-model transfer results than cross-model cross-task transfer results.For guided decoding, same-task transfers from NQ-train on 7B to NQ on 13B and from CNN/DM on 7B to XSum on 13B show similar performance improvements, while CNN/DM-to-NQ transfer shows no significant improvement.

5 Discussions and Ablations

The ablations examine chunk size, attention-head selection, layer selection, and qualitative decoding behavior. Improvements are generally robust across chunk sizes, while predictive power is distributed across many heads and layers.

  • Effect of Chunk Size: Lookback Lens guided decoding shows consistent improvements across chunk sizes, reducing the need to optimize chunk size.The study varies chunk sizes from 4, 8, to 16; shorter chunks are slightly preferred for NQ and longer chunks for XSum.
  • Predictive Power of Different Heads: Predictive power is not concentrated in a small subset of attention heads.The analysis ranks heads using the magnitude of coefficients in the all-head linear classifier.
  • Predictive Power of Different Heads: Top-10 attention heads perform worse than all heads, while increasing the number of selected heads improves performance and top-100 heads largely recover the full model’s performance.
  • Predictive Power of Different Heads: Higher lookback ratios in positively weighted heads indicate higher factuality, whereas negatively weighted heads associate lower lookback ratios with greater factuality.
  • Reducing Number of Layers: No tested layer subset recovers the full model’s performance, although middle layers 13-16 and 17-20 are slightly more useful than other layers.
  • Qualitative Study: Greedy decoding produces a nonexistent $100m (£64m) claim, while Lookback Lens guided decoding favors a summary that is factual to the input document.

6 Related Work

The paper distinguishes contextual hallucinations from hallucinations based on parametric knowledge and positions its approach as attention-map-based detection and mitigation. It also relates the method to classifier-guided generation and research on self-attention’s role in model behavior.

  • Hallucinations in LLMs: Contextual hallucinations are outputs inconsistent with provided context, whereas close-book hallucinations rely on parametric knowledge.
  • Hallucinations in LLMs: Unlike prior work focused mainly on internal representations, this work uses attention maps to capture how LLMs process contextual information.
  • Classifier Guided Generation: The method differs from prior classifier-guided generation by using classifiers over attention maps rather than output tokens.
  • Self-attention and Model Behavior: The paper builds on research using self-attention to understand model behavior.

7 Conclusion

The paper introduces Lookback Lens, a lightweight attention-weight classifier for detecting contextual hallucinations and guiding decoding. It reports detection, mitigation, and transfer across tasks and models.

  • Lookback Lens detects contextual hallucinations using a lookback ratio computed solely from attention weights.
  • Lookback Lens Guided Decoding uses the detector to mitigate contextual hallucinations during generation.
  • The method transfers across tasks and models after mapping attention heads, including application of a detector trained on one model to another.

Limitations

The paper identifies three limitations: decoding cannot correct errors absent from sampled candidates, candidate sampling increases inference time, and classifier training requires annotated examples.

  • Lookback Lens Guided Decoding cannot correct an error when the LLM fails to sample the correct chunk among eight candidates.
  • Multiple-candidate sampling increases total inference time despite the Lookback Lens classifier having negligible inference time.
  • The authors leave attention-map intervention for future work as a potentially faster alternative to guided decoding.
  • The Lookback Lens requires around 1k-2k annotated examples to train its classifier.

Ethics Statement

The study uses public datasets without collecting personal information, but deployment may still produce biased, harmful, or offensive LLM outputs.

  • The research uses publicly available datasets and collects no personal information.
  • Real-world deployment still carries risks of biased, harmful, or offensive output from LLMs.

A Data Creation for Lookback Lens

The authors create controlled summarization and question-answering data by generating responses with LLaMA-2-7B-Chat and labeling hallucinated spans. They avoid existing data because source differences could confound hallucination detection.

  • The dataset covers 1,000 CNN/DM summarization examples and 2,655 Natural Questions examples, with contexts limited to three documents per question.
  • LLaMA-2-7B-Chat greedily generates responses for both tasks so hallucinated and non-hallucinated examples share the same source distribution.
  • The authors extract hallucinated and non-hallucinated spans together with attention-map lookback ratios to train the classifiers.
  • Predefined-span labeling treats annotated hallucinated spans as negative data and several remaining response regions as non-hallucinated.
  • Sliding-window labeling uses fixed eight-token chunks, marking a chunk hallucinated when it overlaps any annotated hallucinated span.
  • The authors generate their own data because externally generated examples could let classifiers distinguish response sources rather than hallucinations.

B.2 Human Evaluation on GPT-4o Evaluation

The appendix documents human checks of GPT-4o evaluation, evaluation prompts, model configurations, inference procedures, and classifier-training details. GPT-4o evaluation achieved high reported accuracy on both summarization and question answering.

  • Human evaluation: The summarization evaluation used documents, ground-truth summaries, model summaries, and GPT-4o judgments in a human-checking interface.
  • Human evaluation: 97.1% final accuracy supported GPT-4o as an automatic evaluator for the summarization dataset.
  • Human evaluation: 94.0% majority-vote accuracy on 50 Natural Questions examples supported using GPT-4o to verify generated answers.
  • Evaluation setup: The appendix provides GPT-4o prompt templates for truthfulness and span-level hallucination annotation on CNN/DM, XSum, and Natural Questions.
  • Evaluation setup: The study evaluates MT-Bench variants with GPT-4 and GPT-4o using separately documented prompt templates.
  • Models and implementation: Experiments use LLaMA-2-7B-Chat, LLaMA-2-13B-Chat, and a DeBERTa-V3-based hallucination evaluation model under their stated licenses.
  • Models and implementation: Guided decoding runs on A6000 and V100 GPUs, with reported per-example generation times of 20-30 seconds for 7B and 40-60 seconds for 13B.
  • Models and implementation: All reported scores come from a single run because computation for the large models was limited.

C.2 Dataset Details

The appendix lists datasets used across experiments and reports analyses of Lookback Lens features, hidden-state alternatives, and attention-block outputs. These analyses find that compact lookback-ratio features can outperform substantially larger hidden-state representations, while classifier behavior depends on all heads collectively.

  • Dataset Details: The experiments use CNN/DM, Natural Questions, XSum, and MT-bench datasets with the listed sample sizes and licenses.CNN/DM and XSum each contribute 1000 test examples; Natural Questions uses 2655 test examples and 2499 training examples; MT-bench uses 80 examples.
  • Lookback-Ratio Analysis: Lookback-ratio visualizations compare the top-10 positive and negative classifier heads during a hallucinated NQ span.Positive heads, especially the top-ranked heads, show lower lookback ratios, whereas negative heads show slightly higher ratios; the classifier still uses all heads with their weights and bias.
  • Hidden-State Comparisons: Concatenating four hidden-state layers produces a 16384-dimensional feature, yet remains less effective for hallucination detection than Lookback Lens.The four layers are 32, 28, 24, and 20; the Lookback Lens feature for the 7B model has 1024 dimensions.
  • Hidden-State Comparisons: Pooling hidden states across all layers yields 4096-dimensional features, but these results are still worse than Lookback Lens.Concatenating all layers would exceed 100k dimensions and make classifier training extremely slow, so max or average pooling is used instead.
  • Feature Design: The analyses indicate that feature design matters more than simply adding layers, allowing a compact 1024-dimensional lookback-ratio feature to outperform 10-times-larger hidden-state features.The additional experiment with attention-block outputs finds no significant difference from the main output choice, while Lookback Lens still outperforms those baselines.
Loading 2407.07071v2…