PaperlayerView exact revision on arXiv

Source-linked AI summary

LLM Lies: Hallucinations are not Bugs, but Features as Adversarial Examples

Jia-Yu Yao, Kun-Peng Ning, Zhen-Hui Liu, Mu-Nan Ning, Yu-Yang Liu, Li Yuan

arXiv:2310.01469v3cs.CLcs.AI

TL;DR

The paper addresses why LLMs hallucinate and whether hallucinations extend beyond training-data effects. It uses theoretical analysis and experiments to construct hallucination attacks, finding that both semantic-preserving and nonsensical prompts can trigger predefined false responses, and examines an uncertainty-based defense.

  • Problem

    LLM hallucinations fabricate non-existent facts or inappropriate information, while their existence and pervasiveness remain unclear beyond training-data explanations.

  • Method

    The paper analyzes transformer manipulability and uses gradient-based token replacement to construct weak-semantic and OoD hallucination attacks targeting predefined mismatched replies.

  • Results

    Both mainstream open-source models failed to resist hallucination attacks; Vicuna-7B reached a 92.31% success rate under the weak semantic attack.

  • Takeaways & Limitations

    Hallucinations may be another view of adversarial examples and a fundamental LLM characteristic, motivating uncertainty-constraint defenses without additional adversarial training.

  • Takeaways & Limitations

    The prompt candidate set grows exponentially with sequence length, making direct search for the best adversarial prompt computationally complex.

Abstract

from arXiv · show

Large Language Models (LLMs), including GPT-3.5, LLaMA, and PaLM, seem to be knowledgeable and able to adapt to many tasks. However, we still cannot completely trust their answers, since LLMs suffer from \textbf{hallucination}\textemdash fabricating non-existent facts, deceiving users with or without their awareness. However, the reasons for their existence and pervasiveness remain unclear. In this paper, we demonstrate that nonsensical prompts composed of random tokens can also elicit the LLMs to respond with hallucinations. Moreover, we provide both theoretical and experimental evidence that transformers can be manipulated to produce specific pre-define tokens by perturbing its input sequence. This phenomenon forces us to revisit that \emph{hallucination may be another view of adversarial examples}, and it shares similar characteristics with conventional adversarial examples as a basic property of LLMs. Therefore, we formalize an automatic hallucination triggering method as the \textit{hallucination attack} in an adversarial way. Finally, we explore the basic properties of attacked adversarial prompts and propose a simple yet effective defense strategy. Our code is released on GitHub\footnote{https://github.com/PKU-YuanGroup/Hallucination-Attack}.

1 Introduction

The paper argues that hallucinations can arise from adversarial prompting beyond training-data effects, including semantically consistent perturbations and nonsensical random-token prompts. It formalizes these triggers as hallucination attacks and reports theoretical, experimental, and defense analyses.

  • LLM hallucinations fabricate non-existent facts or inappropriate information, creating risks in applications such as law and medical consultation.
  • Nonsensical OoD prompts composed of random tokens can elicit responses that appear true but are false, despite lacking similarity to the training dataset.
  • The hallucination attack targets predefined mismatched replies by perturbing prompts through semantic-preserving token replacement or nonsensical OoD construction.
  • Examples show Vicuna-7B producing the same fabricated fact from both a semantically consistent perturbed prompt and a meaningless random-token prompt.
  • The paper characterizes hallucinations as sharing adversarial-example properties: perturbed inputs can preserve semantics or depart from training distributions while producing mismatched outputs.
  • The study combines theoretical and experimental evidence of adversarial-prompt vulnerability with a simple uncertainty-constraint defense strategy.

2 What leads to Hallucination?

The paper defines hallucinations as outputs that conflict with reality and investigates how transformer attention and input perturbations can produce them. Theoretical analysis and experiments show that perturbing token embeddings can induce arbitrary outputs, including hallucinatory responses.

  • Hallucination definition: Hallucinations are responses that do not satisfy the set of reality and facts.
  • Transformer mechanism: A simplified decoder-only transformer represents input tokens as embeddings and generates attention outputs through query, key, and value transformations.
  • Theoretical mechanism: Theorem 1 states that properly perturbing input embeddings can produce arbitrary rational or irrational responses from an attention block.
  • Theoretical mechanism: Embedding perturbations can construct predefined tokens and continuous attention maps that lead to hallucinations, indicating hallucination is a characteristic enabled by the attention mechanism.
  • Experimental verification: Figure 3 tracks attention-map fluctuations during optimization toward a target map associated with a hallucinatory response.
  • Experimental verification: 99.86% of 32k tokens were successfully output through perturbation in the Vicuna-7B experiment.

3 Adversarial Attack Induces Hallucination

The hallucination attack constructs adversarial prompts that elicit predefined fabricated responses through gradient-based token replacement. It uses weak semantic and OoD modes, with search approximations to avoid exponentially large prompt-candidate spaces.

  • Attack Pipeline: The attack pipeline combines hallucination-data generation, gradient-based token replacement, weak semantic attacks, and OoD attacks.It first constructs hallucination data, then triggers responses from semantically constrained or random-token prompts.
  • Hallucination Data Generation: Hallucination data are created by collecting common-sense questions, obtaining correct answers, and randomly replacing subjects, predicates, or objects to fabricate mismatched facts.The resulting dataset contains nonsensical question–answer pairs used as target hallucinations.
  • Gradient-Based Token Replacement: Gradient-based token replacement selects alternative tokens using the gradient direction that increases the target hallucination’s log-likelihood.For each position, the method computes a first-order likelihood change and selects top-k replacement candidates.
  • Search Strategy: The prompt candidate set has exponential size in prompt length, so direct adversarial-prompt search can be exponentially complex.The method randomly samples B candidates and selects the one maximizing log-likelihood for the next iteration.
  • Weak Semantic Attacks: Weak semantic attacks replace only a few tokens so prompts remain semantically consistent to humans while eliciting hallucinatory responses.Because no perfect human-comparable feature extractor is available, semantic preservation is approximated by constraining the number of replaced tokens.
  • OoD Attacks: OoD attacks begin with random-token sequences and seek predefined hallucinations without semantic constraints.The method therefore targets hallucinations from prompts that are nonsensical and outside the training-data distribution.

4 Experiment

Experiments show that weak semantic and nonsensical OoD prompts can trigger predefined hallucinations in open-source LLMs, while entropy-based refusal partially mitigates these attacks.

  • Hallucination attacks: 92.31% success rate was achieved by weak semantic attacks against Vicuna-7B.Both mainstream open-source models failed to resist the hallucination attacks, and OoD prompts also elicited predefined hallucinations with high probability.
  • Hallucination attacks: Weak semantic attacks replaced only several tokens while preserving prompts’ basic human-perceived semantics, yet elicited completely fake facts from Vicuna-7B.Similar phenomena occurred on LLaMA2-7B-chat.
  • Hallucination attacks: Meaningless OoD prompts far outside the training distribution still elicited reasonable-sounding but factually mismatched hallucinations from Vicuna-7B.The optimized prompt elicited the predefined Donald Trump hallucination.
  • Hallucination attacks: 34.6% increase in attack success rate occurred when LLaMA2-7B-chat OoD initialization length increased from 20 to 30 tokens, reaching 65.38%.The reported rates were 30.77% at length 20 and 65.38% at length 30.
  • Threshold defense: Entropy threshold 1.6 left all raw prompts answerable while refusing 46.1% of OoD prompts and 61.5% of weak semantic prompts.Higher thresholds made the defense ineffective against hallucination attacks.

5 Related Work

Related work attributes hallucinations mainly to training data, memorization, overfitting, and corpus-based learning processes, while adversarial research studies intentionally perturbed inputs that produce incorrect outputs.

  • Hallucination explanations: Prior accounts explain hallucinations through noisy data, overfitting, memorization, and corpus-based word-frequency heuristics.These views characterize hallucination as fabricating answers by finding similar corpus information in parameterized memorization.
  • Position of this work: This work differs by examining hallucinations beyond training-data explanations and by directly constructing adversarial prompts that elicit mismatched replies.The distinction motivates the hallucination-attack formulation.
  • Adversarial attacks: Adversarial examples use small, intentionally worst-case perturbations to make models output incorrect results and are described as a fundamental feature of deep neural networks.The paper applies this adversarial framing to prompts for LLMs.

6 Conclusion

The conclusion presents hallucination as another view of adversarial examples and proposes uncertainty constraints as a defense without additional adversarial training.

  • Conclusion: Theoretical and experimental evidence supports hallucinations as another view of adversarial examples and an inherent characteristic of LLMs beyond training data.The paper links hallucination generation to vulnerability to adversarial prompts.
  • Conclusion: Uncertainty constraints provide a simple defense for adversarially perturbed prompts without requiring additional adversarial training.The authors suggest this understanding should inform more comprehensive LLM evaluation.

A.1 Token Perturbation

Embedding perturbation experiments examine how input-token changes affect attention outputs during hallucination-response generation.

  • Token perturbation: Failed perturbation output tokens are rarely used in common queries.The failure analysis focuses on tokens produced during perturbation.
  • Token perturbation: The experiments provide additional details and results on embedding perturbation attacks for generating hallucinatory responses.They specifically study direct perturbations to the model’s input sequence.
  • Token perturbation: MSE optimization adjusts ∆ so Vicuna-7B attention blocks produce attention maps similar to ˆoHa, which generates a hallucinatory response.The optimization targets attention-map behavior rather than only the final token.

A.2.2 Attention Experiment Result

The attention experiment shows that perturbation optimization makes all attention maps increasingly similar to a constructed target. Additional tables document weak-semantic and OoD attack results across Vicuna and LLaMA2-7B-Chat.

  • Attention experiment: All 32 attention heads become increasingly similar to the target attention map during perturbation optimization.The similarity approaches an almost identical match as optimization proceeds.
  • Weak-semantic attacks: Tables 5 and 6 report weak-semantic hallucination attacks on Vicuna-7B and LLaMA2-7B-Chat.The tables mark differences between original and attacked prompts.
  • OoD attacks: Tables 7 and 8 report OoD attacks on LLaMA2-7B-Chat and Vicuna-7B, respectively.Their annotations identify successful defenses, failed attacks, or elicited pre-defined hallucinations.

C Experiments Settings

The experiments use a gradient-based hallucination attack initialized either from the original question or random tokens. The algorithm iteratively searches token replacements to maximize the likelihood of a target hallucination under attack-specific constraints.

  • Experimental settings: Experiments use 8 NVIDIA A100 GPUs, with 128 epochs for weak-semantic attacks and 1000 epochs for OoD attacks.The prompt is initialized from the original question for weak-semantic attacks and from random tokens for OoD attacks.
  • Weak-semantic settings: Weak-semantic attacks use top-k 256, batch size B=1024, and target Vicuna-7B and LLaMA2-7B-Chat.Their maximum repeat epochs are 128, with optimization stopping when hallucination is triggered.
  • OoD settings: OoD attacks use batch size B=1024, target Vicuna-7B and LLaMA2-7B-Chat, and prompt length l=20.Their maximum repeat epochs are 1000, with optimization stopping when hallucination is triggered.
  • Algorithm: The attack requires an LLM, epoch count T, batch size B, top-k parameter k, and semantic constraint parameter δ.The algorithm then samples a non-sense question-answer pair and initializes an adversarial prompt.
  • Algorithm: At each iteration, candidate token replacements are searched to maximize log p(ỹ|x̃), with a semantic-distance constraint applied only to weak-semantic attacks.OoD attacks optimize without that constraint and stop when the target response is triggered or the epoch budget is exhausted.

D.1 Limitation

The study is constrained by its access requirements, data scope, and evaluation procedure. It also identifies adversarial training as an unresolved defense direction with substantial resource demands.

  • Limitations: The hallucination attack requires access to the target LLM’s weights.This constrains applicability to models whose parameters are available.
  • Limitations: Experiments are conducted only on WiKi data because of limited data.The paper therefore leaves broader data coverage untested.
  • Limitations: Dataset construction and result evaluation rely on human feedback rather than an automatic model semantic evaluator.This limits the evaluation methodology used in the study.
  • Future work: Adversarial training is identified as future defense work, but it generally requires more resources for LLMs than for traditional models.Efficient approaches such as LoRA are proposed as a challenge for future fine-tuning-based defenses.
Loading 2310.01469v3…