The Troubling Emergence of Hallucination in Large Language Models -- An Extensive Definition, Quantification, and Prescriptive Remediations

Vipula Rawte; Swagata Chakraborty; Agnibh Pathak; Anubhav Sarkar; S. M Towhidul Islam Tonmoy; Aman Chadha; Amit P. Sheth; Amitava Das

Source-linked AI summary

The Troubling Emergence of Hallucination in Large Language Models -- An Extensive Definition, Quantification, and Prescriptive Remediations

Vipula Rawte, Swagata Chakraborty, Agnibh Pathak, Anubhav Sarkar, S. M Towhidul Islam Tonmoy, Aman Chadha, Amit P. Sheth, Amitava Das

arXiv:2310.04988v2Oct 8, 2023cs.AI

TL;DR

The paper addresses the lack of uniform and programmatic methods for evaluating and distinguishing LLM hallucinations. It proposes a fine-grained taxonomy, the Hallucination Vulnerability Index, and mitigation strategies, while showing that hallucination patterns vary across model sizes and categories.

Problem
There is no uniform evaluation metric or programmatic method for measuring LLM hallucinations and distinguishing hallucinated from factually correct output.
Method
The paper categorizes hallucinations by orientation, degree, and type, defines HVI for comparing vulnerability, and proposes detection-and-replacement mitigation using open-source LLMs.
Results
Hallucination patterns vary by category and model size: complex categories become more prevalent, while Virtual Voice increases from GPT-3.5 to GPT-4.
Takeaways & Limitations
HVI is intended to support comparison of LLM hallucination vulnerability and aid the wider scientific community and policy-makers.
Takeaways & Limitations
The annotation considered only one hallucination category per sentence, despite acknowledged multi-class and multi-label instances.

Abstract

from arXiv · show

The recent advancements in Large Language Models (LLMs) have garnered widespread acclaim for their remarkable emerging capabilities. However, the issue of hallucination has parallelly emerged as a by-product, posing significant concerns. While some recent endeavors have been made to identify and mitigate different types of hallucination, there has been a limited emphasis on the nuanced categorization of hallucination and associated mitigation methods. To address this gap, we offer a fine-grained discourse on profiling hallucination based on its degree, orientation, and category, along with offering strategies for alleviation. As such, we define two overarching orientations of hallucination: (i) factual mirage (FM) and (ii) silver lining (SL). To provide a more comprehensive understanding, both orientations are further sub-categorized into intrinsic and extrinsic, with three degrees of severity - (i) mild, (ii) moderate, and (iii) alarming. We also meticulously categorize hallucination into six types: (i) acronym ambiguity, (ii) numeric nuisance, (iii) generated golem, (iv) virtual voice, (v) geographic erratum, and (vi) time wrap. Furthermore, we curate HallucInation eLiciTation (HILT), a publicly available dataset comprising of 75,000 samples generated using 15 contemporary LLMs along with human annotations for the aforementioned categories. Finally, to establish a method for quantifying and to offer a comparative spectrum that allows us to evaluate and rank LLMs based on their vulnerability to producing hallucinations, we propose Hallucination Vulnerability Index (HVI). We firmly believe that HVI holds significant value as a tool for the wider NLP community, with the potential to serve as a rubric in AI-related policy-making. In conclusion, we propose two solution strategies for mitigating hallucinations.

1 Hallucination: The What and Why

The paper frames LLM hallucination as a significant risk accompanying generative AI capabilities and argues that existing characterization lacks uniformity and sufficient granularity. It introduces a structured taxonomy, a dataset, HVI, and mitigation strategies to study and alleviate hallucination.

Motivation: LLM hallucinations create risks of false information and misuse alongside the benefits of generative AI.The paper connects these concerns to calls for AI-training moratoria, copyright questions, and regulatory proposals.
Taxonomy: The paper distinguishes factual mirage and silver lining orientations, each divided into intrinsic and extrinsic forms with mild, moderate, and alarming severity.These distinctions profile hallucination by prompt factuality, orientation, and degree.
Taxonomy: It categorizes hallucination into six types: acronym ambiguity, numeric nuisance, generated golem, virtual voice, geographic erratum, and time wrap.The categories provide a finer-grained vocabulary for analyzing hallucination behavior.
Resources: HILT is a publicly available dataset containing 75,000 text snippets generated by 15 contemporary LLMs with human annotations.The annotations cover the proposed hallucination categories.
Measurement: HVI provides a quantitative spectrum for evaluating and ranking LLMs according to their vulnerability to hallucination.The paper positions HVI as a resource for hallucination research, mitigation assessment, and potentially AI policy-making.
Mitigation: The proposed mitigation approach identifies high-entropy words in outputs from high-HVI LLMs and replaces them using a lower-HVI LLM.The paper presents this as one of two strategies for alleviating hallucination.

2 A Holistic View of the Hallucination Spectrum: its Types and Scales

The paper develops a holistic framework for hallucination spanning orientations, severity levels, and six categories, addressing fragmented prior definitions and task-specific studies. It also introduces HILT, HVI, and mitigation strategies for studying and alleviating hallucination.

Scope and motivation: Prior work studied hallucination notions loosely and in isolation, whereas this paper examines it across NLG tasks using fundamental text-generation principles.The authors state that hallucination can occur in any NLG task and that their findings can extend across NLP tasks.
Orientations: Factual Mirage concerns factually correct prompts, while Silver Lining concerns factually incorrect prompts; each has intrinsic and extrinsic subcategories.FM includes distortion or hallucination of correct prompts, whereas SL involves elaborate narratives based on incorrect prompts.
Categories of hallucination: The six hallucination types are Acronym Ambiguity, Numeric Nuisance, Generated Golem, Virtual Voice, Geographic Erratum, and Time Wrap.These categories cover imprecise acronym expansions, inconsistent numerical values, fabricated personalities, unverifiable quotations, incorrect locations, and fused timelines.
Degrees of hallucination: Hallucination severity is annotated as mild, moderate, or alarming, labeled 0, 1, and 2 respectively.Mild hallucinations are superficial; moderate hallucinations introduce fictitious or tangential facts; alarming hallucinations radically diverge from the prompt.
Dataset construction: HILT uses NYTimes tweets and Politifact headlines as prompts for 15 LLMs, generating 75,000 text passages categorized into factual mirage and silver lining.Each model produces 5,000 entries, divided into 2,500 FM and 2,500 SL passages; the table caption reports 129K annotated sentences overall.

4 Hallucination Vulnerability Index (HVI)

The paper introduces HVI as a comparative spectrum for evaluating and ranking LLMs by hallucination vulnerability. It analyzes how vulnerability varies across hallucination categories, model size, and RLHF usage.

HVI quantifies and ranks LLMs according to their vulnerability to producing hallucinations.The measure accounts for hallucinated-sentence ratios, factual-mirage and silver-lining tendencies, and is scaled from 0–100 for interpretability.
Larger LLMs without RLHF are prone to both orientations of hallucination.
Numeric Nuisance and Acronym Ambiguity are mild categories whose silver-lining orientation decreases as LLM size grows.
Time Wrap and Geographic Erratum become more prevalent with larger models, while Virtual Voice increases markedly from GPT-3.5 to GPT-4.
Generated Golem, Virtual Voice, and Geographic Erratum are rarely observed in smaller LLMs such as T5 and Dolly.
The relationship between LLM size and hallucination is not necessarily direct because training-data quality, factual training, overconfidence, and other factors also contribute.

6 Hallucination Mitigation Strategies

The paper explores automatic and human-in-the-loop strategies for mitigating hallucination after generation. Its black-box entropy replacement approach targets simpler categories, while its gray-box fact-checking approach addresses more complex cases.

The paper proposes automatic high-entropy replacement and human-in-the-loop sentence-level fact-checking as two mitigation directions.The automatic method replaces high-entropy words from high-HVI outputs using a lower-HVI LLM; the human-in-the-loop method flags susceptible sentences for review.
Automatic mitigation: ENTROPYBB identifies high-entropy words and replaces them with predictions from a lower-HVI LLM.Consecutive high-entropy words are masked as one unit before replacement, which is especially effective for Generated Golem and Acronym Ambiguity.
Automatic mitigation: For GPT-3 content, albert-large-v2 performs best at detecting high-entropy words, while distilroberta-base performs best at replacing them.
Human-in-the-loop mitigation: FACTUALITYGB retrieves 20 search results, selects relevant sentences, and uses textual entailment to classify generated sentences as support, refute, or not enough information.
Comparative performance: ENTROPYBB primarily addresses Acronym Ambiguity and Numeric Nuisance, whereas FACTUALITYGB is more applicable to complex hallucinations.The reported comparison motivates combining black-box and gray-box approaches.

7 Conclusion and Future Avenues

The paper positions its benchmark and HVI as resources for continued scientific and policy work on LLM hallucination. It also presents the two mitigation techniques as baseline methods for future research.

The benchmark and HVI are intended to serve the wider scientific community and aid policy-makers.
The benchmark and HVI will remain publicly open for further collaborative updates.
The two proposed mitigation techniques can serve as baselines.

8 Discussion and Limitations

The discussion positions HVI and the benchmark as resources for evaluating hallucination risks and mitigation, while acknowledging limitations from evolving models, annotation simplification, model coverage, and technique-specific distinctions.

HVI is proposed as a framework for assessing hallucination risk, mitigation, and machine-generated content.
The paper presents HILT and HVI as publicly accessible resources intended to support future research, policy-making, and collaborative updates.
The annotation scheme assigns one hallucination category per sentence despite acknowledged multi-class and multi-label instances.
The defined hallucination categories may become incomplete as new categories and combinations emerge with advances in LLMs.
The study’s selection of 15 contemporary LLMs may not encompass all available models in the rapidly changing field.
FACTUALITYGB distinguishes different entities through entailment, whereas ENTROPYBB handles token-level distinctions such as 1789 versus 1889.

9 Ethical Considerations

The paper warns that hallucination findings could be misused to generate fake news that is indistinguishable from human-written content.

The authors discourage malicious use of their findings to create AI-generated fake news indistinguishable from human-written content.

Frequently Asked Questions (FAQs)

The FAQs clarify the paper’s scope, explain why hallucinations are studied across prompt factuality, and describe HVI, mitigation strategies, and annotation procedures.

Scope and motivation: Hallucinations can sometimes have beneficial effects in computer vision, such as in-painting occluded image content, but are usually undesirable in text.
Scope and motivation: The study focuses on text hallucination because downstream impacts are worsened by the lack of a programmatic way to distinguish hallucinated from factual output.
Scope and motivation: Hallucinations are examined for both factually correct and factually incorrect prompts, motivating the Factual Mirage and Silver Lining data.
HVI and mitigation: The mitigation approach detects and replaces high-entropy points because they are more uncertain and more likely locations for hallucination.
HVI and mitigation: HVI is intended as a uniform score specifically measuring LLM hallucination tendency, unlike general downstream-task metrics.
HVI and mitigation: The choice between black-box and gray-box mitigation methods is described as dependent on the LLM and task.
Annotation procedure: Annotation uses three layers—orientation, category, and degree—applied through a sentence-wise procedure.
Annotation procedure: The HILT dataset was constructed with a web interface displaying the prompt and generated text to annotators.

B.4 Selecting quality annotators on AMT

The annotation process addressed crowdsourcing noise through in-house calibration, annotator screening, and compensation adjustments, while using a structured three-level label scheme.

Annotator quality control: A 2,000-sample in-house annotation process supported guideline development and annotator selection before crowdsourcing.
Annotator quality control: Only AMT annotators exceeding 90% accuracy on the in-house dataset were selected for the pilot task.
Annotator quality control: Compensation was iteratively adjusted because both insufficient pay and excessive wages could undermine annotation quality.
Label scheme: The annotation scheme labels orientation, category, and degree in three successive levels, including miscellaneous category 7 when needed.
Outcome: Qualified annotators and suitable compensation rates improved the quality of crowdsourced annotations.

B.5 Inter-Annotator Agreement

Agreement between three annotators was assessed with Fleiss’s kappa and Krippendorff’s alpha across NYT and Politifact annotations. Agreement was substantial to almost perfect overall, with orientation and category reaching nearly or more than 80%, while degree varied slightly.

Agreement measures: Fleiss’s kappa and Krippendorff’s alpha measured agreement among three annotators across the NYT and Politifact datasets.Fleiss’s kappa handles multiple annotators, while Krippendorff’s alpha accommodates nominal and ordinal data.
Agreement results: Nearly or more than 80% agreement was obtained for orientation and category annotations in both datasets.The passage reports substantial to almost perfect agreement across all three annotation types.
Agreement results: Degree annotation showed slight variation because annotators differed in their subjective assessment of the percentage of hallucination in a sentence.The passage attributes this variation to differing interpretations of percentage.
Datasets: HILT used NYT for factually correct prompts and Politifact for factually incorrect prompts.These two datasets supported curation of the hallucination dataset.

F Mitigation Techniques: The specifics

The paper illustrates its mitigation techniques through pre- and post-mitigation examples and tables of overall hallucination drops across model combinations. The reported maxima vary by generating model, with the largest listed drop reaching 10.66.

Illustrative examples: Tables 7 and 8 provide pre- and post-mitigation examples identifying hallucinated, high-entropy, replaced, uninformative, and refuted text fragments.The labels distinguish hallucinated fragments, high-entropy fragments, replacements, highlighted text where no information was found, and textual-entailment refutations.
Illustrative examples: Tables 9 to 23 illustrate overall hallucination drops across 16 combinations of four LLMs for multiple generating models.Rows represent models detecting high-entropy words, and columns represent models replacing those words.
Overall hallucination drops: Table 9 reports a 10.66 maximum overall hallucination drop for GPT-3, using albert-large-v2 for detection and distilroberta-base for replacement.The tables vary the detector and replacer across 16 combinations of four LLMs.
Overall hallucination drops: Maximum drops are also reported for Dolly, OPT, Alpaca, T0, XLNet, and T5 across detector–replacer combinations.The listed maxima are 8.20, 8.58, 9.80, 8.40, 9.00, and 7.90, respectively.

F.2 Evaluation strategy - how to determine no hallucination after mitigation?

The evaluation sampled 2,000 balanced instances to assess hallucination after ENTROPYBB and FACTUALITYGB. The findings indicate that ENTROPYBB mainly addresses simpler categories, whereas FACTUALITYGB applies more broadly to complex cases.

Sampling and evaluation: 2,000 data points were manually evaluated, comprising 500 instances each of IFM, EFM, ISL, and ESL.The sample was designed to maintain a balanced distribution across the six hallucination categories.
Sampling and evaluation: FACTUALITYGB treated hallucination as absent when highlighted sentences were rewritten by humans.This evaluation rested on the assumption that human rewriting removes hallucination.
Comparative findings: ENTROPYBB primarily tackles less complex categories such as acronym ambiguity and numeric issues.The result is reported from the relative analysis in Fig. 5 and Table 24.
Comparative findings: FACTUALITYGB is more applicable to complex hallucination cases than ENTROPYBB.The passage presents the two techniques as complementary in their category coverage.
Comparative findings: Table 24 compares Silver Lining and Factual Mirage HVI scores before and after ENTROPYBB and FACTUALITYGB across all six categories.The table reports the comparative HVI analysis for both mitigation techniques.