PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-contradictory Hallucinations of Large Language Models: Evaluation, Detection and Mitigation

Niels Mündler, Jingxuan He, Slobodan Jenko, Martin Vechev

arXiv:2305.15852v3cs.CLcs.AIcs.LG

TL;DR

Language models often generate self-contradictory hallucinations, but existing factuality methods may require external retrieval or model access. This paper introduces a prompting-based pipeline to trigger, detect, and iteratively mitigate contradictions in black-box LMs. The framework finds contradictions frequently, detects them accurately, reduces them while preserving text quality, and applies to both open-domain generation and question answering.

  • Problem

    Language models generate hallucinated content, including self-contradictions, while some such contradictions cannot be verified using external text.

  • Method

    The paper uses prompting to generate constrained sentence pairs, detect contradictions, and iteratively revise conflicting information without requiring external knowledge.

  • Results

    The framework identifies self-contradictions across four instruction-tuned LMs and two tasks, achieving around 80% F1 detection and reducing up to 89.5% of self-contradictions while preserving fluency and informativeness.

  • Takeaways & Limitations

    The approach complements retrieval-based methods and is applicable to black-box LMs because many self-contradictions cannot be verified online.

  • Takeaways & Limitations

    The current framework handles contradictions between two sentences sampled at the same text position and does not address other hallucinations such as consistently incorrect content.

Abstract

from arXiv · show

Large language models (large LMs) are susceptible to producing text that contains hallucinated content. An important instance of this problem is self-contradiction, where the LM generates two contradictory sentences within the same context. In this work, we present a comprehensive investigation into self-contradiction for various instruction-tuned LMs, covering evaluation, detection, and mitigation. Our primary evaluation task is open-domain text generation, but we also demonstrate the applicability of our approach to shorter question answering. Our analysis reveals the prevalence of self-contradictions, e.g., in 17.7% of all sentences produced by ChatGPT. We then propose a novel prompting-based framework designed to effectively detect and mitigate self-contradictions. Our detector achieves high accuracy, e.g., around 80% F1 score when prompting ChatGPT. The mitigation algorithm iteratively refines the generated text to remove contradictory information while preserving text fluency and informativeness. Importantly, our entire framework is applicable to black-box LMs and does not require retrieval of external knowledge. Rather, our method complements retrieval-based methods, as a large portion of self-contradictions (e.g., 35.2% for ChatGPT) cannot be verified using online text. Our approach is practically effective and has been released as a push-button tool to benefit the public at https://chatprotect.ai/.

1 INTRODUCTION

The paper focuses on self-contradictory hallucinations, where language models generate logically inconsistent sentences in the same context. It proposes a prompting-based pipeline to trigger, detect, and mitigate these contradictions across generation and question answering.

  • Reasoning about Self-contradictory Hallucinations: Self-contradiction occurs when an LM generates two logically inconsistent sentences given the same context.The paper treats contradiction as evidence of non-factuality because both sentences cannot be simultaneously correct.
  • Reasoning about Self-contradictory Hallucinations: The proposed pipeline generates constrained sentence pairs, detects contradictions with prompting strategies, and iteratively edits conflicting information while preserving fluency and informativeness.Because it operates through prompting, the framework applies directly to black-box LMs.
  • Significance of Self-contradiction: 17.7% of all sentences generated by ChatGPT contain self-contradictions in open-domain text generation, and 35.2% cannot be verified using Wikipedia or web search.These findings motivate complementing retrieval-based approaches with self-contradiction analysis.
  • Effective Detection and Mitigation: ∼80% F1 score is achieved for contradiction detection, while mitigation reduces up to 89.5% of self-contradictions while maintaining informativeness and fluency.The introduction presents these as representative effectiveness results for the proposed framework.
  • Generality: 74.2% to 83.8% precision detects 12.7% to 38.0% of self-contradictions in both vanilla and retrieval-augmented question answering.This demonstrates applicability beyond open-domain text generation.
  • User-friendly Tool and Open-source Repository: The authors release a user-friendly tool that warns about hallucinations and automatically mitigates them, along with public code and datasets.The tool is accessible at chatprotect.ai.

2 RELATED WORK

Prior work studies language-model capabilities, hallucinations across generation tasks, and retrieval-based factuality methods. This paper emphasizes black-box applicability and combines hallucination detection with mitigation.

  • Large Language Models: Modern instruction-tuned LMs include proprietary black-box systems and emerging open-source models such as Vicuna and Llama2.Instruction-tuned LMs are increasingly used in daily life.
  • Large Language Models: Comprehensive benchmarks assess overall LM capabilities, whereas this work focuses specifically on addressing hallucinations.The paper positions its contribution as extending beyond general capability evaluation.
  • Hallucinations in Natural Language Generation: Hallucination research spans translation, summarization, dialogue, question answering, and open-domain text generation.Compared with related work, this framework is designed for black-box LMs and additionally addresses mitigation.
  • Hallucinations in Natural Language Generation: Retrieval-based approaches address factuality using external knowledge, while this paper targets hallucinations through prompting without requiring model internals or external retrieval.The distinction is central to the paper’s positioning relative to prior methods.

3 DEFINING AND MOTIVATING SELF-CONTRADICTIONS

The paper defines self-contradiction as logical inconsistency between two sentences generated by the same LM under the same context. It motivates this focus because contradiction guarantees non-factuality and can support mitigation without external knowledge.

  • Language Models: An LM generates text as a sequence of sentences conditioned on a user prompt specifying the task.The paper denotes multi-sentence generation as x ∼LM(· | p).
  • Self-contradiction of LMs: A self-contradiction is a pair of logically inconsistent sentences generated by the same LM from the same context.The shared context constrains both sentences to describe the same subject.
  • Self-contradiction vs. Non-factuality: 63.1% of ChatGPT self-contradictions contain two non-factual sentences, although at least one sentence in every contradiction is factually incorrect.Removing conflicting information therefore strictly decreases non-factuality.
  • Self-contradiction vs. Non-factuality: Sampling-based alternatives require tens of samples for reasonable detection accuracy and cannot handle mitigation.The paper attributes this contrast to their not exploiting the connection between self-contradiction and non-factuality.
  • Self-contradiction vs. Knowledge Retrieval: Self-contradiction detection and mitigation can use logical reasoning alone, while 35.2% of ChatGPT self-contradictions cannot be verified or refuted through online text.This motivates complementing retrieval-based hallucination methods.

4 TRIGGERING, DETECTING AND MITIGATING SELF-CONTRADICTIONS

The framework generates alternative sentences under shared contexts, detects contradictions with an analyzer LM, and revises only conflicting content. It repeats local mitigation to reduce remaining contradictions while preserving coherent, informative text.

  • Pipeline Components: The framework separates generation, detection, and revision between a generating LM and an analyzer LM.The analyzer performs reasoning tasks on outputs produced by the generating model.
  • Mitigation: The reviser removes conflicting information from contradictory sentence pairs while retaining non-conflicting content and coherence.Sentences without detected contradictions remain unchanged to preserve fluency and informativeness.
  • Trigger: The trigger algorithm extracts contexts for each sentence and generates an alternative sentence constrained to the same scope.The context must balance subject alignment with enough freedom to permit contradiction.
  • Detection: The detector predicts whether two generated sentences contradict each other within their context, distinguishing this task from general natural language inference.The same-LM origin and contextual interpretation are specific requirements.
  • Mitigation: The mitigation procedure repeats local revisions across sentence pairs and may remove remaining contradictory sentences after several iterations.The experiments use an example of three iterations, after which the process is expected to converge.

5 INSTANTIATION TO TEXT GENERATION TASKS

The framework is instantiated for open-domain text generation through prompts that trigger, detect, and revise self-contradictory sentence pairs. It generates initial text, extracts contexts, applies prompted detection and revision, and can be adapted to other free-form tasks.

  • Generating Initial Text: The pipeline generates initial encyclopedic text about a user-specified entity before processing its sentences for self-contradictions.The user prompt takes the form “Please tell me about t,” and the model generates a sequence of sentences reflecting its internal knowledge.
  • Defining Context: For each sentence, the context combines the entity, preceding sentences, and an information-extracted relation triple.CompactIE extracts the relation triple from the sentence without receiving external information.
  • Trigger: The trigger prompt constrains the entity, prefix, and relation subject and relation while omitting the object, prompting generation of a single aligned sentence.The omitted object leaves freedom for the model to fill the relation, and system instructions plus few-shot demonstrations enforce one-sentence output.
  • Detect: The detector receives the context and sentence pair, produces an explanation through chain-of-thought prompting, and ends with a parseable Yes-or-No contradiction judgment.The example correctly identifies inconsistent birth years, while the explanation improves interpretability.
  • Revise: The reviser is instructed to remove conflicting information while retaining an informative, coherent sentence, and the framework can generalize to question-answering prompts.In the example, revision eliminates the problematic birth date and returns a faithful, fluent sentence.

6 EXPERIMENTAL EVALUATION

The evaluation measures self-contradiction triggering, detection, and mitigation across four instruction-tuned LMs, primarily on open-domain text generation and additionally on question answering. Results show frequent contradictions, effective detection and mitigation, model-dependent performance, and complementarity with retrieval-based methods.

  • Experimental setup: The evaluation covers four instruction-tuned LMs across open-domain text generation and question answering.The open-domain MainTestSet contains 360 descriptions covering 30 entities, with a second 100-entity test set used for validation.
  • Evaluation metrics: The pipeline evaluates triggering frequency, classification precision, recall and F1, then contradiction removal, informativeness, and fluency.Fluency is assessed through the increase in perplexity between original and revised text.
  • Open-domain results: 15.7% to 22.9% of sentences contain triggered self-contradictions across gLMs, while 35.2% of ChatGPT contradictions cannot be verified online.Contradictions occur across popular and lesser-known entities, making retrieval-based resolution incomplete.
  • Open-domain results: ∼80% F1 is achieved for detection, while mitigation removes 76.3% to 89.5% of contradictions without harming fluency or informativeness.The results use ChatGPT as the analysis model across different generation models.
  • Open-domain results: Detection and mitigation performance varies across analysis models: GPT-4 and ChatGPT are strong, whereas open-source models show lower recall and weaker removal.Llama2-70B-Chat substantially outperforms the earlier Vicuna-13B version in the reported comparison.
  • Efficiency: The method makes a constant number of LM queries per sentence, so total queries grow linearly while prompt length grows quadratically with text length.The authors consider the resulting costs acceptable when trustworthiness is highly demanded.
  • Question answering: 12.7% to 38.0% of retrieval-augmented or vanilla answers contain detected self-contradictions with 74.2% to 83.8% precision.Contradictions remain even after retrieval augmentation, supporting the method’s complementary role.

7 CONCLUSION AND DISCUSSION

The paper frames self-contradiction as a detectable form of hallucination and presents a prompting-based pipeline that exposes, detects, and mitigates it. The approach works with black-box LMs without external retrieval, but its current scope is limited to contradictions between sentences sampled at the same position.

  • Conclusion: Self-contradictions reveal non-factuality because two logically inconsistent sentences cannot both be correct.Removing conflicting information is presented as reducing non-factuality.
  • Conclusion: The prompting-based algorithm triggers, detects, and mitigates self-contradictions across four instruction-tuned LMs and two tasks.The framework is designed for black-box LMs and does not require retrieval of external knowledge.
  • Limitations and future work: The current framework handles contradictions between two sentences sampled at the same text position, leaving cross-position detection for future work.The paper also identifies consistently incorrect content as a different hallucination type.

A MORE DETAILS ON EXPERIMENTAL SETUP

The experiments use diverse Wikipedia-derived entities, collaborative human annotation, and multiple LM-serving APIs to evaluate self-contradictions and their verifiability.

  • Test-set construction: Test sets balance human and non-human entities by sampling Wikipedia biographies, general articles, and featured articles.Featured articles add more popular entities to the sampled sets.
  • Human annotation: Human annotation jointly labels self-contradiction, factuality, and verifiability of contradictory information.Two annotators contribute to each decision, followed by discussion and adjudication of remaining disagreements.
  • Human annotation: Verifiability annotation requires Internet searches covering entity, related-entity, and potentially non-English Wikipedia articles.
  • Human annotation: Factuality is assigned when annotators find supporting text on the Internet; otherwise, the information is labeled non-factual.
  • Compute: ChatGPT and GPT-4 use the OpenAI API, Vicuna-13B uses FastChat on NVIDIA A100 GPUs, and Llama2-70B-Chat uses Together AI.

B MORE EXPERIMENTS ON OPEN-DOMAIN TEXT GENERATION

Additional experiments examine self-contradiction across entities, compare detection approaches, test temperature robustness, and analyze triggering, mitigation, and LM-combination choices.

  • Distribution: Self-contradictions remain common across entities and popularity levels in ChatGPT generations.
  • Robustness: Detection performance remains robust across tested temperatures, with precision, recall, and F1 showing very small variance when analyzer temperature changes.
  • Trigger ablation: The proposed triggering prompt outperforms Continue, Rephrase, and Q&A baselines by producing more self-contradictions and higher detection precision.Continue can impose insufficient constraints, while Rephrase preserves semantic meaning and overly restricts the alternative sentence.
  • Mitigation: After three mitigation iterations, removing sentences for remaining predicted contradictions preserves fluency and informativeness.Naively dropping all predicted contradictions initially produces significantly lower informativeness and fluency.
  • LM combinations: Combining different generator and analyzer LMs achieves higher F1 than using the same LM, while GPT-4-generated pairs are harder to classify.The paper attributes the latter difficulty to more nuanced sentence-pair differences.
  • 2ndTestSet: 12.7% of sentences are predicted self-contradictory on 2ndTestSet, and mitigation removes them while retaining 100.9% of predicted non-contradictory sentence pairs.Perplexity increases by only 0.42.

C MORE REAL-WORLD EXAMPLES

Examples show that the framework removes self-contradictions through local revisions while preserving unaffected text and improving coherence, factuality, and certainty.

  • Sentence-level examples: Mitigation eliminates contradictory sentence pairs and produces coherent revisions, including cases involving geographical locations and wedding dates.Some examples expose internal uncertainty when one sentence is correct and the other is wrong.
  • Text-level examples: Text-level mitigation changes only sentences where self-contradictions are detected and leaves other sentences unchanged.The paper links local changes to maintaining overall fluency and informativeness.

A Self-contradiction Unverifiable using Online Text

The examples illustrate both a case where online text cannot verify the contradiction and a model-specific detection limitation.

  • Unverifiable contradiction: When neither Wikipedia nor web search contains relevant information, ChatGPT detects the contradiction and removes a sentence, improving overall factuality.
  • Model limitation: Vicuna-13B has much lower detection recall because it struggles to identify conflicting information and generates erroneous explanations that cause false negatives.

D COMPLETE PROMPTS

This section provides full prompts, ablation configurations, evaluation materials, and examples illustrating the method’s detection and mitigation behavior.

  • Prompts of Our Method: The full prompts for sentence generation, contradiction detection, and revision are provided in Figures 13–15, with demonstrations used to enforce one-sentence generation.The generation prompt uses three-shot demonstrations with Douglas Adams, Kayne West, and Angela Merkel.
  • Prompts of Our Method: The adapted detection prompt compares an original sentence with 20 alternatives, first checking for contradictions and then judging factual correctness.This adaptation supports comparison with SelfCheckGPT on non-factuality prediction.
  • Prompts of Ablation Baselines: Baseline prompts are supplied for Continue, Rephrase, Q&A, and Multi-path, with controls matching sentence-pair counts or demonstrations where applicable.Multi-path samples five explanations at temperature 1, whereas the proposed method samples one explanation at temperature 0.
  • Additional Evaluation Materials: The supplementary materials include ablations for generation, detection, mitigation, temperature choices, and the relationship between the generation and analysis models.They also provide entity lists, popularity and sentence-count information, real-world hallucination examples, and mitigation examples across entities.
  • Real-world Examples: A Llama2-70B-Chat contradiction illustrates a case that cannot be verified through Wikipedia or web search but is detected and removed by ChatGPT.The accompanying table presents this as improving the overall factuality of the text description.
Loading 2305.15852v3…