PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Alignment for Factuality: Mitigating Hallucinations in LLMs via Self-Evaluation

Xiaoying Zhang, Baolin Peng, Ye Tian, Jingyan Zhou, Lifeng Jin, Linfeng Song, Haitao Mi, Helen Meng

arXiv:2402.09267v2cs.CLcs.AI

TL;DR

LLMs can generate inaccurate statements despite holding relevant knowledge, and such hallucinations undermine trust and practical applicability. This paper uses self-evaluation and confidence tuning to generate training signals for factuality alignment, reporting improved factual accuracy across three tasks.

  • Problem

    LLMs may produce plausible but inaccurate statements even when they hold relevant knowledge, undermining trustworthiness and practical use, especially in high-stakes tasks.

    Source: §1 · p. 1

  • Method

    The framework prompts LLMs to evaluate their own responses using internal knowledge, tunes confidence estimation and calibration with SK-TUNING, and uses the resulting scores as DPO training signals.

    Source: §1 · p. 2

  • Results

    Self-Alignment for Factuality improves LLaMA-family factual accuracy across three knowledge-intensive tasks and outperforms representation-editing and consistency-based confidence methods.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The findings support self-evaluation-based alignment as an approach to improving factual accuracy without external knowledge or human intervention.

    Source: §6 · p. 9

    Experiments in this study focus on 7B-scale LLaMA-family models; effectiveness on larger or RLHF-fine-tuned models remains to be investigated.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Despite showing increasingly human-like abilities, large language models (LLMs) often struggle with factual inaccuracies, i.e. "hallucinations", even when they hold relevant knowledge. To address these hallucinations, current approaches typically necessitate high-quality human factuality annotations. In this work, we explore Self-Alignment for Factuality, where we leverage the self-evaluation capability of an LLM to provide training signals that steer the model towards factuality. Specifically, we incorporate Self-Eval, a self-evaluation component, to prompt an LLM to validate the factuality of its own generated responses solely based on its internal knowledge. Additionally, we design Self-Knowledge Tuning (SK-Tuning) to augment the LLM's self-evaluation ability by improving the model's confidence estimation and calibration. We then utilize these self-annotated responses to fine-tune the model via Direct Preference Optimization algorithm. We show that the proposed self-alignment approach substantially enhances factual accuracy over Llama family models across three key knowledge-intensive tasks on TruthfulQA and BioGEN.

1 Introduction

LLMs can produce plausible but inaccurate statements, including when they hold relevant knowledge, undermining trust and practical use in high-stakes settings. The paper proposes using self-evaluation and confidence tuning to align responses with factuality, reporting improvements across three tasks and two benchmarks.

1 Introduction

  • Hallucinations can undermine LLM trustworthiness and practical applicability, particularly in high-stakes tasks.Source: §1 · p. 1
  • The paper targets the “knowing”–“telling” gap: models may hold relevant knowledge yet fail to convey accurate information consistently.The introduction describes cases where an LLM gives an inaccurate answer at one inference time and a correct one at another.Source: §1 · p. 1
  • The authors motivate self-evaluation as an alternative to consistency-based confidence, which may not reflect a model’s internal knowledge.They argue that models show potential to evaluate generated responses even when directly producing the correct response is challenging.Source: §1 · p. 2
  • The framework prompts a model to assess its own responses’ factuality using internal knowledge, then uses confidence scores as DPO training rewards.SELF-EVAL elicits factuality confidence, while SK-TUNING improves confidence estimation and calibration to strengthen self-evaluation.Source: §1 · p. 2
  • Self-Alignment for Factuality improves LLaMA-family factual accuracy across MCQA, short-form, and long-form generation on TruthfulQA and BioGEN, surpassing representation-editing and consistency-based confidence methods.The approach relies solely on the model’s internal knowledge.Source: §1 · p. 2

2 Related work

Related work addresses hallucinations through correction, inference-time intervention, and alignment training, with limitations including dependence on model capabilities, domain-specific data, or extensive human labels. The paper distinguishes its aim of expressing known truths from honesty-based approaches and consistency-based confidence estimation.

2 Related work

  • The paper targets reliable communication of accurate information when the model has sufficient knowledge, rather than broader factuality enhancement.It frames this target as “unfaithful hallucination,” where factually incorrect statements occur despite relevant knowledge.Source: §2 · p. 2
  • Hallucination mitigation includes post-hoc correction, inference-time intervention, and alignment training, with trade-offs in capability dependence, data generalizability, and human-label costs.Post-hoc methods analyze consistency among generated responses; interventions manipulate internal representations; alignment training uses supervised fine-tuning or RLHF.Source: §2 · p. 3
  • Unlike Yang et al. (2023)’s honesty-based fine-tuning to acknowledge limitations, this work guides models to state truthful information when they hold pertinent knowledge.Source: §2 · p. 3
  • Unlike Tian et al. (2023a)’s consistency-based confidence estimation, SELF-EVAL-SKT uses heterogeneous data to enhance confidence estimation capabilities.Source: §2 · p. 3

3 Self-Alignment for Factuality

Self-Alignment for Factuality uses an LLM’s internal knowledge to evaluate its own responses, then uses those evaluations to build preferences for alignment. SELF-EVAL estimates factuality, SK-TUNING aims to improve its confidence calibration, and DPO aligns the model using the resulting preference data.

3.1 Overview

  • The framework proceeds from generating candidate responses, through factuality scoring, to creating preference data for alignment.The overview describes these as three steps for Self-Alignment for Factuality.Source: §3.1 · p. 4
  • SELF-EVAL scores candidate responses using the LLM’s intrinsic knowledge; for long-form responses, it evaluates atomic claims and averages their factuality scores.Each claim is converted into an atomic question and scored as p(True|q, c), producing an average factuality score for the response.Source: §3.1 · p. 4, §3.2 · p. 4

3.3 Alignment Tuning with DPO

  • DPO aligns the LLM using preference pairs formed by ranking candidate responses by factuality and selecting preferred and dis-preferred responses.The DPO policy is initialized from the base reference policy, with β controlling deviation from that reference.Source: §3.1 · p. 4, §3.3 · p. 5

3.2 Factuality Self-Evaluation

  • SELF-EVAL prompts the LLM to judge whether its response is valid using only its internal knowledge, with p(True|q, a) representing its factuality estimate.SELF-EVAL-P(TRUE) uses a True/False question-and-answer prompt, with few-shot prompts to encourage well-structured answers.Source: §3.2 · p. 4
  • SK-TUNING addresses overconfidence in SELF-EVAL-P(TRUE) by training the model to better assess response factuality and calibrate its confidence scores.For training data, candidate answers are checked against a golden answer and converted into True/False prediction pairs; the model is fine-tuned on these pairs.Source: §3.2 · pp. 4–5, Figure 3 · p. 5

4 Experiments

The experiments evaluate self-alignment on knowledge-intensive question answering and generation, with SELF-EVAL-SKT improving factuality across tasks and outperforming comparison methods. Additional analyses compare GPT-4 pairwise judgments and alternative confidence-estimation approaches.

4.1 Setup

  • The evaluation covers MCQA, short-form TruthfulQA generation, and long-form BioGEN generation, using Accuracy, Truthfulness, Informativeness, True∗Info, FActScore, and response-level fact counts.Open-ended self-alignment uses dataset prompts; BioGEN uses five-shot prompting and TruthfulQA tasks use six-shot prompting.Source: §4 · p. 5, §4.1 · p. 6, Table 1 · p. 7
  • LLAMA-7B and LLAMA2-7B are fine-tuned for five epochs on constructed preference data; reported results are means over three runs.Source: §4.1 · p. 6

4.2 Main Results

  • SELF-EVAL-SKT raises TruthfulQA MC Accuracy by roughly 13%, reaches True∗Info 45.75% (LLAMA-7B) and 53.42% (LLAMA2-7B), and improves BioGEN FActScore approximately 4%.These are reported results for MCQA, short-form generation, and long-form generation, respectively.Source: §4.2 · p. 6, Table 1 · p. 7
  • Without labeled domain-specific data, SELF-EVAL-SKT surpasses ITI and DOLA on TruthfulQA True∗Info while retaining strong True and Info scores.The authors say this indicates a balance between providing accurate information and acknowledging limitations.Source: §4.2 · p. 7

4.3 Pairwise Evaluation

  • GPT-4 pairwise evaluations show SELF-EVAL-SKT outperforming FACTTUNE-MC and SELF-EVAL-P(TRUE) across factuality, helpfulness, relevance, and naturalness.Figure 4 reports win rates for the two comparisons on BioGEN.Source: §4.3 · p. 7, Figure 4 · p. 7

4.4 Self-Alignment with Varying Factuality Estimation Methods

  • Semantic-equivalence and universal-self-consistency variants improve factuality over base models, but perform below SELF-EVAL-SKT.The variants use response clusters to identify preferred responses; the USC cluster is identified using GPT-3.5-turbo.Source: §4.4 · pp. 7–8, Table 2 · p. 8

5 In-dpeth Analysis of SELF-EVAL

The analysis tests how well SELF-EVAL estimates answer correctness across multiple-choice datasets and whether SK-TUNING improves those estimates. SK-TUNING performs better than P(TRUE) on the reported estimation metrics and yields better calibration on CommonSenseQA.

5 In-dpeth Analysis of SELF-EVAL

  • For MCQA, SELF-EVAL predicts whether each answer option is correct by reformulating questions as True/False queries.Source: §5 · p. 8

5.1 Setup

  • The analysis uses TruthfulQA, CommonSenseQA, OpenBookQA, MedQA, and MMLU, evaluating answer selection with Accuracy and correct-versus-incorrect discrimination with AUROC.Source: §5.1 · p. 8

5.2 Results

  • Across five MCQA tasks, SELF-EVAL-SKT outperforms SELF-EVAL-P(TRUE) on Accuracy and AUROC; on CommonSenseQA, it is better calibrated than the substantially overconfident P(TRUE).Accuracy measures selection of the correct answer; AUROC measures discrimination between a correct answer and a randomly sampled incorrect answer.Source: §5.2 · pp. 8–9, Figure 5 · p. 8

6 Conclusion

The paper presents Self-Alignment for Factuality as a way to use an LLM’s self-evaluation to improve factuality without external knowledge or human intervention. Experiments show improved factual accuracy across three tasks, with potential applications in legal, medical, and educational domains.

6 Conclusion

  • Self-Alignment for Factuality uses an LLM’s self-evaluation to generate factuality confidence scores that serve as training signals, with SK-TUNING intended to improve confidence estimation and calibration.The authors describe the findings as a promising starting point for investigating factuality self-alignment.Source: §6 · p. 9
  • The approach improves factual accuracy for LLaMA-family models across three knowledge-intensive tasks, and the authors report SK-TUNING also improves factuality estimation across five MCQA tasks.The paper presents wider applications, including legal, medical, and educational fields, as a potential direction.Source: §6 · p. 9

Limitations

The authors characterize their results as preliminary and identify limits in the tested models and confidence-estimation methods. They also suggest combining the framework with decoding strategies as a possible extension.

Limitations

  • The authors regard the findings as preliminary, with further avenues of research remaining.They suggest combining the framework with high-performing methods such as DOLA as a potential route to more accurate and factual improvements.Source: Limitations · p. 9
  • Experiments cover 7B-scale LLaMA-family models; testing 13B and 70B variants and RLHF-fine-tuned models such as LLaMA2-CHAT remains future work.The authors anticipate possible gains for larger models based on prior findings about self-evaluation ability.Source: Limitations · p. 9
  • Different factuality-estimation approaches substantially affect framework performance, and the authors identify more efficient confidence-estimation and calibration methods as future work.They specifically note SK-TUNING’s role in refining confidence estimation and calibration for factuality improvement.Source: Limitations · pp. 9–10

Ethics Statement

The paper describes privacy and fairness considerations for its datasets and prompts, summarizes related mitigation approaches, and specifies how it constructed BioGEN examples.

Ethics Statement

  • The study uses established knowledge-intensive benchmark datasets without personally identifiable information and designs GPT-4 prompts to exclude discriminatory language.The authors state that the research follows ethical guidelines and principles.Source: Ethics Statement · p. 10

A A brief summary of recent hallucination mitigation approaches.

  • Table 4 summarizes related hallucination-mitigation approaches, separating prompting-engineering methods from model-development methods.The caption defines MCQA as multiple-choice question answering and Gen. as open-end text generation.Source: A A brief summary of recent… · p. 14

B Data statistics and task descriptions for main experiments.

  • BioGEN examples use “Write a biography of <Entity>” prompts for entities sampled from Min et al. (2023b), with training and validation responses generated by GPT-4.The paper directs readers to Table 5 for task descriptions and dataset details.Source: B Data statistics and task… · p. 14

C Implementation details.

The implementation uses factuality confidence scores to rank candidate answers or responses and create DPO preference data. Separate procedures describe SK-TUNING confidence evaluation and GPT-4 pairwise comparisons.

C Implementation details.

  • For generation tasks, the implementation samples candidate responses, scores factuality with SELF-EVAL-SKT, and uses ranked responses to form DPO preferences.For long-form generation, GPT-3.5 extracts atomic claims and converts them into questions; claim-level confidence scores are averaged into a response score.Source: C Implementation details · pp. 14–15
  • MCQA preference data ranks answer options by SELF-EVAL-SKT factuality scores, marking the highest-scoring answer preferred; evaluation selects the option with the highest p(True).The task-description caption says MC accuracy compares conditional probabilities of candidate answers irrespective of the other options.Source: C Implementation details · p. 14, Table 5 · p. 15

D Evaluation of the confidence estimation ability of LLMs.

  • Confidence-estimation evaluation reports Accuracy and AUROC, with Accuracy based on whether the correct answer receives the highest ranking among options.For SELF-EVAL-P(TRUE) and SELF-EVAL-SKT, the task is reformulated as true/false evaluation.Source: D Evaluation of the confidence… · p. 16

E Pairwise comparisons.

  • Pairwise automatic evaluations use GPT-4 to compare outputs on factuality, helpfulness, relevance, and naturalness.The evaluation prompt is provided in Table 7.Source: E Pairwise comparisons · p. 16

F Qualitative Analysis.

A qualitative comparison of two biographies found significantly fewer factual errors in outputs from self-alignment with SELF-EVAL-SKT than in LLAMA2-7B outputs. The authors say this aligns with automatic evaluation and indicates improved factuality and reliability in generated biographies.

F Qualitative Analysis.

G Analysis on the Factual Errors after Self-Alignment

The analysis examines 100 incorrect TruthfulQA answers from LLAMA2-7B with SELF-EVAL-SKT and identifies five challenging error types. The authors theorize that pre-training data quality may contribute to complex-query mistakes and suggest human-annotated data as a possible response.

G Analysis on the Factual Errors after Self-Alignment

  • The authors inspected 100 incorrect TruthfulQA answers from LLAMA2-7B with SELF-EVAL-SKT and summarized five challenging error types in Table 9.Source: G Analysis on the Factual Errors after… · p. 16
  • Figure 6 reports GPT-4 pairwise BioGEN comparisons on factuality, helpfulness, relevance, and naturalness.Source: Figure 6 · p. 16
  • Two error types are insufficient precise knowledge that leaves the model vulnerable to confusing descriptions, and definite answers to questions without definitive answers or clear conditions.Examples include answering “Los Angeles” instead of “Banos” and naming UC Berkeley as the biggest university without specifying a country.Source: G Analysis on the Factual Errors after… · pp. 16–17
  • Other errors include accepting misleading question premises and repeating superstitions, such as claiming that a black cat brings bad luck.Source: G Analysis on the Factual Errors after… · p. 17
  • The model also answers debatable questions, such as naming the world’s most interesting man.Source: G Analysis on the Factual Errors after… · pp. 17–18
  • The authors theorize that pre-training data quality, including possible superstitions or inadequate coverage of complex queries, may underlie these mistakes.They suggest future work could use high-quality human annotations and train models to express uncertainty or clarify false premises.Source: G Analysis on the Factual Errors after… · p. 18

I Error analysis on Claim Extraction and Question Generation

The authors audited GPT-3.5-turbo’s claim extraction and question generation on biographies produced by LLAMA2-7B. Their manual checks found high qualification rates for claims and questions, with some potentially ambiguous claims.

I Error analysis on Claim Extraction and Question Generation

  • The audit sampled 10 of 50 LLAMA2-7B biographies and yielded 208 claims with 208 corresponding questions.GPT-3.5-turbo was used for claim extraction and question generation.Source: I Error analysis on Claim Extraction… · p. 18
  • Claims scored 100% on coverage, consistency, completeness, and naturalness, and 96.15% on absence of ambiguity.Eight of 208 claims might be ambiguous, but each was followed by a clear claim; the authors considered their effect on evaluation minimal.Source: I Error analysis on Claim Extraction… · p. 18
  • The authors suggest that more detailed prompt instructions and examples could avoid potentially ambiguous claims.Source: I Error analysis on Claim Extraction… · pp. 18–19
  • Generated questions scored 100% on targeting factual knowledge, completeness, naturalness, and absence of ambiguity.The authors report that all reviewed questions were high quality, including questions for claims that might be ambiguous.Source: I Error analysis on Claim Extraction… · p. 19
Loading 2402.09267v2…