PaperlayerView exact revision on arXiv

Source-linked AI summary

Factuality of Large Language Models: A Survey

Yuxia Wang, Minghan Wang, Muhammad Arslan Manzoor, Fei Liu, Georgi Georgiev, Rocktim Jyoti Das, Preslav Nakov

arXiv:2402.02420v3cs.CLcs.AI

TL;DR

Factual errors in LLM responses limit their real-world utility, while evaluating open-ended generations remains challenging. This survey reviews factuality research and its evaluation and improvement approaches, identifies major issues, and points to future research directions.

  • Problem

    Factual errors limit LLMs’ real-world utility, and automatically evaluating factual accuracy in open-ended generations remains challenging.

    Source: §1 · p. 1, §6 · p. 8

  • Method

    The survey synthesizes work on LLM factuality, covering evaluation and improvement methods across stages and identifying challenges and future directions.

    Source: §1 · p. 1, §7 · p. 9

  • Results

    The survey identifies three major issues in LLM factuality research and points to promising future research directions.

    Source: §7 · p. 9

  • Takeaways & Limitations

    The survey highlights calibration research that makes models aware of knowledge limitations and improves the reliability of output confidence.

    Source: §1 · p. 1

    The rapidly evolving field may have led to omitted studies, and vision-language factuality and other modalities receive limited discussion.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large language models (LLMs), especially when instruction-tuned for chat, have become part of our daily lives, freeing people from the process of searching, extracting, and integrating information from multiple sources by offering a straightforward answer to a variety of questions in a single place. Unfortunately, in many cases, LLM responses are factually incorrect, which limits their applicability in real-world scenarios. As a result, research on evaluating and improving the factuality of LLMs has attracted a lot of attention recently. In this survey, we critically analyze existing work with the aim to identify the major challenges and their associated causes, pointing out to potential solutions for improving the factuality of LLMs, and analyzing the obstacles to automated factuality evaluation for open-ended text generation. We further offer an outlook on where future research should go.

1 Introduction

Chat-oriented LLMs offer convenient answers but can produce factual errors that limit their real-world utility. This survey reviews factuality research, its challenges, and potential directions for evaluation and improvement.

1 Introduction

  • Chat-oriented LLMs can replace searching, extracting, and integrating information across sources, but fabricated statements and factual errors limit their utility.The paper frames reliable factuality as important to developing more useful LLMs.Source: §1 · p. 1
  • Existing surveys either lack in-depth discussion or focus too narrowly to address fundamental challenges, solutions, and ambiguous factuality concepts.The authors position their survey as a response to these gaps, emphasizing recent studies in a rapidly evolving field.Source: §1 · p. 1
  • The survey covers factuality categorizations, evaluation methods, and mitigation techniques for language and vision modalities.It also explores calibration, including models’ awareness of knowledge limitations and the reliability of their output confidence.Source: §1 · p. 1

2 Background

The background distinguishes factuality from hallucination and from broader notions of trustworthiness. These concepts overlap in evaluation contexts but refer to different properties of generated content.

2 Background

  • The survey aims to clarify the boundaries among hallucination, factuality, relevance, and trustworthiness to reduce conceptual confusion.Table 1 compares surveys on LLM factuality; its caption identifies evaluation and improvement as comparison dimensions.Source: §2 · p. 1, Table 1 · p. 2
  • In LLMs, hallucination can include divergence from user input, prior context, or established world knowledge; factuality hallucination concerns discrepancies with verifiable real-world facts.The survey distinguishes factual inconsistency and factual fabrication as forms of factuality hallucination.Source: §2 · p. 2
  • Factuality concerns a model’s ability to learn, acquire, and use factual knowledge, while trustworthiness extends beyond factual accuracy.Trustworthiness is described across truthfulness, safety, fairness, robustness, privacy, ethics, transparency, and accountability.Source: §2 · p. 2
  • Factuality concerns generating content consistent with factual information and world knowledge, whereas hallucination and factual errors are related but distinct challenges.A response can be factually correct yet hallucinate by going off-topic or adding details beyond what was requested.Source: §2 · pp. 1–2

3 Evaluating Factuality

Evaluating factuality in open-ended generations is difficult, and the section reviews datasets, metrics, and evaluation strategies across answer formats. It also highlights limits in automating evaluation, especially for long-form text.

3.1 Datasets and Metrics

  • The survey categorizes factuality datasets by answer space and the difficulty of automatic quantification, including four broad response types.Table 2 distinguishes open-ended generation, yes/no answers, short-form or entity-list answers, and multiple-choice QA; the survey also notes the lack of comparative analysis across factuality aspects.Source: Table 2 · p. 3, §3.1 · p. 3
  • Open-ended long-form factuality is difficult to assess automatically because the output space lacks definitive gold-standard responses.Studies constrain answers to yes/no, short phrases, or entity lists; long-form evaluation commonly uses human experts or retrieval-based fact-checkers. A reported verifier achieved F1 0.63 with Google evidence for false claims and F1 0.53 with PerplexityAI.Source: §3.1 · p. 3
  • Automatic fact-checkers decompose documents into atomic claims and verify them against retrieved evidence, but their verification is not guaranteed to be fully accurate.FactScore uses the percentage of claims verified true to represent a document’s factual status, with dataset-level averages used to assess models.Source: §3.1 · p. 3

3.2 Other Metrics

  • Other measures include hallucinated named-entity error, entailment ratio, and the hallucination vulnerability index.A hybrid benchmark uses human scoring when external retrieval cannot resolve a claim, but may be affected by human annotation bias.Source: §3.2 · p. 4

4 Improving Factuality

The survey organizes factuality improvements across the LLM lifecycle, from training through inference and post-processing, and treats retrieval augmentation as usable at multiple stages. It reviews methods alongside their practical constraints and the challenges of evaluating automatic fact-checkers.

4 Improving Factuality

  • Factuality interventions span pre-training, tuning, inference, and post-processing; retrieval augmentation can be integrated at multiple stages.The survey groups approaches by lifecycle stage and discusses retrieval augmentation across training and inference.Source: §4 · p. 4

4.1 Pre-training

  • High-quality pre-training corpora and retrieval-augmented pre-training can improve factuality, but corpus curation grows harder at scale and RETRO adds about 25% computation.RETRO is reported as more accurate and less prone to hallucination than GPT; its performance may suffer from inaccurate, biased, or outdated retrieval data.Source: §4.1 · p. 4

4.2 Tuning and RLXF

  • Domain-specific SFT and knowledge injection can improve factual accuracy, while refusal-aware or behavioral tuning can manage knowledge gaps but requires curation and may reduce parametric-knowledge retention.R-tuning targets questions beyond parametric knowledge; BeInfo emphasizes selecting correct information and communicating when information is unavailable.Source: §4.2 · p. 5
  • DPO with factuality feedback and hallucination-augmented recitations offer tuning approaches, but cross-domain generalization and broader application remain under-explored.DPO is evaluated in-domain on biographies and medical queries; HAR uses counterfactual data to encourage attribution to context.Source: §4.2 · p. 5

4.3.1 Decoding Strategy

  • Decoding strategies trade diversity, context use, or speed against factuality: factual-nucleus sampling reduces randomness over sentence generation, CAD suits high-quality contextual evidence, and DoLa adds 1.01x–1.08x decoding time.Top-p sampling is reported as less factual than greedy decoding despite greater diversity; CAD prioritizes current context over model priors.Source: §4.3.1 · pp. 5–6

4.3.2 ICL and Self-reasoning

  • In-context examples can teach fact editing, but generalizing new facts while preserving predictions on unrelated questions remains difficult; multi-agent debate is another factuality strategy.The survey reports that more agents and longer debates tend to produce better results, and that debate can complement other generation methods.Source: §4.3.2 · p. 6
  • Retrieval can condition generation on evidence, verify claims afterward, or correct errors during generation; pre-generation retrieval may constrain creativity, while post-generation checking risks hallucination snowballing.Real-time generate-verify-correct methods can prevent snowballing, but latency is a concern for high-throughput, responsive systems.Source: §4.3.2 · pp. 6–7

4.4 Automatic Fact Checkers

  • Fact-checkers combine claim processing, retrieval, and verification, but systems differ in whether they detect or correct errors and whether they operate on documents or atomic claims.Some approaches edit whole documents, while FacTool and Factcheck-GPT address atomic claims; evaluating fact-checker effectiveness remains challenging.Source: §4.4 · p. 7

5 Factuality of Multimodal LLMs

The survey groups multimodal factuality errors into object existence, attributes, and relationships, and reviews evaluation and mitigation approaches. Evaluation ranges from object-focused benchmarks to human review, while mitigation includes fine-tuning, inference-time correction, and representation learning.

5 Factuality of Multimodal LLMs

  • Multimodal factuality errors concern false claims about an image’s objects, their attributes, or their relationships.Relationship errors include false descriptions of relative positions and interactions.Source: §5 · p. 8
  • CHAIR evaluates object-existence accuracy in captions against a predefined set of COCO objects, while POPE uses positive and negative binary-choice prompts to query object presence (Rohrbach et al., 2018; Lin et al., 2014; Li et al., 2023).The survey notes that COCO’s frequent use in training can make CHAIR misleading as a sole evaluation basis.Source: §5 · p. 8
  • GAVIE uses GPT-4 assistance for visual-hallucination evaluation, and human evaluation has also been used to avoid inaccuracies and systematic biases (Li et al., 2023; Gunjal et al., 2023).Source: §5 · p. 8
  • Mitigation approaches include fine-tuning on specialized instructions or human preference pairs, inference-time correction, and representation learning (Liu et al., 2023; Sun et al., 2023; Yin et al., 2023a; Zhou et al., 2023; Leng et al., 2023; Jiang et al., 2023).The survey describes fine-tuning as producing models with fewer hallucinated responses and names specialized correction models as inference-time approaches.Source: §5 · p. 8

6 Challenges and Future Directions

The survey identifies challenges in factuality evaluation and improvement, including reliable fact-checking, open-ended evaluation, and retrieval. It points to future work on decoding anchors, iterative correction, and better retrieval algorithms.

6 Challenges and Future Directions

  • Because language-model training maximizes sentence probability rather than factual-statement probability, fluent, coherent models are not guaranteed to answer factually.Source: §6 · p. 8
  • Open-ended factuality evaluation lacks a unified framework, making comparisons across enhancement studies difficult; automated fact-checker quality is itself hard to quantify.Current studies use human judgments or automated checkers, creating what the survey calls a chicken-and-egg situation.Source: §6 · p. 8
  • Retrieval latency and difficulty finding pertinent evidence can bottleneck RAG, and some facts require information from multiple sources and multi-hop reasoning.Source: §6 · p. 8
  • Models may answer correctly in some trials despite errors in others, motivating the question of whether an anchor could guide decoding toward a factually correct path (Tian et al., 2023).The survey also asks whether iterative detection and correction early in generation can improve subsequent sentences, and whether correct, relevant text can serve as an anchor.Source: §6 · p. 9
  • The survey calls for better retrieval techniques because unreliable online information, including fake news, can compromise retrieved knowledge and lead to responses based on incorrect input.Source: §6 · p. 9
  • Reliable, efficient fact-checkers remain a key challenge; GPT-4-based verification can be costly and sensitive to prompt changes and model updates, while task-specific fine-tuned NLI models may be more viable.Source: §6 · p. 9

7 Conclusion

The conclusion presents a survey of LLM factuality research, covering evaluation, improvement methods, and their challenges. It also identifies major issues and promising directions for future work.

7 Conclusion

  • The survey reviews factuality studies on evaluation and improvement methods, their challenges, and future research directions.Source: §7 · p. 9
  • Its coverage of improvement methods spans pre-training, supervised fine-tuning, inference, and post-processing.Source: §7 · p. 9
  • The authors identify three major issues and point to promising future research directions.Source: §7 · p. 9

Limitations

The survey may omit studies because the field is evolving rapidly, and its treatment of multimodal factuality is limited in scope.

Limitations

  • Despite an extensive literature review, some studies may have been omitted because factuality research is rapidly evolving.Source: Limitations · p. 9
  • Vision-language factuality is only briefly covered; mitigation techniques for it and factuality in video and speech models need more comprehensive discussion.Source: Limitations · p. 9

Related papers

Loading 2402.02420v3…