PaperlayerView exact revision on arXiv

Source-linked AI summary

Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificity

Cunxiang Wang, Xiaoze Liu, Yuanhao Yue, Xiangru Tang, Tianhang Zhang, Cheng Jiayang, Yunzhi Yao, Wenyang Gao, Xuming Hu, Zehan Qi, Yidong Wang, Linyi Yang, Jindong Wang, Xing Xie, Zheng Zhang, Yue Zhang

arXiv:2310.07521v3cs.CL

TL;DR

LLM factuality matters because inaccurate outputs can undermine reliability across applications, yet existing work lacks a holistic overview spanning the field. This survey synthesizes definitions, impacts, evaluation, mechanisms, and enhancement strategies across standalone and retrieval-augmented LLMs, including domain-specific approaches. It reports factual inaccuracies in evaluated models and concludes that factuality evaluation and model internals remain incompletely understood.

  • Problem

    Existing factuality studies address evaluation and improvement, but a holistic overview spanning the full spectrum of factuality in LLMs remains elusive, despite its importance for reliable applications.

  • Method

    The survey organizes research around factuality’s definition and impact, evaluation, underlying mechanisms and error causes, and enhancement approaches for standalone, retrieval-augmented, and domain-specific LLMs.

  • Results

    FActScores for biographies generated by evaluated commercial LLMs ranged from 42% to 71%, with factual precision decreasing as entity rarity increased.

  • Takeaways & Limitations

    The survey provides a structured guide for examining and improving factual reliability across standalone and retrieval-augmented LLM settings.

  • Takeaways & Limitations

    Factuality evaluation remains difficult, and how LLMs store, update, and produce facts is not yet fully revealed; continual training and retrieval also have limitations.

Abstract

from arXiv · show

This survey addresses the crucial issue of factuality in Large Language Models (LLMs). As LLMs find applications across diverse domains, the reliability and accuracy of their outputs become vital. We define the Factuality Issue as the probability of LLMs to produce content inconsistent with established facts. We first delve into the implications of these inaccuracies, highlighting the potential consequences and challenges posed by factual errors in LLM outputs. Subsequently, we analyze the mechanisms through which LLMs store and process facts, seeking the primary causes of factual errors. Our discussion then transitions to methodologies for evaluating LLM factuality, emphasizing key metrics, benchmarks, and studies. We further explore strategies for enhancing LLM factuality, including approaches tailored for specific domains. We focus two primary LLM configurations standalone LLMs and Retrieval-Augmented LLMs that utilizes external data, we detail their unique challenges and potential enhancements. Our survey offers a structured guide for researchers aiming to fortify the factual reliability of LLMs.

1 INTRODUCTION

This survey frames factuality in LLMs as an important reliability challenge and organizes prior work across definitions, evaluation, mechanisms, and enhancement. It covers both standalone and retrieval-augmented LLMs, with attention to domain-specific factuality.

  • Motivation: LLMs reduce knowledge-base maintenance costs while supporting flexible, context-aware reasoning, but may still generate non-factual content.The survey motivates factuality research by contrasting the practical advantages of LLMs as knowledge carriers with concerns about unreliable outputs.
  • Research landscape: Existing factuality research spans factoid question answering, fact checking, external retrieval, continual pretraining, and supervised fine-tuning.The survey identifies these efforts but notes that they do not yet provide a holistic overview of factuality in LLMs.
  • Research gap: Prior surveys leave important areas undercovered, including domain-specific factuality and outdated information.The authors position their survey as the first comprehensive study focused on factuality in large language models.
  • Scope and organization: The survey examines factuality definitions and impacts, evaluation methods and quantitative assessment, underlying mechanisms, and enhancement approaches.Its structure covers metrics, benchmarks, evaluation studies, causal analysis, and improvement techniques.
  • Scope and organization: The survey distinguishes standalone LLMs from Retrieval-Augmented LLMs as its two primary usage settings.Examples include ChatGPT for standalone use and BingChat for retrieval-augmented use.

2 FACTUALITY ISSUE

The factuality issue concerns whether LLM outputs follow established, determinable facts, and it is linked to hallucination, outdated information, and domain-specific knowledge gaps. The survey limits its scope to selected LLM settings and emphasizes that factuality errors can affect real-world decisions.

  • Definition: Factuality is the capability to generate content consistent with commonsense, world knowledge, and domain facts grounded in reliable sources.The survey considers only clear, truth-determinable, undisputed facts.
  • Sources of error: Factuality errors arise from domain knowledge deficits, outdated information, missing memorization, forgetting, and reasoning failures.The survey analyzes causes across model, retrieval, generation, and other operational levels.
  • Factuality and hallucination: Hallucination concerns baseless or unwarranted content, whereas factuality concerns learning, acquiring, and using factual knowledge.A response can diverge from prompt specifics while remaining factually accurate, making hallucination distinct from factuality error.
  • Settings: The survey focuses on standard LLMs and Retrieval-Augmented LLMs, including the retrieval of pertinent knowledge snippets.Retrieval augmentation is treated as a prevalent approach for improving factuality.
  • Scope boundary: The survey does not heavily focus on summarization because source texts may be non-factual and summarization has distinct coherence, conciseness, and relevance challenges.Some overlapping retrieval-related work is still discussed.
  • Impact: Factuality failures can produce societal or economic losses and affect legal, medical, and other high-accuracy applications.Reported concerns include hallucinated legal authorities, inaccurate medical recommendations, important omissions, and erroneous responses.

3 FACTUALITY EVALUATION

The survey organizes LLM factuality evaluation around rule-based, neural, human, and LLM-based metrics, alongside benchmarks and empirical studies. Automated methods offer consistency and scalability, while human evaluation captures linguistic and contextual nuances that automated systems may miss.

  • Factuality evaluation covers metrics, benchmarks, empirical studies, and domain-specific assessments of generated content.
  • The survey groups evaluation metrics into rule-based, neural, human, and LLM-based categories.
  • Rule-based metrics are widely used because they are consistent, predictable, easy to implement, and reproducible, but they may miss contextual and linguistic nuances.
  • Human evaluation captures language and context nuances that automated systems may miss, whereas automated evaluation provides consistency and efficient large-scale measurement.
  • FActScores for biographies generated by several commercial LLMs range from 42% to 71%, with factual precision decreasing as entity rarity increases.
  • LLM-based evaluation can improve efficiency and scalability, but requires validated evaluator models and suitable decoding strategies for accurate results.

3.2 Benchmarks for Factuality Evaluation

Factuality benchmarks span general knowledge, truthfulness, hallucination detection, reasoning, self-knowledge, and up-to-date information. They use varied task designs and evaluation protocols to test both factual accuracy and adaptation to changing knowledge.

  • General factuality benchmarks: MMLU measures multitask accuracy across 57 subjects, testing both world knowledge and problem-solving ability.Its subjects range from mathematics and history to computer science and law.
  • General factuality benchmarks: TruthfulQA evaluates whether models avoid false answers to 817 questions across 38 categories, including health, law, finance, and politics.Questions include misconceptions or false beliefs that may induce misleading answers.
  • Domain and reasoning benchmarks: C-Eval assesses advanced knowledge and reasoning in Chinese contexts through multiple-choice questions spanning 52 disciplines and four difficulty levels.C-Eval Hard targets especially challenging subjects, while subsets assess specific abilities.
  • Domain and reasoning benchmarks: GPT-4 was the only evaluated model to exceed 60% average accuracy on C-Eval, while GPT-4, ChatGPT, and Claude were top performers despite not being Chinese-specialized.The benchmark emphasizes Chinese culture, history, laws, and other interests of Chinese users.
  • Self-knowledge and reasoning: SelfAware tests answerable and unanswerable questions to assess whether models recognize what they do not know.GPT-4 achieved an F1 score of 75.5 versus 85.0 for humans; larger models and in-context learning improved performance.
  • Dynamic factuality: REALTIMEQA and FreshQA evaluate responses to current or changing information, exposing weaknesses caused by outdated training or insufficiently informative retrieval.FreshQA uses RELAXED and STRICT human-evaluation modes, while FRESHPROMPT integrates real-time search data.

3.3 Factuality Evaluation Studies

Factuality evaluation studies use human annotation, claim-level checks, self-evaluation, and reference-free assessment across standalone and retrieval-augmented settings. Their findings show both useful evaluation mechanisms and limits in models’ ability to judge factual correctness reliably.

  • Evaluation methods: Manakul et al. classify generated biography sentences as major inaccurate, minor inaccurate, or accurate, then derive passage scores and precision-recall curves.They assess inter-annotator agreement with Cohen’s κ.
  • Evaluation methods: BSChecker extracts subject-predicate-object claims and compares them with references using entailment, contradiction, or neutral labels.The framework detects factual errors at triplet-level granularity.
  • Knowledge evaluation: An information-theoretic metric measures whether target knowledge is present by comparing prediction distributions before and after knowledge injection.GPT-3.5 is evaluated on the T-REx and LAMA datasets for knowledge probing.
  • Adversarial evaluation: A case study compiled 50 adversarial questions based on false premises, all answered incorrectly by GPT-3.5.The dataset targets model behavior when users’ questions presuppose false information.
  • Cross-domain evaluation: FACTOOL audits chatbot factuality across knowledge, code, math, and science prompts using claim-level, response-level, and weighted claim-level accuracy.It evaluates GPT-4, ChatGPT, Claude, Bard, and Vicuna, while FACTOOL itself leverages GPT-4.
  • Limits of automated evaluation: LLM-based factuality judgments can diverge from human assessment: GPT-3.5 could not reliably judge generated answers, and model factuality indicators did not significantly correlate with human evaluations.The findings cover open-question answering and summary factuality scoring.
  • Retrieval-augmented evaluation: RAGAS evaluates retrieval-augmented generation without references by assessing context relevance, context use fidelity, and generated-content quality.Its quality dimensions include faithfulness to the supplied context.

3.4 Evaluating Domain-specific Factuality

Domain-specific factuality evaluation uses benchmarks tailored to specialized tasks, datasets, and models across fields including law, finance, medicine, and biology. These resources support comparison of domain capabilities while reflecting each field’s particular knowledge and reasoning demands.

  • Domain-specific benchmarks organize evaluation by domain, tasks, datasets, and the LLMs assessed.Table 7 summarizes these dimensions for specialized factuality evaluation.
  • Finance: FLARE evaluates financial understanding and prediction using task-specific metrics, including accuracy, weighted F1, entity-level F1, and Matthews correlation coefficient.It compares FinMA and several general-purpose or finance-oriented LLMs under differing shot settings.
  • Medicine: Medical evaluation resources include CMB, a Chinese and culturally localized benchmark, and Huatuo-26M, a dataset with over 26 million medical QA pairs.Huatuo-26M covers diseases, symptoms, treatments, and drug information, while CMB evaluates both prominent and localized Chinese LLMs.
  • Biology: GeneTuring contains 12 tasks with 50 question-answer pairs each across nomenclature, genomic location, functional analysis, and sequence alignment.GeneGPT was evaluated on nine tasks related to NCBI resources.
  • Law: LegalBench contains 162 tasks spanning six legal-reasoning types and evaluates 20 open-source and commercial LLMs.Its interdisciplinary construction involved substantial contributions from legal professionals.
  • Law: LawBench evaluates legal-detail memorization, text understanding, and legal problem solving across 20 tasks and 51 LLMs.The evaluated models include multilingual, Chinese-oriented, and law-specific systems; GPT-4 ranked highest, while reliable legal performance remains unresolved.

4 ANALYSIS OF FACTUALITY

The survey analyzes factuality through knowledge storage, awareness, contextual use, conflicts, and error causes. It finds that LLM factuality depends on pre-training information and balancing internal knowledge with external context, while errors arise at multiple operational levels.

  • Analysis dimensions: LLMs process factual content through knowledge storage, awareness, contextual use, and responses to conflicting information.The analysis examines both parametric and retrieved knowledge across these dimensions.
  • Knowledge storage: Causal tracing studies indicate that factual knowledge is primarily stored in early-layer feed-forward networks.The survey presents this as evidence about how factual information is organized within model parameters.
  • Knowledge awareness: LLMs often misjudge their factual knowledge boundaries and show excessive confidence, although retrieval enhancement can partly compensate.SelfAware evaluates recognition of answerable and unanswerable questions, while other work studies confidence and perplexity.
  • Parametric knowledge: Fact-based answering depends strongly on the amount of relevant information encountered during pre-training, limiting performance on long-tail knowledge.The survey therefore identifies retrieval augmentation as a complementary approach for such knowledge-intensive tasks.
  • Contextual influence: Larger models may favor parametric knowledge over counterfactual context, while reducing context noise improves controllability without resolving robustness.Prompt templates are also reported as a strategy for more faithful generation.
  • Knowledge conflict: Knowledge conflicts increase prediction uncertainty, and larger models tend to default to their parametric knowledge when context contradicts learned information.The survey frames factuality as requiring effective use of both internal knowledge and external context.
  • Root causes: Factual errors arise from model-level, retrieval-level, generation-level, and miscellaneous causes, including domain knowledge deficits and outdated information.Training-data coverage and update timing constrain what models can know in specialized or recently changed domains.

5 ENHANCEMENT

The survey groups factuality-enhancement methods into standalone LLM techniques, retrieval-augmented generation, and domain-specific approaches. It also cautions that reported improvements across methods may not be directly comparable because experimental settings differ.

  • Enhancement methods target standalone LLMs, retrieval-augmented generation, and domain-specific factuality.Figure 2 organizes the methods into these three primary areas.
  • The enhancement discussion spans generation, retrieval augmentation, inference-phase methods, and domain-specific factuality improvements.These phases are outlined as the scope of the section.
  • Performance metrics from different enhancement studies may not be directly comparable, even on the same metric and dataset.The studies use distinct zero-shot, few-shot, or full experimental settings and compare improvements against baseline LLMs.

5.1 On Standalone LLM Generation

Standalone LLM factuality enhancement methods target training data, supervised knowledge injection, model editing, inference-time steering, and multi-agent elicitation. The survey organizes these approaches while noting that reported results across studies may not be directly comparable.

  • Standalone LLM enhancement methods fall into three categories: improving unsupervised training data, using supervised data, and eliciting knowledge during generation.The elicitation category includes multi-agent collaboration, prompting, and novel decoding methods.
  • Improving Factual Knowledge from Unsupervised Corpora: Pretraining enhancements address redundant or uninformative data through deduplication, informative-token masking, topic prefixes, and sentence-completion objectives.These methods aim to improve factual knowledge acquisition and reduce problems such as immemorization and forgetting.
  • Enhancing Factual Knowledge from Supervised Data: Supervised fine-tuning injects labeled or structured knowledge and can improve the model’s reasoning and factual knowledge.Approaches include knowledge-graph triplet training, contrastive learning with negative samples, and precise parameter adjustments.
  • Enhancing Factual Knowledge from Supervised Data: Knowledge-graph enhancement strategies operate before training, during training, or after training by combining factual data, textual and structural knowledge, and knowledge-guided objectives.
  • Model Editing: Model editing changes fact-related weights or activations more precisely than direct fine-tuning, but robustness and generalization remain open questions.Weight-based methods include KN, ROME, KE, and MEND, while Inference-Time Intervention adjusts activations during inference.
  • Optimally Eliciting Factual Knowledge from the Model: Multi-agent debate improves factual accuracy across tasks and can be applied directly to existing black-box models through shared prompts.Multiple model instances debate answers and reasoning before reaching a consensus.

5.2 On Retrieval-Augmented Generation

Retrieval-Augmented Generation addresses outdated information and memorization limits in standalone LLMs, while introducing retrieval and interpretation challenges. The survey organizes RAG approaches around normal retrieval, interactive retrieval, model adaptation, and additional knowledge bases.

  • RAG supplements standalone LLMs with external information but can suffer from insufficient retrieval and misinterpretation of related data.These challenges motivate interactive retrieval mechanisms.
  • Other approaches adapt LLMs to RAG settings or retrieve from external parametric memories and knowledge graphs.REPLUG fine-tunes retrieval models while treating the LLM as a black box, whereas other methods extend the available knowledge base.
  • Normal RAG Setting: Normal RAG retrieves external data and passes it to the LLM during generation through document loading, transformation, embedding, storage, and retrieval modules.The workflow includes chunking documents, searching vector stores, and selecting retrieved data.
  • Interactive Retrieval: Interactive retrieval methods incorporate reasoning, actions, or feedback to obtain more appropriate information during generation.Examples include CoT-based retrieval, agent-based API use, and self-reflection after failure signals.
  • Interactive Retrieval: FLARE dynamically decides when and what to retrieve by monitoring low-probability tokens and using the current sentence as a query.Across multihop QA, commonsense reasoning, long-form QA, and open-domain summarization with GPT-3.5, it outperforms baselines; both query-generation methods perform comparably.

5.3 Domain Factuality Enhanced LLMs

Domain knowledge deficits constrain LLM use in specific fields, so the survey categorizes domain-factuality enhancement techniques and the domains represented in its review. The techniques include continual pretraining, continual supervised fine-tuning, training from scratch, and external knowledge.

  • Domain knowledge deficit limits LLM application in specific fields and motivates domain-specific factuality enhancement.
  • The survey covers healthcare, finance, law, geoscience, education, food testing, and home renovation domains.Table 9 uses H, F, L, G, E, FT, and HR as domain abbreviations.
  • Continual Pretraining: Continual pretraining updates a pretrained model with domain-specific data so it remains relevant as new information becomes available.
  • Continual SFT: Continual SFT fine-tunes models on labeled or annotated domain data to learn domain nuances and provide accurate, contextually relevant information.
  • Train From Scratch: Training from scratch builds domain understanding with minimal prior knowledge and can suit entirely new domains or tasks.The approach may require substantial resources and does not leverage pre-existing knowledge.
  • External Knowledge: External knowledge augments internal model knowledge with databases, websites, or structured repositories for fact verification and additional information.The approach is especially relevant to dynamic or rapidly changing content.

5.4 Healthcare domain-enhanced LLMs

Healthcare-focused LLMs apply varied training, retrieval, knowledge-graph, and dialogue strategies to medical tasks. The survey describes systems for consultation, question answering, recruitment, de-identification, genomics, guideline recommendation, and molecular prediction.

  • Healthcare LLMs support medical text classification, question answering, dialogue, de-identification, recommendation, and molecular tasks.Examples include CohortGPT, ChatDoctor, DeID-GPT, BioMedLM, DoctorGLM, BioGPT, GeneGPT, Almanac, and MolXPT.
  • HuatuoGPT continually pretrains Baichuan-7B and Ziya-LLaMA-13B-Pretrain-v1 on ChatGPT-distilled and doctor-generated data for medical consultation.The survey reports state-of-the-art performance in medical consultations.
  • Zhongjing uses 70,000 authentic doctor-patient dialogues in CMtMedQA to support complex dialogue and proactive inquiry.Its backbone is Ziya-LLaMA-13B-v1.
  • LLM-AMT augments GPT-3.5-Turbo and LLaMA-2-13B with a hybrid retriever containing 51 MedQA textbooks and Wikipedia for open-domain medical QA.
  • DISC-MedLLM combines medical knowledge graphs, reconstructed real-world dialogues, and human-guided preference rephrasing to build high-quality SFT data.It is applied to Baichuan-13B-Base.
  • Other systems use knowledge graphs, biomedical corpora, web APIs, or retrieval for specialized medical tasks.CohortGPT uses graph assistance, BioMedLM uses PubMed, GeneGPT uses NCBI Web APIs, and Almanac retrieves medical guidelines.

5.5 Legal domain enhanced LLMs

Legal-domain LLMs are adapted through legal-text training, supervised task design, domain-knowledge injection, retrieval, and error handling. The survey links these approaches to legal assistance, explanation, and factuality concerns such as hallucination.

  • Legal LLMs provide assistance with legal questions, document generation, legal advice, and explanations of legislative terms.
  • Legal-text corpora, retrieval modules, and continual domain development are presented as ways to address hallucination and support reliable legal services.The survey describes these innovations as opening further research directions in legal applications.
  • LawGPT 1.0 is a GPT-3 model fine-tuned on a large legal-text corpus for conversational legal assistance.
  • Legal-domain enhancement combines domain-knowledge injection, supervised fine-tuning, and retrieval modules to improve factuality during generation.These methods are described for LLaMA-based Chinese legal tasks.
  • ChatLaw improves factuality through data screening and self-attention-based error handling using legal regulations, judicial interpretations, and real consultation data.

5.6 Finance Domain-enhanced LLMs

Finance-domain-enhanced LLMs combine specialized language modeling with extensive financial and business data to support finance and e-commerce tasks. The surveyed examples illustrate domain-tailored training and broad financial NLP evaluation.

  • Finance-domain LLMs are designed for financial text analysis, e-commerce settings, and business-related instruction processing.
  • E-commerce domain-enhanced LLMs: EcomGPT is an e-commerce model fine-tuned on EcomInstruct, a 2.5-million-example dataset spanning product information, user reviews, and other tasks.Its training uses atomic and Chain-of-Task tasks with BLOOMZ as the backbone.
  • Finance domain-enhanced LLMs: BloombergGPT is a specialized 50-billion-parameter financial model trained on 363 billion tokens combining Bloomberg financial sources with general-purpose data.Financial data accounts for 51.27% of its training data.
  • Finance domain-enhanced LLMs: BloombergGPT is evaluated on financial NLP tasks including sentiment analysis, named entity recognition, binary classification, and question answering.

5.7 Other Domain-Enhanced LLMs

Other domain-enhanced LLMs adapt general models, domain corpora, instruction datasets, and external knowledge to specialized settings. The examples cover geoscience, renewable energy, ocean science, education, food testing, and home renovation.

  • Geoscience and Environment domain-enhanced LLMs: K2 applies continual training on 5.5 billion geoscience tokens and GeoSignal fine-tuning, while GeoBench evaluates geoscience performance.
  • Geoscience and Environment domain-enhanced LLMs: HouYi targets renewable energy using more than 1.1 million academic literature sources in REAP and fine-tuning from general LLMs such as ChatGLM-6B.
  • Geoscience and Environment domain-enhanced LLMs: OceanGPT uses DoInstruct to obtain ocean-domain data and achieves higher ocean-science knowledge expertise on OceanBench.
  • Education domain-enhanced LLMs: GrammarGPT supports Chinese grammatical error correction through ChatGPT-generated and human-annotated data, heuristic guidance, and an open-source design.
  • Food domain-enhanced LLMs: FoodGPT addresses food-testing hallucinations by combining structured knowledge, scanned documents, incremental pretraining, and a knowledge graph external base.
  • Home renovation domain-enhanced LLMs: ChatHome combines domain-adaptive pretraining and instruction tuning on professional articles, standards, and web content for home renovation.

6 CONCLUSION

The survey organizes factuality research across definitions, implications, evaluation, underlying mechanisms, and enhancement methods for standalone, retrieval-augmented, and domain-specific LLMs. It concludes that fully factual LLMs remain difficult because evaluation and factual storage, updating, production, and enhancement techniques retain unresolved limitations.

  • The survey covers factuality definitions and implications, evaluation metrics, benchmarks, studies, and domain-specific evaluations.
  • It examines mechanisms underlying factuality and enhancement techniques for standalone, retrieval-augmented, and domain-specific LLMs.
  • Challenges: Factuality evaluation remains difficult because natural-language variability and nuance complicate assessment.
  • Challenges: How LLMs store, update, and produce facts remains insufficiently understood.
  • Challenges: Continual training and retrieval show promise but retain limitations, leaving fully factual LLMs as an ongoing challenge.
Loading 2310.07521v3…