Source-linked AI summary
Factuality Challenges in the Era of Large Language Models
Isabelle Augenstein, Timothy Baldwin, Meeyoung Cha, Tanmoy Chakraborty, Giovanni Luca Ciampaglia, David Corney, Renee DiResta, Emilio Ferrara, Scott Hale, Alon Halevy, Eduard Hovy, Heng Ji, Filippo Menczer, Ruben Miguez, Preslav Nakov, Dietram Scheufele, Shivam Sharma, Giovanni Zagni
TL;DR
Large language models create a veracity problem because they can produce hallucinated or misleading content and can be used to scale malicious information operations. This paper synthesizes these risks, reviews technological and institutional responses, and concludes that responsible use requires improved safeguards, evaluation, regulation, coordination, and AI literacy.
Problem
LLMs can mislead users through unreliable outputs and exploitative applications, while factuality evaluation remains difficult because benchmarks, metrics, and ground-truth references have important limitations.
Method
The paper reviews LLM veracity risks and discusses technological, regulatory, coordination, and AI-literacy strategies for addressing them.
Results
The paper concludes that no single solution fully mitigates LLM-related harms and presents combined safeguards including hallucination control, knowledge editing, evidence support, retrieval, regulation, coordination, and literacy.
Takeaways & Limitations
Responsible LLM deployment requires coordinated technological development, adaptive regulation, and broad AI literacy to reduce risks while retaining benefits.
Takeaways & Limitations
Factuality assessment remains an open problem because maintaining accurate ground-truth references is costly, subject to data drift, and vulnerable to benchmark and metric limitations.
Abstract
from arXiv · showhide
The emergence of tools based on Large Language Models (LLMs), such as OpenAI's ChatGPT, Microsoft's Bing Chat, and Google's Bard, has garnered immense public attention. These incredibly useful, natural-sounding tools mark significant advances in natural language generation, yet they exhibit a propensity to generate false, erroneous, or misleading content -- commonly referred to as "hallucinations." Moreover, LLMs can be exploited for malicious applications, such as generating false but credible-sounding content and profiles at scale. This poses a significant challenge to society in terms of the potential deception of users and the increasing dissemination of inaccurate information. In light of these risks, we explore the kinds of technological innovations, regulatory reforms, and AI literacy initiatives needed from fact-checkers, news organizations, and the broader research and policy communities. By identifying the risks, the imminent threats, and some viable solutions, we seek to shed light on navigating various aspects of veracity in the era of generative AI.
1 Introduction
LLMs have evolved from statistical language models into highly capable systems that generate increasingly human-like language, but their widespread adoption raises serious factuality and misuse concerns. The paper examines hallucinations, malicious applications, and strategies for preserving veracity while retaining benefits.
- Technological development: LLMs evolved from frequency-based language models into systems capable of memorizing, reasoning about, and generating increasingly human-like natural language.Despite architectural and training differences, they principally perform next-word prediction from a prompt.
- Public adoption: ChatGPT reached 100 million monthly active users two months after launch, accelerating public attention to conversational LLMs.Subsequent innovations and open-source alternatives further expanded the technology’s profile and availability.
- Public adoption: The proliferation of LLMs has raised societal concerns as rapid adoption increasingly involves professionals and people with limited or no technical knowledge.The paper also notes that a small group of well-funded organizations dominates advances because training requires extensive computational resources.
- Factuality risks: LLMs can produce factually false statements and persuasive explanations, making it difficult to distinguish fiction from fact when fluent, confident language masks inaccuracies.These risks are especially concerning in sensitive information-seeking contexts where users may rely on chatbot responses.
- Malicious use: Malicious users can exploit LLMs to generate credible inauthentic content at scale, including improved phishing messages and manipulative information.The paper connects these capabilities to potential harms across politics, finance, health, and society.
- Scope and contribution: The paper surveys technological, regulatory, and literacy responses intended to reduce LLM-related risks while preserving their potential benefits.It frames responsible use as requiring action from individuals, organizations, and governments.
2 Risks, Threats, and Challenges
LLM factuality risks arise both from unreliable outputs that users may trust and from deliberate misuse that scales deception. The paper identifies evaluation, sourcing, presentation, access, personalization, impersonation, and detection challenges that complicate fact-checking.
- Core risks: LLM veracity problems have two connected sides: unreliable outputs from flawed data or hallucinations, and intentional malicious services such as scams and disinformation.Low-credibility malicious content can also become training data, linking accidental unreliability with deliberate abuse.
- Core risks: Chatbots can generate false or misleading content that spreads rapidly across platforms, aggravating information-quality and reliability problems.Their coherent output and lack of credible sourcing make generated claims difficult to assess.
- Output characteristics: LLMs often struggle with factual accuracy despite fluency, while confident and anthropomorphic presentation can make erroneous content appear authoritative and persuasive.The paper notes that fluent style can shape perceptions of truth even for claims from unreliable sources.
- Fact-checking challenges: Private, proprietary chatbot interactions make misinformation harder for fact-checkers to detect and correct than widely circulated claims in traditional settings.A chatbot’s misinformation may be buried among many true and false statements.
- Access and traceability: Open access to models can accelerate innovation and democratization, but growing model diversity makes malicious use more difficult to trace.The paper contrasts locally accessible models with ChatGPT’s rate-limited API.
- User perception: The halo effect can lead users to generalize a model’s competence from one topic to open-domain reliability, including during urgent events outside its training knowledge.The paper gives pre-2019 training as an example of why a chatbot might respond incorrectly about COVID-19.
- Evaluation: Factuality evaluation remains unreliable because benchmarks can be contaminated, metrics have documented biases, and maintaining accurate ground truth is costly and vulnerable to data drift.The paper reports mixed performance on misinformation-detection tasks involving scientific and social COVID-19 claims.
- Malicious scaling: Personalization, style imitation, and automated variation enable targeted deception and can help malicious content evade fact-checking based on circulation volume.Generated messages may combine factual and false statements or imitate journalists, politicians, and other trusted figures.
3 Addressing the Threats
The paper outlines technological, evaluative, regulatory, provenance, and educational responses to LLM factuality threats, while emphasizing that current solutions remain limited. Proposed approaches include grounding generation in external knowledge, improving safety and evaluation, authenticating content, and increasing public awareness.
- Technological Responses: Training large LLMs is often beyond academic resources, motivating research on smaller models and targeted safety interventions.Training a multi-billion-parameter model from scratch takes several months and hundreds of GPUs, whereas smaller models are more feasible in academia.
- Technological Responses: Current safety and hallucination-control efforts span data cleansing, instruction-tuning, hidden prompts, deployment filters, self-consistency, and cross-model verification.
- Technological Responses: Retrieval-augmented generation incorporates external context to reduce inaccurate outputs, but requires efficient large-scale retrieval and robust evaluation.
- Evaluation: Existing benchmarks and evaluation metrics have limitations because training contamination and distribution shifts can undermine factuality assessment.Benchmark datasets may overlap with training data, while common metrics assume similar training and evaluation distributions that do not fit zero-shot and in-context settings.
- Provenance and Regulation: Cryptographic provenance could help establish whether text was altered and potentially limit the spread of harmful AI-generated content.
- Provenance and Regulation: Regulatory efforts include AI-content watermarks, transparency obligations, high-risk AI frameworks, and restrictions on misleading tools, while open-source models may evade regulation.
- Public Education: Public education should explain deceptively polished chatbot content, but skepticism may also reduce trust in credible sources and increase vulnerability to conspiracy theories.
4 Fact-Checking Opportunities
LLMs can support fact-checkers by processing, organizing, and matching claims across large volumes of content, although they do not themselves possess a concept of truth. They may also assist with stance detection and provide interfaces to verified domain-specific corpora, but plausible errors remain possible.
- Fact-Checking Support: LLMs can transcribe and summarize media, extract claims, organize claim frames, and identify repeated or semantically equivalent claims for fact-checking workflows.These capabilities support cross-media and cross-lingual checking and help fact-checkers monitor large volumes of potentially misleading information.
- Stance Detection: ChatGPT and other LLMs show indications of usefulness for stance detection, with recent studies reporting stronger performance there than on emotion or pragmatic analyses.
- Domain-Specific Chat Support: Domain-specific chatbots can provide interfaces to factually verified article collections, but LLMs may still insert plausible incorrect information.
5 Conclusion
The paper calls for urgent, coordinated action to reduce harms from generative AI while preserving its benefits. Its agenda combines regulation, AI literacy, evidence-supporting technology, external knowledge, and international collaboration.
- Conclusion: Users should verify basic facts in LLM-generated text, regardless of fluency, while society develops regulation, education, and collaboration responses.
- Coordination and Collaboration: Global coordination should combine research investment, adaptive legislative safeguards, and collaboration between political and industry leaders.
- Regulations: Comprehensive regulation should address intentional and inadvertent harms, high-risk technology production and sales, ethical industry standards, journalism guidance, labeling, and sector-specific evaluation.
- Promoting AI Literacy: AI literacy efforts should reach all ages, integrate ethics into graduate curricula, and teach digital consumers about GenAI harms and causes.
- Technological Development: Technological development should provide upfront information about LLM limits, implement conversational guardrails, support evidence, use external knowledge, and detect coordinated harmful content.
Funding Information
The paper acknowledges support from multiple academic, governmental, nonprofit, and industry organizations.
- The authors report funding and support from the Institute for Basic Science, National Research Foundation of Korea, Wipro AI, the European Union, the National Science Foundation, DARPA, the Knight Foundation, and Craig Newmark Philanthropies.