Factuality Enhanced Language Models for Open-Ended Text Generation

Nayeon Lee; Wei Ping; Peng Xu; Mostofa Patwary; Pascale Fung; Mohammad Shoeybi; Bryan Catanzaro

Source-linked AI summary

Factuality Enhanced Language Models for Open-Ended Text Generation

Nayeon Lee, Wei Ping, Peng Xu, Mostofa Patwary, Pascale Fung, Mohammad Shoeybi, Bryan Catanzaro

arXiv:2206.04624v3Jun 9, 2022cs.CLcs.AIcs.CYcs.LG

TL;DR

Pretrained LMs can generate nonfactual open-ended text, motivating methods that measure and improve factuality. The paper introduces FactualityPrompts, analyzes model size, decoding, and training, and proposes factuality-enhancing interventions. Larger LMs are more factual on its benchmark, while factual-nucleus sampling and factuality-enhanced training improve factuality while addressing quality or diversity trade-offs.

Problem
Pretrained LMs have limited factual accuracy in open-ended generation, while prior work mainly evaluates factuality in task-specific fine-tuned models.
Method
The paper builds FactualityPrompts and factuality metrics, then proposes factual-nucleus sampling plus TopicPrefix and sentence completion loss for factuality-enhanced training.
Results
Larger LMs are more factual, factual-nucleus sampling improves factuality while maintaining quality and diversity, and TopicPrefix plus sentence completion loss reduces factual errors.
Takeaways & Limitations
Factuality and diversity involve a trade-off that should be considered when designing LMs for different task goals.
Takeaways & Limitations
The decoding experiments mainly use a 61.3B LM because it balances learning capacity with resource cost.

Abstract

from arXiv · show

Pretrained language models (LMs) are susceptible to generate text with nonfactual information. In this work, we measure and improve the factual accuracy of large-scale LMs for open-ended text generation. We design the FactualityPrompts test set and metrics to measure the factuality of LM generations. Based on that, we study the factual accuracy of LMs with parameter sizes ranging from 126M to 530B. Interestingly, we find that larger LMs are more factual than smaller ones, although a previous study suggests that larger LMs can be less truthful in terms of misconceptions. In addition, popular sampling algorithms (e.g., top-p) in open-ended text generation can harm the factuality due to the ''uniform randomness'' introduced at every sampling step. We propose the factual-nucleus sampling algorithm that dynamically adapts the randomness to improve the factuality of generation while maintaining quality. Furthermore, we analyze the inefficiencies of the standard training method in learning correct associations between entities from factual text corpus (e.g., Wikipedia). We propose a factuality-enhanced training method that uses TopicPrefix for better awareness of facts and sentence completion as the training objective, which can vastly reduce the factual errors. We release our code and FactualityPrompts benchmark at: https://github.com/nayeon7lee/FactualityPrompt.

1 Introduction

The paper addresses nonfactual generations from pretrained LMs in open-ended text generation, where existing factuality work has largely focused on downstream tasks. It introduces a benchmark, analyzes model and decoding behavior, and proposes factuality-enhancing methods.

Pretrained LMs model token correlations and can generate nonfactual information, raising concerns for applications such as content creation and dialogue.
Existing factuality methods primarily target fine-tuned models and source-target consistency in summarization, question answering, and table-to-text generation.
Plain-text factual sources such as Wikipedia offer scalable knowledge without the human-annotation bottleneck of structured knowledge graphs.
The paper builds open-ended factuality benchmarks and metrics, studies LMs from 126M to 530B parameters, and finds named-entity factual error falls from 63.69% to 33.3%.
It proposes factual-nucleus sampling and factuality-enhanced training to improve factuality while addressing generation quality, diversity, and entity-association errors.

2 Related Work

Prior work studies factuality in misconceptions, parametric knowledge, downstream NLG, and human-feedback systems. This paper instead focuses on factual accuracy in general-purpose open-ended generation.

TruthfulQA reports that larger web-trained LMs can be less truthful about false beliefs or misconceptions, whereas this paper studies factual rather than conceptual knowledge.
Parametric-knowledge research commonly evaluates factual recall with cloze-style question answering and attempts to inject or edit knowledge in LMs.
Other factuality research targets task-specific LMs for summarization, data-to-text, and dialogue, unlike this paper’s general-purpose open-ended setting.
Human-feedback and source-citation approaches improve factuality but motivate this paper’s focus on human-free mitigation as a less expensive direction.

3 FACTUALITYPROMPTS and Evaluation Metrics

The FactualityPrompts framework evaluates open-ended LM continuations by identifying checkworthy text, supplying relevant Wikipedia evidence, and measuring factuality and quality. Its NEER and EntailR metrics correlate with human factuality judgments.

The evaluation framework generates continuations, identifies checkworthy factual claims, prepares ground-truth knowledge, and computes factuality and quality measures.
FactualityPrompts includes factual and nonfactual prompts to test how prompt factuality affects LM continuations.
Ground-truth evidence is selected at document or sentence level, using FEVER Wikipedia annotations for documents and automatic sentence selection for sentence-level evaluation.
NEER measures the fraction of generated named entities absent from the ground-truth Wikipedia document, while EntailR measures the fraction of generations entailed by an entailment model.
Human annotations of 200 continuations show strong correlation between factuality judgments and the automatic metrics, with lower NEER indicating better factuality.

4 Factuality Analysis of Pretrained LMs

The analysis examines model size, prompt type, and decoding. Larger models are more factual, factual prompts reduce errors, and greedy decoding improves factuality at the cost of diversity and quality; factual-nucleus sampling targets that trade-off.

Model Size: The study compares LMs with 126M, 357M, 1.3B, 8.3B, and 530B parameters across factuality-related behaviors.
Prompt Type: Factual prompts consistently produce fewer nonfactual generations than nonfactual prompts, with the NEER gap increasing from 4% to 7% as model size grows.
Decoding Algorithm: Greedy decoding is more factual because it minimizes randomness, but it sacrifices generation diversity and quality.
Error Types: Greedy-generation errors include named-entity mix-ups and fabricated facts, reflecting wrong entity associations from next-subword training.
Decoding Algorithm: Factual-nucleus sampling offers better factuality-versus-diversity and factuality-versus-repetition trade-offs than nucleus sampling.

5 Factual-Nucleus Sampling

Factual-nucleus sampling dynamically reduces sampling randomness later within each sentence, targeting better factuality while preserving generation quality. Its components improve factuality and quality relative to standard decoding, with a stated diversity–factuality trade-off.

Evaluation: The proposed sampling algorithm is designed to achieve a better trade-off between generation quality and factuality than existing decoding algorithms.The evaluation uses a 1.3B LM and compares multiple nucleus and factual-nucleus settings.
Method: Factual-nucleus sampling dynamically adapts the nucleus probability p during each sentence to reduce harmful later-stage randomness.The method uses λ-decay, p-reset, and an ω lower bound.
Ablation results: 12.5% lower NEER and 10.1% higher EntailR result from adding λ-decay with λ = 0.5 to top-p 0.9.This setting also makes diversity and repetition resemble greedy decoding.
Ablation results: 0.26 diversity gain follows p-reset with λ = 0.5, while factuality changes negligibly.p-reset mitigates repetition without much factuality loss.
Quality–factuality trade-off: ×11 diversity and ×4.6 repetition improvements over greedy occur with p=0.9, λ=0.9, ω=0.3.The method still under-performs top-p 0.9 on diversity, but its perplexity does not exceed top-p.

6 Factuality-Enhanced Continued Training

Factuality-enhanced continued training combines TOPICPREFIX with sentence completion loss to improve factual associations learned from text. The method reduces factual errors, although domain-adaptive training on Wikipedia alone can hardly improve factuality.

Method: TOPICPREFIX and sentence completion loss form the proposed factuality-enhanced continued-training method.TOPICPREFIX improves awareness of document-level facts, while sentence completion focuses learning on later sentence tokens.
Sentence Completion Loss: Sentence completion loss masks prediction losses before a pivot t, emphasizing subwords later in each sentence during autoregressive training.The pivot is needed only during training, not inference.
Pivot strategies: SCHALF uses t = 0.5× sentence-length, while SCRANDOM and SCROOT use random and dependency-based pivots; SCHALF performs on par with the complex strategies.The authors recommend SCHALF as the default option.
Baseline: Wikipedia-only domain-adaptive training can hardly improve generation factuality.The result challenges the assumption that restricting continued training to factual-domain text is sufficient.
TOPICPREFIX results: 42.1% to 27.6% NEER reduction follows TOPICPREFIX-preprocessed Wikipedia continued training with factual-nucleus decoding.This configuration also reports NEER 27.6% versus 39.9% and repetition 8.0% versus 33.1% against 1.3B greedy decoding.
Sentence completion results: 8.7% to 17.4% EntailR and 27.6% to 23.6% NEER changes follow adding TOPICPREFIX + SCHALF over TOPICPREFIX alone.The improvement is reported as consistent across pivot-selection strategies.
Model scale: 14.5% NEER and 25.5% EntailR are achieved by the method on the 530B LM, the strongest reported factuality results.The 530B model produces less diverse generations despite improved generation quality.

7 Conclusion

The paper presents a benchmark and methods for improving factuality in open-ended generation, while identifying a trade-off between diversity and factuality.

Factual-nucleus sampling improves generation factuality at inference time.The method dynamically targets factuality during decoding, while retaining the computational cost of regular top-p sampling.
Combining sentence completion loss with TOPICPREFIX preprocessing improves factuality during continued training.The approach is designed to improve the model’s awareness of facts in the training corpus.
The results reveal a trade-off between diversity and factuality in language-model design.The authors frame this trade-off as a consideration when prioritizing model attributes for a task’s final goal.

A Generation Examples

The examples contrast generations from pretrained models using greedy or top-p sampling with factuality-enhanced generations using factual-nucleus sampling. They illustrate factual, nonfactual, repetitive, and incorrectly targeted outputs across several prompts.

The example annotations use green for factual text, red for nonfactual text, and strikethrough for repetition.The authors caution that the annotations were checked extensively but are not guaranteed to be perfect.
Medical: For the lung-cancer prompt, the factuality-enhanced output lists symptoms and diagnostic procedures, whereas greedy and top-p outputs repeat or provide unrelated content.The factuality-enhanced example includes coughing, chest pain, hoarseness, weight loss, chest X-ray, and biopsy; the top-p output discusses oxygen masks.
Actor: For the film prompt, the factuality-enhanced output correctly describes DiCaprio’s supporting role, while greedy and top-p outputs provide unrelated or incorrect film details.The factuality-enhanced continuation identifies his role as Johnny Depp’s character’s brother in What’s Eating Gilbert Grape.
Company: Factuality-enhanced generation supplies relevant biographical facts for the KFC prompt, unlike the repetitive greedy output and unrelated top-p continuation.The factual-nucleus example discusses Colonel Harland Sanders and KFC’s founding, while the other outputs repeat or discuss unsupported payment-system claims.
History and Sports: The examples include factuality-enhanced continuations about San Francisco’s Spanish settlement and Michael Jordan’s basketball achievements.The San Francisco example describes the Presidio and Mission Dolores, while the Jordan example lists major awards and scoring records.

C Experiment Details

The experiments construct factual and nonfactual prompts from FEVER and apply TOPICPREFIX preprocessing to Wikipedia text, with specified training and model-architecture details.

TOPICPREFIX: TOPICPREFIX preprocessing prepends an entity name to Wikipedia sentences during training.The example repeats “Barack Obama ==>” before sentences describing Barack Obama, making the topic explicit.
Training details: The 1.3B and 530B factuality-enhancement runs use different learning rates and batch sizes, with maximum input sequence length 2048 for both.The 1.3B configuration uses learning rate 2e-6 and batch size 64; the 530B configuration uses 1e-5 and 512.
Model and data setup: All language models with different sizes are pretrained on the same corpus, and inference uses maximum subword sequence length 150.Architecture details are reported in Table 6, while data statistics for FACTUALITYPROMPTS are reported in Table 7.
Prompt construction: Factual and nonfactual prompts are constructed from FEVER’s SUPPORTED and REFUTED claims, respectively.Claims are filtered to remove prompts that are too short or otherwise unsuitable for providing context.

E Limitations and Societal Impact

The paper notes that factuality enhancement has societal benefits for safer deployment, but continued pretraining of large language models carries a negative carbon footprint.

Continued pretraining of large language models has a negative carbon footprint.The authors contrast this cost with the intended benefit of reducing fake information and supporting safer deployment.

F.1 Ablation Study of Sentence Completion Loss

Sentence completion loss alone has little effect on factuality, but combined with TOPICPREFIX it produces a significant boost on both factuality metrics.

Sentence completion loss alone is indifferent to standard factual-domain adaptive training, showing negligible difference in factuality.
Combining sentence completion loss with TOPICPREFIX significantly improves both factuality metrics.
The ablation uses 3000 Factual Prompts to measure the stand-alone and combined effects of sentence completion loss.

F.2 Experimental Results with Perplexity

The proposed decoding and training methods preserve generation fluency in experiments with a 1.3B pretrained LM, outperforming Top-p 0.9 on perplexity.

PPL scores for all proposed decoding choices are 1.9∼4.1, lower than the Top-p 0.9 score of 12.0.PPL is evaluated on generated text from the 1.3B pretrained LM as a fluency measure.
The results consistently indicate that the proposed decoding and training methods do not harm generation fluency.
NEER denotes named-entity error, EntailR entailment ratio, Div. distinct 4-grams, and Rep. repetition; arrows indicate whether higher or lower is better.
Table 9 reports factuality for a 1.3B LM under different decoding algorithms, with p, λ, and ω specifying nucleus probability, decay factor, and decay lower bound.

G Generation Examples (Continued)

The generation examples contrast greedy decoding with the paper’s method across factual prompts, showing more complete and factually aligned continuations in several cases.

For the Pompeii prompt, greedy decoding repeats the prompt and image-caption text, whereas the comparison continuation adds contextual details about the site’s entrance.
For David Beckham, the comparison continuation supplies career facts, including league titles in four countries, while greedy decoding produces only a brief ambassador statement.
For Andy Warhol, the comparison continuation develops the meaning of his book creation, while greedy decoding gives a short graphic-artist continuation.
Other examples include Jordan’s Olympic career, Tiber Oil Field, Naples, and The Blue Lagoon, illustrating varied factual-prompt continuations.
For Doctor Who, the comparison continuation gives a coherent BBC, Doctor, Gallifrey, and TARDIS description, whereas greedy decoding repeats a generic show synopsis.