Source-linked AI summary
Generating Benchmarks for Factuality Evaluation of Language Models
Dor Muhlgay, Ori Ram, Inbal Magar, Yoav Levine, Nir Ratner, Yonatan Belinkov, Omri Abend, Kevin Leyton-Brown, Amnon Shashua, Yoav Shoham
TL;DR
Language-model factuality evaluation often samples facts from the model, which can underrepresent rare or domain-specific facts. FACTOR transforms a factual corpus into a contrastive benchmark of true and similar false statements, and its scores increase with model size, improve with retrieval, and better track open-ended factuality than perplexity when rankings diverge.
Problem
Existing factuality evaluations do not control the evaluated facts and may underrepresent rare or domain-specific information, despite the need to assess factual errors before deployment.
Method
FACTOR automatically transforms factual corpora into benchmarks where models compare one factual completion with similar non-factual alternatives.
Results
FACTOR scores increase with model size, improve with retrieval augmentation, and better predict factuality in open-ended generation than perplexity when the two rank models differently.
Takeaways & Limitations
FACTOR provides an automatic way to measure domain-specific factuality and assess factuality improvements from retrieval-augmented language models.
Takeaways & Limitations
Automated benchmark generation may not fully satisfy the benchmark requirements, and contradiction prompts may vary in effectiveness across domains.
Abstract
from arXiv · showhide
Before deploying a language model (LM) within a given domain, it is important to measure its tendency to generate factually incorrect information in that domain. Existing methods for factuality evaluation of LLM generation focus on facts sampled from the LM itself, and thus do not control the set of evaluated facts and might under-represent domain specific or rare facts. We propose FACTOR: Factual Assessment via Corpus TransfORmation, a scalable approach for evaluating LM factuality. FACTOR automatically transforms a factual corpus of interest into a benchmark evaluating an LM's propensity to generate true facts from the corpus vs. similar but incorrect statements. We use our framework to create three benchmarks: Wiki-FACTOR, News-FACTOR and Expert-FACTOR. We show that: (i) our benchmark scores increase with model size and improve when the LM is augmented with retrieval; (ii) benchmark score and perplexity do not always agree on model ranking; (iii) when perplexity and benchmark score disagree, the latter better reflects factuality in open-ended generation, as measured by human annotators. We make our data and code publicly available in https://github.com/AI21Labs/factor.
1 Introduction
FACTOR addresses the need to evaluate factuality over controlled, potentially rare facts rather than only facts sampled from a model. It transforms factual corpora into contrastive benchmarks and shows that benchmark accuracy captures factuality differently from perplexity.
- Motivation: Existing sampling-based factuality methods may overrepresent high-likelihood common facts and underrepresent rare or domain-specific facts, leaving controlled factuality evaluation unavailable.Perplexity is also not directly linked to factuality because it is affected by many linguistic phenomena.
- FACTOR: FACTOR transforms factual corpora into benchmarks contrasting each true statement with similar false variations, measuring whether models rank the true completion highest.The framework uses InstructGPT to generate false variations and defines accuracy as the percentage of examples where the factual completion receives the highest likelihood.
- Benchmarks: The framework produces Wiki-FACTOR, News-FACTOR, and Expert-FACTOR benchmarks spanning encyclopedic knowledge, news, and domain-specific question answering.The evaluated models range from 110M to 66B parameters across GPT-2, GPT-Neo, and OPT families.
- Results: 58% Wiki-FACTOR, 68% News-FACTOR, and 55% Expert-FACTOR were the highest reported scores, while retrieval augmentation consistently improved FACTOR scores.These results indicate that the benchmarks remained challenging for the largest evaluated models.
- Results: FACTOR accuracy and perplexity can rank models differently, and manual annotation of 1,200 completions supports FACTOR accuracy as predictive of factuality in open-ended generation.The OPT-66B model is cited as having higher perplexity but better Wiki-FACTOR accuracy than GPT-J-6B.
2 Related Work
Prior factuality evaluations commonly assess consistency with references, probe internal knowledge with slot filling, or score sampled generations. FACTOR instead evaluates factuality in natural open-ended generation while controlling the evaluated facts.
- Existing evaluation: Reference-based methods evaluate whether generated text is supported by a source, while slot-filling methods probe factual knowledge in simplified synthetic settings.These approaches address factuality through downstream consistency or masked factual completion tasks.
- FACTOR: FACTOR evaluates natural open-ended text generation, where context may be needed to reason about a statement that is not explicitly present in the context.This setting differs from summarization, where the factual statement is supported by the source text.
- Existing evaluation: Sampling-based approaches score facts generated by the model but lack control over which facts are evaluated and favor common facts.This creates a contrast with FACTOR’s corpus-based selection of factual statements.
- Contrastive evaluation: FACTOR follows contrastive evaluation by perturbing factual claims into similar positive and negative examples, using automatically generated non-factual alternatives.Contrastive factuality datasets have also used human annotation, rules, or model-based heuristics to obtain negative examples.
3 The FACTOR Evaluation Approach
FACTOR constructs a four-choice factuality task from a corpus: one original completion is contrasted with three generated contradictions. A pipeline filters and selects fluent, contradictory, diverse alternatives before assembling domain benchmarks.
- Evaluation task: Each FACTOR example contains a prefix and four completions, with the corpus continuation as the sole factual answer and three non-factual alternatives.Models are evaluated by whether the factual completion receives the highest mean log-probability.
- Benchmark properties: The benchmark seeks contradictions that are factually inconsistent, similar to the factual completion, and difficult to reject for fluency or style alone.Logical and self-consistent alternatives make factual discrimination harder.
- Error types: Non-factual completions cover predicate, entity, circumstance, coreference, and link errors adapted from an existing factual-inconsistency typology.These error types vary how alternatives conflict with the prefix and factual completion.
- Dataset construction: 52FACTOR generation follows four stages: selecting a prefix and completion, generating contradictions, filtering candidates, and selecting the final alternatives.The pipeline selects a sentence completion, prompts InstructGPT for contradictions, filters non-contradictory or non-fluent outputs, and chooses one completion per error type when possible.
- Dataset construction: Automatic filtering uses NLI to identify contradictions and language-model scores to remove completions substantially less fluent than the original.The NLI threshold is manually validated, and the fluency margin is set to τLM = 0.2.
- Datasets: The resulting Wiki-FACTOR, News-FACTOR, and Expert-FACTOR datasets contain 2994, 1036, and 236 examples, respectively.Manual validation found that nearly all generated completions contradicted the originals, were fluent, and were self-consistent; error types were diverse.
4 Experimental Setup
The experiments evaluate open-source GPT-2, GPT-Neo, and OPT models on three FACTOR datasets, with matched sequence lengths and held-out benchmark corpora. Retrieval-augmented variants use in-context document retrieval without additional training.
- Evaluation data: Experiments cover Wiki-FACTOR, News-FACTOR, and Expert-FACTOR, with perplexity measured on 300 Wikipedia articles containing approximately 367K tokens.The three benchmarks represent Wikipedia, news, and expert question answering domains.
- Models: The study evaluates four GPT-2, five GPT-Neo, and eight OPT models spanning 110M to 66B parameters.Sequence length is capped at 1024 tokens for direct comparison.
- Data separation: The benchmark corpora were excluded from model training, with News-FACTOR and Expert-FACTOR also created after the models’ data cutoff dates.Wiki-FACTOR uses a validation split that was not part of the examined models’ training sets.
- Retrieval augmentation: Retrieval-augmented evaluation uses IC-RALM, prepending retrieved documents to the language-model input without additional training or specialized architecture.A retriever is called every s tokens using a query formed from the last ℓ tokens.
5 Factual Knowledge Evaluation Results
FACTOR accuracy generally rises with model size, but scores vary across model families and domains. Retrieval augmentation improves FACTOR accuracy, while perplexity and factuality-based rankings can diverge.
- 5.1 Model Scaling and Benchmark Results: 58.0% for GPT-NeoX-20B, 68.1% for OPT-66B, and 55.9% for OPT-30B are the largest reported scores on Wiki-FACTOR, News-FACTOR, and Expert-FACTOR, respectively.These ceilings indicate that the benchmarks remain challenging.
- 5.1 Model Scaling and Benchmark Results: Larger models generally outperform smaller models within the same model family, while different model families obtain different FACTOR scores.The evaluation covers GPT-2, GPT-Neo, and OPT models across Wiki-FACTOR, News-FACTOR, and Expert-FACTOR.
- 5.2 Retrieval Augmentation: IC-RALM consistently improves factual accuracy for GPT-Neo and OPT models on Wiki-FACTOR.The figure compares the base models with their IC-RALM variants.
- 5.2 Retrieval Augmentation: IC-RALM’s relative gains in FACTOR accuracy are more moderate than its reported relative perplexity gains on WikiText-103.The comparison motivates further analysis of the relationship between the two metrics.
- 5.3 FACTOR Accuracy and Perplexity: FACTOR accuracy offers a complementary view of model performance because perplexity improvements do not necessarily imply better factuality.The two metrics can produce substantially different pairwise model rankings.
6 Factuality in Open-Ended Generation
The authors test whether FACTOR accuracy predicts factuality in open-ended generation using manually annotated generations. Models that score better on FACTOR produce more true claims, including when perplexity is similar.
- Method: The study conditions models on a shared prefix and completion context, then manually labels generated text as true, false, or neutral relative to a minimal factual claim.The minimal claim is identified from the factual and non-factual completion pair.
- Experimental Design: The evaluation compares GPT-J 6B and OPT-66B on examples where each model is respectively right or wrong about the factual-versus-non-factual completion preference.Three generations were sampled for 100 examples from each subset and model, producing 1200 generations before filtering.
- Method: Overall FACTOR accuracy is computed by weighting true-generation rates from the right and wrong subsets according to their Wiki-FACTOR distribution.Neutral generations are excluded from the factuality analysis.
- Results: OPT-66B generated true claims 4.6% of the time when it was wrong on FACTOR pairs, compared with 46.6% when it was right.This links FACTOR pairwise correctness with factuality in free-text generation.
- Results: OPT-66B generated true claims 38.8% of the time versus 24.8% for GPT-J 6B, matching their Wiki-FACTOR ranking despite similar Wiki perplexity.The result supports FACTOR as a better proxy for domain-specific factuality than perplexity in this comparison.
7 Discussion
FACTOR transforms a factual corpus into a contrastive benchmark of factual statements and non-factual variants. The experiments show that it ranks models differently from perplexity and better aligns with factuality in open-ended generation.
- Contribution: FACTOR compares an LM’s likelihood of factual claims with similar non-factual variants to measure its propensity to generate factual information.The benchmark is constructed automatically from a corpus of factual statements.
- Motivation: Perplexity can reflect factors other than factual knowledge because it does not contrast factual statements with false alternatives.FACTOR focuses the language-modeling task on factuality through a contrastive approach.
- Findings: FACTOR ranks models differently than perplexity and is more aligned with factuality in open-ended generation.The authors identify negative examples as important for evaluating factuality.
- Relation to Prior Work: FACTOR’s controlled evaluation and generation fact-checking are complementary approaches that together provide a more holistic assessment of LM factuality.The distinction is between evaluating a controlled set of facts and checking facts in model generations.
Limitations
The authors identify limitations in automated benchmark quality and in extending FACTOR construction across domains. Domain-general corpora may require more intricate methods to identify factual sentences.
- Benchmark Construction: Automated FACTOR benchmarks may not fully satisfy the benchmark requirements defined by the authors.This limitation is analyzed in the paper’s benchmark-construction discussion.
- Domain Generalization: Constructing FACTOR benchmarks for different domains can be challenging because identifying factual sentences is harder outside knowledge-intensive corpora.In knowledge-intensive domains, nearly every sentence may contain factual information; general cases require a more intricate approach.
Ethics Statement
FACTOR enables efficient automatic factuality evaluation, but human evaluation remains necessary for models deployed in sensitive settings.
- FACTOR supports automatic evaluation of language models’ factuality and methods for mitigating factuality risks.
- Automatic evaluations may be insufficient in sensitive deployment settings, where human evaluation is required.
A Technical Details of FACTOR Data Pipeline
The FACTOR pipeline generates and filters factual and incorrect completions using targeted edits, then compares benchmark scores with perplexity and human factuality judgments.
- The pipeline identifies entity, predicate, coreference, circumstance, and link edits using part-of-speech tags, named entities, and temporal or causal link words.
- Candidate completions are filtered for contradiction and fluency using NLI and language-model thresholds selected from manual annotations.
- Figure 5 compares Wiki-FACTOR scores with perplexity across GPT-Neo, OPT, and GPT2 model families.
- Most generations were neutral for both OPT-66B and GPT-J 6B, highlighting a limitation of sampled-based assessment of model factual knowledge.
- FACTOR-accuracy often exceeded the 25% random baseline, suggesting models could predict some unseen facts.
- Higher News-FACTOR scores may reflect related training documents or prefixes that imply unseen facts, but analyzing these cases is left for future work.
C Dataset Licenses
The supplementary materials document corpus licenses, prompt templates for generating five error types, and manual factuality annotation results for two language models.
- Table 5 documents the licenses associated with each corpus used in the paper.
- Prompt templates generate entity, circumstance, coreference, predicate, and link errors, with context omitted for link prompts.
- The prompts ask models to plan local edits before generating candidate completions, while link prompts provide only the completion.
- Table 6 reports manual factuality annotations for OPT-66B and GPT-J 6B across right and wrong subsets, including weighted averages.