Source-linked AI summary
Evaluating the Factual Consistency of Large Language Models Through News Summarization
Derek Tam, Anisha Mascarenhas, Shiyue Zhang, Sarah Kwan, Mohit Bansal, Colin Raffel
TL;DR
LLMs can hallucinate, motivating a benchmark for testing whether they prefer summaries factually supported by news documents. FIB pairs verified consistent summaries with annotated inconsistent ones and evaluates model scoring preferences. LLMs generally prefer consistent summaries, but often favor inconsistent summaries when those summaries occur verbatim in the document; length-normalized PMI is the most effective scoring choice in the reported evaluation.
Problem
Factual inconsistency is widespread in natural language generation, but evidence about factual consistency in large language models is needed under the newer prompting-based paradigm.
Method
FIB compares model scores for manually verified consistent summaries and manually annotated inconsistent summaries, using accuracy across paired news-document examples.
Results
LLMs generally score factually consistent summaries higher, but almost always score extracted inconsistent summaries higher; length-normalized PMI most effectively detects inconsistency.
Takeaways & Limitations
FIB provides a benchmark for probing LLM factual consistency and identifies extractive inconsistency as a failure case for current models.
Takeaways & Limitations
FIB measures factual consistency only for news summarization, so generalization to scientific articles or question answering is unclear.
Abstract
from arXiv · showhide
While large language models (LLMs) have proven to be effective on a large variety of tasks, they are also known to hallucinate information. To measure whether an LLM prefers factually consistent continuations of its input, we propose a new benchmark called FIB(Factual Inconsistency Benchmark) that focuses on the task of summarization. Specifically, our benchmark involves comparing the scores an LLM assigns to a factually consistent versus a factually inconsistent summary for an input news article. For factually consistent summaries, we use human-written reference summaries that we manually verify as factually consistent. To generate summaries that are factually inconsistent, we generate summaries from a suite of summarization models that we have manually annotated as factually inconsistent. A model's factual consistency is then measured according to its accuracy, i.e.\ the proportion of documents where it assigns a higher score to the factually consistent summary. To validate the usefulness of FIB, we evaluate 23 large language models ranging from 1B to 176B parameters from six different model families including BLOOM and OPT. We find that existing LLMs generally assign a higher score to factually consistent summaries than to factually inconsistent summaries. However, if the factually inconsistent summaries occur verbatim in the document, then LLMs assign a higher score to these factually inconsistent summaries than factually consistent summaries. We validate design choices in our benchmark including the scoring method and source of distractor summaries. Our code and benchmark data can be found at https://github.com/r-three/fib.
1 Introduction
FIB evaluates whether large language models prefer factually consistent summaries over factually inconsistent ones, addressing factual inconsistency in LLM-generated text. The benchmark supports evaluation across 23 models from six families.
- Factual inconsistency is widespread in natural language generation, and summarization models can generate content that contradicts source documents.
- FIB measures whether a model assigns a higher probability score to a factually consistent summary than to a factually inconsistent summary.Accuracy is the proportion of document-summary pairs where the consistent summary receives the higher score.
- The benchmark contains over 3,500 manually annotated summary pairs drawn from the XSum dataset.
- The evaluation covers 23 LLMs from six model families, including BLOOM, OPT, GPT, and T0, ranging from 1B to 176B parameters.
- FIB contributes a benchmark and annotated summaries for probing LLM factual consistency and evaluates models across multiple families.
2 Related Work
Prior work introduced labeled datasets and metrics for evaluating factual consistency, while FIB evaluates the factual consistency of language models themselves through conditional continuations.
- FRANK and GO FIGURE provide model-generated summaries with factuality labels for meta-evaluation or metric training.
- Reference summaries are not always factually consistent, so FIB manually verifies references before pairing them with inconsistent summaries.
- Related datasets address text simplification or checking claims against knowledge bases, whereas FIB focuses on factual consistency of conditional model continuations.
- Existing factuality metrics use entailment or question-generation and answering approaches to compare summaries with source documents.
3 FIB: Factual Inconsistency Benchmark
FIB pairs document summaries labeled for factual consistency and evaluates whether models score the consistent summary higher. Its construction combines verified reference summaries, inconsistent model outputs, and length-normalized PMI scoring.
- Each FIB example pairs a document with one factually consistent summary and one factually inconsistent summary.A consistent summary must be inferable solely from the document, even if its extra information is true.
- FIB compares extractive CNN/DM summaries with abstractive XSum summaries.Extractive summaries occur verbatim in documents, whereas abstractive summaries do not.
- Dataset Construction: Gold summaries are created by manually labeling reference summaries and minimally editing inconsistent ones until they are document-consistent.Edited summaries were approximately 90% of XSum references and 30% of CNN/DM references.
- Dataset Construction: Factually inconsistent summaries are retained from outputs of 15 extractive CNN/DM models and 7 generative XSum models after annotation.
- Dataset Construction: The benchmark contains 3,124 XSum pairs across 500 documents and 457 CNN/DM pairs across 96 documents.
- Scoring Function: Length-normalized PMI addresses both short-summary bias and high summary-only likelihood in log-likelihood scoring.The normalization averages token-level scores.
4 Experiments
The experiments evaluate 23 LLMs across six model families and test how benchmark construction choices affect factual-consistency preferences. Models generally favor factually consistent summaries, but extractive summaries and distractor-generation methods substantially influence results.
- Main Results: Models generally prefer Gold summaries over factually inconsistent model-generated summaries, although accuracy remains far below 100%.
- Main Results: Performance generally increases slightly with scale within a model family, while zero-shot models perform remarkably similarly across families.T0 is an exception: T0-11B underperforms T0-3B.
- Main Results: On CNN/DM, models rarely prefer factually consistent reference summaries over factually inconsistent model-extracted summaries, but they prefer factually consistent extracted summaries when both alternatives are extractive.This pattern suggests a preference for text copied verbatim from the input regardless of factual consistency.
- Alternative Summaries: On XSum, models are nearly at chance when distinguishing factually consistent model-generated summaries from Gold summaries, whereas CNN/DM models consistently prefer factually consistent extracted summaries.The results indicate that preference for factually consistent model-generated summaries depends on whether summaries are extractive.
- Alternative Summaries: MFMA-generated alternatives are rarely preferred over Gold summaries, with accuracy between 85% and 100% across models.FactCC produces somewhat plausible inconsistent summaries for zero-shot decoder-only LLMs, but performs less effectively across other architectures and training schemes.
5 Analysis
The analysis examines when models prefer factually inconsistent model-generated summaries over Gold summaries, including cases involving extrinsic hallucinations and model-specific scoring behavior.
- BLOOM examples show factually inconsistent model-generated summaries receiving higher scores than Gold summaries.The examples involve extrinsic hallucinations that add false information rather than merely omitting document-supported information.
- Models used to generate summaries often score their own factually inconsistent summaries highly, producing the lowest accuracies along the heatmap diagonal.This indicates poor performance when an evaluated model scores summaries generated by the same model.
- In most cases, Gold summaries are preferred less than 50% of the time against factually inconsistent model-generated summaries.The analysis reports that summarization models tend to assign higher scores to their own factually inconsistent outputs.
- BLOOM and T5-large are exceptions because other models almost always assign low scores to their factually inconsistent summaries.The paper leaves the explanation for this model-specific trend to future work.
6 Conclusion and Takeaways
The paper introduces FIB as a benchmark for evaluating factual consistency in language models and applies it to 23 large language models. It finds that models generally prefer factually consistent summaries, but often prefer factually inconsistent extracted summaries, while length-normalized PMI performs best for detecting inconsistency.
- FIB benchmarks factual consistency in language models, and the study evaluates 23 large language models.The benchmark and evaluation are presented as the paper’s main contributions.
- LLMs generally assign higher scores to factually consistent summaries than to factually inconsistent summaries, except for factually inconsistent extracted summaries.The paper identifies extracted summaries as the main stated exception to the general preference.
- Length-normalized PMI enables models to most effectively detect factually inconsistent summaries.
7 Limitations
FIB measures factual consistency only for news summarization, so the extent to which its results generalize to other domains and tasks remains unclear.
- FIB is limited to evaluating factual consistency for news summarization.The paper specifically identifies scientific articles and question answering as settings where generalization is unclear.
- The results may not generalize to domains such as scientific articles or tasks such as question answering.
B Sample Edited Summaries
The appendix presents examples of original factually inconsistent XSum reference summaries alongside edited versions made factually consistent.
- Table 2 shows original factually inconsistent reference summaries and edited factually consistent summaries from XSum.
C Sample Model-Extracted factually inconsistent
This section presents examples of model-extracted factually inconsistent summaries from CNN/DM. The examples are shown in table 3.
- The examples are presented as part of the paper’s sample of model-extracted factually inconsistent summaries.
- Table 3 presents examples of documents with model-extracted factually inconsistent summaries.
- The examples are drawn from CNN/DM.
D Models Used to Generate Summaries
The paper uses multiple extractive and abstractive summarization models to generate summaries for XSum and CNN/DM. It applies fixed decoding and prompting procedures, then evaluates model performance across scoring functions and alternative-summary sources.
- Generation procedure: Greedy decoding is used for all models with a maximum generation length of 50 tokens.
- Models: The summary-generation models include extractive systems such as Textrank, PacSum, and MatchSumm, alongside neural models including BART, PEGASUS, and T5.
- Prompting: The evaluation uses three prompt templates for all models, replacing [input] with the document.
- Evaluation: Performance is reported across XSum and CNN/DM, multiple scoring functions, and alternative summaries generated by different methods and models.
K Accuracies from Models Used to Generate Summaries
This section reports model accuracies across datasets, scoring functions, and alternative-summary sources. The accompanying tables organize results for various alternatives, including MFMA, FactCC, factually consistent model-generated summaries, and FIB.
- Same-model alternatives: Table 40 reports XSum performance when the same models generate the factually inconsistent summaries.