Source-linked AI summary
How Decoding Strategies Affect the Verifiability of Generated Text
Luca Massarelli, Fabio Petroni, Aleksandra Piktus, Myle Ott, Tim Rocktäschel, Vassilis Plachouras, Fabrizio Silvestri, Sebastian Riedel
TL;DR
The paper addresses whether fluent language-model generations convey factual world knowledge and how decoding affects that property. It evaluates generated sentences with Wikipedia-based automatic and human fact checking across decoding strategies, finding a tradeoff between verifiability and repetitiveness and proposing DELAYEDBS as a hybrid strategy.
Problem
It is unclear whether language models can produce long, fluent, grammatical sequences that reflect factual world knowledge rather than merely fluent text.
Method
The paper generates sentences from Wikipedia prompts under varied model and decoding configurations, then evaluates them with automatic and human fact checkers using verifiability and repetitiveness metrics.
Results
Sampling produces less repetitive but less verifiable text, whereas likelihood-based decoding is more verifiable but highly repetitive; DELAYEDBS combines these properties more favorably.
Takeaways & Limitations
Decoding strategy strongly shapes the balance between factual verifiability and repetition, making hybrid decoding a supported way to improve that tradeoff.
Takeaways & Limitations
The study treats each generated sentence as a single claim, although some generations contain multiple independently assessable claims.
Abstract
from arXiv · showhide
Recent progress in pre-trained language models led to systems that are able to generate text of an increasingly high quality. While several works have investigated the fluency and grammatical correctness of such models, it is still unclear to which extent the generated text is consistent with factual world knowledge. Here, we go beyond fluency and also investigate the verifiability of text generated by state-of-the-art pre-trained language models. A generated sentence is verifiable if it can be corroborated or disproved by Wikipedia, and we find that the verifiability of generated text strongly depends on the decoding strategy. In particular, we discover a tradeoff between factuality (i.e., the ability of generating Wikipedia corroborated text) and repetitiveness. While decoding strategies such as top-k and nucleus sampling lead to less repetitive generations, they also produce less verifiable text. Based on these finding, we introduce a simple and effective decoding strategy which, in comparison to previously used decoding strategies, produces less repetitive and more verifiable text.
1 Introduction
The paper asks whether fluent, grammatical language-model generations also express factual world knowledge, especially as adoption of generated text increases. It evaluates verifiability and repetitiveness across decoding strategies and introduces DELAYEDBS to address their tradeoff.
- 1 Introduction: The study evaluates generated text for repetitiveness and verifiability, using Wikipedia-based fact checking with automatic and human assessment.The methodology generates sentences from Wikipedia prompts, classifies them as supported, refuted, or unverifiable, and defines metrics from these outcomes.
- 1 Introduction: The experiments focus on large autoregressive transformer models, which remain mostly grammatical and fluent while varying substantially in repetitiveness.The analysis therefore combines repetitiveness with verifiability rather than treating grammaticality as the primary differentiator.
- 1 Introduction: Sampling methods such as top-k and nucleus produce less repetitive text but fewer supported and more refuted statements than likelihood-based decoding.The reported findings identify a tradeoff between naturalness, repetitiveness, and factual verifiability.
- 1 Introduction: DELAYEDBS injects stochasticity at the beginning of a sentence before switching to beam search, producing less repetitive and more verifiable text.The strategy is designed to combine sampling’s diversity with likelihood-based decoding’s verifiability.
- 1 Introduction: The paper’s contributions include a measurement methodology, a broad decoding-strategy comparison, a new decoding strategy, and annotation-based validation.The validation assesses both the automatic fact-checking system and the resulting findings.
2 Related Work
Prior work has improved controllability, diversity, fluency, and detection of generated text, but has provided limited insight into factual verifiability under different decoding strategies. This paper instead studies that relationship directly.
- 2 Related Work: CTRL enables conditioning generated text on style, content, domains, entities, and relationships, but does not assess the verifiability of its generations.This motivates separating controllability from factual verification.
- 2 Related Work: Top-k and nucleus sampling were proposed to diversify generation, while prior work showed that decoding strategy strongly affects fluency and repetitiveness.The present paper extends this decoding-strategy discussion to verifiability.
- 2 Related Work: Unlikelihood training reduces probabilities for tokens appearing in undesirable contexts, but testing alternative training objectives is outside this paper’s scope.The approach is described as orthogonal to decoding strategy.
- 2 Related Work: Other work detects machine-generated text or generates malicious and fake-news claims, whereas this paper investigates verifiability in existing language models across decoding strategies.Its focus is the relationship between decoding choices and factual status, not direct optimization for factual or malicious news.
- 2 Related Work: Because repetitiveness is identified as a major generation problem, the paper includes it as a central evaluation dimension alongside verifiability.This emphasis follows prior generation studies.
3 Background
Autoregressive language models select tokens from a context-dependent vocabulary distribution, and decoding strategies determine how those selections form complete sequences. The paper contrasts stochastic sampling, likelihood-based search, repetition blocking, and their hybrid combination.
- 3 Background: At each position, an autoregressive model updates its context with the selected token and repeatedly chooses the next token from the resulting distribution.This defines the left-to-right open-ended generation process.
- 3 Background: Sampling-based methods introduce stochastic decisions to increase output diversity and avoid repetitions.Top-k samples from the k highest-probability tokens, whereas nucleus sampling uses the smallest token set whose cumulative probability exceeds p.
- 3 Background: Beam search approximately maximizes whole-sequence likelihood by retaining β prefixes and extending them iteratively; greedy decoding is beam search with β=1.Likelihood-based strategies navigate a large sequence space using practical search heuristics.
- 3 Background: Group-diverse and sibling-diverse beam search modify beam search with diversity penalties to preserve more varied hypotheses.The former penalizes similarity between groups, while the latter penalizes candidate rank relative to its source hypothesis.
- 3 Background: N-gram blocking prevents already observed n-grams from being generated again as a simple repetition-reduction mechanism.The method targets repeated surface forms during decoding.
- 3 Background: DELAYEDBS samples the first L tokens and then completes the sentence with beam search, allowing L to tune the balance between sampling and likelihood-based behavior.Smaller L approaches beam search, while larger L approaches sampling.
4 Evaluating Verifiability
The evaluation treats generation and fact checking as black boxes, classifies generated sentences by factual status, and measures both supported content and repetition. Its metrics distinguish overall support from support among verified sentences and include unique variants to expose repeated claims.
- 4 Evaluating Verifiability: The fact checker labels each generated sentence SUPPORTED, REFUTED, or NOT ENOUGH INFO, with the first two labels counted as verified.Generated completions are segmented into sentences and evaluated over the first k sentences.
- 4 Evaluating Verifiability: The setup combines a text generator with a fact checker, typically implemented using information retrieval and stance detection components.Both components are treated as black boxes for the evaluation.
- 4 Evaluating Verifiability: The evaluation computes metrics over generated sentence sequences produced from a set of prefixes P.The prefixes serve as the data source for the verifiability probe.
- 4 Evaluating Verifiability: Supports Per Generation (SPG) measures the fraction of supported sentences among all generated sentences, while Supports Per Verified (SPV) measures the fraction among verified sentences.SPG acts like recall and SPV like precision.
- 4 Evaluating Verifiability: Unique-support metrics address the possibility that repeated supported sentences can inflate SPG and SPV.Equivalent sentences share a factuality label and the same supporting evidence; only the first occurrence counts as unique.
5 Methodology
The methodology evaluates generated text across decoding strategies using Wikipedia-based prefixes, language models, repetitiveness metrics, and automated fact checking. It combines controlled generation and sentence processing to classify outputs as supported, refuted, or unverifiable.
- Wikipedia titles and descriptions form non-overlapping validation and test prefixes, with 2018 used for parameter sweeps and 2017 for evaluation.
- The study trains Transformer language models at 124M, 354M, and 1.4B parameters on Wikipedia, books, web text, and news corpora.
- It compares top-k, top-p, greedy, beam, diverse-beam, and Delayed Beam Search, with optional n-gram blocking and three seeded generations per prefix.
- Generated texts are segmented into sentences, coreference and entity heuristics are applied, and sentences longer than 50 tokens are treated as unverifiable.
- An off-the-shelf FEVER-trained fact checker retrieves relevant Wikipedia pages and sentences before recognizing textual entailment, while repetitiveness is measured with distinct 4-grams and 4-gram proportions.
6 Results
Sampling strategies generate more diverse but less verifiable text, whereas likelihood-based strategies improve verifiability at the cost of repetition. DELAYEDBS offers a less repetitive alternative with verifiability comparable to likelihood-based methods, while human evaluation confirms the central trade-off.
- Sampling methods achieve greater diversity but lower verifiability, while beam search attains the highest SPG and SPV with substantially more repetition.Sampling also produces more refuted sentences, especially as k or p increases; likelihood-based decoding follows higher-probability paths that better preserve verifiability.
- DELAYEDBS nearly doubles distinct 4-grams while maintaining precision and recall for diverse verifiability comparable to likelihood-based strategies.Sampling only the first high-uncertainty token and then completing with beam search is sufficient to trigger this behavior.
- Blocking 20-grams with beam search gives the best diverse-verifiability performance, whereas DELAYEDBS remains less repetitive.N-gram blocking prevents repeated phrases while preserving a high-likelihood solution, but smaller blocking sizes force less likely continuations and reduce verifiability.
- CCNEWS leads in USPG across decoding strategies and also leads in USPV for top-k and DELAYEDBS, while WIKIPEDIA has low USPG despite the lowest perplexity.The ablation compares training corpora and transformer sizes using USPV versus USPG.
- Sentence average log probability is positively correlated with verifiability, consistent with better performance from maximum-likelihood strategies.The correlation analysis also considers entities, parsing, conjunctions, prefix perplexity, tf-idf overlap, and sentence length.
- Human evaluation confirms that sampling is less repetitive but less supported than beam search, while DELAYEDBS is less repetitive than BS and more supported than top-k.Human annotations also show that n-gram blocking does not fully remove beam-search repetition, and the fact checker has limited recall.
7 Conclusion and Discussion
The paper systematically evaluates verifiability across decoding strategies using automatic fact-checking and human annotations, finding a tradeoff between diversity and factual support. It proposes a hybrid strategy that combines less repetitive sampling with more verifiable likelihood-based generation, while identifying dataset and claim-definition boundaries.
- The study finds that sampling strategies generate less repetitive but less verifiable text than likelihood-based strategies, motivating a hybrid decoding approach.The hybrid strategy combines the non-repetitive nature of sampling with the verifiable generation of likelihood-based methods.
- The analysis evaluates generated text from large autoregressive language models with both an off-the-shelf fact-checker and human annotations.
- The evaluation focuses on the most viewed Wikipedia pages from 2017 and 2018, while random Wikipedia pages produced lower SPG and USPG recall values.The authors suggest investigating relationships between training corpora and generation in future work.
- Each sentence is treated as a single claim, leaving generations containing multiple independently assessable claims for future study.This choice was made to keep the experimental setting clean and avoid noise from an automatic claim extractor.
8 Appendix
The appendix details measurements of generated-text relevance, structural properties, grammatical correctness, and a DELAYEDBS delay-length ablation. The ablation finds that longer delays yield nearly as many supported sentences as top-k decoding.
- Experimental setup: The parameter sweep selects, for each decoding strategy, the configuration with the highest USPG score on the small transformer validation setup.The appendix also reports examples generated by different strategies and analyzes multiple generated-text dimensions.
- Generated-text analysis: spaCy supplies tokenization, sentence splitting, named-entity recognition, and dependency parsing for the structural statistics.The dependency parser is used to count conjunctions in each sentence's dependency tree.
- Ablation study: The delay-length ablation compares supported-sentence counts across delay lengths against a top-k validation-set reference.Figure 5 plots delay length against the number of distinct supported sentences, with horizontal lines showing top-k performance.
- Generated-text analysis: Generated text is evaluated using distinct-sentence counts, named entities, tokens, conjunctions, Wikipedia-page relevance, and grammatical correctness.Relevance is measured with tf-idf, while grammatical correctness uses the link-gram parser success rate.
- Ablation study: With larger delay lengths, DELAYEDBS produces a number of supported sentences very close to top-k decoding.The ablation uses validation-set prefixes, top-k sampling, and varying delay lengths with DELAYEDBS.
- Generated examples: Table 5 contrasts large-transformer generations from different strategies using a cherry-picked repetitive-beam-search example and a random example.Fact-checker-supported sentences are highlighted in green and refuted sentences in red.