Source-linked AI summary
Do Massively Pretrained Language Models Make Better Storytellers?
Abigail See, Aneesh Pappu, Rohun Saxena, Akhila Yerukola, Christopher D. Manning
TL;DR
The paper addresses the limited evidence about whether massive pretraining improves open-ended language generation. It compares GPT2-117 with a specialized story-generation model across automatic analyses and finds stronger context conditioning but persistent repetition and under-diversity under likelihood-maximizing decoding.
Problem
The quality and characteristics of text generated by massively pretrained language models had not been examined through a detailed study.
Method
The study compares GPT2-117 with a state-of-the-art specialized story-generation model across automatic metrics and decoding settings.
Results
GPT2-117 conditions more strongly on context, is more sensitive to event ordering, and uses more contentful words, but matches the Fusion Model’s repetition and under-diversity under low-k decoding.
Takeaways & Limitations
The findings indicate that large-scale pretraining can improve several aspects of story generation without resolving generic, repetitive, and under-diverse output under likelihood-maximizing decoding.
Takeaways & Limitations
Because the study excludes human evaluation, it measures diversity more than overall quality and cannot reliably assess coherence, reasoning, or world knowledge.
Abstract
from arXiv · showhide
Large neural language models trained on massive amounts of text have emerged as a formidable strategy for Natural Language Understanding tasks. However, the strength of these models as Natural Language Generators is less clear. Though anecdotal evidence suggests that these models generate better quality text, there has been no detailed study characterizing their generation abilities. In this work, we compare the performance of an extensively pretrained model, OpenAI GPT2-117 (Radford et al., 2019), to a state-of-the-art neural story generation model (Fan et al., 2018). By evaluating the generated text across a wide variety of automatic metrics, we characterize the ways in which pretrained models do, and do not, make better storytellers. We find that although GPT2-117 conditions more strongly on context, is more sensitive to ordering of events, and uses more unusual words, it is just as likely to produce repetitive and under-diverse text when using likelihood-maximizing decoding algorithms.
1 Introduction
The paper examines whether massive pretraining improves open-ended story generation, comparing GPT2-117 with a specialized neural story-generation model. It studies how pretraining and decoding affect generated text.
- Anecdotally impressive generated text had not been examined through a detailed formal study.
- The study compares GPT2-117 with a state-of-the-art specialized story-generation architecture.
- The analysis asks how open-domain pretraining changes generated text, where it makes no difference, and whether task-specific architecture is necessary.
- The evaluation varies decoding algorithms and produces an objectively defined, richly detailed, reproducible analysis of generated text.
- The authors publicly release generated stories and evaluation code for browsing, independent evaluation, and reproduction.
2 Background
The study uses the WritingPrompts dataset and compares GPT2-117 with the Fusion Model while examining likelihood-based and top-k decoding.
- WritingPrompts contains 303,358 human-written prompt–story pairs from Reddit’s /r/WritingPrompts forum.Its stories average 734.5 words, and the dataset contains 200 million story words.
- The Fusion Model is a state-of-the-art story-generation architecture designed to model long-range context and improve prompt relevance.It combines multi-scale gated self-attention with a two-stage fusion mechanism.
- GPT2 is a large Transformer language model trained on WebText, while GPT2-117 has 117 million parameters and a 1024-token context size.
- Beam search and greedy decoding maximize likelihood, whereas top-k sampling truncates the vocabulary distribution before sampling.Top-k sampling ranges from greedy decoding at k = 1 toward unbiased sampling as k increases.
3 Experimental Details
The experiments fine-tune GPT2-117 and evaluate it against the Fusion Model on filtered WritingPrompts data using matched generation settings and multiple k values.
- WritingPrompts-1024 contains 192,364 training, 11,115 validation, and 10,686 test examples after length filtering.Stories were truncated to 1000 words, and examples exceeding GPT2-117’s 1024 BPE-token context were excluded.
- GPT2-117 is fine-tuned by representing each prompt and story as one delimiter-separated language-modeling sequence.
- 31.544: finetuned GPT2-117’s test word-perplexity, six points lower than the Fusion Model.Word-level normalization enables comparison despite different tokenizations.
- Both models generate 1000 stories from different test prompts using top-k sampling across k values from 1 to vocabulary size.
- Generated stories contain exactly 150 words, and human comparison stories are truncated to 150 words with shorter stories discarded.
4 Story-prompt relatedness
GPT2-117 conditions more strongly on story prompts than the Fusion Model across language-modeling and generation analyses, with semantic similarity generally declining as k increases.
- 80.16%: GPT2-117’s prompt ranking accuracy, compared with 39.8% for the Fusion Model and 10% random chance.The metric measures whether a model assigns a story higher probability under its true prompt than under nine random prompts.
- GPT2-117 generates stories with greater semantic similarity to prompts than the Fusion Model, and similarity decreases as k increases.
- Across n = 1, 2, 3, GPT2-117 copies more prompt n-grams than the Fusion Model, especially when k < 100.
- GPT2-117 uses more prompt named entities than the Fusion Model, although both models use fewer than humans when k is below vocabulary size.
- GPT2-117 conditions on prompts much more strongly than the Fusion Model in both language modeling and generation settings.This result holds despite the Fusion Model’s fusion technique being designed to improve story-prompt relevance.
5 Coherence
Both models detect incoherent sentence order well, with GPT2-117 more sensitive to event ordering and to context position. However, both models are less sensitive to swaps near the beginning of a story.
- Coherence: The coherence evaluation uses the first 15 sentences of each test story and compares the original with 14 versions formed by swapping adjacent sentence pairs.
- Coherence: GPT2-117 reduces the error rate to 2.17%, compared with 3.44% for the Fusion Model, a 36.92% error reduction in detecting swapped sentences.Random guessing yields a 93.33% error rate.
- Coherence: Both models are less sensitive to out-of-order sentences at the beginning of the text than at later positions.GPT2-117 shows this positional pattern more strongly than the Fusion Model.
6 Repetition and rareness
Repetition and lexical diversity are governed primarily by the decoding parameter k rather than by the model’s training scale. GPT2-117 uses somewhat rarer words and is slightly more diverse, but these differences are small compared with the effect of k.
- N-gram repetition: Low-k decoding causes both models to produce substantially less diverse and more repetitive text than human writing.Distinct-n measures the ratio of unique n-grams to total generated n-grams; lower values indicate more within-story repetition.
- N-gram repetition: As k increases, unigram, bigram, and trigram diversity rises toward human levels, with GPT2-117 only slightly above the Fusion Model.The influence of k is much larger than the small difference between the models.
- N-gram repetition: Likelihood-maximizing decoding, including low-k top-k sampling, is identified as a primary cause of repetition.The models can match human repetition rates when sampling from the full, untruncated distribution.
- Rare word usage: GPT2-117 uses rarer words than the Fusion Model at every equal k ≥ 2.Its lower mean log unigram probability indicates greater rare-word usage; the probabilities are calculated over the WritingPrompts training set.
- Conclusion: Training data alone is unlikely to solve repetition and diversity problems because decoding choices have a much larger effect than the model difference.GPT2-117 is only very slightly less repetitive and more rare-word-oriented than the Fusion Model.
7 Syntactic style and complexity
Generated syntax is less diverse and less complex under small-k decoding, but approaches human syntactic patterns as k approaches vocabulary size. This pattern holds across sentence length, part-of-speech distributions, and syntactic diversity, with a remaining proper-noun mismatch.
- Sentence length: Small-k decoding produces shorter sentences than human writing, while sentence length converges toward human levels as k increases.Sentence length is used as an estimate of readability and syntactic complexity.
- Part-of-speech usage: As k approaches vocabulary size, both models closely fit the human part-of-speech distribution, especially GPT2-117.Under low k, a substantially different syntactic distribution emerges.
- Syntactic diversity: Syntactic diversity is very low at small k and reaches human levels as k approaches vocabulary size for both models.The results use distinct-n over part-of-speech n-grams with n = 1, 2, and 3.
- Part-of-speech usage: Both models fail to produce enough proper nouns even when k approaches vocabulary size.This is the stated exception to their otherwise close fit with human part-of-speech distributions.
- Syntactic diversity: Generated text can match human lexical repetition rates yet remain syntactically repetitive, as shown by repeated part-of-speech patterns.This distinction means n-gram repetition and syntactic diversity do not necessarily converge simultaneously.
- Conclusion: Overall, syntactic under-diversity is attributed primarily to low k rather than insufficient training data.As k approaches vocabulary size, generated syntax more closely matches human syntactic patterns.
8 The element of surprise
Likelihood-maximizing decoding makes generated stories increasingly self-confident and repetitive, whereas human text continues to include lower-probability words. Increasing top-k slows this over-confidence, with GPT2-117 showing a distinct high-k probability pattern.
- Both models develop self-reinforcing repetitive text when k is small, while human text does not show a strong rising probability trend.
- Teacher-forced human text shows only a slight probability increase as stories progress, likely because additional context improves prediction accuracy.
- Top-k generation increases token probabilities more rapidly than teacher forcing, especially for smaller k.
- As k increases, both models assign lower probability to generated stories; at full sampling, GPT2-117’s stories are more surprising to itself than human stories.
9 Concreteness
GPT2-117 generates more concrete words than the Fusion Model at the same k, while both models approach human concreteness as k increases. Low-k output combines unusually concrete nouns with unusually abstract verbs.
- GPT2-117 tends to generate more concrete words than the Fusion Model for the same k.
- Both models’ concreteness approaches approximately human levels as k increases.
- At small k, generated nouns are much more concrete than human nouns, while generated verbs are much less concrete than human verbs.
10 Conclusions
GPT2-117 outperforms the Fusion Model on context conditioning, event-order sensitivity, and contentfulness, but low-k decoding leaves both models repetitive and under-diverse. Higher-k output matches human text on many measured metrics, yet still lacks coherence, reasoning, and world knowledge, and the study’s conclusions are limited by its model and evaluation scope.
- The effect of massive pretraining: GPT2-117 conditions more strongly on context, responds more sensitively to event ordering, and uses more rare, concrete words and named entities than the Fusion Model.
- The effect of massive pretraining: GPT2-117’s advantages show that a general-purpose architecture can sometimes outperform a complex task-specific architecture with sufficient pretraining data.
- The effect of k: At small k, both models are equally lexically under-diverse, syntactically under-complex, and repetitive, with increasing over-confidence.
- The effect of k: These generation problems are primarily associated with likelihood-maximizing decoding algorithms rather than insufficient training data.
- The effect of k: Low-k output copies more from prompts, shifts toward verbs and pronouns over nouns and adjectives, and uses fewer syntactic patterns than human text.
- The effect of k: As k approaches vocabulary size, generated text matches human text on most measured metrics but can still lack commonsense reasoning, world knowledge, and multi-sentence coherence.
- Limitations of this study: The study uses only the smallest GPT2 model and does not include human evaluation, emphasizing diversity more than overall text quality.
Appendix
Across decoding settings, GPT2-117 generally shows stronger prompt reuse and slightly greater lexical and syntactic diversity than the Fusion Model, while diversity depends primarily on k. Increasing k also changes sentence length, word concreteness, and probability patterns, but high-k text can become surreal despite fluency.
- Prompt conditioning: GPT2-117 copies more prompt n-grams and uses a larger proportion of prompt named entities than the Fusion Model.Both models generally use fewer named entities than human text when k is below vocabulary size.
- Generation characteristics: Sentence length increases with k for both models, while k = 1 produces a spike caused by long repeating sequences without sentence-ending tokens.For equal k, GPT2-117 generally produces more concrete nouns and verbs than the Fusion Model; noun concreteness decreases and verb concreteness increases as k rises.
- Lexical diversity: Lexical diversity increases with k, with GPT2-117 reaching human levels at k = 2000 for unigrams, k = 800 for bigrams, and k = 600 for trigrams.GPT2-117 is slightly more lexically diverse than the Fusion Model at equal k, but k is the primary determining factor.
- Syntactic diversity: Syntactic diversity also increases with k, with GPT2-117 reaching human levels at k = 6000 for unigrams, k = 9000 for bigrams, and k = 6000 for trigrams.GPT2-117 is slightly more syntactically diverse than the Fusion Model at equal k.
- Word and POS usage: GPT2-117 uses slightly more rare words and fewer stopwords than the Fusion Model, but both models approach human rareness levels only when k is near vocabulary size.The same high-k trend accompanies closer matching to human part-of-speech distributions, including more Numerals and Proper Nouns.
- Generation characteristics: The Fusion Model converges toward the probability it assigns to human-written stories as k approaches vocabulary size, whereas GPT2-117 converges to a lower probability.High-k generations can remain fluent and readable while becoming surreal and bizarre.