Plot Writing From Pre-Trained Language Models

1 Introduction

• Long-form story generation often produces incohesive, repetitive, or content-poor narratives because language models lack global planning, discourse coherence, and common sense.Individual sentences may appear fluent and logical while failing to fit together as a whole.Source: §1 · p. 1
• Existing story-generation systems use prompts, keywords, semantic frames, or summaries as explicit content plans to improve event sequences.These approaches reintroduce planning into a field otherwise dominated by end-to-end learning.Source: §1 · p. 1
• Natural-language-incompatible content plans cannot generally be understood by PLMs without fine-tuning on parallel data, while supervised plans may reproduce predictable common-sense routines rather than interesting stories.The paper identifies lack of natural-language compatibility and reduced surprise as limitations of prior planning approaches.Source: §1 · pp. 1–2
• SCRATCHPLOT prompts an off-the-shelf PLM to plan location, characters, genre, and theme, then generates the story and ranks candidate endings.The framework progressively composes the story without fine-tuning, using the planned attributes as conditioning information.Source: Figure 1 · p. 1, §1 · p. 2
• SCRATCHPLOT yields better stories than fine-tuned parallel-data baselines and standard-prompting PLMs in automatic and human evaluations.The experiments were conducted on English stories, although the authors state that the approach could extend to other languages with suitable PLMs and task descriptions.Source: §1 · p. 2

• SCRATCHPLOT samples location, characters, genre, and theme sequentially, while modeling dependencies such as genre influencing theme.Source: §2 · p. 2
• Self-debiasing encourages generated geographical units and names to differ across task descriptions, while other plot elements are generated without it.Source: §2 · p. 2
• The method fuses sampled plot elements into one task description and generates the story, using male and female character values for the character element.Source: §2 · p. 2
• Because GPT-2 lacks an end-of-sequence token, the story body is generated to a fixed length and truncated at the last complete sentence.Source: §2 · p. 2
• A separate task description generates a short ending from the story body and stops at the first closing quotation mark.Source: §2 · p. 3
• SCRATCHPLOT ranks sampled story-ending pairs with BERT next-sentence prediction and conditioned perplexity to detect coherence and ending quality.Source: §2 · p. 3

RQ1: Ending ranking

• PPL selected more favorable story endings than NSP or random selection in human evaluation, so the authors used PPL for subsequent experiments.On the Story Cloze Test, NSP and PPL performed comparably with DSSM, while PPL outperformed NSP without requiring in-domain training.Source: §3 · pp. 4–5

RQ2: SCRATCHPLOT comparison

• Plan-and-write generated logically sequenced but less diverse stories with poorer plots and characters, whereas ProGen often produced illogical stories from keyword-like plans.The comparison suggests keyword lists work better for short, simple stories than for nuanced cohesive stories.Source: §3 · pp. 5–6
• SCRATCHPLOT performed best on naturalness, interestingness, and cohesiveness compared with the baselines, with especially pronounced gains in interestingness.The human evaluation used scores from 1 to 5 and compared multiple story-generation systems.Source: §3 · p. 6, Table 5 · p. 6

RQ3: Unsupervised content planning

• SCRATCHPLOT outperformed the baseline without content planning on all evaluated story aspects, showing benefits from content plans even when generated plans contain noise.Some generated themes were ambiguous or nonsensical, although locations and person names were generated with high quality.Source: §3 · p. 6
• The baseline without explicit content planning produced less diverse stories and more within-story repetitions, while SCRATCHPLOT showed natural progression instead of late-story looping.The qualitative looping behavior is described as neural text degeneration.Source: §3 · p. 6, Table 7 · p. 6

4.1 Story Content Planning

• Earlier systems used prompts, keywords, semantic frames, summaries, predicate-argument tuples, or latent vectors as intermediate story representations.Source: §4.1 · pp. 6, 8
• ProGen progressively refines keyword representations through multiple fine-tuned seq2seq stages before generating the final story.Source: §4.1 · p. 8
• Heterogeneous plot elements such as cast, location, and genre serve as ScratchPlot’s content plan, which is sampled from PLMs without fine-tuning.Source: §4.1 · p. 8

4.2 Story Ending Generation

• Story-ending work ranges from commonsense-story datasets and external knowledge retrieval to discourse tokens marking introduction, body, and conclusion.Source: §4.2 · p. 8
• ScratchPlot uses natural-language instructions rather than trainable embeddings to signal story endings while relying on PLMs without explicit reasoning.Source: §4.2 · p. 8
• Conditional perplexity was demonstrated to be a more reliable story-ending metric than BERT next-sentence prediction in RQ1.Source: §4.2 · p. 8

• SCRATCHPLOT performs unsupervised content planning for story generation using only pretrained language models.Source: §5 · p. 8
• SCRATCHPLOT achieves strong results compared with supervised baselines fine-tuned on large parallel corpora and a PLM without access to content plans.Source: §5 · p. 8

• SCRATCHPLOT is intended for creative text composition rather than factual reporting about real persons or anecdotes.Source: Ethical Considerations · p. 9
• PLMs may partially recall training data, and the paper notes that the originality of generated stories cannot feasibly be measured.Source: Ethical Considerations · p. 9
• The system sometimes generates famous real-person names and may produce inappropriate or disrespectful stories for particular populations.Source: Ethical Considerations · p. 9
• Manual curation or automatic content filtering can be deployed to mitigate inappropriate generations.Source: Ethical Considerations · p. 9

A Full List of Task Descriptions

• Task descriptions generate each plot element, with the story body and ending sharing a single task description.Source: A Full List of Task Descriptions · p. 11, Table 9 · p. 12

B Post-Processing

• Post-processing removes trailing punctuation from phrase outputs and filters continuations that repeat words from task descriptions.Source: B Post-Processing · p. 11
• First- and second-person pronouns are filtered because they often produce generic or opinionated plot elements.Source: B Post-Processing · p. 11
• Generated stories are filtered when they contain fewer than two essential elements: a male name, female name, or location.Source: B Post-Processing · pp. 11–12
• The appendix summarizes which post-processing rules apply to each output type.Source: B Post-Processing · p. 12

• NLTK word tokenization supports diversity and repetition calculations, while self-BLEU averages sentence_bleu scores across dataset examples.Each example serves as the reference in one self-BLEU round.Source: C Additional Experimental Setups · p. 12
• Generation takes roughly 1 minute per story because the system produces multiple bodies and endings before scoring and selecting the best candidate.Source: C Additional Experimental Setups · p. 12
• Inference uses an NVIDIA Quadro P5000 GPU with 16GB vRAM, but CPU inference is possible because the method requires no fine-tuning.The authors report comparable quality when using GPT2-medium instead of GPT2-XL under resource constraints.Source: C Additional Experimental Setups · p. 12

• Crowdsource evaluations were conducted on Toloka and covered annotation-task specifications, quality control, and annotation statistics.Source: D Details of Crowdsource Evaluation · p. 12

D.1 Annotation Task Specifications

• Fine-grained evaluation assigns each story a 1-to-5 rating for individual aspects so annotators can focus on one aspect without context switching.Source: §D.1 · p. 12
• Naturalness measures whether a story is fluent, understandable, and linguistically natural, allowing minor grammatical errors that do not affect comprehension.Source: §D.1 · p. 12
• Interestingness asks whether a story would interest readers familiar with its genre, whereas cohesiveness evaluates logical consistency, character continuity, and event sequences.Source: §D.1 · p. 12
• Figures document the annotation interface, instructions, audience filter, and post-processing steps used across the evaluation tasks.Source: §D.1 · p. 13, Figure 4 · p. 13, Table 10 · p. 13
• Story-ending evaluation uses pairwise judgments of relevance, logic, conclusiveness, and thoughtfulness.Source: §D.1 · p. 13

D.2 Quality Control

• Quality control selects fluent English speakers among the top-rated 20% who pass training, then limits workloads and removes likely bots or unreliable workers.Source: §D.2 · p. 13

• Fine-grained evaluation tasks paid $0.05 each, took about 30 seconds on average, and used around 40 crowdworkers per aspect.The reported average earning was $6 per hour.Source: §D.3 · p. 13
• Story-ending evaluations paid $0.1 per task, required about 1 minute 13 seconds on average, and involved 20 crowdworkers.Source: §D.3 · p. 13
• The total budget for all crowdsource evaluations was $300, including worker payments, bonuses, and platform fees.Source: §D.3 · p. 13
• Figures provide examples of interfaces and annotation instructions for interestingness, naturalness, cohesiveness, story endings, and pool statistics.Source: Figure 5 · p. 14, Figure 6 · p. 14, Figure 7 · p. 15