PaperlayerView exact revision on arXiv

Source-linked AI summary

Content Planning for Neural Story Generation with Aristotelian Rescoring

Seraphina Goldfarb-Tarrant, Tuhin Chakrabarty, Ralph Weischedel, Nanyun Peng

arXiv:2009.09870v2cs.CLcs.AI

TL;DR

Neural story generation remains fluent locally but often lacks global structure, cohesion, and prompt relevance. The paper learns plot structures with a plot-generation model and Aristotelian rescoring models, finding that the resulting stories are more relevant and higher quality than unplanned or unprincipled baselines.

  • Problem

    Neural story generators often produce fluent sentences without global structure, cohesion, or sustained relevance to the prompt.

    Source: §1 · p. 1

  • Method

    The system generates plots and revises them with an ensemble of rescoring models implementing Aristotelian story-writing principles.

    Source: §1 · pp. 1–2, §3.1 · p. 3

  • Results

    The Aristotelian plot system produces stories that are more relevant and higher quality than stories generated directly from prompts or using plots without Aristotelian rescoring.

    Source: §5 · p. 8, §8 · p. 9

  • Takeaways & Limitations

    Content planning through an interim plot structure can guide neural story generation toward better relevance and quality.

    Source: §8 · p. 9

    Aristotelian plots improve over naive plots but still leave gaps relative to gold plot structures, and better models for incorporating plots remain needed.

    Source: §3.1 · p. 4, §3.3 · p. 5, §8 · p. 9

Abstract

from arXiv · show

Long-form narrative text generated from large language models manages a fluent impersonation of human writing, but only at the local sentence level, and lacks structure or global cohesion. We posit that many of the problems of story generation can be addressed via high-quality content planning, and present a system that focuses on how to learn good plot structures to guide story generation. We utilize a plot-generation language model along with an ensemble of rescoring models that each implement an aspect of good story-writing as detailed in Aristotle's Poetics. We find that stories written with our more principled plot-structure are both more relevant to a given prompt and higher quality than baselines that do not content plan, or that plan in an unprincipled way.

1 Introduction

Neural story generators often produce fluent sentences without global structure, topic relevance, or cohesion. This paper proposes Aristotelian content planning through learned plot structures and rescoring models to guide generation.

1 Introduction

  • Neural narrative systems can imitate fluent human writing while producing stories whose sentences lack cohesion, content, and sustained relevance to the prompt.Source: §1 · p. 1
  • The authors use Aristotle’s ordering of story elements to prioritize event choice and arrangement, followed by character, relevance, and finally style and diction.Source: §1 · p. 1
  • The system treats plot generation as content planning and microplanning before a language model performs surface realization.Source: §1 · p. 1
  • Aristotelian rescoring selects events, characters, and relevant content through event-choice, event-arrangement, character, and relevance models.Source: §1 · p. 2
  • The paper reports strong experimental results against four baselines for its revision-based Aristotelian content-planning approach.Source: §1 · p. 2

2 Background

Prior work shows that intermediate plot representations can make neural story generation more controllable. This paper follows that approach using SRL-based plots extracted from stories, while presenting its pipeline alongside prior content-planning work.

2 Background

  • Content-planning systems generate an intermediate plot from a prompt and then generate a story conditioned on that plot.Source: §2 · p. 2
  • The approach models p(y,z|x) = p(z|x)p(y|z, x), showing the generated story while retaining the intermediate representation during generation.Source: §2 · p. 2
  • Learning a good interim structure makes story generation more controllable than open-domain generation conditioned only on a prompt.Source: §2 · p. 2
  • Because parallel gold-standard plot-story datasets are unavailable at scale, the authors use silver-standard plots extracted from stories, represented with Semantic Role Labelling, coreference resolution, and compression.Source: §2 · p. 2
  • The paper’s pipeline generates a naive plot, revises it with Aristotelian rescorers, and then generates the story.Source: Figure 1 · p. 3

3 Approach

The approach learns plot structures with Aristotelian rescoring models, then combines them with language-model probabilities to guide story generation. It compares rescorer architectures, trains mixture weights, and selects an ensemble for the final system.

3 Approach

  • The system re-ranks plot-model hypotheses with rescoring models that encode event, character, and relevance principles, using learned score weights.The modified decoding objective combines the original plot model with Aristotelian rescorer scores.Source: §3 · p. 3, §3.1 · p. 3

3.1 Aristotelian Rescoring Models

  • Event rescorers distinguish inter-sentence, intra-sentence, and verb-shuffled plots to learn alternative notions of event choice and arrangement.These perturbations respectively shuffle whole sentences, event tuples within sentences, or verbs while preserving arguments and contexts.Source: §3.1 · pp. 3–4
  • Character and relevance rescorers model contextually appropriate entity continuation and prompt–plot compatibility using true-versus-negative examples.Character examples contrast true and randomly sampled entities, while relevance examples pair prompts with true or randomly selected plots.Source: §3.1 · p. 4

3.2 Rescoring Model Architecture

  • RoBERTa achieves the best performance for each Aristotelian concept, so the final system uses RoBERTa for all rescorers.The evaluated alternatives include n-gram XGBoost, an augmented CNN, and RoBERTa-large.Source: §3.2 · p. 5, Table 2 · p. 5

3.3 Mixture Weight Training and Ablations

  • The authors train mixture weights with validation-set margin ranking and evaluate individual and combined rescorer configurations.The objective compares the gold plot with hypothesized plots, and ablations isolate each rescorer’s contribution.Source: §3.3 · p. 5
  • The five-rescorer ensemble performs best, while Inter-event is the strongest individual rescorer, suggesting complementary information across negative-example designs.The authors select the full ensemble with optimal learned weights for the final Aristotelian Plot System.Source: §3.3 · p. 5

4 Experimental Setup

The experiments use the Writing Prompts dataset to compare Aristotelian plot planning with strong baselines and ablations using automatic and human-centered metrics. Evaluation covers plot structure, story diversity and repetition, relevance, and overall quality.

4 Experimental Setup

  • The Writing Prompts dataset provides hierarchical prompt–story pairs and diverse long-form stories for evaluating plot-based generation.Source: §4 · pp. 5–6

4.1 Baselines

  • The study compares two strong external systems with Prompt to Story and Naive Plot ablations, isolating the effects of direct generation and unrescored plot planning.Source: §4.1 · p. 6

4.2 Metrics

  • The evaluation combines automatic metrics on 1,000 test prompts with human judgments of 110 stories, each rated by three judges.Source: §4.2 · p. 6
  • Plot and story metrics measure vocabulary diversity, entity counts, verb diversity, and intra- and inter-story trigram repetition.Source: §4.2 · p. 6
  • Human evaluation defines Relevance as prompt alignment and sustained topicality, while Overall Quality combines coherence, interestingness, and relevance.Source: §4.2 · p. 7
  • Because Likert ratings can suffer central-tendency bias and cognitive load, pairwise comparisons supplement them for close systems.Source: §4.2 · p. 7

5 Results

The Aristotelian Plot system improves human-judged relevance and overall quality relative to plot-free and naive-plot baselines, while automatic metrics expose important differences hidden by lexical scores.

Stories

Plots

  • The Aristotelian plot moves generated plots closer to gold plots in Vocab:Token ratio, while rescored plots contain slightly fewer entities because they are shorter on average.Source: §5 · p. 7

Stories

  • Unique Verbs is reported for comparability but considered uninformative because it is not normalized by length or token count.Source: §5 · p. 7

Human Evaluation

  • Pairwise human comparisons show Aristotelian Plot superior to Prompt to Story and Naive Plot on both Relevance and Overall Quality.Source: §5 · p. 8, Figure 2 · p. 8
  • Likert ratings favor Aristotelian Plot for relevance but Prompt to Story for Overall Quality, with the three BART systems too close for reliable separation.Source: §5 · p. 8

6 Analysis

The analysis finds that Aristotelian plotting improves relevance and overall quality in selected human evaluations, while plot improvements are not fully realized in story text. It also identifies weaknesses in both evaluation and plot incorporation.

  • 8% of prompts have low user confidence and 8% have high user confidence, which motivates separate analysis of minimally and maximally distinguishable systems.Examples for these prompts are provided in Appendix C.Source: §6 · p. 8
  • The Aristotelian plot system is preferred for Overall Quality in all high-confidence comparisons, while high-confidence Relevance prompts favor it in all but one case.Low-confidence Relevance prompts produce essentially split win-rates, and all systems struggle with stylistic and metacontent prompts.Source: §6 · p. 8
  • High plot-word and plot-verb incorporation shows that the story model uses plots, but current incorporation remains below its possible upper bound.Improving story-model incorporation could allow plot improvements to appear more strongly in surface realization.Source: §6 · p. 9, Table 6 · p. 9

7 Related Work

Related work spans story generation without explicit plots, plot-based generation, and rescoring methods for story aspects. The paper distinguishes its approach by applying principled rescoring directly to plot structures.

  • Prior systems generate stories directly from prompts, often adding convolutional sequence modeling, structured commonsense knowledge, or other control mechanisms.These approaches improve story generation without using the paper’s Aristotelian plot-rescoring framework.Source: §7 · p. 9
  • Plot-based systems represent narrative structure through refinement search, plot graphs, event-to-event generation, keyword-based outlines, semantic-role representations, reinforcement learning, or latent variables.The literature explores multiple ways to create or use interim plot structures before text realization.Source: §7 · p. 9
  • Prior rescoring work uses discriminators to model conversational or collaborative story-writing principles, but does not apply rescoring to plot structures.The paper presents plot-level rescoring based on different story-writing principles as an unaddressed direction.Source: §7 · p. 9

8 Conclusion and Future Work

The paper concludes that Aristotelian rescoring combined with interim plot structures produces stories that are more relevant and higher quality than direct-generation and naively planned baselines. It also identifies remaining gaps in plot quality and plot incorporation.

8 Conclusion and Future Work

  • Aristotelian rescoring with interim plot structures yields more relevant and higher-quality stories than direct prompt generation or plots without Aristotelian rescoring.This is the paper’s central conclusion about principled content planning.Source: §8 · p. 9
  • Aristotelian plots improve over naive plots but remain separated from gold plot structures in quality.The remaining gap motivates additional work on plot-generation principles.Source: §8 · p. 9
  • Better story models for incorporating plots could make plot improvements more effective downstream.The authors identify plot incorporation as a further research direction alongside additional story principles.Source: §8 · p. 9

A.1 Data Quality in Mechanical Turk Studies

The study strengthens Mechanical Turk evaluation quality by combining confidence reporting with manually designed attention checks. Failed checks are filtered and evaluations are rerun until complete datasets are obtained.

  • Turkers report confidence in pairwise story comparisons, enabling analysis of cases where systems are minimally or maximally distinguishable.Workers are told confidence does not affect remuneration.Source: §A.1 · p. 12
  • Manual True-False attention checks are created for every human-evaluation prompt, and 15% of responses fail and are excluded.The evaluations are rerun until a full dataset is collected.Source: §A.1 · p. 12
  • Manually authored checks such as “The protagonist is a child” are more effective than extractive keyword checks.Checks with below-80% pass rates are manually reviewed to ensure they are not overly difficult.Source: §A.1 · p. 12

A.2 Dataset Statistics

The appendix documents dataset splits, generation settings, model configurations, rescoring training, mixture-weight tuning, and event-representation choices. These details cover both plot-to-story generation and the component discriminators used in the system.

A.2 Dataset Statistics

  • The Writing Prompts dataset contains 303358 prompt–story pairs, with stories truncated to 1000 words on average 700-word sequences and rare words mapped to UNK.The data are split into 65%/10%/10% for language-model training, validation, and testing, with 10% for rescoring models and 5% for mixture weights and metrics.Source: §A.2 · p. 12

A.3 Implementation Details

  • Generation uses top-k sampling with k = 5 and softmax temperature 0.7, selecting among the most likely next-token candidates at each timestep.The same decoding procedure is described for generating plots and stories from the models.Source: §A.3 · p. 12

A.3.1 Fine-Tuning BART for Prompt to Plot

  • BART is fine-tuned for 19 epochs for one FTB model setting and 17 epochs for another, with MAX TOKENS = 1024 and UPDATE FREQ = 16.Source: §A.3.1 · p. 12, §A.3.2 · p. 12

A.3.3 Fine-Tuning roBERTa-large for Discrimimators

  • The rescoring discriminators use a roBERTa-large checkpoint and are fine-tuned for 2–10 epochs across relevance, entity, inter-event, intra-event, and intraV-event models.Their training uses max-tokens = 1024 and a fixed learning-rate scheduler, with four RTX 2080 GPUs used for training.Source: §A.3.3 · pp. 12–13

A.4 Hyper-Parameters for Mixture Weight Tuning

  • Mixture weights are tuned with SGD on 10,000 held-out validation samples at learning rate 0.001; models train for three epochs but converge in one epoch.One epoch takes 24 hours on one RTX 2080 GPU.Source: §A.4 · p. 13

B Example Training Data for Rescorers

The appendix provides example rescorer training data and generated outputs, including comparisons between high- and low-confidence examples and baseline systems.

B Example Training Data for Rescorers

  • Table 7 presents excerpted positive and negative training examples for all rescorers, using a campfire-legend prompt and distinct context scopes for relevance and entity scoring.The relevance rescorer receives one sentence of plot context, whereas the entity rescorer uses the entire plot up to a randomly selected entity.Source: B Example Training Data for Rescorers · p. 13

C Example Outputs For High and Low Confidence Scores

  • The appendix includes unedited output examples for high- and low-confidence scores, after removing special separation tokens and detokenizing.Line breaks in Tables 8–10 were added by the model.Source: C Example Outputs For High and Low… · p. 13

D Output from Baseline systems for the first example prompt

E Mechanical Turk Study Materials & Compensation Details

The appendix documents the Mechanical Turk evaluation materials, participant instructions, compensation, survey questions, and example stories and scores used in the study.

E Mechanical Turk Study Materials & Compensation Details

  • Human evaluation covered 110 titles with three judges each, producing 330 x 5 long stories for Likert comparisons and 1320 x 2 long stories for pairwise comparisons.Turkers were paid $12/hr, and the total budget was $2k.Source: E Mechanical Turk Study Materials &… · p. 13
  • The appendix includes survey instructions, optional comments, rejection criteria for missing reasoning, and repository availability for the survey HTML.Participants could complete as many tasks as they wanted.Source: E Mechanical Turk Study Materials &… · pp. 13, 18, 19, 20
  • Table 7 illustrates training data for all rescorers, with relevance using one sentence of plot context and entity scoring using the plot up to a sampled entity.The displayed prompt concerns people gathered around a campfire to tell a legend who sits among them.Source: Table 7 · p. 14
  • High-confidence examples report an overall-quality score of 4.67 out of 5 and a relevance score of 4.86 out of 5, while a low-confidence relevance example scores 2.8 out of 5.These values are reported in Tables 8–10.Source: Table 8 · p. 15, Table 10 · p. 16, Table 9
  • Example evaluation materials include prompts about a new Earth season, poison-transmitting humans, ambiguous heroism, and the end of the universe.The appendix also shows generated stories from Aristotelian, naive-plot, and prompt-to-story systems for one evolution prompt.Source: E Mechanical Turk Study Materials &… · pp. 15–16
  • Pairwise surveys ask which story is more relevant or better, collect confidence ratings and explanations, and include attention checks based on the title.The materials also ask participants to disregard punctuation, spelling, and cutoff endings when evaluating stories.Source: Figure 3 · p. 17, E Mechanical Turk Study Materials &… · pp. 18–19

Related papers

Loading 2009.09870v2…