PaperlayerView exact revision on arXiv

Source-linked AI summary

PlotMachines: Outline-Conditioned Generation with Dynamic Plot State Tracking

Hannah Rashkin, Asli Celikyilmaz, Yejin Choi, Jianfeng Gao

arXiv:2004.14967v2cs.CL

TL;DR

Outline-conditioned story generation asks models to turn rough descriptions of key characters and events into coherent narratives while preserving the outline’s plot elements. PlotMachines uses dynamic plot memory and discourse structure to generate such stories, and experiments across three datasets show that large language models struggle with outline-consistent coherence while plot tracking improves narrative tightness. The paper’s outline representation is intentionally limited to automatically extractable events and phrases.

  • Problem

    Outline-conditioned generation addresses how to produce coherent, multi-paragraph narratives from rough, unordered descriptions of key characters and events.

  • Method

    PlotMachines recurrently generates paragraphs while updating dynamic memory for outline points and latent topics, augmented with high-level discourse structure.

  • Results

    Across three fiction and non-fiction datasets, GPT-2 and GROVER struggle with outline-consistent coherence, while PlotMachines achieves tighter narratives than competitive baselines.

  • Takeaways & Limitations

    Dynamic plot state tracking and discourse features are important for composing narratives that better use outlines and maintain narrative flow and ordering.

  • Takeaways & Limitations

    The paper limits plot points to events and phrases that can be automatically extracted and does not release the model trained on the NYTimes data because of fake-news concerns.

Abstract

from arXiv · show

We propose the task of outline-conditioned story generation: given an outline as a set of phrases that describe key characters and events to appear in a story, the task is to generate a coherent narrative that is consistent with the provided outline. This task is challenging as the input only provides a rough sketch of the plot, and thus, models need to generate a story by interweaving the key points provided in the outline. This requires the model to keep track of the dynamic states of the latent plot, conditioning on the input outline while generating the full story. We present PlotMachines, a neural narrative model that learns to transform an outline into a coherent story by tracking the dynamic plot states. In addition, we enrich PlotMachines with high-level discourse structure so that the model can learn different writing styles corresponding to different parts of the narrative. Comprehensive experiments over three fiction and non-fiction datasets demonstrate that large-scale language models, such as GPT-2 and Grover, despite their impressive generation performance, are not sufficient in generating coherent narratives for the given outline, and dynamic plot state tracking is important for composing narratives with tighter, more consistent plots.

1 Introduction

Outline-conditioned story generation asks models to turn rough descriptions of key characters and events into coherent narratives while dynamically tracking how plot elements are interwoven. PlotMachines addresses this challenge with dynamic plot-state memory and discourse structure, and experiments show limitations in large language models and benefits from plot tracking.

  • Task and challenge: Outline-conditioned story generation converts phrases describing key characters and events into a coherent narrative consistent with the outline.The input provides only rough plot elements, so the model must flesh out how they intertwine across the story.
  • Task and challenge: Plot elements can reappear non-linearly, requiring the model to track how outline phrases have been used while writing.Their interweaving must adapt to the composed text while remaining faithful to the outline and overall narrative structure.
  • Approach: PlotMachines transforms outlines into multi-paragraph stories using dynamic memory blocks that track implicit plot states from the outline and generated story.The model also uses high-level discourse labels to represent different parts of the narrative.
  • Contributions: The paper introduces a task formulation, three datasets, PlotMachines, and empirical comparisons with competitive baselines.The datasets cover fiction and non-fiction domains and pair multi-paragraph narratives with automatically constructed outlines.
  • Findings: Large-scale language models such as GPT-2 and GROVER still struggle to generate narratives coherent with input outlines despite impressive generation performance.The experiments motivate dynamic plot-state tracking for composing tighter, more consistent plots.

2 Outline-Conditioned Generation

The paper defines outlines as flexible, unordered collections of concrete plot elements that guide long, multi-paragraph story generation. It constructs datasets from fiction and news corpora while limiting plot points to automatically extractable events and phrases.

  • Task definition: The task takes a plot outline as input and produces a long, multi-paragraph story guided by controllable story elements.It is designed to investigate how generation models plan long narratives around specified plot points.
  • Scope: The work limits plot points to events and phrases because these can be automatically extracted, while leaving alternative event-based planning methods for future work.This is an explicit scope boundary on the outline representation.
  • Outline definition: Plot outlines are unordered lists containing an arbitrary number of multiword plot elements that must appear somewhere in the narrative.Unlike abstract writing prompts, outlines provide concrete points to integrate into the generated story.
  • Task challenges: The task requires appropriate discourse flow and natural outline use, including delaying some points until later narrative stages.For example, a protagonist’s death may be more appropriately introduced near the end.
  • Datasets: Three datasets pair existing multi-paragraph story or news narratives with automatically constructed input outlines.The collection includes Wikiplots, WritingPrompts, and NYTimes, spanning fictional and non-fictional domains.
  • Dataset construction: The outlines are created with RAKE, a domain-independent keyword extraction algorithm based on word frequency and co-occurrence statistics.Overlapping n-gram key points are filtered during extraction.

3 PLOTMACHINES

PLOTMACHINES generates multi-paragraph stories from outlines by combining transformer generation with dynamic memory, preceding context, and discourse representations. The model recurrently updates plot-state memory while generating each paragraph and uses discourse information to structure the narrative.

  • 3 PLOTMACHINES: PLOTMACHINES is an end-to-end transformer built on GPT that recurrently generates paragraphs from an outline.At each step, it conditions generation on the outline, discourse representation, preceding context, and previous memory.
  • 3.1 Outline Representation: The outline is tokenized with delimiters for each plot point and reused as input when generating every paragraph.Plot points are separated by kw tokens and the sequence ends with endkw; the outline is truncated to a maximum length.
  • 3.2 Discourse Representation: Discourse representations mark paragraphs as introduction, body, or conclusion to model stylistic differences across the narrative.The implementation uses special tokens for these discourse roles and assumes the first paragraph is introductory, the last is concluding, and the remainder are body paragraphs.
  • 3.3 Preceding Context Representation: The preceding-context representation is the average of static GPT output embeddings for words in the previous paragraph and initializes the transformer input.During decoding, the model uses its own generated paragraphs to compute subsequent context representations.
  • 3.4 Memory Representation: Memory combines outline-point vectors K with a latent topic matrix D to track mentioned plot elements and semantic content across the story.K is initialized from outline-token embeddings, while D is randomly initialized; memory updates use the preceding paragraph representation and a learned gating mechanism.
  • 3.4 Memory Representation: Transformer blocks use parallel self-attention and memory attention, allowing generated text to attend over the evolving memory matrix.Training uses end-to-end paragraph cross-entropy with gold preceding paragraphs, whereas decoding updates memory from the model’s own predictions.

4 Experiments

Experiments compare PLOTMACHINES with conditional-generation baselines, large pretrained language models, and ablated variants using automatic metrics and human judgments. Across these evaluations, dynamic memory and discourse representations support stronger outline use, narrative flow, and plot consistency.

  • Experimental setup: PLOTMACHINES is evaluated against Fusion, P&W-Static, GROVER, and GPT-based baselines, alongside ablations removing memory, context, or discourse representations.The experiments use automatic metrics, human judgments, and example generations across competitive baselines and model variants.
  • Automatic metrics: PLOTMACHINES achieves comparable or higher ROUGE on all three datasets and improves over GROVER despite using fewer parameters than GROVER.Both GPT- and GPT-2-based PLOTMACHINES variants improve over GROVER, whose model has significantly more parameters than the GPT-based PLOTMACHINES model.
  • Ablations: Removing discourse representations lowers ROUGE, while adding preceding-paragraph context slightly improves performance over GPT-2.The full model’s memory component produces large ROUGE improvements over the no-memory variant, underscoring the importance of plot-state tracking.
  • Human evaluations: PLOTMACHINES is preferred over Fusion and GPT in outline-use judgments, and humans rate it higher for narrative flow and ordering than competing models.PLOTMACHINES is judged to use outlines more naturally, while GROVER is judged to use the outline more but performs worse on narrative flow and ordering.
  • Human evaluations: PLOTMACHINES receives the highest ratings for paragraph transitions and relevance while remaining less repetitive than GROVER or Fusion.GPT is least repetitive but has low transition and relevance scores, whereas PLOTMACHINES combines strong transitions and relevance with lower repetition.
  • Outline usage: GROVER over-repeats outline points, Fusion omits portions of the outline, and PLOTMACHINES more closely balances coverage with the gold stories’ distribution.Gold stories mention each outline point in around one paragraph on average, while PLOTMACHINES is generally more inclusive and follows a beginning-middle-ending structure.

5 Related Work

Prior work addresses state tracking and controllable generation, while outline-conditioned generation offers a more flexible and structured grounding scheme. PlotMachines extends these ideas by tracking plot state and applying discourse structure.

  • State Tracking: State-tracking methods update memory or entity representations during dialogue and other generation tasks.PlotMachines adapts these techniques to story generation by updating plot state after each paragraph.
  • Controllable Story Generation: Prior controllable story-generation work includes plot-controllable, plan-driven, constrained, keyword-conditioned, and creative-generation approaches.
  • Controllable Story Generation: Outline-conditioned generation provides more flexibility than fine-grained plans and more structured grounding than coarse approaches.
  • Controllable Story Generation: PlotMachines addresses task-specific challenges through dynamic state tracking and discourse structure.

6 Conclusion

The paper introduces outline-conditioned story generation and PlotMachines, which generates paragraphs using discourse structure and dynamic plot memory. Quantitative analysis finds that PlotMachines composes tighter narratives from outlines than competitive baselines.

  • Task: The paper introduces outline-conditioned story generation from outlines representing key plot elements.
  • Datasets: Three existing datasets are adapted with plot outlines as inputs for long-story generation.
  • Model: PlotMachines generates paragraphs with high-level discourse structure and dynamic plot memory tracking both the outline and story.
  • Results: Quantitative analysis shows that PlotMachines composes tighter narratives based on outlines than competitive baselines.

A.1 Examples from Training Datasets

The appendix presents full stories corresponding to excerpts from the training-dataset examples. These examples cover the Wikiplots, WritingPrompts, and New York Times datasets.

  • Full Stories: Full stories are presented in Tables 6–8.
  • Full Stories: The full stories correspond to excerpts shown in the main-text dataset subsection.

A.2 Human Evaluation Details

The appendix documents human-evaluation questionnaires, payment rates, full-story comparisons, and paragraph-excerpt ratings. It also explains how one response was reversed for consistent figure interpretation.

  • Questionnaires: Human raters evaluated narrative flow, ordering, outline usage, and full-story comparisons through questionnaires.
  • Questionnaires: Raters were paid $1.00 per HIT for excerpt questionnaires and $2.00 per HIT for full-story questionnaires.
  • Questionnaire Processing: Answers to the second full-story question were reversed in Figure 3 so higher values represented less repetition.
  • Full-Story Study: The full-story study compared PLOTMACHINES with baselines across 20 stories and found a general preference trend for PLOTMACHINES except against GROVER on outline utilization.
  • Excerpt Ratings: Paragraph-excerpt ratings measured repetitiveness, transitions, and relevance, with results reported for Fusion, GPT, GROVER, and PLOTMACHINES.

A.3 Qualitative Examples

Qualitative examples compare GROVER, PLOTMACHINES, and Fusion outputs using annotated outline coverage. They show GROVER restarting stories mid-document, while PLOTMACHINES produces more structured narrative openings and conclusions.

  • Examples annotate outline points by highlighting mentions and distinguishing points mentioned partially or multiple times.The examples use red text for outline points and separate markings for mention frequency.
  • The qualitative comparisons include GROVER, PLOTMACHINES, and Fusion outputs across multiple example documents.
  • GROVER often finishes one story and starts another partway through the document.The examples identify this behavior as a notable discontinuity in GROVER outputs.
  • This mid-document restarting may help explain GROVER’s repeated outline points and judgments of greater repetitiveness and lower consistent relevance.
  • PLOTMACHINES examples demonstrate learned discourse structure through scene-setting introductions and definitive closing actions.Table 11 presents example introduction and conclusion paragraphs from Wikiplots validation outputs.
Loading 2004.14967v2…