PaperlayerView exact revision on arXiv

Source-linked AI summary

Controllable Neural Story Plot Generation via Reward Shaping

Pradyumna Tambwekar, Murtaza Dhuliawala, Lara J. Martin, Animesh Mehta, Brent Harrison, Mark O. Riedl

arXiv:1809.10736v4cs.CL

TL;DR

Neural plot generators often lack control over a specified goal and can lose coherence during generation. This paper adapts a pretrained language model with corpus-derived reward shaping, producing intermediate rewards that guide plots toward target endings. The resulting system reaches specified goals more consistently and yields stories judged to have more plausible event ordering than a baseline.

  • Problem

    Neural language-model approaches to plot generation generally lack user guidance toward a specific goal and can produce stories with weak progression and coherence.

    Source: Introduction · p. 1

  • Method

    The paper analyzes a story corpus to construct dense intermediate rewards and backpropagates them through a pretrained language model using policy gradients.

    Source: Reinforcement Learning for Plot… · p. 2, Reward Shaping · p. 3

  • Results

    The DRL-clustered model reached specified goals 93.82% of the time on average, compared with 37.72% for Seq2Seq and 19.935% for DRL-unrestricted.

    Source: Results and Discussion · p. 4

  • Takeaways & Limitations

    Reward shaping enables controlled neural plot generation with more plausible event ordering and perceived single-plot coherence than the baseline.

    Source: Results and Discussion · pp. 5–6, Conclusions · p. 6

Abstract

from arXiv · show

Language-modeling--based approaches to story plot generation attempt to construct a plot by sampling from a language model (LM) to predict the next character, word, or sentence to add to the story. LM techniques lack the ability to receive guidance from the user to achieve a specific goal, resulting in stories that don't have a clear sense of progression and lack coherence. We present a reward-shaping technique that analyzes a story corpus and produces intermediate rewards that are backpropagated into a pre-trained LM in order to guide the model towards a given goal. Automated evaluations show our technique can create a model that generates story plots which consistently achieve a specified goal. Human-subject studies show that the generated stories have more plausible event ordering than baseline plot generation techniques.

Introduction

Automated plot generation seeks coherent sequences of story events, but neural language-model approaches generally lack user control over a specified goal and lose coherence over longer generations. The paper addresses this with reward shaping that guides a neural language model toward desired plot outcomes.

  • Prior planning-based plot generators can use goals or outcome states to support coherence, but require extensive domain-knowledge engineering.Source: Introduction · p. 1
  • Neural plot generators often cannot accept user guidance toward a specific ending, leaving generated stories without a clear aim.Language-model systems generate each token, word, or sentence by sampling, which limits goal-directed progression.Source: Introduction · p. 1
  • RNN-based generators tend to lose coherence as generation continues because they use only a relatively small part of the token history in practice.Source: Introduction · p. 1
  • The paper introduces reward shaping that reinforces neural-language-model weights to guide plot points toward a given goal.The technique is evaluated against standard language modeling using computational metrics and human judgments.Source: Introduction · p. 1
  • Human evaluations find improved plausible event ordering and plot coherence over a baseline, while computational evaluation also favors the proposed technique.Source: Introduction · p. 1

Background and Related Work

Story-generation research moved from manually engineered symbolic and case-based systems toward neural language models trained on story corpora. Reinforcement learning offers a route to combine neural generation with goal specification and coherence-oriented rewards.

  • Early symbolic planners and case-based systems generated stories mainly in predetermined, well-defined domains using manually engineered knowledge.Symbolic planners could provide long-term causal coherence, but domain restrictions complicated judging algorithmic robustness.Source: Background and Related Work · pp. 1–2
  • Recurrent neural networks learned from large story corpora to predict the next letter, word, or sentence for story and plot generation.Source: Background and Related Work · p. 2
  • Unlike Story Cloze prediction, this work generates an entire story to fit a specified ending; other controllable systems provide themes, topics, or titles.Source: Background and Related Work · p. 2
  • Reinforcement learning addresses neural text-generation coherence issues while allowing a goal to be specified through policy optimization.REINFORCE samples from the current policy and backpropagates received rewards through the policy-model weights.Source: Background and Related Work · p. 2
  • Because plot generation requires longer-horizon coherence than task-based dialogue, the paper uses reward shaping to seek rewards over longer horizons.Source: Background and Related Work · p. 2

Reinforcement Learning for Plot Generation

The system treats plot generation as planning toward a goal event, specializing a pretrained event-based language model with policy gradients and corpus-derived dense rewards. Reward shaping combines distance and frequency information, while verb clustering discourages premature jumps to the target.

Reinforcement Learning for Plot Generation

Initial Language Model

  • Events are encoded as tuples containing WordNet-based subject, object, and modifier representations plus a VerbNet verb class, enabling sequential event generation.Source: Initial Language Model · p. 2

Policy Gradient Descent

  • The policy-gradient update gives larger likelihood shifts to highly rewarding next events, with reward based on their distance from the goal event.Source: Policy Gradient Descent · p. 3

Reward Shaping

  • Reward shaping estimates intermediate usefulness from how close verbs occur to the target and how frequently they precede it in corpus stories.Source: Reward Shaping · p. 3

Final Reward

  • The final reward multiplies normalized distance and frequency components, while verb clustering restricts output verbs to the next cluster to discourage jumping directly to the target.Source: Final Reward · p. 3, Verb Clustering · p. 3

Automated Experiments

The experiments compare baseline and reward-shaped models on goal achievement, perplexity, and story length using a romance-focused subset of the CMU movie summary corpus. DRL-clustered most reliably reaches target verbs, while DRL models substantially reduce perplexity and clustering balances faster goal achievement against premature endings.

Automated Experiments

  • The study measures target-verb achievement, perplexity, and average story length against a baseline event2event model, terminating generation at the goal, end-of-story token, or 15 lines.Goal achievement is the percentage of generated stories ending in admire or marry, with length and perplexity computed for model comparisons.Source: Automated Experiments · p. 3, Experimental Setup · p. 4

Corpus Preparation

  • Experiments use a soap-opera-like cluster from the CMU movie summary corpus, with admire-31.2 and marry-36.2 as target verbs and 90%/10% training/testing data.The corpus was clustered into 100 genres before selecting the romance-focused subset and eventifying its stories.Source: Corpus Preparation · pp. 3–4

Model Training

  • DRL-clustered and DRL-unrestricted continue training from Seq2Seq weights with policy gradients, while only DRL-clustered restricts next-event verbs using clustering.Both DRL variants receive 200 additional training epochs; unrestricted training removes the vocabulary restriction.Source: Model Training · p. 4

Results and Discussion

  • 93.82% of stories reached the target goals with DRL-clustered, versus 37.72% for Seq2Seq and 19.935% for DRL-unrestricted.The testing corpus averaged 22.47%, showing the targets were rare; Table 1 reports the automated comparison.Source: Results and Discussion · p. 4, Table 1 · p. 4
  • DRL models reduced perplexity from the Seq2Seq baseline’s 48.06 to 7.61 and 5.73 for admire, and 7.05 and 9.78 for marry.DRL-unrestricted sometimes achieved lower perplexity but sacrificed goal achievement because its rewards used subsequent verbs rather than verb clusters.Source: Results and Discussion · p. 4
  • DRL-clustered generated slightly shorter stories than the baseline while avoiding both immediate goal jumps and excessively long, rambling plots.Seq2Seq and DRL-unrestricted produced lengths closer to the corpus, whereas clustering encouraged quicker but non-immediate goal attainment.Source: Results and Discussion · p. 4

Human Evaluation

The human evaluation used controlled translations of generated event sequences and showed that DRL stories were judged more plausibly ordered and more likely to follow a single plot than baseline stories, without significant disadvantages on other dimensions.

Human Evaluation

  • The evaluation protocol had translators convert event sequences into concise, faithful natural-language sentences so judges assessed event content rather than sentence-generation quality.Pairs translated plots independently and reached consensus on final versions.Source: Human Evaluation · pp. 4–5, Corpus Creation · p. 5

Corpus Creation

  • The study compared five DRL-clustered stories, five Seq2Seq stories, and three corpus stories, using the same start events and restricting generated stories to 5–10 events.Twenty-six people were trained to translate the events before the human evaluation.Source: Corpus Creation · p. 5

Experimental Setup

  • The study recruited 175 Amazon Mechanical Turk participants, who rated translated plots on five-point Likert scales; data from 25 participants were removed after they failed to answer short-answer questions.Source: Experimental Setup · p. 5

Results and Discussion

  • Repeated-measures ANOVA with Tukey HSD was used to test differences across the three story conditions, with average scores and significance reported in Figure 2.The analysis assumed normally distributed data and nonsignificantly different variances.Source: Results and Discussion · p. 5, Figure 2 · p. 6
  • DRL stories were rated as having more plausible event orderings and as more likely to follow a single plot than Seq2Seq stories (p < 0.05).The DRL was also perceived as having more local causality, but that difference was not statistically significant.Source: Results and Discussion · pp. 5–6
  • For all other evaluated dimensions, DRL stories were not significantly different from baseline stories, indicating no significant degradation beyond goal-directed plot control.Interesting language and repetition were nonsignificant across all three conditions, while grammar differed between corpus and generated stories because generated events more often lacked objects or modifiers.Source: Results and Discussion · pp. 5–6

Conclusions

The reward-shaping approach gives neural plot generation explicit goal control while preserving corpus-like behavior. It reaches pre-specified goals about 93% of the time and improves human judgments of event ordering and coherent single-plot structure.

Conclusions

  • The reward-shaping policy reaches a pre-specified goal about 93% of the time while generating stories probabilistically comparable to the training corpus.The paper also reports improved perplexity against the testing corpus, while noting that corpus similarity is not the most important plot-generation metric because novel plots may be good.Source: Conclusions · p. 6
  • Human evaluation found that reward shaping significantly improves perceived event ordering and the likelihood that generated events form a single coherent plot.The paper presents this as control over neural plot generation through a goal specifying how the plot should end.Source: Conclusions · p. 6
Loading 1809.10736v4…