PaperlayerView exact revision on arXiv

Source-linked AI summary

STORIUM: A Dataset and Evaluation Platform for Machine-in-the-Loop Story Generation

Nader Akoury, Shufan Wang, Josh Whiting, Stephen Hood, Nanyun Peng, Mohit Iyyer

arXiv:2010.01717v1cs.CLcs.HC

TL;DR

Story generation lacks richly contextual datasets and reliable evaluations for long-form creative text. The paper introduces the STORIUM dataset and an author-in-the-loop evaluation platform, finding that edit-based metrics correlate with user judgments and interviews emphasize contextual relevance. The released resources support further research while requiring substantial author effort and remaining tied to the STORIUM community.

  • Problem

    Story generation is difficult to guide and evaluate because existing datasets lack rich context and existing automatic and crowdsourced evaluations are unreliable for long-form creative text.

    Source: §4 · p. 6

  • Method

    The paper builds a 6K-story, 125M-token STORIUM dataset with natural-language annotations and evaluates fine-tuned models through author edits on the STORIUM platform.

    Source: §1 · pp. 1–2

  • Results

    Edit-based automatic metrics correlate with user ratings of generated stories and qualitative interview feedback, while interviews identify contextual relevance as the most important open problem.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    The publicly released dataset and evaluation platform provide resources for more principled research on machine-in-the-loop story generation.

    Source: §1 · p. 2, §7 · p. 9

    Evaluation can take considerable time because it depends on author effort and a relatively small STORIUM community, and the platform is not trivially usable for models trained on other datasets.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Systems for story generation are asked to produce plausible and enjoyable stories given an input context. This task is underspecified, as a vast number of diverse stories can originate from a single input. The large output space makes it difficult to build and evaluate story generation models, as (1) existing datasets lack rich enough contexts to meaningfully guide models, and (2) existing evaluations (both crowdsourced and automatic) are unreliable for assessing long-form creative text. To address these issues, we introduce a dataset and evaluation platform built from STORIUM, an online collaborative storytelling community. Our author-generated dataset contains 6K lengthy stories (125M tokens) with fine-grained natural language annotations (e.g., character goals and attributes) interspersed throughout each narrative, forming a robust source for guiding models. We evaluate language models fine-tuned on our dataset by integrating them onto STORIUM, where real authors can query a model for suggested story continuations and then edit them. Automatic metrics computed over these edits correlate well with both user ratings of generated stories and qualitative feedback from semi-structured user interviews. We release both the STORIUM dataset and evaluation platform to spur more principled research into story generation.

1 Introduction

STORIUM addresses the difficulty of generating and evaluating open-ended long-form stories by combining richly annotated narratives with machine-in-the-loop author evaluation. Its edit-based metric correlates with user judgments, while interviews identify contextual relevance as the key open problem.

  • Existing story datasets and evaluations provide insufficient context or unreliable assessment for long-form creative text.Source: §1 · p. 1
  • STORIUM contributes nearly 6K longform stories with natural-language structural annotations and a platform where authors interact with generated continuations.Annotations include narrative elements such as character goals and abilities.Source: §1 · pp. 1–2
  • The platform records author edits to generated text and uses preserved text to compute the USER metric.Source: §1 · p. 2
  • USER correlates with Likert judgments of linguistic properties such as relevance and coherence.Source: §1 · p. 2
  • User interviews suggest that generating text relevant to the current story context is the most important open problem.Source: §1 · p. 2

2 STORIUM Dataset

STORIUM is a collaborative, game-based storytelling dataset whose cards provide structural annotations for long narratives. The section validates cards’ influence on text and analyzes thematic topics and transitions across stories.

2.1 STORIUM: Gamified Storytelling

  • STORIUM stories are collaboratively written as turn-based games, with scenes, character perspectives, challenges, and cards encoding strengths, weaknesses, items, and goals.Cards played by narrators and characters collectively form structural natural-language annotations that guide the written story.Source: §2.1 · pp. 2–3
  • STORIUM contains 5,743 public stories, 25,092 scenes, 448,264 entries, and 126,041,738 tokens, split into validation and test sets with an 8:1:1 story and token ratio.The dataset is conditioned on 232,596 cards, including 204,698 unique cards.Source: §2.1 · p. 3, Table 1 · p. 3, Table 2 · p. 3
  • 77% of annotated scene entries reference played cards and 80% address the current challenge, indicating that STORIUM’s metadata influences story text.The annotation study covered 235 scene entries; annotators judged card influence and challenge coverage.Source: §2.1 · p. 3
  • Compared with short-story datasets, STORIUM offers considerably longer narratives, averaging 14 narrator prompts and 41 natural-language annotations describing goals, attributes, and key items.Unlike roleplayerguild, STORIUM includes prompts and annotations, while novels provide long-form text without explicit structure.Source: §2.1 · p. 3, Table 1 · p. 3

2.2.1 Topic model specification

  • The topic model omits recurrence because STORIUM contexts do not consistently follow a chronological timeline, and developing a specialized model is outside the paper’s scope.Each entry or challenge is encoded with averaged GloVe embeddings, dictionary-row similarities, softmax topic weights, and contrastive max-margin reconstruction.Source: §2.2.1 · p. 4

2.2.2 Examining topics and their transitions

  • A 50-topic neural model reveals diverse worlds and local narrative transitions, such as weapons topics leading toward technology in Space Adventure and valor in Fantasy Classic.The model uses dictionary learning over entry and challenge text; it analyzes transitions between consecutive entries by the same character across scenes.Source: §2.2 · p. 3, §2.2.2 · p. 4, Table 3 · p. 4

3 Generating Scene Entries

The paper fine-tunes GPT-2 to generate STORIUM scene entries while conditioning on rich narrative context, using segment embeddings and constraint-based token packing. Preliminary user-facing experiments show that including more history and Cassowary packing improves model and user evaluations.

3.1 Model Specification

  • GPT-2 is fine-tuned to generate scene entries conditioned on story context, including challenges, card metadata, character biographies, scene introductions, and preceding entries.The model uses segment embeddings to distinguish context types and autoregressively samples entries at test time.Source: §3 · p. 5, §3.1 · p. 5
  • The model’s compositional segment embeddings let each token condition on multiple metadata segments alongside its token and positional embeddings.This design represents semi-structured scene and card information within the input sequence.Source: §3.1 · p. 5

3.1.1 Context packing

  • STORIUM’s long, heterogeneous contexts exceed GPT-2’s 1024-token limit, making efficient allocation across entries, characters, and challenges necessary.Stories average over 19K tokens, and metadata usage varies substantially between entries.Source: §3.1.1 · pp. 5–6, §3 · p. 5
  • Cassowary packs metadata and story-context tokens by solving prioritized linear constraints, allowing variable bounds and missing-segment collapse instead of fixed-length allocation.The resulting arrangement improves human and automatic evaluation scores.Source: Figure 3 · p. 5, §3.1 · p. 5, §3.1.1 · p. 6

3.2 Hyperparameter Selection

  • Including more story history and applying Cassowary both reduce validation perplexity and increase fluency, coherence, relevance, and likability ratings.These exploratory results motivate selecting the best-performing configuration for full evaluation.Source: §3.2 · p. 6, Table 4 · p. 6
  • User feedback led the final experiments to truncate generated outputs to a maximum of four sentences because longer entries often became unrelated and incoherent.The preliminary study was conducted before the full machine-in-the-loop evaluation.Source: §3.2 · p. 6

4 A Machine-in-the-Loop Evaluation Platform

The evaluation platform places generated scene entries directly in STORIUM authors’ writing workflows, where users edit and rate the outputs. It also introduces USER, an edit-based metric designed to quantify how much generated text authors preserve.

4.1 Evaluation Lifecycle

  • STORIUM authors obtain generated scene entries, edit them by adding or deleting text, and rate relevance, fluency, coherence, and likability on five-point Likert scales.The platform exposes model suggestions within the authors’ existing storytelling workflow.Source: §4.1 · p. 6, §4 · p. 6
  • The publicly accessible platform lets researchers integrate Python-based models by implementing four methods: startup, shutdown, preprocess, and generate.A dedicated web service serves model outputs to STORIUM users.Source: §4.1 · pp. 6–7

4.2 A Metric Over User Edits

  • USER measures the amount of generated text preserved from an entry X in the final published entry Y, treating preserved text as evidence of usefulness.The metric is inspired by the longest common subsequence variant of ROUGE.Source: §4.2 · p. 7
  • Unlike vanilla ROUGE-L, USER matches contiguous substrings containing at least one non-stopword, favoring preserved n-grams or full sentences over stopword-heavy subsequences.This matching design reflects the intended significance of authors retaining coherent generated spans.Source: §4.2 · p. 7

5 Analysis

The analysis evaluates user edits, linguistic properties of preserved text, crowdsourced judgments, and interviews to assess generated story continuations. Relevance emerges as the central weakness and the strongest correlate of user preservation, while crowdsourced evaluation performs poorly with lengthy contexts.

Top-k vs. nucleus sampling

  • USER scores are 15.6 for top-k versus 9.9 for nucleus sampling, with both configurations strongest on fluency and weakest on relevance.Users generally preferred top-k sampling across categories, while low USER scores indicate that most generated text was deleted.Source: §5 · p. 7

USER to Likert scores

  • Relevance has a significantly higher Pearson’s r correlation with USER than the other evaluated properties for top-k sampling.The table caption reports moderately strong correlations between relevance, USER, and the remaining human judgments, with correlations generally significant at p < 0.01.Source: §5 · p. 7, Table 5 · p. 7
  • Nucleus sampling correlates less with relevance but more with the other three properties, possibly because its average scores on those properties are lower.This pattern is presented as a possible consequence of nucleus sampling’s lower average property scores.Source: §5 · p. 8

Linguistic properties of preserved text

  • 29.5% of generated proper nouns are preserved versus 13.5% for other POS tags, while 63% of preserved generated entities already occur in the story context.The remaining 37% of preserved entities are often new character names, which interviews identify as a useful feature.Source: §5 · p. 8

Crowdsourced evaluation is unreliable

  • Mechanical Turk workers showed very low agreement across properties, including fluency, when shown story context, whereas removing context improved agreement for fluency and coherence.Median submit-time analysis indicated that many workers did not actually read the context, despite misleadingly long reported work times.Source: §5 · p. 8, Table 6 · p. 8

Feedback from user interviews

  • Nine of ten interviewed users identified relevance as the model’s primary improvement area, while four noted fabricated facts or newly introduced characters.Three users nevertheless said the model inspired writing or supplied useful character and place names.Source: §5 · p. 8

6 Related Work

STORIUM extends prior story-generation work by combining long, naturally annotated stories with interactive machine-in-the-loop authoring and a user-centered evaluation approach. Its evaluation differs from automatic scoring and crowdsourced assessment by relying on edits from engaged authors.

Related datasets

  • STORIUM stories are much longer than common prior datasets and include natural-language structural annotations for conditioning generation.The dataset contains scene-level cards describing narrative elements such as character goals, abilities, and key items.Source: §6 · p. 9

Machine-in-the-loop storytelling

  • The platform interactively generates story continuations for authors and evaluates a trained model with a large user population rather than using retrieval alone.This builds on machine-in-the-loop storytelling, including Creative Help, while replacing retrieval-based evaluation with a trained model.Source: §6 · p. 9

Evaluation methods

  • Unlike prior automatic scorers and diversity measures, STORIUM evaluates generated stories through direct author interaction and emphasizes human evaluation for quality assessment.Prior work includes learned quality scoring and automated diversity measures, but quality assessments remain best measured through human evaluation.Source: §6 · p. 9

Limitations

STORIUM provides high-quality judgments from invested authors, but its evaluation is costly and tied to the STORIUM community. These constraints prevent instant evaluation and limit straightforward transfer to models trained on other datasets.

Evaluation cost

  • Author effort and the relatively small STORIUM community can make collecting hundreds of judgments take considerable time.The platform is therefore not currently suitable for instant evaluation of generated stories.Source: Limitations · p. 9

Evaluation scope

  • Because the platform is deployed on STORIUM, it cannot be trivially used to evaluate models trained on other story-generation datasets.Its users are mainly invested in writing narratives that follow the STORIUM setting.Source: Limitations · p. 9

7 Conclusion

The paper introduces STORIUM as a dataset and evaluation platform for machine-in-the-loop story generation. It combines annotated long stories with author edits and an edit-based metric that correlates with relevance judgments.

  • STORIUM contains 6K long stories annotated with structural metadata for conditioning language models.Source: §7 · p. 9
  • Real STORIUM authors evaluate generated continuations by adding and removing text to create their own stories.Source: §7 · p. 9
  • The USER metric, computed from author edits, correlates strongly with judgments of generated-text relevance.Source: §7 · p. 9
  • The dataset and evaluation platform are publicly released to support progress in story generation.Source: §7 · p. 9

A Additional Dataset Statistics

The appendix characterizes STORIUM as a highly compositional dataset whose cards influence scene-entry text. It also reports long-tailed distributions and distinguishes card types with different authoring conventions.

B Web Service

The web service uses a modular frontend-backend architecture for serving models and displaying evaluation data. Its interface supports adding models through a small standard API and comparing outputs with ratings and edit-based metrics.

  • The service separates a STORIUM-facing frontend from backend services that serve model outputs.Source: B Web Service · p. 14
  • The dashboard displays ratings, user comments, scene-entry diffs, and Pearson’s r correlations among metrics and user ratings.Source: B Web Service · p. 14
  • New models can be integrated by implementing four methods: startup, shutdown, preprocess, and generate.Source: B Web Service · p. 14
  • The platform uses Docker and ASGI for deployment and runs on an on-premise server with four 2080Ti GPUs.Source: B Web Service · p. 15

C User Story Edit Ratings

USER evaluates story continuations by measuring how much generated text users retain after editing, addressing weaknesses in ROUGE for open-ended long-form text. It correlates with human judgments while producing lower, potentially more discriminative scores.

Loading 2010.01717v1…