Source-linked AI summary
Hierarchical Neural Story Generation
Angela Fan, Mike Lewis, Yann Dauphin
TL;DR
Story generation must capture long-range dependencies and condition on abstract prompts while producing coherent, fluent text. The paper introduces a hierarchical dataset and model with fusion and gated multi-scale self-attention, achieving improvements on automated and human evaluations; judges prefer its stories twice as often as a non-hierarchical baseline.
Problem
Story generation remains difficult because models must handle long-range dependencies and condition on abstract, high-level prompts across several paragraphs.
Method
The paper collects paired prompts and stories and uses hierarchical generation, model fusion, convolutional sequence-to-sequence modeling, and gated multi-scale self-attention.
Results
Human judges prefer the hierarchical model’s stories twice as often as those from a non-hierarchical baseline, while the proposed mechanisms improve automated and human evaluations.
Takeaways & Limitations
The dataset and architectures support longer, more consistent, more fluent, and more topical generated passages.
Takeaways & Limitations
Random sampling can cause tokenization errors, dialogue newline errors, and repetition in generated text; generated prompts can also be generic because rare words are difficult to model.
Abstract
from arXiv · showhide
We explore story generation: creative systems that can build coherent and fluent passages of text about a topic. We collect a large dataset of 300K human-written stories paired with writing prompts from an online forum. Our dataset enables hierarchical story generation, where the model first generates a premise, and then transforms it into a passage of text. We gain further improvements with a novel form of model fusion that improves the relevance of the story to the prompt, and adding a new gated multi-scale self-attention mechanism to model long-range context. Experiments show large improvements over strong baselines on both automated and human evaluations. Human judges prefer stories generated by our approach to those from a strong non-hierarchical model by a factor of two to one.
1 Introduction
Story generation requires long-range thematic consistency, creativity, and high-level planning. The paper addresses these challenges with hierarchical generation, model fusion, and gated multi-scale self-attention, evaluated on a large human-written story dataset.
- Story generation must maintain thematic consistency across long documents while supporting creativity and plot-level planning.
- Hierarchical generation first produces a prompt describing the story topic, then conditions story generation on that prompt to improve plot grounding and reduce topic drift.
- Standard seq2seq models can degenerate into language models that pay little attention to the writing prompt because prompt–story dependencies are complex and underspecified.
- Model fusion trains a second model over a pretrained seq2seq model to improve dependencies between the prompt and generated story.
- 734 words is the average story length in the dataset, motivating convolutional architectures that encode stories in parallel.
- Gated multi-scale self-attention lets the model condition on previous outputs at different time-scales, addressing the bounded context of existing convolutional architectures.
- 303,358 human-generated stories paired with writing prompts were collected, alongside evaluation metrics designed to isolate different aspects of story generation.
- Human judges prefer the hierarchical model’s stories twice as often as those from a non-hierarchical baseline, while fusion and self-attention improve automated and human evaluations.
2 Writing Prompts Dataset
The WRITINGPROMPTS dataset pairs diverse online writing prompts with freely contributed human story responses. It is constructed from scraped forum data, cleaned, split into validation and test sets, and vocabulary-limited for experimentation.
- WRITINGPROMPTS is an online community where users submit story premises and others freely respond with stories.
- Prompts vary widely in topic, length, and detail, while stories must contain at least 30 words and should be inspired by their prompts.
- Three years of prompts and associated stories were scraped through the official Reddit API.
- Automated posts, deleted posts, moderator announcements, moderator comments, and stories shorter than 30 words were removed during cleaning.
- The dataset reserves 5% of prompts for validation and 5% for testing after tokenization with NLTK.
- Experiments cap stories at 1000 words and retain vocabulary items appearing more than 10 times in prompts and stories.
- The resulting vocabulary contains 19,025 prompt words and 104,960 story words, while rare words and misspellings make full-vocabulary modeling challenging.
3 Approach
The approach combines hierarchical planning, convolutional sequence-to-sequence modeling, gated multi-scale self-attention, and model fusion to generate coherent stories grounded in prompts.
- Hierarchical Story Generation: Hierarchical generation first creates a premise and then uses it to condition story generation, enabling explicit high-level planning.The premise provides a structural sketch for the story before passage generation.
- Efficient Learning with Convolutional Sequence-to-Sequence Model: Convolutional encoder-decoder networks replace sequential RNN processing, allowing parallel computation across long story sequences.The model uses attention modules to connect encoder and decoder representations.
- Modeling Unbounded Context with Gated Multi-Scale Self-attention: Self-attention supplements the decoder so it can refer to previously generated words beyond CNNs’ bounded context window.Parallelism limits the computational impact while providing access to long-range context.
- Modeling Unbounded Context with Gated Multi-Scale Self-attention: Multi-head gated attention uses expressive gated networks and different downsampling scales so heads select fine-grained information at distinct time scales.Downsampling reduces the number of tokens in attention maps, making them sharper; the mechanism can optionally ignore past information and excludes the current timestep.
- Improving Relevance to Input Prompt with Model Fusion: Seq2seq models struggle because local language-modeling dependencies are easier to learn than subtle dependencies between prompts and stories.This difficulty contributes to models ignoring prompts and focusing primarily on story language modeling.
- Improving Relevance to Input Prompt with Model Fusion: Fusion trains a second seq2seq model with access to a pretrained model’s hidden states, allowing it to focus on prompt-story dependencies the first model missed.The gated hidden layers are concatenated and passed through further fully connected layers with GLU activations.
4 Related Work
Related work covers sequence-to-sequence generation, hierarchical representations, and language-model integration, while this paper emphasizes textual-premise hierarchy for improving story coherence and structure.
- Story Generation: Seq2seq networks have been applied to machine translation, summarization, Wikipedia article writing, poetry, and other open-ended generation tasks.Prior systems have also generated genre-specific text through intermediate event representations, though those outputs were not raw text.
- Story Generation: The authors report no previous work using hierarchical generation from a textual premise to improve story coherence and structure.This distinguishes textual intermediate premises from approaches whose intermediate outputs are events rather than raw text.
- Hierarchical Generation: Earlier hierarchical approaches learned word-, sentence-, and paragraph-level representations or generated discrete latent variables before producing text.These studies decompose long-sequence generation into multiple levels or condition text on an intermediate representation.
- Language-Model Integration: Prior work combined language models with seq2seq systems through initialization, inference-time prediction combination, or deeper architectural integration.The paper situates its fusion approach within this broader effort to integrate complementary models.
5 Experimental Setup
The experiments compare hierarchical and non-hierarchical story generators using perplexity, prompt relevance, human pairing, and preference evaluations. The setup also tests fusion, self-attention, sampling, and nearest-neighbor baselines.
- Baselines: The study evaluates language models, seq2seq models, ensembles, and KNN as baselines for story generation.
- Attention mechanism: Gated multi-scale attention significantly improves perplexity on the WRITINGPROMPTS dataset.
- Model training: The fusion model is trained by fixing a pretrained Conv seq2seq model and providing its hidden states to a second Conv seq2seq model.
- Generation: Stories are generated with top-k random sampling from the 10 most likely vocabulary candidates rather than beam search.
- Automatic evaluation: Prompt ranking measures whether a model assigns the highest likelihood to the true prompt among one true and nine randomly sampled prompts.
- Human evaluation: Human evaluators pair shuffled stories with prompts in a triple-pairing task, while a separate blind test compares hierarchical and non-hierarchical stories.
6 Results
The results show that hierarchical generation, gated multi-scale self-attention, and fusion improve story quality and prompt relevance across automated and human evaluations. Fusion also matches nearest-neighbor prompt relevance while retaining generative capacity.
- Effect of Hierarchical Generation: Human raters preferred stories from hierarchical generation 67% of the time in a blind comparison.
- Effect of new attention mechanism: Combining the proposed attention additions with the Conv seq2seq baseline reduces perplexity by 9 points.
- Effect of model fusion: Fusion improves human prompt-story pairing accuracy by 7%, whereas ensembling has no effect on this association.
- Effect of model fusion: Fusion substantially improves the likelihood of human-generated stories and outperforms an ensemble despite having fewer parameters.
- Comparison with Nearest Neighbours: Fusion matches nearest-neighbor search for prompt-story connection while producing an unlimited number of stories, unlike KNN.
7 Discussion
The discussion highlights both the generative model’s originality and its remaining quality constraints. Generated stories avoid direct copying, but sampling and prompt generation can produce repetition, tokenization errors, and generic prompts.
- Generation quality: Fusion-generated stories copy 8.9 words on average from training data, compared with 10.2 for Conv seq2seq and 150 for KNN.
- Generation quality: Random sampling can produce tokenization errors, dialogue formatting errors, and repeated text.
- Prompt generation: Generated prompts are fairly generic compared with human prompts because language models struggle to model rare words accurately.
- Prompt generation: The generated prompts often begin with “the man,” while human prompts more often contain unique settings and vocabulary.
- Attention analysis: Fusion decoder attention focuses mainly on salient prompt words, while self-attention can access words generated far earlier in the story.
8 Conclusion
The paper presents a dataset for creative text generation from short writing prompts and reports improvements from hierarchical models, self-attention, and model fusion.
- The authors collected the first dataset for creative text generation based on short writing prompts.
- The dataset requires models to handle longer-range dependencies and condition generation on an abstract premise.
- Automatic and human evaluations show that hierarchical models, self-attention mechanisms, and model fusion improve generated-story fluency, topicality, and overall quality.
Generation with Fusion of Conv seq2seq + Self-Attention
The fusion approach is illustrated against a language-model baseline through generated-story examples. The proposed stories better follow the desired prompt and maintain coherence across sentences and topics.
- The table compares stories generated by the proposed hierarchical fusion approach with stories generated by a language model.
- Fusion-model stories relate more closely to the desired prompt than stories from the language-model baseline.
- The fusion model shows increased coherence between sentences and greater ability to stay on one topic.
9 Appendix of Model Architectures
The appendix specifies architectures and training settings for the models, including convolutional encoder-decoder configurations, self-attention heads, ensembling, and fusion with a pretrained seq2seq model.
- Another Conv seq2seq configuration uses 3 encoder layers, 8 decoder layers, 256-dimensional embeddings, and 4 decoder self-attention heads.
- Two Conv seq2seq models are trained and ensembled by averaging them with equal weights.
- The additional fused model is built on a pretrained seq2seq model.
- The appendix gives a separate fused-model architecture with 5 encoder layers, 5 decoder layers, and 4 decoder self-attention heads.