Source-linked AI summary
A Knowledge-Enhanced Pretraining Model for Commonsense Story Generation
Jian Guan, Fei Huang, Zhihao Zhao, Xiaoyan Zhu, Minlie Huang
TL;DR
Existing story-generation models often produce fluent but repetitive, logically conflicting, and poorly coherent stories from limited context. This paper extends GPT-2 with external commonsense knowledge and a true-versus-fake story classification task, and reports more reasonable stories than strong baselines, particularly in logic and global coherence. The authors also identify remaining logical, entity-related, and chaotic-scene errors.
Problem
Existing models struggle to maintain reasonable plots, event sequences, relevant entities, and long-range coherence when generating stories from limited context.
Method
The model extends GPT-2 with ConceptNet and ATOMIC knowledge and uses auxiliary classification of true versus auto-constructed fake stories during fine-tuning.
Results
The model generates more reasonable stories than strong baselines, particularly in logicality and global coherence.
Takeaways & Limitations
Commonsense knowledge and multi-task learning are associated with more coherent and reasonable generated stories in the reported experiments.
Takeaways & Limitations
The model still produces sentence-level repetition, unrelated entities, conflicting events, and chaotic logic.
Abstract
from arXiv · showhide
Story generation, namely generating a reasonable story from a leading context, is an important but challenging task. In spite of the success in modeling fluency and local coherence, existing neural language generation models (e.g., GPT-2) still suffer from repetition, logic conflicts, and lack of long-range coherence in generated stories. We conjecture that this is because of the difficulty of associating relevant commonsense knowledge, understanding the causal relationships, and planning entities and events with proper temporal order. In this paper, we devise a knowledge-enhanced pretraining model for commonsense story generation. We propose to utilize commonsense knowledge from external knowledge bases to generate reasonable stories. To further capture the causal and temporal dependencies between the sentences in a reasonable story, we employ multi-task learning which combines a discriminative objective to distinguish true and fake stories during fine-tuning. Automatic and manual evaluation shows that our model can generate more reasonable stories than state-of-the-art baselines, particularly in terms of logic and global coherence.
1 Introduction
Open-ended story generation from limited context remains difficult: existing models can be fluent locally but often lose plot logic, relevant entities, and long-range coherence. The paper proposes external commonsense knowledge and auxiliary true-versus-fake story classification to improve these shortcomings.
- Existing models generate locally coherent stories but struggle with coherent plots, reasonable event sequences, and story diversity.
- GPT-2 story outputs exhibit repetition, logic conflicts, and weak long-range coherence despite good fluency and intrasentence coherence.
- External knowledge bases such as ConceptNet and ATOMIC provide semantic and inferential concepts that can expand reasonable story plots.
- The proposed model extends GPT-2 with external commonsense knowledge and post-trains on knowledge examples from ConceptNet and ATOMIC.
- An auxiliary classification task distinguishes true stories from shuffled, negatively sampled, or repeated fake stories to capture causal, temporal, and inter-sentence dependencies.
- Automatic and manual evaluations find that the model generates more reasonable stories than strong baselines, especially in logicality and global coherence.
2 Related Work
Prior work explores conditioning, entity and event representations, planning, pretraining, commonsense knowledge, and auxiliary objectives for language generation. However, improving pretrained models specifically for commonsense stories remains insufficiently investigated.
- Earlier story-generation systems condition on images or short text descriptions, whereas this paper uses only a limited leading context.
- Some prior models represent story entities and events to simplify dependencies between sentences.
- Another line of work first plans story sketches and then generates sentences from those sketches.
- Improving pretrained models to generate commonsense stories remains insufficiently investigated.
- Large-scale pretraining has been used to learn word- or sentence-level representations and reduce reliance on complex task-specific architectures.
- Commonsense knowledge has improved language inference, reading comprehension, dialogue generation, story-ending generation, and essay generation.
- Auxiliary-task and multi-task learning methods have been used to capture contextual dependencies and improve pretrained representations during fine-tuning.
3 Methodology
The framework extends GPT-2 with commonsense knowledge and a multi-task objective for generating stories with reasonable logical and temporal structure. It combines pretrained language modeling, knowledge post-training, and discrimination between true and automatically constructed fake stories.
- Task definition: The task is to continue a one-sentence beginning into a K-sentence story with logical, causal, and temporal connections.
- Pretrained transformer language model: GPT-2 is first pretrained as a left-to-right decoder-only transformer language model using masked self-attention.The model inherits pretrained parameters from large-scale corpus training before subsequent stages.
- Training with commonsense knowledge: Commonsense knowledge from ConceptNet and ATOMIC is converted into readable sentences and used to post-train GPT-2 implicitly.ConceptNet provides relational triples, while ATOMIC provides typed if-then triples describing everyday inferential knowledge.
- Multi-task learning: The model fine-tunes on true and fake ROCStories, where fake stories arise from sentence shuffling, negative replacement, or repetition.These transformations operate only on the K continuation sentences, not the leading context.
- Multi-task learning: The auxiliary classifier distinguishes true from fake stories alongside language modeling to capture story logicality, temporal dependencies, coherence, and repetition.Language-modeling loss is applied only to true stories, whereas classification loss uses both true and fake stories; λ scales the classification term.
4 Experiments
Experiments on ROCStories compare the knowledge-enhanced model with strong baselines and ablations using automatic, manual, coherence, and relation-understanding evaluations. The model generally improves fluency, knowledge coverage, diversity, logicality, and coherence, while repetition remains a limitation relative to some task-specific systems.
- Dataset and setup: The study evaluates four-sentence story generation from first-sentence contexts on ROCStories, using 98,162 five-sentence stories.The original corpus task selects endings, whereas this experiment generates four sentences from the first sentence.
- Baselines and setup: The comparison includes convolutional, planning-based, and GPT-2 baselines, alongside ablations that remove pretraining, external knowledge, or multi-task learning.The authors also report ablation tests using the same network structure.
- Automatic evaluation: Automatic evaluation finds stronger perplexity and BLEU, higher knowledge coverage and distinct-4, and more diverse stories than the GPT-2 variants and baselines overall.Perplexity values are not comparable for systems using word rather than byte-pair tokenization; repetition remains worse than some task-specific architectures.
- Ablation analysis: Removing pretraining raises perplexity, removing external knowledge lowers coverage and distinct-4 while increasing repetition-4, and removing multi-task learning substantially increases repetition-4.The reported ablations associate pretraining with fluency, external knowledge with commonsense coverage and diversity, and classification loss with reduced redundancy.
- Manual evaluation: Manual evaluation shows the model significantly outperforms other baselines in grammaticality and logicality, while multi-task learning further improves logic without affecting fluency.Knowledge-base post-training improves both grammar and logic by providing knowledge for expanding story plots.
- Relation understanding and case study: Relation-ranking and case-study analyses indicate that external knowledge helps capture specific causal relations and expand plots with contextually connected entities and events.The case study illustrates a progression involving driving, a breakdown, calling, being taken home, and getting home.
5 Error Analysis
Manual error analysis shows that the model improves commonsense story generation but still produces substantial logical and coherence errors. Unrelated entities or events and conflicting causal or temporal orders are the dominant error types.
- Residual errors: Many generated stories remain unreasonable despite outperforming state-of-the-art baselines in manual evaluation.The analysis compares the model against Fusion and GPT-2 (Fine-tune).
- Error types: Unrelated entities or events and conflicting logic make up most annotated errors across the evaluated models.Conflicting logic includes wrong causal relations or temporal order.
- Error types: The model further reduces chaotic logic and avoids repetition through external knowledge and multi-task learning, but coherent story generation remains challenging.Typical errors include sentence-level repetition, unrelated entities, conflicting events, and chaotic logic.
6 Conclusions and Future Work
The paper presents a knowledge-enhanced, multi-task pretraining framework for commonsense story generation. It combines pretrained language-model knowledge with external commonsense knowledge and an auxiliary true-versus-fake story classification task, while future work targets knowledge integration during pretraining.
- Conclusions: The framework combines implicit knowledge from pretrained language models with explicit knowledge from ConceptNet and ATOMIC.External knowledge is incorporated through post-training on knowledge examples.
- Conclusions: An auxiliary classification task distinguishes true from auto-constructed fake stories to capture causal, temporal, and inter-sentence dependencies.The fake stories are used to train the model to recognize story logicality and coherence.
- Conclusions: Extensive experiments show that the proposed method outperforms strong baselines and generates more coherent and reasonable stories.The conclusion attributes the improved generation quality to commonsense knowledge and multi-task learning.
- Future Work: Future work would integrate commonsense knowledge at the pretraining stage so models possess it without fine-tuning.The proposed direction differs from the paper's post-training use of external knowledge.