Source-linked AI summary
A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Stories
Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, James Allen
TL;DR
Story understanding needs better evaluation of commonsense causal and temporal relations between everyday events. The paper introduces ROCStories and the Story Cloze Test, then finds that shallow baselines and models struggle on this richer evaluation, motivating deeper semantic representations and story modeling.
Problem
Commonsense story understanding lacks a proper evaluation framework for causal and temporal relations between everyday events.
Method
The paper constructs roughly 50k quality-controlled five-sentence ROCStories and introduces the Story Cloze Test, which requires choosing a sentence to complete a story.
Results
The Story Cloze Test challenges all tested models, and prior narrative-cloze approaches struggle on its richer evaluation.
Takeaways & Limitations
The authors propose Story Cloze as an evaluation for story understanding and script-learning progress, with richer representations and deeper semantic modeling needed beyond shallow performance.
Takeaways & Limitations
The corpus defines stories as causally or logically linked events with shared characters rather than requiring plots, goals, or obstacles.
Abstract
from arXiv · showhide
Representation and learning of commonsense knowledge is one of the foundational problems in the quest to enable deep language understanding. This issue is particularly challenging for understanding casual and correlational relationships between events. While this topic has received a lot of interest in the NLP community, research has been hindered by the lack of a proper evaluation framework. This paper attempts to address this problem with a new framework for evaluating story understanding and script learning: the 'Story Cloze Test'. This test requires a system to choose the correct ending to a four-sentence story. We created a new corpus of ~50k five-sentence commonsense stories, ROCStories, to enable this evaluation. This corpus is unique in two ways: (1) it captures a rich set of causal and temporal commonsense relations between daily events, and (2) it is a high quality collection of everyday life stories that can also be used for story generation. Experimental evaluation shows that a host of baselines and state-of-the-art models based on shallow language understanding struggle to achieve a high score on the Story Cloze Test. We discuss these implications for script and story learning, and offer suggestions for deeper language understanding.
1 Introduction
The paper addresses the difficulty of representing commonsense causal and temporal relations in stories by introducing a high-quality corpus and a new evaluation framework. It argues that existing evaluation and shallow approaches are insufficient for deeper story understanding.
- Commonsense knowledge about daily events is central to interpreting narratives, but noisy weblogs make useful causal information difficult to extract.
- The paper introduces ROCStories, a corpus of 50k high-quality five-sentence commonsense stories containing stereotypical causal and temporal relations.The corpus is also intended for learning narrative schemas and training story-generation models.
- The Narrative Cloze Test evaluates prediction of a held-out event, but the paper identifies a need for a more systematic framework aligned with deeper story understanding.
- The Story Cloze Test asks systems to choose an entire sentence that completes a given story instead of predicting a single event.
- 3,742 doubly verified Story Cloze Test cases were collected, and proposed models include recent state-of-the-art approaches for the narrative cloze task.
- The results suggest that outperforming random or constant-choice performance requires richer event representations and deeper modeling of story semantics.
2 Related Work
Prior work developed narrative representations and evaluated comprehension in constrained settings, but the paper positions its corpus and Story Cloze Test as broader tools for commonsense narrative understanding.
- Narrative chains model partially ordered events sharing a protagonist, with events represented as verbs and typed-dependent participants.
- Language-modeling techniques performed well on earlier narrative-cloze formulations, whereas this paper reports that they struggle on the richer Story Cloze evaluation.
- Existing comprehension frameworks include stories that are mostly fictional, average 212 words, and target children in grades 1–4.
- This paper distinguishes its resource through short, simple stories covering a wide variety of commonsense events for narrative understanding.
3 A Corpus of Short Commonsense Stories
The paper constructs ROCStories, a high-quality corpus of short nonfictional daily-life stories designed to capture commonsense causal and temporal relations. Its collection and analysis procedures aim to provide diverse, coherent narratives suitable for script learning, story generation, and temporal-ordering research.
- ROCStories contains varied commonsense causal and temporal relations between everyday events across many events rather than a single domain or genre.
- The corpus consists of high-quality nonfictional daily short life stories that can support training coherent story-telling models.
- Stories were collected from hundreds of Amazon Mechanical Turk workers as novel five-sentence narratives, with the length chosen to provide context while limiting irrelevant sidetracks.
- Multiple pilots, manual checks, iterative prompt design, and worker qualification procedures were used to improve story coherence and filter unsuitable submissions.
- Prompt constraints required coherent stories with specific beginnings and endings, causal or temporal links, sentence lengths of at most 70 characters, and descriptive titles.
- The corpus exhibits broad event and topic diversity: the two most frequent events, ‘go’ and ‘get’, each comprise less than 2% of all events, while title n-grams span many topics.
- Human shuffling results indicate that coherent stories contain recoverable temporal and causal structure, while sentence order matches TimeML order 55% of the time in another analysis.
4 A New Evaluation Framework
The paper introduces Story Cloze Test, a systematic framework that evaluates story understanding by selecting the correct ending for a four-sentence context. It uses crowd-sourced endings and multi-stage verification to create reliable test cases.
- The Narrative Cloze Test evaluates script learning by predicting a missing event, but later systems could optimize its score without demonstrating narrative knowledge.Recent cloze chains were also not human-verified as gold.
- Story Cloze Test presents a four-sentence context with right and wrong fifth-sentence endings, requiring a system to choose the right ending.The right ending is framed as an entailing hypothesis and the wrong ending as a contradicting hypothesis.
- The framework is intended as a generic evaluation for story understanding and can also evaluate story generation models using the likelihoods they assign to alternative endings.Strong performance is described as demonstrating some level of deeper story understanding, without requiring explicit narrative knowledge learning.
- 13,500 stories were sampled from ROCStories, and workers wrote a right and wrong ending for each story under character-sharing and realism constraints.The ending had to continue the story and remain realistic and sensible in isolation.
- A two-step quality-control process used qualification testing followed by verification of 27,000 full five-sentence stories by three crowd workers each.Cases were retained when the right ending received all 1 ratings and the wrong ending received all 0 ratings.
5 Story Cloze Test Models
The evaluation compares surface, embedding, narrative-chain, and deep semantic models on Story Cloze validation and test sets. Results show that shallow and event-based approaches struggle to capture the richer semantics of story endings.
- The models are evaluated with accuracy on validation and test sets, using only the validation set for tuning.Constant-choose-first provides a baseline, while human performance is included for comparison.
- The baselines include search-frequency and n-gram-overlap methods, along with average Word2Vec similarity between an ending and its context.The frequency baseline compares semantic-role representations of the main events, while overlap uses up to four-gram Smoothed-BLEU.
- Narrative Chains-AP represents events with verbs and typed dependencies and ranks endings using PMI associations learned from the Associated Press portion of Gigaword.Narrative Chains-Stories uses the same model trained on ROCStories.
- DSSM jointly projects the four-sentence context and fifth sentence into a shared vector space, selecting the candidate with the largest cosine similarity.Its configuration uses a 1000-dimensional hidden layer and a 300-dimensional embedding vector.
- 51% constant-choose-first and 100% human performance frame the results, while DSSM achieves the highest accuracy at only 7.2 points above constant-choose-first.Error analysis finds that event-based models miss complex final-sentence meanings because similar syntactic relations can describe semantically different situations.
6 Discussion
The paper contributes ROCStories, Story Cloze Test, and initial models for evaluating commonsense story understanding. Its discussion argues that the test challenges existing systems and should encourage generalization to unseen concepts and situations.
- The paper’s three contributions are the ROCStories commonsense-story corpus, the Story Cloze Test, and initial approaches for the new evaluation framework.The authors describe ROCStories as the first crowd-sourced corpus of its kind and release about 50k stories with validation and test sets.
- The authors expect future progress to depend on models that generalize to unseen commonsense concepts and situations, rather than only increasing training-data size.They note that the dataset is intended to grow to 100k stories.
- Story Cloze Test challenged all tested models and is presented as an evaluation for story understanding and script knowledge learners.The authors encourage reporting results on the Story Cloze test set and conclude that Narrative Cloze does not evaluate deeper knowledge and language understanding.