Source-linked AI summary
Story Ending Generation with Incremental Encoding and Commonsense Knowledge
Jian Guan, Yansen Wang, Minlie Huang
TL;DR
Story ending generation requires modeling context clues spanning a story and using implicit commonsense knowledge to produce reasonable conclusions. The paper introduces incremental encoding with multi-source attention over preceding-sentence representations and ConceptNet graphs. Automatic and manual evaluations report more coherent and reasonable endings than baselines, while the authors identify broader applicability as future work.
Problem
Story ending generation must represent logic and causal information spanning multiple sentences while using implicit knowledge beyond the text surface.
Method
The model incrementally encodes story sentences and uses multi-source attention over preceding-sentence states and ConceptNet knowledge graphs.
Results
The model generates more coherent and reasonable story endings than baselines, with lower perplexity and higher BLEU scores in automatic evaluation.
Takeaways & Limitations
Incremental encoding and multi-source attention capture context clues and commonsense knowledge for story ending generation.
Takeaways & Limitations
The paper identifies applicability to other language generation tasks as future work rather than evaluating that broader scope.
Abstract
from arXiv · showhide
Generating a reasonable ending for a given story context, i.e., story ending generation, is a strong indication of story comprehension. This task requires not only to understand the context clues which play an important role in planning the plot but also to handle implicit knowledge to make a reasonable, coherent story. In this paper, we devise a novel model for story ending generation. The model adopts an incremental encoding scheme to represent context clues which are spanning in the story context. In addition, commonsense knowledge is applied through multi-source attention to facilitate story comprehension, and thus to help generate coherent and reasonable endings. Through building context clues and using implicit knowledge, the model is able to produce reasonable story endings. context clues implied in the post and make the inference based on it. Automatic and manual evaluation shows that our model can generate more reasonable story endings than state-of-the-art baselines.
Introduction
Story ending generation must represent context clues spanning multiple sentences and use implicit commonsense knowledge to produce coherent conclusions. The paper proposes incremental encoding and multi-source attention to address both challenges.
- Motivation: Story ending generation requires logic, causal information, and implicit knowledge beyond the text surface.The ending should reflect the whole story context rather than a single entity or the last sentence.
- Context Clues: Context clues capture chronological and causal relationships among events and entities across the story.The paper emphasizes connected concepts such as Halloween, trick or treat, and monster rather than isolated words.
- Commonsense Knowledge: Commonsense knowledge helps infer reasonable endings that are not explicitly stated in the story.The Halloween example links candy to the story through retrieved ConceptNet knowledge.
- Proposed Approach: The proposed model uses incremental encoding to represent context clues and multi-source attention to incorporate commonsense knowledge.It attends to preceding words and ConceptNet knowledge graphs while encoding each sentence.
- Contribution: The model investigates story ending generation through incremental context clues and implicit knowledge.The stated evaluation shows these techniques capture story coherence and logic.
Related Work
Prior work approaches story generation with rule-based retrieval and neural sequence models, while related research increasingly incorporates entities, event planning, and commonsense knowledge.
- Story Cloze and Classification: Story Cloze Test research selects a correct ending from two candidates using coherence features such as events, sentiment, and topic.The paper positions story ending generation as moving beyond this classification setting toward generation.
- Rule-Based Methods: Rule-based story-generation methods retrieve events from knowledge bases using predefined semantic relations.These methods form stories by satisfying selected criteria.
- Neural Models: Neural story-generation work uses sequence-to-sequence learning and additional content such as photos or independent descriptions.Other models dynamically represent entities or decompose generation into event planning and sentence generation.
- Commonsense Knowledge: Commonsense knowledge has been applied to semantic reasoning, textual entailment, natural language inference, language generation, and cloze-style reading comprehension.These studies motivate using external commonsense knowledge for story generation.
Methodology
The model incrementally encodes story sentences while attending to preceding-sentence states and knowledge graphs, then uses the resulting representations for ending generation.
- Task Definition: Given a sentence sequence X, the task is to generate a one-sentence ending Y that is reasonable in logic.The model selects the ending by maximizing its conditional probability given the story context.
- Incremental Encoding: Incremental encoding attends to the preceding sentence so relationships between adjacent words and sentences are captured implicitly.The method targets logical, temporal, and causal connections in the story context.
- Knowledge Integration: The model combines hidden-state attention with graph representations of words retrieved from ConceptNet.These graph vectors provide one-hop commonsense knowledge for words in the preceding sentence.
- Model Overview: The model's context vector is an attentive read of preceding-sentence hidden states and graph vectors.Figure 2 distinguishes state context, knowledge context, graph vectors, and hidden states in the architecture.
- Encoder-Decoder Framework: The encoder-decoder framework represents the input with hidden states and generates each output word from a distribution conditioned on prior outputs and the encoded context.Attention forms a weighted read of encoder hidden states for decoding.
Incremental Encoding Scheme
The incremental encoding scheme represents context clues by reading story sentences sequentially and attending to the preceding sentence, allowing relationships between adjacent events or entities to be captured. Multi-source attention further combines preceding-sentence states with commonsense knowledge vectors during encoding and decoding.
- Incremental context representation: The incremental encoder reads the current sentence while attending to the preceding sentence to capture relationships between adjacent words.This design targets chronological and causal information that straightforward concatenation or hierarchical encoding may represent inadequately.
- Multi-source attention: Multi-source attention combines a state context vector from preceding-sentence hidden states with a knowledge context vector from preceding-sentence graph vectors.The two components provide textual and commonsense information for representing context clues.
- Commonsense integration: Each word in the preceding sentence retrieves a one-hop ConceptNet graph whose vector contributes to the knowledge context through attention.The graph vector extends a word’s semantic representation through neighboring entities and relations.
- Context propagation: Incremental encoding propagates context clues through the story, so decoding can attend to the last input sentence rather than every context sentence.The decoder uses the propagated information when generating the ending.
Knowledge Graph Representation
The model represents commonsense knowledge with one-hop ConceptNet graphs retrieved for words in the story. Each graph is encoded as a vector using existing graph-attention or contextual-attention methods and is integrated with the encoder through contextual attention.
- ConceptNet graphs: ConceptNet represents commonsense knowledge as triples in which a head concept has a relation with a tail concept.A word’s one-hop graph is retrieved by using that word as a query.
- Representation methods: The paper compares graph attention and contextual attention as two existing methods for representing ConceptNet knowledge.The comparison focuses on using knowledge for story ending generation rather than introducing a new knowledge-representation method.
- Graph representation: The knowledge graph for each word is represented as a set of triples sharing that word as the head concept.The graph vector is computed from the triples associated with the queried word.
- Vector construction: Concept and tail entities use word vectors, while relation vectors are trainable, avoiding a representation gap between graph embeddings and text-contextual embeddings.This choice follows ConceptNet’s use of common words as entities.
- Contextual attention: In contextual attention, the graph vector is computed by attending to knowledge triples according to the word’s encoder hidden state.The resulting triple memory is produced by a BiGRU, while h(x) denotes the hidden state at the encoding position of word x.
Loss Function
The model supervises the incremental encoder with vocabulary predictions at each encoding step, alongside the decoder objective. This auxiliary supervision is intended to improve modeling of chronological and causal relationships between adjacent sentences.
- Encoding supervision: At each encoding step, the model generates a vocabulary distribution in addition to performing the decoding process.The auxiliary prediction task provides supervision directly to the incremental encoding network.
- Joint objective: The loss function is formed from negative data likelihood terms for the encoder and decoder objectives.The encoder term uses reference words from context sentences, while the decoder term uses reference ending words.
- Effect of supervision: Experiments show that supervising the encoding network produces better logic than supervising only the decoding network.The method does not assume that each encoding step has only one correct next sentence.
Experiments
Experiments evaluate incremental encoding and commonsense-aware multi-source attention on ROCStories using automatic and manual measures. The results show stronger ending quality, especially logicality, than baseline encoders.
- Experimental Setup: The evaluation uses 98,162 five-sentence ROCStories, with 90,000 stories for training and 8,162 for evaluation.The generation task uses four-sentence contexts to generate the fifth sentence.
- Evaluation Metrics: Automatic evaluation uses perplexity and BLEU-1/BLEU-2, with lower perplexity indicating better performance.BLEU is limited by the availability of only one reference ending and the possibility of multiple reasonable endings.
- Automatic Evaluation: IE and IE+MSA achieve lower perplexity than the baselines, while IE+MSA(CA) obtains the highest BLEU-1 and BLEU-2 scores.The result indicates greater overlap with reference endings when multi-source attention is used.
- Manual Evaluation: Manual evaluation scores grammar and logicality independently on a 0–2 scale using three annotators and majority voting.The evaluation sampled 200 stories and collected 1,600 endings from eight models.
- Manual Evaluation: IE significantly outperforms Seq2Seq and HLSTM in grammar and logicality, while IE+MSA significantly outperforms HLSTM+MSA in logicality.IE scores 1.84 versus 1.74/1.57 in grammar and 1.10 versus 0.70/0.84 in logicality; IE+MSA scores 1.26/1.24 versus 1.06/1.02 for HLSTM+MSA across GA/CA settings.
- Manual Evaluation: IE+MSA has a larger proportion of endings with good grammar and logicality, and commonsense knowledge improves logicality across model comparisons.The three-annotator kappa is 0.29, indicating fair agreement; an example also shows baselines producing improper, repetitive, or copied content.
Conclusion and Future Work
The paper concludes that incremental encoding with multi-source attention uses context clues and commonsense knowledge to generate more coherent and reasonable story endings. The authors suggest applying these mechanisms to other language-generation tasks.
- Conclusion: The model incrementally encodes story context and attends to preceding words and their knowledge graphs through multi-source attention.This combines contextual clues with commonsense knowledge retrieved from ConceptNet.
- Conclusion: Experiments show that the models generate more coherent and reasonable story endings.
- Future Work: The authors identify applicability to other language-generation tasks as future work.
Appendix A: Annotation Statistics
Annotation agreement was high overall, but logicality was harder to assess consistently than grammar.
- 96% of grammar annotations and 94% of logicality annotations had agreement from at least two of three annotators.
- Three-way agreement was lower for logicality than for grammar, indicating greater annotation complexity for logicality.
Appendix B: Error Analysis
The error analysis identifies grammar, logicality, and other failures, including repetition, incorrect entities, inconsistent events, and difficulty with rare words.
- The authors manually examined 46 bad endings and grouped errors into bad grammar, bad logicality, and other errors.
- One example repeats content, while another predicts daughter instead of the more appropriate entity car.
- Some endings contain events inconsistent with preceding words, such as work on his own following relieved.
- The model also performs poorly with rare words, which the authors suggest may be improved with a copy mechanism.
Appendix C: Attention Visualization
The visualization shows incremental encoding using separate attention over state context and commonsense knowledge, linking important events across adjacent sentences.
- Multi-source attention computes separate state context and knowledge context vectors during incremental encoding.
- The figure’s green graphs represent state attention, blue graphs represent knowledge attention, and each matrix relates two adjacent sentences.
- The visualization uses graph attention as an example for representing commonsense knowledge in the model.
- State attention highlights important context events across sentence boundaries, such as cooking in X1 when encoding X2.
- ConceptNet knowledge is incorporated into each sentence’s encoding through attention to knowledge graphs from preceding sentences.