Source-linked AI summary
Event2Mind: Commonsense Inference on Events, Intents, and Reactions
Hannah Rashkin, Maarten Sap, Emily Allaway, Noah A. Smith, Yejin Choi
TL;DR
The paper addresses the lack of benchmarks for commonsense inference about people’s unstated intents and emotional reactions in short event descriptions. It builds a crowdsourced event-phrase corpus and neural encoder-decoder models, showing inference on previously unseen events and an application to gender bias in movie scripts.
Problem
Existing benchmarks do not adequately cover pragmatic commonsense inference about people’s likely intents and emotional reactions from short event phrases.
Method
The paper constructs a crowdsourced corpus of nearly 25,000 event phrases and trains models to infer participants’ intents and reactions from free-form event text.
Results
Neural encoder-decoder models compose representations of previously unseen events and reason about their participants’ mental states.
Takeaways & Limitations
Commonsense inference over event phrases can support analysis of implicit gender bias in modern movie scripts.
Takeaways & Limitations
Gender is represented as binary because of data limitations, although the paper acknowledges gender is more complex.
Abstract
from arXiv · showhide
We investigate a new commonsense inference task: given an event described in a short free-form text ("X drinks coffee in the morning"), a system reasons about the likely intents ("X wants to stay awake") and reactions ("X feels alert") of the event's participants. To support this study, we construct a new crowdsourced corpus of 25,000 event phrases covering a diverse range of everyday events and situations. We report baseline performance on this task, demonstrating that neural encoder-decoder models can successfully compose embedding representations of previously unseen events and reason about the likely intents and reactions of the event participants. In addition, we demonstrate how commonsense inference on people's intents and reactions can help unveil the implicit gender inequality prevalent in modern movie scripts.
1 Introduction
The paper introduces commonsense inference over event phrases to predict participants’ unstated intents and emotional reactions. It contributes a corpus and model, shows performance on unseen events, and applies the inference to gender bias in movie scripts.
- Motivation: Narrative understanding requires inferring people’s intents and emotional reactions from events, including effects on participants not explicitly mentioned.These inferences support applications such as empathetic dialogue, socially aware advertising, and narrative understanding.
- Motivation: Existing entailment benchmarks do not cover this broader social commonsense reasoning about mental states.The task focuses on pragmatic inference beyond explicit textual entailment.
- Contributions: The paper introduces a task, corpus, and model for stereotypical intents and reactions described in short free-form event text.The corpus covers diverse everyday events and includes inference about people indirectly affected by events.
- Results: Neural encoder-decoder models compose representations of previously unseen events and reason about participants’ mental states.This establishes baseline performance for the new phrase-level inference task.
- Contributions: Around 25,000 event phrases combine extracted story and blog phrases with idiomatic verb phrases, and the corpus is publicly available.The corpus supports the study’s broad coverage of everyday situations.
- Application: Applying the model to modern movie scripts reveals implicit gender bias beyond what previous studies have offered.The application demonstrates a practical use of inferred intents and reactions.
2 Dataset
The dataset targets well-scoped commonsense interpretation of short event phrases, emphasizing intents and emotional reactions linked to event participants. It combines broad event collection with crowdsourced annotations and reports dataset validity, agreement, and composition statistics.
- Scope: The study asks whether models can perform limited, well-scoped commonsense inference on short free-form text called event phrases.It focuses on causal relations between events and people’s mental states, which most existing resources do not cover well.
- Data construction: Event phrases are collected from stories, blogs, syntactic n-grams, and Wiktionary idioms, with person mentions replaced by typed variables such as PersonX and PersonY.The extracted patterns combine verb predicates with partially instantiated arguments; only person mentions are generalized to typed variables.
- Inference types: Intent is the mental pre-condition explaining why an agent causes a volitional event, or “none” when the event is unintentional.For example, taking a stab at something may express wanting to solve a problem.
- Inference types: Reaction is the mental post-condition describing how the agent’s and other participants’ states change after an event.For gifting, the giver may feel good while the recipient may feel grateful or thankful.
- Crowdsourcing: Each phrase is annotated by three crowdworkers for intentionality, up to three agent intents, agent reactions, and applicable reactions of other people.Annotators may mark unintelligible phrases invalid.
- Data quality: 91% of extracted events are rated valid, with average Cohen’s κ = 0.45 for multiple-choice intent and reaction annotations.The task treats variation as expected because pragmatic interpretations can have multiple valid labels and are modeled as conditional language distributions.
- Corpus statistics: 86% of events are judged willingly caused, 26% involve other people, 22% contain blanks, and intent descriptions average 3.4 words versus 1.5 for reactions.These statistics characterize intentionality, participant coverage, event instantiation, and annotation length.
3 Models
The models encode typed-variable event phrases and predict three entity-specific pragmatic inferences—PersonX’s intent, PersonX’s reaction, and others’ reactions—using multitask decoding. They compare vector-based n-gram reranking with sequence generation and test several event encoders.
- The model maps a free-form event pattern with typed variables into predictions for PersonX’s intent, PersonX’s reaction, and others’ reactions.
- Two decoding setups represent outputs as either three distributions over words and phrases or three generated word sequences.
- Event phrases are encoded into an H-dimensional embedding using pooling, a convolutional network, or a bidirectional GRU.
- Although events average 4.6 tokens, ConvNet and BiRNN encoders improve composition of their words.
- The sequence-decoding setup initializes three GRU decoder RNNs with the event embedding and generates intent and reaction descriptions one word at a time.
- Training minimizes cross-entropy for the observed outputs while jointly optimizing all three decoders through multitask learning.
- The dataset contains 24,716 unique events and 57,094 annotations, split into training, development, and test sets using an 80/10/10% division.
4 Empirical Results
The experiments show that compositional encoders and sequence decoders improve commonsense inference, while error analyses expose difficulties with explicit multi-person events and idioms. Interpolations further suggest that the embedding space captures both broad sentiment changes and fine event distinctions.
- Compositional ConvNet and BiRNN encoders moderately improve recall and cross-entropy over less compositional alternatives in both decoding setups.
- BiRNN encoders outperform ConvNets on cross-entropy, with improvements concentrated largely in PersonX intent prediction.Recall for others’ reactions is higher because most events do not involve other people.
- Sequence decoders usually achieve higher human-evaluated precision @10 than corresponding n-gram decoders, especially for intent prediction.The BiRNN sequence model slightly outperforms the other evaluated models and is used in later sections.
- Intent prediction is 10% behind full-development-set performance on the analyzed event subsets, while other-people reaction prediction is harder when those people are explicitly mentioned.Idiomatic events are particularly difficult, likely because their meanings are nonliteral or noncompositional.
- Interpolated event embeddings distinguish positive-to-negative changes and small lexical differences such as “washes” versus “cuts.”Predictions are decoded from points sampled between pairs of event phrases outside the training set.
5 Analyzing Bias via Event2Mind Inference
Event2Mind inference is applied to movie-script events to expose character mental states and analyze gender-associated portrayal patterns. The results indicate that female characters are linked with prosocial and appearance-related inferences, whereas male characters are linked with achievement, violence, and negative reactions.
- Processing of Movie Scripts: The model generates ten probable intent and reaction sequences for events extracted from scene descriptions covering 772 movie scripts and over 21,000 characters.
- Processing of Movie Scripts: Generated inferences are grouped into LIWC categories, aggregated by character, standardized, and correlated with gender using logistic regression with multiple-comparison correction.Scene-description word counts are controlled for because women appear in only 29.4% of scenes.
- Gender Differences: Female-agent events are associated with prosocial intents involving FRIEND, FAMILY, and AFFILIATION, especially around eating and making food.
- Gender Differences: Male-agent events are associated with achievement-related motivations involving ACHIEVE, MONEY, REWARDS, and POWER.
- Gender Differences: Women’s inferred intents and reactions emphasize sexuality, visibility, and perception, while men’s emphasize violence, fighting, and strong negative reactions.The reported category associations are significant after correction, with specified exceptions at p < 0.05 or p < 0.01.
- Implications: The approach converts nuanced implications into explicit statements, helping identify gender bias in modern movie portrayals.
6 Related Work
The paper differs from prior work by targeting commonsense inference about event participants’ intents and reactions rather than linguistic entailment, paraphrase, or other event relations. It also complements commonsense knowledge bases with relations that are less represented in existing resources.
- Prior research addresses event roles, implicit attributes, and event sentiment, whereas this work focuses on generating commonsense inferences from natural-language event descriptions.
- Unlike linguistic entailment work, the paper targets commonsense-based inference and generates intents and reactions rather than only entailed statements, causal explanations, or paraphrases.
- ConceptNet contains only 25% of the paper’s events, and only 12% have relations resembling intent and reaction.The authors present a more detailed comparison with ConceptNet in Appendix C.
7 Conclusion
The paper introduces a corpus, task, and model for inferring stereotypical intents and reactions from everyday event descriptions, including previously unseen events. It also demonstrates that these inferences can reveal implicit gender bias in movie scripts.
- The work combines a new corpus, task, and model for commonsense inference over textually described everyday events.
- The approach supports learning representations for diverse events and reasoning about likely mental states for previously unseen events.
- Event2Mind inference can help reveal implicit gender bias in movie scripts.
A Appendix
The corpus filters event phrases to balance generalizability with sufficient concreteness for annotation.
- Events contain at least two and fewer than five content words, excluding stop words, person tags, and blanks.Phrasal verbs such as “get up” count as content words.
A.2 Annotation Setup
Each event receives independent human annotation, with invalid events excluded from subsequent questions.
- Three Amazon Mechanical Turk raters evaluated each event.Each rater was paid $0.10 per event.
- Valid events required raters to answer how PersonX typically feels afterward.Raters could mark nonsensical events as invalid and skip the remaining questions.
B Event2Mind Training Details
Training uses Adam with specified embedding dimensions, vocabulary preprocessing, frequency thresholds, and an annotation convention for missing responses.
- Adam trains the models for ten epochs in TensorFlow.
- Baseline event embeddings use H = 300, while the BiRNN also uses H = 100.
- The vocabulary includes training tokens appearing at least twice and bigrams or trigrams appearing more than five times.
- Missing intent or reaction annotations become “none,” and leading “to” or “to be” is removed from intent annotations.
- Event selection uses corpus-specific frequency thresholds, including five for ROC Story, 100 for Spinn3r, and the top 10000 Syntactic Ngram events.
C Comparison with ConceptNet
The comparison maps Event2Mind intent and reaction annotations to related ConceptNet relations and finds limited coverage and overlap.
- Six ConceptNet relations are matched to Event2Mind’s intent and reaction dimensions before calculating unigram overlap.Intent mappings include MotivatedByGoal and CausesDesire; reaction mappings include Causes and HasLastSubevent.
- 75% of Event2Mind events are not covered in ConceptNet.Only 12% have an edge among the six relations, and annotation overlap is below 5%.
- Each event in the annotation setup was labeled by three Amazon Mechanical Turk raters.