Source-linked AI summary
It's not Rocket Science : Interpreting Figurative Language in Narratives
Tuhin Chakrabarty, Yejin Choi, Vered Shwartz
TL;DR
Figurative expressions are often non-compositional, and language models struggle to represent their intended meanings. This paper tests idiom and simile interpretation through narrative-continuation datasets and tasks, finding that knowledge-enhanced models improve performance while a human-performance gap remains.
Problem
Figurative expressions often depart from the literal meanings of their words, yet language models struggle to represent those implied meanings.
Source: §1 · p. 1Method
The authors build narrative continuation tasks for idioms and similes and add context- and constituent-meaning knowledge to models.
Source: §1 · p. 2, §3.1 · p. 5Results
Knowledge-enhanced models outperform other models across both datasets and settings, but supervised models remain substantially behind humans.
Source: §1 · p. 2, §6 · p. 12Takeaways & Limitations
The findings show that human-inspired knowledge strategies can bridge some of the performance gap on figurative-language narrative tasks.
Source: §1 · p. 2, §6 · p. 12The knowledge-enhanced models leave room for improvement, and spelling or grammar errors may have affected plausibility judgments of generated continuations.
Source: §1 · p. 2, §5.3 · p. 11
Abstract
from arXiv · showhide
Figurative language is ubiquitous in English. Yet, the vast majority of NLP research focuses on literal language. Existing text representations by design rely on compositionality, while figurative language is often non-compositional. In this paper, we study the interpretation of two non-compositional figurative languages (idioms and similes). We collected datasets of fictional narratives containing a figurative expression along with crowd-sourced plausible and implausible continuations relying on the correct interpretation of the expression. We then trained models to choose or generate the plausible continuation. Our experiments show that models based solely on pre-trained language models perform substantially worse than humans on these tasks. We additionally propose knowledge-enhanced models, adopting human strategies for interpreting figurative language types : inferring meaning from the context and relying on the constituent words' literal meanings. The knowledge-enhanced models improve the performance on both the discriminative and generative tasks, further bridging the gap from human performance.
1 Introduction
The paper targets figurative meaning in narratives, where non-compositional expressions challenge language models, and evaluates models through continuation tasks. Pre-trained models trail humans; knowledge-enhanced approaches improve performance but do not close the gap.
1 Introduction
- Figurative phrases often convey meanings that differ from their constituent words, while similes may be created ad hoc and require multiple reasoning steps.The paper gives “chicken feed” as an idiom whose meaning differs from its literal reading, and a mountain-lake comparison that implies calmness.Source: §1 · p. 1
- The study extracts narratives containing figurative expressions and tests whether models can choose or generate continuations that depend on interpreting them.The two tasks are discriminative selection among candidates and generative production of a coherent, meaning-compliant continuation.Source: §1 · pp. 1–2
- Supervised models remained 10 and 14.6 points behind humans in idiom and simile discriminative accuracy, and 24 and 28 points behind in generative plausibility evaluation, respectively.The reported gaps are for idioms and similes, in that order, across the two task types.Source: §1 · p. 2
- Knowledge-enhanced models use context inferences and constituent-word literal meanings, and consistently outperform other models across both datasets and settings.The generative tasks retain a substantial gap from human performance.Source: §1 · p. 2
- The favored interpretation strategy differs by expression type: context-based generation works well for idioms, while literal-attribute reasoning is favored for similes.The authors note that the knowledge-enhanced models still leave room for improvement.Source: §1 · p. 2
2 Background
The background reviews idiom and simile interpretation, human strategies for understanding figurative language, narrative benchmarks, and commonsense knowledge models. It positions this work as applying those resources to figurative meaning in narratives.
2.1 Idioms
- Idioms have non-literal meanings and are typically non-compositional and fixed; prior work has emphasized identification, while language models encode constituent words but capture little phrase meaning (Yu and Ettinger, 2020).Earlier approaches to idiom interpretation include paraphrase substitution, single-token representations, and character-level composition.Source: §2.1 · pp. 2–3
2.2 Similes
- Simile work has focused mainly on detection, while interpretation research addresses recovering implicit properties and pragmatic inferences (Qadir et al., 2016; Chakrabarty et al., 2021a).Similes may be explicit, with a stated comparison property, or implicit, with that property omitted.Source: §2.2 · p. 3
2.3 Human Processing of Figurative Language
- For unfamiliar idioms, people often infer meaning from context or constituent words; novel similes and metaphors are interpreted through shared literal attributes.In Cooper’s study, context inference succeeded 57% of the time and reliance on constituent words 22%.Source: §2.3 · p. 3
2.4 Narrative Understanding
- Narrative understanding includes question answering and story cloze tasks, which ask models to select or generate plausible continuations.The paper follows the story-cloze setup while targeting figurative-language comprehension.Source: §2.4 · pp. 3–4
2.5 Commonsense Knowledge Models
- Commonsense knowledge models generate implicit event- or concept-centered inferences, and this paper uses COMET and ParaCOMET to supply constituent-word or narrative-context information.ParaCOMET provides discourse-aware paragraph-level knowledge; COMET can generate inferences for new events or concepts.Source: §2.5 · p. 4
3 Data
The datasets cast figurative-language interpretation as narrative continuation: plausible and implausible continuations should depend on understanding the expression. Separate idiom and implicit-simile collections provide the examples and splits for evaluation.
3 Data
- The benchmark tests whether a model that understands a figurative expression can infer or decide what happens next in a narrative, using a story-cloze format.The datasets cover idioms and similes.Source: §3 · p. 4
3.1 Idioms
- The idiom dataset pairs narratives containing listed idioms with crowd-written plausible and implausible continuations conditioned on idiom definitions.The 5,101 tuples were filtered to remove literal uses and split into train, validation, and test sets with no idiom overlap between train and test.Source: §3.1 · pp. 4–5
3.2 Similes
- The simile collection converts explicit corpus similes to implicit forms by removing the stated property, then pairs them with narratives and crowd-written continuations.It contains 520 similes and 4,996 narrative-continuation tuples; the splits have no simile overlap.Source: §3.2 · p. 5
4 Discriminative Task
The section defines narrative continuation choice for idioms and similes, compares zero-, few-shot, and supervised systems, and analyzes inference-based RoBERTa enhancements.
4 Discriminative Task
- The discriminative task asks a model to choose the more plausible of two continuations for a narrative.This follows the story-cloze setup.Source: §4 · p. 5
4.1 Methods
- Few-shot GPT-3 uses six prompting examples within the 2,048-token API limit, with test answers omitted; PET reformulates tasks as cloze questions and trains each model on 100 examples for 3 epochs, selecting the model with best validation accuracy.PET predicts masked tokens and maps them to task-label inventories.Source: §4.1 · p. 6
- Baselines include supervised RoBERTa-large continuation scoring, language-model likelihood ranking, and zero-shot UnifiedQA posed as a plausibility question.RoBERTa scores each narrative-continuation combination; the higher-scoring continuation is selected.Source: §4.1 · pp. 5–6
- Two RoBERTa variants add generated inferences from narrative context or literal constituent words, then sum inference-wise continuation scores.Context inferences come from ParaCOMET; literal-word inferences come from COMET, with relation choices tailored to idioms and similes.Source: §4.1 · pp. 6–7
4.2 Results
- Among zero-shot models, GPT-2 performed worse than GPT-3 and UnifiedQA, each of which performed best on one task.UnifiedQA performed well on idioms, which the authors attribute possibly to its familiarity with the QA format and narrative domain.Source: §4.2 · p. 7
- 12 accuracy points for idioms and 3.5 for similes: PET outperformed few-shot GPT-3, which the authors conjecture reflects 100 versus six training examples.GPT-3’s six-example limit followed from the 2,048-token API limit and the concatenated-prompt setup.Source: §4.2 · p. 7
- Few-shot models underperformed zero-shot models on both datasets; the authors point to pretrained-model advantages and the few examples available for learning the task.They relate GPT-2 and GPT-3’s performance to the similarity between plausibility choice and next-word prediction, and UnifiedQA’s to relevant training.Source: §4.2 · pp. 7–8
- Knowledge-enhanced models narrow the idiom gap, while simile results are mixed; supervised models substantially outperform few-shot and zero-shot models but remain below human accuracy.Human performance is the average accuracy of two native English speakers; the authors assume they knew more common idioms.Source: §4.2 · pp. 7–8, Table 2 · p. 8
5 Generative Task
The generative task tests whether models can continue narratives coherently while honoring an idiom or simile. Knowledge-enhanced models outperformed GPT-2 in human evaluation, with different strategies favored for each expression type.
5 Generative Task
- The task generates a plausible next sentence coherent with the narrative and consistent with the figurative expression’s meaning.The generative experiments included zero-shot, few-shot, supervised, and knowledge-enhanced approaches; the latter condition on individual inferences and ensemble logits across prompts to generate a continuation.Source: §5 · p. 9, §5.1 · p. 9
5.1 Methods
- The knowledge-enhanced generative models use GPT-2 XL with each inference paired with the narrative, then combine generations by ensembling logits across inference prompts.The models are fine-tuned with a language-model objective to predict the plausible continuation from an inference-conditioned input.Source: §5.1 · p. 9
5.2 Results
- Knowledge-enhanced models performed similarly to supervised GPT-2 XL on automatic metrics, with slight preference for the baseline on idioms and for knowledge-enhanced models on similes; both supervised approaches beat zero-shot and few-shot models.The reported automatic metrics were Rouge-L and BERT-Score.Source: §5.2 · p. 10, Table 4 · p. 10
- For both idioms and similes, a knowledge-enhanced model outperformed baseline GPT-2 in absolute and comparative human evaluation; context was favored for idioms and literal knowledge for similes.Workers judged plausibility and preference; 80% of human-written idiom continuations and 88% of simile continuations were judged plausible, with narrative ambiguity offered as a possible explanation for the gap from 100%.Source: §5.2 · pp. 10–11, Table 5 · p. 10
5.3 Error Analysis
- Implausible generations were categorized as inconsistent with the figurative expression, inconsistent with the narrative, or containing spelling or grammar errors.Examples include a brown brownie continuation for a simile whose property is dark blue, purple, and black, and generations that confuse characters; spelling or grammar errors may have affected judgments despite instructions to ignore them.Source: §5.3 · p. 11
6 Conclusion
The paper introduces a narrative-understanding benchmark for idioms and similes, using continuation tasks to test figurative-language interpretation. Pre-trained models struggle in zero-shot and few-shot settings, and supervised models remain behind humans; knowledge-enhanced models narrow that gap.
6 Conclusion
- The benchmark tests figurative-language understanding in narratives through discriminative and generative continuation tasks.It focuses specifically on idioms and similes.Source: §6 · p. 11
- Pre-trained language models struggle in zero-shot and few-shot settings, while supervised models remain significantly behind human performance.The authors report that knowledge-enhanced models bridge some of the human-performance gap.Source: §6 · p. 12
- The analysis also finds that models can generate grammatical, human-like text while remaining inconsistent and limited in distinguishing story characters.Source: §6 · p. 12