Source-linked AI summary
Neural Metaphor Detection in Context
Ge Gao, Eunsol Choi, Yejin Choi, Luke Zettlemoyer
TL;DR
Metaphor detection requires contextual reasoning about whether a described situation could literally occur. The paper applies BiLSTMs with contextualized word representations to classify target verbs and label metaphorical words across sentences. The models improve prior state-of-the-art results on three benchmarks, while jointly labeling sentence words can enhance performance.
Problem
Metaphor detection requires contextual reasoning about whether a described situation could literally occur, across target-verb classification and all-word sentence labeling.
Source: §1 · p. 1Method
Results
7.5 F1 on VUA sequence labeling, 2.5 F1 on VUA verb classification, and 4.9 F1 on MOH-X improve prior state-of-the-art results.
Source: §1 · p. 1Takeaways & Limitations
Sequence labeling enhances performance when jointly predicting the metaphoricity of all words in a sentence.
Source: §6 · p. 5For datasets without annotations for every word, the models assume that each unannotated word is literal.
Source: §3.1 · p. 2
Abstract
from arXiv · showhide
We present end-to-end neural models for detecting metaphorical word use in context. We show that relatively standard BiLSTM models which operate on complete sentences work well in this setting, in comparison to previous work that used more restricted forms of linguistic context. These models establish a new state-of-the-art on existing verb metaphor detection benchmarks, and show strong performance on jointly predicting the metaphoricity of all words in a running text.
1 Introduction
Metaphor detection requires contextual reasoning about whether a situation can literally occur, while prior approaches often used restricted context or weak unigram features. The paper presents end-to-end neural models for two metaphor-detection tasks and reports improved benchmark results.
1 Introduction
- Metaphor detection requires deciding whether a specific situation can actually happen, which calls for contextual reasoning.The example of “examining” a country with a microscope illustrates why literal plausibility depends on context.Source: §1 · p. 1, Table 1 · p. 1
- Earlier approaches often supplied limited context such as SVO triples, while some full-sentence systems relied on unigram features with limited expressivity.The paper argues that full-sentence context supports more accurate prediction.Source: §1 · p. 1
- The paper investigates classifying a target verb and detecting all metaphorical words in a sentence, regardless of part of speech.It uses relatively standard BiLSTM architectures with contextualized word embeddings for both tasks.Source: §1 · p. 1
- The models improve prior state of the art by 7.5 F1 on VUA sequence labeling, 2.5 F1 on VUA verb classification, and 4.9 F1 on MOH-X.The models are based on bidirectional LSTMs and contextualized word representations.Source: §1 · p. 1
2 Task
The paper formulates metaphor detection both as labeling every word in a sentence and as classifying one target verb. Sequence labeling generalizes target-verb classification and can outperform it when all words are annotated.
2 Task
- Sequence labeling predicts a binary metaphoricity label for each word in the sentence.Figure 1 depicts every word as classified.Source: §2 · p. 1, Figure 1 · p. 2
- Target-verb classification predicts one binary metaphoricity label for a specified verb in the sentence.The target is identified by its index.Source: §2 · p. 1
- With accurate annotations for every word, sequence labeling outperforms classification even when evaluation is set up as target-verb classification.The target verb's prediction can be extracted from the full-sentence predictions.Source: §2 · p. 2
3 Model
Both task formulations use sentence-level bidirectional LSTMs and feedforward classifiers, with pretrained and contextualized embeddings as input. The sequence-labeling model predicts labels for each token, while the classification model uses attention to pool sentence states for a target verb.
3 Model
- Both models encode sentences with a bidirectional LSTM and classify using a feedforward network optimized for gold-label log-likelihood.Token representations combine pretrained word embeddings with ELMo contextual vectors.Source: §3 · p. 2
3.1 Sequence Labeling Model
- The sequence-labeling model maps each token's word and ELMo representations through a bidirectional LSTM to a label for that token.When not every word is annotated, the model assumes unannotated words are literal.Source: §3.1 · p. 2
3.2 Classification Model
- The classification model marks the target verb with an index embedding, applies attention to pool LSTM states, and feeds the pooled representation to a classifier.The attention-weighted sum is used to compute the target verb's label scores.Source: §3.2 · p. 2
4 Dataset
The experiments use verb-classification datasets TroFi and MOH-X and the VUA dataset for classification and sequence labeling. The datasets differ in size, sentence characteristics, metaphor prevalence, and evaluation setup.
4 Dataset
- The benchmark suite includes TroFi and MOH-X for verb classification and VUA for tagging, with dataset statistics reported for the classification and VUA sequence-labeling data.The percentage-metaphor statistic is sentence-level in Table 2 and token-level in Table 3.Source: §4 · p. 2, Table 2 · p. 3, Table 3 · p. 3
- MOH-X is a subset of MOH in which a verb and its argument are extracted from each sentence.MOH sentences are shorter and simpler than sentences from resources such as news articles; TroFi and MOH-X also have higher metaphor prevalence than running text in VUA.Source: §4 · pp. 2–3
- MOH-X and TroFi use 10-fold cross-validation, while VUA classification uses its original train-test split with 10% of training set aside for development.The VUA sequence-labeling split follows the verb-classification split; only 11% of its tokens are metaphors.Source: §4 · p. 3
5 Experiments
Experiments compare sequence labeling and verb classification across VUA, MOH-X, and TroFi, with results varying by dataset annotation coverage. They also examine contextualized embeddings, POS-specific performance, and model errors.
Experiment Setup
- Experiments use a lexical-frequency baseline that labels a word metaphorical when metaphorical annotations outnumber literal ones in training, and compare against published feature-based, neural-similarity, and CNN-LSTM systems.Source: §5 · p. 3
Verb Classification Results
- CLS performs better than SEQ on TroFi and MOH-X, while SEQ outperforms CLS on VUA when all words are annotated.The sequence-labeling predictions can also supply target-verb predictions, and accurate labels for context words help the VUA classification setup.Source: Table 6 · p. 4, §5 · p. 4
Sequence Labeling Results
- The VUA sequence-labeling model mainly improves recall, while the lexical baseline performs strongly in precision.The results are reported for all POS tags on the VUA sequence-labeling test set.Source: Table 4 · p. 3, §5 · p. 4
- Adpositions are easiest to identify as metaphorical and are the most frequently metaphorical POS class at 28%, while particles are challenging around multi-word expressions.The POS breakdown covers tags with more than 10% metaphor in the training set.Source: Table 5 · p. 3, §5 · p. 4
Ablation Study
- Contextualized word vectors improve both CLS and SEQ performance by a large margin in the VUA verb-classification ablation.The ablation compares input representations with and without ELMo vectors.Source: §5 · p. 4, Table 7 · p. 4
Error Analysis
- In the sampled VUA development errors, false positives and false negatives were about equally common; borderline cases accounted for about half of false negatives and 20% of false positives.False positives also included implicit arguments, long-range dependencies, rare word senses, and anthropomorphic arguments.Source: §5 · pp. 4–5
- Among 257 development examples missed by CLS but correctly classified by SEQ, SEQ handled personifications, indirect metaphors, and direct metaphors involving uncommon verbs.The examples were specifically sampled from cases where CLS was wrong and SEQ was correct.Source: §5 · p. 5
6 Related Work
Prior metaphor-detection work explored diverse lexical, semantic, and embedding-based features, as well as neural architectures with linguistic features. This paper reports gains from contextualized embeddings and studies sequence labeling alongside classification.
Related Work
- Earlier approaches used features such as concreteness, imaginability, feature norms, sensory information, bag-of-words, and WordNet semantic class.Embedding-based approaches also showed gains on benchmarks, and VUA shared-task systems included LSTMs and CRFs augmented with linguistic features.Source: §6 · p. 5
- The paper finds that contextualized word embeddings improve metaphor detection and that sequence labeling enhances performance when jointly predicting all words in a sentence.The paper also compares sequence labeling and classification approaches.Source: §6 · p. 5
7 Conclusion
Simple BiLSTMs with contextualized word representations establish new state-of-the-arts across existing metaphor-detection benchmarks, while error analysis identifies remaining challenges.