MelBERT: Metaphor Detection via Contextualized Late Interaction using Metaphorical Identification Theories

1 Introduction

• Metaphor represents concepts beyond words’ literal meanings; in “hope is on the horizon,” “horizon” has a context-specific, positive meaning.The paper motivates contextualized meaning as important to NLP tasks including machine translation, sentiment analysis, and dialogue systems.Source: §1 · p. 1
• Automated metaphor detection aims to identify metaphorical expressions in computational models.The paper frames this as a challenging task involving metaphorical expressions of words in sentences.Source: §1 · p. 1
• Existing metaphor-detection approaches use hand-crafted features, recurrent neural networks, or contextualized models such as BERT and RoBERTa.The passage notes that feature-based methods depend strongly on corpus quality and that RNNs are limited in understanding diverse contextual word meanings.Source: §1 · p. 1
• MIP identifies metaphor when a word’s contextual meaning differs from its literal meaning, while SPV flags a target word that is unusual among its surrounding words.The paper adopts both theories, which share an idea but use different detection procedures.Source: §1 · p. 1
• MelBERT encodes a sentence with its target word and the target word separately, then combines MIP- and SPV-inspired signals to classify the target.The contextualized and isolated target-word representations support comparison of contextual and literal meanings; sentence and contextualized target representations support assessment of surrounding-word mismatch.Source: §1 · p. 2
• MelBERT consistently outperforms state-of-the-art metaphor-detection models in F1-score on several benchmark datasets.Its reported advantages combine contextualized word representations, late interaction, and two metaphor-identification theories.Source: §1 · p. 2

2.1 Metaphor Detection

• Hand-engineered metaphor features include abstractness, concreteness, imageability, supersenses, and property norms, but reliance on manually annotated resources makes rare metaphor usages difficult to handle.Sparse distributional features and dense embeddings are described as alternatives.Source: §2.1 · p. 2
• RNN-based metaphor detectors combine recurrent encoders with text and linguistic features, but shallow BiLSTM and CNN models have limits representing varied contextual word meanings.Examples include BiLSTM and CNN with Word2Vec, and BiLSTM using GloVe and ELMo.Source: §2.1 · p. 2
• Contextualized models such as BERT and RoBERTa support metaphor detection; prior systems add linguistic features, external-resource information, or multitask learning.The cited work includes DeepMet, IlliniMet, and a multitask formulation for metaphor detection.Source: §2.1 · p. 2

2.2 Semantic Matching over BERT

• Late-interaction semantic matching encodes the two inputs independently, and MelBERT applies this architecture to a sentence containing a target word and the target word alone.The passage contrasts this with fully interacting BERT query-document pairs, which compute a relevance score.Source: §2.2 · pp. 2–3

3 MelBERT

• The model predicts whether a target word is metaphorical or literal, and can label every sentence word by sequentially changing the target.The section distinguishes target-word classification from sequence labeling.Source: §3 · p. 3

3.1 Motivation

• Late interaction encodes the sentence and target word independently, avoiding unnecessary interactions and allowing the sentence encoding to be reused across target words.The paper adopts this paradigm to compare contextualized and isolated target-word meanings.Source: §3.1 · p. 3

3.2 Model Architecture

• MelBERT uses separate sentence and target-word encoders; when BPE splits a target into multiple tokens, it represents the target with average pooling.The sentence encoder produces contextualized word representations, while the target encoder represents the word without context.Source: §3.2 · pp. 3–4, Figure 2 · p. 4
• MelBERT learns contextual-versus-isolated target meaning gaps with MIP and whole-sentence-versus-target-in-sentence gaps with SPV, then combines both signals for prediction.The MIP and SPV hidden vectors are combined to produce a prediction score.Source: §3.2 · pp. 4–5

4.1 Experimental Setup

• The evaluation uses VUA-18 and its VUA-20 extension, with MOH-X and TroFi reserved for VUA-trained zero-shot tests.VUA covers all POS tags and has about 10% metaphorical words; MOH-X and TroFi annotate verbs and have more than 40%. The study reports precision, recall, and F1, and tunes contextualized-model hyperparameters on VUA-18dev.Source: §4.1 · pp. 5–6, Table 1 · p. 5

4.2 Empirical Results

• 2.8%, 1.0%, and 1.9% F1 gains over DeepMet on VUA-18, VUA-20, and VUA-Verb accompany MelBERT's consistent lead over contextualized baselines.The reported gains over RoBERTa_BASE and RoBERTa_SEQ reach 1.2–1.5% across the three datasets.Source: Table 2 · p. 6, Table 3 · p. 6, §4.2 · p. 7
• MelBERT remains comparable to or better than competitive models across VUA-18 genres, with its clearest F1 leads in Academic and Fiction.Against RNN-based models, reported F1 gains reach 4.9% in Academic, 4.4% in Conversation, 10.2% in Fiction, and 2.8% in News; Conversation and Fiction have more complicated or rare expressions.Source: Table 4 · p. 7, §4.2 · p. 8
• MelBERT has the best F1 across VUA-18 verb, adjective, adverb, and noun targets, and outperforms DeepMet for every POS tag.Against RNN-based models, reported gains reach 5.9% for verbs, 3.4% for adjectives, 14.5% for adverbs, and 10.3% for nouns.Source: Table 5 · p. 7, §4.2 · p. 8
• In zero-shot testing after VUA-20 training, MelBERT achieves F1 = 79.2 on MOH-X, while TroFi is harder for all models and MelBERT beats DeepMet on recall.The authors attribute TroFi's lower overall performance relative to MOH-X to longer, more complicated sentences; the comparison with DeepMet uses a VUA-20-trained model.Source: Table 6 · p. 8, §4.2 · p. 8
• Ablations show the full model performs best with both MIP and SPV, while MelBERT without SPV outperforms the version without MIP.The bagged MelBERT-CV also outperforms DeepMet-CV on all metrics on VUA-18 and VUA-20, and raises recall from 68.6 to 73.7.Source: §4.2 · pp. 7–8, Table 7 · p. 8
• Error analysis identifies borderline or implicit metaphors as particularly difficult, alongside multiple metaphors in one sentence and personification without SPV.Without MIP, the model also struggles with common polysemous words such as “go” and “feel.”Source: §4.2 · pp. 8–9, Table 8 · p. 9

• MelBERT achieves state-of-the-art performance on several datasets.Source: §5 · p. 9
• MelBERT marries pre-trained contextualized models with metaphor identification theories.Source: §5 · p. 9
• The authors describe this as the first work to take full advantage of both contextualized models and metaphor identification theories.Source: §5 · p. 9