PaperlayerView exact revision on arXiv

Source-linked AI summary

Testing the Ability of Language Models to Interpret Figurative Language

Emmy Liu, Chen Cui, Kenneth Zheng, Graham Neubig

arXiv:2204.12632v2cs.CLcs.AI

TL;DR

Figurative language is common, yet its interpretation remains relatively understudied in NLP, leaving open how well language models understand nonliteral phrases. Fig-QA tests this with paired metaphorical expressions and opposing interpretations; models can be fine-tuned to perform well on the task, but zero-shot performance remains far below human performance.

  • Problem

    Figurative language is relatively understudied in NLP, and it remains unclear whether language models can interpret nonliteral phrases that may rely on commonsense or cultural knowledge.

    Source: §1 · p. 1

  • Method

    Fig-QA is a Winograd-style benchmark of paired figurative phrases with divergent meanings, asking models to select the entailed interpretation.

    Source: §1 · p. 2

  • Results

    Fine-tuned models perform well on this task, but zero-shot performance shows a large gap from human performance.

    Source: §8 · p. 9

  • Takeaways & Limitations

    The authors call for further study of nonliteral reasoning, especially in few-shot settings.

    Source: §8 · p. 9

    Fig-QA contains only English examples, leaving figurative language in other languages for future work.

    Source: §9.1 · p. 10

Abstract

from arXiv · show

Figurative and metaphorical language are commonplace in discourse, and figurative expressions play an important role in communication and cognition. However, figurative language has been a relatively under-studied area in NLP, and it remains an open question to what extent modern language models can interpret nonliteral phrases. To address this question, we introduce Fig-QA, a Winograd-style nonliteral language understanding task consisting of correctly interpreting paired figurative phrases with divergent meanings. We evaluate the performance of several state-of-the-art language models on this task, and find that although language models achieve performance significantly over chance, they still fall short of human performance, particularly in zero- or few-shot settings. This suggests that further work is needed to improve the nonliteral reasoning capabilities of language models.

1 Introduction

Figurative language is common but comparatively understudied, and interpreting creative metaphors asks language models to infer meaning rather than merely detect a metaphor. Fig-QA tests this ability with paired metaphorical phrases and their opposing interpretations.

1 Introduction

  • Figurative language is less studied than literal language and may require commonsense or cultural knowledge that language statistics alone do not capture.Prior work often tests metaphor detection or paraphrasing rather than choosing a context-dependent interpretation.Source: §1 · p. 1
  • Fig-QA asks models to choose the entailed interpretation for paired, opposite metaphorical phrases, targeting creative metaphors that may not appear in training data.The benchmark contains 10,256 metaphors and implications.Source: §1 · p. 2
  • Language models can be fine-tuned to perform reasonably well, but their few-shot performance falls significantly short of human performance.The analysis also finds that models underuse metaphorical context and that associating a metaphor with an interpretation is harder than the reverse.Source: §1 · p. 2

2 Dataset Creation and Validation

The dataset was crowdsourced to elicit rare, creative metaphor pairs with literal implications, then manually filtered and released in multiple training-set sizes.

2.1 Crowdsourcing Task

  • Workers generated rare or creative metaphor pairs with different meanings and supplied literal implications; random metaphor-rich words served as optional prompts.Opposing meanings were intended to reduce ambiguity and help prevent annotation artifacts.Source: §2.1 · p. 2

2.2 Data Validation

  • Three authors manually filtered 13,324 collected examples, retaining 10,256 after excluding unintelligible or format-violating items.Exclusions also covered examples that did not make sense given the figurative expression.Source: §2.2 · p. 2

2.3 Final Dataset

  • The medium train, dev, and test splits partition first-stage data; the large train split adds second-stage examples, while the small train split is a random sample.The release provides these named data splits.Source: §2.3 · p. 2

3 Figurative Language Typologies

The analysis characterizes figurative examples by their subject-relation-object structure and by the kinds of commonsense knowledge needed to interpret them.

3 Figurative Language Typologies

  • The typology analysis examines subjects, objects, and relations alongside the commonsense knowledge needed to interpret metaphors.The structure analysis segments phrases using syntactic patterns, then lemmatizes and removes punctuation and selected determiners.Source: §3 · p. 2, §3.1 · p. 3

3.1 Figurative Language Structure

  • The medium train set contains 441 unique subjects, 646 relations, and 1,198 objects; Figure 1 visualizes its 25 most frequent examples of each component.Metaphors and similes are represented as subject, relation, and object, with interpretation inferring a subject attribute from the object's relational attribute.Source: §3.1 · pp. 2–3, Figure 1 · p. 3

3.2 Common-sense Knowledge Types

  • Test-set analysis categorized commonsense needs as object knowledge (68.35%), visual metaphors (14.73%), social understanding (27.55%), and cultural knowledge (16.56%).The categories are not mutually exclusive; the cultural examples reflect US-based crowdworkers.Source: §3.2 · p. 3

4 Baseline Models and Evaluation

The section describes probability-based zero-shot scoring for autoregressive models, transfer and contrastive evaluation for masked models, and the forced-choice and human-evaluation settings.

4.1 Auto-regressive Language Models

  • Autoregressive models score metaphor–interpretation pairs using forward probabilities, while backward probabilities score phrases given interpretations.The method uses length-normalized sentence probabilities as a heuristic to reduce sentence-length effects.Source: §4.1 · p. 4

4.2 Masked Language Models

  • Masked language models cannot directly provide sentence probabilities, so evaluation uses transfer from WinoGrande or NLI datasets and contrastive fine-tuning on Fig-QA.For NLI models, the paper tests both the original three-label setup and a forced binary choice based on entailment and contradiction logits.Source: §4.2 · p. 4

4.3 Forced-choice Paradigm

  • The forced-choice answers constrain each metaphor’s interpretation to a single axis, although other valid interpretations may exist.This motivates also examining generation performance.Source: §4.3 · p. 5

4.4 Human Performance

  • Ten volunteers completed the test set without training examples, with examples divided into shuffled partitions and uncertainty marking allowed.Participants included native and non-native English speakers; they were instructed to flag unfamiliar vocabulary or cultural references.Source: §4.4 · p. 5
  • The study evaluates zero-shot and fine-tuned test accuracy, with fine-tuned results averaged across five seeds.The supplied table caption identifies the metric and averaging procedure but provides no accuracy values.Source: Table 4 · p. 5

5 Results

Fine-tuning brought the best model to within 5% of human performance, while zero-shot models remained 26 percentage points behind. Prompting changes had limited effects, and GPT-3 Davinci’s generated interpretations were often inconsistent.

5.1 Inference Results

  • GPT-3 models outperformed GPT-2 and GPT-neo in zero-shot accuracy, with performance correlating with GPT-3 model size; BERT and RoBERTa fell within the GPT-3 range.GPT-3 Davinci achieved the highest zero-shot test accuracy.Source: §5.1 · p. 5
  • Fine-tuning improved all models; RoBERTa came within 5% of human performance, while the best zero-shot model remained 26 percentage points behind humans.GPT-3 models improved by about 13 percentage points and BERT/RoBERTa by about 25 points; GPT-3 Davinci was not fine-tuned due to budget.Source: §5.1 · p. 6
  • Suffix prompting improved baseline performance by 1–2%, while adding example pairs was generally ineffective.Figure 2 compares prompting methods with autoregressive models.Source: §5.1 · p. 6, Figure 2 · p. 6
  • Backward accuracy was lower than forward accuracy; GPT-3 Curie dropped 7% in the zero-shot case, suggesting reverse phrase selection is harder for models.Source: §5.1 · p. 6
  • Under paired evaluation, autoregressive models’ performance plummeted, masked models also lost accuracy, and human scores were least affected.Source: §5.1 · p. 6

5.2 Generation Results

  • GPT-3 Davinci was 50.8% accurate when literalized metaphors counted as incorrect; 37.7% of phrases had contradictory completions among four generations.At least one completion was correct for 78.1% of phrases, but all four were correct for only 19.3%; excluding literalized metaphors, accuracy was 63.9%.Source: §5.2 · p. 7

6 Performance and Error Analysis

Models often rely on answer probability rather than metaphorical context, and their errors vary with phrase length and the kind of commonsense reasoning required. Fine-tuning improves some categories, but sarcastic metaphors and obvious errors remain difficult.

6.1 Reliance on Probability of Answers

  • Across the analysis, models often predict from answer probability alone, sometimes making the same choice for paired sentences regardless of context.Fine-tuning improves this tendency, and GPT-3 best disentangles answer probability from the probability of an answer given its metaphor, but all three models remain strongly reliant on relative answer probability.Source: §6.1 · p. 7, §6 · p. 7

6.2 Other Factors Influencing Correctness

  • A longer context phrase correlates with lower correctness (r = -0.1544, p = 1.50 × 10−7), while more probable answers correlate with higher correctness (r = 0.1840, p = 3.50×10−10).The answer-probability association is consistent with models tending to select the answer that was already more probable.Source: §6.2 · p. 7

6.3 Qualitative Analysis of Error Trends

  • As models improve, most gains come from object, visual, and social commonsense, while cultural examples show little improvement with training.Humans and trained models also tend to find object and visual commonsense metaphors easier to interpret.Source: §6.3 · p. 8, Table 6 · p. 8
  • Sarcastic metaphors can mislead both humans and models when simple word associations override reading the whole sentence.For example, “bubbly as still water” can mean “bland,” rather than “vivacious.”Source: §6.3 · p. 8
  • GPT-3 Curie shared only 13 of 64 human errors and made many obvious errors, unlike human mistakes involving rare vocabulary or unfamiliar cultural references.One model error interpreted “The ball is a big red sun” as meaning the ball was small.Source: §6.3 · p. 8

7 Related work

Prior work has emphasized metaphor identification and paraphrase, while interpretation of novel figurative language remains less studied. Fig-QA differs through its Winograd-style format and focus on creative, non-conventionalized metaphors.

7.1 Figurative Language Identification

  • Most metaphor identification work uses conventionalized expressions, while word-level annotation poorly captures novel metaphors and extended conceptual metaphors.The cited work also includes probing models for metaphor recognition, but interpretation is described as more difficult and less studied.Source: §7.1 · p. 8

7.2 Figurative Language Interpretation

  • Metaphor interpretation has received less attention than detection, and paraphrase may miss intended emotion and interactions among a metaphor’s subject, relation, and object.Prior narrative work studies plausible continuations of figurative language in stories or dialogues; this work expands study of non-conventionalized metaphors.Source: §7.2 · p. 9

7.3 Other Figurative Language Datasets

  • Fig-QA’s Winograd-style paired choices guard against success based only on memorized word associations, and its creative phrases require more detailed commonsense knowledge.Crowdworkers were instructed to create phrases that may not have appeared in training corpora; unlike riddles, the phrases are not intended to be difficult when properly contextualized.Source: §7.3 · p. 9

7.4 Human Language Processing

  • Even when one answer is obviously wrong if the input is understood, language models still trail human performance on this task.The discussion notes that models excel at word-association tasks, but the dataset includes creative phrases that may take humans longer to process, especially non-native English speakers.Source: §7.4 · p. 9

8 Conclusion

The paper presents a Winograd-like benchmark built from creative human-written metaphors to assess language models’ figurative reasoning. On this dataset, zero-shot models remain far behind humans, though fine-tuning can yield strong task performance.

8 Conclusion

  • The paper introduces a Winograd-like task based on a large collection of creative metaphors written by humans to test language models’ figurative reasoning.The task tests reasoning about figurative language.Source: §8 · p. 9
  • Language models show a large zero-shot performance gap from humans on this dataset, but can perform well on this particular task after fine-tuning.Source: §8 · p. 9
  • The authors identify multimodal metaphors and figurative language in other languages as directions for future work.The current dataset contains only text-based metaphors.Source: §8 · p. 9

9 Ethical Considerations

The paper discusses potential harms from figurative language and limits its dataset to English. It also reports artefact licensing and the compute resources used.

9.1 Potential Risks

  • Figurative language can harm minority and historically disadvantaged groups, and potentially offensive examples were removed based on the authors’ judgment.Generically insulting examples were retained when they did not target a particular individual.Source: §9.1 · p. 10
  • Fig-QA contains only English examples, and the authors leave investigation of other languages to future work.The authors describe the dataset as preliminary and note that all authors speak English.Source: §9.1 · p. 10

9.2 Terms of Use of Artefacts Used

  • The paper lists licenses for the additional datasets and models used, including Apache 2.0 for WinoGrande and BERT and MIT for FEVER-NLI.GPT-3 outputs were obtained through the public API.Source: §9.2 · p. 10

9.3 Computational Infrastructure and Computing Budget

  • The experiments generally used no more than two GPUs and required an estimated 20 GPU hours.The reported computing budget was roughly 100 USD, with roughly 20 USD spent on GPT-3 API credits.Source: §9.3 · p. 10

A Crowdsourcing Details

Crowdworkers generated creative metaphor pairs and literal interpretations through a staged Mechanical Turk process, with manual validation before examples entered the dataset.

A Crowdsourcing Details

  • Workers recruited through Amazon Mechanical Turk had approval ratings above 98% and more than 1,000 completed HITs; collection used two stages.The first stage yielded 1,458 training examples and all development and test examples; the second added the remaining 6,558 training examples.Source: A Crowdsourcing Details · p. 13
  • Workers were paid $0.33 per pair and asked to generate three pairs at a time.The authors estimated an hourly rate of $19.80 based on their own pilot timing, which varied with example creativity.Source: A Crowdsourcing Details · p. 13
  • Instructions requested rare or creative metaphor pairs with opposite or very different meanings, plus literal sentences with the same meanings.The task interface showed examples and provided three random words to help workers get unstuck.Source: A Crowdsourcing Details · p. 13
  • The interface repeated each start phrase before its metaphor fields as workers typed, which the authors found reduced improperly formatted responses.Workers filled five fields for each set of three examples: a start phrase, two metaphorical phrases, and two literal interpretations.Source: A Crowdsourcing Details · p. 13
  • Collected form responses were converted into sentence pairs and each pair was manually validated before dataset inclusion.The authors launched batches of tasks until they had collected the desired quantity of data.Source: A Crowdsourcing Details · p. 13

B Invalid Examples

The appendix describes criteria for excluding invalid examples and gives table captions for accuracy breakdowns and generated completions, without reporting their cell values in these passages.

B Invalid Examples

  • Examples were excluded for being nonsensical, unintelligible due to grammar or spelling errors, or noncompliant with the task format.Tables 7–9 are identified as examples rejected for these respective reasons.Source: B Invalid Examples · p. 13, Table 7 · p. 13, Table 8 · p. 13
  • The authors sought to exclude offensive content and personally identifiable information and omitted WorkerID and other potentially identifying information..Source: B Invalid Examples · p. 13

C Backward accuracies

  • Table 10 is captioned as reporting zero-shot and finetuned backward autoregressive model accuracies on the test set.Source: Table 10 · p. 14

D Paired accuracies

  • Table 11 is captioned as test-set accuracy for pairs where both sentences are correct, with results from one run.Source: Table 11 · p. 14

E.1 Subject; E.2 Relation; E.3 Object

F.1 Subject; F.2 Object

  • Tables 15–16 break down accuracy by WordNet hypernyms in metaphor subjects and objects, showing hypernyms with at least 10 occurrences.Source: Table 15 · p. 15, Table 16 · p. 15

G Generation examples

Related papers

Loading 2204.12632v2…