PaperlayerView exact revision on arXiv

Source-linked AI summary

FLUTE: Figurative Language Understanding through Textual Explanations

Tuhin Chakrabarty, Arkadiy Saakyan, Debanjan Ghosh, Smaranda Muresan

arXiv:2205.12404v3cs.CL

TL;DR

Figurative language remains a challenge for language models. FLUTE uses a model-in-the-loop approach to construct figurative-language data and explanations. T5 fine-tuned on FLUTE produced higher explanation scores than T5 trained on e-SNLI across all four figurative-language types.

  • Problem

    Figurative language remains a bottleneck for NLP, and models struggle with the physical, cultural, and social context in which it is embedded.

    Source: §1 · p. 1

  • Method

    FLUTE uses a model-in-the-loop framework to create figurative-language instances and explanations, including GPT-3-generated metaphor paraphrases.

    Source: §2.1.3 · p. 4, §6 · p. 9

  • Results

    Explanation scores for T5FLUTE were 51.1, 41.3, 24.9, and 22.2 points better than T5e-SNLI for Sarcasm, Simile, Metaphor, and Idiom, respectively.

    Source: §4 · p. 8

  • Takeaways & Limitations

    FLUTE offers a challenging testbed for experimentation toward models that understand figurative language through textual explanations.

    Source: §6 · p. 9

    The study does not explicitly evaluate the faithfulness of model-generated natural language explanations.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Figurative language understanding has been recently framed as a recognizing textual entailment (RTE) task (a.k.a. natural language inference, or NLI). However, similar to classical RTE/NLI datasets, the current benchmarks suffer from spurious correlations and annotation artifacts. To tackle this problem, work on NLI has built explanation-based datasets such as e-SNLI, allowing us to probe whether language models are right for the right reasons.Yet no such data exists for figurative language, making it harder to assess genuine understanding of such expressions. To address this issue, we release FLUTE, a dataset of 9,000 figurative NLI instances with explanations, spanning four categories: Sarcasm, Simile, Metaphor, and Idioms. We collect the data through a model-in-the-loop framework based on GPT-3, crowd workers, and expert annotators. We show how utilizing GPT-3 in conjunction with human annotators (novices and experts) can aid in scaling up the creation of datasets even for such complex linguistic phenomena as figurative language. The baseline performance of the T5 model fine-tuned on FLUTE shows that our dataset can bring us a step closer to developing models that understand figurative language through textual explanations.

1 Introduction

Figurative language remains difficult for language models, and existing figurative NLI benchmarks lack explanations that can help assess whether predictions are genuinely understood. FLUTE addresses this gap with a 9,000-instance benchmark and experiments showing higher-quality explanations from a T5 model trained on FLUTE than from one trained on e-SNLI.

1 Introduction

  • Existing figurative NLI benchmarks can exploit spurious correlations, while no explanation-based dataset for figurative language was available to assess genuine understanding.The cited discussion links such artifacts in NLI to brittleness on out-of-domain or adversarial examples.Source: §1 · p. 1
  • FLUTE provides 9,000 figurative sentence pairs with entailment or contradiction labels and explanations, covering sarcasm, simile, metaphor, and idiom.Both labels and explanations concern the figurative expression itself.Source: §1 · pp. 1–2
  • FLUTE uses GPT-3 to generate literal text with few-shot prompts, with crowdworkers making minimal edits and experts checking quality.The framework is presented as a way to scale dataset creation with limited human involvement.Source: §1 · p. 2
  • T5 fine-tuned on FLUTE produced higher-quality explanations than T5 trained on e-SNLI in automatic and human evaluations.The experiments jointly predict entailment or contradiction labels and explanations.Source: §1 · pp. 2–3

2 Model-in-the-loop for building FLUTE

FLUTE is built by pairing literal premises with figurative hypotheses and generating entailment or contradiction examples through model-in-the-loop workflows. The construction varies by figurative-language type, while explanations are generated and expert-checked for each instance.

2 Model-in-the-loop for building FLUTE

  • FLUTE pairs literal premises with figurative hypotheses and supplies entailment or contradiction labels and explanations for each instance.The paper describes separate construction methods for each figurative-language type and for explanations.Source: §2 · p. 3

2.1.1 Sarcasm

  • For sarcasm, GPT-3 paraphrases negative-emotion literal sentences, then crowdworkers minimally edit the paraphrases into sarcastic contradictions.Experts check the paraphrases and resulting sarcasm; three experts review outputs, and 25% of sarcastic instances were edited for quality.Source: §2.1.1 · p. 3

2.1.2 Simile

  • For similes, the pipeline replaces the simile with its property, prompts GPT-3 for a literal paraphrase, then generates contradictions by inverting the property.The benchmark includes 750 entailment and 750 contradiction pairs.Source: §2.1.2 · p. 4

2.1.4 Idioms

  • For idioms, GPT-3 jointly generates paraphrases and contradictions using the idiom and its meaning, with experts verifying and editing outputs.The benchmark contains 1,000 entailment and 1,000 contradiction pairs; experts edited 23% of generations.Source: §2.1.4 · p. 5

2.2 FLUTE: Generating Textual Explanations

  • GPT-3 generates explanations from the premise, hypothesis, and label; idiom prompts also include the idiom and its meaning.Three experts check explanations for correctness and adequate reasoning detail, editing 21% for sarcasm paraphrases and 27%, 40%, and 10% for simile, metaphor, and idiom respectively.Source: §2.2 · p. 5

3 Experimental Setup

The experiments compare T5 models fine-tuned on e-SNLI and FLUTE, with each jointly predicting an entailment label and explanation. Evaluation combines explanation-aware accuracy, rationale utility, and human judgments.

3.1 Models

  • A joint self-rationalizing T5 model outputs an entailment label followed by a rationale; the two variants are fine-tuned on e-SNLI or FLUTE.The instruction asks whether the figurative hypothesis is entailed or contradicted by the literal premise and requests a sentence explaining the decision.Source: §3.1 · pp. 5–6

3.2 Evaluation Setup

  • The FLUTE test set contains 1,500 instances: 750 sarcasm examples and 250 each for simile, metaphor, and idiom.The e-SNLI model uses 366,603 training examples, while the FLUTE model uses 7,035 samples across the four figurative-language types.Source: §3.2 · p. 6, §3.1 · p. 6
  • Explanation score averages BERTScore and BLEURT; label accuracy is reported at explanation-score thresholds of 0, 50, and 60.Accuracy@0 is ordinary label accuracy, while Accuracy@50 counts correct labels only when explanation score exceeds 50.Source: §3.2 · p. 6
  • Rationale quality measures the change in gold-label prediction accuracy when a rationale is supplied alongside the input.The evaluation uses predicted rationales from each joint model and also computes quality with gold rationales.Source: §3.2 · p. 6
  • Human evaluation asks three workers to judge whether each model’s explanation justifies its answer on 200 instances; workers also categorize shortcomings when they do not answer Yes.Workers’ scores are averaged as Hscore; inter-annotator agreement was Krippendorff’s α = 0.45, indicating moderate agreement.Source: §3.2 · p. 6, Table 3 · p. 7

4 Results and Discussion

Across the reported evaluations, T5 fine-tuned on FLUTE produces higher-quality explanations than the e-SNLI-trained model. The results include explanation-aware accuracy, human judgments, rationale utility, and explanation shortcomings.

4 Results and Discussion

  • T5FLUTE has significantly higher accuracy scores than T5e-SNLI at each explanation-score threshold for every figurative-language type.For T5e-SNLI, accuracy drops almost by half above a 50% explanation-score threshold and falls further above 60%.Source: §4 · p. 7
  • Predicted rationales from the e-SNLI-trained model are less useful for gold-label prediction than those from the FLUTE-trained model.Predicted-rationale simulatability scores are negative, whereas positive scores for gold rationales indicate that FLUTE explanations are useful for model predictions.Source: §4 · p. 7, Table 4 · p. 7
  • T5FLUTE’s Hscore exceeds T5e-SNLI by 51.1, 41.3, 24.9, and 22.2 points for sarcasm, simile, metaphor, and idiom, respectively.Workers answered solid Yes 43.4% more often on average and No 28.5% less often for T5FLUTE explanations; this held despite e-SNLI’s larger dataset.Source: §4 · p. 8
  • Insufficient justification is the most common explanation shortcoming, and T5e-SNLI explanations are at least as often marked deficient across categories.T5e-SNLI explanations are also more frequently judged too trivial, often following a template rather than explaining the reasoning.Source: §4 · p. 8, Figure 2 · p. 8

5 Related Work

FLUTE builds on figurative-language RTE benchmarks and explanation datasets while adding broader coverage and explanations for each NLI instance. Its construction also differs from e-SNLI’s crowdsourcing approach.

5 Related Work

  • Compared with Chakrabarty et al. (2021), FLUTE offers greater diversity, explanations for each NLI instance, and multiple expert quality checks.Unlike Big-bench and IMPLI, FLUTE’s non-entailment examples contradict the metaphor rather than the non-metaphoric sentence content.Source: §5 · p. 8
  • Unlike e-SNLI’s crowdsourced creation, FLUTE uses a model-in-the-loop framework; the paper reports that T5 trained on FLUTE performs notably better for explanation generation.Related explanation datasets include CoS-E, Movie Reviews, and e-SNLI.Source: §5 · p. 8

6 Conclusion

FLUTE is a model-in-the-loop benchmark for figurative-language understanding, with textual explanations across four categories. Baseline models trained on e-SNLI perform poorly, while T5 fine-tuned on FLUTE moves toward understanding figurative language through explanations.

  • T5 fine-tuned on FLUTE moves toward models that understand figurative language through textual explanations, while e-SNLI-trained baseline models perform poorly.The authors present this as a step toward that goal, not as evidence that the goal has been achieved.Source: §6 · p. 9
  • FLUTE contains 9,000 figurative-language instances spanning sarcasm, simile, metaphor, and idioms, collected through a model-in-the-loop framework.The dataset includes free-form textual explanations.Source: §6 · pp. 8–9
  • The authors propose explanation generation for figurative language as a future research direction and position FLUTE as a challenging testbed.Source: §6 · p. 9

Limitations

The dataset covers four figurative-language types but is an initial step toward the broader range of cultural knowledge and contexts involved. Its sarcasm subset captures a common incongruity, and explanation faithfulness is not explicitly evaluated.

  • FLUTE covers four types and is described as a first step toward capturing the wider range of cultural knowledge and contexts involved in figurative language.Source: Limitations · p. 9
  • The sarcasm subset captures the most common context-sentiment incongruity, but other forms include situational, underplayed, and dramatic sarcasm.Examples and explanations for those forms will differ.Source: Limitations · p. 9
  • The study does not explicitly evaluate the faithfulness of model-generated natural-language explanations, though the authors hope to assess it using contemporaneous methods.The authors cite Sia et al. (2022) and Chan et al. (2022).Source: Limitations · p. 9

Ethics Statement

The authors report manually inspecting FLUTE for toxic text and paying Mechanical Turk workers at a rate compliant with the U.S. minimum hourly wage.

Ethics Statement

  • The authors manually inspected FLUTE with experts and report that it contains no toxic text despite using web-trained language models.Source: Ethics Statement · p. 9
  • Amazon Mechanical Turk workers were paid $15 per hour, which the authors state complies with the U.S. minimum hourly wage.Source: Ethics Statement · p. 9

A.1.1 Hyperparameters for the sarcasm dataset

The appendix specifies GPT-3 settings and prompts used in creating sarcasm data and explanations, including prompts for paraphrase generation and entailment explanations.

A.1.1 Hyperparameters for the sarcasm dataset

  • Paraphrase and explanation generation both use temperature 1, a 100-token limit, top p 0.9, and frequency and presence penalties of 0.5.Paraphrase generation uses best of 1; explanation generation additionally stops at a period.Source: §A.1.1 · p. 13

A.1.2 Prompts for generation of Paraphrase from which Sarcasm is created

  • The paraphrase prompt asks for creative, humorous paraphrases that preserve the literal sentence’s meaning while reflecting its associated emotion.Source: §A.1.2 · p. 13

A.1.3 Prompts for generation of Explanation for Paraphrase from which Sarcasm is created (Entailment)

  • For entailment examples, the prompt asks for natural-language explanations that justify the entailment between premise and hypothesis.Source: §A.1.3 · p. 13

You will be presented with examples of some literal

  • The sarcasm-explanation prompt asks annotators to explain why a sarcastic sentence conflicts with its literal counterpart.The examples describe positive reactions to bugs or nighttime noises as inconsistent with the negative or threatening situations.Source: You will be presented with examples of… · pp. 13–14

A.2.1 Hyperparameters for the simile dataset

The simile dataset uses GPT-3-Davinci-002 with separate generation settings for paraphrases, contradictions, and explanations.

A.2.1 Hyperparameters for the simile dataset

  • GPT-3-Davinci-002 generates simile data with separate hyperparameter configurations for paraphrases, contradictions, and explanations.The supplied settings specify temperature, token limits, top-p, and penalties for each generation task.Source: §A.2.1 · p. 14

A.2.2 Challenging instances for the simile dataset

The appendix describes how the authors select challenging simile instances and prompt GPT-3 to generate figurative-language paraphrases, contradictions, and explanations. It also reports generation settings and prompt designs for metaphor and idiom data.

A.2.2 Challenging instances for the simile dataset

  • The authors select simile instances with the lowest averaged RoBERTa logits, targeting cases where the model confuses entailment with contradiction.The selected examples are often ironic similes.Source: §A.2.2 · p. 14

A.2.3 Prompts for Simile Paraphrase Generation

  • Simile paraphrase prompts ask GPT-3 to preserve specified words while producing a creative paraphrase consistent with the literal sentence.The prompt includes example paraphrases.Source: §A.2.3 · p. 14

A.2.4 Prompts for Simile Contradiction Generation

  • Simile contradiction prompts instruct GPT-3 to invert a sentence's meaning with respect to a property using a minimal edit.The prompt illustrates inversion of “impenetrable and inescapable” to “easy to walk into and exit from.”Source: §A.2.4 · pp. 14–15

A.2.5 Prompts for Simile Contradiction Explanation Generation; A.3 Prompts for Simile Entailment Explanation Generation

  • For similes, contradiction explanations justify why the contradictory sentence conflicts with the simile, while entailment explanations justify why an entailment sentence implies it.The examples explain the simile’s figurative meaning in relation to the paired sentence.Source: §A.2.5 · p. 15, §A.3 · p. 15

A.4.1 Hyperparameters for the metaphor dataset; You will be presented with examples of some

  • Metaphor-generation prompts include creative paraphrases consistent with the literal input, with GPT-3-Davinci-002 settings specified separately for paraphrases, contradictions, and explanations.The appendix gives distinct temperatures and token limits for the three tasks.Source: §A.4.1 · p. 15, You will be presented with examples of… · p. 15

You will be presented with examples of sentences; You will be provided with a sentence containing

A.4.3 Prompts for Metaphor Contradiction Generation; You will be provided with a sentence containing

  • Metaphor contradiction prompts request minimal edits, and the appendix contrasts this approach with prior examples that fit a different word sense to the context.The table caption identifies the contrast between prior examples and contradictions generated for this paper.Source: §A.4.3 · p. 15, Table 6 · p. 16

A.5.1 Hyperparameters for the idiom dataset; You will be presented with examples of some input

B Details of Human evaluation

The human evaluation checks whether crowdworkers recognize each NLI relation before judging model-generated explanations. It compares explanations from T5FLUTE and T5e-SNLI using worker ratings and reported shortcomings.

B Details of Human evaluation

  • Workers first identify whether each literal–figurative pair is entailment or contradiction, and submissions are accepted only when that relation is identified correctly.The interface example is shown in Figure 4.Source: B Details of Human evaluation · p. 17, Figure 4 · p. 18
  • Workers compare two explanations without knowing which model produced each one and rate whether each explanation justifies the answer.The options are Yes, Weak Yes, Weak No, and No.Source: B Details of Human evaluation · p. 17
  • For ratings below Yes, workers can identify shortcomings including common-sense violations, insufficient justification, input inaccuracies, triviality, and verbosity.Figure 5 plots shortcomings as a percentage of the sample by type; lower values mean fewer shortcomings.Source: B Details of Human evaluation · p. 17, Figure 5 · p. 21
  • Scores are averaged across three workers per entry, then averaged by figurative-language type for each model.The evaluation also displays shortcomings by figurative-language type in Figure 5.Source: B Details of Human evaluation · p. 17, Figure 5 · p. 21

Related papers

Loading 2205.12404v3…