PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-lingual and Multi-cultural Figurative Language Understanding

Anubha Kabra, Emmy Liu, Simran Khanuja, Alham Fikri Aji, Genta Indra Winata, Samuel Cahyawijaya, Anuoluwapo Aremu, Perez Ogayo, Graham Neubig

arXiv:2305.16171v1cs.CL

TL;DR

Figurative-language research and resources have focused largely on English, even though expressions draw on culturally situated concepts. The paper creates a seven-language dataset and evaluates multilingual models, finding poor zero-shot performance, especially for lower-resource languages.

  • Problem

    Figurative-language processing is understudied outside English, and existing English datasets may not be culturally relevant to other languages.

    Source: §1 · pp. 1–2

  • Method

    The authors create MABL, a seven-language figurative-language dataset, and use it to analyze cross-language patterns and evaluate multilingual models.

    Source: §1 · p. 2

  • Results

    Zero-shot multilingual-model performance is relatively poor, especially for lower-resource languages, while figurative expressions often invoke culture-specific concepts.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings highlight the importance of culturally relevant training data that captures how people view concepts within their cultures.

    Source: §1 · p. 2

    The authors note that it is difficult to disentangle common sense from culture because annotators may treat culture-specific facts as common sense.

    Source: §9 · p. 9

Abstract

from arXiv · show

Figurative language permeates human communication, but at the same time is relatively understudied in NLP. Datasets have been created in English to accelerate progress towards measuring and improving figurative language processing in language models (LMs). However, the use of figurative language is an expression of our cultural and societal experiences, making it difficult for these phrases to be universally applicable. In this work, we create a figurative language inference dataset, \datasetname, for seven diverse languages associated with a variety of cultures: Hindi, Indonesian, Javanese, Kannada, Sundanese, Swahili and Yoruba. Our dataset reveals that each language relies on cultural and regional concepts for figurative expressions, with the highest overlap between languages originating from the same region. We assess multilingual LMs' abilities to interpret figurative language in zero-shot and few-shot settings. All languages exhibit a significant deficiency compared to English, with variations in performance reflecting the availability of pre-training and fine-tuning data, emphasizing the need for LMs to be exposed to a broader range of linguistic and cultural variation during training.

1 Introduction

Figurative language depends on culturally situated concepts, yet empirical work has focused largely on English. The paper introduces MABL, a seven-language dataset, and finds both culture-specific expressions and weak multilingual-model performance, especially for lower-resource languages.

1 Introduction

  • Figurative language carries cultural references and judgments that vary across cultures, as illustrated by contrasting English and Hindi associations with warmth and coldness.The paper notes that such differences may reflect cultural values, history, or other factors.Source: §1 · p. 1
  • MABL contains 6,366 figurative-language expressions in seven languages, addressing the limited study and resources for non-English figurative language.The authors also report that English figurative-language datasets may lack cultural relevance for other languages.Source: §1 · pp. 1–2
  • Figurative expressions often invoke culturally important entities, including food, mythology, famous people, and region-specific plants and animals.Understanding them involves knowing which concepts are culturally significant and how they are viewed in that culture.Source: §1 · pp. 1–2
  • Zero-shot multilingual-model performance is relatively poor, especially for lower-resource languages, and target-language examples can help larger models more in those languages.The authors suggest concept shift plays a larger role than cross-lingual transfer in these cases.Source: §1 · p. 2

2 Linguistic and Cultural Biases of Existing Figurative Language Datasets

A pilot study tested whether translating an English figurative-language dataset could provide culturally relevant examples in other languages. The translations often had accuracy or cultural-relevance problems, motivating collection from native speakers.

2 Linguistic and Cultural Biases of Existing Figurative Language Datasets

  • The pilot translated Fig-QA’s development set into French, Japanese, and Hindi, then asked annotators to assess translation correctness and cultural relevance.Cultural relevance meant whether an expression would make sense to a native speaker from the language’s predominant culture.Source: §2 · p. 2
  • Translated Fig-QA examples included many incorrect translations, particularly in Hindi and Japanese, and some culturally irrelevant examples.The passage attributes problems to expressions that do not translate directly and to implicitly assumed cultural knowledge.Source: §2 · p. 2, Table 2 · p. 2
  • The authors argue that native speakers familiar with a culture can capture shifts in the topics discussed across cultures.This motivates collecting natural figurative-language examples from native speakers.Source: §2 · p. 2

3 The MABL Dataset

MABL covers seven languages selected to span linguistic and cultural diversity. Native speakers generated paired figurative examples, which other fluent workers validated; the released samples serve as test sets without explicit training data.

3.1 Language Selection

  • The seven selected languages span five typologically diverse families across four countries, with selection also shaped by annotator availability and speaker population.The languages are Hindi, Yoruba, Kannada, Sundanese, Swahili, Indonesian, and Javanese.Source: §3.1 · pp. 2–3

3.2 Dataset Collection

  • Native speakers generated paired metaphors beginning with the same words but having different meanings, along with literal interpretations.Novel metaphors were allowed if native speakers could readily understand them.Source: §3.2 · p. 3
  • Separate fluent workers validated samples, rejecting incoherent, offensive, or incorrectly formatted examples; the released samples are test sets with no explicit training set.The paper attributes this release choice to the limited number of samples in each language.Source: §3.2 · p. 3

4 Dataset Analysis

The dataset analysis finds culturally specific concepts and limited overlap among the objects used in figurative expressions, with regional patterns in concept similarity. Annotators also identify differences in the commonsense knowledge associated with examples.

4.1 Concepts expressed

  • Only six objects occur in every language, while the identified concepts include culturally specific foods, religious figures, and popular-culture references.The authors caution that lemmatization may leave related concepts counted separately.Source: §4.1 · p. 3
  • Languages from the same region tend to group together by concept similarity, while English is most similar to Swahili in this dataset.The authors note that English has no particularly closely related language in the dataset and identify nature terms and Christianity references in Swahili as possible explanations for its similarity to English.Source: §4.1 · p. 3, Table 4 · p. 4

4.2 Commonsense Categories

  • Social and object knowledge are the most common annotated categories; Yoruba has especially many social examples, while few examples are labeled cultural.The authors suggest category judgments may differ, since some culturally specific examples could be classified as object or social knowledge.Source: §4.2 · p. 4, Table 6 · p. 4

4.3 Cross-lingual concept distribution

  • XLM-R clusters sentences by language, whereas LaBSE groups sentences from multiple languages by conceptual similarity; examples within clusters reflect language-specific cultural traits.Rice is commonly mentioned in Indonesian languages, where it is a staple food.Source: §4.3 · p. 4
  • A targeted scan of food, weather/season, and friendship metaphors (230, 111, and 307 sentences) surfaced regional references including neem, tamarind, and mythological characters.The authors note that some such comparisons may not be understandable to non-native speakers.Source: §4.3 · pp. 4–5

5 Evaluation and Results

The evaluation finds that multilingual models face both cross-lingual transfer and concept-shift challenges, with concept shift generally larger. Few-shot gains are usually modest, while performance varies with model scale, language resources, and the language being evaluated.

5.1.2 Zero-shot transfer results

  • The translate-test comparison translates target-language test sets into English to distinguish cross-lingual transfer gaps from concept-shift gaps.The authors note that machine-translation errors make these gap estimates upper bounds.Source: §5.1.2 · p. 5
  • Concept-shift gaps exceeded cross-lingual transfer gaps for every language except Swahili across models; XLM-R’s zero-shot Indonesian accuracy was 83.6%.For Indonesian, translating the test set to English did not help XLM-R, whereas it did help mBERT.Source: §5.1.2 · p. 6, Table 8 · p. 7
  • Accuracy increased with model and training-data size, especially for Hindi and Indonesian, while lower-resource languages showed more modest gains.Larger-model accuracy reached 67.58% for Hindi and 78.09% for Indonesian.Source: §5.1.2 · p. 6

5.2.2 Few-shot results

  • Most few-shot accuracy gains were <1%, but 50 examples raised Kannada XLM-Rlarge by 3.83%, Swahili XLM-Rbase by 1.10%, and Sundanese mBERTbase by 2.33%.The authors attribute the usually small gains to the diversity of cultural references, many of which do not repeat.Source: §5.2.2 · p. 6, §5.2.1 · p. 6

5.3 Evaluation of Large Language Models

  • GPT-3 performed poorly on most languages in zero-shot evaluation, despite 74.86% accuracy on the English development set.The authors report both a high concept-shift gap and a comparatively higher cross-lingual gap for GPT-3.Source: §5.3 · p. 7

6 Error Analysis

The error analysis uses English translation to examine model outcomes and compares accuracy on cultural examples with overall accuracy. It also reports that adding target-language examples is most beneficial to XLM-Rlarge beyond 10 examples.

6.1 Effect of English MT

  • Across languages, 53% of examples were correct both before and after translation, 19% became correct after translation, 15% were difficult to translate or mistranslated, and 12% remained incorrect.The analysis concerns XLM-Rlarge predictions before and after English translation.Source: §6.1 · pp. 7–8, Table 9 · p. 7
  • Adding more than 10 target-language examples was most beneficial for XLM-Rlarge.Figure 3 covers adding up to 50 target-language examples to English training data; exact results are in Appendix F.Source: Figure 3 · p. 8

6.2 Cultural Examples

  • XLM-Rlarge accuracy was 61.68% on cultural examples versus 63.99% overall; the preliminary analysis suggests named entities may not be the only cultural challenge.The analysis covers cultural examples in the commonsense categories discussed in Section 4.2.Source: §6.2 · p. 8

7 Related Work

Prior figurative-language research has largely focused on English, while multilingual metaphor work covered a small set of higher-resource languages. The paper positions its dataset as addressing the lack of a large-scale multilingual resource for studying cultural variation.

7.1 Figurative Language

  • English figurative-language work has often trained models to detect metaphors in idioms and conventionalized phrases, rather than test interpretation of figurative meaning.Work on metaphor understanding includes paraphrase detection and generation.Source: §7.1 · p. 8
  • Some studies of metaphor interpretation do not account for context-dependent meanings, while Liu et al. (2022) introduced a dataset requiring inference of a metaphor’s correct meaning.The passage characterizes this as testing deeper semantic understanding.Source: §7.1 · p. 8
  • Earlier multilingual metaphor studies focused on English, Russian, and Spanish, leaving no large-scale dataset for examining cultural variation across metaphors.Those studies used cross-lingual techniques with newspapers and dictionaries.Source: §7.1 · p. 8

8 Conclusion

The paper introduces MABL, a figurative-inference dataset spanning seven languages, and reports substantial cross-language variation in the cultural references used. It finds multilingual models have considerable room for improvement and that cross-cultural shift may contribute to performance degradation from English.

8 Conclusion

  • MABL is a figurative-inference dataset across seven languages, addressing an area the authors describe as relatively understudied in NLP.The conclusion notes that figurative language is especially understudied for non-English languages.Source: §8 · p. 9
  • Figurative expressions vary considerably across languages in the objects they invoke, including food, mythology, religion, and famous figures or events.The authors suggest this variation is likely due to differences in cultural common ground.Source: §8 · p. 9
  • The authors report that multilingual models have substantial room to improve and that cross-cultural shift may significantly contribute to degradation from English.They propose figurative language as a testbed for examining cross-linguistic and cross-cultural variation.Source: §8 · p. 9

9 Limitations

The paper qualifies its cross-cultural analysis through limited annotator and language coverage, narrow example syntax, translation-based analysis, and English-centered transfer evaluation.

9 Limitations

  • Data cover only 2–3 annotators per language across seven languages, and many examples use the narrow “<X> is like <Y>” template.Source: §9 · p. 9
  • Machine-translated test sets may overestimate concept-shift errors because translations can be erroneous, and limited resources prevented human translations.Source: §9 · p. 9
  • English may not be the best transfer source; machine-translating training sets was untested, and another source-language combination may transfer better.Source: §9 · p. 9
  • Annotators may classify culture-specific facts as common sense, making the distinction difficult to disentangle across cultures.Source: §9 · p. 9

A Selected Languages

B Instructions for Annotators

  • Annotators generated metaphor pairs freely, with encouragement to emphasize their culture, rather than receiving prompts based on English metaphorical frames.Source: B Instructions for Annotators · p. 11

C Unique Concepts in Different Languages

D Jaccard Similarity between Concepts

E Training Details

The training appendix specifies the hyperparameter search and selected configurations, reports seed selection, and notes that one experiment reused earlier settings without further optimization.

E Training Details

  • Grid search covered 10, 20, or 30 epochs; learning rates 2 × 10−4, 5 × 10−4, 2 × 10−5, 5 × 10−5, 2 × 10−6, or 5 × 10−6; and batch sizes 32 or 64.Source: E Training Details · p. 12
  • XLM-Rlarge used (20 epochs, 5 × 10−6, batch 32) and (30 epochs, 2 × 10−5, batch 64); mBERTbase used (30 epochs, 5 × 10−5, batch 64).Each model used an A6000 GPU, and each run took on the order of a few minutes.Source: E Training Details · p. 12
  • Because most seeds performed near randomly on English development data, the five best seeds among 1–100 were selected by English development performance.Source: E Training Details · p. 12
  • Section 5.2.2 reused previously selected hyperparameters without further optimization, which may explain lower performance in some settings.Source: E Training Details · p. 12

F Few-shot Full Results

The appendix reports the scope of the few-shot experiment, illustrates figurative examples, and specifies which languages are represented in the commonsense accuracy table.

G Four-Quadrant Examples

F Few-shot Full Results

H Accuracy on Annotated Commonsense Categories

Related papers

Loading 2305.16171v1…