PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Understanding: Evaluating the Pragmatic Gap in LLMs' Cultural Processing of Figurative Language

Mena Attia, Aashiq Muhamed, Mai Alkhamissi, Thamar Solorio, Mona Diab

arXiv:2510.23828v2cs.CL

TL;DR

LLMs’ ability to process culturally grounded meaning remains poorly understood, especially for figurative language that depends on local knowledge and pragmatic nuance. The paper evaluates understanding, pragmatic use, and connotation across Arabic and English figurative expressions, finding a consistent performance hierarchy and greater difficulty using expressions appropriately than interpreting them.

  • Problem

    How well LLMs process culturally grounded meaning remains poorly understood, despite progress in multilingual text understanding and generation.

    Source: §1 · p. 1

  • Method

    The paper evaluates 22 open- and closed-source LLMs on contextual understanding, pragmatic use, and connotation tasks using Egyptian Arabic idioms, multidialectal Arabic proverbs, and English proverbs.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    Performance follows a hierarchy from English proverbs to Arabic proverbs to Egyptian idioms, while pragmatic-use accuracy is 14.07% lower than understanding accuracy.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    Figurative language serves as a diagnostic for cultural reasoning, revealing that models can often interpret figurative meaning but struggle to use it appropriately.

    Source: §7 · p. 9

    The evaluation covers idioms and proverbs, only a subset of figurative language phenomena.

    Source: Limitations · p. 10

Abstract

from arXiv · show

We present a comprehensive evaluation of the ability of large language models (LLMs) to process culturally grounded language, specifically to understand and pragmatically use figurative expressions that encode local knowledge and cultural nuance. Using figurative language as a proxy for cultural nuance and local knowledge, we design evaluation tasks for contextual understanding, pragmatic use, and connotation interpretation in Arabic and English. We evaluate 22 open- and closed-source LLMs on Egyptian Arabic idioms, multidialectal Arabic proverbs, and English proverbs. Our results show a consistent hierarchy: the average accuracy for Arabic proverbs is 4.29% lower than for English proverbs, and performance for Egyptian idioms is 10.28% lower than for Arabic proverbs. For the pragmatic use task, accuracy drops by 14.07% relative to understanding, though providing contextual idiomatic sentences improves accuracy by 10.66%. Models also struggle with connotative meaning, reaching at most 85.58% agreement with human annotators on idioms with 100% inter-annotator agreement. These findings demonstrate that figurative language serves as an effective diagnostic for cultural reasoning: while LLMs can often interpret figurative meaning, they face challenges in using it appropriately. To support future research, we release Kinayat, the first dataset of Egyptian Arabic idioms designed for both figurative understanding and pragmatic use evaluation.

1 Introduction

The paper uses figurative language to evaluate how LLMs handle culturally grounded meaning, addressing a gap in research focused mainly on comprehension rather than pragmatic use. It introduces a cross-lingual evaluation framework and Kinayat, and reports lower performance on Arabic proverbs and Egyptian idioms than on English proverbs.

1 Introduction

  • Prior work has emphasized explaining idioms and proverbs, while pragmatic use additionally requires context sensitivity, affective inference, and social appropriateness.The paper frames figurative expressions as culturally rooted and dependent on shared world knowledge and pragmatic reasoning beyond literal semantics.Source: §1 · p. 1
  • The evaluation tests figurative interpretation, appropriate use in context, and connotative and affective inference across 22 open- and closed-source models.It draws on Jawaher and MAPS for proverb interpretation and introduces Kinayat for Egyptian Arabic idioms.Source: §1 · pp. 1–2
  • Arabic proverb accuracy is 4.29% lower than English proverb accuracy, and Egyptian idiom performance is 10.28% lower than Arabic proverb performance.In the pragmatic-use task, accuracy is 14.07% lower than in understanding; adding an idiom sentence as context improves average precision by 10.66%.Source: §1 · p. 2

2 Related Work

Prior figurative-language benchmarks cover English, multilingual proverbs, and some Arabic resources, but the section describes limited integration of figurative language into Arabic cultural evaluation. The paper situates its broader evaluation alongside these existing datasets and studies.

2 Related Work

  • The related-work section presents a comparative summary of existing figurative-language datasets in Table 2.Source: §2 · p. 2, Table 2 · p. 3
  • English-focused resources assess figurative reasoning through multiple-choice questions, narrative continuation, metaphor interpretation, and pragmatic competence.The PUB benchmark tests distinguishing literal from contextually appropriate figurative meanings in conversation.Source: §2 · p. 2
  • Multilingual resources include MAPS for proverbs in six languages, ProverbEval for cultural proverbs in Ethiopian languages and English, and MABL for metaphor and simile comprehension.Source: §2 · p. 2
  • Arabic cultural benchmarks often omit figurative language or include it only in limited capacity, while Jawaher covers proverbs across 20 dialects and Fann or Flop evaluates Arabic poetry.The passage notes that ArabCulture includes five idiom samples per country and that PALM includes proverbs among its topics.Source: §2 · pp. 2–3
  • Prior studies also examine affective and social dimensions, including valence, arousal, concreteness, and metaphor effects on Arabic sentiment analysis.Source: §2 · p. 3

3 Methodology

The methodology evaluates figurative understanding and use through tasks spanning contextual reasoning, pragmatic use, negation, explanation generation, proverb completion, and connotation. It also describes how idiom-use examples and incorrect explanations are prepared and checked.

3 Methodology

  • The framework extends proverb understanding with negation, contextual reasoning, pragmatic use, and connotation labeling for Arabic idioms, Arabic proverbs, and English proverbs.Source: §3 · p. 3
  • For pragmatic use, GPT-4.1 generates Egyptian Arabic sentences from 150 Kinayat idioms and their explanations, after which a native speaker reviews them.The speaker modified 77 samples (51.3%) through dialectal adjustment, rephrasing, additions, or rewriting.Source: §3 · p. 3
  • The pragmatic-use multiple-choice task asks models to select the idiom matching a sentence, with a plausible incorrect idiom serving as the distractor.Figure 1 provides sample questions and English translations.Source: §3 · p. 3, Figure 1 · p. 4
  • Evaluation uses zero-shot prompts in English and a single run per experiment to maintain consistency across Arabic and English datasets.Source: §3 · p. 3
  • Incorrect explanations are generated with a general prompt or by changing one semantic role in a gold explanation to create a subtler error.Source: §3 · p. 4

Human Verification of Incorrect Distractors

  • A native Arabic speaker rated sampled incorrect explanations for linguistic and cultural plausibility, with average verification scores of 1.72 for idioms and 1.68 for proverbs.Source: Human Verification of Incorrect… · p. 4
  • The task suite also includes contextual MCQ understanding, selecting the incorrect explanation, generating explanations, completing proverbs, and predicting connotation from an expression or its explanation.Connotation labels are positive, negative, or neutral, and evaluation uses samples with unanimous human agreement.Source: Human Verification of Incorrect… · pp. 4–5

4 Experimental Setup

The experimental setup combines proverb and idiom datasets, 22 diverse LLMs, and task-specific evaluation procedures. The section describes dataset scope, model groups, and metrics for multiple-choice and generation tasks.

MulticulturAl Proverbs and Sayings (MAPS)

4.1 Datasets

  • Kinayat contains 325 Egyptian colloquial idioms with MSA explanations, sourced from a 1949 book; preprocessing excluded 10 entries and edited or supplemented some explanations.Source: MulticulturAl Proverbs and Sayings… · p. 5

4.2 Models

  • The evaluation includes 22 open- and closed-source models, including multilingual and Arabic-focused open models and several closed-source model families.Source: §4.2 · p. 5

4.3 Evaluation

  • Multiple-choice tasks are evaluated by mean accuracy with standard error, while generation tasks use BERTScore F1, LLM-as-a-Judge, and human evaluation.GPT-4.1 judges most models; Claude 3.5 Sonnet judges GPT models to avoid self-evaluation bias.Source: §4.3 · p. 5

5 Results

Models perform best on English proverbs, less well on Arabic proverbs, and worst on Egyptian Arabic idioms; pragmatic use and connotation judgments are additional challenges. Contextual information improves understanding, but strong comprehension does not guarantee appropriate use.

5.1 Knowledge and Understanding

  • 76.29% accuracy on Kinayat idioms was below 86.57% on Jawaher Arabic proverbs, while MAPS reached 90.86% without context and 95.66% with context.The authors note that context lengths may contribute to the English–Arabic gap, while idioms’ shorter, less informative contexts may contribute to their difficulty.Source: §5.1 · p. 6
  • Negation reduced average accuracy from 76.29% to 70.97% for idioms and from 86.57% to 82.71% for proverbs.GPT-4o outperformed Claude models on the negation task despite Claude models scoring higher on the original MCQ task.Source: §5.1 · p. 6
  • 75.43% average English-proverb completion accuracy exceeded 10.65% for Arabic proverbs, although models understood Arabic proverbs substantially better than they completed them.Claude 3.5 had the highest completion scores: 93.91% for English and 36.36% for Arabic proverbs.Source: §5.1 · p. 6

Model Ability to Generate Explanations

  • For generated explanations, average idiom scores were 0.65 BERTScore-F1 and 2.19 LLM-as-a-Judge, below proverb scores of 0.68 and 3.06, respectively.On a 220-explanation subset, human and automated scores matched exactly in 62.27% of cases; the automated evaluator’s average score was 0.21 points higher.Source: Model Ability to Generate Explanations · p. 6

5.2 Pragmatic Use

  • 64.45% average pragmatic-use accuracy was 14.07% lower than understanding; adding idiom sentences raised contextual understanding accuracy by 10.66% to 89.18%.Claude 3.5 Sonnet achieved the highest pragmatic-use accuracy, 85.33%.Source: §5.2 · p. 7

5.3 Connotations

  • Among idioms with unanimous annotator agreement, Claude-3.5-Sonnet reached at most 85.58% connotation-labeling accuracy, while judgments remained subjective across expressions.Only 104 of 150 idioms and 105 of 198 multidialectal proverbs had full inter-annotator agreement; evaluation was restricted to unanimous cases.Source: §5.3 · p. 7

6 Ablation and Linguistic Variation Analysis

The analyses find that distractor-generation strategy has limited effects overall, while performance varies across dialects, model groups, and model sizes. Error analysis shows that context can help with some idioms, but comprehension and pragmatic use remain dissociated.

6 Ablation and Linguistic Variation Analysis

  • 86.57% proverb accuracy changed to 86.36% with SRL-based distractors, while idiom accuracy rose from 76.29% to 78.52%; overall effects were minor.Claude models consistently outperformed GPT models on MCQ understanding, although GPT-4.1 generated the distractors and was not evaluated.Source: §6 · p. 8
  • UAE, MSA, Libya, and Oman were the highest-performing dialects on average, while Sudan, Mauritania, Qatar, and Yemen were lowest.UAE and MSA remained in the top four, and Sudan and Mauritania among the lowest three, across distractor-generation prompts; Egypt scored mid-range despite being high-resource.Source: §6 · p. 8
  • Multilingual models generally exceeded Arabic models by 4.72% on English tasks and 2.19% on Arabic tasks, but Arabic models led in completion and pragmatic use.Closed-source models performed best overall; the authors note their parameter counts are undisclosed and may be larger than the evaluated open-source models.Source: §6 · p. 9
  • Pragmatic-use performance correlated more strongly with model size (R2 = 0.6) than MCQ understanding did (R2 = 0.265).ALLaM-7B-Instruct-preview was the most parameter-efficient model for Arabic understanding and pragmatic use; Qwen2.5-7B-Instruct was most efficient for English understanding.Source: §6 · p. 9

6.1 Error Analysis

  • Context sentences reduced errors for some idioms, but not all; models often failed at pragmatic use even when they identified an idiom’s meaning.For two idioms, context reduced error frequency from 9 to 1 and from 9 to 4, while some idioms remained difficult with context.Source: §6.1 · p. 9

7 Conclusion

The paper evaluates LLM understanding and pragmatic use of Egyptian Arabic idioms, multidialectal Arabic proverbs, and English proverbs. It finds consistent performance differences across these forms and releases Kinayat, while identifying broader figurative-language coverage and evaluation settings for future work.

7 Conclusion

  • Models performed best on English proverbs, less well on Arabic proverbs, and worst on Egyptian idioms, with pragmatic use and connotation interpretation posing further challenges.The authors conclude that figurative language can diagnose cultural reasoning, while models may interpret figurative meaning without using it appropriately.Source: §7 · p. 9
  • Kinayat is introduced as the first Egyptian Arabic idiom dataset designed for figurative-understanding and pragmatic-use evaluation.The evaluation covers Egyptian Arabic idioms, multidialectal Arabic proverbs, and English proverbs.Source: §7 · p. 9
  • The authors recommend extending pragmatic evaluation to proverbs and multiple dialects, and assessing pragmatic use in free-form generation.They also identify metaphors, similes, hyperboles, irony, and sarcasm as figurative forms beyond the idioms and proverbs evaluated.Source: §7 · p. 9

Limitations

The study’s scope is limited by its focus on Egyptian idioms and on idioms and proverbs, while connotation labels may be subjective and shaped by cultural perspective.

Limitations

  • The idiom benchmark uses a 1949 source, so some expressions may be outdated and usage may vary across generations and social groups.The pragmatic evaluation uses 150 more familiar idioms from the 325 collected, to reduce the influence of highly unfamiliar expressions.Source: Limitations · p. 10
  • The evaluation covers idioms and proverbs, not other figurative forms such as metaphors, similes, hyperbole, irony, or sarcasm.The paper calls for extending evaluation to these forms for a more comprehensive assessment of Arabic figurative language.Source: Limitations · p. 10
  • Connotation labeling can be subjective, especially when a connotation is weak or context-dependent.Annotation guidelines may not capture multiple cultural perspectives, creating mismatches between annotator intent and dataset construction (Liu et al., 2025).Source: Limitations · p. 10
  • Annotator bias can embed particular worldviews in labeled data, while agreement procedures may miss that language is contextual and values are not universally shared.The cited discussion advocates multilayered or perspectivist models that document diverse meanings and cultural nuance.Source: Limitations · p. 10

Ethical Considerations

The appendix documents dataset ethics, task prompts, and verification of generated incorrect explanations. The prompts cover comprehension, generation, pragmatic use, and connotation tasks.

Ethical Considerations

  • Seven idiom samples considered inappropriate or offensive were removed after manual review.Kinayat is released under CC-BY 4.0, and the dataset includes no annotator personal information; two annotators were coauthors, while a third was paid for the work.Source: Ethical Considerations · p. 10

A Prompt Templates

  • Prompt templates cover MCQ understanding and contextual understanding, incorrect-explanation generation, proverb completion, negation, explanation generation and judging, sentence generation, pragmatic use, and connotation labeling.Templates use “proverb” for Jawaher items and replace it with “idiom” for Kinayat.Source: A Prompt Templates · p. 14

B Verification of Incorrect Explanations

C Data Analysis

The analysis compares sequence lengths and embedding similarities across idioms, proverbs, and task choices. The reported distributions differ by dataset and distractor-generation strategy.

C Data Analysis

  • Arabic proverbs have a higher mean sentence length than Arabic idioms, and English proverbs have a higher mean than Arabic proverbs.The sequence-length distributions cover Kinayat, Jawaher, and MAPS.Source: C Data Analysis · p. 19, Figure 17 · p. 19, Table 5 · p. 19
  • 0.9239 was the mean cosine similarity for SRL-based incorrect explanations, compared with 0.6596 for explanations generated with the general prompt.The general-prompt similarities were roughly normally distributed; the SRL-based similarities were highly right-skewed.Source: C Data Analysis · p. 19
  • 0.743 was the mean cosine similarity between the two idiom options in the pragmatic-use task.Source: C Data Analysis · p. 19

D Additional Results

Additional results show that answer order affects MCQ accuracy, performance varies across datasets and tasks, and closed-source models lead the reported model-group averages. Connotation accuracy is stronger when models use explanations.

D Additional Results

  • 91.95% vs. 79.65% average accuracy was observed when the correct answer was always listed first versus the alternate ordering.The remaining experiments randomized option order to mitigate positional bias.Source: D Additional Results · p. 19, Table 6 · p. 20
  • Mauritania and Sudan were among the lowest-performing dialects under both prompting strategies, while UAE and MSA were among the top-performing groups.The country-level results cover the general and SRL-based prompts and their average.Source: D Additional Results · p. 20, Figure 20 · p. 21, Figure 21 · p. 21
  • 95.66% was the accuracy on English proverbs with context, followed by 90.86% on English proverbs without context, 86.57% on Arabic proverbs, and 76.29% on Egyptian Arabic idioms.For proverb completion, accuracy was 10.64% on Arabic proverbs and 75.43% on English proverbs.Source: D Additional Results · pp. 19–20, Table 8 · p. 21, Table 9 · p. 22
  • On the 150-idiom Kinayat subset, pragmatic-use scores were lowest, MCQ understanding scores next, and contextual MCQ understanding scores highest.Figure 22 visualizes these three tasks.Source: D Additional Results · p. 20, Figure 22 · p. 23
  • 68.17% for proverbs and 72.21% for idioms were the average accuracies when inferring connotations from explanations.On samples with full inter-annotator agreement, Claude Sonnet 4 and Claude 3.5 Sonnet achieved the highest performance; proverb connotations had lower average agreement than idiom connotations, 49.71% vs. 50.10%.Source: D Additional Results · p. 20, Table 12 · p. 24
  • Closed-source models consistently outperformed Arabic and multilingual open-source models across evaluations, although Arabic models led on several specified tasks.Multilingual open-source models slightly outperformed Arabic models on most tasks; the paper notes that closed-source systems are generally assumed to be larger, but their exact sizes are undisclosed.Source: D Additional Results · p. 20, Table 13 · p. 24

E Ablation

Model size correlated significantly with accuracy on some understanding datasets and with pragmatic competence, though the strength of the relationship varied. The section also presents Kinayat idiom examples and summarizes frequent errors across evaluation tasks.

E Ablation

  • Across understanding datasets, size correlations were significant for MAPS + Context (p = 0.0086) and Kinayat (p = 0.0178), but not MAPS (p = 0.1053) or Jawaher (p = 0.0529).The section reports that the MAPS trend was weak and positive, and that Jawaher's trend was borderline.Source: E Ablation · p. 26, Table 17 · p. 26, Figure 23 · p. 26
  • For Kinayat pragmatic use, model size correlated moderately with accuracy (R2 = 0.600, p = 0.0007), versus very weak significant correlations in MCQ and contextual understanding (R2 = 0.265, p = 0.0497 and p = 0.0495).For MCQ Understanding, the reported R2 and p-value are 0.265 and 0.0497; for Contextual Understanding, they are 0.265 and 0.0495.Source: E Ablation · p. 26, Table 18 · p. 26, Figure 24 · p. 26

F Sample Idioms

  • Table 19 provides examples of Egyptian Arabic idioms from Kinayat alongside their explanations.The supplied passage does not describe the specific examples.Source: F Sample Idioms · p. 26, Table 19 · p. 27

G Error Analysis

  • Tables 20–22 summarize frequent errors across 22 models on a 150-idiom Kinayat subset, comparing understanding, contextual understanding, and pragmatic-use tasks.The comparison includes understanding errors before and after contextual sentences are added.Source: G Error Analysis · p. 28, Table 20 · p. 28, Table 21 · p. 28
Loading 2510.23828v2…