Are Multilingual LLMs Culturally-Diverse Reasoners? An Investigation into Multicultural Proverbs and Sayings

1 Introduction

• Proverbs are fixed expressions whose literal forms and interpretations vary across cultures, while their underlying meanings often express universal human experiences.This variation makes them a proxy for cultural common ground; their interpretation also depends on conversational context and can be figurative.Source: §1 · p. 2
• MAPS supports experiments on how multilingual LLMs recognize proverbs, interpret them in context, and reason across cultures.The paper frames these as questions about cultural common ground and cross-lingual reasoning.Source: §1 · p. 2, MAPS · p. 2

MAPS

• MAPS contains proverbs and sayings from six languages, conversational contexts, interpretation tasks, and figurative-use labels.The languages are English, German, Russian, Bengali, Mandarin Chinese, and Indonesian.Source: MAPS · p. 2
• Across the evaluated models, proverb knowledge varies and is biased toward English and Chinese; memorization does not guarantee contextual reasoning, and translated proverbs reveal culture gaps.The paper also reports that figurative proverbs are harder to reason about in many languages.Source: MAPS · p. 2

2 Related Work

• Earlier proverb work recognized proverbs from short stories, while BIG-bench evaluated proverb matching across three languages with a small set.Other work assessed cultural norms in English.Source: §2 · pp. 2–3
• MAPS is presented as the largest multilingual dataset, to the authors’ knowledge, focused on proverbs and sayings with conversational contexts and an inference task.The paper contrasts this scope with prior English-focused conversational or cultural-norm tasks.Source: §2 · p. 3
• MABL (Kabra et al., 2023) studies novel metaphors and cross-lingual transfer, rather than memorization versus contextual proverb reasoning.The paper distinguishes its focus on cultural common ground and multilingual conversational contexts from MABL.Source: §2 · p. 3

3 MAPS — MulticulturAl Proverbs and Sayings

• MAPS combines proverbs, conversational contexts, one correct and one wrong interpretation, and labels indicating figurative use.The dataset covers six languages selected for geographic and typological diversity, while accounting for resource availability.Source: §3 · p. 3, §3.1 · p. 3

3.1 Dataset Creation

• Native-speaker review assessed contextual usage and interpretation answers, accepting answers aligned with one possible interpretation and revising options when needed.Contexts were seeded with model-generated conversations and reviewed by native speakers; contexts for English, Chinese, Russian, and Bengali were completely rewritten.Source: §3.1 · pp. 3–4
• The final dataset contains 2,313 proverbs with conversational context, with a test set and a few-shot train-dev set of 30 examples per language.Proverbs were collected from Wikiquote and Wiktionary; Bengali items were subsampled to keep the dataset roughly balanced across languages.Source: §3.1 · pp. 3–4

3.2 Analysis of MAPS

• Indonesian figurative proverbs more often use animals and concern nature than English figurative proverbs in the sampled analysis.The authors also note culture-specific concepts in the dataset, including dragons and phoenixes in Chinese proverbs.Source: §3.2 · p. 4
• Figure 2’s KDE visualization shows language-specific proverb embedding distributions, with Indonesian partially overlapping English and Chinese and Bengali relatively distinct from Western languages.The authors interpret the distributions as reflecting cultural differences, and report additional analysis to isolate language effects.Source: §3.2 · p. 4, Figure 2 · p. 4

4 Experimental Setup

• The model suite includes masked, encoder–decoder, and causal language models, with most covering all six MAPS languages.BLOOMZ lacks Russian and German coverage; the study also includes three LLaMA-2 sizes despite its primarily English focus.Source: §4 · p. 5
• For memorization, models complete a proverb with its final word removed, and the authors count a match under model-specific completion criteria.For non-MLMs, generated text must start with the missing token or contain the whole proverb; MLMs predict the masked final word.Source: §4 · p. 5
• For contextual inference, the model selects between two answer candidates by comparing their logits under a prompt template.For MLMs, the answer-candidate prediction logits are compared.Source: §4 · p. 5
• The English–Chinese culture-gap case study compares machine translation with human-adapted translations that correct literal and grammatical errors and adjust names and locations.Human adaptation is described as a best-effort attempt to reduce the language gap.Source: §4 · pp. 5–6

5.1 Knowledge of Proverbs

• Proverb memorization improved with model size, but memorization disparities were especially pronounced for lower-resource Indonesian, Bengali, and Russian, potentially due to data exposure.The analysis found no significant attribution to proverb familiarity, length, or figurative status.Source: §5.1 · p. 6

5.2 Reasoning of Proverbs with Conversational Context

• As model size increased, inference accuracy generally improved, but memorization did not indicate reasoning ability, and its benefit was inconsistent between English and Chinese.The reported memorization benefit was evident in English but inconsistent in Chinese.Source: §5.2 · p. 6
• Across English, German, and Russian, models performed worse on figurative-proverb inference; Chinese showed the opposite pattern, while larger models appeared better on Indonesian and Bengali figurative proverbs.The authors describe the latter pattern as an appearance, not a general finding across all languages.Source: §5.2 · p. 6
• Asking models to choose the wrong answer made previously strong models perform badly, except mT0, and enlarged performance gaps across languages as model size increased.The authors also report consistent failures and inverse scaling in many cases when asking for the wrong answer without using “not.”Source: §5.2 · p. 7

5.3 Culture Gaps in mLLMs - A Case Study

• The culture gap for Zh–En was 5.73 for mT0 and 19.40 for LLaMA-2, measured against the stronger of source- and target-language accuracy.Human translation improved over machine translation, which the authors treat as the language gap; the culture gap is the difference between human-translated accuracy and the maximum source- or target-language accuracy.Source: §5.3 · p. 8, Figure 5 · p. 8
• Machine translation produced incomplete or incorrect proverb translations, and human-translated data outperformed machine-translated data in the Zh–En comparison.The paper illustrates errors in translating context-dependent Chinese phrases and calls for further research on cultural awareness and cultural priors in translation models and mLLMs.Source: §5.3 · p. 8

• The study evaluates multilingual LLMs’ proverb memorization, contextual reasoning, and cross-cultural communication.It uses proverbs and sayings from different languages as an investigative tool for cultural common ground.Source: §6 · p. 8
• Knowing proverbs does not guarantee that models can reason with them in context, and mT0’s culturally diverse reasoning is limited.Source: §6 · p. 9
• Zero-shot reasoning ability emerges with model scale, while the ability to answer negative questions inversely correlates with scale.Source: §6 · p. 9
• Language disparities in culturally diverse reasoning grow with model size, and the study identifies culture gaps in cross-lingual communication.Source: §6 · p. 9

7 Limitations

• The dataset has one conversation and one correct–wrong interpretation pair per proverb, and is relatively small compared with many automatically generated benchmarks.The authors describe lexical bias as an intended feature of focusing on cultural common ground.Source: §7 · p. 9
• The evaluation covers general-purpose open-source models; larger or task-specific models may be needed to assess negative questions and culture gaps fully.Closed-source evaluation is outside the paper’s scope, although the public dataset can be used to evaluate such models.Source: §7 · p. 9

A.1 Annotations

• Annotators were recruited through Prolific and needed first-language fluency in the corresponding language and fluency in English; experts included researchers and authors.Source: §A.1 · p. 14
• Conversations were limited to one turn, with a maximum of two, and annotation recorded the contextual meaning and a negative alternative.Annotators checked whether the proverb was used correctly and rewrote incorrect conversations; figurative uses were to match the figurative meaning.Source: §A.1 · p. 14

• Proverbs contain culturally important concepts, with tiger relatively important in Eastern cultures and lion more important in Western cultures.Source: §A.2 · p. 14
• Rice is described as more culturally important in the East, while bread is enjoyed by many people around the world.Source: §A.2 · p. 14
• The table presents selected food and animal concepts drawn from proverbs across languages.Source: §A.2 · p. 14

A.3 Additional Qualitative Analysis of Proverbs

• Similar proverbs across cultures may use locally familiar concepts or, rarely, local place or person names.For figurative proverbs, concept-category distributions differ across English, Indonesian, and Chinese; Indonesian examples more often involve animals and nature.Source: §A.3 · p. 15

A.4 Additional Data Statistics

• Additional dataset statistics report average context tokens and turns, counting each Chinese character as a word for the token average.Source: Table 6 · p. 15, §A.4 · p. 15

A.5 Interpreting the KDE Plot

• Topic-coherent parallel-dataset sentences have overlapping, inseparable embeddings, unlike the cross-language differences visible in proverb embeddings.The KDE comparison samples 400 sentences per language from a parallel multilingual dataset.Source: §A.5 · pp. 15–16, Figure 8 · p. 16

A.6 Data Examples

• MAPS balances its labels and provides example data for every language.Source: §A.6 · p. 16

B Templates

• GPT-3.5 generated seed conversations from a prompt, and a translated template did not improve their quality.Memorization accuracy is the union of examples retrieved by five prompt patterns; the paper also supplies its main inference template and a non-MLM workflow illustration.Source: B Templates · p. 16, Figure 9 · p. 16

D.1.1 Discussions

• The authors use a single correct missing-word prediction as evidence of proverb memorization, while noting masked-token prediction better fits mT0 and XLM-R pretraining.Context can make several plausible completions likely even without knowledge of the proverb.Source: §D.1.1 · p. 18

D.1.2 Additional Results

• With one randomly masked token per proverb, language disparities, low mT0 memorization, and a positive model-size correlation persist.The experiment uses adjusted prompts with <extra_id_0> for mT0 and <mask> for XLM-R.Source: Figure 10 · p. 18, §D.1.2 · p. 18

D.2 Memorized versus Not Memorized

• Memorization benefits inference for English, but not Chinese, among the three best-performing models analyzed.Results are limited to English and Chinese because other groups had fewer than 50 proverbs in the not-memorized group.Source: §D.2 · p. 18, Table 12 · p. 19

D.3.1 Additional Prompt Templates

• Three additional prompt templates preserve the trends: mT0 performs best, larger models improve, and memorization does not indicate inference performance.These experiments do not assess formal reasoning abilities such as mathematical reasoning.Source: §D.3.1 · pp. 18–19

C Cross-lingual Transfer Baselines

• Cross-lingual transfer remains weak for models under a billion parameters, and the performance gap between English and other languages remains significant.The baselines train on English and evaluate on other languages; translate-test baselines are also included.Source: C Cross-lingual Transfer Baselines · pp. 16, 18, Table 10 · p. 19

D.3.2 Additional Models

• Performance on MAPS improves with model size; Aya-101 is generally on par with mT0 13B and performs noticeably better in Bengali.The additional-model results cover Vicuna-V1.5 and Aya-101.Source: §D.3.2 · p. 19, Figure 12 · p. 21

D.4 ‘Negative’ Questions

• Negative-question tests motivate improving such answers; without a question, all tested LLMs scored above random on multiple languages, which the authors interpret as failure.The authors speculate that positive biases in training data may contribute to this behavior.Source: §D.4 · p. 21

D.5 Culture Gaps

• For mT0 on English–Chinese translated data, the reported culture gap is 5.33.The authors report similar results to their earlier English–Chinese experiment.Source: §D.5 · p. 21

D.6 Additional Results on LLaMA-2 with Translations

• Translated-data performance shows significant gaps from English that increase with the language’s geographic distance; the reported gap combines language and culture gaps.The experiment translates all non-English data into English using Google Translate.Source: §D.6 · p. 21, Table 14 · p. 23

D.7 Few-shot (In-context) Evaluation

• The evaluation tests BLOOMZ 7.1B, mT0-XXL 13B, and LLaMA-2 13B with 2-shot and 5-shot demonstrations.Five sets of demonstrations were randomly sampled for each setting.Source: §D.7 · p. 23
• The authors report that degradation with few-shot demonstrations has also been observed in Li et al. (2023b) and Koto et al. (2023).Source: §D.7 · p. 23
• Few-shot demonstrations do not improve performance over zero-shot; model performance consistently degrades with more demonstrations.The authors note that the task’s long context may affect n-shot performance.Source: §D.7 · p. 23, Table 15 · p. 24