PaperlayerView exact revision on arXiv

Source-linked AI summary

MAKIEval: A Multilingual Automatic WiKidata-based Framework for Cultural Awareness Evaluation for LLMs

Raoyuan Zhao, Beiduo Chen, Barbara Plank, Michael A. Hedderich

arXiv:2505.21693v3cs.CL

TL;DR

Cultural awareness is consequential for globally used language models, but evaluating it in open-ended generation across languages remains challenging. MAKIEVAL combines multilingual generation, Wikidata-linked entity analysis, and four complementary metrics; its findings show that cultural awareness varies across models, countries, and languages, with prompting language playing a role.

  • Problem

    Evaluating cultural awareness in natural language generation across languages is challenging, while existing work often relies on English-focused or translation-based setups and fixed cultural entities.

    Source: §1 · pp. 1–2

  • Method

    MAKIEVAL evaluates open-ended multilingual generations by extracting cultural entities, linking them to Wikidata, and measuring granularity, diversity, culture specificity, and cultural consensus.

    Source: §1 · p. 2, §3.2 · p. 3

  • Results

    Cultural awareness varies by model-, country-, and language-specific characteristics, and prompting language plays a role in evoking cultural differences.

    Source: §1 · p. 2, §8 · p. 9

  • Takeaways & Limitations

    The findings support multilingual and multicontextual evaluation of cultural awareness rather than treating it as a simple general bias toward Western culture.

    Source: §8 · p. 9

    MAKIEVAL targets concrete cultural expressions such as food, books, and music, not abstract constructs such as moral reasoning, political ideology, or values.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Large language models (LLMs) are used globally across many languages, but their English-centric pretraining raises concerns about cross-lingual disparities for cultural awareness, often resulting in biased outputs. However, comprehensive multilingual evaluation remains challenging due to limited benchmarks and questionable translation quality. To better assess these disparities, we introduce MAKIEval, an automatic multilingual framework for evaluating cultural awareness in LLMs across languages, regions, and topics. MAKIEval evaluates open-ended text generation, capturing how models express culturally grounded knowledge in natural language. Leveraging Wikidata's multilingual structure as a cross-lingual anchor, it automatically identifies cultural entities in model outputs and links them to structured knowledge, enabling scalable, language-agnostic evaluation without manual annotation or translation. We then introduce four metrics that capture complementary dimensions of cultural awareness: granularity, diversity, cultural specificity, and consensus across languages. We assess 7 LLMs developed from different parts of the world, encompassing both open-source and proprietary systems, across 13 languages, 19 countries and regions, and 6 culturally salient topics (e.g., food, clothing). Notably, we find that models tend to exhibit stronger cultural awareness in English, suggesting that English prompts more effectively activate culturally grounded knowledge.

1 Introduction

The introduction frames cultural awareness as important for avoiding biased or culturally insensitive outputs, yet difficult to evaluate in natural language generation across languages. MAKIEVAL addresses this gap with open-ended multilingual evaluation and four cultural-awareness metrics, then examines how awareness varies by model, country, and language.

1 Introduction

  • Cultural awareness matters because models without it risk biased or culturally insensitive outputs that can perpetuate stereotypes and harm user trust.Source: §1 · p. 1
  • Evaluating cultural differences in open-ended generations is challenging because output specificity, diversity, and cross-lingual consistency can vary across languages.Source: §1 · p. 1
  • Existing evaluation is limited by simplified formats, English-focused or translation-based studies, and fixed entity benchmarks that do not adapt well to emerging cultural expressions.Source: §1 · pp. 1–2
  • MAKIEVAL uses open-ended text evaluation, Wikidata as a language bridge, and structured metrics to support scalable, up-to-date assessment without translation.Source: §1 · p. 2
  • The framework measures granularity, diversity, culture specificity, and cultural consensus; its evaluation spans 13 languages, 19 countries, and 7 models, with awareness patterns linked to model, country, and language characteristics.Source: §1 · p. 2

2 Related Work

Related work spans evaluations of abstract cultural constructs and concrete cultural knowledge, as well as benchmarks with varied formats and coverage. MAKIEVAL differs by using natural open-ended prompts for multilingual scene descriptions rather than constrained entity listing.

2 Related Work

  • Prior work examines abstract cultural dimensions such as values, ideology, gender norms, and stereotypes, as well as concrete cultural knowledge and practices.Source: §2 · p. 2
  • Cultural benchmarks include Japanese- and Italian-focused evaluations, multimodal cultural questions, and Arabic-prompt studies of representation and bias.Source: §2 · pp. 2–3
  • Many benchmarks are single-language, multiple-choice, or region-specific, while multilingual studies have used short-answer or multiple-choice tasks, translation-based alignment, or artifact listing.Source: §2 · p. 3
  • MAKIEVAL instead uses natural open-ended prompts to elicit coherent scene descriptions rather than limiting outputs to entity lists.Source: §2 · p. 3

3 MAKIEVAL

MAKIEVAL generates culturally grounded responses to multilingual prompts, extracts and classifies cultural entities, links them to Wikidata, and compiles matched entities and metadata for analysis.

3 MAKIEVAL

  • The framework has four stages: text generation, cultural entity extraction, Wikidata-based entity matching, and metric-based analysis.Source: §3 · p. 3

3.1 Text Generation

  • Open-ended prompts use country or region templates to support comparison across languages, topics, and places, and each combination receives 500 responses per model.Source: §3.1 · p. 3

3.2 Cultural Entity Extraction

  • GPT-4o-mini extracts topic-relevant entities from generated text and assigns semantic granularity labels, such as specific dishes versus dish types.Source: §3.2 · p. 3

3.3 Wikidata-Based Entity Match

  • Extracted entities are matched to Wikidata labels or aliases in the prompt language, with English fallback, and ambiguous labels are resolved through semantic relations before metadata is compiled for analysis.Unresolved entities are discarded; the metadata includes descriptions and country of origin, and may include an author's or performer's nationality.Source: §3.3 · pp. 3–4

4 Evaluation Metrics

MAKIEval evaluates cultural awareness through four complementary metrics over cultural entities extracted from model responses and linked to Wikidata. The metrics capture detail, breadth, cultural alignment, and agreement across languages.

4 Evaluation Metrics

  • Four metrics evaluate cultural entities extracted from model responses: granularity, diversity, culture specificity, and culture consensus.Entities are linked to Wikidata QIDs, which provide metadata such as country of origin and entity type.Source: §4 · p. 4

4.1 Granularity

  • Granularity scores how finely culturally specific entities are described, with higher values representing finer detail on a predefined cultural scale.Each entity receives a score, and the metric is computed over the entity set.Source: §4.1 · p. 5

4.2 Diversity

  • Diversity counts unique cultural entities generated, measuring the breadth of cultural coverage for a given language and topic.Repeated mentions do not increase the count.Source: §4.2 · p. 5

4.3 Culture Specificity

  • Culture specificity measures the proportion of extracted entities associated with the country or cultural context specified in the prompt.The metric is intended to reflect culturally appropriate, context-aligned content.Source: §4.3 · p. 5

4.4 Culture Consensus

  • Culture consensus measures Jaccard overlap between Wikidata QID sets from responses to the same prompt in two languages.Higher consensus indicates more stable cultural grounding across languages despite surface linguistic variation.Source: §4.4 · p. 5

5 Experimental Setups

The experiments cover seven models, 13 languages, and prompts spanning cultural contexts and everyday topics. The setup also excludes Hausa and Yoruba experiments after open-source models produced poor-quality, often unrelated responses in those languages.

5.1 Models

  • Seven models span open-source and proprietary systems from several regions, with two Llama variants enabling a size comparison and Aya used to assess multilingual generalization.The selection includes Llama-3.1-8B-Instruct, Mistral-7B-Instructv0.1, Qwen2.5-7B-Instruct, ChatGPT-4o-mini, DeepSeek-V3, Llama-3.3-70B-Instruct, and aya-expanse-8b.Source: §5.1 · p. 5

5.2 Languages

  • The study selects 13 languages spanning multiple language families, writing systems, and geographic regions.The languages are Arabic, English, German, Hindi, Italian, Japanese, Korean, Persian, Simplified Chinese, Spanish, Thai, Traditional Chinese, and Turkish.Source: §5.2 · p. 5, Table 2 · p. 6

5.3 Prompts

  • Prompts use explicit cultural context, culturally associated names, or no cultural cue, across food, beverages, clothing, books, music, and transportation.The English food templates illustrate the Neutral, Explicit, and Implicit conditions.Source: §5.3 · p. 6, Table 1 · p. 6
  • Hausa and Yoruba experiments were set aside as future work after open-source models produced very poor, often unrelated text in those languages.The authors initially explored these two low-resource African languages.Source: §5.3 · p. 6
  • Native speakers reviewed prompt meaning and fluency; Mistral and Qwen received added instructions to keep responses in the prompted language.During pilots, Mistral often answered in English and Qwen often answered in Chinese, with Qwen doing so in up to 80% of responses.Source: §5.3 · p. 6

6 Results and Analysis

Across the evaluation, cultural awareness varied with prompt language, model, and cultural context. Granularity, diversity, specificity, and cross-language consensus showed distinct patterns rather than a single uniform effect.

6 Results and Analysis

  • The evaluation used 1,716 prompts across 13 languages, six topics, 19 countries, and contextualized and neutral settings, yielding over 85.8 million model-generated texts.MAKIEval matched 24,042 cultural entities for computing its four metrics.Source: §6 · p. 6

6.1 Granularity

  • Granularity was mostly stable across cultural contexts for a given model and prompt language, while English and Turkish were often lower than Chinese.The comparison covers Chinese, English, and Turkish prompts across models; individual outliers occurred, but no context-specific pattern was visible.Source: §6.1 · p. 7, Figure 3 · p. 7

6.2 Diversity

  • Llama3 had the highest mean diversity at 36.65 and DeepSeek the lowest at 14.62; diversity also averaged 26.518 in a country's native language versus 23.617 in other languages.The native-language advantage was not guaranteed, and diversity also varied by model, prompt language, and country.Source: §6.2 · p. 7, Figure 4 · p. 7

6.3 Cultural Specificity

  • For Llama3, neutral prompts showed high specificity for matching country–native-language pairs, and explicit cultural context increased specificity in most cases.English prompts consistently showed the highest specificity gains from added context across nearly all countries and models.Source: §6.3 · pp. 7–8, Figure 5 · p. 8

6.4 Cultural Consensus

  • For ChatGPT on food, consensus was high between some European language pairs and between Japanese and Korean; larger models had similar consensus values around 0.2.Consensus also differed substantially among smaller models of similar size, so model size was not the only factor.Source: §6.4 · p. 8, Table 3 · p. 8

7 Discussion

MAKIEval's complementary measures capture systematic differences across languages and models, but their interpretation depends on the task. The paper does not treat higher metric values as universally preferable.

7 Discussion

  • Figure 6 presents ChatGPT's cultural consensus between language pairs for the food topic.The accompanying discussion highlights high consensus for some European language pairs and for Japanese and Korean.Source: Figure 6 · p. 9, §7 · p. 9
  • The metrics are complementary measures of cultural knowledge expression, not a prescriptive benchmark where higher values are inherently better.Their relevance varies with downstream-task characteristics, so interpretation requires care.Source: §7 · pp. 8–9
  • No single direction of change is universally optimal: high specificity or granularity may suit shared-cultural settings, while lower values may suit intercultural communication.Cross-lingual consensus may suit knowledge-intensive applications, whereas divergence may benefit creative or locally adapted tasks.Source: §7 · p. 9

8 Conclusion

MAKIEval combines prompt-controlled generation, entity extraction, Wikidata grounding, and four metrics to evaluate cultural awareness across languages, regions, and topics. The results show that cultural awareness takes multiple forms and that multilingual, multicontextual evaluation is needed.

8 Conclusion

  • MAKIEval combines prompt-controlled generation, structured entity extraction, Wikidata grounding, and four cultural metrics to analyze knowledge across languages, countries or regions, and topics.The metrics are granularity, diversity, cultural specificity, and cultural consensus.Source: §8 · p. 9
  • Cultural awareness is more complex than a general Western bias, appearing in entity granularity, specificity, and diversity, while prompting language also shapes which differences emerge.The conclusion additionally highlights English sensitivity to cultural signals, links between countries and local languages, and consensus across regional areas.Source: §8 · p. 9
  • The authors plan to release generated outputs and cultural entities for all model, language, country, and topic combinations to support further analysis.They identify training, alignment, and internal cultural representations as areas for future research.Source: §8 · p. 9
  • The framework and findings emphasize multilingual and multicontextual evaluation for more equitable worldwide evaluation and deployment of language technologies.The authors describe their metrics as actionable signals for improving fairness and inclusivity in global LLM applications.Source: §8 · p. 9

Limitations

The paper reports scope limits, implementation choices, prompt and extraction procedures, and checks of generation and entity-linking quality. Its experiments omit low-resource languages, and its framework targets concrete rather than abstract cultural expression.

Limitations

  • Experiments omit low-resource languages because current models produced low-quality outputs; Wikidata metadata may be incomplete or inaccurate, and abstract constructs such as values remain outside the framework’s scope.The authors say metadata issues are rare and have limited overall impact, though they may add noise to entity-level specificity or consensus metrics.Source: Limitations · pp. 9–10

A Computational Resources and AI Assistance

  • Analyses used publicly available models; locally run LLMs used temperature 0.7, top-p 0.9, and top-k 10 on NVIDIA A100 GPUs, except LLaMA3-70B on an H200 GPU.DeepSeek and ChatGPT were accessed via public APIs. ChatGPT assisted with manuscript editing and coding; the authors said they would release their code, produced data, and documentation.Source: Limitations · p. 10, A Computational Resources and AI… · p. 13

B.1 Prompt Template for Text Generation

  • Templates covered six topics and 13 languages, received review from 15 native speakers, and added language-specific instructions for Mistral and Qwen to resolve response-language mismatch.Food prompts compare neutral, explicit, and implicit framing across 13 languages.Source: §B.1 · p. 13, Figure 7 · p. 14

B.2 Prompt for Entity Extraction

  • Topic-specific prompts extract entities into a dictionary, with branches by topic and additional person and location names; ChatGPT-4o-mini assigns granularity labels whose binary scores are averaged.Specific references score 1 and general categories score 0.Source: §B.2 · p. 15

B.3 Generation Quality in Low-Resource African Languages

  • Hausa and Yoruba were excluded after open-source models produced semantically irrelevant, grammatically incoherent, or prompt-disconnected outputs.Source: §B.3 · p. 15

C.1 Performance of Entity Extraction

  • Extraction was manually checked on 50 sentences per topic in four languages; F1 scores were consistently high, missed entities were rare, and Wikidata disambiguation filters false positives.One native speaker annotated each language; performance varied by English domain, while overall results remained strong.Source: §C.1 · p. 15

C.2 Performance of Wikidata Search

  • Approximately 26%-35% of extracted entities lacked valid QIDs across topics, often because outputs included hallucinated books, songs, or dishes that cannot be trivially verified.Source: §C.2 · p. 15

C.3 Entity Statistics

  • Table 11 reports unique QIDs and labels by topic; a lower resolve rate means more multilingual surface forms are linked to shared entities.Source: §C.3 · p. 16

D Results of Granularity

The reported granularity measure is relatively stable across prompted countries or regions within a model-language pair.

D Results of Granularity

  • Granularity remains relatively stable across prompted countries or regions for a given model-language pair, which may suggest that it is invariant to the referenced country or region.Figure 9 presents radar charts of average granularity across topics by language, with colored lines for countries or regions.Source: D Results of Granularity · pp. 16–17

E Results of Diversity of Native and Non-native Language

DeepSeek has the lowest reported overall diversity, while Llama3-8B and Mistral have higher diversity averages across language, topic, and country combinations.

E Results of Diversity of Native and Non-native Language

  • Figure 8 counts the unique cultural entities, measured as Wikidata QIDs, extracted from model outputs in each language.Source: Figure 8 · p. 17
  • 16.53: DeepSeek has the lowest overall diversity; Llama3-8B and Mistral have higher diversity across language, topic, and country combinations.Table 12 reports the models’ average diversity with variance.Source: Table 12 · p. 17, E Results of Diversity of Native and… · p. 17

F Results of Culture Specificity

Culture specificity measures how outputs change when prompts include cultural cues, while culture consensus measures shared entity interpretations across languages. The reported results show the largest specificity gains in English and higher consensus among European languages.

F Results of Culture Specificity

  • Culture specificity is the difference in entity representations between country- or region-contextualized prompts and neutral prompts; higher values indicate stronger adaptation to cultural cues.Source: F Results of Culture Specificity · p. 17
  • Native language–country pairs often have relatively high specificity even with neutral prompts, suggesting language can act as an implicit cultural signal.Source: F Results of Culture Specificity · p. 17

G Results of Culture Consensus Results

  • Culture consensus is measured by pairwise overlap in entity QIDs across languages responding to semantically equivalent prompts.The overlap reflects the extent to which languages share cultural interpretations of the same prompt.Source: G Results of Culture Consensus Results · p. 17

F Results of Culture Specificity

G Results of Culture Consensus Results

Loading 2505.21693v3…