Source-linked AI summary
BLEnD: A Benchmark for LLMs on Everyday Knowledge in Diverse Cultures and Languages
Junho Myung, Nayeon Lee, Yi Zhou, Jiho Jin, Rifki Afina Putri, Dimosthenis Antypas, Hsuvas Borkakoty, Eunsu Kim, Carla Perez-Almendros, Abinew Ali Ayele, Víctor Gutiérrez-Basulto, Yazmín Ibáñez-García, Hwaran Lee, Shamsuddeen Hassan Muhammad, Kiwoong Park, Anar Sabuhi Rzayev, Nina White, Seid Muhie Yimam, Mohammad Taher Pilehvar, Nedjma Ousidhoum, Jose Camacho-Collados, Alice Oh
TL;DR
LLMs struggle with everyday cultural knowledge that is poorly represented in online and English-centric training data. BLEnD addresses this gap with a multilingual, multicultural benchmark, finding substantially uneven performance across cultures and languages.
Problem
Everyday cultural knowledge is poorly documented in common online training sources, and existing multilingual multicultural benchmarks do not adequately capture it.
Method
BLEnD is a hand-crafted benchmark covering 16 countries or regions and 13 languages, built from native-speaker annotations and localized question templates.
Results
LLMs perform better on highly represented cultures, while performance is lower for less-represented cultures and languages, especially with local-language prompts.
Takeaways & Limitations
BLEnD provides an evaluation of everyday cultural knowledge across diverse regions, cultures, and languages.
Takeaways & Limitations
The benchmark may not fully represent included regions because it typically uses only five annotators per question, sometimes from the same locality.
Abstract
from arXiv · showhide
Large language models (LLMs) often lack culture-specific knowledge of daily life, especially across diverse regions and non-English languages. Existing benchmarks for evaluating LLMs' cultural sensitivities are limited to a single language or collected from online sources such as Wikipedia, which do not reflect the mundane everyday lifestyles of diverse regions. That is, information about the food people eat for their birthday celebrations, spices they typically use, musical instruments youngsters play, or the sports they practice in school is common cultural knowledge but uncommon in easily collected online sources, especially for underrepresented cultures. To address this issue, we introduce BLEnD, a hand-crafted benchmark designed to evaluate LLMs' everyday knowledge across diverse cultures and languages. BLEnD comprises 52.6k question-answer pairs from 16 countries/regions, in 13 different languages, including low-resource ones such as Amharic, Assamese, Azerbaijani, Hausa, and Sundanese. We construct the benchmark to include two formats of questions: short-answer and multiple-choice. We show that LLMs perform better for cultures that are highly represented online, with a maximum 57.34% difference in GPT-4, the best-performing model, in the short-answer format. For cultures represented by mid-to-high-resource languages, LLMs perform better in their local languages, but for cultures represented by low-resource languages, LLMs perform better in English than the local languages. We make our dataset publicly available at: https://github.com/nlee0212/BLEnD.
1 Introduction
BLEND addresses the lack of multilingual, multicultural benchmarks for everyday cultural knowledge by combining human-crafted questions across diverse regions with short-answer and multiple-choice evaluation. Experiments show substantial performance disparities between highly represented and underrepresented cultures.
- LLMs may produce incorrect, incomplete, nonsensical, stereotypical, or culturally inappropriate responses because everyday knowledge is poorly documented in common online training sources.
- BLEND is constructed through question collection, filtering and translation, answer annotation, and answer aggregation, using native-speaker annotators across countries and regions.
- 79.22% versus 12.18%: average short-answer performance is much higher for US culture in English than Ethiopian culture in Amharic.A similar performance gap appears in the multiple-choice format.
- The benchmark uses two formats—short-answer and multiple-choice—to automatically measure LLMs’ cultural knowledge.
- Experiments across 16 LLMs show a significant performance gap between highly represented and underrepresented cultures.
2 Related Work
Prior cultural-knowledge benchmarks often focus on English, formal online sources, single languages, or social-media data. BLEND instead uses human-crafted questions to compare everyday cultural knowledge across 13 languages and 16 regions.
- Earlier studies commonly focus on national culture, English-language data, and objective cultural aspects documented in formal sources.
- Recent benchmarks cover Korean or Indonesian cultural knowledge, but do not compare diverse languages and cultures using the same question set.
- Social-media-based resources capture cultural nuances but may contain noisy assertions and remain limited in language or coverage of everyday behavior.
- BLEND is human-crafted and covers everyday cultural knowledge across 13 languages and 16 countries/regions, including West Java and North Korea.
3 Construction of BLEND
BLEND is constructed from culturally grounded questions and native-speaker annotations spanning diverse countries, regions, and languages. The dataset also measures answer diversity and cross-region similarity through aggregation and lemma overlap.
- Dataset scope: BLEND selects languages with varying resource availability and includes native-speaker inspection for represented countries and regions.North Korea is the stated exception to the native-speaker-origin criterion.
- Question collection: The benchmark uses 500 question templates covering food, sports, family, education, holidays/celebrations/leisure, and work-life.Templates are generated by at least two native annotators per country or region and filtered for duplicates and overly country-specific items.
- Answer annotation: Five native annotators per question provide short answers, with replacement collection when an annotator selects “I don’t know the answer.”Annotators may provide up to three responses when one answer is insufficient.
- Answer aggregation: Reviewers remove invalid answers and group linguistic variants into shared answer terms before producing local-language and English variants with vote counts.The aggregation process addresses both misunderstood questions and natural-language variation.
- Quality and diversity: Annotator agreement averages 3.16 out of 5, while answer uncertainty averages 1.01 “I don’t know” responses per question.Agreement reflects both consensus and substantial variety in answers; uncertainty is highest in Northern Nigeria and lowest in South Korea.
- Cross-region similarity: Indonesia–West Java, United States–United Kingdom, and Spain–Mexico have the most similar responses, consistent with closely aligned cultural backgrounds.Similarity is estimated from shared lemmas in English versions of annotations.
4 LLMs Cultural Knowledge Evaluation
The evaluation measures LLM performance on short-answer and multiple-choice versions of BLEND across local-language and English prompts. Performance is consistently higher for highly represented cultures and languages, while language choice and question type affect results.
- Evaluation setup: BLEND evaluates LLMs using short-answer and multiple-choice tasks, with experiments covering 16 evaluated models.Short-answer scores average direct-answer and target-country persona prompts, and correctness is matched against human annotations.
- Short-answer results: 79.22% for US culture in English versus 12.18% for Ethiopian culture in Amharic illustrates the performance gap across represented cultures and languages.The local-language short-answer evaluation shows lower performance for lower-resource languages.
- Cultural contrasts: GPT-4 shows a 31.63% performance disparity between South Korea and North Korea despite their shared Korean language.Spain and Mexico also show a GPT-4 gap of 4.35%.
- Region-centric models: Region-centric models tend to perform better on their associated country or region, with Qwen1.5-72B strongest on Chinese and HyperCLOVA-X exceeding GPT-4 on North Korean questions.The passage attributes these results to customized datasets containing richer local cultural content and nuances.
- Language effects: Local-language prompting performs better for high-resource cultures such as Spanish and Chinese, whereas English performs better for low-resource cultures such as Azerbaijani, Sundanese, and Amharic.For North Korean questions, both languages perform poorly, though Korean is slightly better.
- Question categories: Models perform worse on food and holidays/celebrations/leisure than on work-life or education, with the category difference significant at p < 0.05.The analysis uses one-way ANOVA and characterizes the lower-performing topics as more subjective.
- Evaluation limitation: The MCQ version is currently available only in English because its cross-cultural distractors require additional translation work.The authors plan to release a multilingual MCQ version.
- Multiple-choice evaluation: MCQ performance is generally higher than SAQ performance, while its country-level pattern correlates strongly with SAQ results at Pearson correlation 0.93.The paper attributes the improvement to multiple-choice questions having well-defined answers.
5 Human Evaluation
Human evaluation categorizes GPT-4’s wrong short-answer responses to identify stereotypical, incomplete, refusal, nonsensical, linguistic, and cross-country errors. Errors are concentrated in underrepresented cultures and low-resource languages.
- Evaluation design: The human evaluation analyzes 120 GPT-4 wrong answers using six categories: stereotypical, partially correct, refusal, nonsensical, unnatural language, and different country’s view.GPT-4 is selected because it is the best-performing model in the short-answer evaluation.
- Stereotypical responses: 48.33% of Ethiopia responses were stereotypical, with food and festival questions producing many attempts at traditional information without sufficient contextual understanding.West Java and Assam also generated many stereotypical responses.
- Incomplete and refused answers: Azerbaijan, North Korea, Northern Nigeria, and Ethiopia had high proportions of partially correct answers or refusals.These countries are all described as underrepresented cultures.
- Language-related errors: Low-resource-language questions elicited long answer lists, refusals, nonsensical outputs, and hallucinations such as the nonexistent Azerbaijani basketball player “Ruslan Cfrov.”The analysis links these patterns to limited question comprehension in low-resource languages.
- Cross-country perspectives: GPT-4 often answered Azerbaijan questions from other Caucasus countries and North Korea questions from South Korea’s perspective.Azerbaijan and North Korea also had the highest ratios of unnatural language.
6 Conclusion
BLEND evaluates everyday cultural knowledge across 16 countries or regions and 13 languages. Results show strong performance for highly represented cultures but substantially lower performance for underrepresented cultures, with language-dependent effects across task formats.
- Benchmark scope: BLEND evaluates cultural knowledge about everyday life in 16 countries or regions and 13 distinct languages.The benchmark is applied to current LLMs across these cultural and linguistic settings.
- Main findings: LLMs perform strongly for highly represented cultures but significantly worse for less-represented cultures and languages, especially when prompted in local languages.This pattern occurs in both short-answer and multiple-choice questions.
- Cultural and linguistic effects: Performance differs between culturally distinct countries sharing a language, and prompting language favors local languages for mid-to-highly represented cultures but English for underrepresented cultures.The conclusion identifies these patterns as evidence of cultural bias and language-dependent performance.
7 Limitations and Future Work
The benchmark’s annotations may not fully represent each country or region because annotator coverage is limited and sometimes geographically concentrated. Short-answer and long-form evaluations also remain difficult, motivating broader annotation and improved evaluation methods.
- Representativeness: Typically five annotators per question, sometimes from one locality, may not fully represent each included country or region.The authors propose increasing annotator numbers and recruiting across diverse regional bases.
- Representativeness: English-proficient academic language experts may bias benchmark construction because they may not represent each population fully.The paper explicitly avoids claiming complete representation of all speakers of any language or region.
- Evaluation: Short-answer evaluation can miss correct responses because word variations are difficult to cover despite human effort and lemmatization or stemming.The dataset also evaluates long-form responses using annotations based on short answers, creating an additional evaluation challenge.
- Future work: Future work should improve evaluation of cultural adaptiveness in long-form natural contexts beyond prompt-based evaluations.The authors identify this as a future direction because current evaluation remains constrained by short-answer annotations.
Checklist
The checklist records that the paper addresses claims, limitations, ethics, reproducibility, and asset documentation through the main text, appendices, figures, and repository links.
- Claims and ethics: The authors state that the paper’s claims and limitations are discussed in the abstract, introduction, and Section 7.The checklist also records discussion of potential negative societal impacts in Section 7.
- Reproducibility: The authors state that code, data, reproducibility instructions, training details, compute information, and error bars are provided or documented.The checklist references the abstract, Appendix C.1, and Figure 3b.
- Assets and compensation: The checklist records that new assets, participant compensation, and asset-related documentation are addressed through the abstract and Appendix B.2.It also records that existing-asset citation and licensing items were not applicable.
- Claims and ethics: The paper reports consent procedures, participant-risk information, and participant instructions in Appendix B.The checklist points to Appendix B.2, B.4, B.5, and B.6 for these materials.
A Dataset Details
The appendix documents BLEND’s public resources, data organization, annotation format, prompt and evaluation scripts, language coverage, ethics procedures, and question-construction guidelines.
- Accessibility and usage: BLEND’s data samples, annotations, prompts, and code are publicly available through its GitHub repository and HuggingFace.The repository includes topic, source-language, English-question, local-language, annotation, and vote-count information.
- Language and annotation coverage: Annotations store local-language and English questions and answers together with vote counts and categories for no-answer or other responses.The South Korea example records “kimchi” with four votes and rice-related answers with three votes.
- Accessibility and usage: The benchmark provides scripts for model inference and short-answer scoring, with configurable models, countries, and languages.Inference results are saved in a default results directory, and evaluation can be run with evaluation/evaluate.sh.
- Language and annotation coverage: BLEND covers 13 languages across countries and regions, with language-resource levels documented in Table 4.The dataset includes local-language and English versions of cultural questions and answers across food, sport, family, education, holidays, and work-life.
- Ethics and participants: The project received KAIST IRB approval, obtained informed consent, and compensated annotators above Prolific’s ethical standards.Participants could withdraw, and the same compensation standards applied to directly recruited low-resource-language annotators.
- Question construction: Question guidelines require culture-specific, short, concrete, distinct questions while discouraging binary and highly stereotypical topics.The guidelines encourage common-sense cultural questions rather than factual questions whenever possible.
B.7 Annotation Analysis
The annotation analysis reports agreement, answer-count, and uncertainty statistics across question categories and countries, alongside the guidelines shown to annotators.
- Annotation statistics: Table 7 measures annotator agreement using the average maximum vote count for answers across categories and countries.A value of 3.00 means that three annotators provided the same answer on average for each question.
- Annotation statistics: Table 8 reports the average number of answers provided per question across categories and countries.A value of 3.00 means that three answers were provided on average for each question.
- Annotation statistics: Table 9 reports the average number of “I don’t know” responses per question across categories and countries.These statistics provide a separate measure of annotator uncertainty.
- Annotation procedure: The annotation guidelines are presented to annotators as part of the benchmark’s data-collection procedure.The supplied figure is identified as the answer annotation guidelines shown to annotators.
C Experimental Settings for LLM Evaluation
The evaluation measures LLM cultural knowledge with short-answer and multiple-choice tasks across countries and languages, using human annotations to determine correctness. It also examines performance by country, language, and question category.
- Evaluation formats: The study evaluates LLMs with short-answer and multiple-choice questions across countries and regions.Short-answer evaluation uses direct and target-country persona prompts, while multiple-choice questions present one correct answer and three wrong options.
- Short-answer evaluation: Short-answer scores average correctness across direct and target-country persona prompts.A response is correct when it is included in the human annotators’ answers for the same question.
- Response matching: Human annotations are normalized for language variation by lemmatizing, stemming, or tokenizing answers and model responses.The pipeline uses language-specific tools, including SpaCy, SparkNLP, jieba, Okt, Qalsadi, CLTK, and Hazm.
- Multiple-choice construction: Multiple-choice distractors are drawn from answers in other countries, filtered for similarity, and completed with GPT-4-generated dummy options when necessary.Questions generally contain one correct answer and three wrong answer options; multiple correct answers produce separate question versions.
- Reported analyses: The experiments report short-answer results in local languages and English, multiple-choice results in English, and category-level averages with Tukey-HSD confidence intervals.The categories include work-life, education, food, and holidays, celebrations, or leisure; annotators also assess answer applicability to the general population.
D.3.2 Human Evaluation Result
Human evaluation reveals culturally inaccurate, stereotypical, hallucinated, and sometimes offensive responses across several regions. Errors often substitute highly associated cultural concepts, perspectives from other countries, or nationally broad stereotypes for context-specific answers.
- Stereotypical food responses: Food-related questions elicited stereotypical substitutions in Algeria, Greece, Indonesia, Assam, and Azerbaijan.Examples include couscous, feta cheese, nasi goreng, Bihu-related foods, and Qutab or Kebab when those answers did not fit the question context.
- Population coverage: Some responses inadequately represented minority or regional populations, including Indonesia’s Christian population and China’s Muslim population.The China example correctly distinguishes Muslim observance of Ramadan from non-Muslim Chinese practices, while the Indonesia example misidentifies Ketupat as Easter food.
- Ethiopia (Amharic): In Ethiopia, responses frequently repeated questions, contained typographic errors, or produced long repeated texts, indicating limited Amharic use.The model also often supplied well-known Ethiopian associations such as Injera for unrelated food questions.
- Cross-country misalignment: Models sometimes answered from another country’s perspective, such as giving a half-mast flag or Independence Day for Indonesian practices.In Indonesia, a yellow flag indicates a death in the area, and Independence Day is not celebrated with fireworks in the described context.
- Iran (Persian): Iranian responses frequently hallucinated names, refused questions about alcohol, and defaulted to the Gregorian calendar instead of the Solar Hijri calendar.The calendar behavior was mostly correct after conversion but was judged culturally insensitive when Persian questions received Gregorian framing.
- North Korea (Korean): North Korean responses were often offensive or biased toward Pyongyang, including repeated use of Kim Jong Un for unrelated questions.The passage attributes the regional bias as potentially stemming from insufficient information about areas outside Pyongyang.