Survey of Cultural Awareness in Language Models: Text and Beyond

1. Introduction

• Western-centric training can contribute to stereotyping, alienation, and culturally insensitive responses, motivating cultural awareness in NLP models.The cited passage also identifies flattening cultural identities as a risk.Source: §1 · p. 2
• The survey reviews cultural inclusion in text and multimodal language systems, drawing on psychology and anthropology and covering data, alignment, evaluation, HCI, ethics, and social research.It positions its coverage across NLP, multimodality, and social science, including images, videos, and audio.Source: §1 · p. 2
• The authors identify four goals: review 300+ papers, summarize data sources and dataset coverage, discuss equity measures, and examine societal impacts and HCI.The review covers benchmarks and methods for cultural inclusion in multimodal language models.Source: §1 · p. 2
• The literature search includes NLP and computer-vision venues, ACL Anthology papers, and recent arXiv submissions; alignment methods are limited to work after 2022.Cultural-inclusion benchmarks are considered from after 2016, while recent social-science work is included given its longer publication cycles.Source: §1 · pp. 2–3
• The paper is organized into data creation, methods and benchmarks across modalities, and observations on inclusion, ethics, social science, and future directions.It identifies research gaps in sections on text, image, and video tasks and proposes future research based on those gaps.Source: §1 · p. 3

Definitions of Culture

• Psychology treats culture as an extension of human behavior, while anthropology abstracts from behavior to focus on context-dependent human interactions.The paper adopts White (1959)’s synthesis of these perspectives.Source: §2 · p. 3
• The two perspectives complement one another: anthropology informs contextual interpretation, while psychology informs processing context and producing a response.The paper argues that current LLM design often mimics behavior without considering context, such as a user's knowledge level or purpose.Source: §2 · p. 3
• Anthropological cultural context spans human concepts and practices, social interactions, and material objects; these dimensions matter for understanding tasks and generating relevant answers.The paper describes norms, values, morals, linguistic forms, artifacts, concepts, and artifact meanings as cultural knowledge relevant to tasks.Source: §2 · pp. 3–5

Cultural Awareness in LLMs

• Cultural awareness entails recognizing a task's social context, interpreting task elements within it, and understanding how cultural elements vary across cultures.Whether culture matters depends on the task; hate-speech norms can vary, whereas mathematical reasoning is given as an example that is not context-sensitive.Source: §2 · p. 4

Cultural Knowledge and Research Scope

• The survey covers methods for adding and evaluating cultural knowledge, but notes that cross-cultural similarities and differences receive less attention than studying cultures in isolation.It warns that insufficient culture-specific data risks flattening identities and erasing cultural boundaries, especially for underrepresented cultures.Source: §2 · p. 5

3.1 Automatic Pipelines & Model-in-the-Loop

• Automatic pipelines commonly filter large multilingual web corpora for specific cultures, then use the refined data to train culture-specific LLMs.Examples of source corpora include Wikipedia, CC100, mC4, and CulturaX.Source: §3.1 · pp. 5–6

3. Data Creation Methodology

• Figure 1 groups example datasets and benchmarks by creation methodology; * marks semi-automatic pipelines and ** marks fully automatic pipelines.The figure presents an overview of data-creation methodologies and examples.Source: §3 · p. 5, Figure 1 · p. 6

3.1 Automatic Pipelines & Model-in-the-Loop

• Model-in-the-loop methods refine cultural data or generate instruction data, often combining LLM-generated content with cultural sources such as surveys and social media.For vision, automatic pipelines often use geolocalized images and metadata or Wikipedia captions, which the survey cautions may be unreliable for cultural datasets and benchmarks.Source: §3.1 · pp. 6–7

3.2 Semi-Automatic: Human-in-the-Loop

• Semi-automatic approaches combine human expertise with machine processing to balance dataset quality and quantity.Examples include human-curated or annotated inputs followed by machine generation, filtering, or evaluation.Source: §3.2 · p. 7

3.3 Manual: Handcrafted from Scratch

• Manual creation is described as the quality benchmark, but its time and financial demands usually produce smaller datasets than automatic or semi-automatic approaches.Human-created examples include instruction-response pairs, feedback, and language-specific corpora.Source: §3.3 · p. 8
• Manual creation can capture undocumented everyday knowledge and support low-resource languages, but web-based image searching may miss ordinary objects and cultures with limited online presence.Native annotators also contribute to vision-language datasets and subjective tasks such as culturally specific hate-speech detection.Source: §3.3 · pp. 8–9

4.1 Cultural Alignment: Methodologies and Goals

• Training-based alignment uses culture-relevant data: continued pre-training uses raw text, while fine-tuning uses labeled task data and instruction tuning supports culturally aware models.Pre-training from scratch is expensive and therefore uncommon; fine-tuning has been applied to task-specific and general-purpose objectives. Studies report cultural competence gains that remain limited, especially in non-Western contexts, and identify potential pre-training bias; adapter methods such as LoRA may generalize and debias better than full-model training.Source: §4.1.1 · pp. 10, 12
• Training-free prompting includes culture-specific language and anthropological reasoning, diversity critique and self-voting, and prompts that ask models to answer from another society’s perspective.Cultural prompting increased alignment with nationally representative survey data; sociodemographic prompting has potential for data augmentation and social-computing simulation, but may elicit cultural biases and varies across models, sizes, and datasets.Source: §4.1.2 · pp. 12–13
• Goal-specific alignment targets applications including content moderation, response diversity, bias mitigation, and offensive-language detection, using tailored datasets, metrics, prompting, or model training.Examples include Indian stereotype and anti-stereotype data, Korean culture-specific bias mitigation data, and a native-language QA dataset with around 64k annotated pairs spanning seven languages, nine regions, and 18 topics.Source: §4.1.1 · p. 12, §4.1.3 · p. 13
• Despite varied alignment methods, most efforts target individual local cultures; comparatively little work develops cross-cultural LLMs with comprehensive knowledge of multiple cultures.Alternative training-based approaches include Modular Pluralism, unsupervised conversion between traditional and simplified Chinese, and preference-tuning to study cross-cultural commonsense morality.Source: §4.1.1 · p. 12, §4.1.3 · p. 14

4.2 Benchmarks and Evaluation

• Text-based cultural benchmarks cover academic and commonsense knowledge, social values, norms and morals, bias and stereotypes, toxicity and safety, emotional and subjective topics, and linguistics.Evaluation is element-specific: commonsense knowledge commonly uses multiple-choice or short-answer questions, while social values are often assessed with surveys such as the World Values Survey.Source: §4.2 · p. 14
• Regional academic benchmarks adapt educational assessments beyond MMLU’s primarily U.S.-focused knowledge; M3Exam spans nine languages, including extremely low-resource Javanese.Examples include IndoMMLU, which covers nine local cultures and eight local languages in Indonesia, and regional assessments in South Korea, Italy, and China. The survey notes that it remains unclear which academic questions require culturally specific information.Source: §4.2.1 · pp. 15–16
• Commonsense benchmarks address the Western focus of earlier resources through single-culture and multicultural datasets, with examples spanning Indonesia, China, Korea, Taiwan, India, Romania, Basque Country, and Arabic regions.IndoCulture uses sentence completion across eleven Indonesian provinces, while other efforts include datasets involving native speakers and locally specific customs, language, and knowledge; evaluation formats include multiple-choice, causal reasoning, and free-form generation.Source: §4.2.2 · pp. 17–19

Benchmarks and Evaluation

• Cultural benchmarks range from country-specific datasets to multilingual cross-cultural tasks, but language and country do not consistently define cultural boundaries.Some work explicitly represents local differences within diverse countries; benchmarks also vary in whether they rely on native annotators, crowdsourcing, or web resources.Source: §4.2.2 · pp. 20, 22
• Many evaluations still rely on multiple-choice questions, while long-form generation is difficult to assess scalably and with culturally specific methods.The survey calls for robust automatic evaluation, particularly for long-form generation tasks.Source: §4.2.2 · pp. 23–24

Social Values

• Social-value studies often use global surveys, but tend toward country-level analysis and find that models reflect values more aligned with WEIRD societies.The survey identifies a need to capture variation in values within countries and beyond national boundaries.Source: §4.2.3 · pp. 24–25

Social Norms and Morals

• Research evaluates social norms and morals with datasets, benchmarks, and extraction frameworks, yet remains largely English-centered despite cross-cultural differences.Reported findings include stronger LLM adaptability to English-centric cultures and moral biases in multilingual models that can misalign with cultural differences and human judgments; multilingual cross-cultural evaluation is needed.Source: §4.2.4 · pp. 26–27

Social Bias and Stereotype

• Culturally situated bias benchmarks extend evaluation beyond English and Western contexts through multilingual word associations, stereotype sentence pairs, and question-answering tasks.WEAT originated in English in the United States, giving its original stereotype and word lists linguistic and cultural biases; later work adapts or extends bias resources for other languages and regions.Source: §4.2.5 · pp. 27–29

Toxicity and Safety

• Cross-cultural toxicity work extends datasets and safety evaluation beyond English, while LLMs generate more harmful content about Indian caste than Western-centric race attributes.Studies include locally annotated datasets, culturally situated safety prompts, and evaluation of generated content as well as user-to-user communication.Source: §4.2.5 · p. 29, §4.2.6 · pp. 29–30

Social Bias

• Embedding-based and probabilistic bias measures are difficult to apply to proprietary LLMs, while cross-cultural evaluations often cover limited bias categories or cultures.The survey calls for methods that evaluate language models more comprehensively across diverse cultures.Source: §4.2.5 · p. 29

Emotional and Subjective Topics

• Emotion and subjective judgments can reflect cultural or geographic biases, while individual differences make consensus difficult and require accounting for evaluators’ backgrounds.Examples include English and American bias in emotion models, difficulty capturing cultural nuance in code-mixed sentiment, and subjective predictions correlated with local infant survival rates.Source: §4.2.7 · pp. 31–32

Linguistics: Language Properties

• Figurative-expression benchmarks show that language-specific metaphors rely on cultural and regional concepts, while cultural implications account for approximately 16% of MorphEval prediction errors.MABL gathers figurative language from native speakers across seven languages; MorphEval includes culturally significant Chinese morphemes.Source: §4.2.8 · p. 33

Linguistics: Linguistic Variety

• Dialect and historical-language resources capture local or past cultures, but work concentrates on Chinese and Arabic varieties, leaving other languages’ varieties underexplored.Studies include endangered dialect resources, dialect benchmarks, and benchmarks for classical and ancient Chinese.Source: §4.2.8 · pp. 33–34

Linguistics: Literary Forms

• Stories and poems are studied for cultural knowledge and bias, but other literary forms—including fiction genres, journalism, and travel writing—remain underexplored.The surveyed work includes fairy-tale bias analysis, culturally diverse narrative datasets, poetry resources, and datasets for motifs, metaphor, and emotion.Source: §4.2.8 · p. 35

Culturally Adapted Translation

• Culturally adapted machine translation has advanced, but culturally specific content remains difficult to translate, and current work often equates culture with language or nationality.The survey identifies generational preferences, regional dialects, and group terminology as opportunities for more nuanced adaptation; in PAR3, experts significantly preferred human translations.Source: §4.2.8 · pp. 36–37

Culturally Adapted Dialogue Systems

• Early task-oriented dialogue datasets were small, unnatural, and weak at capturing cultural nuance; newer efforts use outline-based annotation and avoid translation artifacts.COD covers Arabic, Indonesian, Russian, and Kiswahili, while MULTI3WOZ is designed for culturally adapted multilingual dialogue.Source: §4.2.8 · p. 37

4.2 Benchmarks and Evaluation

• cuDialog uses Hofstede cultural dimensions across 13 cultures and 5 genres to capture implicit dialogue cues, while Humanoid Agents adapt to emotions and relationships (Cao, Chen, and Hershcovich 2024; Wang, Chiu, and Chiu 2023).Humanoid Agents incorporate System 1 thinking, including basic needs, emotions, and relationship closeness.Source: §4.2.8 · p. 37
• CULTURAL CODES operationalizes cross-cultural inference in 794 games with 7k turns and 153 players; RoCulturaBench spans Romanian culture and improves task performance.CULTURAL CODES uses a collaborative two-player Codenames Duet word-reference game; RoCulturaBench was curated by Romanian humanities academics across artistic and scientific contributions, cuisine, and sports.Source: §4.2.8 · pp. 37–38
• Cultural dimensions and sociocultural priors improve performance and alignment with real-world conversational contexts, but dynamic adaptation during interactions remains a gap.Future work could adapt responses in real time to shifts in tone, topic sensitivity, or cultural context.Source: §4.2.8 · p. 38

5.1 Language Output Tasks

• CULTURALVQA evaluations found stronger VLM cultural understanding for North America than Africa, while benchmarks also test culture- and location-specific commonsense.Culture-specific benchmarks include Chinese, Korean, and Hausa examples, while GD-VCR draws on media from four world regions.Source: §5.1.1 · p. 41
• Multilingual image descriptions contained 29.9% more objects, 24.5% more relations, and 46.0% more attributes than monolingual captions.A captioning study also introduced the Cultural Awareness Score and a 1.5k-image dataset with ground truth for cultural background and context.Source: §5.1.2 · p. 42
• Across visual benchmarks, datasets vary in cultural scope, annotation and image-collection methods, and assumptions about whether geography or language represents cultural diversity.The survey identifies consistent benchmark methods and shared standards for measuring cultural understanding as needs for the research community.Source: §5.1.2 · p. 42

5.2 Image Output Tasks

• A multilingual text-to-image model trained on 18 languages outperformed Stable Diffusion on culture-specific concepts, while other work reported homogenization especially for disadvantaged cultures.Studies also examine single-culture and multicultural generation, cultural clusters, and culture-specific fine-tuning.Source: §5.2 · p. 43
• 25 of 27 countries’ users judged generated images less representative without country-specific prompts, while DALL·E-2 was more representative than Stable Diffusion with such prompts.The comparison covered cultural artifacts in the two text-to-image models.Source: §5.2 · p. 44
• Text-to-image cultural evaluation remains inconsistent: common image-quality metrics miss cultural nuances, and studies select evaluation methods independently.Proposed measures include cultural appropriateness, object presence and localization, semantic consistency, visual aesthetics, and cohesion.Source: §5.2 · pp. 44–45

5.3 Art Forms Related Tasks

• Art-related work spans culturally grounded symbolism tasks, cross-cultural artwork retrieval, multilingual annotations, and multimodal knowledge graphs.Examples include a Chinese-art dataset for identifying visual elements, matching symbolic meanings, and explaining conveyed messages.Source: §5.3 · pp. 45–46

5.4 Miscellaneous Tasks

• Other visual tasks include culturally diverse object recognition, reasoning, captioning, and image translation, with M5 drawing on about 38K photos from 63 regions or countries.The survey also notes that some studies model cultural change over time, while language-based image scraping can merge distinct cultures that share a language.Source: §5.4 · p. 46

6.1 Audio and Speech

• Audio research centers on music recommendation and emotion recognition, but few works jointly model audio and semantic text to understand cultural context beyond emotion.The survey identifies IndicVoices and SEACrowd as initial efforts in culturally diverse speech-text data; their controlled recordings may not capture everyday conversational nuances.Source: §6.1 · pp. 47–48

6.2 Video

• Video cultural adaptation mainly targets emotion detection and content moderation, while cultural humor, personality, and multimodal interaction datasets extend the studied tasks.Examples include cross-cultural conversations, affect recognition, and multimedia hate-speech datasets.Source: §6.2 · pp. 48–49
• Day-to-day interactions and activities remain underrepresented in video research, and the survey identifies culture-specific information extraction and culturally conditioned video generation as areas for further work.The survey notes that video perception varies with cultural background.Source: §6.2 · p. 49

7. Language and Region Coverage

• Most studies collect data in English; for most languages at resource level 2 or below, there are at most seven studies, with Swahili the exception.Chinese, Spanish, and German also appear frequently, while Korean, Indonesian, Bengali, and Swahili are relatively active for their resource levels.Source: §7 · p. 49, Figure 15 · p. 50
• Research focuses mainly on WEIRD countries, East Asia, Indonesia, and India, while Africa, Central and South America, Eastern Europe, and Central Asia are underrepresented.Data sources and data-point volumes also vary across regions, contributing to uneven representation.Source: §7 · pp. 50–51, Figure 15 · p. 50
• The coverage analysis uses country-level categories and cannot account for broad cultural groups crossing national borders or finer sub-regional and historical divisions.The survey gives the broad label “Arab world” as an example of culture being grouped across national boundaries.Source: §7 · p. 51

8.1 Ethics of Cultural Alignment

• Cultural mismatch can contribute to misunderstanding, distrust, and conflict, while alignment with opposing or unsafe values can produce unsafe behavior.The section also flags possible harms from cultural personalization, including dependency, polarization, and labor displacement.Source: §8.1 · pp. 51–52
• Dataset performance gains do not establish that models use cultural knowledge, and expert evaluations report failures to adapt outputs for non-dominant cultures or users’ cultural backgrounds.Internet-scraped data can omit everyday cultural knowledge, leaving a limitation for both model designers and practitioners.Source: §8.1 · pp. 51–52

8.2 Accelerating Social Science Research

• Cultural alignment matters for social-science uses of models because flattening identity or misrepresenting non-majority cultures can cast doubt on inferences drawn from their outputs.The section describes uses including content analysis, survey-response simulation, and agent-based modeling.Source: §8.2 · p. 53
• Using social-science sources to align models and then using generated models in surveys risks a feedback loop that misrepresents cultures and misses cultural change.The section cautions against over-relying on survey data that is synthetic rather than directly collected from people in different cultures.Source: §8.2 · p. 53

8.3 Human Computer Interaction and Cultural Alignment

• Most cultural evaluations focus on model capability, while human interaction and broader systemic impacts receive little attention and are rarely assessed before deployment.Risks depend on the use context, users, and application features; model outputs may also affect users’ political values and opinions.Source: §8.3 · p. 54
• An HCI approach would tailor models to cultural needs and applications, involve cultural stakeholders in co-design, and ensure faithful representation of cultural artifacts.The section identifies generative art as a use case where culturally varied aesthetic standards and emotional responses matter.Source: §8.3 · p. 54

Expand Research on Low-Resource Cultures and Languages; Vary Data Collection Strategies Across Cultures

• Low-resource cultures need more data and evaluation, but native-annotator approaches in low-technology-access regions are not scalable and alignment methods must work with relatively small datasets.Data collection should vary with technology access, while human involvement can help standardize quality across approaches.Source: §9 · pp. 54–55

Approach Defining Cultural Boundaries with Caution

• Country and language are imperfect proxies for culture, so benchmarks should define cultural boundaries carefully and draw on sociolinguistic expertise.Countries may contain diverse local cultures, while a single language can span multiple cultural contexts.Source: §9 · p. 55

Ensure Inclusive Cultural Representations

• Cultural datasets need diverse, sizable annotator pools because norms vary within groups, and agreement checks can test whether gold answers reflect cultural context.Small, homogeneous groups may produce incomplete or biased representations.Source: §9 · pp. 55–56

Develop LLMs that can Adapt and Evolve with Cultural Change over Time

• Because cultures change, models should move beyond static alignment toward preserving culture and responding to ongoing transformations.The section notes a gap in research on adapting LLMs as cultures evolve.Source: §9 · p. 56

Alternative Image Data Collection to Mitigate Biases in Web Images

• Web-image biases can omit everyday objects and cultural concepts; local-guided photography, video frames, and regional television are proposed as alternatives.The section names availability, apprehension, and negative-set biases in web images.Source: §9 · p. 56

Expand Cultural Evaluation Methods to Diverse Interaction Settings

• Multiple-choice evaluation misses implicit cues and natural dialogue, motivating long-form conversational assessment and culturally specific, robust automatic evaluation methods.Current evaluations often depend on human judgment or LLM-as-a-judge methods.Source: §9 · p. 56

Balance Development of Culturally-specific LLMs and Comprehensive Universal LLMs

• Research should balance culture-specific models for local needs with comprehensive cross-cultural models serving global audiences.The section notes comparatively less emphasis on models that reason across diverse cultural contexts.Source: §9 · pp. 56–57

Develop Culturally Aware LLMs from User Perspectives

• Observing real-world user interactions can inform practical model development, including responses to offense and possible uses mediating communication across languages and cultures.The section also suggests studying interactions between multiple LLM agents and humans.Source: §9 · p. 57

• The survey defines cultural awareness in language models by drawing on definitions of culture from psychology and anthropology.Source: §10 · p. 57
• It reviews cross-cultural dataset methods, downstream cultural-inclusion strategies, and approaches for benchmarking cultural awareness.Source: §10 · p. 57
• The review also covers HCI, cultural alignment's role in social science research, and ethical issues, and aims to serve as a reference for future research.Source: §10 · p. 57