PaperlayerView exact revision on arXiv

Source-linked AI summary

Toward Culturally Grounded Natural Language Processing

Sina Bagheri Nezhad

arXiv:2603.26013v2cs.CL

TL;DR

Multilingual coverage does not guarantee cultural competence, and the two have often been studied separately. This paper synthesizes more than 50 papers to connect them through the communicative ecologies in which language is used. Across the literature, cultural performance varies with task framing and supervision, supporting evaluation that attends to social context and community validation.

  • Problem

    Linguistic coverage and cultural competence can diverge, while research on multilingual transfer and culture-focused evaluation has often been treated separately.

    Source: §1 · pp. 1–2

  • Method

    The paper synthesizes over 50 papers on multilingual inequality, transfer, cultural evaluation, alignment, multimodal benchmarks, benchmark critique, and community-grounded data practices.

    Source: §1 · p. 2

  • Results

    Across text, values, emotion, conversation, multimodality, and low-resource evaluation, models can transfer technically while still failing culturally [Belay et al., 2025].

    Source: §7 · p. 8

  • Takeaways & Limitations

    Culture-sensitive NLP should report whether evaluations use translated or native-authored items, which varieties and settings they test, and which communities validated the data and labels.

    Source: §7 · p. 8

    The evidence base is uneven across regions, modalities, and tasks, despite spanning more than 50 papers.

    Source: §8 · p. 8

Abstract

from arXiv · show

Multilingual NLP is often treated as a route to global inclusion, but linguistic coverage and cultural competence frequently diverge. This paper synthesizes over 50 papers spanning multilingual performance inequality, cross-lingual transfer, culture-aware evaluation, cultural alignment, multimodal benchmarks, benchmark-design critique, and community-grounded data practices. Across this literature, training data coverage remains important, but outcomes are also shaped by tokenization, prompt language, translated benchmark design, culturally grounded supervision, modality, and who authors or validates evaluation data. We argue that culturally grounded NLP should move beyond treating languages as isolated rows in benchmark tables and instead model communicative ecologies: the institutions, scripts, domains, modalities, and communities through which language is used. We propose a layered evaluation and reporting agenda centered on representation audits, mixed elicitation, ecological validity, community validation, adaptation provenance, within-language variation, and maintenance of living cultural resources.

1 Introduction

Multilingual coverage does not ensure cultural competence: models may transfer technically while misreading local norms, knowledge, or interaction. Synthesizing more than 50 papers, the paper connects multilingual performance research with culture-focused evaluation and argues for studying language in its communicative ecology.

  • Linguistic coverage and cultural competence diverge: models can be fluent across languages yet mis-handle local norms, culturally specific entities, or dominant-worldview assumptions [Belay et al., 2025; Havaldar et al., 2025].Source: §1 · p. 1
  • Coverage and transfer research identifies persistent resource disparities and predictors including pretraining distribution, lexical overlap, script, tokenization, and typology, but does not establish alignment with community values or norms.Source: §1 · p. 1
  • Text, multimodal, and interactional evaluations reveal cultural failures, while adaptation studies indicate that progress depends on locally grounded supervision [Guo et al., 2025; Liu et al., 2025a; Feng et al., 2025].Source: §1 · pp. 1–2
  • Communicative ecologies comprise the social and technical conditions of language use, including institutions, scripts, domains, modalities, and communities of practice.Source: §1 · p. 2
  • The synthesis connects multilingual transfer with culture-oriented evaluation, identifies design factors shaping cultural performance, and proposes communicative ecologies as a research framing.Source: §1 · p. 2

2 From Multilinguality to Cultural Competence

Language coverage and transfer explain some performance variation, but technical transfer does not establish culturally grounded behavior. Culture is often measured through partial proxies, motivating a shift toward communicative ecologies.

2.1 Coverage, transfer, and the limits of language-only evaluation

  • Resource concentration and infrastructure advantages correlate with multilingual performance gaps, which remain when evaluation expands to hundreds of languages and dialects [Joshi et al., 2020; Blasi et al., 2022; Adelani et al., 2024; Team et al., 2022; Aji and Cohn, 2025].Source: §2.1 · p. 2
  • Pretraining data, lexical sharing, script, and tokenization help explain cross-lingual transfer, but say less about local norms, values, or region-specific knowledge [Philippy et al., 2023; Limisiewicz et al., 2023; Hämmerl et al., 2025; Bagheri Nezhad and Agrawal, 2024; Bagheri Nezhad et al., 2025].Source: §2.1 · p. 2
  • Token or script overlap can support transfer while culturally loaded metaphors, local emotional expressions, regional entities, and social norms remain under-specified [Belay et al., 2025].Source: §2.1 · p. 2

2.2 Culture is usually operationalized through proxies

  • Culture is commonly approximated through countries, languages, surveys, or artifacts, so a benchmark may test only a thin slice such as national values or cultural trivia [Adilazuarda et al., 2024; Liu et al., 2025a].Source: §2.2 · p. 2
  • The paper frames these proxies as partial windows and proposes communicative ecologies to situate language use in communities, institutions, domains, and modalities.Source: §2.2 · p. 3, Figure 1 · p. 3

3 What Recent Benchmarks and Adaptation Work Show

Recent work finds cultural gaps across text, multimodal, and interactional evaluation, while showing that task framing and locally grounded supervision shape observed behavior. Benchmark design and representation also determine which cultural knowledge and communities evaluations capture.

3.1 Text and value-oriented evaluation

  • Multilingual and monolingual models can prefer Western-associated entities in Arabic, and multilingual ability does not reliably predict cultural alignment.Source: §3.1 · p. 3
  • Prompt language and persona prompting affect cultural outputs, while tuning can improve alignment across cultures without fully resolving internal diversity or task variation [AlKhamissi et al., 2024].Source: §3.1 · pp. 3–4

3.2 Multimodal, interactional, and local evaluation

3.3 Adaptation and alignment methods

3.4 Evaluation paradigms, representation, and ecological validity

  • Open-ended assessment, community co-design, and task realism affect what cultural competence evaluations represent, while scaled resources raise questions about provenance and loss of local detail [Kabir et al., 2025].Source: §3.4 · p. 5

4 Synthesis: Recurring Findings Across the Literature

Across the literature, data coverage matters but does not determine cultural performance on its own. Benchmark assumptions, supervision, modality, within-language variation, and social context all shape what models learn and what evaluations measure.

4.1 Data quantity remains necessary but insufficient

  • Training-data coverage remains a strong performance predictor, but tokenizer allocation, token alignability, prompt language, and pretraining composition mediate the path from coverage to performance.Source: §4.1 · pp. 5–6

4.2 Benchmark design often imports external cultural assumptions

4.3 Local supervision and participation matter

4.4 Culture is distributed across modalities and within-language variation

4.5 From language lists to communicative ecologies

  • The synthesis recommends treating languages as elements of communicative ecologies, since contextual covariates and social situatedness matter beyond the language present or raw data amount [Bagheri Nezhad et al., 2025; Liu et al., 2025a; Zhou et al., 2025; Rystrøm et al., 2025].Source: §4.5 · p. 6

5 Toward Better Evaluation Protocols

The section proposes evaluating cultural competence through a layered protocol rather than a single benchmark score. It emphasizes representation, elicitation, situated use, supervision provenance, and temporal maintenance.

5 Toward Better Evaluation Protocols

  • Cultural-competence evaluation should separate representation, elicitation, ecological validity, validation, and adaptation-data questions rather than collapse them into one score.Table 2 summarizes practical shortcuts and stronger alternatives.Source: §5 · p. 6, Table 2 · p. 7
  • Benchmark reports should identify authorship, language varieties, targeted groups, and local validation, while treating missing-group analysis as core evaluation (Aji and Cohn, 2025; Alkhamissi et al., 2026).Country and language labels can conceal within-language gaps.Source: §5 · p. 6
  • Mixed elicitation should pair closed-form surveys or multiple-choice QA with open generation, preference judgments, disagreement analysis, and qualitative error coding (Kabir et al., 2025; Havaldar et al., 2025).These protocols can distinguish absent knowledge, plausible cultural variation, and norm violations.Source: §5 · pp. 6–7
  • Cultural-competence claims should be limited to tested settings, including relevant modalities, dialects, code-mixed inputs, local entities, conversation, or agentic interaction.Performance can degrade when models must integrate local cues in more realistic contexts.Source: §5 · p. 7
  • Adaptation reports should document the social provenance of supervision, since native preference data, codesigned datasets, and culture-specific instructions form part of the causal explanation (Guo et al., 2025).Without provenance or stakeholder documentation, benchmark claims should be narrower.Source: §5 · p. 7
  • Cultural resources should be versioned, refreshed, and revalidated as social norms, salient entities, public events, and visible communities change.Static releases risk freezing a historical snapshot of local salience.Source: §5 · p. 7

6 A Research Agenda for Culturally Grounded NLP

The research agenda calls for culturally grounded evaluation and reporting that makes benchmark assumptions auditable and attends to variation within and across communities. It treats participation, adaptation, and resource maintenance as continuing parts of the work.

6 A Research Agenda for Culturally Grounded NLP

  • Separate multilingual coverage from cultural competence by comparing translated and native-authored items, sensitive and agnostic subsets, and monolingual and interactional settings.Aggregate multilingual accuracy alone is insufficient.Source: §6 · p. 7
  • Publish contextual metadata and benchmark slices for scripts, domains, registers, authorship, translation pipelines, represented communities, and validation; make assumptions auditable rather than creating one scalar culture score (Liu et al., 2025a; Alkhamissi et al., 2026).Source: §6 · pp. 7–8
  • Combine closed and open-ended elicitation, pairwise judgments, and qualitative analysis to distinguish norm violations from reasonable local variation (Kabir et al., 2025).Source: §6 · p. 8
  • Prefer native-authored and locally validated data where possible, since native preference data, diverse raters, local prompts, and community-grounded datasets surface different errors and can improve adaptation quality (Guo et al., 2025; Kim and Lee, 2025; Müller-Eberstein et al., 2025).Participation is framed as research infrastructure, not optional review.Source: §6 · p. 8
  • Evaluate multimodal artifacts, local entities, politeness registers, dialects, and narrative traditions beyond text-only settings and standardized varieties (Maji et al., 2025; Aji and Cohn, 2025).Source: §6 · p. 8
  • Treat cultural alignment as task- and community-specific ongoing localization that must accommodate pluralism and change, rather than a one-time correction (AlKhamissi et al., 2024; Liu et al., 2025a; Feng et al., 2025).Source: §6 · p. 8
  • Make community-engaged data creation and participatory design part of the experimental setup, since culturally grounded datasets can affect which harms and categories are surfaced.Source: §6 · p. 8
  • Maintain cultural resources through refresh cycles, re-annotation, and versioned slices rather than one-off releases, to support changing coverage and avoid entrenching majority defaults (Alkhamissi et al., 2026).Source: §6 · p. 8

7 Conclusion

The conclusion argues that multilingual capability does not by itself yield cultural competence. It calls for benchmark design, contextual reporting, community-grounded supervision, and evaluation that recognizes languages within communicative ecologies.

7 Conclusion

  • Across text, values, emotion, conversation, multimodality, and low-resource evaluation, evidence shows that models can transfer technically while still failing culturally (Belay et al., 2025).Source: §7 · p. 8
  • Culture-sensitive NLP should become a standard reporting axis alongside multilingual accuracy, robustness, and safety, with broad-generalization claims specifying tested items, varieties, settings, and validating communities.The conclusion warns that without this shift, systems may remain globally legible yet locally brittle.Source: §7 · p. 8

8 Limitations

The synthesis is limited by uneven literature coverage and by the partial measures of culture used in the underlying work. Because culture-aware evaluation and alignment are changing quickly, the proposed research agenda is a living program.

  • The synthesis’s claims depend on the coverage and quality of literature that is uneven across regions, modalities, and tasks.Source: §8 · p. 8
  • Culture is often operationalized through partial proxies, and no single measure is claimed to capture it exhaustively.Source: §8 · p. 9
  • Rapidly changing culture-aware evaluation and alignment mean the proposed agenda should be read as a living program, not a closed taxonomy.Source: §8 · p. 9

9 Ethical Considerations

Operationalizing culture risks flattening internal diversity, while optimizing average performance may reinforce existing inequalities. Participatory alignment also requires transparency about whose judgments are represented and which disagreements remain unresolved.

  • Country, language, script, and survey averages can flatten internal diversity, especially for Indigenous, diasporic, minoritized, or non-standard language communities.Source: §9 · p. 9
  • Optimizing only for average performance may reinforce inequalities by favoring communities with stronger representation, stable orthographies, or abundant culturally relevant supervision.Source: §9 · p. 9
  • Native preference learning and local annotation do not resolve internal diversity, substantive disagreement, or changing cultural norms, so represented judgments and unresolved disagreements require transparency.Source: §9 · p. 9

Related papers

Loading 2603.26013v2…