PaperlayerView exact revision on arXiv

Source-linked AI summary

DIWALI: Diversity and Inclusivity aWare cuLture specific Items for India: Dataset and Assessment of LLMs for Cultural Text Adaptation in Indian Context

Pramit Sahoo, Maharaj Brahma, Maunendra Sankar Desarkar

arXiv:2509.17399v2cs.CL

TL;DR

LLM evaluation for cultural awareness is constrained by limited culturally grounded datasets and by CSI resources that may omit subregional complexity or contain false positives. DIWALI supplies Indian cultural concepts across facets and subregions for evaluating cultural text adaptation; the authors report inadequate, mostly shallow adaptations by evaluated LLMs.

  • Problem

    Evaluation of cultural awareness is challenging because culturally grounded datasets and metrics are limited, while existing CSI datasets may lack subregional coverage and contain false positives.

    Source: §1 · p. 2

  • Method

    The authors build and manually verify an Indian CSI dataset and evaluate LLM cultural text adaptations using CSI-based, LLM-judge, and human assessments.

    Source: §1 · p. 2

  • Results

    The authors report that LLMs fail to perform cultural text adaptation adequately and mostly produce shallow-level adaptations.

    Source: §9 · p. 9

  • Takeaways & Limitations

    DIWALI provides a resource for assessing cultural text adaptation across Indian subregions and facets.

    Source: §1 · p. 2, §9 · p. 9

    The evaluations cover multilingual, globally diverse models rather than models specifically trained or fine-tuned for country-specific linguistic and cultural contexts.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Large language models (LLMs) are widely used in various tasks and applications. However, despite their wide capabilities, they are shown to lack cultural alignment \citep{ryan-etal-2024-unintended, alkhamissi-etal-2024-investigating} and produce biased generations \cite{naous-etal-2024-beer} due to a lack of cultural knowledge and competence. Evaluation of LLMs for cultural awareness and alignment is particularly challenging due to the lack of proper evaluation metrics and unavailability of culturally grounded datasets representing the vast complexity of cultures at the regional and sub-regional levels. Existing datasets for culture specific items (CSIs) focus primarily on concepts at the regional level and may contain false positives. To address this issue, we introduce a novel CSI dataset for Indian culture, belonging to 17 cultural facets. The dataset comprises ~8k cultural concepts from 36 sub-regions. To measure the cultural competence of LLMs on a cultural text adaptation task, we evaluate the adaptations using the CSIs created, LLM as Judge, and human evaluations from diverse socio-demographic region. Furthermore, we perform quantitative analysis demonstrating selective sub-regional coverage and surface-level adaptations across all considered LLMs. Our dataset is available here: https://huggingface.co/datasets/nlip/DIWALI, project webpage https://nlip-lab.github.io/nlip/publications/diwali/, and our codebase with model outputs can be found here: https://github.com/pramitsahoo/culture-evaluation

1 Introduction

LLMs lack cultural knowledge and competence, while cultural text adaptation requires understanding social norms, historical references, and cultural nuance. DIWALI addresses limited Indian CSI coverage with a sub-regionally grounded dataset and uses it to evaluate LLM adaptations.

1 Introduction

  • Cultural text adaptation requires understanding social norms, subjective historical references, resonating events, and cultural nuances.The paper describes applications including education, dialogue, subtitling, hospitality, advertising, communication, and storytelling.Source: §1 · p. 1
  • DIWALI contains 8,817 cultural concepts across 17 facets and 36 Indian subregions, addressing limited representation in existing datasets.The paper also notes false positives in CANDLE’s cultural concept mappings.Source: §1 · p. 2
  • The paper evaluates seven open-weight models across three LLM families using automatic and human metrics, with a cultural sub-regional analysis.The adaptation task uses existing education-domain text belonging to American culture.Source: §1 · p. 2

2 Culture & Culture Specific Items

The paper treats culture as a complex construct with material and non-material components, and operationalizes it geographically as India and its states and union territories. Culture-specific items are culturally significant concepts, objects, items, or customs that may vary across subregions.

2 Culture & Culture Specific Items

  • Culture is treated as a regional demographic proxy, with India as the region and its states and union territories as subregions.The paper describes culture as encompassing knowledge, beliefs, morals, values, and customs, with both material and non-material components.Source: §2 · p. 2
  • Culture-specific items are culturally important concepts, objects, items, and customs that may vary across subregions.The paper notes that existing CSI datasets such as CANDLE and DOSA cover only subsets of cultural items.Source: §2 · pp. 2–3

3 DIWALI Dataset

DIWALI is a manually verified dataset of 8,817 cultural items across 17 facets and 36 Indian subregions, grouped at the state and union-territory level. Its construction combines facet curation, LLM prompting, web searches, and source and concept checks.

3 DIWALI Dataset

  • DIWALI contains 8,817 culture-specific items across 17 facets, with subregional items grouped at the state and union-territory level.Bodo and Rabha items, for example, are grouped under Assam; food has 1,419 concepts and dance forms 1,105.Source: §3 · p. 3

3.1 Choice of facets

  • The authors curated 17 facets through brainstorming and grouping concepts into material culture, social practices, geography and languages, and names.The dataset extends the limited facet scope of CANDLE.Source: §3.1 · p. 3

3.1.1 Material Culture

  • Material culture covers tangible items through clothing, textiles, jewellery, food, and drinks.These facets include clothing choices, regional weaving traditions, ornaments, food, and traditional drinks.Source: §3.1.1 · p. 3

3.1.2 Social Practices and Activities

  • Social practices and activities include festivals, rituals, traditions, dance forms, traditional games, religion, and arts.These facets represent customs, practices, dance styles, religious activities, and indigenous games.Source: §3.1.2 · p. 3

3.1.3 Geographical locations and Languages

  • Geographical and language facets cover states, capitals, places, architectural styles, and languages and dialects across subregions.The dataset also includes culturally significant names, reflecting subregional naming conventions.Source: §3.1.3 · p. 3, §3.1.4 · p. 3

3.2 Data Construction

  • Initial GPT-4o prompting yielded Wikipedia-sourced concepts, whose limited coverage prompted broader collection using GPT-4o-plus and web searches of official cultural and tourism sites.The authors identify limited Wikipedia coverage and the prompt-based method’s dependence on the LLM’s learned knowledge as constraints.Source: §3.2 · pp. 3–4

3.3 Quality Check

  • Each of the 8,817 items was manually inspected, and items with missing or inaccessible source links were excluded from further validation.Each item was also cross-checked against at least one additional reliable source; incorrect descriptions were discarded.Source: §3.3 · p. 4, §3.3.1 · p. 4, §3.3.2 · p. 4

4 Cultural Adaptation

The paper evaluates American-to-Indian cultural text adaptation by measuring whether models preserve meaning while making text culturally relevant and emotionally resonant. It tests seven models with CSI-based scoring, LLM judges, and human raters across math, dialogue, and story datasets.

4.1 Problem Definition

  • For American-to-Indian adaptation, the task preserves the source text’s semantic intent while aligning it with the target culture and audience.The target text should be culturally relevant and emotionally resonant.Source: §4.1 · p. 4

4.2 Evaluation Dataset

  • Evaluation uses GSM8k and MGSM American-context math test sets, plus 100 DailyDialog and 100 ROCStories examples to test dialogue and story domains.GSM8k and MGSM contribute 1,319 and 250 test samples, respectively.Source: §4.2 · p. 4

4.3 Models

  • Seven 1B–9B open-weight models from the Llama, Mistral, and Gemma families receive English or Bengali instructions to adapt cultural relevance, tone and intent, and sensitivity.The prompt requests structured output containing the adapted text and replaced concepts.Source: §4.3 · p. 4

4.4.1 CSI Adaptation Score

  • The CSI Adaptation Score is the proportion of replaced concepts matched to target concepts, using exact matching first and fuzzy matching when needed.Fuzzy matches must meet a predetermined similarity threshold, set to 80 in these experiments.Source: §4.4.1 · p. 5

4.4.2 LLM as Judge

  • Two LLM judges rate cultural relevance, language fluency, and mathematical integrity on a 0–5 Likert scale.The judges are Llama-3.1-8B-Instruct and Mistral 7B Instruct-v0.3.Source: §4.4.2 · p. 5

4.4.3 Human Evaluation

  • Human evaluation rates cultural relevance for 350 adaptations from seven models, using five socio-demographically diverse Indian annotators; it is limited to English-prompt GSM8k samples.Annotators use a 0–5 Likert scale and assume “Country: India” as the culture proxy.Source: §4.4.3 · p. 5

5 Results and Discussions

DIWALI produces higher adaptation scores than existing CSI datasets across the reported math, dialogue, and story comparisons, though outcomes vary by model and prompt. Human ratings are lower than LLM-judge ratings, and agreement is low for the Mistral model.

5 Results and Discussions

  • The results section compares average adaptation scores from CANDLE, DOSA, and DIWALI using exact and fuzzy matching, with Bengali CSI translations manually verified by a native author.The evaluation also covers the LLM-judge and human-evaluation results.Source: §5 · p. 5, §5.1 · pp. 5–6

5.1 Existing vs. DIWALI: Average Adaptation Scores

  • 0.615 fuzzy and 0.855 exact match for Llama-2-7b-chat-hf on GSM with an English prompt using DIWALI, versus 0.050 and 0.028 with CANDLE.The authors report this pattern across models and say DIWALI is more sensitive in capturing adaptation capabilities.Source: Table 1 · p. 6, §5.1 · p. 6
  • 0.842 exact and 0.573 fuzzy AAS for Llama-2-7B-chat-hf on DailyDialog with DIWALI; all models score substantially higher with DIWALI than CANDLE or DOSA.The authors report the same trend for ROCStories, where the model scores 0.969 with DIWALI versus 0.020 with CANDLE.Source: Table 2 · p. 6, §5.1 · p. 6, Table 3 · p. 6
  • 0.494 exact match for Llama-3.1-8B-Instruct on MGSM with DIWALI, versus ≤ 0.016 with CANDLE and DOSA; a similar DIWALI advantage is reported for Bengali prompts.For Bengali MGSM, Llama-2-7B-chat-hf scores 0.612 with DIWALI versus 0.122 with CANDLE and 0.265 with DOSA; Llama-3.2-1B-Instruct stays near zero across methods.Source: §5.1 · p. 6

5.2 LLM as Judge Results

  • LLM judges rate evaluated models consistently well across cultural relevance, language fluency, and mathematical integrity, with Gemma-2-9B-Instruct and Mistral-7B-Instruct-v0.3 scoring highly for both judges.The cited passage does not provide the score values.Source: §5.2 · p. 7

5.3 Human Evaluation Results

  • 2.68 is the highest average human cultural-relevance score, for Llama-3.1-8B-Instruct, while Llama-3.2-1B-Instruct scores the lowest at 0.75; LLM judges assign higher scores than humans.The authors suggest LLM-as-judge may not be appropriate for evaluating cultural text adaptation.Source: §5.3 · p. 7
  • Inter-annotator agreement is acceptable for Llama- and Gemma-family models, but low for Mistral-7B-Instruct.Agreement is measured using Cohen’s κ on pairs of human raters for each model.Source: §5.3 · p. 7

6 Analyses

The analyses find that cultural adaptations are uneven across Indian subregions and often replace concepts without connecting them to culturally meaningful scenarios. They examine this pattern through subregional heatmaps, examples of adapted text, and evaluation tables.

Sub-regional coverage

  • Food adaptations by Gemma-2-2B-Instruct cluster in Uttar Pradesh, Madhya Pradesh, Maharashtra, and Punjab, with none from Northeastern states.Similar subregional skews appear across the other facets.Source: §6 · p. 7
  • The analysis maps replaced concepts to states or union territories and uses facet heatmaps to examine whether regional-level adaptation fairly represents subregional concepts.Figure 3 presents the Food-facet heatmaps; the remaining 16 facets appear in Appendix I.Source: §6 · p. 7, Figure 3 · p. 8

Evaluation

  • The evaluation tables cover LLM-as-Judge and human cultural-adaptation scores, alongside inter-annotator agreement on Cultural Relevance.The supplied table captions identify the evaluation-score comparison and Cohen’s κ agreement measure.Source: Table 4 · p. 8, Table 5 · p. 8, §6 · p. 8

Surface vs. Deep level

  • LLMs often replace an event without adapting its scenario, leaving the resulting text without meaningful or emotionally resonant cultural connection.Llama-3.1-8B-Instruct changes Tuesday to Diwali while leaving the scenario of selling CDs unchanged.Source: §6 · pp. 7–8

7 Related Work

Prior Indian culture-specific-item resources offer either limited concept coverage or rich participatory data that is difficult to scale. DIWALI addresses these gaps with coverage across 17 facets and 36 Indian subregions.

CANDLE

  • CANDLE contains 650 India-focused concepts across five facets, despite using web-scale text extraction and clustering.Its listed facets are clothing, drinks, cuisines, rituals, and traditions.Source: §7 · p. 8

DOSA

  • DOSA gathers 615 artifacts from 260 participants across 19 Indian states, capturing diverse cultural dimensions through participatory research.The passage notes that participatory methods are difficult to scale.Source: §7 · pp. 8–9

DIWALI

  • DIWALI addresses gaps in prior resources with a cultural-specific-item collection spanning 17 facets and 36 Indian subregions.The related-work discussion describes participatory collection as difficult to scale and existing resources as limited in their representation of Indian concepts.Source: §7 · pp. 8–9

8 Usage of CSIs

Culture-specific items can support cultural competence evaluation, cross-cultural translation, and cultural adaptation. The section presents them as useful resources for identifying and appropriately handling culture-specific concepts across these tasks.

Cultural competence evaluation

  • Culture-specific items can support cultural competence evaluation by providing concepts used to measure LLM cultural biases.The passage cites Culture Bias Score and edit-level analysis as examples of CSI-reliant methods.Source: §8 · p. 9

Cross-culture translation

  • Comprehensive culture-specific-item collections can help translation systems identify and appropriately translate culture-specific entities.The passage connects CSIs to culturally aware translation.Source: §8 · p. 9

Cultural adaptation

  • CSIs can support cultural adaptation in open text generation, including story writing and dialogue.The passage describes CSIs as enabling more holistic adaptation.Source: §8 · p. 9

9 Conclusion

DIWALI provides a manually verified Indian cultural-specific-item dataset spanning 36 subregions and 17 facets, which the paper uses to evaluate LLM cultural text adaptation. Human evaluation finds inadequate and mostly shallow adaptations, motivating further work on models and training methods that grasp cultural context.

Conclusion

  • DIWALI contains 8,817 manually verified cultural concepts across 17 facets and 36 Indian subregions.The paper also uses the dataset to evaluate LLMs on cultural text adaptation.Source: §9 · p. 9
  • Human evaluation finds that LLMs do not perform cultural text adaptation adequately and mostly produce shallow-level adaptations.Source: §9 · p. 9
  • The authors call for future research on models and training methods that can grasp the nuanced context behind cultural concepts.Source: §9 · p. 9

Limitations

The study’s limitations concern model coverage, limited analysis of adaptation depth, and the narrow regional reach of its human evaluation.

Limitations

  • Evaluations cover widely used multilingual models, not models pretrained or fine-tuned specifically for country-level linguistic and cultural contexts.The authors intend to expand the study to country-specific LLMs.Source: Limitations · p. 10
  • Analysis of surface- versus deeper-level generations draws on only a few examples.The authors state that studying cultural “aboutness” requires analysis by a large cultural population.Source: Limitations · p. 10
  • Human evaluation involved five annotators from five socio-demographic regions, and the authors call for broader coverage across such regions.They identify this broader sourcing as needed for more diverse and fair evaluation.Source: Limitations · p. 10

Ethical Considerations

The appendix documents DIWALI’s curation and evaluation procedures, dataset scope, and reporting choices, alongside ethical practices and model-comparison findings.

Ethical Considerations

  • The dataset uses public sources, stores no personal information, and is intended for academic research and cultural analysis with care to avoid misrepresentation.Human evaluators’ personal information was also not collected; annotators were research students whose remuneration came through their research assistantships.Source: Ethical Considerations · p. 10

A CANDLE False Positives

  • A prompt-based relevance check followed by manual inspection removed 393 false-positive Indian cultural concepts from CANDLE.The passage identifies CANDLE concepts as limited and containing false positives; Table 7 concerns false-positive concepts for India.Source: A CANDLE False Positives · p. 13, Table 7 · p. 13

B DIWALI Details

  • DIWALI organizes 17 facets across 36 States/UTs, grouping sub-sub-regional distinctions under broader regions for consistency.The dataset includes facet, concept, description, subregion, and source-link fields; Table 9 provides samples.Source: Table 8 · p. 13, B DIWALI Details · p. 13, Table 9 · p. 14

C Decoding Strategies and Inference Setup

  • Experiments use temperature 0 and greedy decoding for deterministic outputs, with up to 2,048 new tokens per prompt on NVIDIA A100 GPUs.Generation was performed in a single run for the entire dataset; experiments used one or multiple GPUs as needed.Source: C Decoding Strategies and Inference… · pp. 14–15

D.1 Average Input/Output Lengths

  • Table 11 reports average input and output lengths for GSM8k cultural-adaptation examples and gives a rough token-count rule of about four characters per token.The caption provides no specific length values.Source: Table 11 · p. 15

D.2 Significance Testing

  • The analysis reports mean exact-match accuracy with 95% bootstrap confidence intervals and uses paired approximate randomization with Benjamini–Hochberg correction.The bootstrap and sign-flip tests each use 10,000 resamples or permutations; results are called significant at q < 0.05.Source: §D.2 · p. 15
  • All seven models perform significantly better on DIWALI than CANDLE, and every pairwise model comparison on DIWALI is significant after correction.The reported largest pairwise gap is +0.37 pp, for Llama-3.2-1B-Instruct over Gemma-2-9B-Instruct.Source: §D.2 · p. 16, Table 13 · p. 16

E Likert Scale; F Instructions for Human Evaluation; G Human Annotators

H Human evaluation vs. LLM as Judge: Comparison on Cultural Relevance

LLM judges assign higher cultural-relevance scores than human evaluators, while the appendix also presents sub-regional heatmaps and adaptation prompts.

H Human evaluation vs. LLM as Judge: Comparison on Cultural Relevance

  • Across evaluated models, LLM judges give cultural-relevance scores about 0.5 to 2.5 points higher than human evaluators.The authors hypothesize that judges overvalue surface-level entity changes such as proper-noun swaps, and caution that this may make them unreliable for deeper adaptation nuances.Source: H Human evaluation vs. LLM as Judge:… · p. 16

I Sub-regional Adaptation Coverage

J Prompts

  • The adaptation prompts require contextually appropriate cultural substitutions while preserving original meaning, logical sense, tone, and numerical or mathematical content.They also instruct against transliteration, solving the problem, or introducing factual errors.Source: J Prompts · p. 26
  • The prompt requests a JSON array containing adapted text and only meaningful changed concepts in a replacement dictionary.Unchanged terms are excluded, and the instructions specify a single-line valid JSON response.Source: J Prompts · p. 26
Loading 2509.17399v2…