PaperlayerView exact revision on arXiv

Source-linked AI summary

NormAd: A Framework for Measuring the Cultural Adaptability of Large Language Models

Abhinav Rao, Akhila Yerukola, Vishwa Shah, Katharina Reinecke, Maarten Sap

arXiv:2404.12464v10cs.CL

TL;DR

Prior evaluations often test what LLMs know about cultural norms, rather than whether they can apply them to user-specific situations. NormAd assesses social-acceptability judgments across levels of cultural context using NormAd-Eti, and finds that models struggle across these settings, with stronger performance for English-speaking countries. The benchmark uses etiquette situations from 75 countries, while its cultural-norm source captures dominant narratives rather than all within-country variation.

  • Problem

    Prior work has largely probed cultural knowledge directly, leaving less evidence about models' ability to apply it flexibly to user-specific scenarios.

    Source: §1 · p. 1

  • Method

    NormAd evaluates social-acceptability judgments across country, value-plus-country, and explicit rule-of-thumb contexts, instantiated in NormAd-Eti's etiquette situations from 75 countries.

    Source: §1 · p. 2, §7 · p. 9

  • Results

    Models struggle across context levels, particularly COUNTRY and COUNTRY +VALUE, lag behind humans, and perform best for English-speaking countries; they also do better on norm-conforming situations.

    Source: §7 · p. 9

  • Takeaways & Limitations

    The findings underscore the need for better contextualization and more nuanced cultural adaptability in LLMs.

    Source: §7 · p. 9

    NormAd uses etiquette norms as a proxy for culture, and its Cultural Atlas source captures dominant national narratives while leaving within-country variation underrepresented.

    Source: §8 · p. 9

Abstract

from arXiv · show

To be effectively and safely deployed to global user populations, large language models (LLMs) may need to adapt outputs to user values and cultures, not just know about them. We introduce NormAd, an evaluation framework to assess LLMs' cultural adaptability, specifically measuring their ability to judge social acceptability across varying levels of cultural norm specificity, from abstract values to explicit social norms. As an instantiation of our framework, we create NormAd-Eti, a benchmark of 2.6k situational descriptions representing social-etiquette related cultural norms from 75 countries. Through comprehensive experiments on NormAd-Eti, we find that LLMs struggle to accurately judge social acceptability across these varying degrees of cultural contexts and show stronger adaptability to English-centric cultures over those from the Global South. Even in the simplest setting where the relevant social norms are provided, the best LLMs' performance (< 82\%) lags behind humans (> 95\%). In settings with abstract values and country information, model performance drops substantially (< 60\%), while human accuracy remains high (> 90\%). Furthermore, we find that models are better at recognizing socially acceptable versus unacceptable situations. Our findings showcase the current pitfalls in socio-cultural reasoning of LLMs which hinder their adaptability for global audiences.

1 Introduction

LLM cultural evaluations have often tested what models know about cultural norms rather than whether they can apply them flexibly to user-specific scenarios. NORMAD evaluates social-acceptability judgments across levels of cultural context, and experiments find persistent difficulties and stronger performance for English-speaking and European norms than African-Islamic norms.

1 Introduction

  • The authors argue that cultural adaptability matters for inclusive and effective service across evolving cultures; otherwise, users risk disproportionate service quality and cultural alienation.They discuss user-provided cultural context as one approach to support adaptation.Source: §1 · pp. 1–2
  • Earlier evaluations largely probed cultural knowledge directly, leaving less evidence about whether models apply it flexibly to user-specific scenarios.The introduction frames cultural adaptability as adjusting outputs to the user's cultural context.Source: §1 · p. 1
  • NORMAD evaluates social acceptability across specific country contexts, abstract values paired with countries, and fine-grained rules-of-thumb.The framework tests whether models can apply cultural norms to described situations, rather than only answer direct questions about cultural knowledge.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • NORMAD-ETI contains 2.6k social-situation descriptions from 75 countries to evaluate judgments of etiquette across cultural contexts.The benchmark draws on Cultural Atlas norms and includes question-answer pairs about whether specific actions are socially acceptable.Source: §1 · p. 2
  • Models struggle with social-acceptability judgments across cultural contexts, with particular difficulty in VALUE and COUNTRY settings and stronger performance for English-speaking and European norms.The introduction also reports that larger models and preference-tuning methods improve performance, with gains more pronounced for English-speaking and European norms than African-Islamic cultures.Source: §1 · p. 2

2 Related work

Prior LLM culture research has mainly assessed cultural knowledge, values, biases, and persona adoption, while this work focuses on applying cultural knowledge in user-specific situations. It also frames its scenarios as potentially pluralistic because social acceptability can depend on context.

2.1 Culture in LLMs

  • Most prior evaluations examine cultural knowledge, values, artifacts, or biases, rather than applying cultural knowledge to user-specific scenarios.Related work also describes efforts to adopt regional personas, fine-tune social norms, or improve culture-specific task performance.Source: §2.1 · pp. 2–3

2.2 On Value Pluralism and Personalization of LLMs

  • The paper treats its work as pluralistic because a situation may have conflicting social acceptabilities depending on context.Related approaches include benchmarks, pluralistic or prosocial fine-tuning, and modular frameworks for value pluralism.Source: §2.2 · p. 3

3 NORMAD Evaluation Framework

NORMAD measures cultural adaptability through judgments of described social situations, rather than cultural knowledge alone. It varies the available cultural context from country information to abstract values and explicit rules-of-thumb.

3 NORMAD Evaluation Framework

  • NORMAD defines cultural adaptability as personalizing or adapting model outputs toward multicultural users and evaluates performance across diverse user-provided cultural contexts.The framework is intended to measure adaptability, in contrast to work that primarily measures cultural knowledge.Source: §3 · p. 3
  • The framework uses free-text social situations with multiple characters, intentionally omitting explicit cultural or geographical markers.Each scenario asks whether a character's action is socially acceptable.Source: §3 · p. 3
  • The RULE-OF-THUMB setting provides the detailed norm needed to answer, making the task an entailment problem for humans and models.The example rule is to avoid eating with the left hand.Source: §3 · p. 3
  • The COUNTRY setting requires models to infer norms from a country name, while VALUE +COUNTRY adds an abstract value to that context.In the example, the model must infer a dining norm from India, either from the country alone or alongside the value of hygiene in dining.Source: §3 · p. 3

4 NORMAD-ETI Construction

NORMAD-ETI is built from etiquette norms into grounded situations, then filtered and human-validated. The construction also measures human judgments under different levels of cultural context.

4 NORMAD-ETI Construction

  • NORMAD-ETI’s pipeline combines situation generation, automatic filtration, human validation, and a small-scale assessment of human performance.The benchmark is grounded in Cultural Atlas material and spans 75 countries.Source: §4 · p. 3, Figure 2 · p. 4

4.1 Social Situation Description

  • Cultural Atlas etiquette norms were transformed into nine situations per country and subcategory, with three examples per label and corresponding ROT and VALUE.Neutral situations used norms from a different Inglehart-Welzel cluster, and descriptions omitted direct geographic indicators.Source: §4.1 · pp. 4–5

4.2 Automatic Filtration

  • After filtration, the benchmark contains 2,633 stories across 75 countries and three labels.Three rounds regenerated descriptions inconsistent with their gold labels; further checks discarded 73 ROT-alignment and 20 VALUE-abstraction mismatches.Source: §4.2 · p. 5

4.4 Verification of Human Performance

  • Humans achieved 95.6% accuracy with Situational Description + ROT, with κ = 0.95 agreement between annotators.The assessment sampled 480 cases across subcategories, labels, and cultural bins.Source: §4.4 · p. 6

Situational Description + VALUE + COUNTRY

  • Humans from India, China, and South Korea achieved 90% accuracy with Situational Description + VALUE + COUNTRY.The study averaged results across three countries; inter-rater agreement was Krippendorff’s α = 0.45 and Fleiss’s κ = 0.63.Source: Situational Description + VALUE +… · p. 6

5 Experimental Setup

The experiments assess open- and closed-source models across cultural-context levels. Accuracy on a ternary acceptability label is the evaluation metric.

5 Experimental Setup

  • The experimental setup evaluates language models’ ability to adapt across varying levels of cultural context.The evaluation uses NORMAD-ETI.Source: §5 · p. 6, §5.1 · p. 6

5.1 Models

  • The evaluated models span open- and closed-source systems with varied parameter counts and fine-tuning objectives.The model set is used to assess cultural adaptability on NORMAD-ETI.Source: §5.1 · p. 6

5.2 Setup and Metrics

  • Models judge social acceptability under ROT, VALUE + COUNTRY, and COUNTRY contexts, with accuracy measured against ternary Yes, No, and Neutral labels.ROT tests contextual reasoning, while VALUE + COUNTRY and COUNTRY assess retrieval of relevant knowledge and its application in reasoning; temperature is set to 0.0.Source: §5.2 · p. 6

6 LLM Culture Adaptability Results

Models show substantial difficulty judging social acceptability across cultural contexts, with performance varying by cultural zone, etiquette category, and ground-truth label. Larger models and KTO perform better in some settings, but important gaps remain.

6.1 How well do models perform across different levels of cultural contexts?

  • 59–63% accuracy for VALUE + COUNTRY and 51–56% for COUNTRY were the best models’ results, while human accuracy reached 90% in those settings.Under RULE-OF-THUMB, GPT-4 reached 87.6%, Mistral-7b-Instruct 81.8%, and Llama-2-70b-chat 71.3%, versus 95.6% for humans.Source: §6.1 · pp. 6–7
  • Performance improves with model size on RULE-OF-THUMB, but the largest models may underperform with COUNTRY context, possibly because it provides insufficient cultural cues.The authors note that the study data was generated with GPT-4, which may give it an unfair advantage, although GPT-4 still struggles.Source: §6.1 · p. 7

6.2 How well do models perform across the Inglehart-Welzel (IW) cultural map?

  • In COUNTRY + VALUE, top models perform better on English-speaking than African-Islamic cultural zones, while poorer-performing models underperform across zones.DPO and KTO bring marginal gains over PPO for 7b models; for 30b models, performance improves significantly, with KTO most effective under RULE-OF-THUMB.Source: §6.2 · pp. 7–8, Figure 4 · p. 7, Figure 5 · p. 8

6.3 What is the performance across subcategories of NORMAD-ETI?

  • Models underperform on Gifting, even with RULE-OF-THUMB, while they excel in Eating and improve in Visiting and Basic Etiquette.Gifting norms can specify details such as presentation, number, and color; COUNTRY + VALUE or RULE-OF-THUMB mitigates the Gifting drop for some, but not all, models.Source: §6.3 · p. 8

6.4 How well do models do across social acceptabilities (Yes/No/Neutral)?

  • Models perform better on norm-conforming “Yes” situations; “Neutral” accuracy reaches only 42% even under RULE-OF-THUMB, versus 98% for humans.Models improve on norm-violating “No” situations as more context is supplied, but performance degradation on Neutral suggests potential overconfidence.Source: §6.4 · pp. 8–9, Figure 6 · p. 8

7 Conclusion

The paper introduces NORMAD and its NORMAD-ETI benchmark to evaluate LLMs’ cultural adaptability across social contexts. Experiments find persistent performance gaps, cultural-zone skew, and difficulty with Gifting and nonconforming situations.

7 Conclusion

  • NORMAD evaluates contextual adaptability, and NORMAD-ETI contains 2.6k etiquette situations from 75 countries across norm-specificity levels, four subcategories, and three labels.The four subcategories are Basic Etiquette, Eating, Visiting, and Gifting; the labels are Yes, No, and Neutral.Source: §7 · p. 9
  • Models struggle across context levels, particularly COUNTRY + VALUE and COUNTRY; larger models and KTO perform better, but English-speaking zones outperform others.Models also struggle with Gifting norms and perform better on situations conforming to social norms.Source: §7 · p. 9
  • The findings underscore the need for better contextualization and more nuanced cultural adaptability in LLMs.Source: §7 · p. 9

8 Limitations

The evaluation is bounded by its choice of cultural proxy, dataset representation, language, and treatment of culture as static. These constraints limit which aspects of cultural diversity and change the benchmark can capture.

8 Limitations

  • NORMAD uses etiquette-related social norms as a semantic proxy for culture, so broader proxies are needed to capture the full cultural landscape.The framework uses a black-box evaluation approach and analyzes demographically informed axes.Source: §8 · p. 9
  • The Cultural Atlas captures each country's dominant cultural narrative, leaving variation within countries, regions, and social classes unrepresented.The authors call for resources that capture these diverse perspectives.Source: §8 · p. 9
  • Evaluations are conducted only in English; multilingual and linguistic-variation testing is beyond this study's scope.The authors encourage future work on multilingual reasoning performance and its correlation with cultural adaptability across languages.Source: §8 · p. 9
  • NORMAD-ETI treats dynamic, multifaceted culture as static variables, which may perpetuate stereotypes and miss changing cultural nuances.The authors envision allowing users to specify their own norms and values in future work.Source: §8 · p. 9

Ethics Statement

The ethics statement frames cultural adaptation as complex: it may affect trust and social risks, and should be guided by user preferences rather than imposed norms. The appendices describe prompts for generating, filtering, eliciting judgments, and validating norm abstractions.

Ethics Statement

  • Highly adaptable systems risk mimicry perceived as manipulative and may amplify polarization; the authors say adaptation should follow user preferences, not impose norms.They present NORMAD as a measurement framework and NORMAD-ETI as a proxy, not as a prescription of cultural standards.Source: Ethics Statement · p. 10

A.1 Situation Generation Prompts

  • The prompts elicit stories from cultural backgrounds and ask for socially acceptable actions, including scenarios where a person follows or disregards a Rule-of-Thumb.The stories are to omit nationality and explicit cultural context.Source: §A.1 · p. 14
  • The situation-generation prompts define a Rule-of-Thumb as a culturally appropriate action that may be inappropriate elsewhere, and a Value as its generalized form.They also specify culture-agnostic rule wording and avoid prompts that explicitly state cultural context or explain a faux pas.Source: §A.1 · p. 14

A.2 Filtration prompt

  • A filtration example checks whether a story about accepting a drink offer is relevant to the stated Rule-of-Thumb.The example asks whether the action fits the rule about declining twice before accepting on the third offer.Source: §A.2 · p. 15

A.3 Responses extraction from LLMs

  • Figure 7 supplies the prompt used to elicit an answer or label from each model.The appendix identifies it as the judgment prompt for all models.Source: §A.3 · p. 15, Figure 7 · p. 15

A.4 Validation Prompts

  • GPT-4 validation prompts check whether a Rule-of-Thumb follows from a cultural background and whether a Value abstracts a Rule-of-Thumb.Source: Figure 8 · p. 15, Figure 9 · p. 15

A.5 Dataset Statistics

The appendix reports dataset-statistics and human-validation materials, but the supplied passages provide table scopes and procedures rather than table values. They also describe annotator recruitment and a column-mapping table.

A.5 Dataset Statistics

  • Table 2 presents dataset statistics across Inglehart-Welzel clusters and labels.Source: Table 2 · p. 16

B.1 Statistics

  • The study qualified 69 annotators from the USA, Mexico, and India and paid $0.3 per HIT.The passage says this yielded over $15 per hour and was commensurate with U.S. minimum-wage standards and much higher than Mexico or India.Source: §B.1 · p. 16

B.2 Mturk Annotator PPA Scores

  • Table 3 reports Fleiss fixed-marginal multirater κ and pairwise agreement scores for the MTurk human-validation study.Accuracy is calculated by majority voting against ground-truth labels.Source: Table 3 · p. 16

B.3 Human Verification Scores: Situation + COUNTRY + VALUE

  • For Situation + VALUE + COUNTRY, the study sampled 12 data points and recruited three annotators from each country.Accuracy uses majority voting; Fleiss κ and Krippendorff’s α are also calculated.Source: Table 4 · p. 16

B.4 Column Mapping from the cultural atlas

C F1-scores over NORMAD-ETI across all models

The supplied model-score passages report precision, recall, and F1 for two models across contextualizations, while the accompanying figures identify accuracy breakdowns by context, cultural zone, and social-acceptability category. Their captions specify comparisons but not outcomes.

C F1-scores over NORMAD-ETI across all models

D Model Accuracies

  • Accuracy figures compare contextualizations across Inglehart-Welzel bins, etiquette subaxes, and social-acceptability categories against an unconditioned baseline.The captions identify country, value or country+value, and rule-of-thumb contexts, with dashed lines denoting baseline performance.Source: Figure 10 · p. 19, Figure 11 · p. 20, Figure 12 · p. 21

D.1 Effect of RL alignment optimization on model performance

  • Figure 13 presents LLaMa-1 accuracies to show the effect of preference alignment across different contexts.Source: Figure 13 · p. 22

D.2 How well do models perform across IW bins?

  • Figure 14 presents model-wise accuracies across cultural zones and contexts.Source: Figure 14 · p. 22

D.3 Model Training paradigms

The appendix lists training paradigms for evaluated models, illustrates GPT-3.5 failure cases, and provides few-shot situation-generation examples and Amazon Mechanical Turk study materials.

D.3 Model Training paradigms

  • The evaluated GPT models use SFT plus PPO, with GPT-3.5-turbo and GPT-4 listed under that paradigm.Table 6 presents the training paradigms of models evaluated against NormAd.Source: §D.3 · p. 23, Table 6 · p. 23

D.4 Failure cases of GPT3.5 in NORMAD-ETI

  • GPT-3.5 struggles with nuanced gifting and eating cases, including when situations are contextualized with a rule-of-thumb.The table notes difficulty reasoning about numbers and actions in gifting; for eating, it reports difficulty with VALUE that is rectified by rule-of-thumb context.Source: Table 7 · p. 24

D.5 An example few-shot for situation generation

  • The few-shot examples pair cultural backgrounds and values with situations and explanations judging social acceptability.One example describes Atlantican respect for personal space and treats a host’s unsolicited hug as not justified; another applies Kalevonian shoe-removal etiquette to a home visit.Source: §D.5 · pp. 25–26
  • Figure 15 shows a few-shot prompt for generating social situations that adhere to the “yes” label.The examples also include a Kalevonian guest entering a friend’s home without removing shoes, judged unacceptable in that context.Source: §D.5 · p. 26, Figure 15 · p. 27

E Amazon Mechanical Turk Annotation Study

Loading 2404.12464v10…