PaperlayerView exact revision on arXiv

Source-linked AI summary

Large language models underestimate and partly misrepresent cultural variation in everyday norms

Kimmo Eriksson, Irina Vartanova, Pontus Strimling

arXiv:2609.30896v1cs.CY

TL;DR

Everyday norms vary across societies, but evidence about whether LLMs capture that variation across concrete behaviors is limited. Comparing model estimates with GSEN ratings, the paper finds that four LLMs substantially underestimate cultural differences and often misidentify their pattern. Their representation of these differences is uneven rather than reliable overall.

  • Problem

    Whether LLMs represent cultural variation across specific everyday behaviors and situations remains an open question requiring comparable society-level ratings.

    Source: Introduction · p. 2

  • Method

    The study compares LLM estimates with GSEN ratings of 150 everyday scenarios across 90 societies.

    Source: Introduction · p. 2

  • Results

    LLMs estimated between-society norm differences at less than half their measured size and often poorly identified which societies judged behaviors more or less acceptable.

    Source: Discussion · pp. 8–9

  • Takeaways & Limitations

    Cultural differences in everyday norms are only weakly and unevenly represented by the four frontier LLMs tested.

    Source: Discussion · p. 9

    Retrospective estimates of change compare different survey respondents and may also reflect changes in instruments and sample composition, not only changes in norms.

    Source: Supplementary Tables · p. 26

Abstract

from arXiv · show

A key aspect of culture is a society's norms about everyday behavior. How accurately do large language models (LLMs) represent cultural differences in such norms? To answer this question we used the recent Global Study of Everyday Norms (GSEN), which collected ratings of 150 scenarios in 90 societies, as the human benchmark. We prompted GPT-5 to estimate each society's average rating for every scenario, and later repeated the benchmark in three other LLMs: GPT-5.4, Claude Opus 4.6, and Gemini 3.1 Pro. Compared to GSEN estimates, all four LLMs misrepresented cultural variation in two ways. First, they greatly underestimated its magnitude, estimating differences between societies to be, on average, less than half their measured size. Second, for many scenarios the LLMs poorly identified the pattern of variation, that is, which societies judged the behavior less acceptable and which societies judged it more acceptable. The pattern of variation was identified better for scenarios that elicit concerns about vulgarity, especially scenarios involving kissing and flirting. We also found that norms in more developed societies tended to be estimated somewhat more accurately, and that prompting in local survey languages rather than English produced only a modest improvement in accuracy. Local-language prompting also reduced, but did not remove, the underestimation of between-society differences. Cultural differences in everyday norms are only weakly and unevenly represented by LLMs.

Significance Statement

Across four frontier LLMs, cultural differences in everyday norms were represented unreliably: their estimated magnitude was less than half the measured size, and their pattern was often poorly identified.

Significance Statement

  • Four frontier LLMs underestimated cultural differences and often failed to identify which societies judged behaviors more or less acceptable.The study compared model estimates with ratings of 150 everyday scenarios in 90 societies.Source: Significance Statement · p. 1

Introduction

The introduction asks whether LLMs can represent variation in everyday norms across societies, using a common set of concrete scenarios and society-level ratings. It motivates tests of variation across behaviors and societies, and of local-language prompting.

Introduction

  • Earlier LLM research examined broad cultural values and contested moral issues, while everyday norms concern specific behaviors in particular situations.The cited examples include Tao et al. (2024) on cultural values and Ramezani and Xu (2023) on moral issues.Source: Introduction · p. 2
  • GSEN provides ratings of 150 scenarios across 90 societies, enabling comparisons of society-level acceptance across a common set of everyday situations.More than 25,000 participants rated scenarios formed by combining 15 behaviors with 10 situations.Source: Introduction · p. 2
  • The study separates accuracy in estimating the magnitude of cultural variation from accuracy in identifying its pattern across societies.The two dimensions can diverge: a model may capture one while misestimating the other.Source: Introduction · p. 2
  • The authors also examine whether accuracy varies by behavior or society and whether local-language prompts improve estimates over English prompts.The study begins with an exploratory GPT-5 benchmark and later replicates findings in GPT-5.4, Claude Opus 4.6, and Gemini 3.1 Pro.Source: Introduction · pp. 2–3

Results

Across four LLMs, estimated between-society differences were less than half their measured size, and cultural patterns were weakly identified for many scenarios. Accuracy varied by behavior and society, while alternative prompts produced only limited or inconsistent improvements.

LLMs make societies look too similar

LLMs identify the pattern of cultural differences only weakly

  • Mean cross-cultural correlations ranged from .23 to .33, and 49% to 66% of scenarios fell below r = .30, depending on the LLM.Correcting for survey sampling noise and repeated-answer reliability raised the correlation range to .26 to .40.Source: LLMs identify the pattern of cultural… · p. 4

LLMs represent cultural differences better for some behaviors than for others

  • Accuracy was relatively high for kissing and flirting but low for ordinary behaviors, and the positive link with vulgarity held across all three replication LLMs.Vulgarity predicted accuracy more strongly than inconsiderateness or lacking sense, and the association persisted in robustness checks.Source: LLMs represent cultural differences… · pp. 4–6

LLM estimates were somewhat more accurate for more developed societies

  • With society-level rating offsets removed, error fell with development in all four LLMs, but adjustment for profile atypicality and rating dispersion erased most or all of the gradient.The averaged error–HDI correlation was r = -.72; the four LLM-specific correlations ranged from r = -.52 to -.77.Source: LLM estimates were somewhat more… · p. 6, Figure 3 · p. 6

How do alternative prompts affect the findings?

  • Local-language prompts modestly improved accuracy, yet all four LLMs still underestimated between-society variation; added variation matched surveyed differences in only two LLMs.Across 16 societies and 30 scenarios, within-society error fell by 0.02 to 0.11 scale points, mean cross-cultural correlations rose by .04 to .12, and 22% to 69% of the compression shortfall was removed.Source: How do alternative prompts affect the… · p. 7
  • Compression remained below one and the vulgarity gradient positive across all 12 model-by-prompt combinations, while targeting students produced little improvement.Student-only and full-sample means correlated at r = .98; student-targeted estimates correlated with student means at r = .30 versus r = .28 for general prompts.Source: How do alternative prompts affect the… · p. 7
  • Without a named society, three LLMs produced an unconditioned profile closest to their United States profile, and the fourth placed it second closest.Distances ranged from 0.11 to 0.20 scale points; the resemblance was specific to each LLM’s own United States representation, not to surveyed United States norms or general typicality.Source: How do alternative prompts affect the… · pp. 7–8

Discussion

The four LLMs substantially underestimated between-society differences in everyday norms and poorly identified their pattern for many scenarios. Accuracy varied across behaviors and societies, while the benchmark and analyses leave important qualifications on interpretation.

  • LLMs estimated between-society differences at less than half their measured size and poorly identified which societies judged many behaviors more or less acceptable.The four tested models were therefore not reliable sources of information about how everyday norms differ between societies.Source: Discussion · pp. 8–9
  • Local-language prompting modestly improved accuracy and cultural differentiation, but substantial compression remained and the analyses could not distinguish limited knowledge from failed elicitation or calibration.The LLMs did not generally compress numerical differences within societies; their ratings across scenarios were more dispersed than survey ratings.Source: Discussion · p. 8
  • Pattern accuracy was considerably better for behaviors eliciting vulgarity concerns, but this interpretation is provisional because the analysis covered only 15 behaviors.The authors suggest such differences may be more explicit in text or linked to purity concerns.Source: Discussion · p. 8
  • Estimation errors were lower for more developed societies, but this association may reflect norm-profile properties rather than greater society-specific knowledge.When no society was named, estimates most closely resembled those produced when prompts named the United States.Source: Discussion · p. 8
  • The human benchmark used unweighted convenience-sample means rather than nationally representative estimates, although student-only and full-sample society means were almost identical.Explicitly asking about university students’ norms did not appreciably improve pattern accuracy, making population mismatch unlikely to explain the poor representation generally.Source: Discussion · pp. 8–9

Materials and Methods

The study benchmarks LLM estimates against GSEN society-level ratings across everyday scenarios, then assesses between-society magnitude, pattern, and within-society error.

Human benchmark

  • GSEN surveyed 25,422 participants in 90 societies and assessed 150 behavior-in-situation scenarios using six-point appropriateness ratings aggregated into society-level means.The grid contained 13,468 of 13,500 possible society-scenario cells; some kissing and flirting items and two other scenarios were not rated everywhere.Source: Human benchmark · p. 9

LLM estimation task

  • Four LLMs estimated each named society’s mean scenario rating on the survey scale, with five queries per society-scenario cell and valid numeric responses averaged.The models’ reasoning configurations differed, limiting comparisons of model capability; a prompt-format check restored a missing space for a subset of cells.Source: LLM estimation task · p. 9

Measures of accuracy

  • Compression measures the LLM-to-survey ratio of between-society SDs, cross-cultural accuracy correlates estimates with survey means across societies, and within-society MAE measures average absolute error across scenarios.Offset-adjusted MAE removes each society’s overall rating-level difference; split-half corrections account for sampling error in survey and LLM estimates.Source: Measures of accuracy · p. 10

Compression analysis

The analyses test whether LLMs compress between-society variation, whether pattern accuracy rises with vulgarity concerns, and whether accuracy varies with societal development and norm-profile properties.

Compression analysis

  • The preregistered compression test averages scenario-level LLM-to-survey SD ratios and supports underestimation when the entire 95% percentile bootstrap interval lies below one.Sensitivity analyses resampled behaviors and recalculated compression using a noise-corrected survey SD.Source: Compression analysis · p. 10

Vulgarity-gradient analysis

  • The vulgarity-gradient analysis tests whether vulgarity predicts cross-cultural accuracy after controlling for available between-society variation, with behavior-clustered and alternative-marker sensitivity analyses.The preregistered test uses a positive vulgarity coefficient with robust p < .05; additional checks resample behaviors and compare vulgarity with inconsiderate and lacks-sense responses.Source: Vulgarity-gradient analysis · pp. 10–11

Society-level analyses

  • Society-level analyses correlate 2021 HDI with offset-adjusted MAE and test whether societal atypicality and response dispersion account for the development-accuracy association.HDI was available for 85 societies; the simultaneous regression standardized the outcome and predictors and was fitted separately for each LLM.Source: Society-level analyses · p. 11

Alternative prompt conditions

The supplementary analyses vary prompt targets, languages, wording and scales, and describe exploratory analyses and confirmatory preregistration. They also document the GPT-5 collection correction and the confirmatory study's preregistration deviations.

Alternative prompt conditions

  • Prompt-condition analyses varied target population, society naming, wording, response scale and language, with some comparisons restricted to matched cells and repetitions.The university-student condition used GPT-5 alone; other conditions used all four LLMs, and the no-society condition covered all 150 scenarios.Source: Alternative prompt conditions · p. 11
  • Local-language comparisons covered 16 societies, while a crossed design prompted about Brazil, China, Germany and Turkey in each of their four languages.The analyses also compared LLM deviations from scenario means with corresponding survey deviations.Source: Alternative prompt conditions · pp. 11–12
  • The university-student condition compared student-targeted GPT-5 estimates with student survey means and general-prompt estimates with both student and full-sample means.The analysis used 54 societies meeting the prespecified response threshold and compared cross-cultural accuracy across the same cells.Source: Alternative prompt conditions · p. 12
  • The no-society analysis compared each LLM's unconditioned profile with its society-conditioned profiles and ranked profiles by distance from the mean.It also ranked societies by distance from the LLM's unconditioned profile.Source: Alternative prompt conditions · p. 12

Additional exploratory analysis

  • Supplementary exploratory analyses compared estimated norm changes across survey periods and measured how well models reproduced societies' scenario-level deviations.The deviation analyses examined regional recovery and associations with society characteristics, perceived tightness and the individualizing-binding axis.Source: Additional exploratory analysis · p. 12
  • Other exploratory analyses compared LLM estimates with individual respondents and reported within-society regional accuracy and agreement among confirmatory LLMs.The respondent comparison measured how often individual ratings differed from other respondents' mean more than LLM estimates differed from society means.Source: Additional exploratory analysis · p. 12

Preregistration, data, and code

  • The confirmatory preregistration tested whether vulgarity predicted cross-cultural accuracy and whether LLMs compressed between-society variation; both hypotheses were supported in all three replication models.Gemini returned reasoning tokens despite the preregistered no-reasoning setting, but neither confirmatory result depended on that deviation.Source: Preregistration, data, and code · pp. 12–13
  • The exploratory GPT-5 collection was corrected after confirmatory preregistration, while its hypotheses' decision rules applied only to the other three models.The GPT-5 preregistration specified the benchmark, temporal task and individual comparison but no hypotheses; the timing of one temporal-reporting choice differed between exploratory and confirmatory studies.Source: Preregistration, data, and code · p. 13

Ethical approval

The study reused previously published GSEN human-participant data and collected only AI-generated model outputs. The authors also describe their AI-assisted manuscript process, funding and competing-interest status.

Ethical approval

  • The study collected no new human-participant data; its new data were AI-generated outputs, and the original GSEN approvals are reported elsewhere.The authors state that no additional ethical approval was required.Source: Ethical approval · p. 13

Use of AI

  • AI tools assisted with code, tables, figures and manuscript drafting, but the authors reviewed and revised outputs and accept responsibility for the study.They state that no AI-generated text was incorporated without substantive human review.Source: Use of AI · p. 13

Funding

  • The Knut and Alice Wallenberg Foundation supported the research through grant 2022.0191.Source: Funding · p. 13

Competing interests

Supplementary Information

The Supplementary Information contains methods, Tables S1–S21 and Figures S1–S6 referenced by the main text.

  • The listed authors are Kimmo Eriksson, Irina Vartanova and Pontus Strimling, with affiliations at four Swedish institutions.The affiliations include the Institute for Futures Studies, Mälardalen University, Uppsala University and Linköping University.Source: Supplementary Information · p. 15
  • The supplement comprises supplementary methods, Tables S1–S21 and Figures S1–S6 referenced in the main text.Source: Supplementary Information · p. 15

Supplementary Methods

The supplementary methods detail the GSEN sample and scenario grid, model prompts and settings, response handling, data correction, sampling-noise adjustments and society-level measures.

Sample and grid

  • GSEN was administered across 125 sites, with local translations and a partly incomplete scenario-by-society grid.Kissing items were absent in Kuwait and Saudi Arabia, flirting items in Kuwait, and two items in Mongolia; regional analyses use four GSEN regions.Source: Sample and grid · p. 15

Elicitation prompt

  • A verbatim prompt asked models to estimate average society ratings for behavior-in-situation scenarios on the −2.5 to +2.5 scale.A second collection restored a missing space before the scenario wording for the temporal comparison and Table S13.Source: Elicitation prompt · pp. 15–16

Generation settings

  • Model endpoints used non-equivalent reasoning configurations, limiting comparisons of model capability.Gemini returned reasoning tokens on every call despite the minimum budget; GPT-5 and GPT-5.4 differed in reasoning-token use, while Claude's extended thinking was disabled by default.Source: Generation settings · p. 16

Valid responses

  • Ratings came from bare-number responses; GPT-5 refusals and empty or failed calls were handled separately from valid ratings.The 84 GPT-5 calls without content were not retried, a deviation affecting 0.10% of its calls; every affected cell retained four of five repetitions.Source: Valid responses · p. 16

Correction of the exploratory GPT-5 collection

  • A parsing fault left many GPT-5 ratings out of the stored column, so the exploratory collection was corrected by parsing saved raw replies.The correction did not change the confirmatory decision rules, which applied only to the other three LLMs.Source: Correction of the exploratory GPT-5… · pp. 16–17

Sampling noise and reliability

  • Survey-mean reliability was estimated from respondent halves, and its square root set the correlation ceiling for a true-means predictor.LLM reliability was similarly estimated from call halves; corrected correlations excluded scenarios without positive estimated LLM reliability, and noise-corrected compression used between-half covariance.Source: Sampling noise and reliability · p. 17

Deviation measures and society-level predictors

  • Deviation accuracy is the within-society correlation of LLM and survey deviations across scenarios; deviation recovery is the corresponding regression slope.Society-level measures include individualizing-binding indicators, vulgarity weight, perceived tightness and strictness; predictors were tested with OLS, using West as the regional reference.Source: Deviation measures and society-level… · pp. 17–18

Supplementary Tables

The supplementary tables test alternative computations and prompt conditions, and extend the analyses to language, unconditioned profiles, historical change, and society-level factors. They also document important limits on interpreting some comparisons.

Tables S7–S8 and S17

Tables S9–S10

  • Table S9 compares English and local-language prompts across 16 societies; the Algeria–Saudi Arabia estimated-difference slopes remain near zero in both languages.The reported slopes range from −.02 to .10 in English and −.19 to .06 in Arabic.Source: Supplementary Tables · pp. 21–22, Table S9 · p. 21

Tables S1–S2 and S12–S13

  • The supplementary analyses vary compression calculations, measurement-precision corrections, respondent weighting, sparse-cell exclusions, and prompt spacing.The repaired-spacing condition changes only the omitted space before the scenario; the other conditions inherit that prompt fault.Source: Supplementary Tables · pp. 18, 24, 25, Table S1 · p. 18, Table S2 · p. 18

Table S14

  • Across models, the no-society (unconditioned) profile is compared with the mean of each model’s profiles for 90 named societies, and the closest named-society profile is identified.For Claude Opus 4.6, Australia is closest and the United States ranks second at 0.17 points.Source: Supplementary Tables · p. 25, Table S14 · p. 25

Table S15

  • The two-survey change analysis estimates retrospective differences between waves, not change tracked within one population over time.Wave differences also include instrument and sample-composition changes, as well as possible changes in norms.Source: Table S15 · p. 25, Supplementary Tables · p. 26

Tables S16 and S18–S20

Supplementary Figures

The supplementary figures break down variation and accuracy across scenarios, behaviors, and societies. They also show how the society-level comparisons are defined and qualified.

Figure S1

  • Across the four LLMs, 83%–93% of scenarios have estimated between-society variation below the survey variation.The figure plots each scenario's ratio; a ratio below 1 means societies are estimated too close together.Source: Figure S1 · p. 30

Figure S2

  • For GPT-5, mean cross-cultural correlation is .27, compared with a mean within-society correlation of .86.The cross-cultural correlations are across societies for each scenario; within-society correlations are across scenarios.Source: Figure S2 · p. 30

Figure S3

  • Behavior-level cross-cultural accuracy is averaged across the ten situations per behavior, with behaviors ordered by their four-model mean.These descriptive behavior means are distinct from scenario-level inferential analyses that account for clustering within behaviors.Source: Figure S3 · p. 31

Figure S4

  • Figure S4 plots the exploratory GPT-5 vulgarity gradient after controlling both axes for survey between-society standard deviation.The fitted slope therefore represents the gradient with that standard deviation held constant.Source: Figure S4 · p. 32

Figure S5

  • Figure S5 relates each society's atypicality to its within-society error after removing mean signed differences from both measures.The plot covers the 85 societies with available HDI data and marks societies more than two residual standard deviations from the fitted line.Source: Figure S5 · p. 32

Figure S6

  • Figure S6 compares deviation accuracy with five society-level measures across 88–90 societies, using nominal 90% Fisher intervals.The intervals assume independent societies; shared language, geography, and history may induce dependence, and they are not simultaneous bounds across models.Source: Figure S6 · p. 33

SI References

The supplementary references include prior work on everyday norms and cross-cultural variation.

References

  • Eriksson et al. (2025) is cited for research on everyday norms becoming more permissive over time and varying across cultures.The reference is published in Communications Psychology.Source: SI References · p. 33
  • The listed works address everyday norms over time and cross-national cultural differences.These are the two entries supplied in this reference section.Source: SI References · p. 33
  • Gelfand et al. (2011) is cited for a 33-nation study of differences between tight and loose cultures.The reference is published in Science.Source: SI References · p. 33
Loading 2609.30896v1…