PaperlayerView exact revision on arXiv

Source-linked AI summary

ChronoCulture: Evaluating Cultural Awareness Across Time in Large Language Models

Rusali Saha, Nitya Khurana, Ruihong Huang, Kuan-Hao Huang

arXiv:2610.20472v1cs.CLcs.CY

TL;DR

Most cultural benchmarks treat culture as a static country or regional attribute, despite attitudes changing over time. ChronoCulture evaluates change with longitudinal survey pairs across 72 countries. Across six LLMs, models often predict attitudes at one time but do not reliably capture change, and specifying a year does not consistently improve alignment.

  • Problem

    Most cultural benchmarks treat culture as a fixed country or community attribute, leaving how LLMs represent evolving attitudes across time as an open question.

    Source: §1 · p. 1

  • Method

    ChronoCulture uses 10,360 temporal pairs from longitudinal surveys across 72 countries, comparing full response distributions for the same question and country across years.

    Source: §1 · pp. 1–2, §3.4 · p. 3

  • Results

    Across six LLMs, models often predict attitudes at one time but do not reliably capture change, and their predictions for different years tend to stay close together.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    A model matching one survey wave may still miss how that culture has changed, so evaluating cultural awareness requires comparison across time.

    Source: §5 · p. 8

    Coverage is limited to the countries, topics, and years in the WVS and ISSP, ending at 2022 and focusing on reported attitudes and values.

    Source: §6 · p. 9

Abstract

from arXiv · show

Cultural attitudes and social norms evolve over time, yet most cultural benchmarks for large language models (LLMs) treat culture as a static attribute of a country or region. We introduce ChronoCulture, a benchmark of 10,360 temporal pairs built from longitudinal survey data across 72 countries. Each pair contains the same survey question asked in the same country in an older and a newer year, with full population response distributions as ground truth. Across six LLMs, we find that models can often predict a country's attitudes at one point in time but do not reliably capture how those attitudes change. Their predictions for different years tend to stay close together, failing to capture much of the observed change. Specifying the intended year in the prompt also offers little overall improvement in alignment with survey responses. These findings suggest that models often present a largely static picture of cultural attitudes, highlighting the need to evaluate cultural awareness across time.

1 Introduction

Most cultural benchmarks assess a country or community at a single point in time, although attitudes can change. ChronoCulture evaluates that change with longitudinal survey pairs, and the authors find that LLMs often fail to capture it reliably.

1 Introduction

  • Existing cultural benchmarks largely evaluate culture as a fixed country or community attribute, against a single cultural snapshot.Cultural attitudes and social norms may evolve, and the extent of change can vary across societies.Source: §1 · p. 1
  • A single-time evaluation provides only a partial view; a fuller evaluation compares predictions for the same question and country across periods.The motivating Canada example shows survey attitudes toward abortion shifting between 1999 and 2022 while model predictions remain similar.Source: §1 · p. 1, Figure 1 · p. 1
  • Across six LLMs, models often predict attitudes at one point in time but do not reliably capture how they change.Predictions for different years tend to stay close together despite observed change, and naming the intended year does not consistently improve alignment.Source: §1 · p. 2
  • ChronoCulture comprises 10,360 temporal pairs across 72 countries, using full population response distributions rather than single answers.Each pair compares the same survey question in the same country across an older and a newer year.Source: §1 · pp. 1–2
  • The evaluation framework tests whether LLMs predict both periods, capture cultural change and stability, and benefit from specifying the intended year.The authors also release code to rebuild the benchmark and run evaluations.Source: §1 · p. 2

2 Related Works

Prior cultural benchmarks and survey-based opinion evaluations generally assess cultural knowledge or alignment across groups and waves. ChronoCulture instead tests how the same population’s attitudes change over time, using question-level response distributions.

Socio-Cultural Knowledge Evaluation for Large Language Models

  • Most existing cultural benchmarks treat culture as a static attribute of a country, language, or region, unlike ChronoCulture’s focus on change over time.The surveyed benchmarks cover everyday knowledge, cultural values and norms, and multimodal cultural knowledge.Source: §2 · p. 2

Temporal-Aware Question Answering

  • Temporal question-answering benchmarks primarily address changes in factual knowledge, while related vision-language work examines cultural content from historical periods.ChronoCulture instead evaluates changes in population attitudes using longitudinal survey responses.Source: §2 · p. 2

Survey-based Opinion and Value Alignment

  • Survey-based opinion benchmarks compare model outputs with human opinions across groups and countries, while Liu et al. (2026) examine alignment across WVS waves.ChronoCulture asks whether models capture how the same population’s attitudes change over time.Source: §2 · p. 2

Concurrent Work

  • Daryani et al. (2026) track countries’ overall cultural positions, whereas ChronoCulture compares full response distributions for individual questions within each country.The question-level comparison can identify changes that may be hidden in an aggregate measure.Source: §2 · p. 2

3 Benchmark Creation

ChronoCulture evaluates cultural attitudes over time by pairing the same survey question within a country across years and using full response distributions. The benchmark draws on longitudinal surveys and evaluates both the size of observed shifts and whether models capture them.

3 Benchmark Creation

  • ChronoCulture uses full response distributions rather than single answers, preserving within-population diversity while comparing the same question and country across time.The benchmark is designed to examine both attitudes at different periods and how they change.Source: §3 · pp. 2–3

3.1 Data Sources

  • WVS Waves 3–7 (1995–2022) and ISSP data spanning 1985–2022 provide individual responses with country and survey year.Both are international survey programs collecting longitudinal data on social attitudes and values.Source: §3.1 · p. 3

3.2 Variable Selection

  • 408 longitudinal variables across 72 countries meet criteria including repeated waves, consistent ordered scales, and at least 500 respondents per country and wave.The selection excludes demographic metadata, behavioral frequency items, and country-specific questions that are not comparable across nations.Source: §3.2 · p. 3

3.3 Gold Response Distribution

  • Survey-provided weights adjust response proportions for demographic representation, producing each year’s gold population response distribution.Invalid responses such as “don’t know” are removed before calculating the proportions.Source: §3.3 · p. 3

3.4 Temporal Pairs

  • 10,360 temporal pairs across 72 countries yield 20,720 dated prompts, pairing the oldest available year with the most recent ISSP year or WVS Wave 7.Each pair uses the same question and country at an older and a newer time point.Source: §3.4 · p. 3, Figure 3 · p. 3

3.5 Categorization of Temporal Pairs

  • Normalized Wasserstein distance between the two gold distributions accounts for response-option order; its values group pairs into stable, moderate, strong, and very strong shifts.The distance is normalized to [0, 1], with larger values indicating greater changes in cultural attitudes.Source: §3.5 · p. 4, Table 1 · p. 4

3.6 Why Distribution Evaluation?

  • 68.2% of pairs with shifted distributions keep the same mode, while 8.5% of stable pairs change mode, so modal-answer scoring can misrepresent change.A mode can miss a substantial distribution shift or suggest change when the distribution changed little.Source: §3.6 · p. 4, Figure 4 · p. 4

3.7 Prompt Construction and Prediction

  • For each temporal pair, models receive separate older- and newer-year prompts with the country, year, question, and response options, and predict percentages for each option.Predictions are compared with the gold distribution using normalized Wasserstein distance.Source: §3.7 · pp. 4–5

4 Experiments and Results

Across six models, the experiments find that predictions often fail to track cultural change, and naming a year offers little overall alignment benefit. Accuracy and temporal performance also vary with the size and topic of shifts and across regions.

4 Experiments and Results

  • Six LLMs are evaluated with the same prompts and model-specific knowledge filters, against the goal of changing predictions when attitudes shift and preserving similarity when they remain stable.Source: §4 · p. 5

4.1 Cultural Knowledge Filter

  • The evaluation retains, separately for each model, the 75% of question–country combinations with the smallest undated-prediction distance to any survey year, about 7,700 temporal pairs per model.This filter is intended to focus temporal evaluation on questions where the model shows some baseline knowledge.Source: §4.1 · p. 5

4.2 RF1: Predicting Both Periods Accurately Becomes Harder as Cultural Change Increases

  • As shift increases, both-period accuracy falls while one-period-only accuracy exceeds 50% for strong and very strong shifts; in 80–87% of shifted pairs, both predictions align with the same gold year, versus 9–15% aligning separately.Reversed pairs are rare.Source: §4.2 · pp. 5–6, Figure 5 · p. 6, Figure 6 · p. 6

4.3 RF2: Models Follow the Direction of Larger Cultural Shifts but Underestimate Their Magnitude

  • As human shifts grow, all six models capture a smaller fraction of change (median r < 1) even as directional alignment improves; identical outputs persist, and stable pairs show predicted change in 29.9–35.5% of cases.The magnitude ratio decreases with larger shifts, while directional alignment improves.Source: §4.3 · pp. 6–7, Figure 7 · p. 7

4.4 RF3: Naming a Year Does Not Consistently Improve Cultural Alignment

  • Across models, specifying a year helps 20–44% of items but hurts a similar share, leaving mean benefit near zero or slightly negative; older-year prompts help more under very strong shifts, while newer-year prompts offer little benefit.Without a year cue, the newest year is the closest match in only 46–48% of cases.Source: §4.4 · pp. 7–8, Figure 8 · p. 7

4.5 Findings by Topic and Region

  • Under larger shifts, predictions are less accurate for political and economic topics than for family and personal life, and both-period accuracy is higher in Western countries than in several other regions.For democracy, accuracy drops from 71% on stable pairs to 17% for larger shifts; for strong and very strong WVS shifts, accuracy is 35% in Western countries versus 16% on average in MENA and South and Southeast Asia.Source: §4.5 · p. 8

5 Conclusion

Across six models, temporal cultural awareness remains limited: models often capture attitudes at one point but portray changing cultures as largely static. They capture the direction of strong shifts better than their magnitude, with the gap widest for political and economic topics and many non-Western populations.

5 Conclusion

  • Across topics and regions, predictions for different years tend to stay close together, and models capture the direction of strong attitude shifts better than their magnitude.The gap is widest for political and economic topics and many non-Western populations.Source: §5 · p. 8
  • Naming a year in the prompt does not reliably improve alignment, so matching one survey wave can miss how that culture has changed.This matters as LLMs are increasingly used to simulate survey responses.Source: §5 · p. 8
  • Future work could explore temporal-source training, predicting changes between response distributions, and expanding coverage where temporal alignment is weaker.These directions could help models better represent cultures as they evolve.Source: §5 · p. 8

6 Limitations

ChronoCulture is limited to longitudinal survey coverage and ordered, consistent response items, and represents populations with country-level distributions. Its construction uses weighted response proportions, a normalized Wasserstein distance, and prompts targeting specific years.

6 Limitations

  • ChronoCulture covers only countries, topics, and years in WVS and ISSP; it ends in 2022 and measures reported attitudes and values, not customs or material culture.Wave 8 of WVS had not been released at the time of writing, and the benchmark requires consistent, ordered response options across survey years.Source: §6 · p. 9
  • Country-level response distributions preserve opinion diversity but can hide differences among demographic, regional, or cultural groups within countries.The authors suggest newer survey waves and finer-grained population data as possible extensions.Source: §6 · p. 9

A.1 Data Sources and Topic Coverage

  • WVS waves 3–7 cover 1995–2022, while ISSP repeated-topic modules cover 1985–2022, providing repeated survey questions for temporal comparisons.The WVS waves were selected because they provide repeated items across many countries.Source: §A.1 · p. 12

A.2 Computing Gold Response Distributions

  • Gold response distributions are computed by removing invalid answers and dividing each valid option’s total respondent weight by the total weight; the proportions sum to one.In the example, weights yield 50% Agree and 50% Disagree despite two respondents selecting Agree and one selecting Disagree.Source: §A.2 · pp. 12–13

A.3 Wasserstein Distance Metric

  • The normalized Wasserstein distance compares distributions over ordered response options and is scaled by the maximum possible distance on the question’s response scale.The scale normalization accommodates questions with different numbers of response options.Source: §A.3 · p. 13

A.4 Prompt Example

  • Temporal prompts are paired across years, with the newer-year version changing the target year and undated prompts omitting it; the same prompts were used across models.The example’s newer-year prompt targets 2020.Source: §A.4 · p. 13, Figure 9 · p. 13

A.5 Modal-Answer Evaluation

The modal-answer evaluation tests whether Llama-3.3-70B selects the most common response in both periods, but correct modal labels can conceal distributional error.

A.5 Modal-Answer Evaluation

  • The evaluation reuses temporal pairs, requires a 10-percentage-point gap between the two most common responses, and excludes moderate shifts.Pairs are retained only when the model answers at least one period correctly; stable pairs and strong or very strong pairs with changing modes are included.Source: §A.5 · p. 13, Figure 10 · p. 14
  • In 24.4% of shifted pairs, the model chose the correct mode while capturing less than half of the distributional shift.The authors therefore treat modal-answer accuracy as a supplementary diagnostic rather than a primary measure.Source: §A.5 · p. 14
  • 86.3% of stable pairs were correct in both periods, versus 11.9% of strong-shift and 14.8% of very-strong-shift pairs.The evaluation retains 1,351 pairs: 1,129 stable, 151 strong, and 71 very strong.Source: §A.5 · p. 13, Table 5 · p. 14

B Experimental Setup

The setup uses fixed inference settings and a cultural-knowledge filter that selects combinations by their undated prediction’s distance to available survey distributions. The filter retains roughly 75% per survey, with selection stratified by survey and year coverage.

B.1 Models and Inference Settings

  • Models used temperature 0, top-p 1.0, at most 120 output tokens, and one generation per prompt; two models used the OpenAI API and others ran locally with vLLM.Local inference used NVIDIA H100 NVL GPUs; model-specific hardware and software versions are specified in the passage.Source: §B.1 · p. 14
  • Models were instructed to return JSON response percentages, which were parsed, clipped at zero, and normalized to sum to one; invalid or nonpositive outputs were excluded.Repeated runs were nearly identical for Llama-3.3-70B, and findings held under alternative prompt instructions.Source: §B.1 · p. 14

B.2 Cultural Knowledge Filter: Scores and Retention

  • The knowledge filter measures each undated prediction’s distance to its closest distribution across all available survey years, not just the paired years.A smaller K means the prediction closely matches at least one observed survey distribution.Source: §B.2 · p. 14
  • Combinations are ranked within each survey and available-year-count group, retaining the 75% with smallest K; dated predictions are reserved for temporal evaluation.Stratification addresses the tendency for K to decrease as the number of available years increases; the figure’s pooled 75th percentile of 0.122 is reference only.Source: §B.2 · p. 14, Figure 11 · p. 15, Table 6 · p. 15
  • For Llama-3.3-70B, most K values fall below 0.12, but a long tail extends beyond 0.30 and no natural cutoff appears.Below 50% retention, the very-strong category falls below 100 pairs, too few for category-level comparison.Source: §B.2 · pp. 14–15, Table 7 · p. 15

C.1 Full Period-Accuracy Results

Across models, accurate predictions for both periods become less common as cultural shifts grow. Sensitivity checks and period-alignment results show that predictions often fail to distinguish the two periods.

C.2 Sensitivity to Accuracy Tolerance

  • For all six models, both-period accuracy falls from stable to very strong shifts at each of four error tolerances.The tested tolerances are τ = 0.05, 0.10, 0.15, and 0.20; Figure 12 displays prediction-to-gold distances and older-versus-newer errors for Llama-3.3-70B.Source: §C.2 · p. 15, Figure 12 · p. 16, Table 9 · p. 16

C.3 Temporal Alignment

  • Most predictions favor the same survey period, while correctly separated predictions remain a minority even as human shift increases.Reversed pairs are comparatively uncommon; Table 10 includes pairs with zero model shift.Source: §C.3 · p. 15, Table 10 · p. 17

C.1 Full Period-Accuracy Results

  • Both-period accuracy declines with human shift for every model, while newer-only accuracy exceeds older-only accuracy.Undated predictions are accurate for both periods in 64–88% of stable and 48–68% of moderate pairs, but only 16–25% of strong pairs and none of the very strong pairs.Source: §C.1 · p. 15, Table 8 · p. 16

C.4 Results by Topic and Region

Tracking cultural change is difficult across topics and regions, but performance varies across groups. Underestimation of change persists when identical model predictions are included.

C.4 Results by Topic and Region

  • Both-period accuracy is lower for strong+ than stable shifts in every model, topic, and region; naming a period changes accuracy little across groups.Mean human shift is broadly similar across WVS topics (0.15–0.17) and regions (0.154–0.188), and ranges from 0.12 to 0.17 across ISSP topics.Source: §C.4 · pp. 17–18
  • Under strong+ shifts, politics and economics are among the hardest topics, while family and personal-life topics are more accurate.WVS democracy and political engagement accuracy is 17% (range 11–23%); ISSP family accuracy is 40%. Models move in the right direction on selected political and economic topics for 45–52% of pairs, versus 94% for family and 82% for life priorities.Source: §C.4 · p. 17
  • Under WVS strong+ shifts, accuracy is 35% for Western countries versus 16% for both South and Southeast Asia and MENA; ISSP regional differences are smaller.The WVS ordering persists when each country is weighted equally (33% for Western countries versus 16–17% for South and Southeast Asia and MENA).Source: §C.4 · p. 18

D.2 Magnitude Ratios Including Identical Predictions

  • Including identical predictions, the median magnitude ratio is at most 0.46 in every human-shift category across all six models.For Llama-3.3-70B, the ratio including identical predictions remains close to 0.30 across categories.Source: §D.2 · p. 18

D.3 Stability Preservation and False Drift

Models usually preserve stable predictions on stable pairs, but false drift remains common. Supplemental analyses show survey-specific differences and illustrate limited or mismatched changes in individual examples.

D.3 Stability Preservation and False Drift

  • All six models preserve stability in 64.5–70.1% of stable pairs, while 29.9–35.5% exhibit false drift, usually small rather than large.Stability is defined as model shift dm ≤ 0.03; false drift is dm > 0.03 when the gold pair is stable.Source: §D.3 · pp. 18, 20, Table 13 · p. 20

D.4 Stability Preservation by Survey

  • False drift is more frequent in WVS than ISSP; in WVS, stability preservation only narrowly exceeds combined false drift, 50.6% to 49.4%.Large false drift is more than twice as frequent in WVS; pooled percentages are closer to ISSP results because ISSP contributes more stable pairs.Source: Table 14 · p. 20, §D.4 · p. 20

D.5 Human and Model Shift Across Temporal Pairs

  • Figure 15 plots model shift against human shift; points below the equal-magnitude diagonal indicate that the model underestimates change.The horizontal axis is human shift dh and the vertical axis is model shift dm; stable pairs are included for context.Source: §D.5 · p. 20, Figure 15 · p. 20

D.6 Illustrative Patterns of Predicted Change

  • Four examples show distinct patterns: matching the direction but understating change, predicting no change, reversing direction, and overstating change.Figure 16 illustrates these patterns for the United States, Australia, India, and South Korea, respectively.Source: §D.6 · p. 20, Figure 16 · p. 21

E.1 Default Period of Undated Predictions

  • For every model, the newest year is the most common default for undated predictions, but earlier years account for more than half of items.Source: Table 15 · p. 22

E.2 Equal-Choice Comparison

  • For Llama-3.3-70B, the newest year is selected only slightly more often than under equal choice.The baseline assigns each available survey year an equal chance for each item, then averages the matching percentages.Source: Table 16 · p. 22, §E.2 · p. 22

D.7 Examples of Limited Model Change

  • In four selected examples, gold distributions differ substantially while model predictions overlap or remain close; Morocco and Egypt have identical predictions across periods.The examples are shown as cumulative response distributions.Source: Figure 17 · p. 21, §D.7 · p. 22

E.3 Overall Effect of Naming a Year

  • Table 17 reports an overall comparison of the effect of naming a year.Source: §E.3 · p. 22

E.4 Effect of Naming a Year by Human Shift

Naming a year has shift-dependent effects: it is associated with small benefits for very strong shifts when the older year is specified, but not consistently when the newer year is specified.

E.4 Effect of Naming a Year by Human Shift

  • About 0.03 in W, naming the older year benefits very strong shifts across all six models, while naming the newer year has intervals including zero in five models and below zero for Qwen3.8-Flash.For stable and moderate shifts, intervals for naming either year lie below zero in all six models.Source: §E.4 · p. 22

F Robustness Analyses

Robustness checks varied shift thresholds, knowledge-filter retention, repeated runs, prompt instructions, and sampling assumptions; the main temporal patterns persisted across the tested settings.

F Robustness Analyses

  • Across models, naming a year has near-zero mean benefit overall; older-year prompts help after very strong shifts, while newer-year prompts help little.For Llama-3.3-70B, naming either year helps more often than it hurts only for very strong shifts.Source: Table 17 · p. 23, Table 18 · p. 23, Table 20 · p. 23

F.1 Robustness to Human-Shift Thresholds

  • Across every tested threshold and model, accuracy falls as human shift grows, separated pairs remain a minority, and most stable pairs keep steady predictions.The median magnitude ratio falls while cosine rises; threshold changes alter measure levels, and under the tighter set moderate-shift median ratios exceed 1 for four models.Source: §F.1 · pp. 23–24, Table 22 · p. 25, Table 23 · p. 25

F.2 Robustness to Knowledge-Filter Retention

  • At 50%, 75%, and 100% retention, the main trends persist, including near-zero or slightly negative benefit from specifying a year.Undated predictions match the newest year in 46–47% of cases, and 63.7–66.9% of stable pairs keep steady predictions.Source: §F.2 · p. 24, Table 26 · p. 28, Table 27 · p. 28

F.3 Robustness to Repeated Runs and Prompt Instructions

  • Five repeated runs reproduce predictions in 90.5% of comparisons, and alternative survey-framing instructions change individual predictions without displacing the main temporal patterns.Mean distances from the original instruction are 0.069 and 0.060; under all three instructions, accuracy declines with human shift and magnitude ratios remain below 1.Source: §F.3 · p. 24

F.4 Survey Sampling Noise

  • Under a 500-respondent sampling simulation, the median distance is 0.016, and all strong and very strong shifts exceed their pair-specific 95th-percentile noise estimates.The stable threshold of 0.03 is within differences sampling alone can produce; the estimates are conservative because most surveys include more than 500 respondents.Source: §F.4 · p. 24

Related papers

Loading 2610.20472v1…