PaperlayerView exact revision on arXiv

Source-linked AI summary

Cultural Divergence Preservation: Diagnosing Flattening and Caricature in LLM-Simulated Survey Populations

Yeeun Chae, Yewon Choi, Seunghyun Lee, IL Im

arXiv:2609.29928v1cs.CLcs.AI

TL;DR

Existing survey-fidelity metrics do not directly measure whether differences across countries are preserved. The paper introduces CDP, a divergence measure calibrated to human data, and finds that it tracks controlled flattening and exaggeration while conventional fidelity rankings can favor the most flattened prompting method.

  • Problem

    Distributional fidelity metrics do not explicitly evaluate whether cross-country variation is preserved, so flattening or exaggeration may coexist with similar distribution-matching scores.

    Source: §1 · p. 1

  • Method

    CDP compares synthetic cross-country response divergence with a human reference, using a one-time domain calibration to identify flattened or amplified divergence.

    Source: §1 · p. 2, §3.1 · p. 2

  • Results

    Across real generations, DeepP-I is the most flattened in all eight model–domain blocks (CDP: 14–59%), while Cultural prompting amplifies divergence.

    Source: §4.3 · p. 4, §5 · p. 4

  • Takeaways & Limitations

    CDP complements rather than replaces fidelity metrics by directly diagnosing attenuation or amplification of cross-country divergence.

    Source: §5 · p. 4

    CDP measures only the magnitude of country-associated divergence, not whether deviations match human directions or country ordering, or country-specific fidelity.

    Source: Limitations · p. 5

Abstract

from arXiv · show

Large language models (LLMs) are increasingly used as synthetic survey respondents to estimate population response distributions. In cross-cultural survey simulation, evaluations should assess not only distributional fidelity within countries but also whether differences across countries are preserved. However, existing distance-based metrics such as Jensen--Shannon divergence (JSD) do not directly capture such cross-country differences. To address this limitation, we introduce Cultural Divergence Preservation (CDP), a reference-light diagnostic based on a one-time human calibration. CDP identifies reduced cross-country divergence as cultural flattening and increased divergence as cultural caricature. To evaluate CDP, we conduct experiments across four LLM backbones, three persona-based prompting methods, and two survey domains, the World Values Survey (WVS) and the Big Five Personality Test. The results reveal a systematic discrepancy between conventional fidelity metrics and CDP. Controlled experiments show that CDP changes monotonically as cross-country divergence is attenuated or amplified, while the corresponding changes in JSD remain relatively small. In our audit of real LLM generations, DeepPersona-Inspired prompting is frequently favored by conventional fidelity metrics but exhibits the strongest flattening in every model--domain block. CDP thus complements fidelity metrics by directly quantifying the attenuation or amplification of cross-country divergence.

1 Introduction

Synthetic survey evaluations need to assess both country-level distributional fidelity and whether cross-country differences are preserved. The paper introduces CDP to diagnose flattening or exaggeration that conventional distribution-matching metrics may not distinguish.

1 Introduction

  • Conventional distributional metrics measure overall error but do not explicitly assess whether cross-country variation is preserved.Synthetic populations can suppress or exaggerate cross-country differences while receiving similar distribution-matching scores.Source: §1 · p. 1
  • Cultural Divergence Preservation (CDP) quantifies cross-country response divergence relative to a human reference and distinguishes flattened, preserved, or exaggerated divergence.A one-time human calibration for each survey domain can be reused across models and prompting methods.Source: §1 · p. 2
  • The evaluation combines controlled attenuation and amplification experiments with an audit of four LLMs, three prompting methods, and two survey domains.The domains are the World Values Survey and Big Five Personality Test.Source: §1 · p. 2
  • DeepP-I, frequently favored by conventional fidelity metrics, exhibits the strongest flattening in all eight model–domain blocks.The result shows that country-level fidelity rankings do not separately characterize divergence preservation.Source: §1 · p. 2

2 Related Work

Prior synthetic-survey evaluations emphasize within-population fidelity, while other work examines subgroup diversity, cultural profiles, or country rankings. The paper positions CDP as a summary of divergence among full country-level distributions relative to a reusable human baseline.

2 Related Work

  • Prior synthetic-survey evaluations compare model-generated and matched human response distributions within populations using reference-based fidelity metrics.Examples include Santurkar et al. (2023), Durmus et al. (2023), AlKhamissi et al. (2024), and Ma et al. (2025).Source: §2 · p. 2
  • Research documents both subgroup-diversity flattening and stereotypical caricature, while persona conditioning does not consistently eliminate either distortion.Detecting these distortions generally requires condition-matched human data; approaches that reduce that dependence assess other fidelity dimensions.Source: §2 · p. 2
  • Cultural profiles, country rankings, and value-signal measures capture cultural variation but do not summarize divergence among full country-level distributions against a reusable human baseline.CDP complements these approaches by calibrating aggregate divergence from the cross-country centroid against the corresponding human level.Source: §2 · p. 2

3 Cultural Divergence Preservation

The paper measures cross-country response divergence using country distributions and their equal-weight item centroids, then expresses synthetic divergence relative to a human reference. This separates divergence preservation from country-wise distributional fidelity.

3.1 Definition

  • For each survey item, the method computes the equal-weight centroid of normalized country response histograms.The histograms are vectors of response-option proportions in the data being evaluated; equal country weights make the centroid reflect cross-country structure rather than national sample sizes.Source: §3.1 · p. 2
  • Jensen–Shannon divergence is bounded between 0 and 1 with base-2 logarithms, equals zero for identical distributions, and increases with dissimilarity.The paper uses this divergence function to compare country distributions with their centroid.Source: §3.1 · p. 2
  • Cultural Divergence (CD) is the mean Jensen–Shannon divergence between each country’s response distribution and the cross-country centroid.CD is zero when all countries share the same distribution and increases as their distributions become more divergent.Source: §3.1 · p. 2

3.2 Human Reference

  • CDP expresses synthetic CD as a percentage of domain-specific human CD: below 100% indicates flattening, while above 100% indicates caricature.The human CD is estimated once per domain and reused across synthetic conditions, but must be recomputed if the country set, item set, or response scale changes.Source: §3.2 · p. 3
  • Average country-wise fidelity does not determine whether cross-country divergence is attenuated or amplified, because fidelity JSD and CD compare different distribution pairs.Fidelity JSD compares each synthetic country distribution with its human counterpart; CD compares each country distribution with the cross-country centroid.Source: §3.2 · p. 3

4 Experiments and Results

The experiments test CDP with controlled changes to cross-country divergence and an audit of persona-based survey simulations. The audit finds that fidelity rankings and divergence preservation can disagree, while CDP responds monotonically to controlled flattening and exaggeration.

4.1 Experimental Setup

  • The audit covers WVS and Big Five surveys, three persona-construction methods, and four open-weight models.The WVS uses six items across six countries; Big Five uses the 50-item inventory across three countries, with 300 simulated respondents per country, method, and model.Source: §4.1 · p. 3

4.2 Controlled Flattening and Exaggeration

  • The controlled transformation scales each country–item deviation from the centroid: γ below 1 moves distributions inward, while γ above 1 amplifies deviations.For γ above 1, negative bins are clipped and histograms renormalized; for γ ≤ 1, item-level centroids are preserved.Source: §4.2 · pp. 3–4
  • Under controlled perturbations, CDP decreases monotonically with flattening and reaches 231–254% at γ = 1.5, while induced JSD costs remain small relative to real-LLM medians.At 20% centroid mixing, WVS retains 62% of human divergence at a 0.5% JSD cost; complete collapse costs 9.3% and 2.3% of the LLM median for WVS and Big Five, respectively.Source: §4.2 · p. 4

4.3 Country-Level Fidelity and Divergence Preservation

  • DeepP-I has the lowest raw WD in seven of eight blocks and the lowest nWD and JSD in six, yet shows the strongest flattening in every block (CDP: 14–59%).Cultural is over-divergent (184–501%), while PHub-I is intermediate (44–99%).Source: §4.3 · p. 4
  • The DeepP-I human–synthetic CD gap excludes zero in seven of eight model–domain blocks, while all eight gaps remain positive under leave-one-country-out analysis.Llama-2-13B on Big Five is the sole bootstrap exception.Source: §4.3 · p. 4

5 Conclusion

The conclusion distinguishes country-level distributional fidelity from preservation of cross-country divergence. Controlled perturbations show CDP tracks flattening and exaggeration even when induced JSD is small, while real generations reveal divergent rankings between fidelity metrics and CDP.

5 Conclusion

  • Controlled perturbations show CDP tracks flattening and exaggeration even when induced JSD is small relative to observed LLM error.Figure 2 compares CDP with unmodified human divergence and induced JSD cost with the same-domain median across real LLM conditions.Source: §5 · p. 4, Figure 2 · p. 4
  • DeepP-I is frequently favored by fidelity metrics but is consistently the most flattened, while Cultural prompting amplifies divergence.CDP complements rather than replaces existing metrics by diagnosing both distortions.Source: §5 · p. 4, Table 1 · p. 4

Limitations

CDP measures the magnitude of country-associated divergence, not whether country-specific patterns match human counterparts. Its empirical scope and reference requirements also constrain interpretation and reuse.

Limitations

  • CDP does not assess country-specific fidelity, recover human country ordering, or detect inversions in country patterns; it should not be used as a general quality score or downstream-fidelity predictor.It requires a domain-level human reference, which must be recomputed when the country set, items, or response scales change; observed country differences may also reflect survey year, translation, response styles, or sample composition.Source: Limitations · p. 5
  • The empirical scope is four open-weight models, two survey domains, and English-language prompts; Big Five includes only three countries, and generation-seed sensitivity uses three seeds per condition.The Llama-2-13B Big Five bootstrap gap includes zero, so statistical separation is supported in seven rather than all eight blocks.Source: Limitations · p. 5
  • The primary WVS analysis is unweighted, and broader validation should examine weighting choices explicitly.Applying within-country W_WEIGHT changes aggregate human CD by −0.80% and changes neither model-by-metric winners nor CDP interpretation classes; future work should extend coverage and add direction-sensitive and item-dependence diagnostics.Source: Limitations · p. 5

B WVS and Big Five Dataset Details

The dataset details identify World Values Survey items and Big Five personality items rated on a five-point scale.

B.1 Survey Items

  • Table 2 presents the World Values Survey items.Source: Table 2 · p. 9
  • Table 3 presents Big Five personality items rated on a five-point scale, from 1 (Disagree strongly) to 5 (Agree strongly).Source: Table 3 · p. 9

B.2 Human Sample Sizes

Table 4 reports country-level human sample sizes for the WVS and Big Five.

B.2 Human Sample Sizes

  • Table 4 reports human sample sizes by country for the WVS and Big Five.Source: Table 4 · p. 9

C Controlled Flattening and Exaggeration Details

Controlled manipulations show that cross-country divergence changes monotonically as it is attenuated or amplified, while the associated JSD costs remain small relative to real-condition medians.

C Controlled Flattening and Exaggeration Details

  • CD falls monotonically as country histograms move toward the equal-weight centroid, and complete convergence costs 9.33% of the WVS and 2.28% of the Big Five real-condition JSD medians.At λ = 1, every country distribution equals the centroid, making mean JSD to the original human distributions exactly the human CD at λ = 0.Source: C Controlled Flattening and… · p. 10
  • CDP rises monotonically with amplification, reaching 231–254% at γ = 1.5 with a JSD cost of 0.6–3.9% of the same-domain real-condition median.For γ ≤ 1, the manipulation is the mixing grid; stronger amplification is not evaluated because boundary clipping becomes non-negligible for some WVS histograms.Source: C Controlled Flattening and… · p. 10
  • Before boundary clipping, JSD to the original distribution grows approximately with (γ − 1)^2, so moderate amplification can produce a large relative CDP change at small JSD cost.Source: C Controlled Flattening and… · p. 10

D Country-Level Cultural Divergence

Country-level results broadly support the aggregate pattern of caricature and flattening, while showing that it does not hold in every individual cell.

D Country-Level Cultural Divergence

E.1 Respondent-Level Bootstrap and Country Composition

The appendix tests whether the reported divergence patterns withstand respondent resampling, country omissions, generation seeds, and WVS sampling weights. DeepP-I flattening and Cultural amplification persist across the reported robustness checks, while weighting leaves interpretation classes unchanged.

E.1 Respondent-Level Bootstrap and Country Composition

  • DeepP-I’s human–synthetic gap excludes zero in seven of eight blocks, and remains positive under every leave-one-country-out check; Cultural’s gap remains negative under every matched omission.The Big Five leave-one-country-out results involve only two countries and are described as a weak sensitivity check, not an uncertainty interval; BCa intervals are degenerate on Big Five and are treated as corroborating evidence only on WVS.Source: §E.1 · pp. 10–12, Table 8 · p. 13

E.2 Country-Level Discordance and Threshold Sensitivity

  • At the descriptive cutoff t = 0.5, 37 of 108 country conditions meet both low-JSD and low-CDPc criteria, and all use DeepP-I or PHub-I.Across cutoffs t = 0.4, 0.5, and 0.6, the counts are 31, 37, and 42 without changing the represented methods; conditions overlap and are not independent replications.Source: §E.2 · p. 12, Table 9 · p. 13

E.3 Generation-Seed Sensitivity

  • Across three generation seeds, DeepP-I remains under-divergent in all eight blocks and Cultural remains over-divergent in all eight.DeepP-I has the lowest JSD in the same six of eight blocks; only three of 48 WD and nWD winner comparisons change, and the sole direction-class change is near the human reference.Source: §E.3 · p. 12, Table 11 · p. 14

E.4 WVS Sampling-Weight Sensitivity

  • WVS weighting changes none of the 12 block-level CDP interpretation classes or 72 country-cell classes, and changes none of the 12 model-by-metric winners.The largest absolute metric change is 4.20%; aggregate human CD changes from 0.021905 to 0.021730 (−0.80%).Source: §E.4 · p. 12

Related papers

Loading 2609.29928v1…