The Limits of Simulated Societies: How Post-Training and Survey Fine-Tuning Erase Cross-Cultural Variance

• Prior work reports that demographic personas can flatten within-group diversity, but whether existing training paradigms mitigate or amplify this flattening remains unresolved.Interview-grounded agents can match participants’ two-week test–retest consistency 85% of the way, but such data is prohibitively expensive for most global survey datasets.Source: §1 · p. 2
• The authors evaluate WVS personas across 12 countries using eleven zero-shot models and five WVS-fine-tuned variants, measuring point quality and dispersion retention.Their four-metric framework includes accuracy, MAE, Wasserstein-1 distance, and deviation ratio; they term alignment-induced output-variance compression consensus collapse.Source: §1 · p. 2
• 57.9% accuracy is achieved by the most accurate WVS-fine-tuned model, Tulu 3 70B-DPO, which retains only half the human response spread (σr = 0.50).Source: §1 · p. 2
• Nigeria retains 11% of human spread (σr = 0.11), versus 70–87% for WEIRD countries, and GRPO does not restore spread under either tested reward design.Prior mixing raises σr from 0.51 to 0.62 and reduces W1 from 0.54 to 0.51 on a held-out split, but the underlying collapse resists simple correction.Source: §1 · p. 2
• Sampling the two DPO models up to T = 1.0 leaves W1 unchanged, so broader sampling does not recover the lost spread in those tests.Source: §1 · p. 2

• Prior work uses LLMs as computational stand-ins for social agents and formalizes subpopulation representative modeling for generalizing from empirical data via in-context learning.Other work reports an AUC of 0.86 for retrodicting individual opinions in unsurveyed years, with weaker performance on entirely unasked questions (Kim and Lee, 2023).Source: §2 · p. 2
• Prior studies find that demographic prompting flattens within-group diversity and that performance degrades for non-Western populations; this paper quantifies the WEIRD/non-WEIRD asymmetry.The reported dispersion-retention ranges are σr = 0.70–0.87 for WEIRD countries and σr = 0.11–0.60 for non-WEIRD countries, with 70B alignment fine-tuning widening the gap.Source: §2 · p. 3
• Earlier survey fine-tuning work reports improved distributional alignment, while this study extends evaluation to a cross-national benchmark and tests whether another training objective avoids collapse.The Demographic Contrastive DPO approach pairs respondents with contrasting demographics who answered the same question differently.Source: §2 · p. 3
• Research on alignment bias documents skew toward some demographic views; this paper compares aligned and unaligned variants and finds that 70B fine-tuning amplifies the WEIRD/non-WEIRD gap.At 9B, the Qwen family shows only a small gap at every training stage.Source: §2 · p. 3
• Interview-based agents can match participants’ own two-week test–retest consistency 85% of the way, placing structured personas at a less information-rich point on an input-richness spectrum.This work provides diagnostic evidence at the structured-persona point, where most cross-national survey simulation research operates (Park et al., 2024).Source: §2 · p. 3

3.1 Dataset

• Each WVS record pairs a respondent’s 59–70-word value-encoded persona with a question and ground-truth ordinal answer.Personas encode demographics, Inglehart–Welzel cultural-map axes, interpersonal trust, civic engagement, and religious identity.Source: §3.1 · p. 3

3.2 Question Split Strategy

• The evaluation trains on 100,000 respondent–question pairs, including 50,000 for GRPO, and tests on 10,000 pairs from six held-out questions.The test set includes high-consensus, contested, and culturally sensitive questions and covers Values, Social Trust, and Political clusters; Economic appears only in training.Source: §3.2 · p. 4, Table 1 · p. 4

3.3 Prompt Format

• All chat models receive the same three-turn respondent-simulation prompt, while the base model receives the same content as a plain completion prompt.The study evaluates eleven zero-shot models and five WVS-fine-tuned variants, spanning SFT, DPO, and GRPO configurations.Source: §3.3 · p. 4

3.4 Evaluation Metrics

• Accuracy and MAE measure pooled point prediction, while W1 and σr measure per-cell distribution distance and dispersion retention, respectively.σr = σpred/σtruth; 1.0 is perfect retention, values approaching 0 indicate mode collapse, and values above 1 indicate over-dispersion.Source: §3.4 · p. 4

3.5 Inference

• Primary evaluations use greedy decoding (T = 0), and both WVS-fine-tuned DPO models are additionally tested at T = 0.5, 0.7, and 1.0.The sweep tests whether broader sampling dissolves consensus collapse.Source: §3.5 · p. 4

4.1 Alignment reduces human variance

• 57.9% accuracy accompanies σr = 0.50 for Tulu 3 70B-DPO (+WVS), while unaligned Qwen 3.5 9B retains σr = 0.73 at 38.4% accuracy.Across WVS fine-tuning runs, SFT and DPO improve point accuracy by 8.2–14.6 points while compressing dispersion by ∆σr of −0.05 to −0.14; sparse-reward GRPO leaves accuracy unchanged and σr within 0.03 of baseline, whereas distribution-shaped GRPO reaches σr = 0.21.Source: §4.1 · pp. 4–5, Table 2 · p. 5, §4.3 · p. 6

4.2 The collapse is not recoverable by raising the sampling temperature

• At T = 1.0, W1 stays at 0.73 for Qwen and σr shifts only from 0.50 to 0.51 for Tulu, so temperature does not recover collapse.Tulu’s non-WEIRD σr remains 0.33–0.34 across the tested temperatures; the authors did not test temperatures above 1, nucleus or min-p sampling, or diversity-seeking prompts.Source: §4.2 · pp. 5–6, Table 3 · p. 6

4.3 Reinforcement learning on Qwen 3.5 9B does not reverse the collapse

• Sparse-reward GRPO leaves Qwen near baseline, while distribution-shaped reward raises accuracy to 47.5% and lowers W1 to 0.84 but reduces σr from 0.70 to 0.21.The distribution-shaped reward is maximized by emitting the cell’s modal answer and reaches σr = 0.00 on Q1.Source: §4.3 · p. 6

4.4 The collapse is most severe for underrepresented populations

• The WEIRD/non-WEIRD gap opens at Tulu SFT and deepens with survey fine-tuning; Nigeria retains only 11% of human response spread.The authors say non-WEIRD populations may be less represented in pre-training data, but they do not measure that representation; the Qwen family is an exception, with gaps at most 0.08 and none after SFT.Source: §4.4 · p. 6, Table 4 · p. 6, Table 5 · p. 7

4.5 Question-level variation reveals where collapse matters most

• For Q1, all models reach about 88–90% accuracy while σr stays below 0.25; for Q182, Tulu-DPO reaches 52.3% accuracy versus Qwen zero-shot’s 9.9%, with σr of 0.53 versus 0.68.On Q189, Tulu-DPO predicts “never justifiable” for all 1,544 respondents, although human responses vary substantially across countries; the question was not administered in Great Britain.Source: §4.5 · p. 7
• Across post-trained models outside Qwen 3.5 9B, dispersion retention differs by 0.20–0.43 between WEIRD and non-WEIRD populations despite broadly comparable MAE.The Llama 3.1 70B base has no gap (1.21 vs. 1.22); within Qwen, the gap is at most 0.08 and disappears after SFT (0.68 vs. 0.68).Source: Figure 3 · p. 7

4.6 How Much Can a Post-Hoc Fix Recover?

• On held-out data, prior mixing raises σr from 0.51 to 0.62 and lowers W1 from 0.54 to 0.51, but Nigeria reaches only σr = 0.36.The held-out evaluation uses 8,005 pairs; the authors characterize recovery as bounded, with large residual gaps for non-WEIRD populations.Source: Table 6 · p. 8, §4.6 · p. 8, Table 7 · p. 8

5.1 Why value-encoded personas are not enough

• Nigeria retains σr = 0.11 despite personas encoding key WVS dimensions; the authors suggest missing biographical detail explains part, but not all, of the deficit.A persona-conditioned Bayes-optimal point predictor retains σr ≈0.31 on average, yet Nigeria, Ethiopia, and Kenya fall below that floor.Source: §5.1 · pp. 8–9
• Under cross-entropy, a point predictor using incomplete inputs has variance no greater than the target variance; interview transcripts provide richer context and match participants’ two-week retest accuracy 85% of the way.The paper presents structured personas as one point on a possible continuum from demographics to dense biographical context.Source: §5.1 · p. 9

5.2 Implications for the field

• Accuracy alone can mislead when evaluating SRM systems; the authors recommend reporting W1 and σr alongside it.The most accurate model in the study is also the least distributionally faithful.Source: §5.2 · p. 9
• Underrepresented and non-Western populations experience the greatest collapse; the authors suggest alignment objectives may contribute, but did not measure training-data coverage directly.The proposed explanation is qualified as plausible rather than directly tested.Source: §5.2 · p. 9
• Decoding settings do not appear to reverse collapse, so the authors point to richer individual context, post-hoc methods, or prompt-level distribution elicitation as possible interventions.Prompt-level elicitation restored GPT-4o’s spread at a substantial accuracy cost.Source: §5.2 · p. 9

6 Conclusion

• 57.9% accuracy still leaves the best model with only half the human spread (σr = 0.50).Tulu 3 70B-DPO fine-tuned on WVS keeps 11% of the spread for Nigeria, versus 70–87% for WEIRD populations.Source: §6 · p. 9
• General post-training removes half the spread with minimal accuracy gain, and survey fine-tuning buys accuracy at a further cost.The paper terms this systematic compression of response spread consensus collapse.Source: §6 · p. 9
• Sampling at up to T = 1.0 leaves W1 unchanged for both fine-tuned DPO models, while prior mixing raises Nigeria’s σr only from 0.13 to 0.36.GRPO on Qwen 3.5 9B leaves the model at its pre-trained prior under a sparse accuracy reward and collapses variance further under a distribution-shaped reward.Source: §6 · p. 9

Limitations

• The evaluation uses six held-out WVS questions from one survey instrument.Replication on other instruments, including Pew, ANES, and Eurobarometer, is identified as valuable future work.Source: Limitations · pp. 9–10
• The authors hypothesize, but have not tested, that the proposed cross-entropy mechanism is instrument-agnostic.The WVS encodes Inglehart–Welzel dimensions directly into personas, and the documented country asymmetry mirrors patterns reported across other cultural instruments.Source: Limitations · p. 10
• Two test questions overlap persona indices, and the authors did not rebuild personas after removing those items.Excluding the two test items leaves the accuracy–diversity ordering unchanged.Source: Limitations · p. 10
• The decoding sweep covers temperatures only up to 1.0, and higher temperatures, alternative samplers, and diversity-seeking prompts were not tested on fine-tuned models.The evaluation also uses English prompts, ordinal scales, and a standardized neutral persona prompt; generalization beyond those settings requires investigation.Source: Limitations · p. 10
• Prompt-level distribution elicitation was tested only on GPT-4o, while the prior-mixing parameter was selected on a stratified 20% development split.The authors say a larger or independently collected development set would strengthen the prior-mixing claim.Source: Limitations · p. 10

Ethical Considerations

• The authors caution against substituting simulated responses for engagement with real communities, particularly marginalized populations, and describe geographic labels as analytical shorthand.The study uses public WVS data and reports adherence to model-provider terms and licenses.Source: Ethical Considerations · p. 10

A Question Inventory

• The test set covers three of four question clusters, with no held-out Economic item, so generalization to that cluster is untested.Q57 is used in persona construction and excluded from evaluation.Source: A Question Inventory · p. 11, Table 8 · p. 12

B Persona–Index Overlap

• Two test questions are components of RESEMAVAL encoded in personas, but excluding both leaves the accuracy–diversity ordering unchanged.The affected items are Q29 and Q182; four other test questions are outside the persona indices.Source: B Persona–Index Overlap · p. 12, Table 9 · p. 12

C Bayes-Optimal Variance Floor

• Because personas do not fully determine responses, σr = 1 is unattainable for any point predictor.The estimated attainable floor uses identical-persona cells in the WVS microdata and explained variance under the law of total variance.Source: C Bayes-Optimal Variance Floor · p. 12

Q In index Tier adds corr(tier, answer)

• The estimated oracle σr averages 0.31 across questions, with per-question values ranging from 0.08 for Q1 to 0.47 for Q182.The comparison is with Tulu 3 70B-DPO (+WVS).Source: Q In index Tier adds corr(tier, answer) · p. 12

D Temperature Analysis by Subgroup

• Across T = 0 to T = 1.0, Tulu 3 70B-DPO (+WVS) remains at σr 0.33–0.34 for non-WEIRD countries and 0.76 for WEIRD countries, with the gap unchanged.For Qwen 3.5 9B-DPO, W1 is flat in both groups; non-WEIRD σr rises from 0.59 to 0.64 without W1 improvement, which the authors read as added noise.Source: D Temperature Analysis by Subgroup · pp. 12–13, Table 11 · p. 13, Table 12 · p. 13

E Prompt-Level Remedies on GPT-4o

• The GPT-4o comparison uses a greedy baseline, population-draw prompting at T = 1.0, and greedy distribution elicitation followed by sampling from the stated distribution.The test uses 2,130 respondent–question pairs, with 30 pairs per country–question cell and identical respondents in every condition.Source: E Prompt-Level Remedies on GPT-4o · p. 13

Q Baseline Population Distribution

• On GPT-4o, distribution elicitation restores Q189 from σr 0.00 to 1.02 and lowers W1 to 0.57, but costs 12 accuracy points and over-disperses Q1 (σr = 2.02).The probe was run only on GPT-4o and does not establish whether elicitation recovers spread erased by survey fine-tuning.Source: Table 13 · p. 13

F Prior-Mixing α Sweep

• The α selection rule is based on a development-set sweep.The sweep uses n = 1,995 and routing seed 42.Source: F Prior-Mixing α Sweep · p. 14, Table 14 · p. 14
• W1 falls from 1.07 at α = 0 to 0.58 at α = 0.95, versus 0.60 for pure DPO.The sweep uses 1,995 development examples and routing seed 42.Source: Table 14 · p. 14
• α = 0.95 is the largest value whose development-set W1 is no worse than pure DPO’s while σr improves from 0.52 to 0.60.Table 14 reports the sweep underlying this selection rule.Source: Table 14 · p. 14, F Prior-Mixing α Sweep · p. 14

G Per-Question Accuracy and Dispersion Retention

• On Q182, Tulu 3 70B-DPO (+WVS) reaches 52.3% accuracy versus 9.9% for Qwen 3.5 9B zero-shot and 33.0% for GPT-4o, with σr = 0.53 versus 0.68 for Qwen.The WVS-fine-tuned models and GPT-4o score 33–79% on Q182 and Q189, compared with 7–19% for Qwen zero-shot and sparse-reward GRPO.Source: G Per-Question Accuracy and Dispersion… · p. 14, Table 15 · p. 14
• On Q64, unaligned models have σr > 1.0, suggesting that some of their preserved spread may be noise rather than signal.Table 16 reports overdiversification on this question.Source: G Per-Question Accuracy and Dispersion… · p. 14, Table 16 · p. 14
• On high-consensus Q1, all models reach approximately 88–90% accuracy but σr < 0.25, a collapse the authors describe as harmless when ground truth is near-unanimous.Table 15 identifies Q1 as high-consensus across all models.Source: G Per-Question Accuracy and Dispersion… · p. 14, Table 15 · p. 14

H Prompt Template

• The chat models receive the same verbatim three-turn prompt, with persona, question, and response bounds filled per sample and no other prompt engineering, few-shot examples, or chain-of-thought instructions.The base model receives the content as a single completion prompt and is decoded greedily for at most 8 tokens.Source: H Prompt Template · p. 14, Q Tulu Qw.DPO Qw.SFT GPT-4o Qw.ZS GRPO · p. 14

G Per-Question Accuracy and Dispersion Retention

• On Q189, Tulu 3 70B-DPO (+WVS) and GPT-4o have σr = 0.00 despite substantial human variation, while unaligned models retain σr > 1.0.The table identifies this as complete collapse for both models.Source: Table 16 · p. 14, G Per-Question Accuracy and Dispersion… · p. 14

Q Tulu Qw.DPO Qw.SFT GPT-4o Qw.ZS GRPO

• The prompt asks the model to simulate a respondent from a supplied profile and return only one integer within the question’s response bounds.It includes a system instruction to follow the specified format without additional commentary.Source: Q Tulu Qw.DPO Qw.SFT GPT-4o Qw.ZS GRPO · p. 14

I Models and Hyperparameters

• Custom fine-tuning uses LoRA with rank 16 and α = 16; the 70B model also uses 4-bit NF4 quantization.The work runs through Unsloth on a single 80 GB-class NVIDIA GPU.Source: I Models and Hyperparameters · p. 14
• Qwen 3.5 9B SFT uses 100,000 respondent–question pairs from 15 training questions, with 5% held out for early stopping.The sample covers all 12 countries and is drawn using seed 3407.Source: I Models and Hyperparameters · pp. 14–15
• Demographic Contrastive DPO trains on 100,000 pairs, using each respondent’s answer as chosen and another respondent’s answer to the same question as rejected.The method is applied to Qwen 3.5 9B and replicated on Tulu 3 70B.Source: I Models and Hyperparameters · p. 15
• Qwen 3.5 9B GRPO uses 50,000 training samples and eight generations per prompt, with both accuracy-based and distribution-shaped rewards evaluated.The distribution-shaped reward is based on relative human response frequency within each country–question cell, estimated from training questions.Source: I Models and Hyperparameters · p. 15
• The evaluation covers eleven zero-shot checkpoints and five WVS-fine-tuned variants using SFT, DPO, or GRPO.Table 17 lists training methods, parameter counts, precision, and key hyperparameters.Source: Table 17 · p. 15
• The evaluated checkpoints are served through a mix of OpenRouter, the Hugging Face Inference API, and local Unsloth runs.The Qwen comparison in Table 2 contrasts an API-served zero-shot run with locally served fine-tuned runs.Source: Table 17 · p. 15