Assessing the Reliability of Persona-Conditioned LLMs as Synthetic Survey Respondents

1 Introduction

• Persona-conditioned LLM outputs are increasingly used as synthetic survey responses and in social simulations, motivating assessment of their survey-grounded external validity and subgroup fidelity.Here, external validity means agreement with population response patterns, while subgroup fidelity concerns within-group agreement and preservation of between-group differences.Source: §1 · p. 1
• Errors may concentrate in particular survey items and demographic groups even when aggregate agreement appears satisfactory, potentially affecting downstream conclusions.Prior work also cautions that model outputs can misrepresent marginalized groups when treated as observations.Source: §1 · p. 1
• Existing evidence leaves unclear whether multi-attribute personas improve on a matched no-persona control or instead redistribute errors across items and demographic subgroups.Simultaneously conditioning on attributes such as age, education, and employment may let a model match some attributes while distorting others.Source: §1 · p. 2
• The study evaluates two open-weight chat models against U.S. World Values Survey microdata, isolating persona prompting with a matched vanilla comparison and a random-guesser baseline.Its diagnostics examine both item- and attribute-level agreement, including hard and soft similarity measures.Source: §1 · p. 2
• The authors discuss downstream risks for synthetic survey respondents and persona-conditioned agents when miscalibration affects particular population subgroups.The contribution focuses on locating where models match survey-grounded patterns and where they deviate.Source: §1 · p. 2

2 Related Work

• Research spans synthetic survey respondents, persona alignment, social simulations, and LLM-generated data as computational social science inputs.Across these applications, survey-grounded fidelity and the assumptions behind synthetic data remain central concerns.Source: §2 · p. 2, §2.4 · p. 3

2.1 LLMs as Synthetic Survey Respondents

• LLMs approximate some survey marginals, but results vary across instruments, countries, and modeling choices; few studies isolate if and how persona prompting changes item-level agreement and subgroup differences within a single standardised survey such as the WVS.Source: §2.1 · p. 2

2.2 LLM Personas, Personality, and Alignment

• Persona and trait conditioning can steer outputs, but existing evaluations often emphasize coherence or coarse alignment rather than item-level survey validity and subgroup fidelity.Prior findings include systematic persona-related reasoning differences [17], caricatured group portrayals [10], and shifts that do not consistently improve demographic alignment.Source: §2.2 · p. 3

2.3 LLM Agents and Social Simulations

• Social-simulation studies often assume synthetic populations reproduce human attitudes and group differences, an assumption the authors say has not been convincingly validated.The paper presents its evidence as a challenge to that assumption and argues for alignment and failure-mode checks in synthetic populations.Source: §2.3 · p. 3

2.4 LLM Data for Computational Social Science

• Work on synthetic data documents risks of unfaithful distributions, propagated bias, and survey responses with reduced variance or prompt sensitivity.Bisbee et al. caution that aggregate agreement may still be insufficient for inference; this study supplies evidence for a standardized public-opinion instrument.Source: §2.4 · pp. 3–4

3.1 Dataset and Preprocessing

• The evaluation uses U.S. respondent-level WVS-7 microdata as ground truth and a curated subset of 31 items with consistently specified questions, options, and response codes.The shared response specification supports uniform decoding across conditions.Source: §3.1.1 · p. 4

3.1.2 Persona attributes.

• Eight respondent attributes define persona clauses and subgroup partitions; unavailable attributes are omitted, while non-substantive entries are filtered without imputing missing values.The attributes are gender, age, education, employment, occupation, income, religion, and ethnicity.Source: §3.1.2 · p. 4
• Normalized entropy indicates variation across all eight attribute marginals, ranging from 0.691 for gender to 0.372 for income.The reported values are gender 0.691, age 0.657, education 0.538, occupation 0.537, employment 0.478, ethnicity 0.446, religion 0.431, and income 0.372.Source: §3.1.2 · p. 4

3.2 Evaluation Setup

• The study compares persona-based and matched vanilla responses against WVS-7 microdata, with a uniform random guesser as a chance baseline, using the same pipeline for Llama-2-13B and Qwen3-4B.The objective is to isolate the incremental effect of multi-attribute persona prompting and assess both aggregate external validity and subgroup fidelity.Source: §3.2 · p. 4

3.2.1 Prompting conditions and matched comparison.

• Persona-based prompts add a respondent profile across eight demographic attributes, while vanilla prompts omit demographic clauses and the random guesser samples uniformly from valid options.The matched conditions are intended to isolate demographic personalization; alternative conditioning options are left to future work [6].Source: §3.2.1 · p. 4

3.2.2 Decoding protocol.

• Each respondent–item instance receives one completion with a 40-token cap and temperature 0.3, so results are point estimates under fixed decoding rather than estimates of generation variance.Repeated runs, random seeds, and prompt paraphrases are not evaluated.Source: §3.2.2 · p. 4

3.2.3 Response parsing and quality control.

• A shared parser normalizes outputs and matches an option index or uniquely identifying text to valid WVS-7 responses; unparseable outputs can make retained persona counts differ by model.The routine is applied identically to persona and vanilla conditions.Source: §3.2.3 · p. 4

3.2.4 Evaluation metrics.

• Hard similarity measures accuracy, while soft similarity uses normalized match distance for ordinal items and reduces to binary mismatch for non-ordinal or non-substantive responses.Subgroup fidelity applies both measures within demographic strata; Table 1 reports overall HS and SS for persona-based, vanilla, and random-guesser conditions, alongside users and significance comparisons.Source: §3.2.4 · pp. 4–5, Table 1 · p. 5

Overall external validity

• The matched design compares PB with V on WVS-7 U.S. respondent microdata and 31 items, isolating demographic clauses as the difference between conditions.Results use hard and soft similarity and are contextualized against a uniform random-guesser baseline.Source: §4 · p. 5
• PB does not yield consistent aggregate gains over matched V across models, although both outperform random guessing and Qwen3-4B scores higher overall than Llama-2-13B.For Llama-2-13B, PB slightly reduces HS and SS; for Qwen3-4B, it modestly raises HS while leaving SS unchanged, with no statistically significant PB–V differences.Source: §4 · p. 5

Item-level persona effects

• On Q83, PB raises HS for both models, but SS improves only for Qwen3-4B, showing exact-match changes can diverge from ordinal-distance alignment.For Llama-2-13B, unchanged SS despite higher HS suggests any exact-match gains are offset by unchanged or larger ordinal distances on other mismatches.Source: §4 · p. 5

Subgroup fidelity and small-group instability

• Subgroup scores show no consistent PB improvement, and the largest shifts tend to occur in small-n strata, where estimates have higher variance.Uneven error redistribution may undermine subgroup fidelity and distort downstream analyses relying on subgroup comparisons or simulated agent populations.Source: §4 · p. 5

Item-level persona effects

• Most item-level PB and V scores track closely, but a small set drives the largest HS and SS swings; Qwen3-4B shows larger swings than Llama-2-13B.The figure compares HS and SS scores by item and plots item-wise PB–V differences; positive differences indicate PB outperforms V.Source: §4 · p. 5, Figure 1 · p. 6

Limitations

• Findings are limited to U.S. WVS-7 respondents and a curated item subset, and may not transfer to other countries or questions with different cultural contexts and response distributions.The study also evaluates only two open-weight models under a fixed pipeline.Source: §4 · p. 6

Conclusions

• Persona prompting did not produce a consistent aggregate gain, while effects varied across items and subgroup fidelity sometimes degraded in low-support strata.Both models exceeded random performance, and Qwen3-4B showed higher overall survey-grounded agreement than Llama-2-13B.Source: §5 · p. 6

Practical implications

• For synthetic polling, agent initialization, or subgroup analysis, the authors recommend reporting matched vanilla baselines and auditing item- and subgroup-level behavior rather than only aggregate scores.Source: §5 · p. 6

Future work

• Future work should broaden models and geographic scope, incorporate repeated runs, test prompt designs and languages, and better respect survey design and distributional targets.Source: §5 · p. 6

• The World Values Survey is a long-running cross-national survey of population responses across social, cultural, political, ethical, and economic issues.Source: A Selected WVS-7 items · p. 8
• 31 WVS-7 questions, curated from a wave of 259 questions, form the study’s selected item set.Source: A Selected WVS-7 items · p. 9, Table 3 · p. 9
• The selected questions span diverse themes and probe non-factual attitudes, making them suitable for examining response variation and subgroup differences.Source: A Selected WVS-7 items · p. 9

• YAML templates standardize question headers, option headers, and response-option layout across survey items.Source: B Prompt Templates, Answer Formatting,… · p. 9
• Non-scale options are listed verbatim, while scale items use a fixed 10-point instruction anchored at 1 and 10.Source: B Prompt Templates, Answer Formatting,… · p. 9
• The prompt dataset supports selectable design variants and logs each active variant with its respondent–item instance for stratified analysis.Source: B Prompt Templates, Answer Formatting,… · p. 9
• Chat prompts use a system-message and user-query envelope, keeping persona text in the instruction context without altering response-option encoding or order.Source: B Prompt Templates, Answer Formatting,… · p. 9