When Persona Attributes Improve Population Alignment in Large Language Models

1 Introduction

• Prior annotation studies link greater human agreement to better LLM reproduction, but whether this relationship generalizes to survey tasks remains unclear (Brown et al. (2025), Sarumi et al. (2024)).Source: §1 · p. 2
• Human response variation—meaningful population heterogeneity—is proposed as a potential explanation for why persona prompting reproduces survey response distributions differently across tasks.The concept captures variation in participants’ responses to a survey question, rather than treating it as a measurement artifact.Source: §1 · pp. 2–3
• Response variation is measured using normalized entropy for nominal responses and dissention for ordinal responses, across survey questions with varying levels of variation.Source: §1 · p. 3
• The study benchmarks persona-attribute selection methods for each target survey question, addressing the importance of attribute choice and prior blockwise changes that ignored the prediction task.Source: §1 · p. 3
• The study asks whether human response variation explains task-level performance differences and whether attribute-selection methods improve persona prompting.Source: §1 · p. 3
• The evaluation uses national-level US and German data from the World Value Survey, the US General Social Survey, and the German General Social Survey.Source: §1 · pp. 3–4

2.1 Persona Prompting Studies Report Mixed Results

• Although persona prompting has many applications, few prior works explain when it succeeds; the authors propose human response variation as a plausible explanation for mixed performance.Related work also links outcomes to attributes correlated with the target and to variation in human annotations.Source: §2.1 · pp. 4–5
• The authors hypothesize that persona prompting helps more on high-variation questions, hurts on low-variation questions when attributes add noise, and that this pattern reverses without personas.They reason that persona attributes can model deviations from an LLM’s default response, whereas on concentrated questions the default may already represent the majority view.Source: §2.1 · p. 5

2.2 Persona Attribute Selection Matters

• Irrelevant attributes can lower task performance by up to 30 percentage points (Luz de Araujo et al., 2025), while explicit attributes outperform names and political ideology appears most influential.Giorgi et al. report gender as the least important attribute.Source: §2.2 · pp. 5–6
• This paper benchmarks task-specific persona-attribute selection, comparing statistical approaches using full-survey associations with LLM-based approaches that may leverage broader world knowledge.The authors note that relationships used by LLMs may not be evident in an isolated survey or captured by traditional statistical methods.Source: §2.2 · pp. 6–7
• The authors assume LLM-optimized attribute sets will outperform statistical baselines that rely on survey data alone.They hope LLM-based selection can leverage the models’ internal knowledge representations.Source: §2.2 · p. 7

3.1 Measuring Human Response Variation

• Nominal responses use normalized Shannon entropy, while ordinal responses use the inverted Tastle and Wierman (2006) consensus measure, called dissention.The methods are designed for different response types; entropy is invariant to response locations on an ordinal scale.Source: §3.1 · p. 7

3.2 Selecting Persona Attributes

• They benchmark target-specific attribute sets using survey-data baselines or LLM selection from question texts, with LLMs choosing five variables by set selection or scoring.Scoring rates each variable from 0 to 100 and selects the five highest-ranked; set selection chooses the five most important variables directly.Source: §3.2 · pp. 8–9

3.2.1 Stability

• LLM-based selection runs 100 times per target-question and model, reduced to 30 for the two largest models, with variable order shuffled between runs.Persona attributes are drawn from the most frequently selected variables for set selection or the highest average-scoring variables for scoring.Source: §3.2.1 · pp. 9–10

3.3 Predicting Survey Responses

• Zero-shot prediction uses fixed prompts and generation settings, supplying the persona description, target question, and response options to elicit the most likely answer.The no-persona baseline repeats predictions as many times as the survey has participants; few-shot prompting and fine-tuning are left for future work.Source: §3.3 · p. 10
• A no-persona baseline repeatedly predicts each target response without specifying a persona, matching the number of predictions to the survey’s participant count.Alternative approaches such as few-shot prompting and fine-tuning are deferred to future work.Source: §3.3 · p. 10

3.4 Evaluating Survey Response Prediction Performance

• Population-level performance uses Jensen-Shannon Distance between aggregated persona predictions and human response distributions; normalized JSD ranges from 0 to 1, with lower values indicating better similarity.Source: §3.4 · p. 10

4.1 Survey Data

• Experiments use GSS and WVS data from the United States and GGSS and WVS data from Germany, with WVS directly comparable across countries.The study uses the latest available data from these four surveys.Source: §4.1 · p. 11, Table 2 · p. 11

4.2 Target Questions

• Target questions are behavioral or attitudinal variables, with the five lowest- and five highest-variation nominal and ordinal variables selected when at least 500 participants answered.Variation scores are ranked separately for nominal and ordinal variables.Source: §4.2 · p. 11

4.3 Model Selection

• The experiments use Qwen and Llama models from 3B to 70B parameters, keeping the same model for persona-attribute selection and response prediction.The varied families, providers, and sizes are intended to test model-choice effects and the approach’s generalizability.Source: §4.3 · p. 12
• The authors characterize data leakage from the surveys into the models as practically impossible for GSS and GGSS, and highly unlikely for the two WVS surveys.Source: §4.3 · p. 12
• Persona prompting performs better on high-variation questions, while statistical attribute-selection baselines outperform LLM-based selection approaches.The overview compares performance using Jensen-Shannon Distance, for which lower is better.Source: Figure 3 · p. 13

5.1 RQ1: Impact of Human Response Variation on Persona Prompting Performance

• Persona prompting predicts responses better for high- than low-human-response-variation questions, while no-persona predictions show the reverse pattern.The contrast suggests relevant persona attributes help LLMs model variation in human responses.Source: §5.1 · pp. 13–14
• The variation-related performance pattern generally holds when results are disaggregated by model family and survey.Source: §5.1 · p. 14

5.2 RQ2: Performance Ranking of Attribute Selection Approaches

• Statistical selection using Correlation and Feature Importance aligns predictions better with survey responses than LLM-based Set Selection and Scoring.LLM-based selections also perform worse and are inconsistent in their chosen attributes.Source: §5.2 · p. 14
• Set Selection performs better than Scoring, and persona prompting outperforms the no-persona baseline across all tested attribute-selection approaches.Differences between selection approaches are more pronounced for high- than low-human-response-variation questions.Source: §5.2 · p. 14
• The performance ordering of attribute-selection methods is consistent across model families and surveys.Source: §5.2 · p. 14

5.3.1 Stability of the Set Selection Approach

• Set-selection stability increases with model size: Llama3.3-70B selects the same attribute as most important in 58% of runs and above 80% for some questions.Source: §5.3.1 · p. 14, Figure 4 · p. 15

5.3.2 Stability of the Scoring Approach

• Larger models, especially Llama-3.3-70B-Instruct, vary sharply by target—from nearly reproducible rankings to high fluctuation—while smaller models fluctuate highly regardless of target.Source: §5.3.2 · p. 15
• Scoring rankings are weakly reproducible (mean Kendall’s W 0.25–0.5); consistency rises with Llama size but not Qwen size, whose largest model has the lowest mean correlation.Source: §5.3.2 · p. 15, Figure 5 · p. 16

• Practitioners can first use human-response-variation measures to target highly contested questions, then use same- or similar-question survey data to select attributes that best align predictions with responses.Source: §6 · p. 16
• The proposed framework could support adaptive survey design that shows selected questions to particular participant groups, improving resource allocation if suitable questions and well-modeled subgroups can be identified.Source: §6 · pp. 16–17
• A statistical baseline using correlations between target and input variables selects the best persona attributes, outperforming more complex LLM- or semantic-similarity-based approaches.Source: §6 · p. 17
• The findings provide no evidence that correlations among selected features matter; the best-performing approach focuses on each input variable’s correlation with the target.Source: §6 · p. 17
• The findings offer a first indication, not a definitive answer, that LLM response representations may reflect human-survey correlations, since internal-representation selectors did not outperform survey-data-informed alternatives.Source: §6 · p. 17
• Future work could inspect activations during persona-conditioned response generation and compare them with input-attribute representations, which remain hidden when only generated outputs are observed.Source: §6 · p. 17

• The study did not vary attribute-selection designs, models, persona formats, generation parameters, or response parsing, since systematically testing them would intensify computational demands.Source: Limitations · p. 18
• Random selection, using all attributes, and theory-informed selection could clarify when and why selection matters, but translating theories into ranked attributes is difficult.Source: Limitations · p. 18
• A traditional survey-imputation baseline is absent, limiting performance context, although prior work found persona-prompted LLMs competitive with a random-forest classifier (Rupprecht et al., 2026).Source: Limitations · p. 18
• The study lacks qualitative comparisons of selected attributes across methods, models, surveys, and countries to reveal biases, model-specific patterns, or cultural sensitivity.Source: Limitations · p. 18
• The authors also note wider social-simulation challenges: limited human diversity, systematic biases, sycophancy, alienness, and generalizability across contexts.Source: Limitations · p. 18

• Persona prompting risks essentialist profiling from insufficient demographics and flattening or caricaturing people; the authors seek to reduce essentialism by exploring attributes beyond usual socio-demographics.Source: Ethical Considerations · p. 19
• Reusing existing survey data and using locally hosted models avoids adding collection risks and prevents data leakage to hosted models.Source: Ethical Considerations · p. 19
• Labor displacement, environmental harms from increased LLM use, and malicious use remain relevant societal concerns for LLM applications.Source: Ethical Considerations · p. 19
• The authors caution against replacing human survey participants with silicon samples while advocating continued evaluation to inform when LLM-based survey methods should be used.Source: Ethical Considerations · p. 19

7.1 Paraphrasing of Survey Questions

• Survey-question paraphrasing used GPT-5.1 through the OpenAI API with default parameters, and the generated paraphrases were manually assessed as sufficiently high-quality and suitable.Source: §7.1 · p. 24
• Figure 6 documents the user prompts used to paraphrase survey questions, with placeholders replaced by the corresponding information.Source: Figure 6 · p. 24
• Figure 7 documents the batch-paraphrasing prompt, which is appended to the base prompt for every batch except the first.Source: Figure 7 · p. 24

7.3 Response Prediction

• For each target question, the response-prediction approach generates synthetic distributions using persona descriptions built from the most important attributes, with a separate prompt per participant.The prompts require a reasoning stream and a dictionary-formatted answer with a short explanation and final response.Source: §7.3 · p. 26

7.3.1 Persona Prompting Approaches

• Figure 10 documents the system and user prompts for persona-based response prediction.Source: Figure 10 · p. 26

7.3.2 No Persona Baseline

• Figure 11 documents the system and user prompts for the no-persona response-prediction baseline.Source: Figure 11 · p. 26

8.1 Survey Data

• The study converts survey codebooks into machine-readable question and response-option formats, then annotates variables by scale and information type to guide persona selection and response-variation measures.It also removes direct duplicates; nominal, ordinal, and continuous scales are distinguished.Source: §8.1 · p. 27

8.2 Target Questions

• For each survey, 20 target variables are grouped by nominal or ordinal scale and low or high human response variation, with their response distributions shown in Figures 12–15.Tables 5–8 provide the target question texts, human response variation scores, and valid-response counts; variable IDs link the tables and figures.Source: §8.2 · p. 28, Figure 12 · p. 29, Figure 13 · p. 30
• Tables 5–8 list selected target questions and their human response variation scores and valid-response counts, while variable IDs connect the tables to the figures.The tables cover GSS, GGSS, WVS_DE, and WVS_US.Source: §8.2 · p. 28, Table 5 · p. 33, Table 6 · p. 34

8.3 Model Selection

• Although WVS data release timing made leakage theoretically possible, dataset complexity and comparable predictive performance suggest no actual leakage occurred; GSS and GGSS leakage was practically ruled out.GSS and GGSS were released after the known model cutoffs and Llama releases, and less than two months before Qwen releases.Source: §8.3 · p. 37

9.1 Disaggregation based on Model Family

• The variation effect was smaller in German than US surveys, and statistical-baseline attributes particularly impaired low-variation predictions by moving LLMs away from the correct baseline response.Source: §9.1 · pp. 37–38
• Persona-prompted LLMs predict high-response-variation questions better, whereas the no-persona baseline performs better on low-variation questions; this pattern holds for both Llama and Qwen models.Source: §9.1 · p. 37, Figure 16 · p. 39

9.2 Disaggregation based on Survey

• Statistical attribute-selection approaches retained their performance advantage over LLM-based approaches across model families and surveys, with differences generally less pronounced for US surveys.Source: §9.2 · p. 38, Figure 16 · p. 39, Figure 17 · p. 40

9.1 Disaggregation based on Model Family

• Survey-level exceptions include Llama reversing the variation pattern for GGSS and some Qwen attribute-selection methods favoring low-variation questions; the general finding holds for the other German survey.Source: §9.1 · p. 37, Figure 17 · p. 40

9.3.1 Stability of the Set Selection Approach

• Selection stability increased with model size: top-attribute recurrence was 15.1% for Qwen2.5-3B, 14.8% for Llama3.2-3B, 32.4% for Qwen2.5-7B, and 21.6% for Llama3.1-8B.Source: §9.3.1 · p. 41
• Llama3.3-70B selected the same top attribute in 57.3% of runs on average and selected all five attributes consistently for some target questions.Source: §9.3.1 · p. 41

9.3.2 Stability of the Scoring Approach

• Llama scoring was most erratic across surveys, while Qwen showed little survey-specific difference beyond GSS and less pronounced model-size differences than Llama.Source: §9.3.2 · p. 43
• For both model families, scoring concordance was much higher on GSS than other surveys, potentially because GSS has fewer predictors and possible rankings.Source: §9.3.2 · p. 43, Figure 19 · p. 44, Figure 20 · p. 45