PaperlayerView exact revision on arXiv

Source-linked AI summary

When Persona Attributes Improve Population Alignment in Large Language Models

Leon Fröhling, Jens Rupprecht, Markus Strohmaier, Claudia Wagner

arXiv:2609.02526v1cs.CLcs.CY

TL;DR

The paper addresses why persona prompting produces mixed results in survey response prediction and how persona attributes should be selected. It introduces human response variation as a potential explanation, benchmarks selection methods across surveys, countries, and LLMs, and reports implications for when persona prompting may be useful. The evaluation is limited to zero-shot persona prompting, leaving few-shot prompting and fine-tuning for future work.

  • Problem

    Persona prompting has shown mixed results, while the effects of human response variation and different persona attribute-selection methods remain unclear.

  • Method

    The paper measures human response variation and compares task-specific persona attribute-selection approaches for survey prediction across US and German survey data.

  • Results

    The study evaluates persona prompting across national surveys from the US and Germany, including the World Value Survey, the US GSS, and the German GGSS.

  • Takeaways & Limitations

    Human response variation and attribute selection are investigated as factors that may identify when persona prompting is useful for survey prediction.

  • Takeaways & Limitations

    The evaluation uses zero-shot persona prompting and leaves few-shot prompting and fine-tuning for future work.

Abstract

from arXiv · show

Large Language Models (LLMs) are increasingly used to predict the responses of human participants in survey panels. Towards that goal, persona prompting has recently emerged as a technique to inform and align large pretrained language models. Persona prompting refers to the practice of using short textual descriptions of 'personas' in prompts to steer the LLM's generations. Personas describe individuals through different attributes such as their socio-demographics, attitudes, or behaviors, with the aim of aligning LLMs to produce responses that correlate with the corresponding human responses. Yet, recent work has produced mixed and partly conflicting results of persona prompting without clear patterns of success and failure. Among the few consistent findings is that the selection of persona attributes matters, and that using more attributes does not necessarily lead to better performance. It remains unclear how different attribute selection methods perform and how to choose among them. In this paper, we propose that observed human response variation of a survey question is a potential explanation for the mixed performance observed so far. In addition, we compare the performance of persona prompting associated with different methods for selecting persona attributes. We evaluate these methods on four different (general) social surveys across two countries, six LLMs, and twenty prediction tasks per survey. Our work helps to identify when persona prompting can be expected to be useful in survey prediction tasks, and provides new insights on the effectiveness of different attribute selection methods for LLM-based survey prediction using persona prompting.

1 Introduction

The introduction argues that mixed persona-prompting results may reflect variation in human survey responses and motivates task-specific attribute selection. It frames two research questions about response variation and attribute-selection methods across surveys, countries, and models.

  • Persona prompting results are mixed, leaving the conditions for faithfully reproducing human judgments and behaviors poorly understood.
  • The paper defines human response variation as variation in participants’ responses to a survey question and treats it as meaningful population heterogeneity.
  • The authors propose differences in human response variation as a potential explanation for performance differences across survey response prediction tasks.
  • Because surveys contain diverse attributes and questions with varying response variation, they support evaluating task-specific persona construction.
  • The study asks whether human response variation explains persona-prompting differences and whether attribute-selection methods improve survey prediction.
  • The evaluation uses national-level data from the US and Germany drawn from the World Value Survey, the US GSS, and the German GGSS.

2 Background

The background reviews mixed persona-prompting findings and develops human response variation and attribute relevance as explanations for performance differences. It motivates comparing task-specific selection methods rather than fixed attribute blocks.

  • Prior work indicates that attribute selection matters: correlated or relevant attributes improve performance, whereas irrelevant attributes can reduce it.
  • Prior studies report mixed or inconsistent persona-prompting gains, and few explain when the approach works in a given context.
  • Human response variation is hypothesized to matter because personas provide attributes needed to model deviations from an LLM’s default response.
  • Persona prompting is hypothesized to perform worst on concentrated-response questions, where additional attributes may steer models away from the majority-aligned default.
  • The paper therefore benchmarks approaches for selecting an optimal attribute set separately for each survey response prediction task.
  • Explicit attributes outperform inferred names in one study, with political ideology appearing most influential and gender least important.

3 Methodology

The methodology measures human response variation, selects persona attributes for each target variable, and evaluates zero-shot persona prompting. It compares statistical and LLM-based selection approaches while holding prediction prompts and generation parameters fixed.

  • 3.1 Measuring Human Response Variation: Human response variation is measured with normalized Shannon entropy for nominal variables and dissention for ordinal variables.
  • 3.1 Measuring Human Response Variation: Both dispersion measures range from 0 to 1, with values near 0 indicating minimum and values near 1 maximum human response variation.
  • 3.2 Selecting Persona Attributes: The study compares statistical baselines with LLM-based approaches that select persona attributes for each target variable independently.
  • 3.2 Selecting Persona Attributes: The set-selection approach prompts an LLM to choose the five most important variables from the other survey-question texts.
  • 3.2 Selecting Persona Attributes: The scoring approach ranks survey variables from 0 to 100 for importance and selects the five highest-ranked variables.
  • 3.2 Selecting Persona Attributes: LLM-based selection is repeated across runs, then attributes are aggregated by selection frequency or average score.
  • 3.3 Predicting Survey Responses: Prediction uses zero-shot persona prompting, while alternative few-shot prompting and fine-tuning are left for future work.

4 Experimental Setup

The study evaluates persona prompting across four social surveys from Germany and the United States, selecting behavioral/circumstantial and attitudinal target variables by human response variation. It compares multiple persona-attribute selection strategies and LLM configurations.

  • Datasets: Four surveys from Germany and the United States provide the evaluation data.The surveys are GGSS, WVS-DE, GSS, and WVS-US.
  • Target variables: For nominal and ordinal variables, the study selects five questions with the lowest and five with the highest human response variation scores.Variables answered by fewer than 500 participants are excluded.
  • Target variables: Target variables represent behavioral/circumstantial or attitudinal information rather than demographic or contextual characteristics.These variable types are selected as the relevant target-response use case.
  • Models: The experiments use Qwen and Llama models spanning 3B to 70B parameters for persona-attribute selection and survey-response prediction.The model range is used to examine model-choice effects and generalizability.
  • Experimental comparison: The pipeline compares statistical and LLM-based attribute selection approaches across the selected surveys.The overview includes Correlation, Feature Importance, Set Selection, Scoring, and a No Persona baseline.

5 Results

Persona prompting performs better for high-human-response-variation questions, while statistical attribute-selection methods outperform LLM-based selection approaches. These patterns generally hold across model families and surveys.

  • Human response variation: Persona prompting performs better on high-human-response-variation questions than on low-variation questions across all attribute-selection approaches.Without persona conditioning, the LLM baseline shows the reverse pattern.
  • Human response variation: Including relevant persona attributes allows LLMs to model variation in human responses more meaningfully.The comparison is based on predicted versus actual human response distributions.
  • Attribute selection: Statistical methods based on Correlation and Feature Importance produce better alignment than the No Persona baseline across tested approaches.The performance differences between selection methods are larger for high-variation questions.
  • Attribute selection: LLM-based attribute selection performs worse than statistical selection and is inconsistent across runs.Among the two LLM-based approaches, Set Selection performs better than Scoring.
  • Attribute selection: The performance ordering of attribute-selection methods is consistent across model families and surveys.The disaggregated analyses generally preserve the human-response-variation effect as well.

Appendix 9.2).

The stability analyses show that larger models select attributes more consistently in the set-selection approach, while scoring-based importance rankings remain generally unstable across independent runs.

  • Set Selection: Set-selection stability increases with model size for both Llama and Qwen models.Llama3.3-70B selects the same attribute as most important in 58% of runs across target questions.
  • Set Selection: The set-selection analysis compares attribute-selection frequencies across repeated independent runs against perfect stability and random guessing.The curves show how often variables are selected among the top five attributes.
  • Scoring: Scoring-based importance rankings generally have weak to moderate agreement across independent runs, with mean Kendall’s W values between 0.25 and 0.5.Kendall’s W measures concordance among the rankings.
  • Scoring: For Llama models, ranking stability generally increases with model size, but the largest Qwen model has the lowest mean correlation across target questions.The model-size pattern therefore differs between the two model families.
  • Scoring: Larger Llama models show high variation across target questions, whereas smaller models fluctuate highly regardless of the target question.Some target variables have nearly reproducible rankings while others do not.

6 Discussion and Implications

The discussion proposes a practical two-step framework: use human response variation to identify suitable survey questions, then use existing survey data to select persona attributes. Results favor correlation-based statistical selection over more complex LLM- or similarity-based methods, while leaving the mechanisms inside LLMs unresolved.

  • Practical implications: A two-step framework first identifies survey questions likely to suit persona prompting using human response variation, then selects attributes using existing survey data.The second step aims to improve alignment between predicted and actual responses and inform decisions before committing additional resources.
  • Practical implications: Questions with high human response variation are generally presented as the most suitable targets for persona-based simulation.The discussion links this suitability to questions where population responses exhibit meaningful heterogeneity.
  • Attribute selection: Correlation-based statistical selection produces the best persona attributes and outperforms more complex LLM- or semantic-similarity-based approaches.The result favors attributes correlated with the outcome variable rather than approaches relying on model-internal representations or semantic similarity.
  • Attribute selection: Unlike traditional feature-selection guidance, correlation among selected attributes does not appear to matter for persona prompting; individual correlation with the target is decisive.The authors report no evidence that redundancy among selected features affects performance in this setting.
  • Interpretation and limitations: The findings provide an initial indication that LLM persona responses may use associations also observable in human survey data, but do not establish this definitively.LLM-based selection approaches with access to internal representations did not outperform survey-data-informed alternatives.
  • Interpretation and limitations: Future work should inspect the parameters and connections activated during persona-conditioned generation to assess whether selected attributes actually influence responses.This direction connects persona-attribute selection with faithfulness in explainable AI.

Limitations

The paper acknowledges practical, methodological, and analytical limitations, including omitted baselines, attribute-selection alternatives, model and prompting choices, and qualitative analyses.

  • The study omits several potentially relevant factors because it cannot consider every influence factor simultaneously.
  • The evaluated attribute-selection approaches do not include random selection, using all attributes, or theory-informed selection.Theory-informed selection is difficult because constructs may lack a single theory or an agreed ranking of attributes.
  • The experiments do not vary several model and response-generation choices, including model types, persona formats, generation parameters, and response parsing.Systematically varying these factors would substantially increase computational demands.
  • The paper lacks a traditional survey-imputation baseline for contextualizing response-prediction performance.The authors note that persona-prompted LLMs were already competitive with a random forest imputation baseline in most settings from previous work.
  • The study does not qualitatively compare selected attributes across approaches, LLMs, surveys, or countries.Such analyses could examine biases, model-specific patterns, and cultural adaptability or sensitivity to nuance.
  • The authors identify broader concerns about generalizability, human diversity, systematic biases, sycophancy, alienness, and non-human-like mechanisms in LLM social simulations.

Ethical Considerations

The paper discusses privacy, profiling, and broader societal risks of LLM-based social simulations while cautioning against replacing human survey participants outright.

  • The paper connects individual-level persona alignment with risks of essentialism and profiling.
  • Persona prompting can risk essentialism and profiling when individuals are categorized using an insufficient set of demographic information.This may flatten representation and caricature individuals.
  • The authors argue that exploring attributes beyond socio-demographics may reduce essentialism risks.
  • Using human participant data raises privacy concerns, especially the collection or disclosure of increased quantities of sensitive information.The authors state that they reuse already collected survey data and use locally hosted models to avoid data leakage into hosted models.
  • The paper situates its work within concerns about labor displacement, environmental harms, malicious use, and non-human-like social-simulation mechanisms.
  • The authors caution against outright replacement of human survey participants while advocating continued methodological evaluation of LLM-based survey applications.They frame evaluation as necessary for informed decisions about when and when not to use these methods.

7 Appendix: Prompts

The appendix documents survey-question paraphrasing, persona-attribute selection, and response-prediction prompts, including a no-persona baseline.

  • 7.1 Paraphrasing of Survey Questions: GPT-5.1 generated the paraphrases, which were manually assessed and judged sufficiently high in quality and suitability.
  • 7.1 Paraphrasing of Survey Questions: Survey questions were paraphrased into concise, self-contained key-value representations of participant attributes and attitudes.
  • 7.1 Paraphrasing of Survey Questions: Batch paraphrasing prompts were appended to the base prompt for all but the first batch.
  • 7.2 Attribute Selection: The appendix documents two LLM-based persona-attribute selection approaches: set selection and scoring.
  • 7.2 Attribute Selection: Set selection asks an LLM to choose the five most important non-target survey variables for predicting a target response.
  • 7.3 Response Prediction: Persona response prediction generates one response per participant from descriptions built with important attributes, using a constrained JSON output format.
  • 7.3 Response Prediction: The no-persona baseline uses a prompt that asks the LLM to predict a respondent’s answer without a described profile.

8 Appendix: Experimental Setup

The appendix describes survey preprocessing, variable annotation, target-question selection, response-variation groupings, and LLM release-date checks for the experiments.

  • 8.1 Data Preparation: Survey codebooks and responses were transformed and standardized into machine-readable formats, with custom parsers used where necessary.
  • 8.1 Data Preparation: Variables were annotated by scale and information type, and direct duplicates were removed before selecting relevant variables.
  • 8.1 Data Preparation: Nominal variables have unordered or categorical response options, whereas ordinal variables have ordered options with defined extremes.
  • 8.1 Data Preparation: Continuous variables use open-ended numeric response fields and mainly represent survey weights or income variants.
  • 8.2 Target Questions: Figures 12 to 15 organize 20 target variables per survey by nominal versus ordinal scale and low versus high human response variation.
  • 8.2 Target Questions: Tables 5 to 8 provide target-question text, human response variation, valid-response counts, and linking variable IDs.
  • 8.2 Target Questions: Table 5 identifies GSS targets by nominal and ordinal scale and by lowest or highest human response variation.
  • 8.3 LLMs: The release timing and data organization suggest that actual leakage into the LLMs did not occur, while leakage was practically impossible for GSS and GGSS.

9 Appendix: Results

Persona-prompted LLMs generally predict responses better for questions with high human response variation, while the no-persona baseline shows the opposite pattern. Attribute-selection performance ordering is largely consistent across model families and surveys, with model size improving set-selection stability.

  • Human response variation: Persona-prompted LLMs perform better on high-variation than low-variation target questions, whereas the no-persona baseline performs better on low-variation questions.This pattern holds across both Llama and Qwen families, though some survey-specific exceptions occur.
  • Human response variation: Statistical attribute-selection baselines perform worse on low-variation questions than on high-variation questions, with data-selected attributes sometimes moving models away from the baseline response.The performance gap is particularly pronounced for statistical baselines.
  • Cross-model and cross-survey consistency: Attribute-selection methods retain the same general performance ordering across model families and surveys, although differences are generally smaller for US-based surveys.The ordering is especially pronounced for high-response-variation questions.
  • Cross-model and cross-survey consistency: The effect of human response variation is mostly survey-consistent, but exceptions include Llama models on GGSS and several Qwen selection methods.For Qwen models, HRV, Set Selection, and Scoring perform better on low-variation questions in some cases.
  • Selection stability: Scoring stability varies by model family and survey: GSS concordance is higher than for other surveys, potentially because GSS has fewer predictor variables and rank variations.Llama scoring is particularly variable across GGSS and WVS_DE, while Qwen shows limited survey-specific differences.
Loading 2609.02526v1…