The Prompt Makes the Person(a): A Systematic Evaluation of Sociodemographic Persona Prompting for Large Language Models

• Research lacks clear guidance on demographic descriptors and persona viewpoints, leaving the effects of sociodemographic prompt strategies on group representation uncertain.Source: §1 · p. 2
• The evaluation varies role-adoption formats and demographic priming, testing group representation in open-ended self-descriptions and bios and opinion alignment in survey responses.Source: Figure 1 · p. 1, §1 · p. 2
• LLMs more often stereotype nonbinary, Hispanic, and Middle Eastern personas, while interview-style and name-based prompts reduce stereotyping and improve opinion alignment.Source: §1 · p. 2
• OLMo-2-7B achieves the best simulation performance, while larger Llama-3.3-70B is less effective at simulating demographic groups.Source: §1 · p. 2

2 Sociodemographic Persona Prompting

• Sociodemographic persona prompting steers an LLM to align its behavior with a specified sociodemographic group or person.Source: §2 · p. 2
• The authors sampled 47 papers from a curated dataset, extracted their original prompts, and identified recurring prompt patterns.Source: §2 · p. 2
• The framework distinguishes role adoption, or how the persona perspective is induced, from demographic priming, or which descriptors signal identity.Source: §2 · p. 2

2.1 Role Adoption Formats

• The authors hypothesize that role format and whether the model responds as or about a persona can trigger different narratives and portrayals.Source: §2.1 · p. 2
• Direct prompts answer as the assigned persona, third-person prompts answer about a hypothetical person, and interview prompts continue a Q&A from one speaker’s perspective.Source: §2.1 · pp. 2–3

2.2 Demographic Priming

• Demographic priming may invoke different group representations; the study contrasts implicit names and titles with explicit natural-language descriptors and structured category-value descriptions.Source: §2.2 · p. 3
• The study uses last names, titles, and pronouns to imply demographic groups in its name-based prompts.Source: §2.2 · p. 3

3 Experimental Setup

• Prompts vary role adoption and demographic priming, include two alternative phrasings, cover 15 intersectional race/ethnicity and gender groups (eight in the closed-ended task), and are evaluated in English.Source: §3 · p. 3

3.1 Tasks

• Open-ended evaluation uses self-descriptions and social-media bios, sampling 100 responses per persona prompt to compare longer and shorter portrayals.Source: §3.1 · p. 3
• The survey task uses 100 English-language questions from ATP waves 54, 82, and 92, answered as assigned personas and compared with corresponding U.S. demographic opinion distributions.Source: §3.1 · p. 4

3.2 Models

• Five instruction-tuned models span Llama-3, OLMo-2, and Gemma-3 model families and vary in parameter size.Source: §3.2 · p. 4

3.3 Evaluation

• The two task types are evaluated using criteria aligned with their distinct objectives.Source: §3.3 · p. 4

3.3.1 Open-Ended Evaluation

• Open-ended responses are cleaned and assessed for language switching, marked-word counts, group-classifier distinctiveness, and within-group semantic diversity.The marked-word and classifier measures assess distinct aspects of demographic-specific language patterns; greater semantic diversity is treated as desirable.Source: §3.3.1 · pp. 4–5

3.3.2 Closed-Ended Evaluation

• For survey responses, opinion distance is the average Wasserstein distance between model and human answer distributions, with lower values indicating better alignment; group variation is expected to reflect empirically grounded opinion differences.Source: §3.3.2 · p. 5

4.1 Demographic Representativeness

• Nonbinary self-descriptions have more marked words and lower semantic diversity, while male personas are among the most diverse; Middle Eastern and Hispanic personas also fare less favorably.The authors interpret these patterns as more flattened portrayals of marginalized gender and race/ethnicity groups.Source: §4.1 · p. 5, Figure 2 · p. 6
• Non-English self-descriptions and bios are most frequent for Hispanic personas, reaching up to 10.5%; non-English responses are rare for other racial groups, at up to 0.8%.The authors say this indicates that LLMs fall victim to the ‘perpetual foreigner stereotype’ when simulating Hispanic personas.Source: §4.1 · p. 6
• On OpinionsQA, Black personas have the lowest opinion distance from human responses and White personas the highest; two models perform worse than the 0.25 ± 0.002 random baseline.The reported distance is average Wasserstein distance.Source: §4.1 · p. 6, Figure 3 · p. 6

4.2 Strategies for Persona Prompting

• Names and interview-style prompting reduce marked words and increase semantic diversity; interview prompting also improves OpinionsQA alignment for OLMo-2-7B.Names and interview prompting yield near-zero non-English response rates and interview style appears to mitigate language switching associated with explicit demographic priming.Source: §4.2 · p. 6, Figure 3 · p. 6, Figure 4 · p. 7

4.3 Model Comparison

• OLMo-2-7B performs best on marked-word and semantic-diversity measures, while Llama-3.3-70B and Gemma-3-27B perform worst; the two larger models also exceed the random baseline on OpinionsQA distance.Llama-3.3-70B assigns probability above 0.999 to a single answer option in 88% of instances, which the authors say likely contributes to its high Wasserstein distance.Source: Figure 4 · p. 7, §4.3 · pp. 7–8

4.2 Strategies for Persona Prompting

• Marked-word-count disparities between demographic groups decrease significantly with names and interview-style prompting across all models.For OLMo-2-32B and OLMo-2-7B, this pattern generalizes across open-text measures; effects for other models are more mixed.Source: Table 1 · p. 7, §4.2 · p. 7

5 Analysis of Marked Words

• The authors analyze the most frequent marked words across demographic-group and prompt-type combinations, building on categories described by Cheng et al. (2023a).They compare the share of personas in associated groups whose self-descriptions include the category words.Source: §5 · p. 8
• The analysis identifies four marked-word patterns: imposed resilience for Black personas, identity as the primary frame for non-White groups, religiosity linked to Middle Eastern identity, and gender focus for nonbinary personas.Examples include “resilience” and “strength,” “heritage” and “culture,” “faith” and “muslim,” and “gender” and “identity,” respectively.Source: §5 · p. 8
• For the first two categories, names and/or interview-style prompting consistently lower the share of stereotyped persona descriptions across models.The paper reports that the additional stereotype categories follow the same pattern.Source: §5 · p. 8

Personas in LLMs

• LLM personas span psychometric, occupational, and demographic dimensions used to guide behavior and tailor responses; this paper studies demographic personas as synthetic samples of human subpopulations.Personas were first widely used in Interaction Design to represent user archetypes.Source: §6 · p. 8
• Research on inducing personas in LLMs is active, and prior studies use them to create synthetic samples of particular human subpopulations.The passage cites Argyle et al. (2023), Jiang et al. (2023), Shu et al. (2024), and Liu et al. (2024).Source: §6 · p. 8

Demographic Biases in LLMs

• Prior work reports mixed survey-based representativeness results, mismatches with human behavior, and flattened portrayals of some groups, particularly marginalized people.Studies have assessed these issues using surveys, content analysis, free-text self-descriptions, and behavioral simulations.Source: §6 · p. 8

• LLMs often stereotype marginalized groups, especially nonbinary, Hispanic, and Middle Eastern personas.Source: §7 · p. 8
• Table 2 highlights four problematic stereotype patterns: imposed resilience, identity as the primary frame, religiosity conflation, and disproportionate focus on nonbinary gender identity.Source: Table 2 · p. 9
• Interview-style prompts can mitigate stereotypes, enhance diversity, and improve alignment; last-name priming also shows promise in mitigating bias.Source: §7 · p. 9
• Name-based priming carries risks: first names can entrench gender biases, and models may misinterpret last names, posing validity challenges.The authors therefore use titles to signal gender and recommend careful implementation and critical reflection on names.Source: §7 · p. 9
• LLMs switch language during simulation only for Hispanic personas, a new observation the authors identify for future study.Source: §7 · p. 9
• The authors stress documenting and justifying prompt design choices and release their code and datasets.Source: §7 · p. 9

Limitations

• The study covers English and only gender and race, constraining generalizability to other languages and demographic dimensions.Its data-driven prompt set is not exhaustive, and other descriptors such as “girl” or “father” were not systematically compared.Source: Limitations · p. 10
• OpinionsQA uses multiple-choice responses, and answer-option order was neither randomized nor controlled, potentially influencing response patterns and observed alignment.The authors also note limitations of the multiple-choice evaluation paradigm.Source: Limitations · p. 10
• Persona prompts were always placed in user messages; system-prompt placement was not systematically compared, though the authors anticipate it could reduce role violations, particularly for Gemma-3-27b.User-message placement was chosen because it was most common in the reviewed studies.Source: Limitations · p. 10

Ethical Considerations

• The authors do not advocate using sociodemographic steering to replace human survey respondents, and caution that quantitative representativeness measures need qualitative analysis and task context.They also acknowledge limited demographic coverage, while noting that role-adoption findings may adapt to categories such as political leaning or education.Source: Ethical Considerations · p. 10
• Name-based priming uses U.S.-centric race categories and surnames, risks narrow group representation and unintended geographic associations, and should not infer gender from first names.The authors use titles to signal gender to avoid misgendering or excluding nonbinary and gender-diverse people.Source: Ethical Considerations · pp. 10–11

A Review of Persona Prompt Categories

• A review of 47 papers identified role adoption and demographic priming as two recurring dimensions; direct, third-person, and interview formats were common role-adoption categories.The review found 44 papers using natural-language persona prompts and three using alternatives such as fine-tuning or soft prompting.Source: A Review of Persona Prompt Categories · p. 14

B Sociodemographic Prompt Construction

• Prompt construction drew on reviewed templates, using explicit descriptors or last names for race and ethnicity and descriptors for gender and other demographic attributes.The study sourced names from prior work except for Middle Eastern groups, for which it selected common names from Turkey, Iran, and Saudi Arabia; measures aggregated responses by demographic category, with alternative per-name calculations yielding similar overall patterns.Source: B Sociodemographic Prompt Construction · p. 14

C Computational Details

• Inference used Llama-3.1-8B-Instruct and OLMo-2-7B-Instruct on one A100 GPU, while three larger models ran on two H100 GPUs in parallel.The authors used model-creator-recommended parameters and temperature-scaled log probabilities for OpinionsQA answer-option distributions.Source: C Computational Details · pp. 14, 17

D Preprocessing of LLM Responses

• For open-text measures, demographic markers, names, and repeated pronouns were replaced with “[identity]” before assessing stereotypes and flattening.Source: D Preprocessing of LLM Responses · p. 17

D.1 Identification of Role Violations

• Role-violation labels were generated with Qwen2.5-32B and checked on 300 sampled responses; human annotators had mean Cohen’s κ of 0.925, and Qwen classification was 100% accurate.Role violations included refusals and responses identifying as an AI assistant.Source: §D.1 · pp. 17–18

E.1 Statistics on Role Violations

• Interview prompts generally produced more role violations; Llama-3.3-70B, OLMo-2-32B, and OLMo-2-7B had none or very few, up to 2.4%.Gemma-3-27b had notable difficulties for some prompt-type and demographic-group combinations; the authors suspect a conflict with the system prompt’s AI-assistant directive.Source: §E.1 · p. 18

E.2 Social Media Bios

• Social-media-bio results overall aligned with those for self-descriptions.Source: §E.2 · p. 18

E.3 Classification Accuracy

• Classification accuracy exceeded 0.9 across demographic groups and prompt types, suggesting demographic information remained recoverable rather than homogenized.The authors describe this as a possible indication, not a definitive explanation.Source: §E.3 · p. 18

E.4 Disparities Between Demographic Groups

• For OLMo-2-7B and OLMo-2-32B, name priming and interview format significantly reduced group disparities across open-text measures except accuracy; other models and the closed-ended task showed mixed results.The analysis used standard deviations across groups as disparity measures and OLS regressions to assess prompt-type effects.Source: §E.4 · p. 18

E.5 Alignment with OpinionsQA

• OpinionsQA evaluation reports Wasserstein opinion distance and majority-option match; higher majority-match values indicate better alignment with human responses.Source: §E.5 · p. 18

E.6 Analysis of Stereotype Categories

• p < 0.001: interview and name coefficients were significantly negative across all models and stereotype categories, indicating reduced expression of the specified stereotypes in self-descriptions.The regressions used direct role adoption and explicit demographic priming as reference categories.Source: §E.6 · p. 18

F.1 Regression Analysis

• Across bios, Middle-Eastern and Hispanic personas had less favorable outputs; names and interview prompts reduced marked words and increased semantic diversity across models.Source: Figure 8 · p. 19, Figure 9 · p. 19
• For Hispanic personas, explicit and structured demographic priming produced more non-English bios than name-based priming, while interview format mitigated language switching.Source: Figure 10 · p. 20
• Name-based priming significantly improved all open-text measures across models; interview-format role adoption improved marked-word count and accuracy across all models and alignment in four of five models.Source: §F.1 · p. 20
• Across both tasks, values averaged over demographic groups show names and interview-format prompts yield the lowest (best) accuracy for all models; Gemma-3-27b and Llama-3.3-70B are higher, while OLMo-2-32B and OLMo-2-7B are lowest.Source: Figure 12 · p. 21
• Opinion distance was generally lowest for Black respondents’ answer distributions and highest for White respondents’; majority match favored Asian respondents and White females and exceeded the random baseline for most conditions.Source: Figure 13 · p. 22, Figure 14 · p. 23
• Prompt-dimension effects exceeded alternative phrasing effects across measures and models, although phrasing was significant in some model-measure combinations and OLMo2-7B was most sensitive.Source: §F.1 · p. 22

F.2 Validation of Log Probabilities for OpinionsQA

• For OLMo-7B on OpinionsQA, log-probability distributions closely matched 100-generation answer distributions, with Wasserstein distance 0.034 ± 0.025.Source: §F.2 · p. 22

F.1 Regression Analysis

• Names and interview format reduced stereotype-linked self-descriptions, although Gemma-3-27b reached 100% role violations in some interview-format/nonbinary combinations.Source: Figure 7 · p. 19, Figure 15 · p. 23