Sociodemographic Prompting is Not Yet an Effective Approach for Simulating Subjective Judgments with LLMs

1 Introduction

• Whether LLMs show systematic alignment with particular social groups across subjective NLP tasks remains underexplored, despite concerns that biased outputs may inadequately represent diverse experiences.Prior evidence cited in the introduction concerns value-based questions, leaving other subjective tasks in need of investigation.Source: §1 · p. 1
• Social context and identity shape language perception, so the same text may be judged polite or inoffensive by one group and differently by another.The introduction frames faithful adjustment to specific perspectives as steerable pluralism and notes limitations in current steerability.Source: §1 · pp. 1–2
• Sociodemographic prompting has shown potential, but its effectiveness is debated because prompt wording, structure, order, model, and task can affect outcomes.Prior work also suggests that this prompting can exacerbate stereotypes or reduce performance on some tasks.Source: §1 · p. 2
• The study evaluates nine LLMs on demographic differences in politeness and offensiveness, two related but distinct judgments involving interpersonal status and distance versus violations of social norms.Offensiveness is narrower than impoliteness because language can vary in politeness while still being perceived as non-offensive.Source: §1 · p. 2
• LLMs’ predictions align more closely with White participants’ judgments than with those of Asian or Black participants, and sociodemographic prompting risks reinforcing racial and gender biases.The authors report that intrinsic biases persist in these subjective judgment tasks.Source: §1 · p. 2

2 Methods

• POPQUORN supplies 45,000 annotations from a demographically representative U.S. sample; the analysis focuses on men and women and Asian, Black, and White annotators because those groups have sufficient annotations.The study analyzes offensiveness and politeness ratings on a 5-point Likert scale and computes average scores for each identity group.Source: §2 · p. 2
• Overall mean scores are 1.88 for offensiveness and 3.31 for politeness, while Black and Asian annotators have diverging means and higher variance than the overall distribution.Men, women, and White annotators’ scores closely mirror the overall distribution.Source: §2 · p. 2
• The study tests nine open- and closed-source LLMs, including FLAN, Tulu2, GPT, Llama, Mistral, and Qwen models.The evaluated set includes models of different sizes and families.Source: §2 · p. 2
• Prompts ask models to predict politeness and offensiveness scores; preliminary tests check response validity, and robustness checks vary templates and answer-option order.The authors report minor performance differences across templates and option orders.Source: §2 · pp. 2–3

3 Are Model Predictions Closer to Certain Demographic Groups?

• Baseline prediction error is the absolute difference between an identity-free LLM prediction and a specific demographic group’s human rating.Separate linear mixed-effects models compare target-group error with a reference group while controlling for instance-level variation.Source: §3 · p. 3
• A positive regression coefficient means target-group prediction errors are larger than reference-group errors, so predictions are closer to the reference group.The models use instance ID as a random effect; aggregated results appear in Figure 1 and detailed regressions in Appendix D, Table 5.Source: §3 · p. 3
• For offensiveness, baseline prediction errors show no significant gender differences except for FLAN-UL2; for politeness, predictions tend to align more closely with women’s ratings except for GPT-3.5 and Tulu2-7B.The authors also report greater politeness bias in larger Tulu2 and GPT models, suggesting that scaling alone may not reduce bias.Source: §3 · p. 3
• LLMs’ predictions for both politeness and offensiveness are consistently closer to White annotators’ ratings than to Black or Asian annotators’ ratings.The section interprets this pattern as evidence that models align more with some demographic groups than others on subjective perception tasks.Source: §3 · p. 3

• The analysis compares prediction errors with and without demographic tokens, controlling for instance-level variation; points above zero indicate increased errors and points below zero indicate improvement.Source: §4 · p. 3
• Adding an ethnicity token improves GPT-3.5 and FLAN-UL2 predictions of Asian participants’ offensiveness ratings, but effects vary by model and task.Source: §4 · p. 3
• Identity tokens do not consistently improve group-specific ratings and can increase errors, including for Black participants’ politeness ratings with GPT-3.5 and GPT-4.Source: §4 · pp. 3–4, Figure 2 · p. 4

• Across both tasks, all tested LLMs’ answers are closer to White annotators’ labels than to those of other demographic groups.Source: §5 · p. 4
• The authors caution that LLMs may be unsuitable for large-scale annotation without care because their subjective judgments align unevenly across groups.Source: §5 · pp. 4–5
• Newer models’ reduced alignment under identity prompts may partly reflect guardrails that increase refusals on sensitive language, limiting prompting functionality.Source: §5 · p. 5

• Across nine models and both tasks, predictions are closer to White people’s perceptions than to Asian and Black people’s perceptions.Source: §6 · p. 5
• Adding identity tokens does not consistently improve models’ prediction of demographic-specific ratings.Source: §6 · p. 5
• The authors advise caution when using LLMs as judges on subjective tasks.Source: §6 · p. 5

• Research simulating group perspectives should avoid introducing problems such as AI-generated impersonation.Source: §7 · p. 5
• The study uses only men and women labels because POPQUORN contains no additional gender identities, leaving non-binary representation outside its evaluation.Source: §7 · p. 5
• The experiments used inference without fine-tuning or training; all but GPT-3.5 and GPT-4 ran on one A5000 GPU in about six hours per setting.Source: §7 · p. 5

8 Limitations

• The study evaluates only a limited number of models because of computational cost.The authors plan to release scripts for future evaluation of other models.Source: §8 · p. 5
• Sparse labels exclude several identity groups, and the two tasks use Reddit comments and polite emails, limiting generalization to other datasets and settings.The analysis also excludes intersectional identities and compares ordinal model predictions with fractional average annotations.Source: §8 · p. 6
• Some models, particularly GPT3.5 and Tulu2, have relatively high refusal rates, especially for offensiveness ratings and prompts involving Black people.The authors note that guardrails may affect findings when models decline to respond over privacy or ethical concerns.Source: §8 · p. 6

A Data

• Tables 1 and 2 report descriptive statistics for offensiveness and politeness ratings by demographic group.Figure 3 visualizes annotation distributions for both tasks across demographic groups.Source: Table 1 · p. 8, Table 2 · p. 8, A Data · p. 8

B Prompts

• The example prompt asks for an offensiveness rating on a 1-to-5 scale and requests an answer without an explanation.Table 3 provides an example prompt, while Table 4 lists the templates used in the study.Source: B Prompts · p. 8, Table 3 · p. 8, Table 4 · p. 8
• Prompt templates have limited influence on correlations between baseline predictions and aggregated U.S.-population annotations.The robustness check tests four open-source models and uses the overall correlation as a reference point.Source: B Prompts · p. 8

C Experimentation details

• The experiments generally follow developer-recommended defaults and set temperature to 0 for more deterministic generation.This setting is intended to examine the most likely answer for a question and group association.Source: C Experimentation details · p. 8
• The experiments use three NVIDIA RTX A5000 GPUs, and a full run takes around six hours of GPU time.Source: C Experimentation details · p. 8
• The authors checked compliance with the corresponding data and model licenses and used accelerate, datasets, numpy, pandas, seaborn, statsmodels, and transformers.Source: C Experimentation details · p. 8

D Regression Results

• Figure 3 presents annotation distributions from demographic groups for offensiveness and politeness.Source: Figure 3 · p. 9
• Figure 4 indicates that model performance changes little across different prompt templates.Source: Figure 4 · p. 9
• Table 5 reports regression results for gaps between zero-shot predictions and demographic-group labels; Tables 5 and 6 include estimated fixed effects, standard errors, and significance stars.One, two, and three stars mark p-values below 0.05, 0.01, and 0.001, respectively.Source: D Regression Results · p. 8, Table 5 · p. 9

E LLM Guardrails

• Tables 7 and 8 summarize invalid-response percentages across nine models with and without demographic information in prompts.The section notes that implicit guardrails may lead models to refuse potentially harmful queries.Source: E LLM Guardrails · p. 9