LLM-as-a-Demographic: Whom Sociodemographic Prompting Helps, and Whom It Hurts

Daniela Occhipinti; Andrea Piergentili; Marco Guerini

Source-linked AI summary

LLM-as-a-Demographic: Whom Sociodemographic Prompting Helps, and Whom It Hurts

Daniela Occhipinti, Andrea Piergentili, Marco Guerini

arXiv:2609.00222v1Aug 31, 2026cs.CL

TL;DR

Subjective LLM judging must account for whose judgments a model reproduces, not only aggregate accuracy. The paper compares predicted and human group-level label distributions across 23 open-weight models and finds that demographic conditioning often benefits already-favored majority groups while harming minority-group alignment. These results motivate caution when using demographic prompting to estimate group judgments.

Problem
Because subjective annotators disagree, it remains unclear whether demographic conditioning moves LLM judgments toward target groups’ distributions rather than stereotypes.
Method
The study compares 23 open-weight LLM judges with human demographic-group label distributions across three subjective tasks and unconditioned, single-attribute, and intersectional profiles.
Results
Unconditioned judges align most with White, college-educated annotators, while conditioning produces asymmetric group-level effects, especially harming minority-group alignment on offensiveness.
Takeaways & Limitations
Demographic prompting should be used cautiously for estimating group judgments because its benefits are smallest for the groups it is intended to represent.
Takeaways & Limitations
The findings are based on three English tasks from one annotator-level corpus and may not transfer to other languages or annotation schemes.

Abstract

from arXiv · show

Large language models (LLMs) are increasingly used as judges for subjective tasks, where annotators disagree and the relevant question is not only how accurate a judge is, but whose judgments it reproduces. Sociodemographic prompting conditions the judge on an annotator's demographic profile to align its judgments with the corresponding group's. We test whether this alignment emerges distributionally, comparing the predicted label distributions of 23 open-weight LLMs on three subjective tasks against those of real annotator groups, under three conditions: no demographic information, single-attribute profiles, and intersectional profiles over gender, age, race, and education. Three findings emerge. First, a judge prompted with no demographics is not perspective-neutral: models best reproduce the judgments of White, college-educated annotators. Second, demographic conditioning is asymmetric: it moves the judge toward majority groups and away from minority groups, most strongly on offensiveness, where intersectional profiles amplify the harm. Third, by comparing base and instruct models we identify instruction-tuning as a possible source of the asymmetry. Demographic conditioning should therefore be used with caution to estimate group judgments: conditioning moves predictions away from the reference distributions of the minority groups the method is often invoked to serve.

1 Introduction

The paper asks whether demographic prompting makes LLM judges reproduce specific groups’ judgment distributions rather than stereotypes. Across subjective evaluation settings, it studies default demographic alignment, conditioning effects, and their equity across groups.

Motivation: Prior work leaves untested whether demographic conditioning moves predictions toward target groups’ label distributions rather than stereotypes.The paper frames this as a distributional evaluation problem for subjective judgments, where annotator disagreement is informative.
Approach: The proposed framework evaluates LLM judges against each demographic group’s judgment distribution instead of an aggregated gold label.It adopts a perspectivist view in which group-level distributions are the evaluation target.
Approach: The study evaluates 23 open-weight LLMs on politeness, intimacy, and offensiveness with unconditioned, single-attribute, and intersectional profiles.Each condition is scored against the matching human group’s label distribution.
Findings: Unconditioned judges tend to align with White, college-educated annotators, indicating that a judge without demographics is not perspective-neutral.The paper treats this default alignment as a central research question rather than assuming demographic neutrality.
Findings: Averaged conditioning effects can conceal opposite group-level changes, with judges moving closer to some groups and farther from minority groups.The paper decomposes effects by group to reveal this asymmetry.

2 Related Work

Related work positions demographic prompting within LLM judging, persona-based generation, demographic simulation, and distributional evaluation. It emphasizes that persona conditioning may activate stereotypes instead of faithfully reproducing group perspectives.

LLM-as-a-Judge: LLM judges evaluate text at scale, but their agreement with human judgments varies across tasks and can be sensitive to response presentation.This motivates examining subjective judgments beyond a single accuracy measure.
Persona-Based Generation: Persona prompting assigns roles or identities to steer model behavior, but the assigned persona may activate stereotypes rather than a group’s genuine perspective.Prior studies associate such conditioning with degraded performance for some identities or increased toxic output.
Demographic Simulation with LLMs: Unconditioned models can align more closely with some demographic groups than others, while demographic conditioning is fragile and sensitive to evaluation setup.This literature motivates testing group-specific alignment directly.
Distributions as Targets: The paper evaluates judges against full demographic-group label distributions rather than a single aggregate label.This distributional target preserves disagreement among annotators as part of the evaluation signal.

3 Methodology

The study evaluates LLM judges against demographic groups’ full human label distributions across three subjective tasks. It uses ordinal-distribution similarity, controlled prompting configurations, and matched references to isolate demographic-conditioning effects.

Data and tasks: The evaluation retains intimacy, offensiveness, and politeness from DEMO, all using five-point ordinal ratings linked to annotator demographics.The dataset maps self-reported gender, age, race, and education profiles to individual annotations, enabling group-specific reference distributions.
Data and tasks: Sparse demographic categories are removed before scoring so both unconditioned and conditioned predictions use references from retained groups.A cell is dropped when any attribute in its conjunction is excluded, because its reference depends on the full conjunction.
Distributional targets: Each text–group pair compares the model distribution p(i,g) over five answer points with the matching human distribution q(i,g).The model distribution is read from answer-token log-probabilities, while the human reference aggregates labels from annotators matching the demographic profile.
Evaluation metrics: Distributional evaluation preserves ordinal structure because EMD weights misplaced probability mass by the distance it travels across the rating scale.The resulting bounded similarity score s ranges from 0 for maximal divergence to 1 for a perfect match and is comparable across tasks and configurations.
Evaluation metrics: The study reports both similarity s and change Δs, with Δs isolating whether demographic conditioning moves predictions toward or away from a group reference.For the same text and group, Δs > 0 indicates movement toward the group and Δs < 0 movement away from it.
Generation and configurations: Models produce full five-point distributions in one forward pass under a shared prompt template, with configurations differing only in profile information and reference group.The study also tests an interview-style profile template, while paired main results use identical wording for conditioned and unconditioned predictions.

4 Results and Discussion

Unconditioned judges systematically align most closely with White and college-educated annotators rather than acting as neutral annotators. Conditioning improves base-model alignment but is uneven for instruction-tuned models, harming minority-group alignment most strongly on offensiveness.

RQ1: Default Profiles of LLM-Judges: Every instruction-tuned judge aligns more closely with White than Black or Asian annotators and with college-educated than less-educated annotators.The average White–Black alignment gap is 0.047 for instruction-tuned models and 0.089 for base models.
RQ2: The Effect of Demographic Conditioning: Conditioning improves alignment for 10 of 11 base models, but helps only five of 12 instruction-tuned models and hurts the rest.The mean conditioning gain for base models rises from +0.014 to +0.032 under mode accuracy, while instruction-tuned models remain near zero.
RQ2: The Effect of Demographic Conditioning: Conditioning gains are essentially unrelated to model scale: log parameter count and ∆s correlate at r = 0.016 across 23 judges.The largest gains occur for base Qwen models, including +0.041 for Qwen 8B and +0.044 for Qwen 14B.
RQ3: Conditioning Helps Majority Groups and Hurts Minority Groups: Instruction-tuned conditioning is positive for White, Man, College degree, and High school groups but negative for Asian, Woman, Graduate degree, and Black groups.The largest group-specific harm is for Black annotators, at −0.033.
RQ3: Conditioning Helps Majority Groups and Hurts Minority Groups: Intersectional effects are non-additive: Man × White benefits by +0.009, while Woman × Black is harmed by −0.036.For Black profiles, the Man effect dilutes harm to −0.019, whereas the Woman effect deepens it to −0.036.
RQ3: Conditioning Helps Majority Groups and Hurts Minority Groups: Minority-group harm concentrates on offensiveness, where alignment falls by −0.074 for Black annotators and −0.035 for Asian annotators and women.Woman × Black shows the largest effect in the study, at −0.087; intimacy shows weaker harm and politeness shows none for minority profiles.
RQ3: Conditioning Helps Majority Groups and Hurts Minority Groups: Conditioning often activates stereotypes rather than reproducing group judgments, such as shifting the Black offensiveness profile +0.080 against a human deviation of +0.052.On politeness, the Asian profile shifts ratings upward despite Asian annotators rating −0.027 below the pooled mean.
RQ3: Conditioning Helps Majority Groups and Hurts Minority Groups: On offensiveness, larger upward conditioning shifts are associated with lower alignment, with r = −0.93 across judges and within-profile r = −0.92.Women provide a concrete case: their offensiveness ratings are near the pooled mean, but the Woman profile shifts instruction-tuned judges upward by +0.039.

5 Conclusions

Across 23 open-weight LLM judges, demographic conditioning benefits groups the models already favor but harms minority groups, especially on offensiveness. Base models respond positively across groups, whereas instruction-tuned models can exaggerate or introduce group differences.

Every model aligns more closely with White than with Black or Asian annotators, and with college-educated than less-educated annotators.
Conditioning instruction-tuned judges appears neutral on average but produces gains for favored groups and losses for minority groups.
Conditioning harms minority-group alignment most on offensiveness, where intersectional profiling sharpens rather than repairs the disparity.
Conditioning improves base-model alignment for every group, while instruction-tuned models respond with expectations about groups rather than activating their perspectives.

Limitations

The study's conclusions are limited to three English tasks drawn from one annotator-level corpus. The authors therefore do not claim that the observed pattern transfers to other languages or annotation schemes.

The evidence covers three English tasks from one annotator-level corpus, limiting claims about transfer to other languages or annotation schemes.

Ethics Statement

The analysis uses coarse demographic categories and existing public annotations to define group reference distributions, while excluding categories too sparse for reliable comparison. It cautions against treating demographic prompting as a substitute for recruiting annotators from the groups of interest.

No new annotations were collected, and no annotator is identifiable in the analysis.
Coarse demographic categories define group reference distributions rather than serving as proxies for individual perspectives.
The results caution against using demographic prompting as a substitute for recruiting annotators from the groups of interest.
Categories with too few annotators for stable group-level distributions are removed before scoring.

B Prompt Templates

The experiments compare unconditioned and profile-conditioned prompts using a shared task template, then test whether conditioning effects depend on an interview-style framing. The alternative framing preserves negative effects for instruction-tuned models while base models remain positive.

Main prompt template: Each task uses a two-message prompt in which the assistant continues an answer prefix by producing the next option letter.
Main prompt template: Conditioned prompts prepend a Your Profile line listing active attributes in the fixed order race, age, gender, education.
Main prompt template: Unconditioned prompts remove the profile line while keeping the question identical; base models concatenate the same messages without a chat template.
Interview-style template: The robustness experiment changes only the lines preceding the unchanged question, items, and options by using an interview-style profile statement.
Robustness comparison: For instruction-tuned models, the self-report template changes Black-group ∆s from −0.027 to −0.035 and lowers average ∆s by 0.020.
Robustness comparison: Base-model groups change by no more than 0.007 under the alternative wording, while conditioning remains positive for all sixteen groups.

C Per-Group Alignment of the Unconditioned Judge

Unconditioned judges show a shared demographic alignment rather than neutrality, with White and college-educated annotators generally closest to model predictions.

White is the closest race group in every base model on every task.
College degree is the closest education band in every base model on every task.
In instruct models, White remains ahead of Black in all 12 models on politeness and intimacy, but in 9 on offensiveness.On offensiveness, the mean White–Black gap drops to +0.023.
Education alignment reverses by task in instruct models: College degree leads on politeness, while High school or below leads in 9 of 12 offensiveness models.The closest education band splits evenly on intimacy.

D Per-Model Conditioning Effects

Conditioning effects vary by model and task, with base models showing larger positive effects and offensiveness producing the only negative instruct mean.

On every task, hollow marker pairs shift right more often than filled pairs, and the base mean exceeds the instruct mean.
Politeness has the smallest conditioning effects in both model pools.
The largest effects occur on offensiveness for instruct models and intimacy for base models.
Offensiveness is the only task with a negative instruct mean, placing the pooled instruct figure below zero.
The tasks differ in effect dispersion rather than direction.

E Rating-Shift Faithfulness for Intersectional Profiles

Intersectional profiles shift predictions mainly toward their race attribute, while distributional scoring requires controls because spread can inflate alignment gaps.

Each gender×race profile shifts toward its race attribute, with magnitudes attenuated toward the gender component.
Reference deviations use annotators matching both attributes, producing wider intervals than single-attribute comparisons.
Spread alone can inflate both human-reference and prediction-side alignment under distributional scoring.
Table 5 reports group deviation from the pooled human mean and conditioning-induced rating shifts for gender×race profiles.Faithful conditioning is represented by ∆≈Human.
Density matching retains one random annotation per group and averages recomputed gaps over 20 draws to equalize reference width.
After density matching, base-pool gaps shrink substantially and sometimes change sign, whereas instruct race gaps remain positive in every model.

F.2 Mode Accuracy

Mode accuracy removes benefits from prediction spread and confirms that base-model conditioning changes selected options, while instruct effects remain near zero.

Instruct-pool gaps largely persist under mode accuracy, remaining positive for race in every instruct model.
Base-pool gaps largely close under mode accuracy, with a few changing sign, indicating that spread explains much of the raw gaps.
Under mode accuracy, base-model conditioning gains double from +0.014 to +0.032 and remain positive in 10 of 11 models.The gain reflects changes in which option the base model selects.
The two controls attenuate base contrasts while leaving every instruct contrast standing, except Gemma 4B and OLMo 7B.
Option tokens carry at least 96.7% of next-token mass in base models and 99.7% in instruction-tuned models.On offensiveness, demographic profiles shift that share by under 10^-4.