PaperlayerView exact revision on arXiv

Source-linked AI summary

Alignment by Stereotyping: How LLMs Sacrifice Individual Distinctiveness for Cultural Adaptation

Qishuai Zhong, Zongmin Li, Siqi Fan, Aixin Sun

arXiv:2609.05993v1cs.CL

TL;DR

The paper asks whether demographic profiles improve value alignment by serving individuals or by substituting group prototypes for individual differences. Across seven models, it tests profile and dialogue conditions and finds that profiles often raise alignment while compressing responses toward group centroids; real dialogue history partially counteracts this pattern.

  • Problem

    It is unclear whether demographic conditioning genuinely tracks individual values or achieves group-level alignment while erasing individual distinctiveness.

    Source: §1 · p. 1

  • Method

    The authors compare no-context, demographic-profile, and dialogue-history conditions across seven models, using WVS values, a no-profile baseline, and permutation tests.

    Source: §1 · p. 1

  • Results

    Profiles improve group-average alignment for 5 of 7 models, but permutation tests identify compression toward group centroids rather than individual modeling as the source of those gains.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Dialogue history partially counteracts stereotyping, supporting individual interaction history over demographic labels as a design principle within the reported scope.

    Source: §6 · p. 9

    WVS is a single-time-point measure of stated preferences rather than a perfect proxy for true individual values, and may understate within-group diversity.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Large language models are increasingly deployed for personalized interaction, and demographic conditioning via user profiles is a widely adopted strategy for cultural adaptation. We ask whether this approach genuinely serves individual users or achieves accuracy by erasing individual distinctiveness. Studying seven models including frontier GPT-5.1 on the World Values Survey, we find that demographic profiles improve value alignment accuracy for most models, but at a systematic cost to individuality. That is, models pull responses toward demographic group centroids rather than preserving individual differences, a behavioral pattern we term alignment by stereotyping. Permutation tests (10,000 permutations, six demographic attributes, seven models) certify that top-performing models compress individuals far above the human baseline; within-family scaling amplifies this tradeoff while degrading intrinsic cultural understanding. Using a synthetic dialogue dataset validated on real human-chatbot conversations from PRISM (Kirk et al., 2024), we further show that distributing demographic signals across conversational turns partially suppresses prototype retrieval compared to compact demographic labels, a finding validated on real conversations via PRISM but requiring replication at larger scale.

1 Introduction

The paper asks whether demographic conditioning serves individuals or improves group-level alignment by sacrificing individual distinctiveness. Across seven models, it finds that profiles often increase alignment by compressing responses toward demographic group centroids, while dialogue history partially reverses this pattern.

1 Introduction

  • Prior value-alignment measures can miss individual errors because high group-average alignment may result from predicting demographic centroids rather than individuals.The paper uses “stereotyping” for applying group prototypes regardless of individual evidence, without implying intentional bias.Source: §1 · p. 1
  • The study asks whether profiles improve group-average value alignment at the cost of distinctiveness, and whether models adapt to individuals or compress them into group prototypes.Source: §1 · p. 1
  • The evaluation compares no-context, explicit-profile, and dialogue-history conditions across seven models, using a no-profile baseline and permutation tests to assess demographic homogenization.Tests cover 10,000 permutations, six attributes, and seven models.Source: §1 · pp. 1–2
  • The authors construct 2,000 synthetic value-eliciting conversations across career and investment advice, with validation against 80 real PRISM conversations.The synthetic dialogues are generated by an agentic pipeline.Source: §1 · p. 2
  • Profiles raise group-average alignment for 5 of 7 models, but permutation tests attribute the gains to compression toward group centroids rather than individual modeling.The paper calls this the alignment-homogenization paradox.Source: §1 · p. 2
  • Dialogue history partially reverses profile-driven stereotyping by distributing demographic signals across conversational turns.The dialogue finding was validated on real PRISM conversations.Source: §1 · p. 2

2 Related Work

Prior cultural-value evaluations measure group-level alignment, leaving individual preservation largely untested. The paper positions its contribution as testing whether demographic conditioning compresses individual responses toward group prototypes.

2 Related Work

  • Prior work finds that LLM outputs respond to demographic cues, but evaluates alignment by comparing model averages with demographic averages rather than testing individual-value preservation.The cited benchmarks include WVS and Hofstede-derived measures.Source: §2 · pp. 2–3
  • Personalization and bias studies leave open whether demographic conditioning erodes within-group diversity or achieves high alignment through stereotyping.The review notes that these works do not quantify this individual-level cost or statistically test the proposed compression pattern.Source: §2 · p. 3
  • The paper extends prior studies by measuring individual value preservation and testing whether explicit profiles compress responses toward group centroids across seven models.It also adds a no-profile baseline.Source: §2 · p. 3

3 Methodology

The methodology evaluates value alignment and individual preservation across profile and dialogue conditions, using WVS-based profiles, multiple complementary measures, and permutation tests. It tests seven models spanning four architectural families.

3 Methodology

  • The study evaluates whether demographic profiles improve group-average alignment at a cost to individual distinctiveness and whether the change reflects prototype compression rather than individuation.Source: §3 · p. 3

3.1 Values Assessment Instrument

  • The instrument samples 1,000 anonymized WVS profiles and 55 ordinal items from 11 value categories, giving each category equal weight.Rankings remained invariant under random item subsampling and leave-one-category-out checks (Spearman ρ = 1.00).Source: §3.1 · p. 3

3.2 Evaluation Conditions

  • The three evaluation conditions are no user context, an explicit demographic profile prepended to the query, and dialogue history replacing the profile label.The dialogue pipeline generates career- and investment-advice conversations using five coordinated roles.Source: §3.2 · pp. 3–4, Figure 2 · p. 4

3.3 Metrics

  • VAA measures per-user Pearson correlation between model and human response vectors; Homog Rate measures how often model responses lie closer to a group centroid than the user's ground truth.The human baseline for homogenization is approximately 50%; the other metrics provide corroborating views of individuality cost.Source: §3.3 · p. 4

3.4 Permutation Test Statistical Validation

  • The permutation procedure shuffles demographic group labels 10,000 times for each model-attribute pair and tests whether observed homogenization exceeds the null distribution.Across 42 pairs, five high-z models remain significant after Bonferroni adjustment (z ≥4.67; α ≈0.001).Source: §3.4 · pp. 4–5

3.5 Models

  • The study evaluates seven models from Llama, Qwen, DeepSeek, and GPT families, including open-weight, closed-source, and reasoning-augmented systems.The set includes GPT-5.1.Source: §3.5 · p. 5

4 Results

Demographic profiles improved group-average value alignment for most models, but the gains accompanied stronger responses toward demographic group prototypes. Dialogue-based personalization reduced homogenization, while within-family scaling increased it and weakened default culture quality.

4.2 RQ2: What Drives the Improvement?

  • Profiles raised group-average VAA for 5 of 7 models (∆= +0.029 to +0.269), while five showed permutation-certified stereotyping across all six attributes.The reported alignment gains were driven by demographic stereotyping rather than individual modeling.Source: §4.1 · p. 5, Table 4 · p. 6, §4.2 · p. 6

4.1 RQ1: Does Demographic Conditioning Improve Value Alignment?

  • NONE VAA ranged from 0.336 for Llama-3.1-70B to 0.599 for Qwen2.5-7B, despite near-identical default responses across models.Inter-model Pearson correlations were 0.59–0.93; Qwen defaults were stronger than Llama defaults.Source: §4.1 · p. 5
  • Models with weaker NONE VAA gained more from profiles; Llama-3.1-70B gained +0.269 and QwQ-32B +0.215 from weak defaults.Their NONE VAA values were 0.336 and 0.400, respectively.Source: §4.1 · p. 5

4.2 RQ2: What Drives the Improvement?

  • The results distinguish self-sufficient stereotypers, demographic-dependent models, a default-sufficient model, and a stochastic model.GPT-5.1, Qwen-72B, and DeepSeek-V3 had the highest homogenization (80–83%); Qwen-7B still homogenized 69.4% despite a VAA decrease of −0.049.Source: §4.2 · p. 6

Within-family scaling decreases default culture

  • Scaling increased homogenization and weakened default culture: Llama 8B→70B rose from 28.0% to 55.3%, and Qwen 7B→72B from 69.4% to 83.2%.Within both families, larger variants gained PROFILE VAA through stronger exploitation of demographic stereotypes, not stronger intrinsic cultural knowledge; more families are needed to establish generality.Source: Within-family scaling decreases… · p. 5, §4.2 · pp. 6–7, Figure 5 · p. 7

4.2 RQ2: What Drives the Improvement?

  • Under PROFILE, CP was negative for all seven models, VR was below 1 for six, and BA exceeded 1 for five; dialogue lowered BA below 1 for five while CP remained negative.QwQ-32B was the VR exception (1.037).Source: §4.2 · p. 7
  • Dialogue homogenization averaged 43.8% (range 20–72%) versus 66.2% under PROFILE (28–83%); five models fell below the 50% human baseline.Dialogue VAA was modestly below PROFILE for most models, with career differences of −0.006 to −0.078 and investment differences of −0.010 to −0.070.Source: Within-family scaling decreases… · p. 6, Table 4 · p. 6, §4.2 · p. 7
  • On 80 real PRISM conversations, three of four tested models showed significant de-homogenization from PROFILE to dialogue, and the homogenization ranking was preserved.Llama-3.1-70B showed a positive but non-significant trend (∆dist = +0.044, 95% CI: [−0.00, +0.10]).Source: §4.2 · p. 7

5 Further Analysis: Converging Evidence for Prototype Retrieval

Further analyses found response distances increasing with demographic age-group separation for GPT-5.1, and dialogue reversal broadly tracking profile stereotyping intensity. The study also applied centroid-distance comparisons to 80 real PRISM conversations.

5 Further Analysis: Converging Evidence for Prototype Retrieval

  • Table 5 applies profile and dialogue centroid-distance measures to 80 real human–chatbot conversations; higher distance indicates greater individuation from group centroids.Its ∆dist represents de-homogenization from profiles to genuine dialogue.Source: Table 5 · p. 8
  • For GPT-5.1, PROFILE response distances increased with age span between groups (Spearman ρ = 0.64), while Llama-3.1-8B showed no such gradient.The authors identify training-data density as an alternative explanation that would require probing experiments beyond this study’s scope.Source: §5 · pp. 7–8
  • Models with profile homogenization above 79% showed the largest dialogue reversals (GPT-5.1: −48 pp; DeepSeek-V3: −40 pp), while Llama-3.1-8B showed the smallest (−9 pp).Permutation significance, distance-scaling gradient, and dialogue reversal magnitude ranked models consistently.Source: §5 · p. 8

6 Discussion

Demographic profiles can improve group-average alignment by compressing responses toward demographic centroids, rather than preserving individual differences. Default culture, scaling, and input format qualify this pattern, while dialogue history partially suppresses prototype retrieval.

6 Discussion

  • GPT-5.1 distances increase with age span (Spearman ρ = 0.64), while Llama-3.1-8B (Homog Rate 28.0%) shows no gradient; human responses provide a baseline.Figure 6 plots pairwise L2 response distances for all 10 age-group pairs under PROFILE conditioning.Source: Figure 6 · p. 8
  • For 5 of 7 models, profiles raise average alignment accuracy by up to 0.27 through demographic compression; the benchmark should also measure individual preservation.The proposed criterion shifts from group-level VAA to individual preservation rate.Source: §6 · p. 8
  • Within the Llama and Qwen families, larger variants achieve higher PROFILE VAA through stronger stereotyping while their default cultures weaken.Lower default quality predicts greater stereotyping reliance for most, but not all, models.Source: §6 · p. 8
  • All seven models have moderately to strongly correlated defaults (r = 0.59–0.93), suggesting a shared cultural baseline with ecosystem-level implications for homogeneity.Source: §6 · p. 9
  • Real conversational history partially offsets demographic stereotyping, with the largest benefit where stereotyping is most severe.In PRISM, demographic keyword density does not predict centroid distance (ρ = 0.007–0.216, all p > 0.05, five models); preliminary value anchors instead raise GPT-5.1 homogenization from 81.4% to 93.3%.Source: §6 · p. 9
  • Accumulated context may reintroduce stereotyping when it converges toward demographic summaries rather than specific episodes.The authors suggest memory-augmented agents accumulate individual interaction history rather than demographic labels.Source: §6 · p. 9

7 Conclusion

The paper finds that demographic profiles improve group-average Value Alignment Accuracy for most tested models, but responses are pulled toward demographic group centroids rather than individuals. Real dialogue history partially counteracts this pattern, suggesting that how demographic signals are presented matters.

7 Conclusion

  • For 5 of 7 models, profiles improve group-average Value Alignment Accuracy, but permutation tests identify controlled response-level stereotyping toward demographic centroids rather than individual modeling.Source: §7 · p. 9
  • Weaker default cultures are associated with greater reliance on demographic shortcuts, while within-family scaling can increase stereotyping precision and breadth.Source: §7 · p. 9
  • Real dialogue history partially counteracts stereotyping across all seven models, suggesting distributed conversational signals suppress prototype retrieval more than compact profile labels.The framework combines a no-profile baseline, permutation-test validation, and cross-dataset replication.Source: §7 · p. 9

Limitations

The evidence is bounded by synthetic dialogue design, an English-only WVS evaluation, narrow domains, limited model coverage, and WVS as an imperfect individual-level measure.

Limitations

  • Synthetic dialogues may miss real conversational diversity, and GPT-5 simulation could partially inflate GPT-5.1 de-homogenization; PRISM validation covers only 80 conversations across two domains.The evaluation also tests generalization beyond career and investment advice only indirectly.Source: Limitations · p. 9
  • English-only WVS questions may disadvantage models with less English pretraining, while untested sensitive domains and seven models limit generalization.The evaluated models reflect coverage as of early 2026; WVS preferences also capture a single time point and may underestimate within-group diversity.Source: Limitations · pp. 9–10

B.2 Human Annotation Study

  • Human annotation supports overall dialogue quality acceptability but has near-chance item-level agreement, limiting dialogue-by-dialogue quality distinctions.The GPT-4.1 evaluation is a consistency check, not an independent quality guarantee; human scores suggest it may overestimate quality in less common contexts.Source: Table 7 · p. 12, Table 9 · p. 13, §B.2 · p. 13

C Cross-Format Consistency Details

  • Cross-format ratios below 1 indicate same-user responses are more similar across PROFILE and DIALOGUE than random pairings, but consistency does not establish individual faithfulness.QwQ-32B has the lowest ratios yet also shows strong demographic homogenization; all models except QwQ-32B have significant tests reported.Source: C Cross-Format Consistency Details · p. 14

E Example of Cross-Format Inconsistency

  • The same Llama-3.1-8B user profile produced option 3 under PROFILE but option 1 with contradictory rationales across two DIALOGUE topics.The example illustrates that signal processing can vary with conversational context.Source: E Example of Cross-Format Inconsistency · pp. 15–16

F PRISM Validation Details

  • In five PRISM-tested models, demographic keyword density did not significantly predict centroid distance under DIALOGUE.The analysis uses Spearman correlation between keyword density and centroid distance.Source: F PRISM Validation Details · p. 16, Table 11 · p. 16

G Dialogue Length: Three Model-Dependent Profiles

Dialogue-length effects differed across three models in the career domain: GPT-5.1 showed a threshold, DeepSeek-V3 a gradient, and Qwen2.5-72B persistent homogenization.

G Dialogue Length: Three Model-Dependent Profiles

  • The test truncates career dialogues at K = 1, 3, or 5 turns for three models and 200 matched users per model; five turns is the design maximum, not demonstrated saturation.The threshold's location and the length at which Qwen2.5-72B might de-homogenize remain untested.Source: G Dialogue Length: Three… · pp. 16–17
  • At K = 5, GPT-5.1 homogenization fell to 23.6%, DeepSeek-V3 to 36.6%, while Qwen2.5-72B remained at 72.9%, above the human baseline.GPT-5.1's transition occurred between three and five turns; DeepSeek-V3 de-homogenized from K = 1, whereas Qwen2.5-72B changed little across tested lengths.Source: G Dialogue Length: Three… · p. 16, Table 13 · p. 17
  • For GPT-5.1, demographic keyword quantity and density were dissociated from the homogenization drop, while individuating user-word counts rose from 25 to 172 across tested dialogue lengths.The analysis cannot separate accumulating individuating content from other quantities that increase with dialogue length; its pattern is consistent with the PRISM keyword-density null.Source: G Dialogue Length: Three… · p. 17

H Anchor Experiment Details

A preliminary GPT-5.1 experiment tested whether adding WVS-derived value anchors alongside a demographic profile changes homogenization.

H Anchor Experiment Details

  • The experiment retained the demographic profile while prepending value statements derived from each user's WVS responses, so it did not isolate anchor-only effects.The anchors were demographically correlated by construction.Source: H Anchor Experiment Details · pp. 17–18
  • For GPT-5.1, homogenization rose from 81.4% with PROFILE alone to 93.3% with profile plus WVS-derived anchors.The authors interpret this as a proof-of-concept risk, not a controlled causal finding, because the anchors may compound demographic signals.Source: H Anchor Experiment Details · p. 18
  • The observation covers only GPT-5.1 and requires investigation across models and anchor types.The authors identify retained profiles and demographic correlation in the anchors as confounds.Source: H Anchor Experiment Details · p. 18

I Corroborating Metrics Under Profile and Dialogue Conditioning

Corroborating metrics show centroid pull across models, while dialogue conditioning reduces group-structured effects; separate checks address alignment tradeoffs and GPT-5.1 run stability.

I Corroborating Metrics Under Profile and Dialogue Conditioning

J.1 Effect Sizes and Run-to-Run Stability

  • Qwen2.5-7B homogenized 69.4% of users despite ∆VAA = −0.049, showing the alignment-homogenization pattern also occurs when average alignment worsens.The reported paired effect size was d = −0.55.Source: §J.1 · p. 18, Table 16 · p. 19
  • GPT-5.1's run-to-run standard deviation extrapolated to n = 1,000 was 0.0007, about 40 times smaller than its ∆VAA of +0.029.Five independent passes had n = 25 users each; the authors conclude decoding noise does not threaten the effect estimates.Source: §J.1 · pp. 18–19

J.2 Item-Selection Robustness

Resampling leaves model rankings unchanged, and alternative same-metric baselines remain below the five permutation-certified models’ profile Homog Rates.

J.2 Item-Selection Robustness

  • Spearman ρ = 1.00 in every resampling case, and VAA and Homogenization Rate rankings matched the full-set ranking.The checks used 100 random 33-item subsets and 11 leave-one-category-out 50-item subsets.Source: §J.2 · p. 19
  • 55.3%–83.2% profile Homog Rates for the five permutation-certified models exceeded all three baseline variants: 50.1%, 50.1%, and 47.2%.The reported ≈50% split-half, median-centroid baseline falls within the 47–50% range of these same-metric variants.Source: §J.2 · p. 19

K AI Writing and Coding Assistance

  • Claude Code assisted with experiment scripts and manuscript preparation, while GPT-5.6 assisted with formatting and presentation review; authors reviewed all AI-assisted outputs.The authors retained responsibility for research design, experimental decisions, analysis, interpretation, and final manuscript content.Source: K AI Writing and Coding Assistance · p. 19

L Data and Code Usage Terms

  • The code is released under the MIT License, while released data artifacts are intended for non-commercial research use only.The synthetic dialogue dataset and evaluation outputs derive from WVS Wave 7 and WorldValueBench, which are available for academic research.Source: L Data and Code Usage Terms · pp. 19–20

Related papers

This paper cites

Loading 2609.05993v1…