Source-linked AI summary
Large Language Models Portray Socially Subordinate Groups as More Homogeneous, Consistent with a Bias Observed in Humans
Messi H. J. Lee, Jacob M. Montgomery, Calvin K. Lai
TL;DR
The paper asks whether LLMs reproduce human-like bias in perceived group homogeneity, a form of bias beyond stereotypical trait association. Using ChatGPT-generated texts about intersectional identities, it finds that racial/ethnic minority groups are portrayed as more homogeneous than White Americans, while gender effects are smaller and group-dependent.
Problem
The paper addresses limited evidence on whether LLMs reproduce perceived-homogeneity bias, which may influence users’ perceptions and attitudes toward social groups.
Source: §1.1 · p. 2Method
The authors analyzed ChatGPT-generated texts about eight intersectional racial/ethnic and gender identities using controlled prompts and homogeneity measures.
Source: §1.3 · p. 2, §2.1 · p. 3Results
ChatGPT portrayed African, Asian, and Hispanic Americans as more homogeneous than White Americans; women were slightly more homogeneous than men, with gender effects consistent in African and Hispanic Americans but not Asian and White Americans.
Source: §4 · pp. 9–10, §6 · p. 11Takeaways & Limitations
The findings identify homogeneity bias as an additional way LLM representations may narrow the portrayed range of human experience across social groups.
Source: §6 · p. 11The study used group labels to signal identities, although identities can be signaled in many different ways.
Source: §5 · p. 10
Abstract
from arXiv · showhide
Large language models (LLMs) are becoming pervasive in everyday life, yet their propensity to reproduce biases inherited from training data remains a pressing concern. Prior investigations into bias in LLMs have focused on the association of social groups with stereotypical attributes. However, this is only one form of human bias such systems may reproduce. We investigate a new form of bias in LLMs that resembles a social psychological phenomenon where socially subordinate groups are perceived as more homogeneous than socially dominant groups. We had ChatGPT, a state-of-the-art LLM, generate texts about intersectional group identities and compared those texts on measures of homogeneity. We consistently found that ChatGPT portrayed African, Asian, and Hispanic Americans as more homogeneous than White Americans, indicating that the model described racial minority groups with a narrower range of human experience. ChatGPT also portrayed women as more homogeneous than men, but these differences were small. Finally, we found that the effect of gender differed across racial/ethnic groups such that the effect of gender was consistent within African and Hispanic Americans but not within Asian and White Americans. We argue that the tendency of LLMs to describe groups as less diverse risks perpetuating stereotypes and discriminatory behavior.
1 INTRODUCTION
The paper examines whether LLMs reproduce human-like biases in perceived group homogeneity beyond stereotypical trait associations. It introduces homogeneity bias in LLM representations and tests ChatGPT’s descriptions of intersectional racial/ethnic and gender groups.
1 INTRODUCTION
- AI systems can reproduce or amplify human biases, including performance disparities and stereotypical trait associations in generated content.Source: §1 · p. 1
1.1 Biases beyond trait association
- Human research links perceived group homogeneity to social status and power, with socially subordinate groups generally viewed as less diverse than dominant groups.Source: §1.1 · p. 2
- Perceived homogeneity is a form of stereotyping associated with prejudice and discrimination, motivating study of whether LLM outputs may influence users’ group perceptions.Source: §1.1 · p. 2
1.2 Homogeneous narratives of marginalized groups in LLMs
- Prior LLM research found marginalized groups represented through homogeneous, essentialized narratives, but this work introduces similarity-based measurement of group representations.Source: §1.2 · p. 2
1.3 This work
- The study tests whether ChatGPT depicts African, Asian, and Hispanic Americans, and women, as more homogeneous than socially dominant comparison groups.Source: §1.3 · pp. 2–3
2 METHOD
The study generated controlled texts about eight intersectional identities and measured their semantic similarity using sentence embeddings. Mixed-effects models tested race/ethnicity, gender, and interaction effects across text formats.
2.1 Data
- The dataset represented African, Asian, Hispanic, and White Americans crossed with men and women to form eight intersectional groups.Source: §2.1 · p. 3
- Using the OpenAI API’s gpt-3.5-turbo model as of 25 July 2023, the study obtained 500 completions for each prompt across four racial/ethnic groups, two genders, and varied 30-word formats.Source: §2.1 · p. 3
2.2 Measure of text homogeneity
- Text homogeneity was measured as standardized pairwise cosine similarity between sentence embeddings generated from each prompt’s texts.Source: §2.2 · pp. 3–4
- The preregistered measure used the second-to-last BERT-base-uncased layer, with BERT, RoBERTa, and Sentence-BERT specifications added for robustness.Source: §2.2 · pp. 3–4
2.3 Testing group differences
- Linear mixed-effects models treated race/ethnicity, gender, and their interactions as fixed effects and text format as a random intercept.Source: §2.3 · pp. 4–5
- Likelihood-ratio tests assessed model fit, while estimated-marginal-means comparisons examined gender differences within each racial/ethnic group.Source: §2.3 · p. 5
3 RESULTS
ChatGPT portrayed racial/ethnic minority groups as more homogeneous than White Americans and women as slightly more homogeneous than men. Gender effects were consistent within African and Hispanic Americans but varied across models within Asian and White Americans.
3.1 Main effect of race/ethnicity
- African, Asian, and Hispanic Americans had standardized cosine similarity values 0.33, 0.31, and 0.18 standard deviations greater than White Americans, respectively.The race/ethnicity effect replicated across all six alternative measurement strategies.Source: §3.1 · p. 5, Figure 1 · p. 6
3.2 Main effect of gender
- Women’s cosine similarity was 0.037 standard deviations greater than men’s, indicating a statistically significant but substantially smaller gender effect than the race/ethnicity effect.The gender effect also replicated across all six alternative measurement strategies.Source: §3.2 · p. 6, Figure 2 · p. 8
3.3 Interaction effect
- Gender effects differed across racial/ethnic groups: African and Hispanic American women consistently showed greater similarity than men, whereas effects varied within Asian and White Americans across models.In the preregistered BERT−2 analysis, White American men and women did not differ significantly, but alternative specifications produced variations.Source: §3.3 · pp. 6, 8
- Interaction models fit better than models without interactions, including for the BERT−2 analysis and several alternative model specifications.Source: §3.3 · pp. 6–8
3.4 Homogeneity bias and topical alignment
- Subordinate racial/ethnic groups were discussed more often in terms of hardship and adversity, but their greater homogeneity persisted both within and outside those topics.Topical alignment partly accounted for the bias, but could not fully explain it because homogeneity differences also occurred within topics.Source: §3.4 · p. 9
- Sentence-embedding homogeneity captured semantic or syntactic similarity that topic models did not, suggesting other representational elements also contribute to the bias.Source: §3.4 · p. 9
4 DISCUSSION
ChatGPT portrayed socially subordinate racial/ethnic groups and women as more homogeneous than socially dominant groups, with gender effects varying across racial/ethnic groups. The discussion considers possible training-data sources while emphasizing that these mechanisms remain speculative.
4 DISCUSSION
- ChatGPT portrayed African, Asian, and Hispanic Americans as more homogeneous than White Americans, while women were more homogeneous than men.The gender difference was smaller than the race/ethnicity difference.Source: §4 · p. 9
- The gender effect was consistent for African and Hispanic Americans but not for Asian and White Americans, underscoring intersectionality in representational bias.Source: §4 · p. 10
4.1 Where might these biases be coming from?
- The proposed training-data explanations remain speculative because the size and opacity of LLM training data make homogeneity bias difficult to confirm directly.Comparing otherwise equivalent models trained on balanced and unbalanced corpora is suggested as a way to examine causality.Source: §4.1 · p. 10
- Training-data selection bias could affect homogeneity because groups represented more frequently may provide models with more varied examples.The authors propose examining how different levels of group representation affect homogeneity in generated text.Source: §4.1 · p. 10
- Stereotypical trait associations in training data could reduce the diversity of words and ideas that LLMs associate with subordinate groups.The authors identify this as another potential source requiring further study.Source: §4.1 · p. 10
5 LIMITATIONS AND FUTURE DIRECTIONS
The study’s limitations concern the short, prompt-dependent texts, the use of group labels, and the limited range of identities examined. Future work should test longer texts, alternative identity signals, additional groups, and links between training-data stereotypes and homogeneity.
Text length
- 30-word ChatGPT generations provided a useful measurement unit, but longer texts remain necessary to assess whether homogeneity bias extends beyond this setting.The authors expect the bias could amplify across sentences and paragraphs, while noting that longer texts may better capture coherence and interconnectedness.Source: §5 · p. 10
Identity signaling
- Using group labels captures only one way identities are signaled, so future studies should examine names and alternative labels.The authors note that performance is heavily influenced by prompts and call for tests of generalizability across identity-signaling methods.Source: §5 · p. 10
Prompt design
- The prompt design may disadvantage comparisons involving typically unmarked terms such as “White American” and “man.”Because these terms nevertheless produced more varied representations, the authors suggest the observed homogeneity bias may be underestimated.Source: §5 · pp. 10–11
Group coverage
- The study prioritized large U.S. population subsets, leaving smaller groups and non-binary identities outside its scope.Including Native or Middle Eastern Americans and people with non-binary gender identities would expand generalizability; smaller groups may show stronger bias.Source: §5 · p. 11
6 CONCLUSION
The study identifies homogeneity bias in LLM representations of socially subordinate groups. Race/ethnicity had a larger effect than gender, and gender effects varied across racial/ethnic groups.
Main conclusion
- LLMs depicted socially subordinate groups as more homogeneous than dominant groups, with race/ethnicity exerting a larger effect than gender.The conclusion frames this as a new bias concerning variability in group representations.Source: §6 · p. 11
Intersectional effects
- Gender effects were consistent within African and Hispanic Americans but inconsistent within Asian and White Americans.Thus, the gender pattern depended on racial/ethnic group membership.Source: §6 · p. 11
Implications
- The observed bias raises concerns about erasing diverse subordinate-group experiences and reinforcing stereotypes.The authors call for strategies that improve fairness, equity, and inclusivity in generated content.Source: §6 · p. 11
A.1 Face validity of the cosine similarity measurements
The face-validity check used randomly selected pairs of ChatGPT-generated stories and linked cosine similarity to semantic overlap. Lower similarity corresponded to diminishing overlap in meaning.
Face validity
- As cosine similarity decreased, the semantic overlap between paired ChatGPT-generated stories about White American men diminished.The examples were arranged in descending cosine similarity in Table A1.Source: §A.1 · p. 14
A.2 Topical alignment alone does not explain homogeneity bias
The authors tested whether shared topics, especially hardship and adversity, explained homogeneity bias. Follow-up analyses showed that topical alignment alone could not fully account for the effect.
Analysis
- Structural topic modeling was used to identify hidden topics and relate topic prevalence to document-level covariates in ChatGPT-generated text.The model was fitted after preprocessing the text.Source: §A.2 · p. 14, §A.2.1 · p. 14
Topical alignment
- Subordinate racial/ethnic groups were discussed more often in terms of hardship and adversity, motivating a test of topical alignment as an explanation.The authors hypothesized that more shared topics could produce higher cosine similarity.Source: §A.2 · p. 14
Follow-up results
- Follow-up studies found homogeneity bias in texts unrelated to hardship and adversity, indicating that topical alignment alone could not fully explain it.The result was supported by analyses of texts about hardship and adversity, texts not about those topics, and an exploratory analysis.Source: §A.2 · p. 14
A.2.1 Hardship and adversity.
Topic modeling identified hardship and adversity as prominent themes in ChatGPT-generated texts, with subordinate racial/ethnic groups discussed in these terms more often than White Americans.
A.2.1 Hardship and adversity.
- Topics 1 and 10 were the corpus’s two most prevalent topics and were associated with hardship and adversity.Source: §A.2.1 · p. 14
- The analysis used structural topic modeling to estimate topic proportions, identify each document’s majority topic, and compare topic distributions across racial/ethnic groups.Texts were preprocessed through stemming, lower-casing, and removal of stopwords, numbers, punctuation, and custom frequent words.Source: §A.2.1 · p. 14
- 41.86%, 26.15%, and 18.65% of African, Asian, and Hispanic American texts, respectively, had Topic 1 or 10 as their majority topic, versus 3.57% for White Americans.These differences were significant for all three subordinate racial/ethnic groups.Source: §A.2.1 · p. 14
- The study also examined whether greater topical alignment around hardship and adversity could explain the observed homogeneity differences.Homogeneity was assessed using cosine similarity between sentence embeddings.Source: §A.2.1 · p. 16
A.2.2 Homogeneity bias in texts not about hardship and adversity.
Follow-up prompts reduced hardship-related content, yet ChatGPT still generated more homogeneous texts for African, Asian, and Hispanic Americans than for White Americans.
A.2.2 Homogeneity bias in texts not about hardship and adversity.
- Texts were encoded as sentence embeddings, pairwise cosine similarity was standardized, and a race/ethnicity mixed-effects model compared groups.Source: §A.2.2 · p. 16
- African, Asian, and Hispanic American texts had higher cosine similarity than White American texts, with differences of 0.15, 0.16, and 0.30 standard deviations, respectively.The race/ethnicity model fit better than a model without race/ethnicity, χ2(3) = 10243.13, p < .001.Source: §A.2.2 · pp. 16–17, Figure A2 · p. 17
A.2.3 Homogeneity bias in texts about cooking.
Across cooking prompts, hardship-focused subsets, topic distributions, and robustness analyses, ChatGPT generally portrayed subordinate racial/ethnic groups as more homogeneous than White Americans, with smaller gender effects and a non-robust interaction.
A.2.3 Homogeneity bias in texts about cooking.
- African, Asian, and Hispanic Americans had higher cosine similarity than White Americans in cooking texts, supporting homogeneity bias outside hardship-related content.All three comparisons were significant: t(19,669) = 34.22, t(19,647) = 26.16, and t(18,484) = 68.93, respectively, ps < .001.Source: §A.2.3 · p. 17, Figure A3 · p. 18
A.2.4 Homogeneity bias in texts about hardship and adversity.
- In Topic 1 texts, African, Asian, and Hispanic Americans each exceeded White Americans in cosine similarity (all three p < .001), and they likewise exceeded White Americans in Topic 10 texts.Topic 1 comparisons had t(26,385.27) = 75.00, t(28,298.17) = 59.04, and t(29,850.68) = 90.09; reported Topic 10 comparisons had t(3,989.36) = 28.31 and t(3,993.86) = 19.05, both p < .001.Source: §A.2.4 · p. 18
- The hardship-subset results indicated that homogeneity bias was not fully attributable to topical alignment and may involve semantic or syntactic alignment captured by sentence embeddings.Source: §A.2.4 · pp. 17, 19
A.3 Distribution of topics
- The supplementary topic analysis used STM majority topics and summed proportions within each group’s top one to five topics.Source: §A.3 · p. 19
- White American texts were not the most topic-dispersed: they ranked second for the top one to three topics and first for the top four and five topics.This contradicted the expectation that White American texts would have the smallest concentration among the groups.Source: §A.3 · p. 19, Table A2 · p. 19
A.5 Robustness to pre-processing steps
- The robustness analysis found that race/ethnicity and gender effects persisted after removing words signaling race/ethnicity and gender during preprocessing.Source: §A.5 · p. 20, §A.5.1 · p. 20
A.5.1 Main effect of race/ethnicity.
- Women’s texts were 0.073 standard deviations more similar than men’s, while race/ethnicity effects remained larger across all three minority groups.The race/ethnicity differences were 0.34, 0.28, and 0.18 standard deviations relative to White Americans.Source: §A.5.1 · p. 20, §A.5.2 · p. 20
A.5.2 Main effect of gender.
- Women exceeded men in cosine similarity within every racial/ethnic group, but the interaction effect was not entirely robust to preprocessing.The interaction model fit better than one without the interaction term, χ2(3) = 6,961.27, p < .001.Source: §A.5.3 · pp. 20–21