Source-linked AI summary
Marked Personas: Using Natural Language Prompts to Measure Stereotypes in Language Models
Myra Cheng, Esin Durmus, Dan Jurafsky
TL;DR
Existing methods provide limited coverage of the prevalence and nuances of stereotypes in LLM outputs, especially for intersectional demographic groups and patterns missed by lexicons. Marked Personas generates personas for marked groups and unmarked defaults, then identifies distinguishing words; GPT-3.5 and GPT-4 produce more stereotypes than human-written portrayals, while the method reveals additional harmful patterns.
Problem
Existing stereotype measures rely on manually constructed data and can miss group-specific patterns, including othering and seemingly harmless tropes.
Method
Marked Personas prompts LLMs to generate personas for target and unmarked demographic groups, then uses distinguishing words to measure stereotypes without lexicons or data labeling.
Results
GPT-3.5 and GPT-4 personas contain more stereotypes than human-written portrayals using the same prompts, and the method identifies harmful patterns overlooked by existing measures.
Takeaways & Limitations
Marked Personas captures stereotypes and harmful patterns across singular and intersectional demographic groups, including positive-seeming stereotypes missed by lexicon-based measures.
Takeaways & Limitations
The study is limited in scope to models available through the OpenAI API and requires defining marked and unmarked identities a priori.
Abstract
from arXiv · showhide
To recognize and mitigate harms from large language models (LLMs), we need to understand the prevalence and nuances of stereotypes in LLM outputs. Toward this end, we present Marked Personas, a prompt-based method to measure stereotypes in LLMs for intersectional demographic groups without any lexicon or data labeling. Grounded in the sociolinguistic concept of markedness (which characterizes explicitly linguistically marked categories versus unmarked defaults), our proposed method is twofold: 1) prompting an LLM to generate personas, i.e., natural language descriptions, of the target demographic group alongside personas of unmarked, default groups; 2) identifying the words that significantly distinguish personas of the target group from corresponding unmarked ones. We find that the portrayals generated by GPT-3.5 and GPT-4 contain higher rates of racial stereotypes than human-written portrayals using the same prompts. The words distinguishing personas of marked (non-white, non-male) groups reflect patterns of othering and exoticizing these demographics. An intersectional lens further reveals tropes that dominate portrayals of marginalized groups, such as tropicalism and the hypersexualization of minoritized women. These representational harms have concerning implications for downstream applications like story generation.
1 Introduction
Marked Personas is an unsupervised, lexicon-free framework that generates demographic personas and identifies words distinguishing marked groups from unmarked defaults. It finds more stereotypes in GPT-3.5 and GPT-4 personas than in human-written portrayals and surfaces harmful patterns missed by existing measures.
- Existing stereotype measures rely on manually constructed unnatural templates or human-written sentences and may miss group-specific othering and seemingly harmless tropes.
- Marked Personas prompts an LLM to generate natural-language personas for target demographic groups alongside personas for corresponding unmarked defaults.The framework requires specifying the demographic group of interest and its unmarked default groups.
- Marked Words identifies statistically significant words distinguishing personas of marked groups from unmarked groups, including at intersections of demographic identities.The method uses differences across personas to surface stereotypes and essentializing narratives without requiring a handcrafted lexicon or labeled corpus.
- The method identifies harmful patterns overlooked by state-of-the-art stereotype measures, including positive-seeming stereotypes and essentializing narratives.The framework also supports analysis of stereotypes and other harmful patterns in downstream applications such as LLM-generated stories.
- GPT-3.5 and GPT-4 personas contain more stereotypes than human-written texts generated with the same prompts.
2 Background and Related Work
The paper grounds its approach in markedness, defines stereotypes as socially hierarchical group associations, and situates the work among existing methods for measuring bias and intersectional harms in language models.
- Markedness: Markedness treats dominant groups as linguistically unmarked defaults and non-dominant groups as explicitly marked by additional explanation or modification.The concept extends from grammatical features to social categories such as gender and race.
- Stereotypes: Stereotypes are traits broadly associated with demographic groups in ways that reify existing social hierarchies.
- Bias Measurement: Existing methods measure social bias and stereotypes in large language models, but a comparison finds they satisfy at most 3 of 5 desiderata.
- Intersectionality: Intersectionality frames sexism and racism as interrelated systems of oppression that can produce compounding bias and unique harms for multifaceted identities.The paper builds on prior work studying intersectional biases in word embeddings and language models.
3 Methods
The methods generate natural-language personas for intersectional demographic groups, then identify words distinguishing marked groups from unmarked defaults to measure stereotypes without handcrafted lexicons or labeled data.
- 3.1 Personas: Generating Intersectional Portrayals: Zero-shot natural-language prompts produce personas portraying individuals belonging to specified demographic or intersectional groups.The study also uses human-written personas generated with the same prompt for comparison.
- 3.2 Marked Words: Lexicon-Free Stereotype Measurement: Marked Words identifies statistically distinguishing words between marked-group personas and corresponding unmarked-group personas.The method uses weighted log-odds ratios with an informative Dirichlet prior and z-scores for significance.
- 3.2 Marked Words: Lexicon-Free Stereotype Measurement: The framework requires specifying target groups and corresponding defaults rather than constructing an extensive dataset, lexicon, or labeling scheme.It is intended to cover many dimensions and intersections of identity.
- 3.2 Marked Words: Lexicon-Free Stereotype Measurement: Markedness treats socially or linguistically default categories as unmarked and compares other groups against those defaults.For English web-trained LLMs, the study uses White as the racial default and man as the gender default, while noting that category status is context-dependent.
- 3.2.1 Robustness Checks: Other Measures: Robustness checks compare Marked Words with Jensen-Shannon Divergence and one-vs-all SVM classification.The alternative methods identify distinguishing words but lack Marked Words’ theoretically informed measure of statistical significance.
4 Experiments
The experiments evaluate GPT-4 and GPT-3.5 persona generations across racial, ethnic, gender, and intersectional groups, comparing generated outputs with human-written personas and testing robustness across word-identification methods.
- 4 Experiments: The study reports main-text results for GPT-4 and GPT-3.5, while other models have limitations including incoherent persona descriptions.GPT-3.5 generations were produced in December 2022, whereas the other generations were produced in May 2023.
- 4 Experiments: The evaluation covers 5 racial or ethnic groups, 3 genders, and 15 gender-by-race or ethnic groups, including nonbinary gender.The intersectional groups combine each race or ethnicity with man, woman, or nonbinary person.
- 4 Experiments: 2700 personas are generated, with 90 personas for each of 15 intersectional groups across both models.The personas use 15 samples for each of six prompts, and the generated outputs are compared with human-written personas.
- 4 Experiments: Marked Words identifies statistically significant distinguishing words across demographic axes, with JSD and SVM used as robustness checks.The SVM evaluation uses an 80% training and 20% test split stratified by demographic group.
- 4 Experiments: Figure 1 compares the average percentages of words from Black and White stereotype lexicons in generated and human-written personas.Error bars denote standard error; generated portrayals contain more stereotypes than human-written ones.
5 Persona Evaluation: Comparison to Human-written Personas
Generated GPT-3.5 and GPT-4 personas contain more lexicon-measured racial stereotypes than human-written personas, but lexicons miss broader and potentially harmful representational patterns.
- Lexicon-Based Comparison: Generated personas contain more stereotype-lexicon words than human-written personas using the same prompts.For GPT-4, Black stereotypes are more prevalent in Black personas and White stereotypes are more prevalent in White personas.
- Lexicon-Based Comparison: A GPT-4 Black persona describes a tall, athletic Black man, illustrating words included in the Black stereotype lexicon.The example uses “tall” and “athletic” as lexicon-matched attributes.
- Shortcomings of Lexicons: Human-written personas contain a broader distribution of stereotype words, whereas generated personas mainly contain words that appear positive in sentiment.This distribution limits what the lexicon comparison reveals about generated portrayals.
- Shortcomings of Lexicons: Lexicons may miss harmful, one-dimensional narratives when broader phrases dominate marked-group personas but are absent from unmarked-group personas.This motivates the unsupervised Marked Personas framework.
- Shortcomings of Lexicons: GPT-3.5 generated White personas contain higher rates of Black stereotype words than generated Black personas.Words such as “tall” and “athletic” occur in both generated Black and White personas, producing an inconsistent lexicon-based pattern.
6 Analyzing Marked Words: Pernicious Positive Portrayals
Marked Words reveals that generated personas often encode harmful stereotypes through positive-seeming language, especially in portrayals of marginalized and intersectional groups. These patterns include essentialism, othering, racialized appearance, tropicalism, religious conflation, and resilience narratives.
- Sentiment and Positive Stereotyping: Positive-sentiment words can still encode harmful stereotypes and essentializing narratives.The analysis emphasizes that replacing negative language with positive language does not remove harmful social context.
- The Myth of Resilience: Intersectional personas surface distinctive stereotypes, including the “strong, resilient Black woman” archetype.Strength and resilience are significantly associated with non-white personas, especially Black women, and Figure 3 reports similar concentration patterns for resilient and resilience.
- Appearance: Appearance terms for minority groups can objectify and exoticize them, including hypersexualized portrayals of Asian women.Asian women are associated with almondshaped, petite, and smooth, while Latina women are linked to tropicalism through words such as vibrant, curvaceous, rhythm, and curves.
- Markedness, Essentialism and Othering: Marked groups are associated with more stereotyped and less everyday descriptions than unmarked White or male groups.Unmarked personas include neutral terms such as good, friendly, and casually, while marked personas reproduce specific archetypes.
- Markedness, Essentialism and Othering: Middle-Eastern personas disproportionately connect identity with religion, reproducing a criticized conflation of Middle-Eastern identity, Arab identity, and Muslim identity.The paper links these patterns to dehumanizing and demonizing representations of Middle-Eastern people.
- Limitations of Anti-stereotyping: Apparently anti-stereotypical words can remain essentializing when they are mentioned only for groups historically denied the corresponding trait.Words such as powerful and independent appear for marginalized groups but not unmarked individuals, potentially reinforcing demographic-based narratives.
7 Downstream Applications: Stories
The paper applies Marked Words to LLM-generated stories to examine whether downstream creative applications reproduce markedness and stereotypes beyond lexicon-based measures. The stories contain essentializing narratives and group-specific stereotypes, including martial arts for Asians and resilience narratives.
- 7 Downstream Applications: Stories: Marked Words applied to generated stories identifies stereotypes and essentializing narratives beyond explicit descriptor words.For unmarked groups, significant words besides explicit descriptors are neutral terms such as town and shop.
- 7 Downstream Applications: Stories: Stories about Asians contain martial arts as a significant group-associated term, illustrating a stereotype that is not overtly negative but still contributes to representational harm.The analysis also identifies the myth of resilience among the patterns discussed in generated stories.
8 Recommendations
The paper recommends addressing positive stereotypes and analyzing identity intersections when measuring or mitigating representational harms. It also frames LLM outputs as containing markedness and othering despite equivalent group labels in prompts.
- 8 Recommendations: Equivalent demographic labels in prompts can still yield generations containing markedness and othering.The authors describe this as revealing bias in LLMs’ collective views of society.
- Addressing Positive Stereotypes and Essentializing Narratives: Positive sentiment is insufficient for evaluating stereotypes because seemingly positive language can perpetuate harmful narratives and discrimination.The paper recommends examining perniciously positive words, essentializing narratives, and other contextual effects rather than simply replacing negative language.
- An Intersectional Lens: Intersectional analysis should account for distinctive stereotypes and unique power differentials associated with combinations of identity axes.The paper reports that intersectional personas contain distinctive stereotypes not captured by examining individual identity axes alone.
- 8 Recommendations: The sources do not establish whether positive stereotypes arise from bias mitigation, training data, or other model components.The paper states that OpenAI does not release its bias mitigation techniques, leaving these contributors unclear.
9 Limitations
The study’s conclusions are bounded by its model, cultural, linguistic, demographic, and methodological scope. It evaluates OpenAI API models, focuses on American stereotypes in English, covers selected identity groups, and requires markedness assignments in advance.
- 9 Limitations: The study evaluates only models available through the OpenAI API and is not a complete systematic probing of all stereotypes in open-ended outputs.The authors characterize the study as offering insight into patterns rather than exhaustive coverage.
- 9 Limitations: The analysis draws on American stereotypes and English, although stereotypes vary across cultures.The paper also notes that many demographic categories and identity markers remain unexplored beyond five race/ethnicity and three gender groups.
- 9 Limitations: Marked Personas currently requires defining marked and unmarked identities a priori rather than discovering the default class unsupervised.Personas are generated with explicit group labels before marked and unmarked groups are compared.
- 9 Limitations: Studying harms by demographic group may reify socially constructed categories, while focusing on OpenAI models may contribute to their dominance and widespread use.These are identified as potential risks of the paper itself.
- 9 Limitations: Marked Words and JSD are not interchangeable in general settings because they use different prior distributions.Marked Words uses other texts in the dataset as its prior, whereas JSD uses only the texts being compared.
- 9 Limitations: Neutral prompts were selected instead of sentiment-modified prompts because negative prompting raises ethical concerns and changes the distribution of top words.The paper treats harmful correlations under neutral prompts and nonnegative sentiments as a distinct dimension for analysis.
D.1 Results for GPT-4
The appendix reports stereotype comparisons for GPT-4 and contrasts stereotype prevalence across two text-davinci versions, finding broader prevalence in text-davinci-002 outputs.
- The full list of GPT-4 top words appears in Tables A2, A3, and A4.
- text-davinci-002 outputs broadly contain more stereotypes than text-davinci-003 outputs.The comparison uses the same parameters and prompts and treats text-davinci-003 as a more recent, conservative GPT-3.5 estimate.
D.3 Results for ChatGPT
ChatGPT sometimes refuses or inconsistently generates requested personas, while its stereotype patterns are analyzed alongside results for other models and demographic groups.
- ChatGPT is inconsistent at generating the desired personas for some prompts, with later prompts often producing refusal-like outputs.Across six prompts, the percentages of outputs containing “language model” were 0%, 0%, 77%, 67%, 100%, and 100%.
- text-davinci-003, text-davinci-002, ChatGPT, and GPT-4 are the only listed models that generate coherent single-person personas in zero-shot prompting.OPT, BLOOM, and smaller GPT-3.5 models cannot produce such coherent descriptions in that setting.
- ChatGPT outputs contain more stereotypes than human-written portrayals, as shown by comparisons using Black and White stereotype lexicons.The generated white personas also have higher rates of Black stereotypical words than generated black personas.
- ChatGPT’s persona stereotype analysis includes lexicon rates, persona-level occurrence rates, resilience terms, and statistically significant distinguishing words.The appendix provides results for race/ethnic, gender, intersectional, and nonbinary groups, along with prompts, examples, stories, and story-level words.
- The appendix notes that prompts in Table A9 avoid wording that arguably encourages stereotypes in the human-portrayal comparison.The listed materials include human-written prompts, generated persona examples, story prompts, and story examples.