PaperlayerView exact revision on arXiv

Source-linked AI summary

First-Person Fairness in Chatbots

Tyna Eloundou, Alex Beutel, David G. Robinson, Keren Gu-Lemberg, Anna-Luisa Brakman, Pamela Mishkin, Meghan Shah, Johannes Heidecke, Lilian Weng, Adam Tauman Kalai

arXiv:2410.19803v2cs.CYcs.AIcs.CL

TL;DR

Chatbot fairness research must address open-ended, user-driven interactions rather than only institutional decisions. This paper introduces a scalable counterfactual LMRA methodology for measuring first-person name bias, finding systematic demographic differences and evidence that post-training RL reduces some biases.

  • Problem

    Open-ended chatbot uses and diverse user-driven scenarios require fairness methods beyond institutional decision-making and classification-based measures.

    Source: §1 · p. 1, §1.1 · p. 2

  • Method

    The paper varies user names counterfactually and uses an LMRA to quantify harmful stereotypes and describe demographic response differences across chatbot interactions.

    Source: §1 · p. 2, §1.2 · pp. 5–6

  • Results

    The methodology identifies subtle systematic biases across models, tasks, and demographic groups, while post-training RL is associated with biases roughly 3–12 times smaller than before RL.

    Source: §1.2 · p. 6, §7 · p. 25

  • Takeaways & Limitations

    Systematic public bias evaluation can support shared understanding and improvement across stages of the machine-learning and development pipeline.

    Source: §7 · p. 25

    Name counterfactuals do not detect biases arising from group differences in writing styles or topics, and the study covers only binary gender, four races, largely U.S.-based name statistics, and text chats.

    Source: §6 · pp. 24–25

Abstract

from arXiv · show

Evaluating chatbot fairness is crucial given their rapid proliferation, yet typical chatbot tasks (e.g., resume writing, entertainment) diverge from the institutional decision-making tasks (e.g., resume screening) which have traditionally been central to discussion of algorithmic fairness. The open-ended nature and diverse use-cases of chatbots necessitate novel methods for bias assessment. This paper addresses these challenges by introducing a scalable counterfactual approach to evaluate "first-person fairness," meaning fairness toward chatbot users based on demographic characteristics. Our method employs a Language Model as a Research Assistant (LMRA) to yield quantitative measures of harmful stereotypes and qualitative analyses of demographic differences in chatbot responses. We apply this approach to assess biases in six of our language models across millions of interactions, covering sixty-six tasks in nine domains and spanning two genders and four races. Independent human annotations corroborate the LMRA-generated bias evaluations. This study represents the first large-scale fairness evaluation based on real-world chat data. We highlight that post-training reinforcement learning techniques significantly mitigate these biases. This evaluation provides a practical methodology for ongoing bias monitoring and mitigation.

1 Introduction

This paper introduces a scalable counterfactual approach to measuring first-person fairness in open-ended chatbot interactions, examining user-name-associated demographic bias across genders and races in real-world chats. It situates this approach within prior language-model bias evaluation methods and outlines its methodology, experiments, mitigation resources, limitations, and appendices.

1.3 Related work

  • The work addresses a gap between chatbot use and traditional third-person fairness research, which centers on institutional decisions such as hiring, lending, and sentencing.Prior research reports mixed demographic bias findings across decision-making, linguistic, question-answering, and other tasks, motivating fairness evaluation for everyday user-driven chatbot scenarios.Source: §1 · p. 1, §1.1 · pp. 2–3, §1.3.1 · p. 7

1.2 Summary of methods and results

  • The study evaluates first-person fairness toward chatbot users by varying names associated with two genders and four races across 66 tasks, nine domains, six language models, and millions of chats.The approach is methodologically replicable for name-sensitive models and supports monitoring deployed systems, although the private data prevent direct reproduction.Source: §1 · p. 2, §1.2 · p. 6

1 Introduction

  • The LMRA compares hundreds of thousands of response pairs, identifies harmful stereotypes and aggregate axes of difference, and generates concise task-level explanations while reducing human exposure to non-public chats.Human ratings closely match LMRA gender-bias ratings but align less strongly for racial bias and feature labels; axes of difference require more computation and are not entirely human-consistent.Source: §1 · p. 2

1.1 First-person fairness and user name bias

  • Counterfactual name analysis compares response distributions for the same first user message under different names, accounting for open-ended and nondeterministic chatbot outputs.The method measures quality, harmful stereotypes, and general differences, but cannot capture all group differences or determine real-world effects.Source: §1.1 · pp. 2–4

1.2 Summary of methods and results

  • Gender analyses find no statistically significant quality differences, task-dependent harmful stereotypes, and strong LMRA–human agreement, whereas racial stereotypes are generally lower, quality differences remain nonsignificant, and human agreement is weaker.The largest harmful gender stereotypes occur for White-sounding names and the smallest for Asian-sounding names; racial stereotype ratings are slightly higher in travel and can arise in restaurant recommendations.Source: §1.2 · pp. 5–6
  • Post-training models show biases roughly 3–12 times smaller than their pre-reinforcement-learning versions across four models and tasks, indicating that reinforcement learning reduces certain biases.The evaluation also finds harmful gender stereotypes below 0.1% overall, but above 2% for ChatGPT-3.5-turbo on “write a story,” while other models remain below 1% across tasks.Source: §1.2 · pp. 5–6

1.3.1 Bias by task

  • 1.3.1 Bias by task: Prior studies examine demographic differences using user profiles, templated completions, sentence pairs, self-descriptions, and cover-letter generation.Some work reports less accurate responses for users with lower English proficiency or education status, while Morehouse et al. (2024) find no significant difference in cover-letter quality.Source: §1.3.1 · pp. 7–8

1.3.2 Other related work

  • 1.3.2 Other related work: Related LLM methods identify chat domains and tasks, analyze commercial-chatbot usage, evaluate outputs, enumerate distributional differences, and interpret model behavior, but differ in bias focus, data sources, or task breadth.The paper also connects its scope to LLM risk taxonomies, broader demographic and political biases, proposed mitigations, and evidence that RL reduces unsafe outputs (Achiam et al., 2023).Source: §1.3.2 · p. 8
  • 1.3.2 Other related work: The paper’s organization covers chatbot name storage, methodology, experiments, API-based profile simulation, limitations, conclusions, and appendices on tasks, disparities, prompts, names, stereotypes, and racial bias.The appendices include a hierarchy of nine domains and sixty-six chat tasks, quality disparities, comparisons with decision-making prompts, and crowd-worker versus LMRA stereotype ratings.Source: §1.3.2 · p. 8

2 Name-sensitive chatbots

Chatbots can access user names through mechanisms such as Memory and Custom Instructions, enabling name-based fairness evaluations. These mechanisms produce highly correlated harmful gender-bias measurements while Custom Instructions also support arbitrary user profiles.

  • ChatGPT can access names through Memory or through profiles supplied in Custom Instructions.Names may be stated explicitly or implicitly, and users can remove memories or disable the feature.Source: §2 · p. 9
  • Harmful gender-bias measurements from Memory and Custom Instructions are highly correlated, with r=0.97.Custom Instructions provide greater flexibility because profiles can indicate attributes such as religion, sexual orientation, age, or disability.Source: §2 · p. 9
  • Custom Instructions let researchers simulate ChatGPT behavior using arbitrary user profiles for future bias research.Source: §2 · p. 9

3 Methodology

The methodology evaluates first-person fairness by generating counterfactual responses for demographic name groups, measuring harmful stereotypes, and enumerating interpretable response differences across chatbot tasks. It combines LMRA-based analysis with order-bias controls, stochasticity-aware metrics, and statistical feature selection.

3 Methodology

  • Counterfactual responses are generated for paired demographic name groups from independently sampled prompts, enabling pairwise fairness analysis of stochastic chatbot behavior.The framework supports gender, racial, and intersectional group comparisons using finite, disjoint name sets.Source: §3 · p. 9

3.1 Identifying domains and tasks

  • Nine manually selected domains and their task typology are produced by combining LMRA classification with programmatic frequency analysis and refinement.The categories prioritize interpretability, task-specific bias coverage, and common use cases while balancing specificity against coverage.Source: §3.1 · pp. 9–10

3.2 Quality differences

  • Quality differences are evaluated with LMRA ratings across nine response criteria, alongside refusal-rate comparisons between demographic groups.The criteria include quality, relevance, accuracy, completeness, clarity, conciseness, tone and politeness, engagement, and ethical considerations.Source: §3.2 · p. 10

3.3 Harmful-stereotype pairs

  • The harmful-stereotype metric subtracts reverse-direction stereotype rates from forward rates, reducing spurious bias caused by random response variation.Identical response distributions yield a net harmfulness rating of zero even when individual pairs can appear stereotypical; anti-stereotypes may produce negative values.Source: §3.3 · p. 11
  • Each response pair is evaluated in both orders, while hiding group assignments helps reduce LMRA over-sensitivity and order bias.The method also accounts for randomness from name selection, language-model sampling, and LMRA or human rating variability; gender- and race-implying prompts or responses are filtered.Source: Figure 3 · p. 10, §3.3 · pp. 11–12
  • Human ratings show a strong correlation with LMRA harmfulness ratings for gender stereotypes, supporting the method’s consistency with independent crowd judgments.The comparison uses a privacy-preserving subset of public chats and reports a strong, nearly monotonic relationship across rater genders.Source: §3.3 · p. 12

3.4 Bias Enumeration Algorithm

  • The Bias Enumeration Algorithm identifies interpretable demographic response differences by brainstorming, consolidating, labeling, and statistically selecting candidate bias features.Feature generation uses sampled prompt-response pairs and general, concise descriptions; labeling is order-invariant, and Bonferroni correction controls family-wise error before final selection.Source: §3.4 · pp. 12–14

4 Results

Across six language models, real-chat evaluations found no significant demographic differences in response quality, but detected task-dependent harmful stereotypes and systematic response differences. LMRA gender-bias ratings aligned strongly with human judgments, while post-training substantially reduced harmful gender stereotypes.

4.1 Response Quality Comparison

  • No statistically significant gender or race differences were detected in GPT-4o-mini response quality on random English chats.Source: §4.1 · p. 15

4.2 Harmful stereotype results

  • Below 0.1% harmful gender stereotypes appeared on random English chats for each model, while open-ended tasks such as write a story produced the highest rates and GPT-3.5-turbo exceeded 2% there.Specific-scenario rates were below 1% for models other than GPT-3.5-turbo.Source: §4.2 · p. 15, Figure 5 · p. 16
  • Gender stereotype ratings correlated 0.97 across tasks between reverse and forward measures, while reverse stereotypes were 0.096 as large by linear regression.The 95% confidence interval for the regression estimate was 0.091–0.102.Source: §4.2 · p. 15, Figure 7 · p. 17
  • Memory and Custom Instructions produced highly correlated harmful-stereotype ratings, but Memory rates were higher, with correlation 0.94 and regression slope 2.15.The slope’s 95% confidence interval was 1.98–2.32.Source: §4.2 · p. 15, Figure 8 · p. 18

4.3 Human correlation with LMRA results.

  • LMRA gender-bias ratings showed a strong, monotonic relationship with diverse crowd ratings and aligned more closely with human judgments than race or feature ratings.Ratings from other LMRA families did not show substantially higher agreement with human ratings.Source: §4.3 · pp. 18, 20, Table 2 · p. 20

4.4 Axes of difference

  • The LMRA identified aggregate demographic axes such as simpler language, but its feature sensitivity was not strongly human-aligned, so these findings are proof of concept rather than definitive conclusions.Large datasets can make differences below 1% statistically significant.Source: §4.4 · p. 21

4.5 Effects of post-training

  • Post-training significantly reduced harmful gender stereotypes in every evaluated task, with the pre-RL versus final-model comparison having a best-fit slope of 0.21.The slope’s 95% confidence interval was 0.17–0.24.Source: §4.5 · p. 23, Figure 10 · p. 24

5 Methodological reproducibility

The study describes the prompt and system-message ordering needed to simulate chatbot interactions, while noting that privacy prevents full reproducibility. It documents Custom Instructions as an optional user-controlled input and explains that examples of such prompts improve research accuracy.

5 Methodological reproducibility

  • User prompts cannot be shared for privacy reasons, so the results are not fully reproducible.The paper instead explains how to simulate experiments through API access, although Memory cannot be perfectly simulated there.Source: §5 · p. 24
  • The documented message order is model-specific system message, Custom Instruction system message, then the user prompt.Source: §5 · p. 24
  • Custom Instructions are optional free-text user instructions or information, including a name or business details, and Figure 11 supplies the experimental system message.Source: §5 · p. 24

6 Limitations

The evaluation has important scope and measurement limits: name counterfactuals do not capture all user differences, the study covers only selected languages and demographics, and LMRA judgments may omit human-identified biases.

6 Limitations

  • Name counterfactuals miss biases associated with different writing styles, topics, or limited English proficiency.They therefore provide an imperfect measure of first-person bias.Source: §6 · p. 24
  • All prompts were in English, and the study covers only binary gender, four races, U.S.-based name statistics, and text-based chats.Source: §6 · pp. 24–25
  • Using an LMRA leaves open the possibility that important biases identified by humans are omitted when language models miss them.Source: §6 · p. 25

7 Conclusions

The paper presents a scalable method for measuring name-based bias across many chatbot models, domains, and tasks. It argues that systematic public evaluation can support improvement while requiring complementary metrics and broader future studies.

7 Conclusions

  • The methodology scales across many names, models, domains, and tasks while identifying systematic differences, including small ones, and producing concise descriptions.Source: §7 · p. 25
  • Systematic public bias evaluation can build shared understanding and motivate improvement across multiple stages of machine-learning development.The paper cautions that bias metrics should be paired with complementary measures such as response quality and should not ignore Custom Instructions or Memory.Source: §7 · p. 25
  • Future work should apply LMRA beyond gender, study writing-style and topic effects, and continue investigating first-person bias in multimodal chats.Source: §7 · p. 25

A Details on determining domains and tasks

The appendix documents how domains, tasks, and axes of response differences were elicited, curated, and consolidated. It also reports examples of gender-associated response patterns across diverse chatbot tasks, while cautioning that LMRA feature ratings are not definitive human judgments.

A Details on determining domains and tasks

  • Domains and tasks were first elicited by LMRA prompts, then curated into nine domains and task labels; domain names were mainly presentational after selection.The appendix lists categories spanning art, business, education, employment, entertainment, health-related, legal, technology, and travel requests.Source: A Details on determining domains and… · pp. 30–31

B Details on axes of difference

Art: Generate Creative Prompts

  • For creative prompts, LMRA ratings associated female-sounding names with simpler, more conversational, shorter, and more encouraging responses, while male-sounding names were associated with formality, structure, and detail.Source: Art: Generate Creative Prompts · p. 32

Business & Marketing: Provide Company Information

  • For company information, female-sounding names were associated with concise conversational responses, whereas male-sounding names were associated with formal tone, jargon, detail, and longer responses.Source: Business & Marketing: Provide Company… · p. 32

Education: Write Recommendation Letter

Employment: Career Advice

  • Career-advice responses associated female-sounding names with friendliness, empathy, personability, and concision, while male-sounding names were associated with professionalism, detail, headers, and long-term planning.Source: Employment: Career Advice · p. 35

Employment: Write Performance Review

  • In performance reviews, female-sounding names were associated with simpler, more summary-oriented responses, while male-sounding names were associated with formality, length, direct improvement suggestions, and broader action plans.Source: Employment: Write Performance Review · p. 35

Legal: Review Legal Document

C Beyond names: direct profiles

The study extends counterfactual bias analysis from names to direct demographic profiles, comparing profile responses with a baseline across multiple tasks. Direct-profile results show substantial variation by demographic category and task, while illustrating the approach’s broader scope beyond names.

C Beyond names: direct profiles

  • The direct-profile setup uses a Canada-based baseline and seven demographic profiles, including gender, religion, and disability characteristics.Profiles state the demographic characteristic together with living in Canada, avoiding empty profiles and pairwise comparisons.Source: C Beyond names: direct profiles · p. 36
  • 0.98 correlation links Female-Male harm ratings from name-based profiles with direct Female profiles across 12 tasks, although direct-profile ratings are consistently lower.The comparison follows the same aggregation approach used for harmful-stereotype ratings and accounts for order bias.Source: C Beyond names: direct profiles · p. 36
  • The Female profile exhibited the largest bias, biases were generally largest for Write a Story, and no significant biases were found for the Non-binary profile.Source: C Beyond names: direct profiles · p. 37
  • Low vision showed a large bias in Generate Creative Prompts, while DHH and Low vision generally showed small biases except in Career Advice.Source: C Beyond names: direct profiles · p. 37
  • Religious biases were generally small except in Write a Story, extending the profile analysis to religious and age characteristics.Source: C Beyond names: direct profiles · p. 37
  • The section demonstrates that the name-focused methodology can be extended beyond names, rather than presenting that extension as the work’s main contribution.Source: C Beyond names: direct profiles · p. 37

D.1 Names for gender bias experiments

Gender-bias experiments use highly frequent U.S. names selected to be strongly associated with female or male births. The resulting set contains 30 names, split evenly across the two genders.

D.1 Names for gender bias experiments

  • 30 names were selected from U.S. Social Security births from 1960–2023, comprising the 15 names with the greatest recorded female births and 15 with the greatest male births.Each name had > 500,000 births, with > 98% classified as female or male, respectively.Source: §D.1 · p. 37
  • The female list includes names such as Amanda, Emily, Jennifer, Jessica, Mary, Sarah, and Stephanie.Source: §D.1 · p. 37
  • The male list includes names such as Andrew, Christopher, James, John, Michael, Robert, Thomas, and William.Source: §D.1 · p. 37

D.2 Names for racial/intersectional bias experiments

The racial and intersectional experiments use curated gendered name lists because the U.S. Social Security dataset does not include race. These lists cover Asian, White, Black, and Hispanic female and male names, with figures documenting related bias analyses.

D.2 Names for racial/intersectional bias experiments

E Further details for response quality differences

Additional response-quality analyses find no statistically significant gender differences in 100,000 GPT-4o-mini responses rated by GPT-4o. The corresponding racial analysis uses the same random-prompt design, with greater confidence reported for smaller models rated by larger evaluators.

E Further details for response quality differences

F Chat versus decision-making

Institutional decision-making prompts differ sharply from real chatbot conversations, supporting separate fairness evaluation. The study also validates its LMRA ratings through crowdsourcing and illustrates name-based gender and racial stereotype assessments.

F Chat versus decision-making

G Details of human crowdsourcing study

  • 13,000 judgments from 454 English-speaking workers were collected across gender, race, and feature-labeling experiments.Gender and race pairs received 40 judgments each, while feature-labeling pairs received 50.Source: G Details of human crowdsourcing study · p. 42

G.1 Crowdsourcing examples

  • The crowdsourcing examples use extreme cases from more than 100K LMSYS or WildChat-derived response pairs to test LMRA calibration against human ratings.Ratings use a [-1, 1] scale that maps to the paper’s percentage scale.Source: §G.1 · p. 45

G.1.1 Gender bias crowdsourcing examples

  • Illustrative gender examples show substantial LMRA–human agreement in some cases, including ratings of 0.94 versus 0.10 and 0.98 versus 0.62.Source: §G.1.1 · pp. 46–47

G.2.1 Asian biases

This section presents illustrative Asian-, Black-, and Hispanic–White bias examples and crowdsourced comparisons of response simplicity and technical terminology. It examines demographic differences in harmful-stereotype ratings and response differences, while noting substantial variation and LMRA–human disagreement across prompts.

G.2.1 Asian biases

  • Asian–White examples show strongly stereotyped responses across names, restaurants, geography, explanations, word completion, and benign instructions, with LMRA ratings of 0.83–1.00.Human ratings range from -0.07 to 0.88 across the cited examples, and the largest disagreement occurs for the RLHF explanation, word completion, and benign-instruction prompts.Source: §G.2.1 · pp. 47–48

G.2.2 Black biases

  • Black–White examples associate Black-coded users with stereotyped movie references, couplets, slang, grammar, violence, and athletic answers, with LMRA ratings of 0.88–0.99.Human ratings range from -0.04 to 0.58, including substantial disagreement for bad grammar, the murderer-count question, and the fastest-man question.Source: §G.2.2 · pp. 49–50

G.2.3 Hispanic biases

  • Hispanic–White examples produce stereotyped differences in interpreting Vicuna, language ability, grammar, and misspelled text, while fewer selected examples may explain lower LMRA–human correlation.The passage states that it is unclear whether this reflects fewer ChatGPT biases or worse LMRA performance; cited LMRA ratings are 0.72–1.00 and human ratings -0.24–0.68.Source: §G.2.3 · pp. 50–51

G.3 Feature-labeling crowdsourcing examples

  • Crowdsourcing experiments provide illustrative examples for annotating axes of response differences, including simplicity and technical terminology.The section introduces ratings of which response uses simpler language and which uses more technical terminology.Source: §G.3 · p. 51, §G.3.1 · p. 51, §G.3.2 · p. 53

G.3.1 Simple responses

  • Simple-language judgments vary across paired responses: LMRA and human ratings agree on several contrasts but diverge sharply for date reasoning, grammar correction, and model-update answers.Examples include A-B ratings of 0.99/0.80, -0.91/-0.76, 0.99/0.67, -0.99/0.82, 1.00/-0.55, and -1.00/0.31 for LMRA/humans.Source: §G.3.1 · pp. 52–53

G.3.2 Technical terminology

  • The section then introduces ratings identifying which response uses more technical terminology, with three examples selected for the largest difference.The supplied passage begins the technical-terminology examples but does not provide the full response pair or ratings.Source: §G.3.2 · p. 53
  • LMRA rated the pushiness-response difference 1.00 versus 0.84 from humans.The example contrasts responses describing pushiness as disrespectful with an apologetic, supportive response.Source: §G.3.2 · p. 53
  • LMRA rated the Taiwan-president response difference 1.00 versus 0.76 from humans.The responses differed over whether Lai Ching-te or Tsai Ing-wen was president as of August 2024.Source: §G.3.2 · p. 53
  • LMRA rated the similar-tools response difference 0.99, while humans rated it -0.06.For multimeters, the responses shared four tools but differed on the fifth: power analyzer versus signal generator.Source: §G.3.2 · p. 53
  • LMRA rated the interpersonal-reassurance response difference 0.42, while humans rated it -0.57.The responses both offered help but differed in how they addressed the user’s concern about being disliked.Source: §G.3.2 · p. 54

H Racial and intersectional bias

Racial and intersectional analyses find higher rated gender harms in most domains, with racial harms showing a restaurant-recommendation exception and gender harms varying across racial name groups. The section also details probability-based harmful-stereotype scoring and order-bias correction.

H Racial and intersectional bias

  • Racial-bias results should be interpreted with lower confidence because LMRA labeling was less consistent for racial than gender stereotypes.The section also notes that multimodal racial bias was not covered.Source: H Racial and intersectional bias · p. 54
  • Gender harms were rated higher than racial harms in most domains, except travel, where restaurant recommendations showed ethnicity-matching tendencies.The comparison is qualified because LMRA ratings agreed more closely with humans for gender harms.Source: H Racial and intersectional bias · p. 54, Figure 25 · p. 56
  • Gender harms were most pronounced among typically White names and least pronounced among typically Asian names.These ratings concern harmful gender stereotypes within racial groups, not intersectional Black-gender stereotypes.Source: H Racial and intersectional bias · p. 54, Figure 26 · p. 56

H.1 Further details about harmful stereotypes

  • The harmfulness probability is the smaller matching probability across the two response orders, reducing sensitivity to which response is displayed first.Source: §H.1 · pp. 54, 57
  • Task-level harmful-stereotype ratings are reported for GPT-4o-mini responses generated using Memory.Source: §H.1 · p. 57, Figure 27 · p. 58

I Order biases

  • Order bias is addressed by querying both response orders and combining probabilities with minima before renormalization.The method uses probabilities for A, B, and C, which are usually among the five most likely tokens.Source: I Order biases · p. 59

J Filtering and scrubbing

The filtering pipeline removes prompts that reveal demographic information and masks user names in responses to reduce LMRA sensitivity to explicit group cues. Refusals are analyzed separately because they can create high noise in stereotype estimates.

J Filtering and scrubbing

  • Prompts mentioning or implying the user’s name, gender, or race are removed after PII scrubbing, although they constitute a minuscule fraction of prompts.Source: J Filtering and scrubbing · p. 59
  • User names repeated in chatbot responses are replaced with [NAME] because LMRA can otherwise rate the named response as a harmful stereotype.Source: J Filtering and scrubbing · p. 59
  • Refusals are separated from ordinary responses and checked for group differences because unequal refusals can generate high-noise harmful-stereotype estimates.The paper notes that detecting a 0.1% stereotype signal with 1% refusals requires very large samples.Source: J Filtering and scrubbing · p. 59

Related papers

Loading 2410.19803v2…