Source-linked AI summary
"Kelly is a Warm Person, Joseph is a Role Model": Gender Biases in LLM-Generated Reference Letters
Yixin Wan, George Pu, Jiao Sun, Aparna Garimella, Kai-Wei Chang, Nanyun Peng
TL;DR
LLM-assisted reference-letter generation raises fairness concerns because generated professional documents may reproduce gender bias. This paper evaluates lexical and stylistic bias, plus bias in hallucinated content, across ChatGPT and Alpaca, finding significant gender biases and urging judicious use and further mitigation research.
Problem
The paper addresses limited understanding of fairness issues and associated harms in LLM-generated professional documents, particularly recommendation letters.
Method
The study evaluates lexical content, language style, and hallucination bias in Context-Less and Context-Based LLM-generated reference letters.
Results
Significant gender biases appear in letters generated by ChatGPT and Alpaca, including differences in language style and bias propagation in hallucinated content.
Takeaways & Limitations
AI-assisted reference-letter writing should be used judiciously, with policies and further research aimed at detecting and mitigating fairness issues.
Takeaways & Limitations
The study considers only binary gender, focuses primarily on reference letters, and evaluates a limited set of models because of data, cost, and compute constraints.
Abstract
from arXiv · showhide
Large Language Models (LLMs) have recently emerged as an effective tool to assist individuals in writing various types of content, including professional documents such as recommendation letters. Though bringing convenience, this application also introduces unprecedented fairness concerns. Model-generated reference letters might be directly used by users in professional scenarios. If underlying biases exist in these model-constructed letters, using them without scrutinization could lead to direct societal harms, such as sabotaging application success rates for female applicants. In light of this pressing issue, it is imminent and necessary to comprehensively study fairness issues and associated harms in this real-world use case. In this paper, we critically examine gender biases in LLM-generated reference letters. Drawing inspiration from social science findings, we design evaluation methods to manifest biases through 2 dimensions: (1) biases in language style and (2) biases in lexical content. We further investigate the extent of bias propagation by analyzing the hallucination bias of models, a term that we define to be bias exacerbation in model-hallucinated contents. Through benchmarking evaluation on 2 popular LLMs- ChatGPT and Alpaca, we reveal significant gender biases in LLM-generated recommendation letters. Our findings not only warn against using LLMs for this application without scrutinization, but also illuminate the importance of thoroughly studying hidden biases and harms in LLM-generated professional documents.
1 Introduction
The paper examines gender bias in LLM-generated recommendation letters, motivated by the risks of using automated professional documents without scrutiny. It evaluates bias across generation settings and analyzes lexical content, language style, and hallucinated information.
- Motivation: LLM-generated professional documents may reproduce gender biases that create risks in sensitive applications such as recommendation letters.Prior research links gender bias in professional documents to diminished career opportunities for gender minority groups.
- Evaluation settings: The study compares Context-Less Generation, based on simple candidate descriptions, with Context-Based Generation, which also supplies personal information and experience.CLG isolates biases associated with minimal input, while CBG simulates typical user-assisted letter writing.
- Illustrative finding: ChatGPT-generated letters describe Kelly with warmth and likability, while portraying Joseph with more leadership and agentic characteristics.The example contrasts phrases such as “well-liked member” with “a natural leader” and “a role model.”
- Evaluation scope: The evaluation covers lexical content, language style, and hallucination bias in LLM-generated reference letters.Hallucination bias concerns bias exacerbation in model-hallucinated content.
- Contribution: Benchmarking ChatGPT and Alpaca reveals significant gender biases and motivates scrutiny of LLM use for professional document generation.The authors characterize current LLMs as not mature for generating professional documents and call for research on mitigation.
2 Related Work
Related work establishes gender bias in NLP and professional-document language, while prior hallucination research motivates an NLI-based detection approach. The paper builds on these strands to study bias in LLM-generated reference letters.
- Social bias in NLP: NLP fairness research distinguishes allocational and representational harms and studies bias evaluation and mitigation in NLU and NLG.Prior work includes methods for measuring, controlling, and reducing gender bias in language systems.
- Bias evaluation: Earlier NLG studies measure gender bias using salient word differences, contextual prompts, and real prompts extracted from Wikipedia.Other approaches investigate controlling NLG models to reduce bias.
- Professional documents: Social science research reports gender bias in professional-document language and connects it with differences in application outcomes.Examples include more personal-life mentions for female candidates and more excellent language for male candidates.
- Language agency: Prior work characterizes agency bias through communal descriptions for women and agentic descriptions for men.Agentic characteristics include assertiveness and initiating tasks, whereas communal characteristics concern others’ welfare and helping.
- Hallucination detection: Hallucination detection research includes information-extraction, question-answering, and natural-language-inference approaches.This study adopts an NLI-based approach that compares generated text with the original input context.
3 Methods
The methods evaluate gender bias in reference letters through two generation settings, lexical and stylistic analyses, and a separate assessment of bias in hallucinated content. They combine word-choice statistics, t-tests, and context-sentence NLI.
- Generation settings: The study uses Context-Less Generation with minimal information and Context-Based Generation with contextual biographies.CLG isolates biases influenced by limited input, while CBG approximates a user asking an LLM to write a letter from self-description.
- Biases in Lexical Content: Lexical-content bias is measured as salient frequency differences in word choices, analyzed separately for nouns and adjectives.The method uses Odds Ratio to compare adjective salience between male and female letters.
- Biases in Lexical Content: Odds Ratio compares an adjective’s odds of appearing in male letters with its odds of appearing in female letters.Adjectives are ranked by Odds Ratio to identify those most salient for male and female letters.
- Biases in Language Style: Language-style bias is assessed by comparing gender-group differences in formality, positivity, and agency using statistical t-tests.The bias statistic uses sample means and standard deviations; a value below the significance threshold indicates bias.
- Biases in Language Style: Formality and positivity measure differences in formal sentences and positive-sentiment sentences between male and female documents.Formality proxies professionalism, while positivity proxies excellency in language.
- Biases in Language Style: Language Agency is proposed as a novel metric based on gender differences in the percentage of agentic sentences.The method evaluates significance using t-testing.
- Hallucination Bias: Hallucination Bias measures harmful propagation or amplification of gender bias in information hallucinated during Context-Based Generation.Context-Sentence NLI splits generated documents into sentences and tests whether the source context entails each sentence.
- Hallucination Bias: Hallucinated content is evaluated with the same formality, positivity, and agency metrics used for full generated documents.Amplification is assessed by comparing hallucinated information with the full document for each gender.
4 Experiments
The experiments evaluate gender bias in LLM-generated reference letters under context-less and context-based settings, examining lexical content, language style, and hallucinated content. ChatGPT and Alpaca exhibit significant gender-stereotypical differences across these analyses, including when candidates share identical career information.
- Experiment setup: The study evaluates Context-Less Generation and Context-Based Generation, with CBG intended to approximate users’ real-world reference-letter writing.CLG uses minimal candidate descriptors, while CBG additionally supplies personal information and experience.
- Experiment setup: ChatGPT and Alpaca were selected for CBG after generation-quality screening excluded models that sometimes produced unrelated content.The screening used criteria-based generation success rates to retain valid reference-letter outputs.
- Context-Less Generation: CLG showed higher odds of male-stereotypical traits in male letters and higher odds of female-stereotypical traits in female letters.The reported male-associated traits include Ability, Standout, Leadership, Masculine, and Agentic; female-associated traits include Feminine, Communal, and Personal.
- Context-Based Generation: In CBG lexical analysis, both models used gender-stereotypical words, and salient words were significantly associated with gender-stereotypical lexicons.Examples include “respectful” for males and “warm” for females; WEAT analyses used name-based and career/family-based trait sets.
- Biases in language style: Male-generated documents were significantly higher than female-generated documents in formality, positivity, and agency for both ChatGPT and Alpaca.These results aligned with reported social-science findings on professionalism, excellency, and agency in human-written reference letters.
- Biases in language style: With identical career-related biographies, ChatGPT described female candidates using communal language but male candidates using more agentic narratives.Female examples included “great to work with,” “communicates well,” and “kind,” whereas male examples included “a standout in the industry” and “a true original.”
- Hallucination bias: ChatGPT hallucinations were more formal and positive for male candidates and less agentic for female candidates, while Alpaca hallucinations were more positive for males and less formal and agentic for females.The authors characterize these patterns as gender-bias propagation and amplification in hallucinated content.
5 Conclusion and Discussion
The study finds substantial gender bias in LLM-generated reference letters, including stereotyped language, lexical differences, and amplified bias in hallucinated content. It therefore calls for cautious use, policy development, and further mitigation research.
- Future work: Future work should examine additional professional document categories, demographics, genders, language styles, lexical content, and hallucinated-content bias.The paper identifies mitigation and broader evaluation as open directions.
- Conclusion: The study identifies significant gender biases in LLM-generated recommendation letters across lexical content, language style, and hallucination bias.The evaluation framework covers both context-less and context-based generation scenarios.
- Conclusion: LLMs default to gender-stereotypical content with insufficient context and use different word choices and linguistic styles even when detailed candidate information is provided.The paper reports these patterns across its two generation settings.
- Conclusion: Hallucinated content propagates and can amplify harmful gender biases in generated reference letters.Examples distinguish communal descriptions for female candidates from agentic descriptions for male candidates with the same biography.
- Conclusion: The authors recommend judicious use of AI-assisted writing, comprehensive real-world usage policies, and continued research on detecting and mitigating fairness issues.These measures are presented as ways to reduce potential harms from biased professional documents.
Limitations
The study’s limitations concern the breadth of populations, document types, and models examined. The authors specifically identify binary-gender analysis and reference letters as restricted parts of the scope.
- Limitations: Limited datasets and prior literature restricted the analysis to binary gender.The authors call for extending fairness research to other gender minority groups.
- Limitations: The study primarily focuses on reference letters rather than the broader range of professional documents enabled by LLMs.Suggested extensions include resumes and peer evaluations.
- Limitations: Cost and compute constraints limited experiments to the ChatGPT API and three other open-source LLMs.Future work is encouraged to include more models and demographic backgrounds.
Ethics Statement
The ethics statement situates the study in concerns about biases learned from internet-scale pretraining and their use in professional documents. It reports noticeable harms while acknowledging that the analyzed user base was only a subset of LLM users.
- Ethics Statement: The experiments use LLMs pretrained on broad internet text that have been shown to learn or amplify biases from that data.This motivates examining ethical considerations in professional-document generation.
- Ethics Statement: The study examines ethical risks in professional documents through the representative task of reference-letter generation.The authors frame reference letters as one real-world application of LLMs.
- Ethics Statement: Although the study analyzes only a subset of the representative LLM user base, it uncovers noticeable harms and areas of concern in real-world scenarios.The authors present the findings as grounds for additional caution and equitable development.
- Ethics Statement: The authors characterize LLM tools as useful writing assistants or first drafts that should nevertheless be used with caution because of potential biases and harms.This ethical framing follows from the study’s findings about generated professional documents.
B.1 Failure Analysis
The failure analysis classifies unsuccessful generations and defines a rule-based success criterion. Among the evaluated models, Alpaca achieved a significantly higher generation success rate and was selected alongside ChatGPT for further evaluation.
- Failure Analysis: Unsuccessful LLM generations fall into three types: empty content, repetitive content, and task divergence.Task divergence means generating text other than a recommendation letter.
- Failure Analysis: A successful generation is nonempty, non-repetitive, and task-following according to three rule-based detection criteria.The implementation checks for nonempty output, no long continuous strings, and inclusion of the word “recommend”.
- Failure Analysis: Alpaca achieves a significantly higher generation success rate than the other investigated LLMs.The reported comparison motivates choosing ChatGPT and Alpaca for subsequent evaluation experiments.
C Prompt Design
The prompt design uses distinct descriptors and prompt templates for context-less and context-based reference-letter generation, with enumerated CLG combinations for analysis.
- C.1 Descriptors for CLG task: CLG uses two gender descriptors, Kelly and Joseph, selected to reduce name overlap across genders and span varied ages and occupations.
- Prompt Design: The prompt materials support systematic querying across the defined CLG descriptor combinations and CBG context-based inputs.
- C.2 Prompts for CLG Task: The CLG prompt template combines descriptor metadata to generate 120 reference letters for analysis across Kelly and Joseph.
- C.3 Prompts for CBG Task: The CBG task uses prompts that query reference-letter generation with candidate context.
- C.3 Prompts for CBG Task: The appendix provides a sample prompt for professional-document generation in the CBG scenario.
D Dataset Statistics: WikiBias-Aug
This section reports statistics for the preprocessed WikiBias-Aug dataset.
- D Dataset Statistics: WikiBias-Aug: WikiBias-Aug is the preprocessed dataset used to provide context information for context-based reference-letter generation.
E Sample Reference Letter Generations
The appendix points to sample generations for ChatGPT and Alpaca under context-less and context-based scenarios, including illustrative letter excerpts.
- Context-Less Generation: ChatGPT sample generations are provided for context-less reference-letter generation.
- Context-Based Generation: ChatGPT sample generations are provided for context-based reference-letter generation.
- Context-Based Generation: Alpaca sample generations are provided for context-based reference-letter generation.
- Generated Letters: One displayed generation recommends Joseph for writing work and describes him as dedicated, hardworking, and reliable.
- Generated Letters: Another displayed generation recommends Kiril Renner as an accomplished actress and highlights her career achievements and awards.
- Generation Context: The context-based generation setup uses anonymized, gender-balanced biography contexts to reduce pre-existing gender biases and name-triggered hallucinations.
F.2 Agentic Classifier Training Details
The classifier-training details describe a synthetic biography dataset and the lexicons used to characterize agentic and communal language alongside related categories and gendered word lists.
- F.2 Agentic Classifier Training Details: The classifier dataset contains 400 unique ChatGPT-generated biographies per category with an even distribution.
- F.2 Agentic Classifier Training Details: Initial biographies are sampled from the Bias in Bios dataset, which derives from online Common Crawl biographies.
- Lexicon Categories: The investigated lexicons include ability, standout, leadership, masculine, feminine, agentic, communal, professional, and personal categories.
- Gendered Word Lists: Gendered word lists are used for WEAT testing, alongside the broader lexicon categories.
- Preprocessing Pipeline: Figure 2 depicts the preprocessing pipeline used to construct the WikiBias-Aug corpus.
- Agentic and Communal Lexicons: Agentic terms include assertiveness, confidence, ambition, dominance, independence, and intellect.
- Agentic and Communal Lexicons: Communal terms include affection, helpfulness, kindness, nurturing, warmth, caring, and assistance.
- Professional and Personal Lexicons: Professional terms cover work and career language, whereas personal terms cover family and domestic relationships.