PaperlayerView exact revision on arXiv

Source-linked AI summary

Do Large Language Models Discriminate in Hiring Decisions on the Basis of Race, Ethnicity, and Gender?

Haozhe An, Christabel Acquaye, Colin Wang, Zongxia Li, Rachel Rudinger

arXiv:2406.10486v1cs.CL

TL;DR

Whether LLMs make different hiring decisions based on applicants’ perceived race, ethnicity, and gender is an open question against a backdrop of name-based differences in human hiring. The paper varies applicant names in hiring-email prompts and finds that LLMs often favor White applicants over Hispanic applicants, with Hispanic male names least favored in many settings.

  • Problem

    Evidence that race-sounding names affect human hiring motivates asking whether LLM hiring decisions similarly vary by perceived race, ethnicity, and gender, with implications for representational and allocational harms.

    Source: §1 · p. 1

  • Method

    The study varies first names statistically associated with race, ethnicity, and gender in templates asking LLMs to write hiring-decision emails, then compares group acceptance rates.

    Source: §2 · p. 2

  • Results

    LLMs tend to favor White applicants over Hispanic applicants, and Hispanic male applicants receive the least favorable treatment in many settings.

    Source: §1 · p. 2, Acceptance rates are uniformly lowest… · p. 4

  • Takeaways & Limitations

    The findings warrant concern about implicit model biases because they could produce representational and allocational harms in downstream tasks.

    Source: §5 · p. 5

    The study covers 40 occupational roles at a coarse level, leaving many of the 867 SOC occupations untested.

    Source: Limitations · p. 6

Abstract

from arXiv · show

We examine whether large language models (LLMs) exhibit race- and gender-based name discrimination in hiring decisions, similar to classic findings in the social sciences (Bertrand and Mullainathan, 2004). We design a series of templatic prompts to LLMs to write an email to a named job applicant informing them of a hiring decision. By manipulating the applicant's first name, we measure the effect of perceived race, ethnicity, and gender on the probability that the LLM generates an acceptance or rejection email. We find that the hiring decisions of LLMs in many settings are more likely to favor White applicants over Hispanic applicants. In aggregate, the groups with the highest and lowest acceptance rates respectively are masculine White names and masculine Hispanic names. However, the comparative acceptance rates by group vary under different templatic settings, suggesting that LLMs' race- and gender-sensitivity may be idiosyncratic and prompt-sensitive.

1 Introduction

The study tests whether LLM hiring decisions vary with the race, ethnicity, and gender associated with applicants’ names. Across more than 2 million generated emails, models tend to favor White applicants and give Hispanic names the least favorable treatment.

1 Introduction

  • Prior field experiments found that Black-sounding names receive fewer opportunities than European American-sounding names among similarly qualified applicants (Bertrand and Mullainathan, 2004; Cotton et al., 2008; Kline et al., 2022).Source: §1 · p. 1
  • The study prompts LLMs to write hiring-outcome emails using applicant names, while varying job title, qualification level, and template phrasing; prompts leave acceptance or rejection to the model.Source: §1 · pp. 1–2
  • The study aims to illuminate LLMs’ representational biases, demonstrate potential hiring harms, and assess LLMs as a low-cost tool for testing social-science hypotheses.The authors frame the question in terms of representational and allocational harms (Blodgett et al., 2020).Source: §1 · p. 1
  • Across over 2 million generated emails, LLMs tend to favor White applicants and give Hispanic names the least favorable treatment.The authors suggest that isolating the influence of names may signal fairness risks in other hiring-pipeline stages or professional workplace settings.Source: §1 · p. 2

2 Experiment Setup

The experiment tests hiring decisions by varying applicants’ first names in email-generation prompts and comparing acceptance rates across demographic groups. It uses controlled prompt templates, multiple LLMs, and checks to validate generated emails and their classification.

Experiment design

  • The study measures demographic differences in hiring decisions by substituting first names associated with race, ethnicity, and gender into prompts and comparing group acceptance rates with the overall average.This first-name substitution method has been used to measure biased or discriminatory behavior in humans and models (Caliskan et al., 2017).Source: §2 · p. 2

Collecting first names

  • The name set contains 300 first names: 50 for each of six race-or-ethnicity-by-gender groups across White, Black, and Hispanic applicants.The researchers selected the names using race/ethnicity data from Rosenman et al. (2023) and gender statistics from the social security application dataset (SSA1).Source: §2 · p. 2

Prompts

  • The 820 prompt templates combine four qualification levels, five phrasings, and 41 occupational roles, including an under-specified setting to isolate name demographics.The five phrasings comprise an initial template and four ChatGPT 3.5 paraphrases; omitting other applicant details was intended to limit confounding and preserve generalizability.Source: §2 · p. 2

Models

  • The experiments use Mistral-Instruct-v0.1, three Llama2 sizes (7b, 13b, and 70b), and GPT-3.5-Turbo, with three random seeds for open-source models.The study generated 756,000 emails for Mistral-7b and Llama2-7b/13b, 48,000 for Llama2-70b, and 19,200 for GPT-3.5-Turbo; the latter two were run at smaller scale due to computational limits.Source: §2 · pp. 2–3

Generation validity

  • Manual checks of 180 Llama2-13b emails found 83%–100% validity rates across models, with no statistically significant within-model differences between demographic groups.Validity required fluent, typical email format and a clear binary acceptance or rejection outcome; pairwise group comparisons had p > 0.05.Source: §2 · p. 3

Email classification

  • A name-redacted SVM classifier trained on 1,200 manually annotated emails labels more than 2 million generated emails as acceptances or rejections, achieving an F1 score of 0.98.Training examples were evenly distributed across gender and race/ethnicity; the F1 score was measured on 170 valid emails sampled from Llama2-13b generations.Source: §2 · p. 3

3 Results and Discussion

Across models and settings, Hispanic male names often receive the least favorable treatment, while other group differences vary by model, occupation, and prompt qualifications. The reported acceptance-rate gaps are generally small but statistically significant.

3 Results and Discussion

  • Table 1 to Table 3 report intersectional-group acceptance rates across templatic settings and models, with group rates tested against the population acceptance rate using permutation tests.The passage points to Appendix C.3 for details of the test.Source: §3 · p. 3

Differences are small but statistically significant

  • 1.60% to 3.78% separates the highest and lowest group acceptance rates across models, and permutation tests find these differences statistically significant.The authors argue that even a 3.78% disadvantage based on name associations is concerning because small differences could accumulate across a hiring pipeline.Source: Differences are small but… · pp. 3–4

Acceptance rates are uniformly lowest for Hispanic male names

  • Hispanic male applicants consistently receive the least favorable treatment in many settings across Mistral-7b, Llama2 models, and GPT-3.5.The authors warn that employer use of LLMs could entrench, systematize, and amplify hiring discrimination against Hispanic applicants.Source: Acceptance rates are uniformly lowest… · p. 4

Some groups exhibit higher acceptance rates

  • White male names and Black female names receive above-average acceptance rates in two and three of five models, respectively; the Black female pattern runs counter to Bertrand and Mullainathan (2004).For Mistral-7b, higher acceptance for Black female names applies only to occupations in the “no formal education” and “postsecondary non-degree award” categories.Source: Some groups exhibit higher acceptance… · p. 4

Llama2-70b shows least variation across demographic groups

  • Llama2-70b has the smallest overall acceptance-rate range at 1.6%, versus 3.56% for Llama2-13b and 3.78% for GPT-3.5.The authors suggest larger models may be more robust and fair in the under-specified email task, but say the cause is inconclusive because training details are not fully public.Source: Llama2-70b shows least variation… · p. 4

Qualifications matter

  • When qualifications are unspecified, female names generally receive higher acceptance rates, but for “somewhat qualified” or “not qualified” candidates, White names—especially White male names—appear most likely to be accepted.The authors speculate, without offering an explanation, that the latter pattern could relate to a gender confidence gap.Source: Qualifications matter · pp. 4–5

Some models exhibit human-like gender-occupation stereotypes

  • For secretary, GPT-3.5 generates fewer acceptance emails for male candidates than female candidates across racial and ethnic groups, but this gender-occupation pattern is not universal across models and roles.The authors characterize gender sensitivity as idiosyncratic and prompt-dependent.Source: Some models exhibit human-like… · p. 5

4 Related Work

Related work connects names with demographic bias and economic opportunity, while this study examines name-based disparities in generated hiring emails and distinguishes its approach from prior hiring audits.

First names, demographic identities, and economic opportunities

  • Names correlate with demographic attributes, and stereotypes linked to names can shape economic opportunities (Bertrand and Mullainathan, 2004; Nunley et al., 2015; Goldstein and Stecklov, 2016; Ahmad, 2020).Source: §4 · p. 5

First name biases in language models

Auditing LLMs in hiring

5 Conclusion

Across many templatic prompts, some LLMs show lower acceptance rates for Hispanic applicants and often higher acceptance rates for White applicants, raising concerns about implicit bias and downstream harm.

5 Conclusion

  • Hispanic applicants’ acceptance rates are systematically below average in multiple cases, while White applicants often have a higher chance of acceptance.The findings identify discrimination in some LLMs against traditionally underrepresented groups.Source: §5 · p. 5
  • The study uses 820 templates and 300 names to generate as many as 756,000 job application outcome emails per model.Source: §5 · p. 5
  • Implicit model biases could cause representational and allocational harms to demographic groups in downstream tasks.Source: §5 · p. 5

Limitations

The study’s scope is limited by the demographic and occupational groups represented, the model sizes tested, and prompts that simplify hiring by omitting most applicant information.

Incomplete representation of demographic identities

  • Limited first-name data allowed thorough study of only Black, White, and Hispanic names and female and male names, leaving many identities and traits unrepresented.Some attributes are not strongly correlated with first names, making names a less feasible proxy; the authors emphasize fair and responsible LLM use.Source: Limitations · p. 6

Incomplete representation of occupations

  • The study tested 40 occupational roles at a coarse level, versus 867 occupations in the 2018 SOC; differential treatment in other occupations remains inconclusive.The authors say extensive experiments are needed to assess whether the pattern extends to other occupations.Source: Limitations · p. 6

A wider range of LLMs could be tested

  • All five tested state-of-the-art models were ≥7b; smaller models were not studied and may behave differently, especially when prompts say applicants are unqualified.The authors leave study of smaller models for future work.Source: Limitations · p. 6

Not simulating the entire hiring process

  • The prompts provide little or no applicant information, unlike real hiring, and use qualification as a summary judgment in place of details such as a resume.The authors note that the focused prompts may reveal representational biases without distraction from additional applicant details.Source: Limitations · p. 6

Ethics Statement · A First Names

The study raises ethical concerns about using LLMs in hiring: models correlate first names with different acceptance rates, while research and warnings about discrimination also carry risks that must be weighed against human decision-making and possible equality gains.

Ethics Statement

  • LLMs correlate distinct first names with varying job-application acceptance rates, raising concerns about their use to allocate economic opportunities.The study calls for vigilance when deploying LLMs in consequential decision-making.Source: Ethics Statement · p. 6
  • Research on race, ethnicity, and gender can itself essentialize or misrepresent individuals within those social categories.The authors identify this as a potential risk of studying discriminatory behavior.Source: Ethics Statement · p. 6
  • The authors believe scalable harms from LLM hiring may outweigh its benefits, but human decision-making is also known to be biased.Employers and policy-makers must weigh the harms of LLM use against those of the human alternative.Source: Ethics Statement · p. 6
  • Warnings about AI decision-making may prevent harmful use, but stigmatizing LLMs could also prevent adoption where they might advance social equality.The authors identify this as a potential risk, not a demonstrated outcome.Source: Ethics Statement · p. 6

A.1 Selection Criteria · A.2 Names Used · B Prompts

The study selects first names using race/ethnicity and gender distributions, then tests hiring prompts that vary in phrasing, occupation, and stated qualifications. The resulting name set contains 300 first names, and the prompts are designed to isolate responses to names from effects of pronouns.

A.1 Selection Criteria

  • The researchers approximate a name’s gender using the SSA dataset and assign the majority gender when it exceeds 50%.Source: §A.1 · p. 10
  • Within each race-and-gender subgroup, the researchers select the 50 names with the highest percentage of the majority race.Source: §A.1 · p. 10

A.2 Names Used

B Prompts

  • To address sensitivity to template phrasing, the researchers use ChatGPT 3.5 to paraphrase an initial template into four variations, creating five base templates.Source: B Prompts · p. 10
  • The researchers later remove pronouns from the template to control for possible effects of “she” and “he” and focus on responses to first names.Source: B Prompts · pp. 10–11

C Additional Experiment Setup Details · C.1 Models

The appendix describes model-use consistency, generation settings, computational constraints, and infrastructure for the hiring-email experiments. Cost limits reduced the scale of Llama2-70b and GPT-3.5-Turbo experiments, while the selected models were used according to their intended instruction-following purpose.

C.1 Models

  • The authors kept hyperparameters consistent across models when possible and used each selected model for its intended instruction-following purpose.They describe the models as fine-tuned to follow human instructions like the designed prompts.Source: §C.1 · p. 11
  • Cost limits reduced Llama2-70b and GPT-3.5-Turbo experiments to seven occupations; Llama2-70b used one seed and GPT-3.5-Turbo used two templates.The passage reports 48,000 emails for Llama2-70b and 19,200 for GPT-3.5; it also reports 756,000 emails for Mistral-7b and Llama2-7b, 70b.Source: §C.1 · p. 11
  • Llama2 generation used temperature 0.6, top_p 0.9, max_batch_size 4, and sequence and generation lengths of 256 across the 7b, 13b, and 70b sizes.The authors used random seeds 1, 42, and 50 for reproducibility.Source: §C.1 · p. 11
  • GPT-3.5-Turbo used temperature 0.6, max_tokens 256, frequency_penalty 0.9, and presence_penalty 1.9, with other settings left at defaults..Source: §C.1 · p. 11
  • Table 5 presents email-generation validity rates and classifier performance for predicting outcomes indicated in valid emails.Source: Table 5 · p. 11
  • Mistral-Instruct-v0.1 generation used temperature 0.6, max_new_tokens 256, do_sample true, and top_p 5, with random seeds 1, 42, and 50.The passage identifies the model as 7b.Source: §C.1 · p. 11
  • The authors state that they followed the model authors’ or company’s terms of use for Llama2, GPT-3.5-Turbo, and Mistral-Instruct-v0.1.Source: §C.1 · p. 11
  • Offline Llama2 and Mistral experiments ran on NVIDIA RTX A5000 and A6000 cards, with Llama2 GPU counts varying by model size.Llama2 7b, 13b, and 70b used one A6000, two A6000, and eight A5000 GPUs respectively; Mistral used one A6000.Source: §C.1 · p. 11

C.2 Email Classification · C.3 Permutation Test

The appendix describes how generated-email outcomes were classified and validated, and how permutation tests assess group acceptance-rate differences and aggregate significance across jobs.

C.2 Email Classification

  • Manual annotation of 1,200 emails was evenly distributed across genders and races/ethnicities, and an SVM with TF-IDF features was trained on 840 samples.The remaining samples were divided into 180 for validation and 180 for testing.Source: §C.2 · pp. 11–12
  • Across intersectional groups within each model, validity-rate standard deviations were relatively small, and no pairwise differences were statistically significant (all p-values > 0.05).Table 6 reports these validity rates; its null hypothesis is that each pair of groups shares the same rate under a binomial distribution.Source: Table 6 · p. 12
  • The classifier achieves 0.97 accuracy on a 180-sample test set evenly distributed across demographic groups.The test set was held out from 1,200 manually labeled emails; 840 samples were used for training and 180 for validation.Source: §C.2 · pp. 11–12
  • For template variations, classifier predictions matched human labels with an F1 score as high as 0.99 for Llama2-70b.The comparison used the same random subset as the validity analysis.Source: §C.2 · p. 12

C.3 Permutation Test

  • The adapted permutation test compares a group's acceptance rate with the overall rate under the null that they are equal, estimating a p-value from 5,000 permutations.It compares the group's observed difference from the global rate with differences after permuting identity labels while keeping the group's size fixed.Source: §C.3 · p. 12
  • For aggregate significance across occupational roles, the study runs separate permutation tests for each job and combines their p-values using Fisher's method.The method is attributed to Fisher (1928).Source: §C.3 · p. 12, §C.2 · p. 12

Related papers

Loading 2406.10486v1…