PaperlayerView exact revision on arXiv

Source-linked AI summary

Gender, Race, and Intersectional Bias in Resume Screening via Language Model Retrieval

Kyra Wilson, Aylin Caliskan

arXiv:2407.20371v3cs.CYcs.AIcs.CLcs.LG

TL;DR

This paper asks whether language-model resume screening disadvantages groups based on race and gender, a concern given the documented biases in AI hiring tools. It audits three embedding models using a retrieval-based screening simulation and finds preferences across identity groups, especially disadvantage for Black men, alongside sensitivity to name frequency and resume length.

  • Problem

    It is unclear whether language models can screen resumes without disadvantaging groups based on protected attributes such as race and gender.

    Source: Introduction · p. 1

  • Method

    The study simulates resume screening with document retrieval and audits three embedding models using real-world resumes and job descriptions augmented with identity-associated names.

    Source: Introduction · p. 2

  • Results

    Resumes with White or male names were preferred over those with Black or female names in up to 85.1% of cases, and Black male names were especially disadvantaged.

    Source: Introduction · p. 2, Discussion · p. 9, Conclusion · p. 10

  • Takeaways & Limitations

    The authors conclude that resume-screening systems need comprehensive audits because outcomes reinforce societal discrimination and vary with name frequency and resume length.

    Source: Discussion · p. 10

    The study represents only two race groups and two gender groups through associated names, a reductive and incomplete representation of identity.

    Source: Future Work · p. 10, Ethical Considerations Statement · p. 11

Abstract

from arXiv · show

Artificial intelligence (AI) hiring tools have revolutionized resume screening, and large language models (LLMs) have the potential to do the same. However, given the biases which are embedded within LLMs, it is unclear whether they can be used in this scenario without disadvantaging groups based on their protected attributes. In this work, we investigate the possibilities of using LLMs in a resume screening setting via a document retrieval framework that simulates job candidate selection. Using that framework, we then perform a resume audit study to determine whether a selection of Massive Text Embedding (MTE) models are biased in resume screening scenarios. We simulate this for nine occupations, using a collection of over 500 publicly available resumes and 500 job descriptions. We find that the MTEs are biased, significantly favoring White-associated names in 85.1\% of cases and female-associated names in only 11.1\% of cases, with a minority of cases showing no statistically significant differences. Further analyses show that Black males are disadvantaged in up to 100\% of cases, replicating real-world patterns of bias in employment settings, and validate three hypotheses of intersectionality. We also find an impact of document length as well as the corpus frequency of names in the selection of resumes. These findings have implications for widely used AI tools that are automating employment, fairness, and tech policy.

Introduction

The paper investigates whether language-model resume screening favors groups based on race, gender, or their intersection, and whether document features affect those outcomes. It simulates screening with retrieval and audits three embedding models, finding group preferences and sensitivity to resume length and name frequency.

Introduction

  • Hiring tools are widely used, yet prior examples include discrimination; the study therefore examines whether model screening disadvantages protected groups.The introduction cites an Amazon resume-screening tool that was scrapped after it was found to discriminate unfairly against women.Source: Introduction · p. 1
  • The study also asks whether name frequency and resume length affect screening outcomes, since these textual features vary across documents.The research questions address race and gender signals separately, their intersections, and the effects of these document features.Source: Introduction · pp. 1–2
  • The study simulates resume screening as a document-retrieval task and examines outcomes by race, gender, and their intersection.Screening is framed as identifying relevant resumes before interviews and selection, when a hiring professional evaluates candidates further.Source: Introduction · p. 2
  • The audit uses three Massive Text Embedding models, over 500 publicly available resumes and job descriptions, and nine occupations.Resumes are augmented with 120 frequency-controlled names associated with White, Black, male, and female identities.Source: Introduction · p. 2
  • Resumes with White or male names were preferred over those with Black or female names in up to 85.1% of cases, and intersectional tests found Black males especially unfavored.Gender differences were driven largely by preferences for Black females over Black males; resume length and name-frequency choices also changed measured bias.Source: Introduction · p. 2

Related Work

Prior work documents discrimination in hiring and biased AI systems, but external audits of AI-mediated resume screening remain limited. Early chatbot studies report biased rankings but cannot inspect internal representations and do not rigorously test document features.

Related Work

  • Resume audit and correspondence studies test discrimination by sending artificial resumes that differ on a protected attribute and comparing response rates.This approach has been used to examine race, queerness, religion, disability, and age [Bertrand and Mullainathan 2004; Mishel 2016; Wright et al. 2013; Hipes et al. 2016; Neumark, Burn, and Button 2016, 2019; Lahey 2008].Source: Related Work · p. 2
  • External gender and racial audits of AI-mediated resume screening tools are absent, while vendor statements about anti-discrimination compliance are often limited.A cooperative audit found no adverse impact, but its assumptions make generalization difficult; another audit found closed-source models can be unstable without testing protected characteristics.Source: Related Work · pp. 2–3
  • Preliminary chatbot studies report biased resume rankings, but their black-box models prevent analysis of internal representations and they omit rigorous tests of length or term frequency.Reported examples include Black women ranked as top software-engineering candidates in 11% of tests and disability-related resumes ranked highest in 25% of cases [Yin, Alba, and Nicoletti 2024; Glazko et al. 2024].Source: Related Work · p. 3
  • Embedding-based retrieval offers a less computationally intensive alternative to generative screening, though it may remain sensitive to spurious prompt features.Generative outputs are sensitive to prompt features unrelated to the task [Sclar et al. 2023].Source: Related Work · p. 3

Data

The study evaluates three high-performing, architecturally similar embedding models on publicly available resume and job-description datasets standardized and filtered to support reproducible screening comparisons.

Massive Text Embedding Models

  • The three selected MTEs share a Mistral-7B-like architecture, with fine-tuning for representational tasks and differing training objectives and data.E5 uses contrastive loss and retrieval data; GritLM combines contrastive loss with next-token prediction; SFR is fine-tuned on E5 for retrieval, classification, and clustering.Source: Data · p. 3, Massive Text Embedding Models · p. 3, Figure 1 · p. 3

Job Description and Resume Stimuli

  • Public resume and job-description datasets were standardized using SOC codes, with low-confidence code assignments removed and occupations required to have at least 20 documents of each type.The filtered corpus contains 554 resumes and 571 job descriptions across nine occupations.Source: Job Description and Resume Stimuli · pp. 3–4

Data

  • The datasets provide free-text job descriptions and resumes, but resume privacy and changing job postings make collection and reproducibility challenging.The study uses publicly available online datasets to support reproducibility.Source: Job Description and Resume Stimuli · pp. 3–4

Approach

The approach encodes job descriptions and resumes with embedding models, ranks their similarity, and tests whether selected resumes are distributed equally among identity groups. It controls identity signals with name augmentation and varies task instructions and name-frequency strategies.

Approach

  • MTE embeddings and cosine similarity rank resumes against each job description; chi-square tests assess whether selected resumes are distributed equally among relevant groups.The dataset has at least 160 resumes for every bias test, exceeding the stated minimum of five observed values for chi-square population estimates.Source: Approach · p. 4

Task-Augmented Job Descriptions

  • Task instructions are appended to job descriptions, which serve as queries, while resumes serve as documents whose similarity estimates relevance.The study uses ten instructions and follows model-specific templates for encoding them.Source: Task-Augmented Job Descriptions · pp. 4–5, Figure 2 · p. 5

Name-Augmented Resumes

  • Resumes receive a variable first name and constant surname, with names selected across four race-gender groups and frequency-matched strategies.The surname Williams is held constant; one name set approximates population-proportional frequencies, while supplemental White names are matched to Black-name frequencies.Source: Name-Augmented Resumes · p. 5, Table 2 · p. 6

Resume Screening

  • The models encode documents to 1,300 tokens, normalize embeddings, and average cosine similarity across task instructions before selecting the most similar resumes.The retrieval framework was validated: same-occupation resumes without names scored significantly higher than different-occupation resumes for every occupation (p<0.001).Source: Figure 3 · p. 6, Resume Screening · p. 6

Experiments

The experiments test retrieval-based resume screening and audit how race, gender, intersectional identity, resume length, and name frequency relate to selection. They use balanced identity groups and compare the most relevant resumes for each job description.

Experiments

  • The four experiments assess retrieval validity, separate race and gender bias, intersectional bias, and the effects of resume length and name frequency.Identity groups are balanced in the initial resume pools; significant deviations from uniform representation are treated as biased outcomes.Source: Experiments · pp. 6–7

Evaluating Retrieval for Resume Screening

  • Cosine similarities for occupation-matched and unmatched resumes are compared to evaluate retrieval as an initial resume-screening stage.The comparison uses resumes without names and job-description embeddings.Source: Evaluating Retrieval for Resume… · p. 7

Evaluating Race and Gender Bias

  • For race and gender audits, each model selects the top 10% of resumes per job description, yielding 27 bias tests per attribute.Groups are formed from 40 names each, with names drawn from the four intersectional groups.Source: Evaluating Race and Gender Bias · p. 7

Evaluating Intersectional Bias

  • Intersectional tests repeat the selection procedure for pairs of identity groups, excluding comparisons that share neither race nor gender.Each intersectional group contributes 20 names.Source: Evaluating Intersectional Bias · p. 7

Evaluating Effects of Length and Frequency

  • The final experiment compares full-length resumes with title-only resumes and tests raw-frequency-matched against population-proportional-frequency-matched names.Both comparisons use cosine similarities and bias tests for non-intersectional race and gender categories.Source: Evaluating Effects of Length and… · p. 7

Results

Retrieval distinguished resumes whose occupation matched the job description, while selection audits found preferences across race, gender, and intersectional comparisons. Shorter resumes and name frequency also affected measured bias.

Verification of Retrieval

  • For every occupation, resumes matching the job-description occupation had higher cosine similarities than unmatched resumes (p<0.001).This supports the use of document retrieval for screening relevance.Source: Verification of Retrieval · p. 7

White and Male Names are Preferred

  • 85.1% of race tests favored White names, while 8.6% favored Black names; 51.9% favored male names and 11.1% female names.The remaining reported tests showed no statistically significant difference; female-name preferences came from one model.Source: White and Male Names are Preferred · pp. 7–8, Figure 5 · p. 8

Nuances of Intersectional Identities

  • White female names were preferred over Black female names in 48.1% of tests, while Black female names were preferred in 25.9%.In Black female-versus-male comparisons, Black female names were preferred in 66.7% of tests and Black male names in 14.8%.Source: Nuances of Intersectional Identities · p. 8
  • Among White names, female names were preferred in 25.9% of tests and male names in 18.5%; no model consistently preferred one group.Significant differences occurred in 44.4% of tests.Source: Figure 6 · p. 8, Nuances of Intersectional Identities · p. 8
  • White male names were significantly preferred over Black male names in all 27 tests (p<0.05).GritLM was the only model that always preferred White female names in the reported comparisons.Source: Nuances of Intersectional Identities · p. 8

Shorter Resumes Result in More Bias

  • 96.2% of title-only race tests showed significant differences, versus 93.7% for full-length resumes.White names were preferred in 62.9% of title-only tests and Black names in 33.3%.Source: Shorter Resumes Result in More Bias · p. 8
  • 85.2% of title-only gender tests showed significant differences, versus 63% for full-length cases; the increase came from preferences for female names.The difference between group selection rates also increased for title-only resumes in both race and gender tests.Source: Shorter Resumes Result in More Bias · p. 8

Frequency Effects Bias Measurements

  • 51.9% of tests using approximately frequency-equivalent DOLMA-corpus names favored Black names, while 22.2% favored White names.This reverses the pattern observed with population-proportional-frequency names.Source: Frequency Effects Bias Measurements · pp. 8–9

Discussion

The models generally favored White and male names, while intersectional tests showed a marked disadvantage for Black men and supported three hypotheses about how race and gender biases interact. Resume length and name frequency also altered outcomes, and the authors identify limits to name-based auditing and the study data.

Discussion

  • Group disparities did not correlate with population statistics, suggesting default model preferences rather than occupational patterns learned during training.The authors suggest White and male defaults may be common across contexts, weakening or obscuring occupation-based nuances.Source: Discussion · p. 9
  • Black male names were preferred over Black female names in 14.8% of tests, while Black female names were preferred in 66.7%.Black male names were preferred over White male names in 0% of tests.Source: Discussion · p. 9, Figure 8 · p. 9
  • The findings support three intersectionality hypotheses: intersectional bias is not reducible to the sum of race and gender biases, race bias aligns more with comparisons among men, and gender bias more with White men versus White women.The last two comparisons were less similar for groups of women and groups of Black men, respectively.Source: Discussion · p. 9
  • Manipulating resume length and name frequency changed both the amount of biased outcomes and which social groups were preferred.The authors note that such naturally varying features are difficult to control in real-world screening.Source: Discussion · pp. 9–10
  • Across models, outcomes favored resumes with White and male names; Figure 9 reports White-male resumes preferred over Black-male resumes in 100% of tests.The authors note that the occupations favoring Black or female names were not those with the largest proportions of workers from those groups.Source: Discussion · p. 9, Figure 9 · p. 10
  • Because the study manipulated names alone, it does not capture other possible identity signals such as education, location, and lexical choices in resumes.The authors note that removing names alone may therefore not address all signals to which screening models are sensitive.Source: Discussion · p. 10
  • The study's data-related limitations include truncated resumes and externally assigned occupation codes that may be less accurate than manual coding.These constraints may limit conclusions about particular occupations.Source: Discussion · p. 10
  • The authors call for auditing proprietary and open-source screening systems and for accounting for structural power imbalances in how these tools are conceptualized and used.They frame comprehensive auditing as necessary to evaluate fairness and improve or remove systems accordingly.Source: Discussion · p. 10

Future Work

The authors identify broader model, identity-group, and resume-length coverage as priorities for future research on risks and transparency in LLM resume screening.

Future Work

  • Future work should assess additional MTEs, more diverse social-group signals, and additional populations beyond White/Black and male/female names, as well as realistic resume-length variations.The authors note that hiring discrimination extends beyond the groups and name signals studied.Source: Future Work · p. 10

Conclusion

The conclusion summarizes a retrieval-based simulation of LLM resume screening, finding societal-default preferences, intersectional disadvantages, and effects of resume length and name frequency.

Conclusion

  • The study finds that models reinforce White and male defaults rather than occupational biases, that Black men face the greatest intersectional disadvantage, and that resume length and name frequency can affect biased outcomes.The authors also report confirming three intersectionality hypotheses in this setting.Source: Conclusion · p. 10

Ethical Considerations Statement

The ethical discussion highlights the tension between using realistic identity signals in resume audits and protecting candidates' privacy. It also notes that representing race and gender through names is reductive and incomplete.

Ethical Considerations Statement

  • Auditing resumes raises a tension between preserving privacy and distributing documents that realistically reflect identity signals such as race and gender.Resumes contain substantial identifiable information, making privacy and confidentiality central concerns.Source: Ethical Considerations Statement · p. 11
  • Audit studies commonly use names to represent race and gender, but this is a reductive and incomplete representation of those identities.The authors call for extending the study as more sophisticated representations of protected groups are developed.Source: Ethical Considerations Statement · p. 11

Appendix: Task Instructions

Table 3 lists the instructions appended to each job description before embedding generation.

Appendix: Task Instructions

  • Table 3 lists the instructions appended to each job description before embedding generation.Source: Table 3 · p. 14

Appendix: Length and Frequency Plots

The appendix reports name-based comparisons, document-length context, and detailed cosine-similarity results. It also shows that same-occupation resumes score higher than different-occupation resumes across the listed models and instructions.

Appendix: Length and Frequency Plots

  • Figure 10 compares selected-group proportions for resumes containing only names and occupation titles, with gray regions marking disparities not significantly different from zero.Source: Figure 10 · p. 15
  • 51.9% of 27 tests preferred resumes with Black names over White names when names were matched by exact corpus frequency, versus 22.2% preferring White names; 25.9% showed no significant disparity.Source: Figure 11 · p. 15

Appendix: Names

  • White first names used in the main analysis were 5.5 times more frequent in the corpus than corresponding Black names, while supplemental White names had roughly equal frequencies.Source: Table 4 · p. 16, Table 5 · p. 17, Table 6 · p. 18

Appendix: Document Lengths

  • Before truncation, resumes averaged 1,154 tokens and varied widely in length; job descriptions were shorter and were not truncated during preprocessing.Source: Figure 12 · p. 19, Figure 13 · p. 19

Appendix: Detailed Instructions Results

  • Same-occupation resumes have significantly higher cosine similarities than different-occupation resumes for every model and task instruction corresponding to Table 3 (p<0.0001).Source: Figure 14 · p. 20

Appendix: Detailed Cosine Similarity Results

  • Cosine similarities were significantly higher for male Accountants and Auditors and Misc. Managers, and for White Chief Executives, Marketing and Sales Managers, and Designers; Misc. Managers favored Black names in another comparison.Source: Figure 15 · p. 21, Figure 16 · p. 22
  • White female names exceeded White male names for Misc. Managers; White male names exceeded Black male names in four occupations, while Black female names exceeded Black male names in six.Source: Figure 17 · p. 22, Figure 18 · p. 23, Figure 19 · p. 23
  • Comparisons between Black and White female names varied by occupation and model; title-only results also varied by group, while equal-frequency Black names scored higher in three occupations.Source: Figure 20 · p. 24, Figure 21 · p. 24, Figure 22 · p. 25
Loading 2407.20371v3…