PaperlayerView exact revision on arXiv

Source-linked AI summary

No Thoughts Just AI: Biased LLM Hiring Recommendations Alter Human Decision Making and Limit Human Autonomy

Kyra Wilson, Mattea Sim, Anna-Maria Gueorguieva, Aylin Caliskan

arXiv:2509.04404v2cs.CYcs.AIcs.CLcs.HC

TL;DR

Whether people can counteract biased AI recommendations in hiring is an open question, with implications for autonomy in high-stakes human-in-the-loop decisions. The study tested 528 participants in 1,526 resume-screening scenarios, examining interactions with racially biased AI across three racial/ethnic comparisons and whether an IAT preceded the task. People selected candidates equally without AI or with unbiased AI, but favored AI-recommended groups up to 90% of the time; completing an IAT beforehand increased stereotype-incongruent selections by 13%.

  • Problem

    Whether people can counteract biased AI recommendations in hiring is an open question, with implications for autonomy in high-stakes human-in-the-loop decisions.

    Source: §1 · p. 2

  • Method

    The study tested 528 participants in 1,526 resume-screening scenarios, examining interactions with racially biased AI across three racial/ethnic comparisons and whether an IAT preceded the task.

    Source: §1 · p. 2

  • Results

    People selected candidates equally without AI or with unbiased AI, but favored AI-recommended groups up to 90% of the time; completing an IAT beforehand increased stereotype-incongruent selections by 13%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    AI-HITL workflows as currently implemented may not effectively mitigate AI bias, and evaluation infrastructure is needed to study risks for multiply marginalized groups.

    Source: §6 · p. 10, §5.2 · p. 10

    The simulated resume-screening setting should be complemented by other screening paradigms and qualitative or observational research with experienced hiring professionals.

    Source: §5.4 · p. 10

Abstract

from arXiv · show

In this study, we conduct a resume-screening experiment (N=528) where people collaborate with simulated AI models exhibiting race-based preferences (bias) to evaluate candidates for 16 high and low status occupations. Simulated AI bias approximates factual and counterfactual estimates of racial bias in real-world AI systems. We investigate people's preferences for White, Black, Hispanic, and Asian candidates (represented through names and affinity groups on quality-controlled resumes) across 1,526 scenarios and measure their unconscious associations between race and status using implicit association tests (IATs), which predict discriminatory hiring decisions but have not been investigated in human-AI collaboration. When making decisions without AI or with AI that exhibits no race-based preferences, people select all candidates at equal rates. However, when interacting with AI favoring a particular group, people also favor those candidates up to 90% of the time, indicating a significant behavioral shift. The likelihood of selecting candidates whose identities do not align with common race-status stereotypes can increase by 13% if people complete an IAT before conducting resume screening. Finally, even if people think AI recommendations are low quality or not important, their decisions are still vulnerable to AI bias under certain circumstances. This work has implications for people's autonomy in AI-HITL scenarios, AI and work, design and evaluation of AI hiring systems, and strategies for mitigating bias in collaborative decision-making tasks. In particular, organizational and regulatory policy should acknowledge the complex nature of AI-HITL decision making when implementing these systems, educating people who use them, and determining which are subject to oversight.

1 Introduction

The paper examines whether people can counteract biased AI hiring recommendations in human-in-the-loop decisions, and whether individual traits or bias training alter their responses. In a resume-screening experiment, people’s choices closely followed biased AI recommendations, while completing an IAT beforehand increased selection of stereotype-incongruent candidates.

1 Introduction

  • The study asks whether human-in-the-loop hiring can counteract AI bias, given that human hiring decisions can themselves be biased and AI bias may threaten autonomy.The authors describe autonomy as the capacity to act on one’s own beliefs, values, motivations, and reasons.Source: §1 · p. 2
  • The study examines whether prior experience, perceptions, explicit beliefs, and implicit associations relate to responses to biased recommendations.Participants completed an IAT before or after screening, and evaluated White, Black, Hispanic, or Asian male candidates represented through names and affinity-group information.Source: §1 · p. 2
  • The experiment included 528 participants and 1,526 resume-screening scenarios across 16 high- and low-status occupations.Candidates were evaluated in a simulated resume-screening task with AI recommendations varying in favored racial identity and bias magnitude.Source: §1 · p. 2
  • People selected White and non-White candidates equally without AI or with unbiased AI, but selected AI-favored candidates up to 90% of the time when recommendations were biased.The authors report that completing an IAT before screening increased selection of stereotype-incongruent candidates by 13%.Source: §1 · p. 2
  • Perceived recommendation quality and importance moderated decisions, while prior hiring or AI experience, IAT scores, and explicit race-status beliefs did not significantly contribute to model fit.The authors identify AI literacy interventions as an area worth further investigation.Source: §1 · p. 3

2 Related Work

Prior work examines reliance on AI recommendations, psychological and task-related influences, and implicit associations in hiring. The paper addresses limited investigation of racial implicit associations and counterfactual AI biases in human-AI hiring decisions.

2 Related Work

  • People may follow incorrect AI advice, and reliance can vary with explanations, psychological factors, and task difficulty.Prior studies report that explanations are not universally effective and that explanations may be more valuable on difficult tasks.Source: §2 · p. 3
  • Hiring AI can be biased and carry possible legal consequences, yet little work investigates implicit bias as a moderator of AI-HITL decisions.Prior research linked traits such as extraversion and self-confidence to recruiters’ trust in unbiased AI recommendations.Source: §2 · p. 3
  • Implicit associations relate to hiring outcomes, but racial associations with specific beliefs had not been studied in AI-HITL hiring.The cited work notes that IATs measure associations through reaction-time differences when sorting words or pictures.Source: §2 · p. 3
  • Prior studies of biased AI interactions focused on observed or stereotype-congruent biases, while findings on whether people amplify or mitigate AI bias were inconsistent.This study uses theoretically informative simulations to examine existing and counterfactual biases and implicit associations in AI-HITL scenarios.Source: §2 · p. 3

3 Data and Methods

The study used a mixed-factorial resume-screening experiment to examine how recommendation bias, candidate race, task order, and job status relate to hiring choices.

3 Data and Methods

  • The 6×3×2×2 mixed-factorial design varied AI recommendation, candidate race, task order, and job status.Participants saw four scenarios, including no recommendation, neutral recommendations, and either congruent or incongruent recommendations at different bias magnitudes.Source: §3 · pp. 3–4

Occupations and Descriptions

  • The 16 occupations comprised eight high-status and eight low-status jobs selected to vary in status and worker demographics.The occupations were chosen using salary-based status categories and information about workforce racial distributions.Source: Occupations and Descriptions · p. 4

Resumes

  • Candidate resumes used validated work histories, with names and affinity-group memberships signaling racial identities.The four work histories selected per occupation had similar validation scores; White candidates had no explicit race label.Source: Resumes · pp. 4–5

AI Recommendations

  • AI recommendations ranged from neutral to severe and modeled both stereotype-congruent and counter-stereotypical preferences.Moderate bias approximated rates from LLM resume-screening simulations, while severe conditions recommended candidates from one racial group and none from the other.Source: AI Recommendations · p. 5

IATs, Explicit Beliefs, and Survey Questions

  • Participants completed a race-status IAT before or after screening, and reported explicit beliefs, AI perceptions, and hiring or AI experience.Explicit beliefs were assessed with competence-related questions about White and non-White groups.Source: IATs, Explicit Beliefs, and Survey… · p. 5

3.2 Participants

  • Of 575 US participants recruited, 528 had usable data.Participants were recruited through Prolific and completed an experiment lasting approximately 25 minutes.Source: §3.2 · p. 5

3.3 Experimental Procedure

  • Participants selected three of five candidates after reviewing each occupation, resumes, and recommendations in four-minute scenarios.The first trial had no AI recommendations; the remaining three used recommendation conditions in randomized order.Source: §3.3 · pp. 5–6

3.4 Analysis

  • The analysis used binomial logistic mixed models to estimate candidate-preference probabilities and test recommendation, job-status, race, and task-order effects.Exploratory models assessed interactions with IAT scores, explicit beliefs, experience, and AI perceptions; stepwise elimination selected predictors for further investigation.Source: §3.4 · pp. 6–7

4 Results

Participants’ preferences varied with AI recommendations and job status, and completing the IAT first was associated with shifts toward stereotype-incongruent choices. Recommendation perceptions did not fully predict behavioral responses.

4.1 AI Recommendation, Race, Task Order, Job Status

  • In severe high-status conditions, participants favored White or non-White candidates 90% of the time under Congruent or Incongruent recommendations, respectively.AI Recommendation had a significant main effect, and AI Recommendation interacted significantly with Job Status.Source: §4.1 · pp. 7–8
  • Most biased recommendations produced significantly different choices from no or neutral recommendations, while no-versus-neutral comparisons were not significant.The reported exceptions to significant differences included specific Moderate recommendation comparisons for low- and high-status jobs.Source: §4.1 · p. 8
  • Participants’ preferences closely tracked AI recommendation rates in Neutral and Moderate conditions, but differed significantly from them in Severe conditions.The severe-condition differences occurred for both recommendation directions and both job-status levels.Source: §4.1 · p. 8
  • Completing the IAT first was associated with a 13.0% increase in preference for Black and a 12.7% increase for Hispanic candidates in high-status jobs.For White-versus-Asian comparisons, the trend was a 12.6% decrease in preference for Asian candidates in low-status jobs; no post-hoc pairwise comparisons were significant.Source: §4.1 · p. 8

4.2 Exploratory Factors

  • IAT scores showed stronger White–high-status associations than Black, Asian, or Hispanic comparisons, but IAT and explicit-belief scores did not significantly contribute to model fit.Hiring experience and AI familiarity also did not significantly contribute; AI recommendation quality and importance did contribute through three-way interactions with job status and recommendation.Source: §4.2 · p. 8
  • Even participants who rated recommendations as poor quality or unimportant still shifted their choices in biased-recommendation scenarios.Among poor-quality raters, White preference for high-status jobs was 44.6% lower in Incongruent/Severe than in None; among those calling recommendations unimportant, choices changed 49.3% on average in Incongruent/Severe conditions.Source: §4.2 · p. 8, Figure 5 · p. 9

5 Discussion

Biased AI recommendations can shift hiring choices toward favored candidates, either reinforcing or countering race-status stereotypes depending on the job context. The discussion proposes system evaluation, bias-awareness training, and AI literacy, while noting limits to the study’s simulated screening setting.

5.1 Societal Impacts of Bias Propagation

  • Participants’ choices tracked biased AI recommendations, with high-status jobs pairing congruent recommendations with more White selections and incongruent recommendations with more non-White selections.Without AI or with equal race preferences, selection rates did not significantly differ across races.Source: §5.1 · p. 9

5.2 Designing Systems to Mitigate Biases

  • Real-world evaluation infrastructure is needed alongside less-biased systems, especially to study harms to people with intersecting marginalized identities.The authors note that third-party fairness audits are rare and companies’ fairness statements are often vague.Source: §5.2 · pp. 9–10
  • Completing an IAT before screening was associated with less stereotypical decisions under biased AI, while IAT scores themselves did not predict decisions.The authors call for further evaluation of AI-HITL interactions, not only final decisions.Source: §5.2 · p. 10

5.3 Strengthening the ‘Human’ in AI-HITL

  • AI literacy should teach people to calibrate judgments of AI performance and recognize biases beyond familiar race-status stereotypes.Participants who considered recommendations unimportant were still influenced by biases favoring less stereotypically associated candidates.Source: §5.3 · p. 10

5.4 Limitations

  • The simulated screening paradigm leaves other settings, variable candidate counts, and experienced hiring professionals for future study.The authors call for complementary research because simulations cannot fully capture proprietary systems across AI-HITL settings.Source: §5.4 · p. 10

6 Conclusion

In a quality-controlled resume-screening task, participants’ choices reflected biased AI recommendations, while choices were similar across races without biased recommendations. The appendices describe occupation selection and the study’s occupation materials and statistics.

6 Conclusion

  • With biased recommendations, participants’ preferences generally did not significantly differ from the AI’s recommendation probabilities; without AI or with equal race preferences, selection rates were equal.The conclusion states that IATs may increase resilience to biased recommendations and calls for further study of their use in hiring.Source: §6 · p. 10

A Procedure for Selecting Occupations

  • Occupations were selected using wage bands and worker demographics, retaining 16 occupations spanning high and low status and demographic similarity to the US population.The selection procedure began with 22 occupations after exclusions.Source: A Procedure for Selecting Occupations · p. 11

B Prompt for Generating Work Histories

C Descriptions

  • Occupation descriptions informed participants about the responsibilities and duties relevant to resume screening and simulation.The appendix provides descriptions for the occupations used in the study.Source: C Descriptions · pp. 11–12

D Occupation Statistics

  • Statistics for the 16 occupations include average annual salary and worker gender and race, drawn from the 2022 American Community Survey five-year estimates.Source: D Occupation Statistics · p. 12

E Work History Validation Experiment

The validation experiment aimed to identify similarly qualified generated work histories for the resume-screening study. Participants rated resumes for occupations under a controlled procedure, and the selected histories excluded identity information until after validation.

E Work History Validation Experiment

Occupation Descriptions

  • The work histories were generated for the occupations in the main study, with start and end dates standardized and identity information omitted until after validation.The researchers generated eight histories per occupation and used the four most similar validation scores as screening stimuli.Source: Occupation Descriptions · p. 12

E.2 Participants

  • Forty-two people completed validation, with two excluded for failing attention checks; participants reported varying hiring or management experience.35.7% reported no such experience, 54.8% a small amount, and 9.5% a great deal.Source: §E.2 · p. 12

E.3 Procedure

  • Participants rated one resume at a time on a five-point qualification scale and completed 16 trials across four randomly selected occupations.A 30-second minimum reading time and attention checks were used; trial order was balanced across occupations.Source: §E.3 · p. 12

E.4 Analysis

The analysis modeled ordinal resume ratings while accounting for participant-level response correlations, then selected the four histories with the smallest aggregate pairwise rating differences.

E.4 Analysis

  • The analysis used a cumulative-link mixed-effects model for ordinal ratings, with resume as a fixed effect and participant as a random effect.Source: §E.4 · p. 13
  • Rather than apply a preset equivalence threshold, the researchers searched for the subset of four resumes with the smallest total pairwise odds-ratio effect sizes.For each four-resume subset, six pairwise effect sizes were summed.Source: §E.4 · pp. 13–14

E.5 Results

Participants rated the work histories as highly qualified overall, and these validated resumes were then used to estimate AI resume-screening bias for the main experiment.

E.5 Results

  • 3.95 was the average resume rating (SD=.99), indicating participants generally considered candidates very qualified.Of 640 collected ratings, 624 were analyzed; each resume had an average of 4.875 ratings.Source: §E.5 · p. 14
  • Twelve of 16 occupation-specific resume subsets had weak average effects (OR ≤2), three moderate effects, and one strong effects.Resumes were selected to be functionally similar in quality, but small quality differences may remain; main-experiment models included a resume random effect to account for them.Source: §E.5 · p. 14

F AI Resume Screening Simulation

  • The resume-screening simulation estimated bias in existing Massive Text Embedding models for use in the main human-subjects experiment.The approach treated job descriptions as queries and resumes as documents, with similarity indicating fit.Source: F AI Resume Screening Simulation · p. 14

F.1 Data and Models

The study simulated resume screening with three text-embedding models, matched candidate resumes and job descriptions, and compared their embedding similarities across racial groups.

Models

  • The simulation used E5-mistral-7b-instruct, GritLM-7B, and SFR-Embedding-Mistral, all fine-tuned from the same pretrained LLM for text-embedding tasks.These tasks include retrieval, clustering, and classification.Source: Models · p. 14
  • The 64 validated work histories were expanded into 16,384 documents using name combinations and additional interests that could signal racial identity.The resumes were embedded with the three models for retrieval against job descriptions.Source: Models · pp. 14–15
  • For each occupation, the selected resume subset minimizes Equation 4 and is the most similar subset; in most best subsets, average pairwise effect-size differences are small (OR ≤2).Source: Table 5 · p. 15

Job Descriptions

  • Each of 16 occupations had eight manually verified job-description variations, yielding 128 descriptions and 1,280 unique task-instruction-prefixed embeddings.The descriptions were varied to reduce dependence on nuances in a single description.Source: Job Descriptions · p. 15

F.2 Approach

  • Resume and job-description embeddings were compared using cosine similarity, then averaged across task-instruction variations for each occupation description.The analysis followed the document-retrieval framework described by Wilson and Caliskan (2024).Source: §F.2 · p. 15
  • The analysis made 3.9 million resume–description comparisons across three models, reduced to approximately 393k scores after averaging task-instruction variations.Within each racial contrast, the top 10% of resumes per description, occupation, and racial group were tested for selection-rate differences across 48 model-by-occupation tests.Source: §F.2 · p. 15

F.3 Results

The appendix details the simulated screening results and the measures and task procedures used to examine racial status associations, AI perceptions, and participant responses.

F.3 Results

  • Across 48 tests per contrast, Black candidates were preferred in 6.25% of white–Black tests, Hispanic candidates in 6.25% of white–Hispanic tests, and Asian candidates in 10.42% of white–Asian tests.White candidates were preferred in most tests; equal-rate results occurred in 2.08% of white–Black and 20.83% of white–Hispanic tests.Source: §F.3 · p. 16, Figure 10 · p. 17, Figure 11 · p. 17

G.1 Status Attributes

  • The IAT used high- and low-status attributes including capable/incapable, competent/incompetent, able/unable, worthy/unworthy, and skilled/unskilled.White–Black and white–Asian targets were faces; white–Hispanic targets were surnames.Source: §G.1 · p. 16, §G.2 · p. 17, Figure 13 · p. 18

H Explicit Beliefs Questionnaire

  • Participants rated 16 randomized explicit-belief questions about how society views racial groups on a five-point scale.Questions covered traits such as confidence, competence, independence, and occupational prestige.Source: H Explicit Beliefs Questionnaire · p. 17

I Survey Questions

  • Participants also reported AI recommendation importance and quality, hiring experience, and familiarity with AI in hiring using three-point response scales.The analysis examined predicted preference for white candidates by reported AI importance and quality across six bias magnitudes.Source: I Survey Questions · pp. 17–18, J Exploratory BLMM · p. 18

K Task Instructions

  • The task asked participants to screen resumes for interviews, sometimes showing AI recommendations as green checks or red crosses.Participants reviewed four scenarios with four minutes allotted to each.Source: Decision-Making · p. 18

L Debrief

  • The debrief described the study as testing whether implicit associations relate to choices when information aligns or conflicts with social associations.The IAT was used to evaluate participants’ implicit associations.Source: L Debrief · p. 19

M IAT Interface

  • The IAT interface used seven blocks, with category pairings changing among racial groups, status words, or combinations of both.Instructions disappeared when the first item appeared at the fixation cross.Source: M IAT Interface · p. 19

N AI Recommendation, Race, Task Order, Job Status R Output

This section reports model outputs and exploratory analyses, then describes a penalized-regression check of the exploratory findings and its scope limits.

N.1 BLMM

  • The BLMM output is reported with fixed-effect coefficients, AIC, log likelihood, and random-effect variances.The corresponding output is shown in Table 7.Source: §N.1 · pp. 19–20, Table 7 · p. 21
  • Figure 15 plots predicted white-candidate preference by reported AI recommendation quality, with X marking the proportion of recommendations favoring white candidates.The figure covers all six bias magnitudes.Source: Figure 15 · p. 20

N.2 ANOVA

  • The omnibus ANOVA output for the fitted BLMM is reported, with pairwise comparisons of significant omnibus effects presented in the main text.The output is shown in Table 8.Source: §N.2 · p. 20, Table 8 · p. 21

O Exploratory Descriptive Statistics

  • Participants’ prior hiring experience was moderately associated with familiarity with AI hiring, and perceived AI recommendation quality was moderately associated with importance.The section also presents distributions of exploratory measures and associations among categorical variables.Source: O Exploratory Descriptive Statistics · p. 20

P.1 Approach

  • Group penalized elastic-net regression was used to check exploratory BLMM findings and identify predictors of screening decisions.Because the method cannot account for random effects, the authors applied it to a binomial logistic model; random effects explained little variance (ICC=.052).Source: §P.1 · pp. 20–21
  • The GENR prediction analysis trained on 80% of 1,526 trials and evaluated on the remaining 20%, using 10-fold cross-validation to select hyperparameters.The best-performing model used α = 0.01 and λ = .515.Source: §P.1 · p. 21

P.2 Results

The exploratory GENR analysis predicts resume-screening choices better than the GLMM, with AI recommendation and job status especially important. A separate variable-importance approach also identifies recommendation quality and importance as relevant to model fit.

P.2 Results

  • 33.784% of test trials were misclassified by GENR, compared with 37.838% by the GLMM, indicating better predictive performance for the sparse model.The analysis describes this as evidence that a sparse model predicts better than one including all possible variables and interactions.Source: §P.2 · pp. 21–22
  • 29.57% of variance was explained by the AI Recommendation × Job Status interaction, the largest contribution reported for the GENR model.Interactions of recommendation quality, explicit score, recommendation importance, AI experience, and IAT score with recommendation and job status each explained at least 5% of variance.Source: §P.2 · pp. 22–23
  • Recommendation quality and importance predicted decisions, while explicit beliefs and IAT scores were useful for prediction despite not emerging as useful for causal explanation.The exploratory analysis also reports that prior hiring or AI experience and implicit biases and explicit beliefs did not moderate hiring decisions, whereas perceived recommendation quality and importance did.Source: §P.2 · p. 23

Q.1 Approach

  • Variable importance was measured as deviance explained by predictors alone or in combination, using a linear model because importance methods for mixed-effects models remain an open research area.The analysis used a BLM rather than a BLMM to assess importance and check exploratory findings from backward stepwise elimination.Source: §Q.1 · p. 23

Q.2 Results

The variable-importance analysis finds that the AI Recommendation × Job Status interaction dominates explained deviance. It also confirms that recommendation quality and importance contribute to model fit.

Q.2 Results

  • 76.7% of total deviance was explained by the Job Status × AI Recommendation interaction, versus 0.1% for Job Status and 14.5% for AI Recommendation alone.Table 9 presents predictor importance as percentages of explained variance in the fit GENR.Source: §Q.2 · p. 23, Table 9 · p. 24
  • AI recommendation importance explained 2.1% of deviance, recommendation quality 0.9%, and hiring experience 0.5%.Recommendation quality and importance were also identified by the alternative variable-importance analysis as relevant to model fit.Source: §Q.2 · pp. 23–24
Loading 2509.04404v2…