Source-linked AI summary
What does it mean to solve the problem of discrimination in hiring? Social, technical and legal perspectives from the UK on automated hiring systems
Javier Sanchez-Monedero, Lina Dencik, Lilian Edwards
TL;DR
The paper examines how three UK-used automated hiring systems understand and attempt to mitigate discrimination, addressing limited scrutiny of such claims outside the US. It finds that UK and EU data-protection requirements create fundamental legal-fit problems, while transparency rights may challenge wholly automated hiring decisions.
Problem
Bias-mitigation claims by automated hiring systems have been rarely scrutinised, with existing analysis focused primarily on the US rather than Europe.
Method
The paper critically examines HireVue, Pymetrics, and Applied using publicly available documents, then situates their design, validation, and bias-auditing approaches in the UK socio-legal context.
Results
The systems provide some information about their workings and approaches to discrimination, but transferring US-developed systems to the UK introduces fundamental legal problems of fit, particularly under data-protection law.
Takeaways & Limitations
GDPR transparency rights may help address candidate–employer information asymmetry and may prohibit wholly automated hiring decisions, though systems developed outside the EU may ignore or poorly implement these rights.
Takeaways & Limitations
The systems’ bias-mitigation frameworks appear limited to single-axis identification and do not substantially address intersectional discrimination.
Abstract
from arXiv · showhide
The ability to get and keep a job is a key aspect of participating in society and sustaining livelihoods. Yet the way decisions are made on who is eligible for jobs, and why, are rapidly changing with the advent and growth in uptake of automated hiring systems (AHSs) powered by data-driven tools. Key concerns about such AHSs include the lack of transparency and potential limitation of access to jobs for specific profiles. In relation to the latter, however, several of these AHSs claim to detect and mitigate discriminatory practices against protected groups and promote diversity and inclusion at work. Yet whilst these tools have a growing user-base around the world, such claims of bias mitigation are rarely scrutinised and evaluated, and when done so, have almost exclusively been from a US socio-legal perspective. In this paper, we introduce a perspective outside the US by critically examining how three prominent automated hiring systems (AHSs) in regular use in the UK, HireVue, Pymetrics and Applied, understand and attempt to mitigate bias and discrimination. Using publicly available documents, we describe how their tools are designed, validated and audited for bias, highlighting assumptions and limitations, before situating these in the socio-legal context of the UK. The UK has a very different legal background to the US in terms not only of hiring and equality law, but also in terms of data protection (DP) law. We argue that this might be important for addressing concerns about transparency and could mean a challenge to building bias mitigation into AHSs definitively capable of meeting EU legal standards. This is significant as these AHSs, especially those developed in the US, may obscure rather than improve systemic discrimination in the workplace.
1 INTRODUCTION
Automated hiring systems are expanding while claims that they mitigate discrimination remain insufficiently scrutinised, especially outside the US. The paper examines three UK-used systems through publicly available materials and situates their approaches within UK and EU legal contexts.
- Automated hiring systems increasingly transform sourcing, screening, interviewing, and selection while promising efficiency, cost savings, and reduced discrimination.
- Research on bias mitigation in automated hiring systems remains early-stage, primarily US-focused, and comparatively underdeveloped in Europe.
- The paper examines three UK-used systems using public documents to assess how they design, validate, and audit tools for bias.
- UK and EU data-protection rights, including transparency protections, provide a legal perspective largely absent from US hiring-algorithm research.
- The analysis treats discrimination and bias as contested concepts, with fairness definitions varying by context and procedural and substantive notions potentially conflicting.
- The assessment does not test the systems’ general scientific validity or effectiveness in evaluating candidates because independent studies were unavailable.
- US-developed bias-mitigation practices may export US legal and societal assumptions to UK workplaces, while opaque systems can leave embedded discrimination unchallenged.
2 THE DATA-DRIVEN HIRING FUNNEL
Automation changes hiring decisions across the funnel, but systems’ definitions of fit and bias can reproduce unequal access while obscuring accountability and power relations. Technical mitigation focused narrowly on model interfaces may therefore miss broader workplace dynamics.
- Automated hiring changes eligibility decisions across sourcing, screening, interviewing, and selection, amid ethical and fairness concerns.
- Hiring for organisational fit gives employers substantial discretion and often relies on historical data about companies or top performers.
- Algorithmic hiring can scale impacts, standardise techniques, obscure accountability, and create a veneer of objectivity despite discrimination at multiple funnel stages.
- Automated hiring can deepen labour-management information asymmetry and constrain candidates through incomplete profiles, proxies, and inferences.
- Bias mitigation typically targets model technicalities and unconscious bias rather than who can adapt to, or is excluded by, quantified assessment practices.
- Ignoring automation’s wider effects on power relations may neutralise challenges in ways that facilitate discrimination under a banner of fairness.
3 TECHNOLOGICAL BIAS AUDITING AND MITIGATION IN HIRING
The three UK-used systems frame bias mitigation differently: Pymetrics and HireVue automate candidate assessment using statistical or model-based techniques, while Applied supports human monitoring through analytics and process interventions.
- 3.1 Pymetrics: Pymetrics profiles cognitive, social, and emotional traits from game behaviour, then compares candidates with models trained on top performers for each role.Candidates receive an aggregated fit-to-role score and are categorized as in-group or out-of-group.
- 3.1 Pymetrics: Pymetrics evaluates fairness across age, gender, and ethnicity using statistical comparisons of individual game scores and aggregated fit-to-role scores.Its audit-AI tool also checks US EEOC fair-treatment requirements, including the 4/5ths rule.
- 3.2 HireVue: HireVue extracts categorical, audio, and video indicators from automated interviews and games designed through Industrial-Organizational psychology research.Its bias strategy includes removing indicators known to adversely affect protected groups and modifying the learning objective for fairness.
- 3.2 HireVue: HireVue’s corrected objective aggregates errors across protected groups, normalizes group influence, and can add penalties for violations of the 4/5ths rule.The proposed penalty depends on a user-specified cost and checks demographic parity in the candidate-evaluation model.
- 3.3 Applied: Applied combines gendered-language analysis, candidate anonymization, chunked assessments, randomized review, multiple scorers, and visual demographic analytics.Unlike Pymetrics and HireVue, it does not automatically assess candidates; it semi-automates discrimination monitoring and mitigation.
4 EVALUATING CLAIMS OF BIAS MITIGATION
The systems operationalize fairness through anti-classification, equal passing rates, or outcome independence, but their audits remain constrained by group definitions, historical reference data, and limited treatment of intersectionality. These computational limitations motivate examining the national legal contexts in which the systems are deployed.
- Fairness concepts: The three systems use varying degrees of anti-classification, classification parity, and calibration to operationalize fairness.Examples include omitting group variables or proxies, comparing passing rates, and requiring outcomes to be independent of group variables.
- Group definitions: Available validation examples define groups mainly by binary gender, US-informed ethnicity categories, and age intervals, while omitting social class and disability.HireVue and Pymetrics audit classification parity through the 4/5ths rule, and Pymetrics additionally compares group scores and passing rates.
- Reference data: Using current or past employees, especially high performers, as reference data may reproduce historical injustice through the algorithmic definition of best performance or fit.Fairness metrics are further limited when rejected candidates are absent, preventing analysis of disparate mistreatment through suitable applicants who were wrongly rejected.
- Intersectionality: The systems largely use single-axis group comparisons, leaving intersectional discrimination insufficiently addressed.Pymetrics compares attributes separately rather than examining combinations such as Hispanic women and white men.
- Computational limitations: HireVue’s abstract penalty-based fairness proposal lacks validation reports, and adding many penalty terms can reduce each term’s influence and create convergence problems.The broader difficulty is expressing intersectionality as a multiple-objective optimization problem.
- Legal context: These computational limitations are evaluated without accounting for the different national contexts and legal frameworks in which AHSs are deployed.The paper therefore turns to the legal framework governing deployment.
5 LEGAL FRAMEWORK
UK employment equality law lacks a prescribed statistical threshold for proving bias, while GDPR provides transparency, access, and objection rights relevant to automated hiring. However, uncertainty remains over when human involvement avoids Article 22 and what meaningful algorithmic information requires.
- US-developed systems may not be optimised for UK/EU data-protection rights, which may be as important as or more important than equality rights for uncovering bias.
- UK equality law provides no fixed outcome ratio for proving bias; significance depends on context, pool size, and underlying proportions.
- Indirect discrimination may be justified when proportionate to a legitimate aim, but these open-textured concepts are difficult to encode in hiring tools.
- GDPR grants data subjects transparency, access, and objection rights concerning solely automated decisions with legal or similarly significant effects.
- Fully automated hiring decisions may trigger Article 22 protections, although valid employment consent and the boundary between hiring and employment remain contested.
- Human rubber-stamping may prevent Article 22 from applying, but GDPR leaves unclear how much genuine human interaction is sufficient.
- Article 15(h) may support meaningful information about automated decision-making, while DPIAs for high-risk processing should consider and address unfairness and bias.
6 DISCUSSION
The three AHSs offer some insight into how automated hiring systems conceptualise and address discrimination, but their bias-mitigation approaches face computational, accountability, transparency, and legal-fit problems. The paper calls for closer assessment of how AHSs should be used to protect candidates and employees.
- The systems provide unusually accessible information about their workings, although exact data sources and models remain obscure and client-dependent.
- Computational fairness is limited by predictive data, quantifying candidate fit, reductionist group categories, and neglect of intersectionality.
- Accountability remains unsettled because responsibility for transparency and bias mitigation may fall on system builders, employers, or another actor.
- US-developed AHSs may fit UK and EU law poorly, particularly GDPR requirements that can challenge information asymmetry or wholly automated hiring.
- The paper identifies a need to extend algorithmic-bias research beyond hiring to firing and in-work conditions in the UK and Europe.
- Limited information about AHS operation, bias mitigation, and deployment is a significant problem, especially because their benefits to employers remain unclear.