PaperlayerView exact revision on arXiv

Source-linked AI summary

Counterfactual, Per-Decision Bias Auditing for Automated Hiring: Localizing and Explaining Disparate Impact in Applicant Tracking Systems

Jay Barach

arXiv:2608.21537v1cs.CYcs.CR

TL;DR

Automated hiring systems need audits that connect population-level fairness standards to individual decisions, but existing group metrics and local explainers leave that connection incomplete. AIBF tests each deployed decision by neutralizing protected proxies and measuring the counterfactual score shift, yielding localized explanations and strong detection performance on real data. Its correction experiment nevertheless shows that residual proxy correlation in merit features prevents legal parity.

  • Problem

    Group fairness metrics summarize populations while local explainers attribute individual predictions, leaving no tool that both localizes decisions and connects them to legal fairness standards.

  • Method

    AIBF neutralizes protected-attribute proxies, re-scores each candidate with the deployed scorer, and uses the counterfactual shift to flag and explain affected decisions.

  • Results

    On two real datasets, AIBF’s per-decision signal is faithful to group disparity and detects counterfactually flipped decisions with an area under the ROC curve of 0.963 versus 0.672 for a group membership baseline.

  • Takeaways & Limitations

    AIBF turns a population-level audit obligation into a ranked, explained worklist for reviewing individual decisions.

  • Takeaways & Limitations

    Correction improves disparate impact but does not reach parity because merit features retain residual proxy correlation, and the audit measures scorer influence rather than real-world causation.

Abstract

from arXiv · show

Automated applicant tracking systems increasingly decide who advances in hiring, and litigation and regulation now demand that those decisions be auditable. Existing tools sit at two extremes. Group fairness metrics such as the disparate impact ratio summarize a whole population but cannot say which individual decisions were unfair or why, while local explainers such as SHAP attribute a single prediction but are not connected to the legal standard by which hiring bias is judged. We present the AI Bias Firewall (AIBF), a method that audits an applicant tracking system one decision at a time. AIBF neutralizes a candidate's protected-attribute proxies, re-scores the decision, and measures the resulting counterfactual shift, which yields a signed per-decision bias in score points, a flag for decisions the protected attributes changed, and a plain-language explanation naming the responsible factors. We evaluate on two real public datasets, Adult and COMPAS, rather than on synthetic data. The per-decision counterfactual shift is faithful, aggregating to reproduce the known group level disparity, for example a mean shift of +7.5 points for the privileged group and -8.0 for the disadvantaged group on Adult, consistent with the measured statistical parity difference. AIBF identifies the decisions that protected attributes flipped with an area under the ROC curve of 0.963 on Adult, against 0.672 for a baseline that flags by group membership, and it identifies the harmed candidates so precisely that reviewing only five percent of decisions surfaces fifty-five percent of them, against six percent under group based review. We also report a limitation: correcting flagged decisions raises the disparate impact ratio substantially but not to legal parity, because features labeled as merit carry residual proxy correlation. AIBF is released under the Apache 2.0 license with code and experiments.

I. INTRODUCTION

Automated hiring decisions increasingly require individual-level auditability, but group metrics and local explainers address different parts of that need. AIBF connects counterfactual per-decision explanations to population-level fairness standards and evaluates the approach on real data.

  • Motivation: Automated screening increasingly shapes hiring decisions, while litigation and regulation demand evidence about how individual decisions were made.Audit obligations require more than aggregate selection rates or impact ratios.
  • Problem: Group fairness metrics identify aggregate disadvantage but cannot locate unfair decisions, explain their factors, or prioritize a limited human review budget.Local explainers provide individual attributions but are not connected here to the legal fairness standard.
  • Method: AIBF neutralizes protected-attribute proxies, re-scores each candidate through the same system, and measures a signed counterfactual shift in score points.The procedure produces a per-decision bias value, a flag for changed outcomes, and an explanation naming responsible factors.
  • Contribution: AIBF provides a counterfactual, faithful, explainable per-decision audit that connects individual attributions to the group fairness standard used in law.Its construction is intended to fill the gap between population metrics and local explainers.
  • Evaluation: AIBF detects counterfactually biased decisions with an area under the ROC curve of 0.963, versus 0.672 for a group membership baseline, on real public datasets.The evaluation also reports improved review efficiency.
  • Limitation: Attribution-based correction improves the disparate impact ratio substantially but does not reach parity because merit features retain residual proxy correlation.The paper positions AIBF as a detection and explanation tool rather than an automated remedy.

II. BACKGROUND AND RELATED WORK

The paper situates AIBF among legal fairness metrics, individual and counterfactual fairness, and local explanation methods. Its distinction is operational: it audits a deployed scorer while preserving connection to population-level standards.

  • Group fairness metrics: The four-fifths rule treats a protected group’s selection rate below 0.80 of the highest group’s rate as adverse impact.Related metrics include demographic parity, statistical parity difference, equalized odds, and equal opportunity difference.
  • Group fairness metrics: Group fairness metrics are appropriate compliance targets but operate at population level and cannot localize or explain a single decision.AIBF is designed to explain and localize disparities rather than replace these metrics.
  • Individual and counterfactual fairness: AIBF operationalizes counterfactual fairness by neutralizing protected proxies and measuring whether the deployed scorer’s decision changes.Unlike causal counterfactual fairness, it does not learn a structural causal model.
  • Local explanation: AIBF builds on additive attribution methods but partitions protected proxies from legitimate qualifications and defines its decision signal through a counterfactual change.This distinguishes its audit role from generic local explanation.

D. Fairness toolkits and hiring specific work

Existing fairness toolkits primarily provide group metrics or mitigation libraries, whereas AIBF supplies an auditable per-decision workflow using the deployed scorer. The method holds merit features fixed while testing protected-proxy influence.

  • Fairness toolkits and hiring-specific work: Open fairness toolkits provide group metrics and mitigation algorithms, leaving users to assemble the auditing workflow.The paper characterizes these tools as operating primarily at group level.
  • Problem formulation: A deployed scorer assigns a score and thresholded decision, with features partitioned into merit factors M and protected proxies P.The counterfactual candidate keeps merit features fixed while protected proxies are set to a common privileged baseline.
  • Problem formulation: A biased decision is operationally defined as one whose outcome differs between the observed candidate and the proxy-neutralized candidate.A rejection that would become an advance is classified as a harmed decision.
  • The counterfactual shift: AIBF computes the score contribution of protected proxies relative to the privileged baseline, with negative Δ indicating that protected attributes lowered the score.The per-decision magnitude is |Δ(x)|, and the audit flags outcome changes.
  • The counterfactual shift: The counterfactual flip is determined by comparing thresholded decisions before and after proxy neutralization.Equation (5) uses the system threshold τ and identifies decisions the protected attributes actually determined.

B. Why not a ratio

A protected-attribution ratio is a poor detector of biased decisions because it ignores whether the decision is near the outcome threshold. AIBF instead uses a counterfactual score shift that connects individual bias to group disparity.

  • 0.265 AUC: the protected-attribution ratio detects counterfactual flips worse than chance on real data.The ratio can be high for candidates deep in the reject region even when protected contributions cannot change the outcome.
  • 0.963 AUC: the counterfactual magnitude detects counterfactual flips on the same data.The shift is defined relative to the decision boundary, so it captures whether protected contributions can change the outcome.
  • AIBF fits a linear reference model to deployed scores and reports additive contributions for each protected feature.For the linear reference, these contributions are SHAP values relative to the privileged baseline and support plain-language rationales.
  • The privileged baseline asks whether the decision would differ if the candidate were treated as a member of the advantaged group.A decision using no protected proxy has exactly zero shift under this baseline.
  • Mean counterfactual shifts reproduce the protected proxies’ contribution to the score gap underlying statistical parity difference.The paper reports that group means of the shift match the sign and relative magnitude of independently computed statistical parity differences on two datasets.

F. Properties

AIBF provides sound, model-agnostic per-decision flags and a ranked batch worklist, with an audit cost of two scorer queries per decision. Its completeness is limited to the protected proxies supplied to it.

  • Properties: A flagged decision demonstrably changes outcome when its protected proxies are neutralized, making the flag sound with respect to the deployed scorer.The flag is based on comparing the original and counterfactual threshold outcomes.
  • Properties: AIBF’s decision signal is model agnostic because the counterfactual shift uses two queries to the true scorer, not the linear reference model.The linear reference is used only to itemize the explanation.
  • Properties: 2n scorer queries: a full audit of n decisions requires two forward evaluations per decision.The stated cost is negligible beside training or the human review informed by the audit.
  • Properties: AIBF does not guarantee completeness because unmodeled protected proxies escape detection.The method detects influence only from the proxies provided to the audit.
  • Per-decision audit: Algorithm 1 returns a bias shift, outcome-change flag, and explanation after neutralizing protected features to the privileged baseline.The explanation retains protected-feature contributions whose absolute magnitude exceeds ϵ.
  • Batch audit: Algorithm 2 appends flagged decisions to a ranked worklist and reports group selection rates, disparate impact ratios, and mean shifts.The worklist is sorted by descending absolute counterfactual shift.

V. EXPERIMENTAL SETUP

The evaluation uses two real public fairness datasets, Adult and COMPAS, with documented favorable outcomes, protected attributes, and dataset composition. Figure 1 depicts the per-decision audit applied in this experimental setting.

  • Datasets: Two real public datasets, Adult and COMPAS, are used instead of synthetic data.Preparation and analysis scripts are released.
  • Adult: Adult treats income above fifty thousand dollars as advancement and uses sex, race, and an over-forty age proxy as protected attributes.Education, weekly hours, and work class are treated as merit features.
  • COMPAS: COMPAS uses low risk assessment as the favorable outcome and tests generalization beyond applicant tracking.The dataset is included as a standard fairness benchmark outside hiring.
  • Dataset composition: Adult’s disadvantaged groups comprise 33.2% female, 14.5% non-white, and 43.8% aged forty or over.Before modeling, favorable income rates differ between the listed groups.
  • Audit procedure: Figure 1 scores each candidate, neutralizes protected proxies to a common privileged baseline, and re-scores the candidate.The resulting shift supports per-decision bias measurement, outcome-change flags, and explanations naming responsible protected factors.
  • Dataset composition: Table II defines base rate as the favorable-outcome rate for each group before any model is applied.

B. The audited system

The audited systems are logistic regressions trained on real benchmark data with protected proxies included, and evaluation compares group fairness, detection, and review metrics. The results show severe disparate impact and faithful counterfactual shifts.

  • The audited system: Each audited scorer is a logistic regression trained on all features, including protected proxies, to model a historically biased system.AIBF accesses this scorer only through queries.
  • Feature encoding: Adult encodes education, weekly hours, and work class as merit features, while sex, race, and age proxy are binary protected features.
  • Metrics and baselines: The evaluation reports disparate impact ratio, statistical parity difference, equal opportunity difference, AUC, average precision, and harmed-decision review recall.Group membership is the baseline for detection and targeted review.
  • Audited systems: 0.078 disparate impact ratio: Adult sex; 0.386 disparate impact ratio: Adult race; both fall below the 0.80 four-fifths threshold.COMPAS race has a 0.645 ratio on the favorable outcome, also below threshold.
  • Faithfulness: 7.49 points: protected proxies raise the privileged Adult sex group’s score on average, while 7.95 points lower the disadvantaged group’s score.The resulting 15.44-point gap is consistent in direction and relative size with statistical parity difference −0.170.
  • Faithfulness: Mean counterfactual shifts are positive for privileged groups and negative for disadvantaged groups across both datasets.Their magnitudes track the measured statistical parity difference.

B. Detection of biased decisions

AIBF detects counterfactually flipped decisions more accurately than group-based signals and concentrates harmed candidates for review, while mitigation remains below parity because merit features retain proxy correlation.

  • Adult detection: 0.963 AUC identifies counterfactual flips on Adult, versus 0.672 for the group membership baseline.AIBF bias magnitude also achieves 0.749 average precision, versus 0.118 for group membership.
  • Adult detection: 0.883 AUC for decision-margin flags shows proximity to the threshold predicts flips but does not establish that protected attributes caused them.AIBF separates threshold proximity from protected-attribute influence and supplies an explanation.
  • Targeted review efficiency: 55% of harmed candidates are surfaced by reviewing the 5% of decisions with the largest AIBF magnitude, versus 6% under group-based review.Reaching half of harmed candidates requires reviewing 1% under AIBF versus 32% under group-based review.
  • Mitigation ceiling and proxy leakage: Adult correction raises disparate impact ratios but remains below the 0.80 four-fifths threshold because merit features carry residual proxy correlation.The reported ratios rise from 0.078 to 0.449 for sex and from 0.386 to 0.582 for race.

E. A worked audit

AIBF turns an individual hiring decision into an actionable audit by showing how protected proxies changed the score and whether they flipped the outcome.

  • Worked audit: −38.1 points separated the rejected candidate’s observed score from the favorable counterfactual score, producing a harmed flip.The observed score was 28.1, while neutralizing protected proxies raised the counterfactual score to 66.3.
  • Worked audit: Female, age, and minority-race proxies contributed −9.83, −7.16, and −4.52 points, while merit features contributed +33.64 points.The explanation identifies the protected factors responsible for the shift and separates them from merit contributions.
  • Worked audit: High-magnitude flagged decisions and near-zero unflagged decisions make a magnitude threshold a clean separator on Adult.Figure 5 reports the distribution of |∆| in score points.
  • Worked audit: AIBF produces an actionable account of why a qualified candidate was rejected, which a group metric cannot provide.The audit indicates that protected proxies subtracted more than twenty-one net points and that the decision warrants review.
  • Discussion: AIBF occupies the middle ground between population metrics and local explainers by combining localization, explanation, and a decision-level counterfactual signal.Its counterfactual values remain faithful to group disparity while identifying individually actionable cases.
  • Practical significance: A ranked, explained worklist makes the population-level audit obligation usable when human reviewers cannot inspect every decision.The review-efficiency result is presented as especially consequential for practice.

A. Deployment

AIBF is designed for batch reporting and inline human review, translating aggregate audit obligations into targeted, explainable intervention at the decision level.

  • Deployment modes: Batch mode attaches a ranked worklist and explanations to the required disparate-impact report, while inline mode routes flagged rejections to human review.The two modes support independent auditing and review before a rejection is finalized.
  • Deployment modes: Per-decision explanations make automated rejections contestable by showing how protected proxies changed the outcome.This supports both reviewer intervention and a candidate’s ability to challenge an opaque rejection.
  • Audit design: Attribution-magnitude ratios can rank important cases incorrectly because they ignore the decision boundary.The paper identifies the counterfactual shift as the appropriate decision-level signal.
  • Audit design: Auditors should compute per-decision counterfactuals before aggregating them into group reports.This preserves localized evidence alongside the aggregate statistic.
  • Review policy: Review should be ranked by counterfactual shift rather than group membership or raw attribution magnitude.Targeted review avoids spending scarce attention on unaffected candidates and reduces overcorrection risk.
  • Review policy: Failure to reach parity after neutralizing explicit protected attributes indicates residual proxy leakage in merit features.The paper recommends scrutinizing those features rather than treating the design as fully group-blind.
  • Deployment safeguards: Thresholds should be calibrated on labeled data for each deployment, while audited protected attributes must remain outside the scorer.The threshold and protected-data handling are deployment-specific safeguards.

E. Generality beyond hiring

AIBF is not specific to hiring, but its scope is bounded by the modeled proxies, the deployed scorer, and the fidelity of its explanation model.

  • Generality beyond hiring: AIBF applies to any thresholded scoring decision with protected proxies, legitimate factors, and query access to the scorer.The paper names lending, insurance, admissions, tenant screening, and pretrial risk assessment as additional domains.
  • Generality beyond hiring: COMPAS provides direct evidence of cross-domain generality because it is a criminal-justice scorer audited with the same core behavior.The reported properties include faithfulness, separation of flipped decisions, and the same correction ceiling.
  • Scope boundaries: AIBF measures the deployed scorer’s influence on protected proxies rather than real-world causation.This operational target is appropriate for auditing a specific scorer but is not a structural causal claim.
  • Scope boundaries: Unmodeled proxies limit both detection and correction, contributing to corrected disparate-impact ratios that do not reach parity.The method audits only the proxies it is given.
  • Scope boundaries: Linear reference explanations can underdescribe strongly nonlinear scorers, although the decision signal remains computed on the true scorer.Additive attribution is exact for the reference model, not necessarily a complete description of the deployed model.
  • Scope boundaries: The experiments audit standard classifiers trained on real public data rather than proprietary production systems, so vendor-specific magnitudes remain unmeasured.Only scoring access to a vendor system could establish those magnitudes.
  • Scope boundaries: A high bias magnitude triggers human review but is not a legal determination of discrimination.The paper cautions against reading its numbers as adjudicating a specific system.

A. Threats to validity

The paper frames AIBF’s evidence around construct, reproducibility, legal compatibility, and deployment-specific thresholding, while retaining explicit limits and future work.

  • Validity: AIBF’s counterfactual flip is an operational analogue of counterfactual fairness for a deployed scorer, but it captures model influence rather than real-world causation.The construct-validity discussion distinguishes the audit target from a causal model of the world.
  • Validity: Released scripts, public datasets, fixed seeds, and command-line experiments make every reported number regenerable and inspectable.The reproducibility claim covers preparation, classification, auditing, efficiency analyses, and figure data.
  • Validity: AIBF supports human and legal judgment while aligning its per-decision reasoning with selection-rate, impact-ratio, transparency, and oversight obligations.The paper positions the method as compatible with Local Law 144, the four-fifths rule, and EU AI Act obligations.
  • Validity: Threshold calibration is deployment-specific because overly aggressive thresholds burden reviewers, permissive thresholds miss harm, and cross-distribution transfer performed poorly.The paper recommends calibration on a labeled sample for each deployment.
  • Conclusion: On two real datasets, AIBF’s signal is faithful to group disparity, detects biased decisions better than a group-membership baseline, and supports efficient explained review.Correction improves the disparate-impact ratio but does not reach parity because merit features retain proxy correlation.
  • Future work: Future work targets causal intervention, proxy discovery, and evaluation on live proprietary applicant-tracking systems.These directions address causal tightness, uncovered leakage, and production measurement.
Loading 2608.21537v1…