PaperlayerView exact revision on arXiv

Source-linked AI summary

Counterfactual Bias Testing for Application Tracking System

Sai Yashwant, Shruti Bansal, Anurag Dubey, Samaroha Chatterjee, Satyam Kumar, Shreyash Gupta, Gantala Thulsiram

arXiv:2608.26899v1cs.AI

TL;DR

Correspondence audits for demographic bias are costly to repeat as candidate-matching models are retrained. The paper combines LLM-generated matched resume variants with a nine-metric fairness audit. Despite Score Delta staying within its ±0.02 PASS band, MARC averaged 4.3 to 5.3 position-changes per variant.

  • Problem

    Repeatedly constructing matched resume variants for demographic-bias correspondence audits is expensive and poorly aligned with modern matching-model retraining cycles.

    Source: §1 · p. 2

  • Method

    The method couples LLM-generated, qualification-matched demographic resume variants with a nine-metric fairness suite spanning counterfactual, group-fairness, and merit-aware families.

    Source: §1 · p. 2, §5 · p. 21

  • Results

    4.3 to 5.3 average position-changes per variant (MARC) occurred while Score Delta stayed within its ±0.02 PASS band for all nine variants.

    Source: §4.6 · p. 21

  • Takeaways & Limitations

    This illustrative audit supports checking rank stability and ranking quality rather than relying only on score, retention, or merit-aware rate-gap metrics.

    Source: §4.5 · p. 21

    Results come from one fixed-seed run without repeated trials, and borderline findings may change when jobs, candidates, or generation randomness are resampled.

    Source: §5 · p. 22

Abstract

from arXiv · show

Automated candidate-job matching systems are increasingly classified as high-risk AI under emerging regulation, yet auditing them for demographic bias is expensive: classical correspondence-audit studies require hand-crafted resumes and manual submission, which does not scale to fast pipeline retraining cycles. This paper presents a general, reusable methodology that (1) uses task-specialized LLM agents to synthesize identity-neutral base resumes and inject controlled demographic treatments across five protected-characteristic axes (sex/gender, age, residence, language, disability), producing a K x (1+N) correspondence-audit matrix; (2) qualitatively flags inferred protected characteristics per an EU AI Act-aligned prompt; (3) ranks candidates against a job description via a fine-tuned sentence-embedding model and cosine similarity; and (4) computes a nine-metric fairness suite spanning counterfactual (score delta, mean absolute rank change, flip rate), group-fairness (top-K retention, four-fifths/impact ratio), and merit-aware (Recall@K, nDCG@K, equal opportunity, equalized odds) families, each with bootstrap confidence intervals, significance tests, and Benjamini-Hochberg correction, culminating in an automated PASS/INVESTIGATE/FAIL report with a composite risk score. On an example corpus of 5 job orders, 100 base candidates, and 10 demographic treatments (90 metric x variant evaluations): score shifts, top-K retention, and merit-aware rate gaps stay within tolerance for every treatment, but a rank-stability metric (MARC) and nDCG@K each surface borderline findings - including one on the neutral baseline itself - that a score- or retention-only view would miss. The results argue for multi-metric, multi-family auditing over any single aggregate score, and for LLM-agent-generated audits as a practical, low-cost complement to human-curated audits for any candidate-job matching pipeline.

1 Introduction

The paper integrates LLM-generated correspondence audits with statistically validated, multi-family fairness evaluation for candidate–job ranking. Its example shows that acceptable score and shortlist outcomes can coexist with borderline ranking findings.

1 Introduction

  • Recruitment AI is explicitly classified as high-risk under the EU AI Act [European Parliament and Council of the European Union, 2024].LLMs can inherit and amplify societal biases, so using LLM-derived semantic components does not remove the need to scrutinize automated hiring pipelines [Gallegos et al., 2024, Bommasani et al., 2021, Nadeem et al., 2021, Buolamwini and Gebru, 2018].Source: §1 · p. 2
  • Correspondence audits are costly to repeat after retraining, while existing fairness toolkits neither generate audit data nor natively support paired, counterfactual, per-treatment evaluation.Correspondence audits hold job-relevant qualifications constant while varying protected-characteristic signals [Bertrand and Mullainathan, 2004, Kim, 2017]; existing toolkits instead assume labeled classification or regression tasks and real deployed populations.Source: §1 · p. 2
  • Task-specialized LLM agents generate identity-neutral resumes, inject one demographic treatment axis while holding qualifications fixed, and produce EU AI Act-aligned qualitative bias flags.Separate base-generation and bias-injection phases prevent qualification and demographic signals from covarying by construction. Both job-matching and non-matching controls test wrongful demotion of qualified candidates and wrongful promotion of unqualified candidates.Source: §1 · pp. 2–3
  • Nine metrics span counterfactual, group-fairness, and merit-aware families, evaluating paired rankings produced through structuring, embeddings, and cosine similarity.Each metric receives a bootstrap confidence interval, an appropriate significance test, and Benjamini–Hochberg correction. An HTML dashboard provides PASS/INVESTIGATE/FAIL classifications and weighted aggregate risk, with stringent research and explicitly relaxed operational thresholds; the classification scheme is calibrated against the legal four-fifths rule.Source: §1 · pp. 2–3
  • Three research questions distinguish changes in relevance scores or ranks, movement across top-K shortlisting cutoffs, and disproportionate disadvantage to protected groups.Bias is defined relative to the disadvantaged group, not the favored group; group-fairness and merit-aware metrics distinguish disproportionate burdens from symmetric perturbations.Source: §1 · p. 3
  • 5 job orders, 100 base candidates, and 10 treatments illustrate that acceptable score-delta, top-K-retention, and merit-aware rate-gap metrics can coexist with borderline rank-stability or ranking-quality findings.The treatments comprise one neutral baseline and nine variants; the demonstration argues against reducing fairness auditing to a single number.Source: §1 · p. 3

2 Literature Survey

The survey connects correspondence audits, quantitative fairness metrics, ranking fairness, and LLM-generated test data. It identifies a gap in integrating controlled audit generation, semantic ranking, statistically tested fairness metrics, and automated reporting within one pipeline.

2.1 Correspondence Audits and Algorithmic Hiring Discrimination

  • Correspondence audits isolate demographic treatment effects by holding job-relevant attributes fixed, providing a causal template for testing matching algorithms.Classical studies compare callbacks for matched fictitious applications; the proposed adaptation replaces the employer with the matching algorithm.Source: §2.1 · p. 3
  • LLM screening retains demographic disparities [Wilson and Caliskan, 2024], while existing third-party audits remain largely manual and one-off.Bogen and Rieke [2018] found widespread absence of bias testing, and Raghavan et al. [2020] found vendor mitigation claims frequently mismatched technically defensible fairness definitions. Kim [2017] and Barocas and Selbst [2016] establish that facially neutral algorithmic selection can nonetheless produce actionable disparate impact.Source: §2.1 · pp. 3–4

2.2 Quantitative Fairness Metrics and Toolkits

  • General-purpose fairness toolkits lack native paired-treatment designs, requiring audit-data reshaping that forfeits paired-comparison statistical power.Hardt et al. [2016] define equal opportunity through equal true-positive rates and equalized odds through equal true- and false-positive rates. Kusner et al. [2017] require counterfactual invariance, while Feldman et al. [2015] formalize disparate impact through group-conditional positive-prediction-rate ratios.Source: §2.2 · p. 4

2.3 Fairness in Ranking

  • Ranking fairness requires attention to exposure, demographic retention, and merit-conditioned outcomes because aggregate ranking effectiveness can conceal treatment-specific demotion.Singh and Joachims [2018] emphasize rank-dependent exposure, while Recall@K and nDCG@K measure retrieval effectiveness rather than demographic consistency. The proposed suite combines top-K retention, a four-fifths ratio applied to retention, and top-K equal opportunity/equalized odds.Source: §2.3 · p. 4

2.4 LLM-Based Synthetic Data Generation for Bias Testing

  • LLM-generated audits risk qualification drift and evaluator self-preference; separate base/variant generation and cross-family generation/evaluation are proposed safeguards.Identity-neutral base generation establishes qualifications, while demographic injection is instructed to preserve them. When ranking is LLM-mediated, the methodology recommends a different model family for evaluation to reduce generation-artifact and stylistic self-preference confounding.Source: §2.4 · p. 5

2.5 Gap Addressed by This Work

  • The authors identify no prior framework combining two-phase LLM audit generation, semantic ranking, statistically tested multi-family fairness metrics, and automated threshold-classified reporting.The proposed ranking combines LLM-based resume/job structuring with fine-tuned sentence-embedding cosine similarity. The fairness suite includes metric-specific significance tests and false-discovery-rate correction.Source: §2.5 · p. 5

3 Methodology

The methodology combines task-specialized LLM agents with an agent-agnostic quantitative audit module in a five-stage pipeline. Agent models are deployment choices, and the example implementation uses multiple model families rather than one model throughout.

3 Methodology

  • Five stages cover optional descriptor elicitation, two-phase synthetic resume generation, translation, optional qualitative bias flagging, and quantitative candidate ranking and fairness auditing.Stages 0–3 use task-specialized LLM agents accessed through standard chat/completion API endpoints.Source: §3 · p. 5
  • The quantitative audit is an agent-agnostic statistical module using scientific-computing libraries, an LLM structuring service, and a local sentence-embedding model for ranking.The libraries support tabular processing, numerical computation, statistical testing, and plotting.Source: §3 · p. 5
  • Agent models are interchangeable deployment choices; the example implementation mixes model families so no single model’s idiosyncrasies are shared across every stage.This extends the cross-family rationale behind the recommended generator/evaluator split for ranking. Standard API endpoints allow any instruction-following LLM, in principle, to serve any agent role.Source: §3 · p. 5

3.1 Multi-Agent Synthetic Correspondence-Audit Generation

The pipeline generates identity-neutral base resumes and controlled demographic variants, translates them, optionally flags protected characteristics, and accumulates candidates for quantitative auditing. Matching and non-matching controls and qualification-preservation checks underpin its correspondence-audit design.

3.1.1 Stage 0: LLM-Elicited Bias Descriptors (Descriptor Agent)

  • An optional Descriptor Agent generates demographic-treatment descriptors, replacing manual authoring and supplying the Generation Agent’s candidate_types list.Structured blocks or bullets are parsed into descriptors. If either agent asks a clarifying question, the pipeline sends one scripted follow-up confirming legitimate auditing with fictitious candidates, then uses fallback text splitting if needed.Source: §3.1.1 · pp. 5, 7

3.1.2 Stage 1: Two-Phase Base-Candidate + Bias-Variant Matrix Generation (Generation Agent)

  • Matching and non-matching controls test both identity-based demotion and promotion, motivating retention of Equal Opportunity and Equalized Odds.Without negative controls, a system could pass a demotion-only audit by inflating scores for everyone carrying a particular identity signal.Source: §3.1.2 · pp. 7–8
  • K × (1 + N) resumes pair K identity-neutral bases with N demographic variants each, holding qualifications and match labels fixed by design.Phase A requests M matching and K − M non-matching bases; Phase B receives each base verbatim and changes only the specified demographic signal. Prompts require inline JSON rather than downloadable artifacts, with JSON extraction and fallback text parsing.Source: §3.1.2 · pp. 7–8
  • Qualification-preservation checks reuse ranking embeddings to detect drift, but initial real-resume comparisons are directional and larger human-curated validation remains future work.Sampled BASE/variant pairs are scored against the same job description using the ranking pipeline’s cosine similarity. Large systematic qualification drift would undermine attribution of ranking differences to demographic treatment alone.Source: §3.1.2 · p. 8

3.1.3 Stage 2: Translation (Translation Agent)

  • A dedicated Translation Agent translates every resume into a configurable target language after demographic injection so the treatment signal survives downstream reading.Translation requires one agent call per resume.Source: §3.1.3 · p. 8

3.1.4 Stage 3: EU AI Act-Aligned Qualitative Bias Flagging (Flagging Agent)

  • An optional Flagging Agent produces per-resume, EU AI Act-aligned qualitative analyses of inferred protected characteristics and biases.The analysis enters the audit trail and, alongside candidate_type, feeds the five-axis protected-characteristic classifier. Flagging can be disabled when only quantitative auditing is required.Source: §3.1.4 · p. 8

3.1.5 Candidate Pool Accumulation and Correspondence Pairing

  • Persistent candidate pools accumulate base identities across runs for the same job, enlarging on-demand quantitative audits without regenerating previously audited identities.Legacy simple mode assigns BASE using a configurable label, defaulting to “neutral,” or otherwise the first-seen type. Stage 4 structures and translates pooled resumes, classifies five protected axes, ranks embeddings by cosine similarity within job/variant groups, and exports fairness metrics with Benjamini–Hochberg correction and an HTML report.Source: §3.1.1 · p. 7, §3.1.5 · p. 8

3.2 Protected-Characteristic Axis Taxonomy and Automated Classification

A structured LLM call assigns every candidate protected-axis annotations, using neutral defaults for invalid or failed classifications. The illustrative taxonomy can be refined for intersectional analysis or adapted to other jurisdictions without redesigning the pipeline.

Automated Classification

  • Five protected-characteristic axes receive automated annotations through one structured LLM call, with neutral defaults ensuring completeness when outputs are invalid or classification fails.The classifier reads candidate_type and Stage 3 bias-analysis text at temperature 0, returning constrained JSON with one value per axis for Stage 4.Source: §3.2 · p. 9

Taxonomy Granularity

  • Male-NA and Female-NA combine sex/gender and naming-origin cues, preventing separate attribution of observed effects to either signal.This tractability choice keeps the example matrix small. Independent axes and finer combination-cell stratification enable intersectional analysis without changing metric formulas, but the example corpus does not exercise that capability.Source: §3.2 · p. 9

Regulatory Adaptability

  • The taxonomy and bias-flagging prompt derive from the EU AI Act [European Parliament and Council of the European Union, 2024], but support jurisdiction-specific axes and thresholds.Regional extensions include Colorado’s statute [Colorado General Assembly, 2026], NYC Local Law 144 [New York City Council, 2021], and California’s regulations [California Privacy Protection Agency, 2025], each with its own protected-characteristic scope, audit cadence, and disclosure requirements. Adaptation re-parameterizes axes and thresholds rather than redesigning the pipeline.Source: §3.2 · p. 9

Illustrative Treatments

  • Nine bias variants V01–V09 instantiate illustrative axis-level combinations rather than an exhaustive or fixed treatment set.Stage 1B injects each combination into every base resume; BASE is fixed at (Male-FR, Young, Close, FR only, No). Applications can change combinations, variant counts, or axis granularity without altering the pipeline or metric suite.Source: §3.2 · p. 9, Table 2 · p. 10

3.3 Ranking Pipeline: LLM Structuring, Embedding, and Cosine Scoring

The pipeline structures resumes and job descriptions consistently, embeds them, and ranks candidates by cosine similarity within separate treatment cohorts for paired fairness evaluation. Non-generative scoring avoids same-model self-bias; deployments using LLM scorers instead require a cross-family generator/evaluator split.

Ranking Pipeline

  • One consistent LLM prompt extracts field-ordered English search text, source-language BM25 keywords, and an information-richness score from every resume and job description.JSON-object responses at temperature 0.1 contain Role, Skills, Speciality, Expertise, Education, and Domains fields. Records are cached to disk so unchanged inputs do not trigger repeated LLM calls.Source: §3.3 · p. 10
  • A local sentence-embedding model [Reimers and Gurevych, 2019] converts structured texts into unit-normalized vectors, using the same backbone across baseline and demographic treatments.The instantiation uses an EmbeddingGemma checkpoint [Google DeepMind, 2025], fine-tuned on job–resume pairs with Cached Multiple Negatives Ranking Loss [Henderson et al., 2017]. Inference requires neither external model downloads nor a GPU, but the methodology is backbone-agnostic.Source: §3.3 · p. 10
  • Candidates are ranked by descending cosine similarity within each job–treatment cohort, with scores clipped to [0, 1] before fairness evaluation.Baseline candidates compete only against baseline candidates, and each demographic variant competes only within its own otherwise-identical cohort, avoiding cross-variant comparison artifacts.Source: §3.3 · p. 10
  • Baseline and counterfactual ranking tables are joined on job_id and cv_id to pair baseline and variant ranks and scores for every fairness metric.The counterfactual table additionally records variant_code and the five protected axes; ranking tables may also include is_true_match.Source: §3.3 · p. 10
  • Non-generative embedding scoring avoids same-model self-bias; substituting an LLM scorer requires an independent model family from the resume generator.The embedding model neither authored nor was fine-tuned specifically on the audited resumes, while shared generator/scorer families can confound bias with stylistic self-preference. Whether evaluator capability changes detected bias magnitude remains an open model-swap-ablation question.Source: §3.3 · p. 10

3.4 Fairness Metric Suite

The suite evaluates demographic treatments through pooled, paired comparisons against a fixed reference, combining counterfactual, group-fairness, and label-dependent merit-aware metrics. Its PASS/INVESTIGATE/FAIL classifications use documented thresholds but require supporting statistics and numerical safeguards near boundaries.

3.4 Fairness Metric Suite

  • K × |J| paired observations per metric-and-variant evaluation pool each treatment across base identities and jobs against a fixed reference, BASE by default.Bootstrap confidence intervals use 1,000 resamples and empirical 95% limits, with significance level α = 0.05. The example uses shortlist K = 10 for counterfactual and group-fairness metrics, versus K = 15 for merit-aware metrics.Source: §3.4 · p. 11
  • PASS/INVESTIGATE/FAIL thresholds distinguish statutory transpositions, analogical extensions, and pre-registered policy choices rather than treating every boundary as legally prescribed.Table 3 specifies metric-dependent comparison directions: lower-is-better metrics use upper bounds, while higher-is-better metrics reverse the inequalities.Source: §3.4 · p. 11, Table 3 · p. 12

3.4.1 Counterfactual Metrics

  • Score Delta measures the mean paired score shift, while MARC measures mean absolute rank change normalized by shortlist size K for classification.Score Delta is classified by its absolute mean; both metrics use paired bootstrap confidence intervals and signed-rank significance tests, with Score Delta citing.Source: §3.4.1 · p. 11
  • Flip Rate tracks changes in top-K membership using paired baseline-versus-variant contingency counts.Uncertainty uses the Wilson score interval, and significance uses McNemar’s test. The test uses an exact binomial variant below 25 discordant pairs and continuity-corrected χ2 otherwise.Source: §3.4.1 · p. 11

3.4.2 Group-Fairness Metrics

  • Group-fairness metrics measure equally weighted per-job top-K retention and its ratio to the reference, reporting disparities against the disadvantaged group.Retention intervals bootstrap jobs when at least two are available, otherwise using a Wilson interval on pooled retained/slots counts. Impact Ratio uses paired job-level bootstrapping and Fisher’s exact test on pooled retained/not-retained counts.Source: §3.4.2 · pp. 11–12

3.4.3 Merit-Aware Metrics

  • When relevance labels exist, Recall@K measures true-match capture, while nDCG@K measures ranking quality relative to an ideal relevance ordering.Recall@K macro-averages across jobs containing true matches. nDCG@K averages per-job normalized, logarithmically discounted binary relevance.Source: §3.4.3 · p. 13
  • Equal Opportunity and Equalized Odds adapt Hardt et al. [2016] to top-K outcomes, measuring true-positive-rate disparity and the maximum absolute TPR/FPR gap, respectively.Equal Opportunity pools true matches across jobs and treats smaller gaps as better; Equalized Odds compares both rates with the reference.Source: §3.4.3 · p. 13

3.4.4 Boundary Sensitivity and Numerical Tolerance

  • Boundary-adjacent statuses require raw values, confidence intervals, and p-values; numerical rounding guards against floating-point artifacts changing classifications.The passage identifies MARC variant V06 as only a few percentage points inside INVESTIGATE, not qualitatively distinct from a nearby value across the boundary. It recommends rounding to 4–6 significant digits before threshold comparisons.Source: §3.4.4 · p. 13

3.5 Statistical Validation Layer

The statistical validation layer corrects metric p-values for multiple comparisons before audit reporting and uses fixed-seed bootstrap resampling for reproducibility.

3.5 Statistical Validation Layer

  • α = 0.05 governs Benjamini–Hochberg false-discovery-rate correction across all computed metrics with associated p-values before audit reporting.The correction addresses multiplicity across up to 90 metric × variant comparisons in the example corpus.Source: §3.5 · p. 13
  • All bootstrap resampling uses a fixed random seed for reproducibility.Source: §3.5 · p. 13
  • Table 4 maps each metric family to its supporting classical statistical test and explains the rationale.Source: §3.5 · p. 13

3.6 Status Classification and Composite Risk Scoring

The report classifies metric–variant pairs as PASS, INVESTIGATE, or FAIL and combines their counts into a weighted risk score. Thresholds assign an overall risk label, with failures weighted more heavily than borderline findings.

3.6 Status Classification and Composite Risk Scoring

  • Every metric–variant pair receives a PASS, INVESTIGATE, or FAIL status using Table 3, and the report computes a weighted risk score from their counts.The counts cover all computed metric–variant pairs.Source: §3.6 · p. 13
  • 0.30 is the HIGH RISK threshold for RiskScore; scores from 0.15 to below 0.30 are MEDIUM RISK, and lower scores are LOW RISK.Source: §3.6 · p. 13
  • One FAIL carries the same weight as three INVESTIGATEs, deliberately distinguishing outright failures from borderline findings.The stated intent is to rate reports with many borderline findings but no confirmed failures as materially less severe than reports containing outright failures.Source: §3.6 · p. 13

3.7 Dual-Threshold, Audience-Aware Reporting

The methodology supports paired, explicitly labeled research and operational reporting tiers to communicate audit findings to different audiences without conflating policy choices. Predeclared thresholds and a locked experimental configuration guard against threshold-shopping.

3.7 Dual-Threshold, Audience-Aware Reporting

  • The methodology supports paired, explicitly labeled research and operational tiers for audiences with different abilities to interpret technical statistics.Both tiers must be presented together as distinct policy choices. An illustrative operational tier widens the Score Delta PASS band from ±2% to ±4%.Source: §3.7 · p. 14
  • The stringent research tier surfaces borderline observations as INVESTIGATE rather than PASS and exclusively governs the Section 4 results.This tier is intended to demonstrate rigorous pressure-testing and support legal and regulatory scrutiny; audience-aware reporting aims at communication fidelity rather than leniency.Source: §3.7 · p. 14
  • Standing safeguards require predeclared thresholds and results traceable to one locked configuration and random seed, separating threshold selection from execution to prevent threshold-shopping.Thresholds not directly derived from statute must be documented as policy choices, and repeated-trial threshold tuning is prohibited.Source: §3.7 · p. 14

3.8 Automated Audit Reporting

The report generator produces a dependency-free HTML audit dashboard with risk summaries and per-metric results, alongside a flat tabular export for downstream analysis.

3.8 Automated Audit Reporting

  • A single dependency-free HTML dashboard presents a color-coded risk banner, evaluation and status counts, and a summary of variants triggering FAIL.The header records K, reference variant, α, bootstrap count, and generation timestamp.Source: §3.8 · p. 14
  • Each metric receives a status-colored bar chart with PASS/INVESTIGATE thresholds and a per-variant table reporting values, 95% CIs, BH-adjusted p-values, statuses, and notes.Charts are embedded inline, requiring no external files.Source: §3.8 · p. 14
  • A companion flat tabular export preserves every metric-result field, including family-specific diagnostics, for downstream analysis.Diagnostic columns include flip-in/flip-out rates and per-job counts.Source: §3.8 · p. 14

4 Results and Analysis

On an illustrative corpus rather than a deployed system, the audit reported low overall risk but detected rank-stability and ranking-quality concerns while seven other metrics passed every treatment. The findings support combining metric families and interpreting threshold classifications alongside statistical uncertainty.

4.1 Example Corpus and Experimental Setup

  • 5 job orders, 100 base candidates, and 10 treatments formed an illustrative audit corpus, not an evaluation of a deployed system.The pipeline produced 500 baseline and 5,000 counterfactual rows, with designed match labels alternating by job parity and candidate index; jobs adapted real job-family requirements while removing identifying details. Although resumes could be translated to preserve demographic-signal legibility, ranking always embedded structured English extracted from resumes and job descriptions, avoiding a surface-language ranking confound.Source: §4 · pp. 14–15, §4.1 · p. 15, §4.2 · p. 15

4.2 Aggregate Audit Outcome

  • 90 evaluations yielded 86 PASS, four INVESTIGATE, and no FAIL results, producing a LOW RISK score of approximately 1.5%.Seven metrics passed every treatment, but MARC flagged V06 and nDCG@15 flagged BASE, V04, and V07—findings that score-, retention-, or rate-gap-only auditing would miss.Source: §4.2 · p. 15, Figure 3 · p. 16, Figure 8 · p. 19

4.3 Counterfactual Metrics

  • 0.533 MARC/K for V06 triggered the sole counterfactual INVESTIGATE finding, despite every variant passing score-delta and flip-rate thresholds.Score shifts ranged from +0.0013 (V09) to +0.0074 (V06), within ±0.02; V06 nevertheless showed statistical significance after BH correction (padj = 0.0032). Raw MARC ranged from 4.30 (V07) to 5.33 (V06) positions, while flip rates ranged from 1.6% (V01, V04) to 3.2% (V03, V06), below 5%.Source: §4.3 · p. 15, Figure 3 · p. 16, Figure 2 · p. 16

4.5 Merit-Aware Metrics

  • 0.82–0.88 Recall@15 exceeded the 0.80 PASS threshold across variants, while Equal Opportunity and Equalized Odds gaps also passed.Recall@15 ranged from 0.82 (V04) to 0.88 (V06, V09), versus BASE's 0.84; rate gaps reached at most 0.04, below 0.10, with FPR gaps never exceeding 0.0044. V02, V03, V06, V08, and V09 increased true-positive capture; only V04 decreased it, by two percentage points.Source: §4.5 · p. 17

4.4 Group-Fairness Metrics

  • 0.84–0.92 top-10 retention and impact ratios cleared the 0.80 PASS threshold for every variant, against BASE retention of 1.0.Fisher tests for V02, V03, and V06 were nominally significant before correction, but none remained significant after BH correction (padj ≥0.07). Retention and merit-conditioned rate-gap metrics agreed here, isolating rank-stability and ranking-quality findings from retention-level effects.Source: §4.4 · p. 17, Figure 5 · p. 18, Figure 6 · p. 18

4.5 Merit-Aware Metrics

  • 0.786 for BASE, 0.793 for V04, and 0.780 for V07 placed nDCG@15 below the 0.80 PASS threshold, triggering INVESTIGATE.The BASE finding reflects baseline position-weighted retrieval quality, not demographic bias; merit-aware metrics can therefore identify model- or data-quality concerns independently of treatment effects.Source: §4.5 · p. 17, Figure 8 · p. 19

4.6 Discussion

  • Score stability did not ensure rank stability, while set-membership metrics missed within-list movement and ranking-quality concerns.Aggregate retention and qualified-only rate parity should both be assessed because they can diverge, as Hardt et al. [2016] establish for classifiers. Reports should be read alongside sample size, confidence-interval width, and p-values, with explicit numerical tolerance in continuous-metric threshold comparisons.Source: §4.6 · p. 21

5 Conclusion and Future Discussion

The illustrative audit found uniformly clean results for seven metrics but flags from MARC and nDCG@K, supporting the methodology’s multi-metric design. The conclusion identifies validation, statistical-power, and reproducibility limitations, alongside extensions toward continuous fairness auditing.

5 Conclusion and Future Discussion

  • Seven of nine metrics were uniformly clean across treatments, while MARC flagged one variant and nDCG@K flagged three cells, including the neutral baseline.The example covered 5 job orders, 100 base candidates, and 10 demographic treatments; the seven clean metrics reported 0 FAIL and 0 INVESTIGATE. This divergence would be invisible to an audit relying on a single aggregate fairness score.Source: §5 · p. 21
  • The methodology separates qualifications from demographic signals through two-phase LLM-agent generation and evaluates fairness using nine metrics across three families.The counterfactual, group-fairness, and merit-aware metrics receive appropriate significance tests, bootstrap confidence intervals, and Benjamini–Hochberg correction. An automatically generated HTML report provides threshold classifications and a composite weighted risk score.Source: §5 · p. 21
  • One fixed-seed experimental run cannot establish the stability of borderline findings; repeated-trial validation is required before external or legal reliance.The four INVESTIGATE flags may change with resampled jobs, candidates, or generation randomness; dual-threshold reporting does not replace validation. Deployments also need larger candidate pools, minimum-n power guidance, and reporting warnings because small pools relative to shortlist size K can make rank-and-set metrics degenerate.Source: §5 · p. 22
  • Merit-aware labels and automated protected-axis classifications need human validation to establish their reliability.Example relevance labels follow a designed alternating rule, while the general pipeline uses the resume-generating agent’s self-reported match flag; larger real-job corpora need human-adjudicated relevance judgments. The five-axis taxonomy is a modeling choice with unquantified classification error, motivating inter-rater comparison against human annotations.Source: §5 · p. 22
  • LLM-injected demographic signals may be stereotyped or unrealistic, and synthetic resumes remain imperfect proxies for real candidate populations.Human review and comparison with hand-authored audit resumes could assess injection realism. Correlated attributes, formatting diversity, and implicit identity signals may be underrepresented, so production monitoring is needed to calibrate synthetic findings against real traffic.Source: §5 · p. 22
  • Planned generator, translation-model, evaluator, and embedding-backbone ablations would test generation artifacts, evaluator self-bias, and whether retrieval improvements alter fairness.For LLM-mediated ranking, varying evaluator capability and family—including a same-family generator/evaluator condition—would distinguish task effects from evaluator-specific effects. Comparing base and fine-tuned embedding backbones would test whether retrieval-quality gains change the fairness profile.Source: §5 · p. 22
  • Further deployment work includes toolkit cross-validation, broader jurisdictions and organizational settings, and explicit numerical tolerance for threshold classification.Planned checks compare paired-treatment implementations with Fairlearn, AIF360, and Aequitas, extend taxonomy and calibration beyond EU and four-fifths frameworks, and cover multiple job families, business units, and brands. Boundary-adjacent findings and floating-point reclassification hazards motivate explicit tolerance as the methodology moves toward continuous auditing and external review.Source: §5 · pp. 21–22

Availability Statement

  • Code, prompts, the example corpus, and the HTML report remain internal, with access possible by direct request subject to organizational review.Case-by-case access may support inspection, replication, or independent audit; the artifacts are not publicly released alongside the paper.Source: Availability Statement · pp. 22–23

Related papers

Loading 2608.26899v1…