Tolerance-Based Fairness Auditing: Violation Certification and Sensitivity Screening

1 Introduction

• Tolerance-based auditing distinguishes acceptable nonzero disparities from violations and addresses whether the disparity lies outside the prespecified tolerance region.The framework considers a prespecified adverse direction and restricts the disparity parameter to nonnegative values.Source: §1 · p. 2
• Violation certification prioritizes protection against false violation declarations, while sensitivity screening prioritizes reducing missed violations near the tolerance boundary.The introduction associates false declarations with regulatory, organizational, or decision costs and missed disparities with early risk discovery.Source: §1 · pp. 2–3
• For violation certification, constrained empirical likelihood uses least-favorable-point calibration, with a one-sided lower confidence bound serving as a numerical certificate.The framework also establishes asymptotic false flagging rate control when this procedure is combined with Benjamini–Hochberg for disjoint subpopulations under suitable regularity conditions.Source: §1 · pp. 3–4
• For sensitivity screening, split and adjusted split empirical likelihood use an adaptive boundary-proxy principle to increase rejection probability at the tolerance boundary for greater sensitivity.The introduction says adjusted split empirical likelihood further enhances detection power and describes a trade-off between detection sensitivity and false alerts.Source: §1 · pp. 3–4
• Simulations illustrate finite-sample trade-offs between error control and detection under skewed and heavy-tailed distributions.The introduction characterizes the procedures as having distinct operating behavior for their different auditing objectives.Source: §1 · pp. 3–4
• The framework accommodates nonzero but tolerable disparities and provides a common testing setup for violation certification and sensitivity screening.The introduction also characterizes the constrained procedure's lower confidence bound as quantifying how far disparity can be certified to exceed tolerance at a prescribed confidence level.Source: §1 · pp. 3–4

2 Preliminaries

• A group-wise performance disparity concerns whether a subgroup's conditional expected metric differs from a target.The setup treats subgroup membership through covariates and uses the same covariate vector for model inputs and group definitions for notational simplicity.Source: §2 · pp. 4–5
• For statistical parity, the disparity is the group's positive-prediction probability minus the overall positive-prediction probability.The example sets the performance metric to the indicator of a positive prediction.Source: §2 · p. 5
• The empirical-likelihood setup forms moment conditions from each observation's metric, target, candidate disparity, and subgroup membership.The stated asymptotic conditions include independent, identically distributed observations and subgroup probability and conditional metric variance bounded away from zero.Source: §2 · pp. 5–6
• Under the stated conditions, the empirical log-likelihood ratio has a Wilks-type chi-squared limit, and replacing the target with its estimator preserves that limit.The preliminary point-null testing results are distinguished from the later composite tolerance hypothesis, which concerns an interval of acceptable disparities.Source: §2 · pp. 6–7

3 Tolerance-based fairness auditing

• Least-favorable-point calibration supports violation certification, while adaptive boundary-proxy calibration supports sensitivity screening.The procedures are intended for different auditing objectives rather than as interchangeable tests.Source: §3 · p. 7

3.1 Violation certification under a tolerance constraint

• CEL calibrates at the tolerance boundary, controlling false declarations at nominal level α there and with rejection probability converging to zero at fixed interior null points.The boundary is the least-favorable null configuration; the interior protection can make the test more conservative.Source: §3.1 · pp. 8–9
• With disjoint groups and the stated boundedness and known-parameter assumptions, applying BH(q) to CEL p-values asymptotically controls the false flagging rate at level q.The passage attributes p-value independence to disjoint groups.Source: §3.1 · p. 10

3.2 Sensitivity-oriented screening

• SEL screens for potentially problematic groups by combining centered baseline and boundary-oriented evidence with a data-dependent gate, without claiming uniform level-α control over the composite null.Screening is an early-warning signal, not a formal violation certificate, and permits increased boundary rejection probability in exchange for sensitivity.Source: §3.2 · pp. 10–11

3.2.1 Adaptive boundary-proxy inference

• The adaptive boundary-proxy principle retains centered baseline evidence and activates boundary-oriented evidence only when estimated disparity supports a tolerance violation, without changing the prespecified threshold.In the split-sample construction, paired differences provide baseline evidence and paired sums provide boundary evidence and the gate.Source: §3.2.1 · pp. 11–12
• The split empirical-likelihood procedure’s boundary proxy is inactive at fixed interior null points, nondegenerate at the tolerance boundary, and active under violations, yielding less conservative interior screening and greater near-boundary sensitivity than least-favorable violation certification.The procedure remains consistent under fixed alternatives.Source: §3.2.1 · p. 14
• At the tolerance boundary, screening admits an increased rejection probability, which is a Type I increase under the composite null rather than a violation-certification guarantee.Corollary 2 characterizes screening sensitivity under local and fixed alternatives, and the stated operating profile explicitly treats the boundary increase as a trade-off.Source: §3.2.1 · p. 15

3.2.2 Power-enhanced screening

• ASEL provides a power-enhanced implementation of the adaptive boundary-proxy principle, replacing boundary component T2 with ℓ2(0); for G > ε0, T1 = Op(1) while T3 diverges to +∞.Consequently, the gate event 1(T3 > cα2) has probability tending to 1.Source: §3.2.2 · pp. 15–16
• Under fixed alternatives, Ta exceeds Ts with probability tending to one, so at the same threshold Ta has asymptotically no smaller rejection probability and typically higher power.Source: §3.2.2 · p. 16
• ASEL preserves the adaptive boundary-proxy operating structure in the interior and at the tolerance boundary while strengthening detection under violations.Source: §3.2.2 · p. 16
• CEL2 has relatively high rejection within the tolerance region; CEL and SEL2 attain maximal size at the boundary, and SEL2 reduces CEL’s excessive conservativeness within the null region.Source: §3.2.2 · p. 17
• ASEL achieves the highest power under alternatives, while CEL is the most conservative and SEL and ASEL are less conservative alternatives.The theoretical limiting rejection probabilities are shown in Figure 1.Source: Figure 1 · p. 17, §3.2.2 · p. 17

4.1 Single group auditing

• Across the designs, CEL controlled false violation declarations at the prescribed level, while ASEL delivered the strongest power gains among the proposed procedures.The results distinguish CEL’s role in violation certification from SEL and ASEL’s greater alignment with sensitivity-oriented screening.Source: Figure 2 · p. 18, Figure 3 · p. 19, Figure 4 · p. 20
• Under normality, EL rejection frequencies approached one deeper inside the null, while CEL and the one-sided t test approached zero; SEL and ASEL approached nominal rejection rates inside the null and exceeded them at the boundary.These boundary exceedances reflect the screening trade-off between detection sensitivity and false alerts.Source: Figure 2 · p. 18, §4.1 · p. 19
• Under heavy tails, EL had the lowest power overall; CEL outperformed the one-sided t test in small samples, while SEL and ASEL had higher alternative power at larger sample sizes and near-nominal interior-null rejection rates.The heavy-tailed location-shift design is symmetric, and its qualitative patterns resemble those under normality.Source: Figure 3 · p. 19, §4.1 · p. 19
• Under skewness, the one-sided t test lost power in small and moderate samples, whereas CEL was more powerful and more robust to skewness.The skewed design also shows SEL improving power near the tolerance boundary; as the parameter moves into the null interior, CEL and the t test become increasingly conservative while SEL and ASEL remain near nominal as sample size increases.Source: §4.1 · pp. 19–20, Figure 4 · p. 20

4.2 Simultaneous subgroup auditing

• Across 26 disjoint groups under normal and exponential designs, the study compares false discoveries and detection power for CEL, SEL, and ASEL; BH is applied to unadjusted SEL and ASEL screening scores as exploratory benchmarks, without claiming FFR control.The study uses 100 replications, with all methods applied to the same observations within each replication.Source: §4.2 · pp. 20–21
• Under the global null, CEL-BH’s empirical FFR approaches nominal near the tolerance boundary, while t-BH is conservative and SEL-BH and ASEL-BH inflate FFR at the tested group sizes.Under the exponential design, t-BH’s empirical FFR is zero; under the normal design, it remains below nominal.Source: §4.2 · p. 22
• With π1 = 0.5 under the exponential design at δ = 0.10, CEL-BH achieves power 0.4692; both CEL-BH and t-BH have empirical FFR below 0.05, at 0.0305 and 0.0081, respectively.Source: §4.2 · p. 22
• CEL-BH offers stronger false-discovery protection than SEL-BH and ASEL-BH while exceeding t-BH in power, whereas ASEL-BH gains detection sensitivity at weak to moderate signals with inflated false flagging.The summary characterizes CEL-BH as particularly advantageous over t-BH under the exponential design and cautions that directly applying BH to unadjusted SEL and ASEL screening scores inflates FFR when true null hypotheses are present.Source: §4.2 · p. 24

Audit approach

• The audit constructs tolerance-based bounds on true predictive-performance disparity, extending statistical evidence about parity into explicit bounds.It builds on the same predictive-parity benchmark used in earlier COMPAS claims.Source: §5 · pp. 24–25

Overall African-American group

• At ε0=0.01, CEL certifies and SEL/ASEL alert at both nominal levels; at ε0=0.03, none does.At the 3-percentage-point tolerance, the lower bounds also fall below the threshold.Source: §5 · pp. 25–26
• At the 90% nominal level, for the African-American group, CEL yields a formal lower bound for violation certification of 2.51%, while SEL and ASEL yield screening lower bounds of 2.09%.Source: §5 · p. 25, Table 3 · p. 26
• At 90%, CEL's 2.51% lower bound exceeds Cherian and Cand`es (2024)'s bootstrap-based 1.87% bound.Both bounds are reported for the COMPAS dataset at the same nominal level.Source: §5 · p. 26

African-American age/sex subgroups

• At both nominal levels, all four procedures have strictly positive lower bounds for All and <25 groups; 90% regions are narrower and more sensitive for detecting nonzero disparities than 95% regions.Source: §5 · p. 26

Age/sex intersection subgroups

• Against the Caucasian reference, younger male subgroups have higher COMPAS PPV; males under 25 have the strongest signal, with a 90% lower bound of at least 6.55%.The analysis defines these subpopulations by intersections of age and sex.Source: Figure 7 · p. 27, §5 · p. 28

6 Conclusion

• The framework uses CEL for violation certification and SEL and ASEL for sensitivity-oriented auditing.CEL uses least-favorable-point calibration; SEL and ASEL instantiate an adaptive boundary-proxy principle through split empirical likelihood.Source: §6 · p. 30
• Sensitivity-oriented auditing uses adaptive boundary-proxy evidence aggregation, with SEL and ASEL exhibiting the screening behavior characterized by the theory.The conclusion identifies this principle as a distinct component of the framework.Source: §6 · p. 30
• Simulations under normal, heavy-tailed, and skewed distributions illustrate distinct finite-sample operating characteristics across the two auditing objectives.The experiments illustrate trade-offs between error control and detection sensitivity.Source: §6 · p. 30
• The COMPAS application interprets observed disparities in light of both the prespecified tolerance and the auditing objective.It distinguishes formal violation certification from sensitivity-oriented screening.Source: §6 · p. 30
• A powerful procedure for sensitivity screening remains technically challenging when the null and alternative hypotheses are interchanged.The conclusion identifies this as a direction for future research.Source: §6 · p. 30
• Sensitive attributes may be unavailable because of privacy protections or difficulty identifying reliable proxy variables, motivating future work on settings with missing attributes.The conclusion identifies extending tolerance-based auditing to these settings as an important topic for future research.Source: §6 · p. 30

8 Funding

• Chuanlong Xie was supported by the National Natural Science Foundation of China (NSFC.12201048).Source: §8 · p. 30
• Lixing Zhu was supported by grants NSFC12131006 and NSFC12471276 from the National Natural Scientific Foundation of China.Source: §8 · p. 30
• The acknowledgments also name grant CI2023C063YLL from the Scientific and Technological Innovation Project of China Academy of Chinese.Source: §8 · p. 30

9 Supplementary Material

• The supplementary material contains an extension to lower-tail tolerance auditing and all proofs for the theoretical results presented in the main text.Source: §9 · p. 31

• The heading names fairness auditing as its topic.Source: Supplementary Material: Tolerance-Based · p. 35
• It specifically names violation certification.Source: Supplementary Material: Tolerance-Based · p. 35
• The final “and” leaves the heading’s paired subject unstated.Source: Supplementary Material: Tolerance-Based · p. 35

A Extension to lower-tail tolerance auditing

• CEL certifies a lower-tail tolerance violation only when the upper confidence bound falls below −ε0.If the bound is no smaller than −ε0, the audit does not provide sufficient evidence to certify a violation.Source: A Extension to lower-tail tolerance… · p. 35
• The lower-tail extension changes the boundary anchor and reverses the gate direction while retaining the split procedures’ critical value.In the null interior the gate is asymptotically inactive; under the alternative the corresponding boundary component is activated and diverges.Source: A Extension to lower-tail tolerance… · pp. 35–36
• SEL and ASEL are sensitivity-oriented screening procedures, not procedures targeting uniform level-α control over the composite null.Their boundary rejection frequencies above α indicate screening behavior, not calibrated violation certification.Source: A Extension to lower-tail tolerance… · p. 36
• Compared with CEL, ASEL has higher rejection frequencies over much of the alternative region, while SEL gains are smaller and mainly near the null boundary.The passages characterize these patterns as greater sensitivity to lower-tail violations, more pronounced for ASEL.Source: A Extension to lower-tail tolerance… · p. 37
• In the small-sample heavy-tailed setting, CEL has slightly higher rejection frequencies than the Student t test at some alternative parameter values.Source: A Extension to lower-tail tolerance… · p. 37
• As the true disparity moves into the null interior, CEL and the t test rejection frequencies approach zero, while SEL and ASEL remain close to nominal level.Figure A.1 reports this pattern for normal and heavy-tailed designs.Source: Figure A.1 · p. 37, A Extension to lower-tail tolerance… · p. 37

B Technical proofs

• At the tolerance boundary, ASEL’s gate activates with limiting probability α2; conditional rejection given activation tends to one, while the closed-gate statistic is T1.The asymptotic independence of T1 and the gate determines the closed-gate rejection behavior.Source: B Technical proofs · pp. 37–38

B.1 Related lemmas

• The recalled empirical-likelihood lemmas give chi-squared limits with one degree of freedom under their stated conditions, including when unknown θP is replaced by a consistent estimator.Source: §B.1 · p. 38
• For independent test statistics and any configuration of false nulls, the recalled Benjamini–Hochberg result controls the false flagging rate at α.Source: §B.1 · p. 38

B.2 Derivation of the constrained empirical likelihood statistic

• For tolerance-based certification, the null restricts εG to [0, ε0] and Tc uses the constrained likelihood; Tc is zero when the estimate is at most ε0.The derivation uses the likelihood at ε0 relative to its unconstrained maximum when the estimate exceeds ε0.Source: §B.2 · p. 39

B.3 Proofs of the Theorems and Corollaries in section 3

• For fixed alternatives, consistency of the empirical-likelihood ratio yields a rejection probability converging to one.Source: §B.3 · pp. 40, 45
• The scalar empirical-likelihood expansion is applied to the estimating equation, yielding a uniform local expansion for the likelihood components.Source: §B.3 · pp. 41–42
• At the boundary, the joint central limit theorem gives asymptotic independence of the empirical-likelihood components, with T1 asymptotically equivalent to a squared Gaussian component.Source: §B.3 · p. 43