VFR-Audit: Verdict-Level Reliability for Fairness Audits in Hospital Length-of-Stay Prediction

Md Jannatul Rakib Joy; Viet Vo; Caslon Chua

Source-linked AI summary

VFR-Audit: Verdict-Level Reliability for Fairness Audits in Hospital Length-of-Stay Prediction

Md Jannatul Rakib Joy, Viet Vo, Caslon Chua

arXiv:2608.30846v1Aug 31, 2026cs.AI

TL;DR

Fairness audits need to determine whether thresholded verdicts remain reproducible across resampling, audit sizes, and hospitals, beyond uncertainty in continuous metrics. VFR-Audit introduces VFR and three reliability axes to make this assessment operational. On Texas-100X, intervention could satisfy point-estimate fairness while verdicts still flipped, and VFR-Audit achieved the same DI pass as B3 at 0.36 pp lower accuracy cost.

Problem
Fairness verdicts may change with cohort composition, audit size, and hospital shift, while existing uncertainty methods do not directly provide scalable binary-verdict stability claims.
Method
VFR-Audit uses the Verdict Flip Rate under stratified bootstrap resampling alongside within-cohort stability, audit-size guidance, and cross-hospital agreement.
Results
VFR-Audit raises the same four-fifths-rule pass as B3 at 0.36 pp lower accuracy cost; 11 of 28 post-intervention cells still flipped under bootstrap despite point-estimate fairness.
Takeaways & Limitations
Fairness status should be reported together with audit reliability because point-estimate passes can remain unstable under resampling or site shift.
Takeaways & Limitations
Texas-100X is a retrospective administrative discharge dataset, and the leakage-sensitivity analysis does not establish prospective admission-time deployability.

Abstract

from arXiv · show

Fairness audits in clinical Artificial Intelligence convert continuous fairness metrics into binary pass-or-fail verdicts against operational thresholds, where hospital governance boards, payers, and regulators act on the resulting verdicts. Such audits are repeated over time and across hospital sites, thus the same verdict can flip between pass and fail across audits. Existing uncertainty methods such as Bayesian posteriors, bootstrap confidence intervals, and permutation tests address verdict instability only at the continuous-metric level. Converting metric-level uncertainty into a verdict-stability claim remains a manual step that scales poorly across the (model, metric, attribute) cells an audit covers. Existing uncertainty methods also leave open whether bias-mitigation steps, such as reweighing or per-group threshold shifts, yield a stable passing verdict at the cost of model discrimination measured as AUROC or AUPRC.To address this verdict-stability gap, we propose VFR-Audit, a framework built around the Verdict Flip Rate (VFR), a scalar bounded between 0 and 0.5 that measures the probability of verdict reversal under stratified bootstrap resampling. VFR-Audit reports VFR alongside three reliability axes, namely within-cohort resampling stability, audit-size sensitivity, and cross-hospital verdict agreement via Fleiss' kappa.

1 Introduction

Fairness verdicts can change across cohorts and deployment settings, creating a need to assess reproducibility beyond point-estimate fairness. VFR-Audit addresses this gap with VFR and three reliability axes, demonstrated on Texas-100X.

Cohort composition, subgroup size, and hospital case mix can change fairness verdicts for an unchanged model, especially near operational thresholds.The paper frames reproducibility under cohort resampling, audit-size variation, and hospital shift as the central problem.
On Texas-100X, 146 of 336 unintervened baseline cells reversed verdicts under stratified bootstrap resampling.The cohort contains 9.25×10^5 discharge records from 441 hospitals and spans 12 classifiers, seven fairness metrics, and four protected attributes.
After the canonical intervention, 11 of 28 cells still flipped despite point-estimate fairness across all four protected attributes.Intersectional reweighing alone failed the four-fifths rule, whereas per-cell threshold shifting produced the all-four-DI pass.
VFR measures binary-verdict instability under stratified bootstrap resampling as the smaller pass or fail count divided by K, bounded between 0 and 0.5.VFR equals zero when all resamples agree and 0.5 when pass and fail counts are equal.
VFR-Audit combines within-cohort resampling stability, audit-size guidance, and cross-hospital agreement into a four-tier per-cell reliability classification.The framework is intended to operationalise repeated deployment audits across the covered reliability dimensions.

2 Related Work

Prior LOS and clinical-AI fairness studies report predictive performance and subgroup disparities, but generally do not provide a scalar instrument for verdict stability. VFR-Audit targets this missing combination of resampling stability, cross-site agreement, and audit-size guidance.

2.1 Length-of-Stay Prediction: LOS prediction studies report binary and regression performance across hospital datasets, while fairness coverage in the LOS literature remains sparse.Reported examples include AUROC, R^2, MAE, and subgroup AUC comparisons across several cohorts.
2.1 Length-of-Stay Prediction: Table 1 compares LOS-focused literature across six audit-relevant axes, including verdict stability, cross-site validation, and audit-size guidance.“n/r” indicates that an axis was not reported.
2.2 Fairness Audits in Clinical AI: Existing clinical-AI fairness frameworks such as FINS and MACAIF do not define a scalar measure of binary-verdict instability or validate it on a state-wide clinical cohort.FINS targets subset selections, while MACAIF is a clinician-facing dashboard layered on an adversarial tool.
2.3 Audit-Uncertainty and Verdict Stability: Established fairness uncertainty methods estimate metric uncertainty through Bayesian posteriors, bootstrap confidence intervals, permutation p-values, and sample-size formulae.These approaches are presented as related uncertainty-quantification branches rather than as a unified verdict-level instrument.
2.3 Audit-Uncertainty and Verdict Stability: Prior verdict-stability work shows that fairness conclusions can change with training seeds or predictive multiplicity, but does not provide the proposed stratified-bootstrap governance instrument.The related LOS literature also lacks reported cross-hospital verdict agreement and audit-size guidance.

3 Proposed Method

VFR-Audit treats thresholded fairness verdicts as the operational object and evaluates their reliability under resampling, audit-size variation, and hospital-grouped validation. Its three axes distinguish verdict instability, metric stabilization needs, and cross-hospital agreement for fixed trained models and audit protocols.

Axis 1: Resampling stability: VFR measures binary-verdict instability across stratified bootstrap resamples, bounded in [0, 0.5].VFR equals the smaller of pass and fail counts divided by the number of resamples.
Framework scope: VFR-Audit applies reliability analysis to fixed trained models and fixed prediction scores on held-out audit data.The procedure separates model training from audit-reliability estimation.
Axis 1: Resampling stability: With 500 resamples, Axis 1 reports VFR, the dominant verdict, and a stability margin for each (model, metric, attribute) cell.A cell is practically stable when VFR ≤ 0.10, corresponding to at least 90% agreement with the dominant verdict.
Axis 2: Audit-size sensitivity: Axis 2 estimates the minimum audit size N* where a metric’s coefficient of variation falls below 5% across 30 repetitions.The method sweeps an audit-size grid and produces a coefficient-of-variation curve and minimum reliable size.
Axis 3: Cross-hospital agreement: Axis 3 evaluates cross-hospital verdict agreement using hospital-grouped validation and Fleiss’ kappa.It produces a fold-by-cell verdict matrix and a metric-level Fleiss’ kappa across protected-attribute cells.
Combined reliability tier: The combined reliability tier is conjunctive: Practical-Stability requires VFR ≤ 0.10, N* ≤ N_field, and kappa ≥ 0.40.Violating one, two, or all three conditions yields Caution-Required, High-Variance, or Catastrophic-Instability, respectively.

4.1 Experimental Setting

The experiment uses the Texas-100X multi-hospital cohort to compare four fairness-audit baselines across classifiers, fairness metrics, and protected attributes. VFR-Audit applies threshold adjustment with a hard all-four-DI constraint and evaluates audit reliability, while holding the audit setup constant across baselines.

Dataset: Texas-100X contains 925,128 inpatient discharge records from 441 Texas hospitals, with records split into training and test partitions for LOS prediction.The binary target is LOS exceeding three days, with an 80/20 stratified split using random seed 42.
Models: Twelve standard tabular classifiers were trained with fixed, model-specific hyperparameters frozen before fairness auditing.This prevents fairness outcomes from influencing model configuration.
Audit instrument: The audit evaluates seven group-fairness metrics across race, sex, ethnicity, and age, producing 28 cells per model and 336 cells across the twelve-model panel.Operational thresholds include DI ≥0.80, |SPD|, |EOpp|, |EOD|, and |PP| ≤0.10, TI ≤0.10, and |Cal| ≤0.05.
Baselines: Four baselines use the same dataset, split, feature pipeline, bootstrap seed, and hospital-fold partition so differences in VFR, N* and Fleiss’ kappa reflect the intervention.The baselines are Real-only, reweighing, threshold-shifting, and VFR-Audit.
VFR-Audit: VFR-Audit starts from threshold shifting and greedily adjusts group-specific decision thresholds while preserving DI ≥0.80 for all four protected attributes.Candidate moves are accepted when they improve the selected VFR objective; ties favor higher accuracy within the intervention-selection partition.
Dataset exploratory analysis: Age has the largest cohort base-rate gap, making it the binding constraint for DI-targeted interventions, while race is a secondary constraint.The reported Age and Race base-rate gaps are 0.399 and 0.189, respectively; the race-by-ethnicity coding is also non-standard.

4.2 Evaluation Questions and Results

Across evaluation questions, VFR-Audit achieves all-four protected-attribute DI passage while improving verdict reliability over threshold-shifting, but residual instability, audit-size sensitivity, and uneven cross-hospital agreement remain.

EQ2. Fairness-Intervention Comparison: VFR-Audit and Threshold-shifting satisfy the four-fifths rule on all four protected attributes, whereas Real-only and Reweighing satisfy it on only 1 of 4.Reweighing can move fairness away from the threshold: Race-DI falls from 0.644 to 0.575, and its best swept Age-DI is 0.283.
EQ2. Fairness-Intervention Comparison: VFR-Audit reaches accuracy 0.8352, mean VFR 0.0809, and maximum VFR 0.476, improving on Threshold-shifting’s 0.8316, 0.0863, and 0.490.The improvement versus Threshold-shifting is +0.36 percentage points in accuracy and reflects verdict reliability rather than a new point-estimate fairness pass.
EQ3. Verdict-Reliability Analysis: On Baseline 1, 146 of 336 audit cells reverse at least once, while 259 of 336 satisfy the operational stability cutoff VFR ≤0.10.Race-axis cells are most unstable, including VFR values of 0.480 for EOpp, 0.386 for EOD, and 0.458 for PP.
EQ3. Verdict-Reliability Analysis: After all-four-DI point-estimate passage, 11 of 28 VFR-Audit cells still flip, with 21 of 28 meeting VFR ≤0.10.VFR-Audit redistributes instability: Age DI and SPD decrease, while Race DI and SPD increase, leaving residual instability concentrated on Race-axis cells.
EQ4. Adoption Cost: Only 11 of 28 VFR-Audit cells reach CV <5% at field-realistic audit sizes; 17 of 28 exceed CV =50% at N=10^3.Eight cells require the full 185,026-record test partition, so quarterly audits of approximately 5 × 10^3 records may require pooling across quarters or sites.
EQ5. Robustness Checks: At K_hosp =20, Fleiss’ κ is moderate for EOpp (0.465) and EOD (0.587), but slight or below-chance for DI (−0.035), SPD (0.117), PP (0.315), and Cal (0.003).Theil reaches κ=1.000 because it satisfies the threshold on every fold; κ is descriptive agreement, not a causal transportability estimate.
EQ1. Predictive Utility: AUROC remains 0.9528 while VFR-Audit reduces accuracy by 4.24 percentage points and F1 by 4.64 percentage points relative to the unintervened XGBoost baseline.Threshold shifting preserves score ranking, so AUROC preservation is mechanically expected.
EQ5. Robustness Checks: Across the twelve-classifier panel, Race-axis threshold shifting moves every classifier above the four-fifths rule at an accuracy cost of 4–6 percentage points.On the canonical model, VFR-Audit preserves the all-four-DI point-estimate passage while reducing aggregate verdict instability.

5 Discussion

VFR-Audit separates fairness status from verdict reliability, showing that point-estimate passes can remain unstable and that mitigation can redistribute instability across protected attributes. Its reliability conclusions persist under stricter validation-audit and admission-lean checks.

Reliability reporting: VFR-Audit jointly reports the dominant verdict, VFR, minimum reliable audit size, and cross-hospital agreement to separate fairness status from verdict reliability.Calibration remains a failure mode on this cohort, with VFRCal ≈0, a consistent Fail verdict, and κCal ≈0.
Cost-benefit trade-off: VFR-Audit pays 4.24 pp in accuracy relative to Baseline 1, 0.36 pp less than Threshold-shifting while achieving the same point-estimate DI pass.Both methods pass the four-fifths rule on every protected attribute on the point estimate.
Mitigation mechanism: Reweighing fails on this dataset because the Age base-rate gap creates a feature-based floor on the achievable selection-rate ratio.The Elderly LOS > 3 d rate is 0.606 versus 0.207 for young adults, a 2.93× ratio; reweighing changes the optimization objective but not the input features.
Robustness: The audit-reliability conclusions persist under both the stricter validation-audit protocol and the admission-lean sensitivity check.These checks support robustness beyond the primary split and feature configuration.

6 Conclusion

VFR-Audit is a model-agnostic audit layer for testing whether thresholded fairness verdicts remain reliable under resampling, audit-size variation, and hospital shift. On Texas-100X, its robustness checks support reporting audit reliability alongside fairness status when making deployment decisions.

Conclusion: VFR-Audit assesses whether thresholded fairness verdicts remain reliable under cohort resampling, audit-size variation, and hospital shift.The framework is model-agnostic.
Conclusion: On Texas-100X, point-estimate fairness can remain unstable even after intervention satisfies the four-fifths rule.The conclusion motivates reporting reliability alongside fairness status for deployment decisions.
Conclusion: Stricter validation-audit splits, admission-lean features, and an independent seed support the conclusion that audit reliability should accompany fairness status.These are the robustness checks named in the conclusion.