Source-linked AI summary
Distill-and-Compare: Auditing Black-Box Models Using Transparent Model Distillation
Sarah Tan, Rich Caruana, Giles Hooker, Yin Lou
TL;DR
Black-box risk scoring models are widely used in high-stakes settings but are often proprietary or opaque. The paper introduces Distill-and-Compare, which trains transparent models on black-box scores and outcomes, and finds evidence that ProPublica’s COMPAS data may omit key features.
Problem
Black-box risk scoring models support high-stakes decisions, but auditors may lack unrestricted API access, model forms, training data, or all features used by the model.
Method
Distill-and-Compare trains a transparent mimic model on black-box risk scores and a second transparent outcome model on ground-truth outcomes, then compares them and tests for missing features.
Results
The statistical test found positive correlation for COMPAS, indicating that ProPublica’s data may be missing key features used in COMPAS; other datasets showed little or no evidence of missing key features.
Takeaways & Limitations
Transparent model comparisons can surface previously unknown differences and biases in complicated real-world data without requiring advance selection of protected features.
Takeaways & Limitations
COMPAS mimicry may be difficult because ProPublica’s data may omit key features and contains fewer than 7,000 samples.
Abstract
from arXiv · showhide
Black-box risk scoring models permeate our lives, yet are typically proprietary or opaque. We propose Distill-and-Compare, a model distillation and comparison approach to audit such models. To gain insight into black-box models, we treat them as teachers, training transparent student models to mimic the risk scores assigned by black-box models. We compare the student model trained with distillation to a second un-distilled transparent model trained on ground-truth outcomes, and use differences between the two models to gain insight into the black-box model. Our approach can be applied in a realistic setting, without probing the black-box model API. We demonstrate the approach on four public data sets: COMPAS, Stop-and-Frisk, Chicago Police, and Lending Club. We also propose a statistical test to determine if a data set is missing key features used to train the black-box model. Our test finds that the ProPublica data is likely missing key feature(s) used in COMPAS.
1 INTRODUCTION
Risk scoring models guide high-stakes decisions but are often proprietary or opaque. Distill-and-Compare audits them using only audit data containing model scores, outcomes, and some or all features.
- Motivation: Risk scoring models are used in criminal justice, finance, hiring, and other critical domains, increasingly for high-stakes decisions.These models are designed to predict future outcomes such as loan defaults.
- Limitations of existing audits: Existing audits often perturb protected features and retrain models or probe model APIs, but proprietary creators may restrict API access and withhold model details.Commercial model creators often do not release the model form or training data.
- Audit setting: The audit setting provides model-produced risk scores, ground-truth outcomes, and some or all features without allowing new queries to the black-box API.The audit data may differ from the original training data and may omit features used by the black-box model.
- Approach: The paper proposes Distill-and-Compare to analyze feature effects, followed by a statistical test for additional black-box features absent from the audit data.Both steps use audit data containing black-box risk scores and ground-truth outcomes.
- Contributions: The contributions include auditing four risk scoring models, calibrating scores to remove scale distortions, and estimating confidence intervals for iGAM comparisons.The paper also proposes a test for missing key features used to train the black-box model.
2 AUDIT APPROACH
The audit trains transparent models on black-box scores and ground-truth outcomes, then compares their feature contributions. A shared interpretable model class and calibration make these comparisons meaningful, while a statistical test assesses missing features.
- Distill and Compare: Distill-and-Compare treats the black-box risk scoring model as a teacher and trains a transparent mimic model on its risk scores.The audit data supplies samples and teacher outputs without requiring new API queries.
- Distill and Compare: The outcome model is trained on the same audit data to predict binary ground-truth outcomes and is distinct from the mimic model.The two models therefore target risk scores versus outcomes.
- Distill and Compare: Both models use the same interpretable model class and features, so differences can reflect discrepancies between black-box risk scores and ground-truth outcomes.Greater mimic fidelity and outcome accuracy increase confidence in interpreting these differences.
- Testing for missing features: The missing-feature test checks whether mimic error is positively correlated with outcome-model error when omitted features help predict the ground-truth outcome.This requires training both models before testing the errors.
- Choice of model class: iGAM represents predictions through visualizable feature or pairwise contributions, allowing two models to be compared by subtracting corresponding contributions.The model uses a logistic link for classification and an identity link for regression.
- Calibrating model inputs: Calibration transforms risk scores so they are linearly related to outcomes on a shared logit-probability scale before mimic training.Calibration is applied to Chicago Police and Lending Club, but not COMPAS or Stop-and-Frisk.
- Comparing models: Feature-contribution differences are computed on the same data sample; positive values mean the mimic assigns more risk than the outcome model.Negative values indicate the converse.
3 RESULTS
Experiments show that transparent mimic models can recover known black-box behavior and expose disagreements with ground-truth outcome models. Across audits, the approach identifies feature use, model differences, and evidence that COMPAS data may omit key features.
- Stop-and-Frisk: The Stop-and-Frisk mimic model recovered coefficients (3, 1, 1) for PS, AS, and Bulge, assigning 0 to remaining features.This matches the known functional form of the risk scoring model.
- Chicago Police: For Chicago Police, the mimic model assigned importance to the eight reported features but none to the eight reported-unused features.The outcome model assigned importance to the unused features, indicating available predictive signal that the black-box model did not use.
- Interpretation: The experiments confirm that mimic models can provide insight into black-box models while using outcome information available in audit data.Observed differences cannot be assigned a definitive explanation without further testing, but transparent model comparisons surface differences for investigation.
- COMPAS: COMPAS mimic and outcome models agreed on Number of Priors but differed for older ages, young individuals, race groups, and gender.For ages greater than 70, confidence intervals did not overlap; the mimic model also predicted low risk for young individuals without corresponding outcome evidence and showed different race and gender effects.
- Lending Club: Lending Club interactions showed higher mortgage-associated default risk in 2007–2008 than from 2009 onward, with a larger outcome-model range than mimic-model range.The difference could indicate conservative updating with lag in the Lending Club risk scoring model.
- Missing features and fidelity: The missing-feature test found evidence that ProPublica COMPAS data may omit key features, while Lending Club and Stop-and-Frisk showed correlations indistinguishable from zero.For COMPAS, no tested mimic model, including random forests, could mimic COMPAS well; the paper also notes small sample size as another possible explanation.
4 DISCUSSION
Excluding protected features does not eliminate bias because correlated features can encode their signal. Distill-and-Compare therefore includes protected features in transparent mimic models to inspect whether the black-box model uses them.
- Removing race or gender does not prevent bias when correlated features, such as income or education, remain available to the model.
- Excluding protected features can make bias harder to detect or correct because it becomes distributed across correlated features.
- The mimic model intentionally includes protected features to reveal whether the black-box model assigns them predictive signal.
5 CONCLUSION
The paper audits opaque risk-scoring models using transparent, comparable models when only scored audit data and outcomes are available. Its usefulness depends on model fidelity, outcome accuracy, and sufficiently complete audit features, while avoiding the need to preselect suspected biases.
- Distill-and-Compare operates without black-box API access, using risk scores, ground-truth outcomes, and features in an audit data set.
- The audit is more effective when the shared transparent model class is faithful to black-box scores and accurate on ground-truth outcomes.
- Transparent models can expose unknown sources of bias without requiring auditors to choose protected features in advance.
A A NEW CONFIDENCE INTERVAL ESTIMATE FOR IGAM
The appendix introduces bootstrap-of-little-bags confidence intervals for iGAM feature contributions and differences between mimic and outcome models. The estimate is conservative, and the resulting intervals are pointwise rather than uniform.
- Bootstrap-of-little-bags estimates variability in iGAM feature contributions and in differences between mimic and outcome contributions.The procedure uses repeated inner and outer data splits, producing KL trained models whose average supplies the mean contribution.
- The method constructs pointwise confidence intervals for individual feature contributions and mimic–outcome contribution differences.
- The variance estimate is conservative, overestimating true variability and reducing the chance of mistaking random noise for real differences.
- The intervals describe variability at specific feature values rather than the entire feature effect; uniform intervals require adjusting the critical value.
B CALIBRATION PLOTS
Calibration plots compare empirical probabilities with risk scores on probability and logit-probability scales. Linear relationships permit direct mimic training, whereas nonlinear relationships require monotonic risk-score transformations first.
- Figure 6 plots empirical probability against risk score on probability and logit-probability scales, with fitted red lines and learned blue monotonic curves.
- COMPAS and Stop-and-Frisk show relationships supporting direct mimic training on raw risk scores.
- Chicago Police and Lending Club show nonlinear relationships, so their risk scores must be calibrated before mimic training.
- Figure 7 checks whether the transformed risk score and logit empirical probability have a linear relationship.