Source-linked AI summary
Auditing Black-box Models for Indirect Influence
Philip Adler, Casey Falk, Sorelle A. Friedler, Gabriel Rybeck, Carlos Scheidegger, Brandon Smith, Suresh Venkatasubramanian
TL;DR
Black-box models are difficult to audit for indirect feature influence, particularly when protected attributes affect outcomes through proxies. The paper introduces feature obscuring and differential accuracy analysis to audit such influence without retraining, and reports validation across datasets and models.
Problem
Black-box auditing commonly measures direct influence, but proxy features can preserve a protected attribute’s signal and conceal its indirect influence.
Method
The paper obscures a feature’s direct and proxy information through a theoretically justified procedure, then quantifies influence by comparing model behavior before and after obscuring.
Results
The procedure is experimentally validated on public datasets and varied models, with synthetic and feature-proxy results supporting its handling of indirect influence.
Takeaways & Limitations
The approach can reveal attribute influence that remains when the attribute is not directly used by the model, while preserving the existing black-box model.
Abstract
from arXiv · showhide
Data-trained predictive models see widespread use, but for the most part they are used as black boxes which output a prediction or score. It is therefore hard to acquire a deeper understanding of model behavior, and in particular how different features influence the model prediction. This is important when interpreting the behavior of complex models, or asserting that certain problematic attributes (like race or gender) are not unduly influencing decisions. In this paper, we present a technique for auditing black-box models, which lets us study the extent to which existing models take advantage of particular features in the dataset, without knowing how the models work. Our work focuses on the problem of indirect influence: how some features might indirectly influence outcomes via other, related features. As a result, we can find attribute influences even in cases where, upon further direct examination of the model, the attribute is not referred to by the model at all. Our approach does not require the black-box model to be retrained. This is important if (for example) the model is only accessible via an API, and contrasts our work with other methods that investigate feature influence like feature selection. We present experimental evidence for the effectiveness of our procedure using a variety of publicly available datasets and models. We also validate our procedure using techniques from interpretable learning and feature selection, as well as against other black-box auditing procedures.
I. INTRODUCTION
Black-box models are difficult to audit for feature influence, especially when protected attributes affect outcomes indirectly through proxy features. The paper proposes obscuring feature information and measuring accuracy changes to quantify such influence without retraining the model.
- Black-box models make it difficult to understand how features influence decisions, including decisions involving race, gender, or age.
- Standard audits often examine direct influence by perturbing a feature, but indirect influence can persist through related proxy features.
- The paper estimates a feature’s information content by predicting it from the remaining features, then minimally obscures that information before testing the model.
- The proposed technique quantifies indirect influence through differential feature-influence analysis before and after obscuring.
- Experiments validate the approach across public datasets and varied models, with additional comparisons to interpretable learning, feature selection, and black-box auditing procedures.
II. CONCEPTUAL CONTEXT
Existing black-box audits commonly measure direct feature effects through perturbation, but random perturbations can damage useful proxy information and obscure the feature’s relative influence. The paper instead uses directed, minimal perturbations based on whether a feature remains predictable from the others.
- The approach operates without retraining the black-box model, unlike methods that investigate feature influence through retraining or feature modification.
- Direct influence is commonly measured by replacing a feature with random noise and observing deterioration in model accuracy.
- Random perturbations may remove useful task information in proxy features and prevent clean measurement of the perturbed feature’s relative effect.
- The proposed perturbation is directed and deterministic: a feature is obscured when it can no longer be predicted from the remaining data, while changing the data minimally.
A. Obscuring data with respect to a feature
The method defines feature removal through unpredictability from the remaining data, then measures indirect influence as the model’s accuracy difference on original versus obscured data. Balanced error rate is used to handle class imbalance and skewed feature values.
- Balanced error rate is the unweighted average of class-conditioned errors and is chosen because it is more sensitive to class imbalance.
- A feature is considered removed when it can no longer be predicted from the remaining data.
- An epsilon-obscure dataset is defined so that the target feature cannot be predicted from the obscured remaining data.
- Indirect influence is the difference in classifier accuracy between the original dataset and its obscured version.
- The obscurity parameter is typically treated as large, such as above 0.5 for binary classification.
IV. COMPUTING INFLUENCE
Gradient feature auditing (GFA) estimates indirect influence by obscuring one feature while minimally changing related feature distributions. For numerical features conditioned on categorical ones, it uses a median distribution and earthmover-distance repair.
- GFA estimates indirect influence by applying an obscuring procedure feature-by-feature and comparing influence before and after obscuring.The procedure can remove full or partial effects, producing both scores and feature rankings.
- For numerical W and categorical O, GFA represents W conditionally on each value of O and constructs a median distribution A.A is chosen so that O is maximally obscured while W changes minimally.
- The numerical repair, called ObscureNumerical, moves conditional distributions toward A using earthmover distance.The method minimizes the summed distance between conditional distributions and A.
- The numerical-categorical procedure does not apply when the obscured and removed features have other type combinations.The paper introduces additional procedures for those cases.
A. Removing numerical features
The paper handles categorical features with exact-metric earthmover repair and supports partial influence removal. It also uses binning for numerical features and reports rankings alongside influence scores.
- A. Removing numerical features: Numerical features are binned and treated as categorical labels before applying the obscuring procedure.The bins use the Freedman-Diaconis rule, while lower-order numerical bits may be treated as measurement error.
- A. Removing numerical features: For categorical features, the exact metric makes earthmover distance equal to the L1 distance between distributions.The minimizing distribution is obtained component-wise by taking medians across conditional distributions.
- A. Removing numerical features: The categorical repair computes earthmover flows between each conditional distribution and the median distribution, producing fewer changes than arbitrary value replacement.The repair uses min-cost flows after computing the component-wise median.
- A. Removing numerical features: The resulting obscured data is used to compute X \ϵ Xi, with categorical or numerical procedures selected according to feature types.The full construction assumes complete removal, but the algorithms also support partial removal.
- A. Removing numerical features: Feature rankings complement influence scores when comparing GFA with auditing methods whose scores may not be directly meaningful.
C. Obscuring, ANOVA, and the F-test
The theoretical analysis shows that obscuring a feature with Gaussian conditionals makes the class-conditional distributions identical, causing the F-test to fail to reject equality. This connects the procedure to ANOVA.
- C. Obscuring, ANOVA, and the F-test: Under Gaussian conditionals, the obscuring procedure creates data on which the F-test does not invalidate the null hypothesis that the conditionals are identical.The paper presents this as a theoretical justification and a generalization of ANOVA.
- C. Obscuring, ANOVA, and the F-test: When the original classes are balanced Gaussians with means v−1 and v1 and shared variance σ2, obscuring replaces both conditional means with their average.The obscured conditional distribution is N(1/2(v−1 + v1), σ2).
- C. Obscuring, ANOVA, and the F-test: The F statistic is the ratio of between-group sample variance to in-group sample variance.With µ=(v−1+v1)/2 and δ=(µ−v−1)2=(µ−v1)2, the analysis relates δ to explained variance and σ2 to unexplained variance.
- C. Obscuring, ANOVA, and the F-test: After obscuring, both class-conditional distributions are identical Gaussians, and the between-group parameter δ concentrates near zero as sample size increases.The proof uses equal expectations and Hoeffding tail bounds.
V. EXPERIMENTS
The evaluation uses five datasets spanning synthetic, socioeconomic, criminal-justice, and materials-science settings, together with opaque SVM and neural-network models. The datasets and GFA code are made available online.
- V. EXPERIMENTS: The experiments use five datasets selected to balance easy replication with practically relevant domains.The paper states that the datasets and GFA code are available online.
- V. EXPERIMENTS: The synthetic dataset contains 6,000 items, balanced across two classes, with five features including correlated row-number features, a random feature, and a constant feature.
- V. EXPERIMENTS: Adult Income contains 48,842 people with 14 census attributes, while German Credit contains 1,000 people with 20 attributes.Adult Income uses its original split; German Credit uses a random training-test split.
- V. EXPERIMENTS: The Recidivism Prediction dataset contains 38,624 prisoners from 15 states, tracked for three years after release.It includes criminal-history and demographic attributes.
- V. EXPERIMENTS: Dark Reactions contains 3,955 hydrothermal synthesis experiments described by 273 attributes, with ionic-crystal production as the classification variable.
- V. EXPERIMENTS: The models include notoriously opaque support-vector machines and feedforward neural networks, with feedforward models trained separately across the datasets.The supplied implementation passage specifies softmax input layers and training settings for several feedforward models.
A. Black-box feature auditing
GFA evaluates feature influence by progressively obscuring variables and measuring changes in model accuracy across multiple datasets and model types. The resulting plots support influence rankings while also exposing model- and dataset-specific behavior.
- Adult income data: On Adult Income, race has small influence across models, whereas age strongly affects SVM and FNN accuracy but has much less impact on the decision tree.
- Adult income data: On the FNN Adult Income model, partially obscuring some features initially increases accuracy, consistent with reducing noise in a non-optimal model.
- Figure 1 organizes obscurity-versus-accuracy plots by model columns and dataset rows, enabling comparisons across C4.5 trees, SVMs, and FNNs.
- Running time: The experiments include model-training, auditing, and data-writing times ranging from 13 seconds to just over three hours.
B. Auditing for consistency
Because German Credit accuracy results are noisy, the paper introduces consistency against the model’s unobscured predictions and uses interpretable models to compare indirect and direct influence rankings. These analyses produce smoother rankings and identify features whose indirect influence differs from direct use.
- Auditing for consistency: Consistency replaces original labels with unobscured model predictions, so the baseline is always 100% and obscuring measures degradation relative to those predictions.
- Auditing for consistency: Under consistency, German Credit accuracy degrades smoothly, and credit amount, checking status, and existing credits rank as the top three features across all models.
- Direct influence audit: A model of a model trains an interpretable model on the black-box model’s predicted labels to audit direct feature influence.
- Direct influence audit: Feature importance in the interpretable model is estimated from the probability that each feature appears on paths from the root to training-set leaf nodes.
- Comparing audits: For Adult Income under C4.5, marital-status, education, race, age, and capital-gain are outliers where indirect and direct rankings differ, suggesting reconstructible proxy information.
D. Comparison to previous work
GFA differs from prior auditing methods by capturing both direct and indirect influence through proxy variables. Comparisons on synthetic and Adult Income data show that GFA identifies proxy-related feature importance that direct-influence approaches miss.
- Synthetic data: All models achieve 100% accuracy on the synthetic data, making GFA consistency and accuracy scores identical.
- Synthetic data: GFA captures both direct influence and indirect influence through proxy variables, unlike Henelius et al.’s direct-influence focus.On synthetic data, features B and C proxy feature A even though the decision tree directly splits only on A.
- Adult Income data: Henelius et al. and GFA produce different Adult Income rankings, with marital-status, fnlwgt, sex, education, education-num, and capital-gain as outliers in Figure 5.The figure relates Henelius et al. influence scores to GFA consistency scores using linear regression and a 95% confidence interval.
- Adult Income data: Education and education-num are perfectly reconstructable from the remaining Adult Income attributes and receive tied GFA importance scores of 0.65.Education-num is a numerical representation of education, making it a perfect proxy.
- Adult Income data: GFA assigns similar importance to marital-status and relationship, with consistency scores of 0.82 and 0.84 and accuracy scores of 0.77 and 0.81.Both features are highly, but not exactly, predictable from the remaining attributes, consistent with close but imperfect proxies.
- Related auditing methods: Datta et al.’s two-step procedure identifies proxy variables before ranking their individual direct influence, but does not account for their indirect influence on outcomes.
E. Comparison to feature selection
GFA and feature selection generate rankings from different questions: GFA measures feature importance for a specific existing model, while feature selection evaluates predictive importance for a future model.
- Adult Income and German Credit rankings show weak correlations between feature selection and GFA, including weaker consistency-ranking correlations for C4.5.The C4.5 consistency correlation was an order of magnitude weaker.
- Feature selection evaluates features with respect to an as-yet uninstantiated model, whereas GFA evaluates importance to a specific instance of an existing model.
- Feature selection indicates direct feature influence, while GFA focuses on indirect influence.
VI. DISCUSSION
Feature influence depends on the interaction between a model and its data. The discussion positions GFA among auditing, interpretability, fairness, privacy, and feature-selection approaches while noting that auditing methods remain difficult to compare.
- A useful audit should respond when obscuring creates differences the classifier uses, but remain insensitive to differences irrelevant to classifier behavior.This sensitivity to irrelevance is identified as an important property of a good auditing procedure.
- Comparing auditing approaches remains challenging because different methods can agree or disagree and use computational or statistical metaphors.The paper uses predictive power rather than hypothesis testing for its obscuring procedure.
- Relating predictive power to hypothesis testing could provide a combined mathematical and computational framework for evaluating black-box models.The paper presents this as a possible direction that might help unify existing audit approaches.
- Black-box audits connect to privacy-preserving data mining, security, interpretability, feature selection, and algorithmic fairness, but their trust models and goals differ.The paper identifies indirect-influence auditing as relevant to understanding whether variables affect decisions in fairness assessments.