Source-linked AI summary
Evaluating and Aggregating Feature-based Model Explanations
Umang Bhatt, Adrian Weller, José M. F. Moura
TL;DR
The paper addresses the lack of quantitative guidance for choosing among many feature-based explanation functions. It defines sensitivity, faithfulness, and complexity, then develops aggregation methods for combining explanations and optimizing these criteria. The resulting framework includes lower-complexity aggregates and an aggregate Shapley-value explanation designed to minimize sensitivity.
Problem
The growing number of feature-based explanation functions makes it difficult to choose which function best captures how a model reaches a specific output.
Method
The paper defines three evaluation criteria and develops an aggregation framework, including procedures for reducing complexity and an aggregate Shapley-value explanation.
Results
The paper reports empirical findings supporting AVA when low sensitivity is desired; AVA also has lower sensitivity in all experiments and can produce less complex explanations.
Takeaways & Limitations
The framework provides practitioners with a principled way to evaluate feature-based explanations and learn aggregates optimized for criteria desired by end users.
Takeaways & Limitations
Faithfulness estimates based on sampled feature subsets may be inaccurate, and learning global feature importances remains future work.
Abstract
from arXiv · showhide
A feature-based model explanation denotes how much each input feature contributes to a model's output for a given data point. As the number of proposed explanation functions grows, we lack quantitative evaluation criteria to help practitioners know when to use which explanation function. This paper proposes quantitative evaluation criteria for feature-based explanations: low sensitivity, high faithfulness, and low complexity. We devise a framework for aggregating explanation functions. We develop a procedure for learning an aggregate explanation function with lower complexity and then derive a new aggregate Shapley value explanation function that minimizes sensitivity.
1 Introduction
The paper addresses the difficulty of choosing among growing numbers of feature-based explanation functions by proposing quantitative criteria and aggregation methods. It defines sensitivity, faithfulness, and complexity as desirable properties and introduces aggregation, including an aggregate Shapley-value explanation.
- The paper targets the difficulty of selecting an explanation function that captures how a model reaches a specific output.
- It proposes three criteria for feature-based explanations: low sensitivity, high faithfulness, and low complexity.
- The aggregation framework combines multiple explanation functions into an aggregate explanation function.
- Two techniques reduce explanation complexity by aggregating explanation functions.
- The aggregate Shapley-value explanation uses nearest neighbors to minimize explanation sensitivity and resemble human reasoning in medical settings.
2 Preliminaries
The paper formalizes supervised classification explanations as feature-importance vectors produced for a predictor and an input. It also defines distance metrics and evaluation criteria over predictors, explanations, and inputs.
- A black-box predictor maps an input x ∈ R^d to an output f(x) ∈ Y in supervised classification.
- An explanation function maps a predictor and input to feature-importance scores φ_x ∈ R^d.
- Candidate explanation functions are represented as G_m = {g_1, g_2, . . . , g_m}.
- The framework uses distance metrics over explanations and inputs, while an evaluation criterion returns a scalar μ(f, g; x).
- The dataset D contains input-output pairs, and D_x denotes all inputs x_i in D.
3 Evaluating Explanations
The paper evaluates explanations using sensitivity, faithfulness, and complexity. These criteria respectively assess local smoothness, correspondence between attributions and output changes, and the concentration of feature contributions.
- The evaluation framework defines three desiderata: low sensitivity, high faithfulness, and low complexity.
- Low Sensitivity: Low sensitivity requires nearby inputs with similar model outputs to receive close explanations, implying local smoothness around x.
- Low Sensitivity: Average sensitivity evaluates explanation variation using a distribution P_x over inputs centered at x.
- High Faithfulness: Faithfulness measures whether attribution sums for selected features correlate with the model-output change caused by replacing those features with a reference baseline.
- High Faithfulness: Estimating faithfulness from randomly sampled fixed-size subsets may produce an inaccurate criterion estimate.
- Low Complexity: Complexity is the entropy of the fractional contribution distribution P_g, with equal contributions across features representing a complex explanation.
4 Aggregating Explanations
The paper aggregates multiple explanation functions using convex combinations or distance-based centroids, obtaining consensus explanations with formal sensitivity and error properties.
- An aggregate explanation combines candidate functions through a consensus rule h(Gm) designed to satisfy a chosen criterion at least as well as individual candidates.
- 4.1 Convex Combination: Convex combinations can produce aggregates whose average sensitivity is no greater than that of the individual explanation functions.Under the stated uniform-distribution assumption, the optimized combination is no more sensitive than the least sensitive candidate.
- 4.2 Centroid Aggregation: For centroid aggregation, ℓ2 distance with p = 2 yields the feature-wise sample mean, whereas ℓ1 distance with p = 1 yields the feature-wise sample median.
- 4.2 Centroid Aggregation: Rank-valued attributions can be aggregated using the Kemeny-Young rule under Kendall-tau distance.
- 4.2 Centroid Aggregation: For mean aggregation, the paper analyzes the error relative to an optimal explanation and states a bound on aggregate error.The analysis defines each candidate’s error as its distance from the optimal explanation and averages across inputs.
- 4.2 Centroid Aggregation: Aggregation can reduce noise and yield better explanations than using one explanation function alone, with typical aggregate error below the expected individual error.
5 Lowering Complexity Via Aggregation
The paper develops two iterative procedures for combining explanations while reducing complexity: directional walks and convex-hull region shrinking.
- The goal is an aggregate gagg(f, x) that incorporates all candidate explanations while having entropy no greater than each candidate’s entropy.
- The example shows that averaging can increase complexity: two minimum-complexity candidates have complexity 0, while their sample mean has maximum complexity log(2).
- 5.1 Gradient-Descent Style Method: The directional-walk method moves candidate explanations toward the sample mean, and the sample mean toward candidates, only when complexity decreases.
- 5.1 Gradient-Descent Style Method: The final directional-walk aggregate is the lowest-complexity location among the 2d walks, with entropy guaranteed not to exceed every candidate’s entropy.
- 5.2 Region Shrinking Method: In the region-shrinking method, the convex hull of candidate explanations is iteratively reduced by selecting minimum-complexity points on pairwise line segments.
- 5.2 Region Shrinking Method: After K iterations, gagg is the minimum-complexity member of the new candidate set, whose minimum complexity decreases or remains constant each iteration.
6 Lowering Sensitivity Via Aggregation
The paper constructs AVA by aggregating Shapley explanations from nearby training points, producing an explanation for a test point that is less sensitive to input perturbations.
- Aggregate Valuation of Antecedents: AVA explains a test point using the Shapley-value explanations of its k nearest neighbors under distance metric ρ.The neighbors are identified in the training dataset before their explanations are aggregated.
- Aggregate Valuation of Antecedents: The aggregation weights each neighbor’s Shapley contribution by inverse distance to the test point and normalizes the weights to sum to one.This makes AVA a convex combination of neighboring explanation functions.
- Aggregate Valuation of Antecedents: The resulting AVA explanation is itself a vector of Shapley values by linearity of Shapley values.The proof uses additivity under scalar multiplication and summation of characteristic functions.
- Aggregate Valuation of Antecedents: Because AVA is a specific convex combination, the paper states that it has lower sensitivity than using the Shapley explanation of the test point alone.The same aggregation perspective contrasts AVA with fixed-baseline or straight-line attribution paths.
- Aggregate Valuation of Antecedents: AVA’s neighbor-based construction is motivated by medical reasoning, where past similar patients complement present patient attributes when diagnosing rare diseases.The paper presents this as an analogy between nearby training points and past patients.
7 Experiments
Experiments evaluate explanation criteria across four datasets and several explanation methods. The reported results support AVA for reducing sensitivity, with complexity benefits when the feature dimension is small.
- Experimental setup: Experiments use Adult, Iris, MIMIC, and MNIST with SHAP, Shapley Sampling, Gradient Saliency, Grad*Input, Integrated Gradients, and DeepLift.The tabular models are multilayer perceptrons, while MNIST uses a convolutional neural network.
- Faithfulness: Faithfulness is measured by correlating attribution sums for perturbed feature subsets with changes in predicted logits relative to baseline values.The evaluation averages multiple runs and varies the subset size.
- Sensitivity: Sensitivity is evaluated using mean and maximum explanation distances between test points and nearby training points within radius r.The procedure averages results over ten random runs and depends on dimensionality d and radius r.
- Sensitivity: For MIMIC, weighted-least-squares Shapley values are reported as the least sensitive explanation method.The paper also reports that sensitivity decreases as the neighborhood radius increases.
- Complexity: Both aggregation approaches can lower explanation complexity slightly, although image data may be less suitable than tabular data for complexity optimization.The qualitative image example illustrates complexity reduction, where minimum complexity would concentrate attribution on one pixel.
- AVA: AVA lowers Shapley-explanation sensitivity across all datasets and is slightly less complex when the number of features d is small.The reported complexity difference depends on the radius r and feature count d.
8 Conclusion
The paper codifies three desirable explanation properties and develops aggregation methods to optimize them, while leaving joint multi-criterion optimization for future work.
- The paper defines low sensitivity, high faithfulness, and low complexity as desirable properties of feature-based explanations.
- It develops an aggregation scheme for combining explanation functions and procedures that target lower complexity or lower sensitivity.
- The framework is intended to help practitioners evaluate explanations and learn aggregates aligned with end-user criteria.
- The paper considers one criterion at a time and identifies multi-objective optimization across criteria as future work.
A Additional Evaluation Criteria
The appendix collects quantitative criteria beyond the paper’s three central properties, covering explanation consistency, surprise, model compatibility, feature relevance, and retraining-based evaluations.
- Consistency criteria: Identity favors explanations that remain identical across repeated calls with the same input, while separability favors different explanations for non-identical inputs.
- Distributional criteria: Conviction measures how surprising an explanation is relative to training-point explanations, with higher values indicating more predictable behavior.
- Distributional criteria: Conditional conviction evaluates surprisal relative to explanations from points sharing the same predicted class.
- Output criteria: Compatibility assesses how closely an explanation recovers the original predictor’s outputs, with values of µCOM nearer zero indicating better compatibility.
- Feature relevance criteria: Deletion and addition evaluate log-odds changes after removing important features or adding them to a baseline, respectively.
- Retraining criteria: ROAR and KAR retrain predictors after removing important or least-important features and compare the resulting accuracy with the original predictor.
B Proofs
The proofs establish aggregation identities by showing that coordinate-wise means minimize squared-distance loss, while coordinate-wise medians minimize absolute-deviation loss.
- Mean aggregation: The squared-distance aggregation proof reduces to showing that the mean minimizes the sum of squared distances among explanation vectors.
- Median aggregation: For ℓ1 distance, the proof defines an absolute-deviation loss over candidate explanations and examines its coordinate-wise derivative.
- Median aggregation: The coordinate-wise median minimizes absolute deviations because it balances the numbers of values above and below it.
B.4 Alternative Proof of Theorem 5
The appendix proves that AVA produces Shapley values through linearity and compares two heuristic strategies for finding aggregate explanations with lower complexity.
- Alternative proof of Theorem 5: Linearity and additivity of Shapley values imply that weighted combinations of neighboring Shapley contribution vectors remain valid Shapley explanations.
- Alternative proof of Theorem 5: The proof concludes that each component of the AVA output is a Shapley value for the characteristic function constructed from neighbors’ average weighted importance scores.
- Complexity optimization: The gradient-descent approach is greedy and component-wise, works best in smooth decreasing regions, and can take sub-optimal steps because it does not backtrack.
- Complexity optimization: The region-shrinking approach repeatedly retains low-entropy points from pairwise combinations, and combining both algorithms can refine candidates toward lower complexity.
D Experimental Setup
Experiments use fixed dataset–model pairings across several tabular, clinical, and image-classification settings. Average sensitivity is computed to compare explanation changes for nearby inputs, alongside prior smoothing and adversarial-training approaches.
- Datasets and models: Experiments cover Iris, Adult, MIMIC-III sepsis diagnosis, and MNIST using MLP or CNN predictors.The datasets span small tabular data, a larger tabular classification task, electronic health records, and digit images.
- Datasets and models: The Iris dataset contains 150 datapoints, four features, and three classes, with a one-layer MLP trained to 96% accuracy.
- Datasets and models: The Adult dataset contains 48,842 datapoints with 38 features and two classes, using label-encoded categorical attributes and a one-layer MLP trained to 82% accuracy.The MLP has 40 hidden nodes and leaky-ReLU activation.
- Datasets and models: MIMIC-III supplies seventeen real-valued features for binary sepsis diagnosis, extracted from critical-care records of over 40,000 patients.The source records include demographics, vital signs, laboratory tests, medications, procedures, notes, imaging reports, and mortality.
- Datasets and models: The MNIST experiment uses 60,000 training and 10,000 test images with a CNN trained to 90% accuracy.The CNN includes two convolutional layers, max-pooling layers, and a final ten-output dense layer.
- Sensitivity evaluation: Average sensitivity assesses whether explanation differences between nearby inputs are proportional to the inputs’ distance, with smoothing and adversarial predictor training as related reduction methods.All experiments keep each dataset paired with a fixed model.