Explaining by Removing: A Unified Framework for Model Explanation

Ian Covert; Scott Lundberg; Su-In Lee

Source-linked AI summary

Explaining by Removing: A Unified Framework for Model Explanation

Ian Covert, Scott Lundberg, Su-In Lee

arXiv:2011.14878v2Nov 21, 2020cs.LGstat.ML

TL;DR

The paper addresses the difficulty of relating diverse model-explanation methods and deciding how their choices differ. It formalizes removal-based explanations around feature removal, analyzed model behavior, and influence summarization, then uses related theories to explain their connections and trade-offs. The framework characterizes 26 existing methods and supports a broader theoretical account of their relationships.

Problem
Relationships and trade-offs among many model-explanation methods remain insufficiently investigated, making the literature difficult to navigate.
Method
The paper defines removal-based explanations and classifies methods by how they remove features, what model behavior they analyze, and how they summarize feature influence.
Results
26 existing explanation methods are unified within the framework, including local and global approaches plus feature attribution and feature selection methods.
Takeaways & Limitations
Cooperative game theory, information theory, and cognitive psychology provide theoretical perspectives on removal-based explanations and their method trade-offs.
Takeaways & Limitations
Distribution-consistent removal can make features appear important even when the model does not functionally use them.

Abstract

from arXiv · show

Researchers have proposed a wide variety of model explanation approaches, but it remains unclear how most methods are related or when one method is preferable to another. We describe a new unified class of methods, removal-based explanations, that are based on the principle of simulating feature removal to quantify each feature's influence. These methods vary in several respects, so we develop a framework that characterizes each method along three dimensions: 1) how the method removes features, 2) what model behavior the method explains, and 3) how the method summarizes each feature's influence. Our framework unifies 26 existing methods, including several of the most widely used approaches: SHAP, LIME, Meaningful Perturbations, and permutation tests. This newly understood class of explanation methods has rich connections that we examine using tools that have been largely overlooked by the explainability literature. To anchor removal-based explanations in cognitive psychology, we show that feature removal is a simple application of subtractive counterfactual reasoning. Ideas from cooperative game theory shed light on the relationships and trade-offs among different methods, and we derive conditions under which all removal-based explanations have information-theoretic interpretations. Through this analysis, we develop a unified framework that helps practitioners better understand model explanation tools, and that offers a strong theoretical foundation upon which future explainability research can build.

1. Introduction

The paper introduces removal-based explanations as a unified class organized by how features are removed, what model behavior is analyzed, and how influence is summarized. This framework connects existing methods and develops theoretical links to cooperative game theory, information theory, and cognitive psychology.

Framework: Removal-based explanations quantify feature influence by simulating feature removal, with methods differing in their removal operations.Examples include replacing features with neutral values, marginalizing over values, or training models for feature subsets.
Framework: Each method is specified by three choices: how it removes features, what model behavior it analyzes, and how it summarizes feature influence.The three choices systematize shared elements across approaches.
Framework: The framework links previously disparate explanation-method classes and makes their relationships and trade-offs easier to analyze.It connects local and global approaches with feature attribution and feature selection methods.
Theoretical connections: Removal-based explanations connect to cooperative game theory, information theory, and cognitive psychology.These connections are used to examine method trade-offs and the cognitive basis of feature removal.
Framework: 26 existing explanation methods are characterized within a unified framework for removal-based explanations.The framework also compares 60+ new approaches created by combining choices across its dimensions.

2. Background

The background positions interpretability as the challenge of explaining complex-model predictions and reviews perturbation-based and existing unification approaches. It motivates a broader framework centered on feature removal and three precise mathematical choices.

Preliminaries: Interpretability seeks insight into model predictions, including local explanations of individual predictions and global explanations across datasets.Feature attribution assigns influence scores, while feature selection isolates relevant features or sets of features.
Preliminaries: Existing categories of interpretability methods are loose collections that often lack a precise shared mechanism.This motivates identifying a common mechanism across otherwise disparate approaches.
Preliminaries: Many explanation methods perturb inputs and measure corresponding changes in model predictions, ranging from gradients to other perturbation operations.Perturbation-based approaches include methods that alter or remove input features.
Related work: Prior unification work related several additive attribution methods to SHAP and connected modified gradient methods.These efforts established relationships among important subsets of the explanation literature.
Related work: The proposed unification is broader because it centers on feature removal and includes both local and global methods.It also analyzes connections with cooperative game theory, information theory, and cognitive psychology.
Related work: The framework is the first described here to consider feature removal, analyzed model behavior, and influence summarization simultaneously.The authors distinguish these three dimensions from choices previously considered separately.

3. Removal-Based Explanations

Removal-based explanations define feature influence through the impact of removing feature groups, and a three-choice framework unifies diverse methods. This framework exposes shared structure, neighboring methods, and unexplored combinations.

Removal-based explanations are specified by feature removal, model behavior, and summary technique.
The three framework dimensions are independent, but all are necessary to fully specify a removal-based explanation.
The framework connects diverse interpretability areas and identifies empty positions as opportunities for new methods.
Disentangling methods reveals that many share choices and differ along only one or two dimensions, clarifying their conceptual and computational trade-offs.

4. Feature Removal

Feature removal requires functions that operate on arbitrary feature subsets while remaining invariant to held-out values. Existing methods implement removal through diverse conventions, including replacement, marginalization, and blurring.

Subset functions accept an input and a feature subset, while their outputs remain invariant to dimensions outside that subset.
A subset extension of a model agrees with the original model when all features are present and provides the basis for removal-based explanations.
Existing methods remove features by setting zeros or defaults, extending neighboring pixel values, blurring, or sampling from generative models.
Other methods remove features by marginalizing with conditional, marginal, product-of-marginals, uniform, or replacement distributions.
Blurring and LIME’s tabular approach retain dependence on removed features, so the authors argue they do not fully remove features.

5. Explaining Different Model Behaviors

Removal-based explanations can target different model behaviors by defining a set function over feature subsets. These targets range from individual predictions and losses to dataset-level performance and model-output dependence.

The framework can explain virtually any function by observing its output, loss, hidden layer, or computation-graph node as inputs are removed.
Prediction-based methods analyze how withholding features raises or lowers an individual model output, optionally transformed into logits or log-odds.
Prediction-loss methods incorporate the true label to quantify whether features make an individual prediction more or less correct.
Dataset-loss methods measure how feature removal degrades model performance across the dataset, whereas output-loss methods compare subset predictions with the full model output.
Shapley Effects instead measure feature influence on model output across the dataset rather than on model performance.

6. Summarizing Feature Influence

Because feature-subset behavior is exponentially large, removal-based explanations summarize influence through attribution scores or selected subsets. Methods differ in summary rules and computational demands, creating distinct trade-offs.

Feature attributions provide real-valued scores for each feature, while feature selection provides coarse binary importance through selected subsets.
Summary techniques include individual removal or inclusion, additive models, Shapley values, low- or high-value subsets, and partitioned subsets.
RISE estimates attribution by averaging set-function values over sampled subsets in which each feature is included.
Shapley Net Effects, IME, Shapley Effects, QII, SHAP, SPVIM, and SAGE use Shapley values for feature attribution.
Exact summarization can require evaluating all 2^d feature subsets, including for Shapley values, RISE, and LIME’s additive model.
Although some techniques are faster or trade computational cost for approximation accuracy, the authors report that all are sufficiently fast for practical use.

7. Game-Theoretic Explanations

Removal-based explanations can be understood as cooperative games whose features are players, and their attribution or selection procedures correspond to allocation strategies or coalition analyses. This perspective connects methods such as Shapley values, Banzhaf values, LIME, and feature-selection approaches while exposing trade-offs among them.

7.1 Cooperative-game foundations: Every removal-based explanation can be interpreted through cooperative game theory because its analyzed set function is a cooperative game.Features act as players, feature sets as coalitions, and the model-derived set function as coalition value.
7.2 Attribution as allocation: Removal-based attribution summaries correspond to allocation strategies, linking feature influence estimates to cooperative-game solution concepts.The framework treats explanation mappings and game-theoretic solution concepts as mappings from games to vectors of player payoffs.
7.3 Modeling cooperative games: The Shapley value is one allocation strategy, while Banzhaf values and individual-player removal or inclusion methods arise from alternative weighting choices.LIME’s weighted least-squares formulation can produce Shapley, Banzhaf, and simpler marginal-contribution quantities.
7.3 Modeling cooperative games: The additive-model view shows that many removal-based feature attributions are additive decompositions of an underlying cooperative game.This connection includes the allocation strategies generated by several attribution techniques, with RISE using a modified Banzhaf value.
7.4 Identifying coalitions using excess: Feature-selection methods identify coalitions by examining excess under equal allocations, separating high-valued from low-valued players.MM considers both coalitions, while other methods focus primarily on either the high-valued or low-valued coalition; several are special cases of MM.
7.5 Summary: The framework uses cooperative-game properties to compare methods and argues that the Shapley value offers the most complete explanation through efficiency and granular contribution accounting.The authors note that whether Shapley efficiency or Banzhaf 2-efficiency is preferable may depend on the use case.

8. Information-Theoretic Explanations

The paper connects removal-based explanations to probability and information theory by requiring feature-removal conventions to respect an underlying data distribution. Under appropriate removal, these explanations admit probabilistic and information-theoretic interpretations, while different choices remain useful for different explanatory goals.

Probabilistic interpretation: A subset function can be interpreted as a conditional distribution over responses given the retained features.For classification, F(xS) represents q(Y | XS = xS); regression uses the corresponding conditional expectation.
Consistency: Unique subset extensions for classification and regression are obtained by averaging the full model output over the conditional distribution of removed features.The same conclusion holds for conditional probabilities and conditional expectations.
Consistency: Feature-removal strategies may achieve consistency under simplifying assumptions such as feature independence, but these assumptions rarely hold in practice.Default-value replacement corresponds to a constant distribution and is justified when the relevant independence assumptions apply.
Conditional-distribution approximations: Accurately modeling the conditional distribution is difficult, especially for high-dimensional or continuous-valued data.Separate models offer strong approximations but do not scale well, whereas generative models amortize many conditional distributions into one model.
Conditional-distribution approximations: Generative-model and optimization-based approaches can approximate conditional marginalization, with reported performance improvements for several adapted explanation methods.The cited results include improvements across numerous metrics and equivalence results for objectives using MSE or cross entropy under specified conditions.
Information-theoretic interpretations: Precise information-theoretic interpretations arise when held-out features are marginalized using their conditional distribution.Under appropriate removal, every removal-based explanation has a probabilistic or information-theoretic interpretation, including mutual-information and variance-decomposition connections in specific cases.
Interpretive trade-offs: No single set function is presented as universally correct; the resulting information-theoretic quantities represent different perspectives for understanding complex models.Distribution-consistent explanations can reveal intrinsic statistical relationships, but may highlight features not functionally used by the model.

9. A Cognitive Perspective on Removal-Based Explanations

Removal-based explanations connect model interpretability to cognitive theories of causality through subtractive counterfactual reasoning. The framework also exposes a practical trade-off between explanations that are simple to understand and those that convey richer information.

9.1 Counterfactual reasoning: Removal-based explanations quantify feature influence by subtracting a feature or event and observing the resulting outcome.This is called a subtractive counterfactual in psychology and corresponds to Mill’s method of difference in philosophy.
9.1 Counterfactual reasoning: The feature-removal principle is widespread in social science theories, helping explain its prevalence in model explanation methods.Many computational methods use feature removal without explicitly drawing on psychology research.
9.1 Counterfactual reasoning: Removing a feature is difficult because the method must retain the observed features while eliminating the information communicated by the removed ones.The framework highlights ambiguity in how to construct the model input after removal.
9.2 Norm theory and the downhill rule: Norm theory supports averaging model outputs over alternative values for removed features, while the downhill rule favors plausible alternatives that increase a story’s coherence.Norm theory does not prescribe one specific distribution for those alternative values.
9.3 Simplicity versus completeness: People report greater satisfaction with explanations citing fewer causes, while longer explanations can require more effort and impede appropriate conclusions.The paper therefore treats simplicity as a design goal and recommends tailoring explanation complexity to users.

10. Experiments

Experiments show that explanation behavior depends strongly on feature-removal and summary choices, with conditional-distribution surrogates and Shapley values often producing strong results. They also reveal important evaluation biases and method-specific limitations.

10.1 Explanation comparisons: Different removal strategies can yield similar explanations when they approximate the conditional distribution, whereas default-value and uniform approaches often diverge.Default and uniform strategies are described as lower-quality conditional-distribution approximations in these experiments.
10.1 Explanation comparisons: Mean when included produces markedly different explanations from other summary techniques, while Shapley and Banzhaf values are relatively close.The similarity reflects their shared use of weighted averages of marginal contributions.
10.2 MNIST: Uniform and product removal strategies frequently produce noisy explanations, while default-value explanations assign zero attribution to zero-valued pixels.This is problematic because zero pixels can nevertheless be informative.
10.2 MNIST: Removing individual features can create noise artifacts when individual removals have negligible model impact, likely because of strong feature correlations.The paper notes that individual removal may be incompatible with close conditional-distribution approximations.
10.2 MNIST: The surrogate model with Shapley values produces the most visually appealing MNIST explanations and roughly corresponds to LossSHAP.The explanations highlight distinguishing digit regions with minimal noise artifacts.
10.2 MNIST: Evaluation metrics can favor explanation methods whose feature-removal choices match the metric’s missing-feature replacement procedure.Zero replacement favors default-value methods, while surrogate handling favors surrogate-based explanations; Shapley values perform best in both cases.
10.2 MNIST: Separate-model evaluation avoids directly mirroring any explanation method but still favors methods that approximate the conditional distribution.It also measures the information content of selected features independently of the original model’s prediction mechanism.

11. Discussion

The framework systematizes removal-based explanations through three mathematical choices and connects disparate methods to game theory, information theory, and psychology. These connections clarify relationships, trade-offs, practical use, and limitations across explanation approaches.

Framework: Each method is specified by how it removes features, what model behavior it analyzes, and how it summarizes feature influence.The behavior may concern an individual prediction or the entire dataset, while summaries produce feature attributions or feature selections.
Framework: Many state-of-the-art methods share a foundation, differ through interchangeable choices, and vary along only one or two framework dimensions.This systematic characterization makes their conceptual and computational advantages easier to compare.
Theoretical connections: Removal-based explanations connect to subtractive counterfactual reasoning, cooperative game theory, and information theory.These perspectives provide richer interpretations of methods and help explain their trade-offs.
Trade-offs: The Shapley value offers a complete model summary but is slower than the fastest removal-based explanations and can be complex or require causal variants.These limitations are especially relevant in model-agnostic settings and when users need easily interpretable explanations.
Theoretical connections: Conditional-distribution marginalization is the only removal approach identified as consistent with standard probability axioms, while alternative strategies can approximate it.The approach also gives removal-based explanations information-theoretic interpretations, though it can be challenging to implement.
Implications: The framework enables practitioners to reason explicitly about tool trade-offs and potentially develop hybrid methods, while guiding researchers in specifying and justifying new approaches.It also supports resolving shortcomings through solutions along different framework dimensions.

Appendix A. Method Details

The appendix details how image-based and instance-wise feature-selection methods formulate feature removal or preservation, often through masks, segmentation, or learned selectors. It also clarifies assumptions and optimization choices behind these methods.

Image-based methods: Meaningful Perturbations modifies images by setting pixels to constants, replacing them with Gaussian noise, or blurring them, with blurring recommended.Its mask controls the Gaussian blur standard deviation through σ · m_i.
Image-based methods: Extremal Perturbations changes Meaningful Perturbations from a removal game to a preservation game and replaces subset-size penalties with constraints.It retains the same blurring operation while maximizing classification probability for a fixed blurred portion.
Image-based methods: FIDO-CA resembles Extremal Perturbations but replaces blurring with features sampled from a conditional generative model.The method can condition on arbitrary feature subsets and recommends a contextual-attention GAN.
Image-based methods: Minimal Image Representation greedily removes image segments that least reduce classification probability until the image is misclassified.The appendix interprets this procedure as solving a constrained optimization problem.
Instance-wise selection: Masking Model methods frame explanation as either finding the smallest destroying region or the smallest sufficient region.The learned explainer model approximates optimization solutions in a single forward pass, with penalties encouraging smoothness and sparsity.
Instance-wise selection: L2X and INVASE use learned selector models for instance-wise feature selection, whereas REAL-X addresses label leakage caused by jointly learning selectors and subset functions.L2X’s information-theoretic interpretation requires specific model and subset-function assumptions.

Appendix B. Additive Model Proofs

The appendix shows that weighted least squares over cooperative-game set functions recovers several familiar feature-attribution methods through different weighting kernels. These kernels produce occlusion-style, univariate, Banzhaf, and Shapley attributions.

Weighted least squares: Weighted least squares provides a common framework for summarizing cooperative-game set functions, with weighting kernels and regularization determining the resulting attributions.Setting the regularizer to zero allows several attribution schemes to arise from different kernels.
Kernel choices: The kernel πInc(S) = 1(|S| ≤1) yields a_i = u({i}) − u({}), matching Occlusion, PredDiff, CXPlain, permutation tests, and feature ablation.The first group corresponds to local importance, while the latter methods summarize global feature importance.
Kernel choices: The kernel πEx(S) = 1(|S| ≥d−1) yields a_i = u(D) − u(D\{i}), matching the univariate predictors approach.This kernel weights coalitions missing no more than one player.
Kernel choices: The unweighted least-squares problem is optimized by the Banzhaf values as its unique coefficients.The appendix verifies optimality by showing that the gradient is zero and the objective is jointly convex.
Kernel choices: A specific weighting kernel makes the weighted least-squares problem optimize the Shapley value.The appendix directs readers to existing proofs of this connection.

C.1 Additive model axioms

The additive-model analysis identifies which Shapley-like axioms weighted least squares satisfies under assumptions on its kernel and regularizer. Feature-selection methods have analogous properties, but not enough to determine a unique axiomatic approach.

Weighted least-squares axioms: The weighted least-squares analysis assumes a nonnegative finite kernel except possibly at the empty and full coalitions, convex regularization, and strict convexity of the weighted objective.Permutation-invariant kernels and regularizers are additionally required for symmetry.
Weighted least-squares axioms: Efficiency holds only when π({}) = π(D) = ∞, imposing b0 = u({}) and Σ_i∈D b_i = u(D) − u({}).These constraints correspond to faithful linear approximations in cooperative game theory.
Weighted least-squares axioms: The dummy property holds for Shapley and Banzhaf kernels but not for arbitrary kernels and regularizers.Additivity holds when the regularizer is Ω = 0, while marginalism follows from identical learned coefficients under identical marginal contributions.
Feature-selection axioms: Feature-selection methods satisfy analogues of symmetry, dummy, and marginalism because they output optimal coalitions rather than attribution vectors.Their axioms describe coalition optimality instead of allocation properties.
Feature-selection axioms: Feature-selection explanations do not appear to satisfy analogues of efficiency or additivity, and their analogous properties do not determine a unique axiomatic method.This contrasts with the stronger characterization available for attribution approaches.

Appendix D. Consistency Proofs

The consistency proofs establish that conditional marginalization gives a unique subset extension of a full model under a feature distribution. Separately trained subset models recover this extension when they optimize population risk under suitable losses.

Unique subset extensions: For classification models estimating conditional probabilities and regression models estimating conditional expectations, probability laws yield unique consistent subset extensions.The classification and regression claims are stated as Propositions 6 and 7.
Unique subset extensions: The consistent subset extension marginalizes missing features using the conditional distribution induced by q(X).Derivations from the full model and from intermediate subsets produce the same result.
Proof strategy: The proof uses countable additivity, Bayes rule, and expectation identities to derive conditional predictions for feature subsets.The displayed derivations are presented first for discrete variables, with analogous reasoning stated for continuous variables.
Separate subset models: Under those population-risk assumptions, separately trained subset models are equivalent to marginalizing missing features from the full model with their conditional distribution.This connects model-removal approaches based on separate subset models to the unified subset-extension framework.
Separate subset models: For regression, population-risk minimization with MSE gives f_S(x_S) = E[Y | X_S = x_S].For classification, cross-entropy or another strictly proper scoring function gives f_S(x_S) = p(Y | X_S = x_S).

E.2 Missingness during training

The section analyzes training-time missingness and surrogate models as ways to approximate conditional marginalization of removed features. Independence between masks and inputs is central, while INVASE can violate this condition.

Missingness during training: INVASE replaces removed features with zeros or other values so the model can recognize missingness and predict from available information.A replacement value observed in the dataset creates ambiguity between genuine and missing values.
Missingness during training: For classification with cross entropy and regression with MSE, the optimal missingness-trained model uses the conditional distribution or expectation of the held-out information.The result extends to other strictly proper scoring functions for classification.
Missingness during training: If masks depend on inputs, selected features can communicate information about held-out features, invalidating the conditional-prediction and information-theoretic interpretation.This issue may prevent INVASE from approximating p(Y | X_S = x_S).
Missingness during training: Training with missing features can approximate conditional marginalization when masks are sampled independently of model inputs.Uniformly random masks or the Appendix E.3 mask distribution satisfy this training requirement.
Surrogate models: Surrogate models trained to match an existing model’s predictions can also approximate conditional marginalization when masks are sampled independently from inputs.The result holds for MSE and cross entropy comparisons, but not for L2X when its mask-input independence condition is unmet.
Mask sampling: Sampling subset sizes uniformly avoids concentrating masks near subsets containing approximately half of the features.The procedure samples k uniformly from 0 through d, then samples k indices without replacement.

Appendix F. Information-Theoretic Connections in Regression

The regression analysis gives information-theoretic interpretations to removal-based set functions under Gaussian assumptions. Without those stronger distributional assumptions, Euclidean-distance interpretations are more conservative.

Information-theoretic connections: Regression set functions can have information-theoretic interpretations when models are evaluated with MSE and model optimality is assumed.The analysis treats conditional expectations, explained variance, and distances involving model outputs or responses.
Conditional expectation: The set function u_x(S) = F(x_S) quantifies the response variable’s conditional expectation and examines each feature’s true relationship with the response.This interpretation is stated directly for the conditional-expectation set function.
Mutual information and KL divergence: Under Gaussian assumptions, squared response distances relate to pointwise mutual information and negative KL divergence.The KL-divergence relationship requires Gaussian conditional distributions for both the full and partial feature settings.
Explained variance: Explained variance relates to mutual information for response variables and model outputs through entropy-maximizing Gaussian bounds.Equality in the first bound requires the relevant conditional distribution to be Gaussian, while the second bound follows from Jensen’s inequality.
Caveat: The KL-divergence interpretation for a deterministic model output is ultimately infinite, making that result somewhat contrived.The Gaussian assumption permits the formal connection, but the resulting divergence does not provide a finite quantity.

G.2 Additional results

Additional experiments examine MNIST explanations and BRCA explanation similarity. LossSHAP highlights both digit pixels and empty regions, while BRCA correlations reveal systematic relationships among removal and summarization strategies.

MNIST explanations: LossSHAP explanations consistently highlight important MNIST digit pixels and empty regions that distinguish digits from alternative classes.The examples include digits such as 3, 4, and 9.
BRCA explanation similarity: Figure 14 compares BRCA dataset loss explanations using mean correlation and Spearman rank correlation.The two measures are shown in the top and bottom portions of the figure, respectively.
BRCA explanation similarity: Shapley-value explanations are relatively similar to explanations using individual-feature removal or Banzhaf values, whereas the include-individual technique differs most.Banzhaf explanations also correlate with mean-based include explanations, as predicted by the theory.