Explaining by Removing: A Unified Framework for Model Explanation

Motivation

• The framework responds to an explainability literature whose relationships, trade-offs, and links to human explanation processes had not been rigorously formalized.The paper presents the framework as a theoretical foundation for understanding existing methods and guiding future work.Source: §1 · p. 2

Framework

• The framework characterizes 26 existing explanation methods—including SHAP, LIME, Meaningful Perturbations, and permutation tests—by how they remove features, what model behavior they analyze, and how they summarize influence.It also integrates local and global approaches with feature attribution and feature selection methods.Source: §1 · pp. 2–3
• The framework exposes relationships among previously disparate method classes and shows that many differences reflect minor, interchangeable design choices.The authors compare 60+ new explanation approaches by mixing choices across the framework’s three dimensions.Source: §1 · pp. 2–3

Information theory

• Conditional-distribution marginalization is identified as the only feature-removal approach consistent with standard probability axioms, while alternatives can approximate it and support information-theoretic interpretations.The result depends on incorporating an underlying data distribution.Source: §1 · p. 3

Cooperative game theory

• Removal-based explanations are implicitly connected to cooperative game theory, which clarifies the advantages of the Shapley value and shows that LIME and Masking Model generalize attribution and selection techniques, respectively.The cited generalizations are LIME and Masking Model.Source: §1 · p. 3

Cognitive psychology

• Feature removal is grounded in social science as subtractive counterfactual reasoning, equivalently Mill’s method of difference from the philosophy of scientific induction (Mill, 1884).This connection supplies a cognitive basis for removal-based explanations.Source: §1 · p. 3

2.1 Preliminaries

• Interpretability addresses local predictions and global model behavior, commonly assigning feature-attribution scores while also using feature selection or relevant feature sets.Existing approaches include perturbation-, gradient-, propagation-based, and inherently interpretable models, but these categories rarely share a precise mechanism.Source: §2.1 · pp. 4–5
• Many explanation methods perturb inputs and measure prediction changes; this work interprets numerous perturbation strategies as simulations of feature removal.The reviewed methods range from infinitesimal gradient perturbations to broader input perturbations (Simonyan et al., 2013; Smilkov et al., 2017; Erion et al., 2019).Source: §2.1 · p. 5

2.2 Related work

• The proposed framework unifies local and global removal-based methods by jointly specifying feature removal, analyzed model behavior, and influence summarization.It is broader than prior unifications and connects explanation methods with cooperative game theory, information theory, and cognitive psychology; the authors describe it as the first framework to consider all three dimensions simultaneously.Source: §2.2 · p. 5

3.1 A unified framework

• The framework unifies 26 explanation methods by specifying how features are removed, what model behavior is analyzed, and how feature influence is summarized.These three independent dimensions are jointly necessary: removal and behavior probe arbitrary feature subsets, while the summary technique condenses exponentially many combinations into an interpretable explanation.Source: §3.1 · pp. 6–7

3.2 Overview of existing approaches

• The framework exposes common choices and trade-offs across diverse methods, with individual predictions often explained and Shapley values serving as the most popular summary technique.Methods can differ along only one or two dimensions, making their relationships easier to analyze than their monolithic algorithm descriptions suggest.Source: §3.2 · p. 7
• The removal-based perspective organizes methods into crowded and isolated regions, while empty combinations identify opportunities for new explanation methods.Examples of crowded regions use conditional feature marginalization with Shapley values, whereas RISE, tabular LIME, and INVASE are relatively isolated; SHAP, SAGE, Occlusion, PredDiff, and conditional permutation tests have neighbors along every framework dimension.Source: §3.2 · pp. 8–9

4.1 Functions on feature subsets

• Subset functions produce outputs determined only by a specified feature subset, remaining invariant to held-out values and thereby accommodating missing data.A subset extension additionally agrees with the original model when all features are present.Source: §4.1 · p. 10

4.2 Removing features from machine learning models

• Blurring in Meaningful Perturbations and Extremal Perturbations, and LIME’s tabular replacement approach, still depend on removed features and therefore do not fully satisfy the subset-function invariance property.The framework’s first dimension is the choice of a subset function, often constructed as an extension of an existing model.Source: §4.2 · pp. 13–14

• Removal-based explanations can explain virtually any function, including predictions, losses, hidden layers, computation-graph nodes, and non-ML functions that accommodate missing inputs.The framework represents the chosen behavior with a set function u : P(D) 7→R over feature subsets.Source: §5 · pp. 14, 16
• Prediction-focused methods explain how withholding features changes an individual model output, optionally transformed into logits or log-odds.Occlusion, RISE, PredDiff, MP, EP, MM, FIDO-CA, MIR, LIME, SHAP, IME, and QII belong to this group.Source: §5 · p. 15
• Other methods target prediction loss, expected loss, or dataset loss to identify features that improve correctness or whose removal degrades overall performance.LossSHAP and CXPlain use true labels; INVASE averages over p(Y | X = x); SAGE, SPVIM, feature ablation, permutation tests, and univariate predictors use dataset-wide loss.Source: §5 · p. 15
• L2X and REAL-X compare subset predictions with the full output, whereas Shapley Effects measure each feature’s influence on output rather than model performance.The former highlights features sufficient to reproduce the full prediction; the latter uses expected loss relative to the full model output.Source: §5 · pp. 15–16
• These set functions span local and global explanations and are mathematically related through added losses or data distributions, including SAGE as the expectation of LossSHAP explanations.The framework exposes these connections by separating each method’s choices rather than treating methods as monolithic algorithms.Source: §5 · p. 16

6.1 Explaining set functions

• Removal-based explanations use explanation mappings to summarize complex set functions as either real-valued feature attributions or binary feature selections.Attributions provide granular influence scores, whereas selections identify a subset of influential features as coarse attributions.Source: §6.1 · pp. 17, 19
• The framework includes individual removal or inclusion, additive models, mean-inclusion scores, Shapley values, and optimization-based subset selections.Shapley-based methods include SHAP and related approaches; selection methods identify low-value, high-value, or partitioned subsets.Source: §6.1 · pp. 17–18

6.2 Complexity and approximations

• Despite sometimes intractable objectives, approximations make removal-based explanations sufficiently fast for practice, with amortized approaches considerably faster and accuracy traded against computational cost.The framework therefore exposes computational trade-offs among explanation mappings while preserving practical usability.Source: §6.1 · p. 19, §6.2 · p. 21
• Many summarization techniques require evaluating all 2^d feature subsets, while individual-feature methods require only d subsets.This makes several methods superpolynomial and potentially intractable for large feature sets without approximation.Source: §6.2 · p. 20
• Sampling, regression on smaller datasets, dynamic programming, continuous relaxations, greedy optimization, and learned explainer models reduce summarization cost.TreeSHAP exploits tree structure, while amortized methods generate explanations in a single forward pass.Source: §6.2 · pp. 20–21

7. Game-Theoretic Explanations

• Every removal-based explanation implicitly defines a cooperative game, enabling game-theoretic interpretations of its feature-removal and summarization procedures.Set functions underlying removal-based explanations are cooperative games, and all methods analyzed can be understood through this lens.Source: §7 · p. 21, §7.5 · p. 27

7.1 Cooperative game theory background

• Cooperative games assign features as players, feature sets as coalitions, coalition outputs as values, and feature effects as marginal contributions, while allocation concepts connect directly to attribution mappings.This terminology supplies the formal bridge between cooperative game theory and removal-based explanations.Source: §7.1 · p. 22, §7.2 · p. 22

7.2 Allocation strategies

• Removal and inclusion of individual features are probabilistic values that use a single marginal contribution, whereas Shapley and Banzhaf values average contributions across coalitions and therefore capture interactions more fully.The simpler methods neglect complex interactions, while Shapley’s efficiency property uniquely distributes the grand coalition’s value among players; whether efficiency or Banzhaf’s 2-efficiency is preferable depends on the use case.Source: §7.2 · p. 24

7.3 Modeling cooperative games

• The additive-model formulation unifies the attribution methods discussed, including marginal contributions, Banzhaf values, and Shapley values, as decompositions of an underlying cooperative game.LIME’s weighted least-squares objective can produce these attribution schemes through different weighting kernels, while RISE uses a modified Banzhaf value.Source: §7.3 · p. 25

7.4 Identifying coalitions using excess

• Feature-selection explanations can be reformulated as minimizing or maximizing coalition excess under equal allocations, identifying influential high- or low-valued feature coalitions.MP, MIR, L2X, EP, INVASE, REAL-X, FIDO-CA, and MM optimize related excess-based objectives; MM considers both complementary coalitions and generalizes some alternatives.Source: §7.4 · p. 27

7.5 Summary

• Shapley values provide the most complete explanation because they satisfy many desirable properties, divide the grand coalition’s value, and give a granular view of each feature’s contribution.By contrast, Banzhaf values violate efficiency, and feature-selection methods provide coarse, hyperparameter-dependent summaries that can select only one feature from a correlated group.Source: §7.5 · pp. 27–28

8.1 Removing features consistently

• Consistent subset functions must obey probability axioms and remain compatible with an underlying feature distribution, although consistency does not guarantee predictive correctness.Inconsistent removal approaches may fail to define valid conditional distributions; allowing alternative q(X) exposes their implicit distributional assumptions.Source: §8.1 · pp. 29–30
• For both classification and regression, the unique consistent subset extension averages full-model outputs over removed features’ conditional distribution.This recovers conditional marginalization, including the conditional SHAP version, SAGE, and related methods (Zintgraf et al., 2017; Aas et al., 2019; Covert et al., 2020; Frye et al., 2020).Source: §8.1 · p. 30

8.2 Conditional distribution approximations

• Several practical strategies approximate conditional marginalization because exact conditional distributions are difficult to model, including independence, parametric, generative, surrogate, missingness-training, and separate-model approaches.Independence-based and default-value methods rely on restrictive assumptions, whereas generative models can sample conditionally; surrogate and missingness objectives can be equivalent under suitable losses and independently distributed subsets.Source: §8.2 · pp. 31–32
• Separate models may provide the best approximation but do not scale to high-dimensional data, while generative, surrogate, and missingness-training approaches trade approximation quality against scalability.Generative models amortize knowledge of exponentially many conditional distributions, and supervised surrogate or missingness objectives require learning at most one additional model.Source: • Separate models · p. 33

8.3 Connections with information theory

• Precise information-theoretic interpretations require marginalizing held-out features with their conditional distribution; under model optimality, subset predictions are Bayes-optimal for the retained features.These assumptions yield conditional probabilities for classification and conditional expectations for regression, enabling probabilistic interpretations of explanation set functions.Source: §8.3 · pp. 33–34
• The cited set functions capture distinct information notions: pointwise mutual information, negative KL divergence, mutual information, and the information or predictive power that selected features communicate about the response variable.Source: §8.3 · p. 35
• When features are removed appropriately, explanation methods quantify feature information, but no single set function is universally correct; the alternatives provide complementary perspectives on complex models.The interpretations approximate these quantities when models are trained toward Bayes classifiers or conditional expectations, even though exact optimality rarely holds in practice.Source: §8.3 · pp. 36–37
• Removal explanations consistent with the observed data distribution can uncover intrinsic statistical relationships and hidden influences, including bias from sensitive attributes.They are also useful when machine learning is used to discover real-world relationships.Source: §8.3 · p. 37
• These explanations may deem features important even when the model does not use them functionally, so causal-analysis-based interventional approaches may better explain prediction mechanisms.The passage points to interventional approaches motivated by causal analysis as an alternative when users prioritize how predictions are calculated.Source: §8.3 · p. 37

9. A Cognitive Perspective on Removal-Based Explanations

• Removal-based explanations have clear parallels with cognitive theories of causality, linking feature removal to counterfactual reasoning and a trade-off between simple and information-rich explanations.This social-science perspective complements the framework’s mathematical analysis.Source: §9 · p. 37

9.1 Subtractive counterfactual reasoning

• Removal-based explanations implement subtractive counterfactual reasoning: removing a feature simulates subtracting an observed fact to quantify its influence on the model’s prediction.This concerns model-focused causality—what makes the model behave this way—not whether a feature causes the real-world outcome.Source: §9.1 · pp. 37–38

9.2 Norm theory and the downhill rule

• Averaging model outputs over alternative feature values has cognitive support from Norm theory, while the downhill rule favors alternatives weighted by their plausibility, motivating conditional marginalization.Norm theory does not prescribe one distribution, but conditional distributions account for the coherence of the full feature set.Source: §9.2 · pp. 38–39

9.3 Simplicity versus completeness

• Removal-based explanations can balance simplicity and completeness through global summaries, feature groups, or sparse selections, because richer explanations may overload users and impede correct conclusions.Users may understand incomplete explanations better when they are tailored to their sophistication and emphasize fewer causes.Source: §9.3 · pp. 40–41

10. Experiments

• The experiments evaluated 80 methods—68 new—across feature-removal strategies, model behaviors, and summary techniques, covering census-income probabilities, MNIST loss, and breast-cancer dataset loss.The study included default replacement, several marginalization distributions, surrogate models, separate models, and summaries based on individual features, mean when included, Banzhaf values, and Shapley values.Source: §10 · p. 41

10.1 Census income

• Across the census and MNIST experiments, conditional-distribution approximations generally produced more coherent explanations, whereas uniform or product-based removal could be noisy or inaccurate under feature dependence.On MNIST, uniform and product removal often produced noisy explanations, while removing individual features could fail when strong correlations made their model impact negligible.Source: §10.1 · pp. 42, 46, §10.2 · pp. 46, 48
• Census-income explanations were most similar when using probabilistic summary values and conditional-distribution removal, whereas mean when included, default values, and uniform marginalization diverged.Shapley and Banzhaf values were relatively close; product and joint marginal approaches were nearly identical, and the surrogate approach most closely matched separate-model explanations on average.Source: §10.1 · pp. 44–45

10.2 MNIST

• MNIST evaluation metrics favored explanations whose removal strategy matched the masking procedure, while a separate missingness-trained model reduced this alignment but still favored conditional-distribution approximations.Zero masking favored default-value explanations, surrogate masking favored surrogate explanations, and Shapley values performed best across the insertion/deletion metrics because they aggregate contributions across subset cardinalities.Source: §10.2 · pp. 49–51
• On MNIST, surrogate/Shapley explanations were most visually appealing and achieved the best evaluation under missingness-trained models, corresponding roughly to LossSHAP.The explanation highlighted digit-discriminating regions with minimal noise, while surrogate-based evaluation significantly outperformed other removal strategies.Source: §10.2 · pp. 48, 50, 51

10.3 Breast cancer subtype classification

• Surrogate removal with Shapley values performed best overall for selecting informative genes in breast-cancer subtype classification, roughly corresponding to SAGE (Covert et al., 2020).The surrogate approach generally outperformed other removal strategies, while mean when included performed comparably well.Source: §10.3 · p. 51
• Feature-removal methods that neglect feature interactions or remove uniformly distributed features performed poorly in BRCA subtype classification.Figure 12 evaluates models trained with 1–20 top genes identified by global explanations, using average loss where lower is better.Source: §10.3 · p. 53, Figure 12 · p. 53
• SAGE ranked genes with documented BRCA associations, including ESR1, CCNB2, and TXNL4B, alongside genes associated with other cancers.The other cancer-associated genes included DDC and GSS.Source: §10.3 · p. 53
• The authors did not assess explanation methods by their ability to identify true associations because literature verification is ambiguous, but global explanations can generate laboratory-testable scientific hypotheses.This use extends beyond confirming known relationships.Source: §10.3 · p. 53

Unified framework

• The framework systematizes 26 methods through three choices: how features are removed, what model behavior is analyzed, and how feature influence is summarized.Removal uses a subset function, behavior is represented by a set function for local or global explanations, and summaries produce feature attributions or selections.Source: §11 · pp. 53–54
• Many state-of-the-art methods share a foundation and differ in only one or two dimensions, making their conceptual and computational trade-offs easier to analyze.The framework reveals interchangeable choices and provides a systematic characterization of the literature.Source: §11 · p. 54

Theoretical connections

• Removal-based explanations are applications of subtractive counterfactual reasoning, while cooperative game theory and information theory clarify how different methods relate and should be interpreted.The framework also connects feature removal to Mill’s method of difference and draws on overlooked perspectives from cognitive psychology.Source: §11 · p. 54

Cooperative game theory

• Game-theoretic analysis connects probabilistic values, weighted additive models, coalitional excess, and Masking Model optimization, enabling comparison through axioms and interpretability.These connections include Ribeiro et al. (2016) and Dabkowski and Gal (2017).Source: §11 · p. 54

Practical trade-offs

• The Shapley value offers the most complete model summary in the authors’ view, but it is slower than the fastest removal-based explanations and can be difficult to interpret or require causal variants.These limitations are especially relevant in model-agnostic settings; approximation methods include ˇStrumbelj and Kononenko (2010), Lundberg and Lee (2017), Lundberg et al. (2020), and Covert and Lee (2021).Source: §11 · p. 54

Information-theoretic explanations

• Conditional marginalization is the only removal strategy identified as yielding predictions that satisfy standard probability axioms, but it can produce confusing importance assignments and equal credit for perfectly correlated features.The approach requires conditional-distribution approximations, which can be obtained through generative models or models trained with missing features; alternative removal and sparsification strategies can address its drawbacks.Source: §11 · pp. 54–55

Implications

• By disentangling methods into explicit dimensions, the framework helps practitioners reason about strengths, weaknesses, and hybrid methods while giving researchers a foundation for more rigorous future work.Future research can distinguish methods’ dimensions and objectives, resolve shortcomings across dimensions, and justify new approaches through connections to game theory, information theory, and psychology.Source: §11 · p. 55

A.1 Meaningful Perturbations (MP) / A.2 Extremal Perturbations (EP)

• MP blurs, replaces, or noises pixels using a continuous mask, while EP reverses the objective into preserving salient information under a fixed-blur constraint.MP’s recommended operation is Gaussian blurring with pixel-specific standard deviation σ · m_i; EP maximizes classification probability while blurring a fixed portion of the image.Source: §A.1 · p. 56, §A.2 · p. 56

A.3 FIDO-CA / A.4 Minimal Image Representation (MIR)

• FIDO-CA replaces EP’s blurring with generative-model samples, whereas MIR greedily removes image segments until the desired class falls below a minimum classification probability.FIDO-CA uses a conditional generative model, preferably a contextual-attention GAN, and retains EP’s preservation-game objective; MIR selects segments that least reduce classification probability.Source: §A.3 · p. 56, §A.4 · p. 57

A.5 Masking Model (MM)

• Masking Model explanations formalize both smallest destroying and sufficient regions, using smoothness and sparsity penalties while learning a global explainer for one-pass predictions.Unlike MP’s per-input optimization, MM learns an explainer model that approximately solves the masking objective in a single forward pass.Source: §A.5 · p. 57

A.6 Learning to Explain (L2X) / A.7 Instance-wise Variable Selection (INVASE)

• L2X and INVASE learn instance-wise selectors, but L2X trains a surrogate to match model predictions whereas INVASE uses separate baseline, predictor, and selector models with missingness introduced during training.L2X’s information-theoretic interpretation requires Bayes-optimal prediction assumptions; INVASE’s baseline primarily reduces variance, while its predictor is treated as the explained model.Source: §A.6 · pp. 57–58, §A.7 · pp. 58–59

A.8 REAL-X

• REAL-X separates subset-function learning from selector learning to prevent label leakage, enabling subsets sampled independently of inputs to approximate Bayes-optimal conditional predictions.It can sample labels from either the true conditional distribution or the model distribution, paralleling INVASE and L2X respectively.Source: §A.8 · p. 59

A.9 Prediction Difference Analysis (PredDiff) / A.10 Causal Explanations (CXPlain)

• PredDiff imputes removed features from bordering pixels and attributes changes in log-odds, while CXPlain measures loss changes but is not causal in the causal-inference sense.CXPlain normalizes attribution values to support a KL-divergence explainer objective and produces attributions in one forward pass.Source: §A.9 · p. 60, §A.10 · p. 60

A.11 Randomized Input Sampling for Explanation (RISE) / A.12 Interactions Methods for Explanations (IME) / A.13 SHAP / A.14 TreeSHAP

• RISE estimates attributions by Monte Carlo sampling randomly masked inputs, whereas IME, SHAP, and TreeSHAP differ in how they marginalize held-out features.SHAP uses Shapley values and KernelSHAP uses weighted least squares; TreeSHAP averages tree paths using dataset frequencies, which is similar but not identical to conditional marginalization.Source: §A.11 · p. 60, §A.12 · p. 61, §A.13 · p. 61

A.15 LossSHAP / A.16 Shapley Net Effects / A.17 Shapley Effects / A.18 LIME

• LossSHAP decomposes prediction loss, SPVIM scales Shapley estimation to larger and nonlinear settings, Shapley Effects target variance reduction under stated MSE and conditional-expectation assumptions, and LIME is deferred to Appendix B.Shapley Net Effects and SPVIM can also use performance measures such as area under the ROC curve or R2; the appendix provides only a cross-reference for LIME.Source: §A.15 · p. 62, §A.16 · p. 62, §A.17 · p. 62

Appendix B. Additive Model Proofs

• LIME becomes weighted least squares over a set function, and familiar attribution values follow from selecting weighting kernels and omitting regularization.The framework calls this the weighted least squares approach to summarizing set functions.Source: Appendix B · p. 63

B.1 Include individual

• Including only coalitions with at most one feature yields ai = u({i}) − u({}), matching Occlusion, PredDiff, CXPlain, permutation tests, and feature ablation.These correspond to local importance methods and global importance summaries, respectively.Source: §B.1 · p. 63

B.2 Remove individual

• Removing at most one feature yields ai = u(D)−u(D\{i}), matching the univariate predictors approach for global feature importance.Source: §B.2 · p. 64

B.3 Banzhaf value

• With uniform coalition weighting, the unweighted least-squares problem has unique optimal coefficients equal to the Banzhaf values ψi(u).The proof establishes optimality by showing the gradient is zero in a jointly convex problem.Source: §B.3 · pp. 64–65

B.4 Shapley value

• A specific weighting kernel makes the weighted least-squares problem optimize the Shapley value, a connection supported by existing proofs.The cited prior work includes Charnes et al. (1988) and Lundberg and Lee (2017).Source: §B.4 · p. 65

Appendix C. Axioms for Other Approaches

• Appendix C identifies which Shapley or Shapley-like axioms apply to other feature-influence summarization approaches.Source: Appendix C · p. 65

C.1 Additive model axioms

• C.1 Additive model axioms: Under nonnegative finite weighting, strict convexity of the weighted least-squares term, and convex regularization, LIME-like mappings can satisfy subsets of the Shapley value axioms.Strict convexity follows from the subset-enumeration design matrix and aligned diagonal weighting matrix; convex regularizers include Lasso and ridge penalties.Source: §C.1 · pp. 65–66
• C.1 Additive model axioms: Efficiency holds only when π({}) = π(D) = ∞, imposing b_0 = u({}) and Σ_i∈D b_i = u(D) − u({}).These constraints define faithful linear approximations in cooperative game theory.Source: §C.1 · p. 66
• C.1 Additive model axioms: Symmetry holds when the weighting kernel and regularizer are permutation-invariant, because strict convexity makes the swapped-coefficient optimum unique.Here, permutation invariance means π depends on subset size and the regularizer is invariant to parameter ordering.Source: §C.1 · p. 66
• C.1 Additive model axioms: The dummy property holds for Shapley and Banzhaf kernels but not arbitrary weighting kernels and regularizers, while additivity requires Ω = 0.Marginalism also holds: games with identical marginal contributions produce identical coefficients and intercepts related by b*_0(u′) = b*_0(u) + c.Source: §C.1 · p. 66, §C.2 · p. 66

C.2 Feature selection axioms

• C.2 Feature selection axioms: MP, MIR, L2X, INVASE, REAL-X, EP, MM, and FIDO-CA admit symmetry, dummy, and marginalism properties analogous to Shapley axioms for optimal coalitions.These methods output an optimal coalition S* rather than an allocation, so the original Shapley axioms do not apply directly.Source: §C.2 · pp. 66–67
• C.2 Feature selection axioms: Coalition symmetry permits replacing one selected player with an interchangeable unselected player, while dummy requires an optimal coalition excluding zero-contribution players.Marginalism requires the same coalition to be optimal in two games exactly when all players’ marginal contributions are identical.Source: §C.2 · pp. 66–67
• C.2 Feature selection axioms: Feature-selection explanations lack apparent efficiency and additivity analogues, and these properties do not suffice to derive a unique axiomatic approach.Thus, their axiomatic characterization is weaker than the attribution case.Source: §C.2 · p. 67

Appendix D. Consistency Proofs

• Both consistency results follow by applying basic probability laws to derive conditional distributions and expectations from q(X) and the full-feature model.The derivations use countable additivity, Bayes rule, expectation definitions, and consistency across alternative feature-subset derivations.Source: Appendix D · pp. 67, 69, 70
• For classification models estimating conditional probabilities, Proposition 6 establishes a unique subset extension consistent with q(X).The extension marginalizes missing features using the conditional distribution induced by q(X).Source: Appendix D · p. 67
• For regression models estimating conditional expectations, Proposition 7 likewise establishes a unique subset extension consistent with q(X).The proof shows that probability laws uniquely constrain the conditional expectation represented by the subset extension.Source: Appendix D · p. 69

Appendix E. Conditional Distribution Approximations

• Several feature-removal approaches approximate marginalizing missing features with their conditional distribution.Appendix E introduces this perspective before detailing separate-model approximations.Source: Appendix E · p. 71

E.1 Separate models

• Shapley Net Effects, SPVIM, original IME, univariate predictors, and feature ablation use separately trained models to predict with missing features.These models define a subset function through predictions from models trained on feature subsets or held-out-feature configurations.Source: §E.1 · p. 71
• These separately trained models remain valid subset extensions because they are invariant to missing features and agree with the full model when all features are present.Their held-out-feature predictions need not explicitly reference the full-feature model to satisfy these subset-extension properties.Source: §E.1 · p. 71
• If each separate model optimizes population risk, the resulting subset function corresponds to conditional marginalization of the full model.For regression, MSE yields conditional expectations; for classification, cross entropy or another strictly proper scoring rule yields conditional probabilities.Source: §E.1 · p. 71

E.2 Missingness during training

• E.2 Missingness during training: Training with missing features, as in INVASE, lets models predict from available information by replacing removed features with defaults or other values.The model is trained to recognize replaced features as missing.Source: §E.2 · p. 71
• E.2 Missingness during training: Replacing features with observed default values creates ambiguity, which can be resolved using an unseen replacement value or an explicit missingness mask.The mask is a binary vector supplied alongside the masked input.Source: §E.2 · p. 71
• E.2 Missingness during training: With masks sampled independently of inputs, the optimal model equals conditional marginalization of missing features for cross entropy, strictly proper scoring rules, and MSE regression.For regression, conditional probability is replaced by conditional expectation.Source: §E.2 · p. 72
• E.2 Missingness during training: INVASE may fail to approximate p(Y | X_S = x_S) because input-dependent masks can communicate information about held-out features.This dependence can invalidate its information-theoretic KL-divergence interpretation.Source: §E.2 · p. 72

E.3 Surrogate models

• E.3 Surrogate models: Surrogate models trained to match an existing model’s predictions on masked inputs include L2X (Chen et al., 2018a), REAL-X, and Frye et al. (2020).Held-out features can be replaced by unseen values or accompanied by a mask input.Source: §E.3 · pp. 72–73
• Under independent mask sampling, the optimal surrogate matches the target in MSE by marginalizing missing features using their conditional distribution.This condition is satisfied by Frye et al. (2020) and Jethani et al. (2021a) (REAL-X), but not Chen et al. (2018a) (L2X).Source: §E.3 · pp. 73–74
• E.3 Surrogate models: To avoid concentrating masks near half-sized subsets, the method samples subset size uniformly, then samples that many feature indices without replacement.This distribution places mass across all subset sizes rather than favoring approximately half the features.Source: §E.3 · p. 74

Appendix F. Information-Theoretic Connections in Regression

• Under Gaussian conditional distributions, squared prediction-error quantities relate to pointwise mutual information and negative KL divergence between conditional response distributions.The mutual-information result assumes a Gaussian conditional distribution for the response, while the KL-divergence relationship assumes Gaussian conditional distributions given X and a feature subset.Source: Appendix F · pp. 74–75
• Explained variance in the response relates to mutual information through Gaussian entropy maximization, with equality conditions tied to Gaussianity and one bound derived using Jensen’s inequality.The set function quantifies explained variance up to a constant value; the first bound is tight when the relevant conditional response distribution is Gaussian.Source: Appendix F · p. 75
• For model outputs, squared deviations from conditional expectations relate to a KL-divergence component, while explained output variance corresponds to the Shapley Effects variance decomposition and mutual information.The KL interpretation is contrived because the divergence is ultimately infinite; the mutual-information bound uses Gaussian entropy maximization and Jensen’s inequality.Source: Appendix F · p. 76
• Regression set functions have information-theoretic interpretations analogous to classification, but conservative interpretation uses Euclidean distances and conditional variances when Gaussian assumptions are avoided.The analysis assumes MSE loss and model optimality; Gaussian distributions are required for the information-theoretic links.Source: Appendix F · pp. 74, 77

Appendix G. Experiment Details

• Appendix G supplies additional details about the models, datasets, hyperparameters, and experimental results used to evaluate the explanation methods.The appendix functions as an experimental specification accompanying the paper’s main analyses.Source: Appendix G · p. 77

G.1 Hyperparameters

• The experiments used LightGBM for census income, a 14-layer CNN for MNIST, and ℓ1-regularized logistic regression for BRCA, with dataset-specific training settings.Census used at most 10 leaves per tree and learning rate 0.05; MNIST used Adam with learning rate 10^-3; BRCA regularization was selected on a validation set.Source: §G.1 · p. 77
• BRCA analysis selected 100 genes from 17,814 using 10 random seeds, retained the best-performing seed, imputed missing values by the mean, and standardized the data.The selected genes and their performance are displayed in Table 9.Source: §G.1 · p. 77, Table 9 · p. 78
• Feature-removal methods used 512 samples for census income and MNIST and 372 for BRCA, with masking-based surrogates tailored to each dataset’s original model.The census surrogate was an MLP, the MNIST surrogate matched the CNN architecture, and the BRCA surrogate was an MLP; masks were appended or represented through the masking layer.Source: §G.1 · pp. 77–78

MNIST explanations

• On MNIST, LossSHAP consistently highlights important pixels within digits and empty regions that distinguish each digit from other possible classes.These examples use the conditional distribution combined with Shapley values.Source: §G.2 · p. 79

BRCA explanation similarity

• On BRCA, Shapley-based explanations are relatively similar to individual-feature removal and Banzhaf explanations, whereas the include-individual technique differs most.Banzhaf explanations also correlate with mean-when-included explanations, as predicted by the theory; Shapley explanations are more similar across removal strategies than across summarization strategies.Source: §G.2 · p. 79, Figure 14 · p. 80