PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Unifying Feature Attribution and Counterfactual Explanations: Different Means to the Same End

Ramaravind Kommiya Mothilal, Divyat Mahajan, Chenhao Tan, Amit Sharma

arXiv:2011.04917v3cs.LGcs.CY

TL;DR

Attributions and counterfactuals offer different forms of model explanation, and their relationship and implications for one another are unclear. The paper unifies them through actual causality and finds that attribution rankings can disagree and top-ranked features are often neither necessary nor sufficient; the results support considering multiple explanation methods.

  • Problem

    Feature attributions and counterfactuals provide different kinds of explanations, and it is unclear whether important attributed features are necessary or sufficient to change a model’s prediction.

    Source: §3 · p. 3

  • Method

    The paper uses actual causality to connect attribution and counterfactual explanations, and uses counterfactual examples to evaluate necessity and sufficiency and generate complementary feature scores.

    Source: §1 · pp. 1–2

  • Results

    Across the examined datasets, top-ranked LIME and SHAP features are often neither necessary nor sufficient, and importance rankings can differ across explanation methods.

    Source: §1 · p. 2, §8 · p. 10

  • Takeaways & Limitations

    Different explanation methods have different objectives, so a single method may not convey the full picture of a model’s prediction.

    Source: §8 · p. 10

    The definitions assume individual features are independent, leaving causal dependencies between features for future work.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

Feature attributions and counterfactual explanations are popular approaches to explain a ML model. The former assigns an importance score to each input feature, while the latter provides input examples with minimal changes to alter the model's predictions. To unify these approaches, we provide an interpretation based on the actual causality framework and present two key results in terms of their use. First, we present a method to generate feature attribution explanations from a set of counterfactual examples. These feature attributions convey how important a feature is to changing the classification outcome of a model, especially on whether a subset of features is necessary and/or sufficient for that change, which attribution-based methods are unable to provide. Second, we show how counterfactual examples can be used to evaluate the goodness of an attribution-based explanation in terms of its necessity and sufficiency. As a result, we highlight the complementarity of these two approaches. Our evaluation on three benchmark datasets - Adult-Income, LendingClub, and German-Credit - confirms the complementarity. Feature attribution methods like LIME and SHAP and counterfactual explanation methods like Wachter et al. and DiCE often do not agree on feature importance rankings. In addition, by restricting the features that can be modified for generating counterfactual examples, we find that the top-k features from LIME or SHAP are often neither necessary nor sufficient explanations of a model's prediction. Finally, we present a case study of different explanation methods on a real-world hospital triage problem

1 INTRODUCTION

Explaining predictions from high-stakes ML models can serve purposes such as identifying important features, debugging, and providing recourse. Attribution methods rank features, while counterfactual methods show minimal input changes; this paper uses actual causality to connect them through necessity and sufficiency.

1 INTRODUCTION

  • The actual-causality framework distinguishes necessity from sufficiency and explains why counterfactual and attribution methods provide complementary views.The paper characterizes counterfactual methods as highlighting necessary feature values and LIME and SHAP as focusing on sufficiency.Source: §1 · pp. 1–2
  • On Adult-Income, German-Credit, and LendingClub, top-ranked LIME and SHAP features are often neither necessary nor sufficient, while attribution rankings also differ across explanation methods.For Adult-Income and German-Credit, more counterfactuals can be generated using features outside the top three than by changing any one of them.Source: §1 · p. 2
  • Counterfactual examples can evaluate the necessity and sufficiency of attribution explanations, and can also generate feature importance scores that complement LIME and SHAP.WachterCF and DiCE importance scores and rankings do not always agree with attribution-based methods; LIME and SHAP also disagree with each other.Source: §1 · p. 2
  • In a hospital triage case study, DiCE, LIME, and SHAP identify triage, chief-complaint, and demographic features, respectively, as most important.The paper also reports that changing LIME’s third-ranked feature yields more valid counterfactuals than changing its top-ranked feature.Source: §1 · p. 2

2 RELATED WORK

Prior work studies multiple desirable properties of explanations and distinguishes feature attribution from counterfactual explanations. This paper connects the two through actual causality and examines their different emphases on necessity and sufficiency.

2 RELATED WORK

  • The related-work discussion covers explanation properties and attribution and counterfactual methods, but excludes work on building intelligible models by design.It frames the paper’s contribution as unifying approaches commonly studied separately.Source: §2 · p. 2, §2.2 · pp. 2–3

2.1 Desirable Properties of an Explanation

  • Explanation properties include soundness, completeness, parsimony, and actionability; counterfactual methods generally prioritize soundness, while data-summary attribution methods prioritize completeness.The related-work discussion also notes that explanations serve different purposes for developers, auditors, and end users.Source: §2.1 · p. 2
  • Necessity and sufficiency are less studied; this paper defines metrics for them using actual causality and computes them with counterfactual examples.The discussion distinguishes this framework from concurrent work proposing an explanation method based on necessity and sufficiency metrics [16].Source: §2.1 · p. 2

2.2 Attribution-based and Counterfactuals

  • Attributions assign feature importance scores, whereas counterfactuals minimally alter inputs to change predictions; the paper treats counterfactuals as another source of attributions and maps the methods to different causal notions.Counterfactuals focus on necessity, while attribution methods tend to focus on sufficiency.Source: §2.2 · pp. 2–3

3 ACTUAL CAUSALITY: UNIFYING EXPLANATIONS

The paper interprets explanations through actual causality: input features are potential causes of a model’s prediction, and explanations can be assessed by necessity and sufficiency. It then relates counterfactual and attribution methods to these properties.

3 ACTUAL CAUSALITY: UNIFYING EXPLANATIONS

  • The framework treats the learned model as a structural causal model in which input features lead to the prediction, and relates explanation methods to whether features cause that output.The definitions assume features can be changed independently; modeling causal dependencies between features is left for future work.Source: §3.1 · p. 3, §3 · p. 3

3.2 Partial Explanation for Model Output

  • An ideal explanation combines actual cause, necessity, sufficiency, and minimality, but partial explanations relax necessity and sufficiency by measuring the fractions of contexts in which they hold.The paper summarizes these properties as (α, β) goodness; high α measures necessity and high β measures sufficiency.Source: §3.1 · p. 3, §3.2 · p. 4

3.3 Unifying Different Local Explanations

  • Counterfactuals correspond to necessity: α measures how often changing a candidate feature subset changes the output, while counterfactual methods search for small such changes.Computing α over contexts is computationally prohibitive, so the paper describes using generated counterfactual examples to support practical analysis.Source: §3.3 · p. 4
  • Attribution methods relate to sufficiency: LIME estimates β over sampled contexts near the input, while Shapley methods average over configurations of other features.The choice of contexts defines the type of attribution explanation.Source: §3.3 · p. 4
  • The framework recommends evaluating both α and β and considering both explanation types to capture complementary aspects of a model’s output.β tests whether holding a feature subset fixed preserves the output as other features change; counterfactual methods can be adapted to test this sufficiency condition.Source: §3.3 · pp. 4–5

4 PROPOSED METHODS

The section links attribution rankings and counterfactuals through tests of necessity and sufficiency and counterfactual-derived feature scores.

4 PROPOSED METHODS

  • The proposed methods use counterfactuals both to assess attribution explanations and to generate complementary feature-importance scores.The assessment measures necessity and sufficiency; the scores reflect how often features change across counterfactuals.Source: §4 · p. 5

4.1 Background: Explanation methods

  • LIME fits a local linear model to perturbed samples, SHAP uses average marginal contributions, WachterCF optimizes for counterfactuals, and DiCE adds diversity for multiple examples.WachterCF's random reinitializations may fail to find unique counterfactuals.Source: §4.1 · p. 5

4.2 Measuring Necessity and Sufficiency

  • Necessity is estimated by restricting counterfactual generation to changing one feature; sufficiency fixes that feature and compares unique-counterfactual generation against the all-feature benchmark.The necessity measure is based on the fraction of test instances for which the requested counterfactuals are generated.Source: §4.2 · p. 5

4.3 Feature Importance using Counterfactuals

  • WachterCFFA and DiCEFA score features by the fraction of counterfactuals that modify them, averaging across counterfactual counts for local scores and across test inputs for global scores.WachterCFFA repeats randomized optimization; DiCEFA generates a diverse counterfactual set.Source: §4.3 · pp. 5–6

4.4 Datasets and Implementation Details

  • Evaluation uses Adult-Income, LendingClub, and German-Credit, with default LIME, SHAP, and DiCE settings and counterfactual loss weights λ1 (0.5) and λ2 (1.0).The reported results are robust to different proximity and diversity hyperparameters.Source: §4.4 · p. 6

5 EVALUATING NECESSITY & SUFFICIENCY

Counterfactual tests show that highly ranked attribution features often fail to be necessary or sufficient, with results varying by dataset and counterfactual method.

Necessity of top features

  • In German-Credit, none of the top features exceed 50% necessity and they are often below 30%, while features outside the top three can achieve almost 100%.Similar results occur in Adult-Income, but are less pronounced.Source: §5 · p. 6
  • In LendingClub, the loan-grade feature is dominant initially, but for nCF > 4 with DiCE, features outside the top three become more necessary.The grade feature is highly correlated with loan repayment status, and its necessity decreases as more counterfactuals are generated.Source: §5 · p. 6
  • Necessity generally tracks LIME/SHAP rankings, yet top-k necessity remains below 100% for nCF > 1, especially in higher-dimensional German-Credit.The Adult-Income exception is LIME's second-ranked feature, often a person's education level.Source: §5 · p. 6

Sufficiency and implications

  • The counterfactual method matters: WachterCF's difficulty finding unique examples can inflate measured sufficiency in LendingClub when nCF > 1.The authors advise choosing the desired necessity or sufficiency property and counterfactual-generation context for the application.Source: §5 · pp. 6–7

Evaluating necessity and sufficiency

  • Top-ranked LIME/SHAP features are often neither necessary nor sufficient; Figure 2 contrasts changing ranked features with changing the remaining features.Necessity generally follows attribution rankings, but features outside the top three often generate counterfactuals more successfully.Source: §5 · pp. 6–7, Figure 2 · p. 7

6 FEATURE IMPORTANCE BY CFS

Feature rankings differ across attribution-based and counterfactual-based methods, including on individual inputs where LIME and SHAP can diverge despite high average-score correlation. The differences become more pronounced in higher-dimensional settings, consistent with the methods capturing complementary notions of importance.

Average feature importance

  • Across three datasets, LIME and SHAP agree on average feature importance, while DiCEFA and WachterCFFA are less similar to LIME, especially in German-Credit.LIME-ranked German-Credit features ranked 13th and 18th become DiCEFA’s top two in 98% of test instances; WachterCFFA makes LIME’s 16th-ranked feature most important.Source: §6 · p. 8

Complementary notions of importance

  • As nCF increases in Adult-Income, counterfactual methods vary more features of differing attribution ranks, highlighting less-important features that can form necessary subsets.LIME and SHAP tend to prioritize sufficiency of features for the original output, whereas DiCEFA and WachterCFFA highlight necessity-related alternatives.Source: §6 · p. 8

Correlation across methods

  • In German-Credit, DiCEFA–LIME/SHAP correlations stay below 0.25 and can become negative as nCF increases.The authors hypothesize that the 20-feature dataset lets DiCE generate counterfactuals using less-important LIME/SHAP features; WachterCFFA has moderate correlation with attribution methods.Source: §6 · p. 8

Differences in feature rankings

  • Paired t-tests compare methods’ rankings for the same feature on individual inputs, extending the analysis beyond average feature-importance scores.The ranking figures are provided in the supplementary materials.Source: §6 · p. 8
  • Individual-input feature rankings often differ significantly across methods, even when LIME and SHAP have high positive correlation in average global importance scores.The pattern holds across datasets and is especially clear for Adult-Income, where LIME and SHAP rank several categorical features differently but agree on hours per week and age.Source: §6 · p. 8

Implications

  • Attributions are sensitive to implementation details, with significant differences both within and across attribution and counterfactual method types.The authors present this as evidence of the difficulty of building a single ideal explanation method.Source: §6 · p. 8

7 CASE STUDY: HOSPITAL ADMISSION

The HospitalTriage case study finds substantial differences among explanation methods on a 222-feature hospital-admission model. The findings include limits to top-ranked features’ necessity and sufficiency, and divergent interpretations of feature importance.

Feature rankings

  • SHAP ranks binary chief-complaint features higher on average, while DiCEFA and LIME have similar mean rankings on chief complaints, which make up 90% of features.DiCEFA and LIME differ on demographics and triage rankings; DiCEFA treats those groups as more important than chief complaints, whereas LIME gives them larger average ranks.Source: Figure 4 · p. 9, §7 · p. 9

Data and model training

  • The case study uses a reproduced deep neural network trained on 252K records and evaluates explanations over 200 test instances.The model has 222 features and test-set precision and recall of 0.81 each, with AUC 0.87; the comparison focuses on DiCE because WachterCF can struggle to generate multiple unique valid counterfactuals when nCF > 1.Source: §7 · p. 9

Similarity between methods

  • LIME and SHAP have almost zero correlation in HospitalTriage rankings; LIME’s average importance-score entropy is 3.2 points higher than SHAP’s.Their correlation is weaker than in Fig. 3b, and SHAP assigns close-to-zero weights to many binary chief-complaint features while LIME assigns them diverse scores.Source: Figure 5 · p. 9, §7 · p. 10

Necessity and sufficiency

  • For LIME, the third-ranked feature yields almost twice as many valid counterfactuals as either of the first two alone because it is often Emergency Severity Index (ESI).ESI is among DiCEFA’s top 10 features in more than 60% of test instances and is a top-three information-gain feature in Hong et al.’s refined XGBoost model.Source: §7 · p. 9, Figure 6 · p. 10

Implications

  • The differing demographic-feature rankings raise fairness questions: DiCEFA and SHAP show demographics can be changed to alter predictions, while the authors caution these may be valid healthcare signals.Hong et al.’s refined model identifies 5 of 9 demographic details and 6 of 13 triage features as important, but only 8 of 200 chief-complaint features.Source: §7 · p. 9

Necessity and sufficiency

  • None of the top three features is sufficient to generate counterfactuals; at nCF = 1 and nCF = 8, keeping each top feature fixed still permits 100% and 68% valid counterfactuals, respectively.The overall fraction of valid counterfactuals decreases as nCF increases, making diverse counterfactuals harder to generate for this dataset.Source: §7 · p. 9, Figure 6 · p. 10

Similarity between methods

  • For HospitalTriage, DiCEFA correlates less with SHAP than with LIME; DiCEFA and SHAP have no average-ranking correlation at nCF = 6 or nCF = 8.The authors attribute the disagreement at higher nCF to DiCE varying more binary features that SHAP gives very low weights.Source: Figure 5 · p. 9, §7 · p. 10

Implications

  • HospitalTriage results show that top-ranked attribution features are not sufficient and are not always the most necessary, while DiCEFA distributes importance more evenly across features.The methods’ differences can substantially change model interpretations, including with respect to fairness.Source: §7 · p. 10

8 CONCLUDING DISCUSSION

The paper’s conclusion is that feature attribution and counterfactual explanations offer complementary views, and that their differences increase with feature dimension. It argues against a uniquely best explanation method and calls for matching methods to users’ questions.

Conclusions

  • Top features identified by LIME and SHAP are neither sufficient nor always necessary, while other features can be meaningful and potentially actionable.The paper presents this as evidence that attribution rankings alone do not capture all useful explanations.Source: §8 · p. 10
  • Counterfactual- and attribution-based importance can be highly correlated on low-dimensional datasets but diverge as feature dimension grows, with differences also appearing within each method type.The authors note no or negative correlations can occur in German-Credit when generating multiple counterfactuals.Source: §8 · p. 10

Discussion

  • The authors reject a one-size-fits-all ideal explanation and recommend finding methods appropriate to the user’s question, since importance can mean actual causes or but-for causes.They describe combining explanations as an opportunity, while acknowledging that leveraging their complementarity is challenging.Source: §8 · p. 10

A.1 Explanation Scores: A Simple Example

A simple threshold-model example contrasts the importance scores assigned by attribution-based and counterfactual-based methods. In the example, counterfactual methods give nearly equal importance to a lower-coefficient feature.

Example

  • For the model at x1 = 1 and x2 = 1, DiCEFA and WachterCFFA give x2 almost equal importance to x1 despite its smaller model coefficient.The table compares explanation scores for x1 and x2; LIME and SHAP are the attribution-based methods, and SHAP scores are shown with median-data and full-training-data backgrounds.Source: Table 1 · p. 12
  • The example uses y = I(0.45x1 + 0.1x2 ≥ 0.5), with x1 and x2 sampled from a uniform distribution on [0, 1].The input being explained is x1 = 1, x2 = 1, with y = 1.Source: Table 1 · p. 12
  • SHAP’s scores are reported for two background choices: median data and the entire training dataset.The table’s second and third columns give each method’s scores for x1 and x2, respectively.Source: Table 1 · p. 12

A.2 Choosing Counterfactual Explanation Methods

The experiments use DiCE and WachterCF as counterfactual methods after screening for subset-restricted and multiple-counterfactual generation; compatibility issues shaped the final choice.

A.2 Choosing Counterfactual Explanation Methods

  • The screening required methods to support both feature-subset counterfactual generation and generation of multiple counterfactuals.Methods that could only be altered in theory for subset-based generation, or lacked explicit multiple-counterfactual support, were filtered out [5] [22] [50].Source: §A.2 · p. 12
  • DiCE and WachterCF were selected as the counterfactual methods after compatibility issues affected comparisons involving DiCE and MOC.DiCE and MOC were the only surveyed libraries directly meeting both criteria, but transferring models between their Python and R implementations caused compatibility issues.Source: §A.2 · p. 12
  • MACE [22] required extensive conversion to logic formulae for new models, while adapting GeCo [50] for feature subsets and Python models was unclear.WachterCF was also included because multiple random seeds can produce multiple counterfactuals despite its default single-counterfactual output.Source: §A.2 · p. 12

A.3 Datasets and Implementation Details

Across the benchmark datasets, DiCEFA rankings remain highly correlated across tested diversity settings, while necessity increases when top-ranked features are allowed to change.

A.3 Datasets and Implementation Details

  • Adult-Income and LendingClub use 8 processed features each, while German-Credit uses all available features for credit-risk prediction.The tasks are income-above-$50,000 prediction, loan-payment classification, and good-versus-bad credit-risk prediction, respectively.Source: §A.3 · p. 13
  • SHAP used KernelExplainer with median feature values as background because large backgrounds were slow, a configuration that may limit its strength.The authors leave exploration of other SHAP configurations to future work.Source: §A.3 · p. 13

A.4 Validity and Stability of DiCE

  • DiCE's default proximity_weight is 0.5 and diversity_weight is 1.0; the parameters trade off counterfactual closeness and diversity.The diversity term encourages different features to change across counterfactuals.Source: §A.4 · p. 13, §A.3 · p. 13
  • Table 2 reports mean percentages of unique and valid counterfactuals for nCF values 1, 2, 4, 6, and 8, plus fully valid instance counts.The mean validity uses a random sample of 200 test instances per dataset.Source: Table 2 · p. 13
  • DiCEFA feature-importance scores remain correlated above 0.96 across diversity_weight settings 1.0, 0.5, and 0.25 on all datasets.Figure 7 shows pairwise correlations between the three diversity-weight versions.Source: Figure 7 · p. 13, §A.4 · p. 13

A.5 Necessity and Sufficiency

  • Necessity increases for top-k features when features through rank k may change, while sufficiency remains identical to the main-paper setting.Figure 8 distinguishes allowing the top-k features to change from fixing them while changing other features.Source: §A.5 · p. 13, Figure 8 · p. 14

A.6 Differences in Feature Rankings

  • Figures 9–10 compare per-feature ranking correlations across test instances for LIME, SHAP, DiCE, and WachterCF; stars encode significance levels.The x-axis is the mean difference in feature rankings over test inputs.Source: Figure 9 · p. 14, Figure 10 · p. 15
Loading 2011.04917v3…