PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluations and Methods for Explanation through Robustness Analysis

Cheng-Yu Hsieh, Chih-Kuan Yeh, Xuanqing Liu, Pradeep Ravikumar, Seungyeon Kim, Sanjiv Kumar, Cho-Jui Hsieh

arXiv:2006.00442v2cs.LGstat.ML

TL;DR

Feature-based explanation evaluations often operationalize feature removal, which can introduce reference-value or generative-model bias. This paper instead evaluates explanations with small adversarial perturbations and derives explanations that optimize those criteria. Experiments report that the explanations capture significant feature sets across multiple domains.

  • Problem

    Feature-removal evaluations can be biased by the chosen reference value, and generative replacements inherit model bias and may be unavailable.

    Source: §1 · pp. 1–2

  • Method

    The paper evaluates relevant and irrelevant feature sets through adversarial robustness and designs explanations to optimize these criteria.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    Experiments demonstrate that the proposed explanations capture significant feature sets across multiple domains.

    Source: §6 · p. 9

  • Takeaways & Limitations

    In a text-classification user study, Greedy-AS explanations matched user intuition the most.

    Source: §5.3 · p. 9

    Greedy-AS may exploit adversarially vulnerable regions that are not semantically meaningful, though the authors report meaningful regions in visualizations.

    Source: §5.2 · p. 8

Abstract

from arXiv · show

Feature based explanations, that provide importance of each feature towards the model prediction, is arguably one of the most intuitive ways to explain a model. In this paper, we establish a novel set of evaluation criteria for such feature based explanations by robustness analysis. In contrast to existing evaluations which require us to specify some way to "remove" features that could inevitably introduces biases and artifacts, we make use of the subtler notion of smaller adversarial perturbations. By optimizing towards our proposed evaluation criteria, we obtain new explanations that are loosely necessary and sufficient for a prediction. We further extend the explanation to extract the set of features that would move the current prediction to a target class by adopting targeted adversarial attack for the robustness analysis. Through experiments across multiple domains and a user study, we validate the usefulness of our evaluation criteria and our derived explanations.

1 INTRODUCTION

Feature-based explanations are widely used, but evaluating them through feature removal can introduce bias. The paper proposes robustness-based criteria using small adversarial perturbations and methods that optimize those criteria.

1 INTRODUCTION

  • Reference-value removal can favor features far from the baseline, while generative replacements inherit the generative model’s bias and may be unavailable.In RGB images, setting features to black can favor bright pixels and omit important dark objects.Source: §1 · pp. 1–2
  • The paper evaluates feature relevance through small adversarial perturbations and uses the criteria to design explanations.Perturbing relevant features tests necessity; perturbing their complement tests sufficiency.Source: §1 · p. 2
  • The proposed explanations can identify pertinent positive and negative features, and the authors report evaluations across image and language datasets, including a user study.Figure 1 illustrates positive and negative features for a digit predicted as “2”.Source: §1 · p. 2, Figure 1 · p. 2

2 RELATED WORK

Prior explanation evaluations include fidelity-based and removal- or preservation-based measures, while counterfactual methods provide a contrastive perspective. This work instead evaluates feature sets through adversarial perturbations.

2 RELATED WORK

  • Fidelity measures compare an explanation’s simplified model with the original, while removal- and preservation-based measures assess prediction changes as salient features are removed.The first family includes completeness, sensitivity-n, local accuracy, and infidelity; the second includes ranking-based criteria.Source: §2 · pp. 2–3
  • Unlike removal-based evaluations, the proposed criteria use smaller adversarial perturbations without operationalizing feature removal.The paper also distinguishes its focus on identifying necessary and sufficient relevant features from general adversarial perturbations.Source: §2 · p. 3
  • Counterfactual explanations ask what to alter to change a prediction, but may identify necessary features without identifying features sufficient for the current prediction.Sampling approaches to identify stable feature sets may be infeasible for high-dimensional images without super-pixels; generative-model approaches depend on an effective model being available.Source: §2 · p. 3

3 ROBUSTNESS ANALYSIS FOR EVALUATING EXPLANATIONS

The paper evaluates feature explanations by measuring adversarial robustness on feature subsets. Paired criteria assess the relevant set and its complement, with targeted robustness extending the analysis to contrasts between classes.

3.1 PROBLEM NOTATION

  • For a K-way classifier, an explanation selects a compact relevant feature set Sr from the input’s features, with the remaining features treated as irrelevant.The notation also denotes the input restricted to a feature set.Source: §3.1 · p. 3

3.2 EVALUATION THROUGH ROBUSTNESS ANALYSIS

  • The framework uses small adversarial perturbations rather than feature removal, motivated by assumptions that relevant features resist perturbations on their complement but are vulnerable when perturbed themselves.These assumptions motivate assessing the explanation through robustness on feature subsets.Source: §3.2 · p. 3
  • Robustness on the irrelevant complement should be higher, while robustness when perturbing the relevant set should be lower, when the explanation covers more pertinent features.The criteria use minimum adversarial perturbation norms with perturbations restricted to feature subsets.Source: §3.2 · p. 4
  • Attribution rankings can be evaluated by plotting robustness against the number of top-ranked features and comparing the resulting AUCs.A larger AUC is better for robustness on the relevant set; a smaller AUC is better for robustness on its complement.Source: §3.2 · p. 4
  • Targeted robustness asks why an input is classified as y rather than a specified class t, highlighting features that contrast the two classes.For estimation, the framework permits attack or verification methods; the paper uses PGD, noting verification methods are too slow and can yield looser estimates.Source: §3.2 · p. 5

4 EXTRACTING RELEVANT FEATURES THROUGH ROBUSTNESS ANALYSIS

The paper formulates feature selection as optimizing robustness criteria under a set-size constraint, then proposes greedy algorithms to approximate the selected explanations. Greedy-AS incorporates interactions among unchosen features through aggregated contribution scores.

4 EXTRACTING RELEVANT FEATURES THROUGH ROBUSTNESS ANALYSIS

  • The feature explainer seeks a size-constrained set that maximizes or minimizes the corresponding robustness objective, forming adversarial or cooperative optimization problems.The discrete optimization is challenging, motivating a greedy estimate.Source: §4 · p. 5

4.1 GREEDY ALGORITHM TO COMPUTE OPTIMAL EXPLANATIONS

  • Vanilla Greedy iteratively adds the feature that best improves the objective until the set reaches K, and can encode selection order with feature weights.The method evaluates individual features at each step.Source: §4.1 · pp. 5–6

4.2 GREEDY BY SET AGGREGATION SCORE

  • Greedy-AS scores unchosen features by expected contribution across subsets, capturing interactions that vanilla Greedy may miss.It uses a regression-based aggregation score related to the Banzhaf value and subsampling to reduce computation.Source: §4.2 · p. 6

5 EXPERIMENTS

Experiments span MNIST, ImageNet, and Yahoo! Answers, using image and text classifiers. They compare Greedy-AS with existing explanations on robustness and other evaluation criteria, then examine qualitative and targeted explanations.

5.1 ROBUSTNESS ANALYSIS ON MODEL INTERPRETABILITY METHODS

  • Greedy-AS consistently outperforms other explanation methods on both proposed robustness criteria.The authors suggest this supports the criteria’s ability to capture characteristics missing from many current explanations.Source: Table 1 · p. 7, §5.1 · p. 7

5.2 EVALUATING GREEDY-AS

  • Greedy-AS performs favorably against other explanations on Insertion and Deletion, using randomly sampled reference values for this comparison.The authors say this further validates the benefits of the proposed criteria, which have a tight connection to optimizing Insertion and Deletion.Source: §5.2 · p. 7, Table 2 · p. 8
  • Greedy-AS stays among the top performers across reference values, while zero-baseline SHAP and IG weaken when evaluation references differ from zero.EG, which averages over multiple baselines, is more stable across reference values; the authors identify baseline changes as a potential risk for evaluations and explanations.Source: §5.2 · p. 8
  • Greedy-AS passes the reported sanity check, and the authors also conduct sensitivity analysis.The checks are reported in Appendices I and J, respectively.Source: §5.2 · p. 8

5.3 QUALITATIVE RESULTS

  • On MNIST, Greedy-AS highlights both crucial positive and pertinent negative pixels; on ImageNet, its explanations focus more compactly on classified objects.Other MNIST explanations tend to emphasize white digit pixels, while Greedy-AS also identifies black regions whose perturbation could change predictions.Source: §5.3 · p. 8, Figure 2 · p. 9, Figure 3 · p. 9
  • For a correctly predicted “sport” example, Greedy-AS highlights five keywords related to that label and is less likely than other methods to select stop words.A user study found that its explanations matched user intuition the most.Source: §5.3 · p. 9, Figure 4 · p. 9
  • Targeted explanations vary meaningfully with the chosen target class, highlighting different regions that could make an input resemble that target.For a digit 7, the regions highlighted against 9 and 2 correspond to pixels that could make it resemble each target.Source: §5.3 · p. 9, Figure 5 · p. 9

6 CONCLUSION

The paper links feature sets to predictions through robustness analysis and develops explanations that optimize its new evaluation criterion. Experiments indicate that these explanations capture significant feature sets across multiple domains.

  • Robustness analysis links a set of features to a prediction by measuring the minimum tolerance of adversarial perturbation.Source: §6 · p. 9
  • The authors develop an explanation method to find important feature sets that optimize this criterion.Source: §6 · p. 9
  • Experiments demonstrate that the proposed explanations capture significant feature sets across multiple domains.Source: §6 · p. 9

A BIAS FOR REFERENCE-VALUE METHODS

The appendix analyzes how reference values can bias both feature explanations and their evaluations. It also describes the robustness-estimation context and experimental implementation.

A.1 BIAS FOR REFERENCE-VALUE BASED EXPLANATIONS

  • When a feature equals the reference value, reference-value-based explanations of the stated form assign it zero attribution, even if it drives the prediction.The analysis identifies IG, LRP, DeepLift, baseline Shapley values, and averaged LRP as biased to reference values; Expected Gradients avoids the stated exact-zero result but may retain a lesser distribution-related bias.Source: §A.1 · pp. 14–15

A.2 BIAS FOR REFERENCE-VALUE BASED EVALUATIONS

  • Deletion and Insertion scores do not change when a feature already equals the replacement reference value, so they can treat crucial features as non-important.The appendix notes that this can occur with references such as blurred images or added noise.Source: A BIAS FOR REFERENCE-VALUE METHODS · p. 14, §A.2 · p. 15

B MORE ON ESTIMATING ADVERSARIAL ROBUSTNESS

  • The framework treats feature-subset perturbation as constrained adversarial robustness and can extend to decision-based classifiers beyond neural networks.Verification methods could approximate robustness, but the authors report they are very slow in practice and often give loose lower bounds on regularly trained networks.Source: B MORE ON ESTIMATING ADVERSARIAL… · p. 15

C IMPLEMENTATION DETAIL

  • Experiments use CNN, pretrained ResNet, and BiLSTM classifiers for MNIST, ImageNet, and Yahoo! Answers, respectively.The reported test accuracies are 99% for MNIST and 71% for Yahoo! Answers; ImageNet uses a pretrained ResNet from PyTorch.Source: C IMPLEMENTATION DETAIL · p. 16
  • Greedy-AS adds the top 5% of features per greedy iteration and estimates robustness with PGD using binary search, a 1.0 step size, and 100 steps.The regression in Eqn. 6 uses coefficients learned from 5000 sampled subsets.Source: C IMPLEMENTATION DETAIL · p. 16
  • Text experiments represent words as embedding vectors, restrict perturbations to selected or unselected word dimensions, and summarize evaluation curves by area under the curves.Curves evaluate relevant-set sizes from 5% through 45% of features; insertion and deletion use the same increments.Source: C IMPLEMENTATION DETAIL · p. 16

D EVALUATION CURVES UNDER ROBUSTNESS-Sr AND ROBUSTNESS-Sr

This section presents robustness-criterion evaluation curves for explanation methods and an AUC table comparing Greedy-AS with its variants. The supplied passages identify the plotted datasets and the preferred direction for each criterion, but do not report comparative outcomes.

E ABLATION STUDY ON GREEDY-AS

Greedy-AS combines greedy feature selection with regression-based approximations of feature contributions, outperforming either component alone in the ablation. The section also presents Insertion and Deletion evaluation curves for MNIST, ImageNet, and Yahoo!Answers.

E ABLATION STUDY ON GREEDY-AS

  • Greedy-AS outperforms both pure Greedy and One-Step Banzhaf under both criteria, with feature-correlation handling and iterative emphasis on important-feature interactions offered as explanations.Pure Greedy ignores feature correlations, introducing noise; One-Step Banzhaf weights interactions among all features equally, whereas Greedy-AS iteratively emphasizes interactions among the most important features.Source: E ABLATION STUDY ON GREEDY-AS · p. 18
  • Greedy-AS combines the original greedy procedure with regression-based approximations of each feature’s contribution.The ablation compares it with pure Greedy selection and a single regression step, termed One-Step Banzhaf.Source: E ABLATION STUDY ON GREEDY-AS · p. 18
  • Figure 9 depicts Greedy-AS selecting less noisy top features.The text links the noise in pure Greedy to its neglect of feature correlations.Source: Figure 9 · p. 18, E ABLATION STUDY ON GREEDY-AS · p. 18

F EVALUATION CURVES UNDER INSERTION AND DELETION

G STUDENT’S t-TEST ON THE PERFORMANCE IMPROVEMENT OF GREEDY-AS OVER EXISTING METHODS

The authors use Student’s t-tests on AUCs to assess whether Greedy-AS improves over existing explanations across evaluation criteria. They report statistically significant performance improvements across the criteria tested.

G STUDENT’S t-TEST ON THE PERFORMANCE IMPROVEMENT OF GREEDY-AS OVER EXISTING METHODS

H ANALYSIS ON THE EFFECT OF REFERENCE VALUE

Greedy-AS remains among the top-performing explanations across reference values, while IG, SHAP, and LOO degrade as the value moves away from zero. A separate sanity check finds Greedy-AS sensitive to model-parameter changes.

H ANALYSIS ON THE EFFECT OF REFERENCE VALUE

  • Greedy-AS performs most stably among the top across reference values, while IG, SHAP, and LOO degrade as the value moves away from zero.Zero is the intrinsic reference value on which these methods rely; Figures 14–17 show evaluation curves and Figure 13 summarizes rankings.Source: H ANALYSIS ON THE EFFECT OF REFERENCE… · p. 21, Figure 13 · p. 21
  • Tables 5 and 6 report Insertion and Deletion AUCs across reference values for MNIST and ImageNet, with higher Insertion and lower Deletion AUCs preferred.Figures 14–17 provide the corresponding evaluation curves.Source: Table 5 · p. 21, Table 6 · p. 21, Figure 14 · p. 22

I SANITY CHECK

  • After randomizing the final fully connected layer, Greedy-AS has lower rank correlation than Grad, IG, and LOO, indicating sensitivity to model-parameter changes and passing the sanity check.The check compares explanations from the original and randomized models.Source: I SANITY CHECK · p. 24, Table 7 · p. 24

J EXPLANATION SENSITIVITY ANALYSIS

The section defines and evaluates explanation sensitivity, reports that Greedy-AS is relatively sensitive, and presents visual and quantitative comparisons across image and text tasks. Its targeted explanations also vary with the chosen target class.

J EXPLANATION SENSITIVITY ANALYSIS

  • The sensitivity measure compares relevant-feature sets within radius r; it is 0 when the set remains unchanged throughout the neighborhood and 1 when some set is disjoint.The MNIST analysis uses the top 20% of features and sets r = 0.1; the authors caution that a constant, meaningless explanation could also have zero sensitivity.Source: J EXPLANATION SENSITIVITY ANALYSIS · p. 24
  • Greedy-AS has relatively large sensitivity; the authors suggest a more robust underlying model or smoothing the explanation as possible ways to reduce it.They relate the result tentatively to the underlying model’s sensitivity and leave further improvement for future work.Source: J EXPLANATION SENSITIVITY ANALYSIS · p. 24, Table 8 · p. 24

K VISUALIZATION ON MNIST

L VISUALIZATION ON IMAGENET

M COMPARISONS WITH OTHER EXPLANATIONS CAPTURING PERTINENT NEGATIVE FEATURES

N QUANTITATIVE RESULTS ON TEXT DATASET

  • The paper conducts quantitative explanation analyses on Yahoo!Answers and presents AUCs for Robustness-Sr and Insertion and Deletion criteria in Table 9.The table caption says higher is better for Robustness-Sr and Insertion and lower is better for Robustness-Sr and Deletion.Source: N QUANTITATIVE RESULTS ON TEXT DATASET · p. 28, Table 9 · p. 28

O QUALITATIVE RESULTS ON TEXT DATASET

  • Figure 22 provides additional examples of explanations for a text-classification model.Source: Figure 22 · p. 29

P USER STUDY ON TEXT DATASET

The user study compares explanation keywords with crowd-sourced labels, finding Greedy-AS highest in overall precision and Anchor and IG strongest on top-1 alignment with user intuition. The section also describes runtime measurements for image explanations on MNIST and ImageNet.

P USER STUDY ON TEXT DATASET

  • Thirty users labeled 30 text examples, and the most frequently selected keywords for each sentence served as its ground-truth explanation.The study used convenience sampling of users with Computer Science backgrounds and collected 10 sets of labels per sentence.Source: P USER STUDY ON TEXT DATASET · p. 30
  • Greedy-AS has the highest overall precision score, while Anchor and IG’s top-1 keywords match user intuition the most.These comparisons use user-labeled keywords as ground truth; Anchor is reported only at precision@1 because it tends to select one keyword per sentence.Source: P USER STUDY ON TEXT DATASET · p. 30, Table 10 · p. 30, Q RUNTIME ANALYSIS · p. 30

Q RUNTIME ANALYSIS

  • Runtime analysis reports average wall-clock time for single-image explanations on MNIST and ImageNet, with Greedy and Greedy-AS timed for the top 20% of relevant features.The supplied passage identifies a runtime table but provides no runtime values.Source: Q RUNTIME ANALYSIS · p. 30, Table 11 · p. 30

Related papers

Loading 2006.00442v2…