PaperlayerView exact revision on arXiv

Source-linked AI summary

Do Feature Attribution Methods Correctly Attribute Features?

Yilun Zhou, Serena Booth, Marco Tulio Ribeiro, Julie Shah

arXiv:2104.14403v2cs.LGcs.CV

TL;DR

Feature attribution methods lack a consensus definition and are difficult to evaluate because natural datasets do not provide ground-truth attributions. The paper creates semi-natural datasets with known attribution targets, evaluates saliency maps, rationales, and attentions, and finds that none performs totally satisfactorily across vision and text. It therefore recommends validating attribution methods against ground truth before deployment.

  • Problem

    Attribution methods lack systematic ground-truth evaluation, while human-alignment and proxy metrics may not establish faithful model reasoning.

  • Method

    The paper modifies natural datasets by weakening original label signals and introducing controlled predictive features, then evaluates attribution desiderata using saliency maps, rationales, and attentions.

  • Results

    Across vision and text, none of the evaluated methods achieves totally satisfactory performance against the induced ground truth.

  • Takeaways & Limitations

    Attribution methods should be dry-run against known ground-truth reasoning before being used to analyze deployed models or detect spurious correlations.

  • Takeaways & Limitations

    Removal-based evaluation can fail with nonlinear interactions, and retraining evaluation can reject valid attributions when alternative features achieve the same accuracy.

Abstract

from arXiv · show

Feature attribution methods are popular in interpretable machine learning. These methods compute the attribution of each input feature to represent its importance, but there is no consensus on the definition of "attribution", leading to many competing methods with little systematic evaluation, complicated in particular by the lack of ground truth attribution. To address this, we propose a dataset modification procedure to induce such ground truth. Using this procedure, we evaluate three common methods: saliency maps, rationales, and attentions. We identify several deficiencies and add new perspectives to the growing body of evidence questioning the correctness and reliability of these methods applied on datasets in the wild. We further discuss possible avenues for remedy and recommend new attribution methods to be tested against ground truth before deployment. The code is available at https://github.com/YilunZhou/feature-attribution-evaluation

1 Introduction

Feature attribution methods are used to identify influential or spurious input features, but their correctness is difficult to evaluate without known ground truth. The paper proposes modified semi-natural datasets that induce ground-truth attributions and uses them to test common methods.

  • Motivation: Attribution methods help inspect whether models rely on genuine signals or artifacts such as cancer-center watermarks in X-ray images.The train-and-interpret pipeline is widely used in medical diagnosis and other applications.
  • Evaluation gap: Direct evaluation on natural datasets is impossible when the spurious correlations of interest are unknown.Existing proxy metrics are limited, notably because they lack ground-truth attributions.
  • Proposed approach: The paper constructs semi-natural datasets by systematically modifying natural data to introduce ground-truth information for feature attributions.The modifications ensure that sufficiently accurate classifiers must rely, sometimes solely, on the introduced features.
  • Scope: The evaluation covers saliency maps, rationale models, and attention mechanisms on image and text data.The authors identify failure modes and recommend testing attribution methods against ground truth before deployment.

2 Related Work

Feature attribution methods assign importance-related scores or selections to input features, but the interpretation of these mathematical quantities requires justification. Existing evaluations rely on human alignment or removal and retraining metrics that can diverge from faithful model reasoning.

  • Attribution methods: Feature attribution methods assign scores whose absolute values informally represent feature importance for prediction or performance.Different methods formalize contribution through sensitivity, relevance, local influence, Shapley values, or filter activations.
  • Attribution methods: Attention mechanisms and rationale models provide alternative attribution interfaces, but attention validity remains debated and rationales select text through a two-stage selector-classifier pipeline.Rationales are often regularized to be succinct and continuous.
  • Interpretation: The mathematical property computed by an attribution method does not automatically justify the high-level interpretation of feature importance.That association requires separate justification.
  • Evaluation limits: Human-alignment evaluations support plausibility rather than faithfulness because models and humans may reach the same prediction through different reasoning mechanisms.Examples include models using watermarks instead of medical signals or subtle textual idiosyncrasies instead of annotated rationales.
  • Evaluation limits: Removal-based prediction-change metrics can fail under nonlinear feature interactions, while retraining metrics can reject valid attributions when alternative features yield the same accuracy.For an OR function, removing the first active feature can appear useless; retraining on x2 can match accuracy achieved using x1.
  • Related approaches: Semi-natural datasets with explicitly defined ground-truth explanations and sanity checks provide complementary ways to evaluate attribution methods.The paper focuses on model-agnostic, dataset-side modifications and identifies additional failure cases.

3 Desiderata for Attribution Values

The paper defines attribution desiderata around recovering features that fundamentally affect model decisions while suppressing features unrelated to the label. It uses dataset modifications and precision-recall metrics to make these requirements testable.

  • Continuous attributions: Correct explanations should not miss features of fundamental importance to the model, denoted FC.Dataset modification makes such usage identifiable by preventing models that ignore introduced features from achieving high accuracy.
  • Continuous attributions: Introduced label noise can force a high-performing model to use a known feature such as an X-ray timestamp rather than genuine medical features.The attribution percentage of the timestamp pixels then measures whether the method identifies that contribution.
  • Continuous attributions: Attribution percentage Attr%(FC) should be approximately 1 when FC contains all features used by the model.The measure sums attribution values over the relevant feature set relative to all features.
  • Continuous attributions: Features non-informative to the label, denoted FN, should not be highlighted, and Attr%(FN) should decrease toward 0 as model performance improves.Reliance on FN is detrimental because it distracts the model from predictive information.
  • Sparse attributions: For explanations selecting k features, precision and recall are preferred because selecting every feature can trivially achieve complete FC coverage.Precision penalizes unnecessary features, while recall measures whether important features are recovered.

4 Dataset Modification with Ground Truth

The dataset modification procedure weakens original label signals and adds label-dependent local input manipulations, creating settings where high accuracy implies reliance on known features. This enables quantitative evaluation of attribution methods against effective regions.

  • Procedure: The procedure uses label reassignment and input manipulation to transform original data into modified data with controlled predictive features.Figure 2 presents the original instance, weakened original signals, and newly introduced predictive features.
  • Label reassignment: Label reassignment preserves the original binary label with probability r and flips it otherwise, bounding feature-only accuracy by p* = max(r, 1 − r).When r = 0.5, existing features are uninformative in expectation.
  • Label reassignment: For K-class classification, a reassignment matrix R determines new labels and bounds expected classifier accuracy by p* = max_i,j R_i,j.This generalizes the binary reassignment scheme.
  • Input manipulation: Input manipulations are selected according to the reassigned label and applied through a function q, optionally including a blank manipulation that leaves the input unchanged.The manipulation choice may be deterministic or stochastic.
  • Input manipulation: Local manipulations modify only part of the input, whose changed features form the effective region φ_l(x).The blank manipulation has an empty effective region.
  • Ground-truth guarantee: When modified-data accuracy exceeds p*, the classifier is guaranteed to rely on the manipulation within the joint effective region φ∪(x).A watermark applied exclusively to one class can make the modified-data bound p̂* = 1 achievable.
  • Evaluation: The evaluation compares saliency maps, attention mechanisms, and rationale models with the desiderata to identify deficiencies and guide improvements.The joint effective region must include all relevant manipulations because absence of a manipulation can itself support a decision.

5 Evaluating Image Saliency Maps

The evaluation tests whether saliency maps identify injected image artifacts whose use is required for high-performing classifiers. Across manipulation type, visibility, training accuracy, and feature correlation, the methods often fail to reflect the model’s actual reliance on those artifacts.

  • Experimental setup: The study uses binary bird-species classification with five injected artifacts—blurring, brightness change, hue shift, pixel noise, and watermarks—and evaluates five saliency-map methods.The models use ResNet-34 classifiers trained on pairs of easily confused species, with some experiments also using visually distinct pairs.
  • Experimental setup: The ground-truth evaluation measures attribution percentage assigned to the joint effective region of the manipulations.Under random label reassignment, only the manipulation correlates with the label, so near-perfect models should have %Attr approximately 1 regardless of effective-region size.
  • 5.1 Attr% by Attributions and Manipulations: 70% of runs exceeded 95% test accuracy, but no method consistently achieved %Attr approximately 1; SHAP performed best at 69% attribution for 40% effective regions on average.Performance varied substantially by manipulation type, and watermark presence was often easier to detect than watermark absence.
  • 5.2 Attribution vs. Test Accuracy: As test accuracy increased, only SHAP on watermarks showed the expected consistent attribution increase; other methods showed mild or noisy trends.The noisy trends suggest that these methods often fail to track increasing model reliance on the manipulation during training.
  • 5.3 Attribution vs. Manipulation Visibility: Except for SHAP on watermarks, methods generally lacked a consistent increase in attribution as manipulation visibility increased.The authors emphasize that less visible artifacts are precisely those interpretability methods should uncover, yet none satisfied this requirement.
  • 5.4 Attribution vs. Original Feature Correlation: Many saliency maps were inconsistent with the Shapley-axiom attribution range as label-correlation strength changed, although SHAP tracked the expected decrease for watermarks.Gradient tracked the predicted range only for the positive class containing the manipulation, and similar versus distinct species pairs showed no clear attribution difference.
  • 5.5 Discussion: Because these failures cast doubt on detecting spurious correlations, the authors recommend dry-running interpretability methods on models guaranteed to use known features before analyzing deployed models.This procedure tests whether the planned method actually highlights features whose causal relevance is controlled by construction.

6 Evaluating Text Attentions

The attention experiments test whether attention scores identify manipulated, label-correlated words and avoid similar non-correlating words. Despite over 97% accuracy, attention patterns vary substantially and generally do not provide reliable attributions.

  • 6.1 Highly Obvious Manipulations: The experiment modifies BeerAdvocate reviews so article words become label-correlated target features, then measures their attribution with %Attr.The selected dataset contains 12,000 reviews split into train, validation, and test sets.
  • 6.1 Highly Obvious Manipulations: Over 97% accuracy coexists with only 8.6% attribution on article words, barely above their 7.9% frequency.Near-perfect models were expected to achieve %Attr ≈1 on the effective region.
  • 6.1 Highly Obvious Manipulations: Attention visualizations show that article words usually do not stand out, and high attention values generally do not correlate strongly with important words.Article attention may stand out only locally relative to neighboring words.
  • 6.2 Misleading Non-Correlating Features: In the CN and NC datasets, both models exceed 97% accuracy, but only the CN model’s displayed run focuses exclusively on correlating articles.The NC model behaves similarly to the earlier experiment rather than clearly isolating the relevant articles.
  • 6.2 Misleading Non-Correlating Features: Across 11 runs, the clean CN attention pattern does not persist, and models sometimes assign higher-than-random weights to non-correlating articles, especially in NC.The variation indicates that attention weights cannot be readily and reliably interpreted as attributions without further validation.
  • 6.3 Discussion: The authors conclude that attention is useful in neural-network architectures but can hardly be interpreted as attribution for model understanding and debugging.They recommend calibrating future proxy metrics against ground truth in controlled settings before applying them to natural datasets.

7 Evaluating Text Rationales

The rationale experiments evaluate whether selected text spans identify manipulated features without misleading users about irrelevant features. Most models select rationales well for obvious manipulations, but rationales can include non-correlating articles and obscure the model’s reasoning.

  • 7.1 Highly Obvious Manipulations: The study evaluates reinforcement-learning and continuous-relaxation rationale models using target selection rates instead of length regularization.The experiments also remove the discontinuity penalty because the ground-truth rationales are not continuous.
  • 7.1 Highly Obvious Manipulations: For obvious manipulations, a good rationale should achieve high precision at low selection rates and high recall at high selection rates.The necessary condition is selecting at least one article word regardless of selection rate.
  • 7.1 Highly Obvious Manipulations: Except for the CR model at the lowest selection rate, all models achieve near-perfect precision and recall while exceeding 97% accuracy.The models are nearly dataset-wide optimal under mini-batch-level selection-rate regularization.
  • 7.2 Misleading Non-Correlating Features: For misleading-feature experiments, rationales consist almost exclusively of article words but may select non-correlating articles and miss correlating ones, reducing precision.The issue is especially evident for the RL model across selection rates from 0.03 to 0.09.
  • 7.3 Discussion: Rationale models can preserve a causal relationship between selected features and prediction while still selecting unnecessary, misleading features that obscure the model’s reasoning.The authors report that this problem is more severe with RL training and suggest post-processing to prune rationales.

8 Conclusion and Future Work

The paper proposes unit-test-like evaluations on semi-natural datasets with induced attribution ground truth and finds that evaluated methods do not perform satisfactorily across vision and text. It discusses broader applicability, limitations, and directions for more realistic evaluations.

  • Conclusion: The evaluation uses carefully modified semi-natural datasets to test whether attribution methods recover ground-truth model reasoning.The procedure introduces manipulations so high-performing classifiers must rely on them, while defining conditions such as zero attribution for features that do not affect decisions.
  • Conclusion: None of the evaluated attribution methods achieves totally satisfactory performance across vision and text domains.The authors describe the conclusions as mostly negative and identify future directions for improving attribution methods.
  • Implications: The study’s mostly negative conclusions cast doubt on using current interpretability methods to identify spurious correlations in deployed systems.The authors connect this concern to the frequent influence of spurious correlations on model decisions.
  • Future Work: More realistic generated features could extend the procedure beyond manually defined artifact features and better simulate scientific-discovery use cases.The paper suggests image inpainting and masked language prediction as possible tools for generating such features.
  • Future Work: The framework is domain-agnostic and can be instantiated beyond image and text data, including graph, speech, and time-series settings.The paper also evaluates feature-selection-style attributions, which are common for text models.
  • Experimental Scope: The manipulation design simulates image artifacts through five types whose effective regions and visibility levels are controlled by parameters.Examples include peripheral blurring, central brightness shifts, and striped hue shifts.

B.2 Saliency Map Methods

The saliency-map evaluation compares five methods across several image manipulations and tests whether attribution tracks effective regions, model accuracy, visibility, and label-correlation strength. The supplied passages specify the evaluation dimensions but not the plotted outcomes.

  • Methods: The study considers five saliency map methods for evaluating image-feature attribution.The supplied method description identifies Gradient and SmoothGrad among the methods.
  • Evaluation Axes: The evaluation plots attribution percentage against the manipulation’s effective-region percentage for every saliency-map and manipulation pair.This tests whether attribution concentrates on the manipulated effective region.
  • Evaluation Axes: The evaluation also plots attribution percentage against test accuracy for every saliency-map and manipulation pair.The passages define the comparison but do not state the plotted results.
  • Evaluation Axes: Manipulation visibility is varied while effective regions remain independent of visibility, enabling fair comparison of attribution percentages.Visibility levels are defined separately for manipulation types such as blurring, brightness, and hue shifts.

B.6 Attribution vs. Original Feature Correlation

The paper defines attribution-related accuracy by simulating access to selected features through conditional data distributions rather than out-of-distribution masked images. It then characterizes boundary cases for empty, full, and feature subsets.

  • Definition: The method estimates expected accuracy when the model is given only feature subset F by comparing predictions on one input with labels from another input sharing F.The expectation is taken over feature values drawn from the marginal distribution of F.
  • Definition: Suppressing information beyond F is modeled as making inputs that agree on F indistinguishable to the model.This avoids evaluating the model on images with other pixels blacked out, which may be out of distribution.
  • Boundary Cases: With balanced labels, a(∅) equals 0.5, while a(FM ∪ FO) equals the normal model accuracy p.The former represents no input information and the latter full access to the input.
  • Boundary Cases: The accuracy using only FO satisfies a(FO) ≤ r because label reassignment weakens FO’s correlation with the label.This links the subset-accuracy definition to the reassignment parameter.
  • Boundary Cases: The definition implies a(FM) = a(FM ∪ FO) = p when FM is perfectly correlated with the label.The paper notes that this implication follows mechanically from the Shapley-value calculation and does not assess whether it is reasonable.

C Additional Results for Attention Mechanism Evaluations

The appendix provides additional visualizations of learned attention distributions, including distributions over articles and non-articles and dataset-specific visualizations for CN and NC.

  • Attention Distributions: Additional attention distributions distinguish articles from non-articles using orange and green bars.The figure provides supplementary visualizations of the learned attention distribution.
  • Dataset Comparisons: Additional attention visualizations compare the CN and NC datasets using bars for correlating articles, non-correlating articles, and other words.CN appears on the left and NC on the right.

D Additional Results for Rationale Model Evaluations

Additional reviews show that the faulty CR model consistently selects the first few words, while comparisons with the RL model reveal different alignment with correlating articles.

  • The faulty CR model consistently selects the first few words across four additional reviews.The figure annotations mark selected non-articles, selected articles, and missed articles.
  • For the same review and target %Sel, the CR and RL models produce different rationale selections.
  • On the CN and NC examples, CR focuses exclusively on correlating articles, whereas RL selects non-correlating articles and misses a correlating article on NC.
Loading 2104.14403v2…