PaperlayerView exact revision on arXiv

Source-linked AI summary

On Baselines for Local Feature Attributions

Johannes Haug, Stefan Zürn, Peter El-Jiz, Gjergji Kasneci

arXiv:2101.00905v1cs.LG

TL;DR

Local feature attributions explain complex models, but their baseline reference can affect the quality of the explanation, and zero is not meaningful for every data type. The paper proposes a taxonomy and evaluates common baselines with attribution models on tabular data, finding strong baseline sensitivity without a universally best method.

  • Problem

    Baselines affect local feature attributions, yet simplistic choices such as zero may not represent missingness across different data domains.

    Source: Introduction · p. 1

  • Method

    The paper proposes a taxonomy of baseline methods and evaluates common baselines with four attribution models on tabular data sets.

    Source: Introduction · p. 1

  • Results

    Attributions were highly sensitive to baseline choice, with no universally best method; expectation, blurred, constant, and Gaussian baselines often performed well, while uniform and maximum-distance baselines generally performed worse.

    Source: Results and Discussion · pp. 5–6

  • Takeaways & Limitations

    Baseline appropriateness depends strongly on the data distribution, so the taxonomy may help compare and select methods for tabular data.

    Source: Introduction · p. 1, Results and Discussion · p. 6

    The reported observations and hypotheses apply to this tabular-data experiment and do not necessarily generalize.

    Source: Results and Discussion · p. 5

Abstract

from arXiv · show

High-performing predictive models, such as neural nets, usually operate as black boxes, which raises serious concerns about their interpretability. Local feature attribution methods help to explain black box models and are therefore a powerful tool for assessing the reliability and fairness of predictions. To this end, most attribution models compare the importance of input features with a reference value, often called baseline. Recent studies show that the baseline can heavily impact the quality of feature attributions. Yet, we frequently find simplistic baselines, such as the zero vector, in practice. In this paper, we show empirically that baselines can significantly alter the discriminative power of feature attributions. We conduct our analysis on tabular data sets, thus complementing recent works on image data. Besides, we propose a new taxonomy of baseline methods. Our experimental study illustrates the sensitivity of popular attribution models to the baseline, thus laying the foundation for a more in-depth discussion on sensible baseline methods for tabular data.

Introduction

Local feature attributions help make complex models more interpretable, but their reference baselines can change attribution quality. The paper proposes a baseline taxonomy and evaluates common methods on tabular data, finding substantial baseline sensitivity and no universally best method.

Introduction

  • Local attribution methods assess feature importance relative to a reference input, often called a baseline.The baseline is closely related to the approximate neutral value a feature would take if considered missing.Source: Introduction · p. 1
  • Because missingness is domain-specific, a zero baseline may be meaningless for nominal features and should be chosen with the data distribution in mind.Zero can represent a missing image pixel but not necessarily a missing nominal feature.Source: Introduction · p. 1
  • Baselines can substantially alter attribution quality, and no method performs best universally; some methods rarely yield discriminative attributions on tabular data.The experiments complement prior image-data work and suggest that uniform and maximum-distance baselines may be poor choices for tabular data.Source: Introduction · p. 1
  • The paper categorizes baseline methods and evaluates common choices with several attribution models on tabular data sets.The taxonomy is intended to help compare and select baselines; the experiments provide a basis for more principled choices in practice.Source: Introduction · p. 1

A Taxonomy of Baseline Methods

The paper proposes a taxonomy to categorize and compare baseline methods, distinguishing them by whether their outputs vary across observations and repeated calls. It then places common methods into these categories and notes a practical constraint on the neutral baseline.

A Taxonomy of Baseline Methods

Categorizing Common Baseline Methods

  • The constant baseline is static and deterministic, while the maximum-distance baseline is dynamic and deterministic.The constant baseline is a fixed value specified once; maximum distance selects the observation furthest away by the ℓ1-norm.Source: Categorizing Common Baseline Methods · p. 2
  • The blurred baseline applies Gaussian blur to an observation and is dynamic and stochastic, but it requires a meaningful neighborhood among features.That neighborhood may not be evident in tabular data, though the method can be applied to numeric tabular data.Source: Categorizing Common Baseline Methods · p. 2
  • Gaussian and uniform baselines are dynamic and stochastic, drawing random samples from feature-wise distributions.The Gaussian distributions are centered at the original feature values, while the uniform distributions span each feature's valid range.Source: Categorizing Common Baseline Methods · p. 2
  • The expectation baseline uses the expectation of a reference sample and is static and stochastic when that sample is randomly drawn from training data..Source: Categorizing Common Baseline Methods · p. 2
  • The neutral baseline was excluded from the experiments because it was specified only for certain neural-network architectures and had no open-source implementation at the time.It is described as static and deterministic when the decision boundary does not shift.Source: Categorizing Common Baseline Methods · p. 2

Experiments

The experiments compare four local attribution methods and baseline choices on tabular data using top-K ablation. Results vary by baseline, with expectation, blurred, constant, and Gaussian baselines frequently producing discriminative attributions, while uniform and maximum-distance baselines generally perform worse.

Experiments

  • The experiments used KernelSHAP, DeepSHAP, DeepLift, and Integrated Gradients, evaluating attribution discrimination by masking top-attributed features with noise and measuring F1 change.For DeepSHAP, DeepLift, and Integrated Gradients, the authors used shallow neural networks; KernelSHAP was applied to an SVM.Source: Experiments · pp. 2–3

Results and Discussion

  • All attribution models were sensitive to baseline choice, and the reported observations are specific to these experiments rather than necessarily general.Figure 1 presents the detailed ablation results across data sets, attribution models, and baselines.Source: Figure 1 · p. 4, Results and Discussion · p. 5
  • Uniform baselines may poorly approximate heterogeneous tabular distributions, while maximum-distance baselines may select unrepresentative outliers.The authors contrast their uniform-baseline results with competitive results reported for image data.Source: Results and Discussion · p. 5
  • Expectation baselines use training-observation expectations, while Gaussian baselines performed well in general and may approximate data distributions sufficiently well.The authors suggest that attributions become more discriminative when baselines more closely follow the data-generating distribution.Source: Results and Discussion · p. 5
  • Stronger blurring makes the blurred baseline approach a low-variance constant, consistent with their nearly identical performance.The authors attribute this relationship to blur reducing variance among input features.Source: Results and Discussion · pp. 5–6
  • On extremely imbalanced Fraud Detection data, all baselines performed equally well, but the ablation test does not explicitly account for class imbalance.The data include PCA-generated features, and removing leading components may remove much discriminative information.Source: Results and Discussion · p. 6
  • KernelSHAP performance decreased as data dimensionality increased, especially on HAR, because its coalition-sample requirements grow exponentially.The authors note that more advanced hardware may improve its performance.Source: Results and Discussion · p. 6
  • Expectation, blurred, constant, and Gaussian baselines frequently produced discriminative attributions; uniform and maximum-distance baselines generally performed worse.The average ablation results across tabular data sets support this pattern.Source: Figure 2 · p. 5, Results and Discussion · p. 6

Conclusion

The paper compares common baseline methods for local attributions on tabular data and proposes a new taxonomy. Its results show that baselines can dramatically affect attribution quality, warranting more attention to sensible baseline selection.

Conclusion

  • Baseline choice can dramatically affect the quality of generated feature attributions.The paper argues that selecting and developing sensible baselines deserves more attention in research and practice.Source: Conclusion · p. 6
  • The paper provides a first empirical comparison of common local-attribution baselines on tabular data, complementing studies on image data.This comparison is paired with a proposed taxonomy of baseline methods.Source: Conclusion · p. 6
  • The proposed taxonomy may support deeper discussion and more principled baseline choices in research and practical applications.Source: Conclusion · p. 6

Related papers

Loading 2101.00905v1…