PaperlayerView exact revision on arXiv

Source-linked AI summary

On quantitative aspects of model interpretability

An-phi Nguyen, María Rodríguez Martínez

arXiv:2007.07584v1cs.LGstat.ML

TL;DR

Programmatic comparison of interpretability methods remains unclear, and explanation quality is contextual. The paper proposes quantitative metrics for interpretability components and evaluates them on benchmark datasets, reporting that method choice depends on the task and target user.

  • Problem

    It remains unclear how to compare interpretability methods programmatically without qualitative assessment or user studies, and explanation quality depends on context.

    Source: §1 · p. 1

  • Method

    The paper proposes quantitative metrics and separates the feature extractor from the explainability method when evaluating interpretability.

    Source: §1 · p. 1, §2 · p. 2, §2.1 · p. 2

  • Results

    The metrics were experimentally validated on different benchmark datasets, and the most appropriate method depends on the task and target user’s trade-off among simplicity, broadness, and fidelity.

    Source: §4 · p. 8

  • Takeaways & Limitations

    The metrics can guide method selection and help choose a smaller subset of explanations for user studies.

    Source: §1 · p. 1

    Mutual-information values do not monotonically track explanation inaccuracy: the Entropy extractor had lower scores than the Random extractor while LIME produced rules more aligned with the true rules.

    Source: §3.1 · p. 5

Abstract

from arXiv · show

Despite the growing body of work in interpretable machine learning, it remains unclear how to evaluate different explainability methods without resorting to qualitative assessment and user-studies. While interpretability is an inherently subjective matter, previous works in cognitive science and epistemology have shown that good explanations do possess aspects that can be objectively judged apart from fidelity), such assimplicity and broadness. In this paper we propose a set of metrics to programmatically evaluate interpretability methods along these dimensions. In particular, we argue that the performance of methods along these dimensions can be orthogonally imputed to two conceptual parts, namely the feature extractor and the actual explainability method. We experimentally validate our metrics on different benchmark tasks and show how they can be used to guide a practitioner in the selection of the most appropriate method for the task at hand.

1 Introduction

The paper proposes quantitative metrics for aspects of interpretability and distinguishes the feature extractor from the explainability method. It frames simplicity and broadness as assessable alongside fidelity, while recognizing that interpretability is contextual.

1 Introduction

  • Because explanation quality depends on the interacting agents and the information sought, a computational benchmark across all methods is unlikely to be possible.Source: §1 · p. 1
  • The paper proposes metrics for three quantitative aspects of interpretability to help practitioners select methods and narrow the explanations presented in user studies.The metrics are not intended to replace user studies.Source: §1 · p. 1
  • Feature extractors supply the representation, while explainability methods provide the explanation; the paper treats these as distinct parts of an interpretability method.The paper uses an alphabet-and-grammar analogy for this separation.Source: §1 · p. 1

1.1 Quantitative aspects: simplicity, broadness, and fidelity

  • Cognitive-science and epistemology work describes explanations as simple and broad, with dimensions including robustness to unimportant details, accuracy, general applicability, and ease of grasp.In machine learning, fidelity—how correct an explanation is—also matters because explanations can be approximations.Source: §1.1 · p. 2

2 Simplicity, broadness, and fidelity in practice

The paper argues that interpretability metrics must account for both the features used and the explanation modality. It proposes measures for feature extractors, example-based explanations, and feature attributions.

2 Simplicity, broadness, and fidelity in practice

  • Because feature representations and explanation modalities differ, the metrics should be tailored to each; the feature extractor and explainability model play orthogonal roles in simplicity, broadness, and fidelity.Changing either component can independently change explanation quality.Source: §2 · p. 2

2.1 Feature Extractor metrics

  • For feature extractors, I(X, Z) monitors simplicity and broadness, while I(Z, Y) monitors fidelity; their trade-off reflects information discarded from the original features versus information retained about the target.A non-injective extractor can discard information and map multiple original samples to one feature-space value.Source: §2.1 · pp. 2–3

2.2.1 Example-based methods

  • Example-based explanations use representative examples or concepts, and may explain a prediction with examples sharing its prediction or with counterexamples.Source: §2.2.1 · p. 3
  • Example-based metrics assess non-representativeness, diversity, and the number of examples; non-representativeness relates mainly to fidelity, diversity to integration, and fewer examples to simplicity.High non-representativeness can also indicate factual inaccuracy.Source: §2.2.1 · p. 3

2.2.2 Feature Attributions

  • For feature attributions, the paper proposes relating attribution magnitude to expected prediction loss when a feature is unknown, with feature values averaged under a density that may be independent or conditional on other features.This choice distinguishes interpretations true to the model from those true to the data.Source: §2.2.2 · p. 3
  • Monotonicity and non-sensitivity assess attribution faithfulness, while the count of nonzero attributions measures complexity.Non-sensitivity checks whether zero attribution is assigned only to features on which the model is not functionally dependent.Source: §2.2.2 · pp. 3–4
  • Effective complexity accounts for feature interactions by finding the smallest set of top-attributed features that keeps expected loss below a chosen tolerance.Low effective complexity yields simpler, broader explanations at a cost of reduced factual accuracy.Source: §2.2.2 · p. 4

3 Experimental validation

Across the experiments, the metrics reveal trade-offs between representative, faithful explanations and diverse, general ones, and distinguish feature-attribution behavior on analytic and text tasks.

3.1 Feature extractor effect on LIME

  • Identity LIME rankings match the tree’s feature importance and evidence signs; Random can mislead despite similar target MI, while Entropy yields better-aligned, simpler, broader rules.The Random extractor alters an important feature in the analyzed sample, and its local approximation score misleadingly suggests a good explanation.Source: §3.1 · p. 5, Figure 1 · p. 5

3.2 Example-based explanations on Rotated MNIST

  • K-medoids yields the most representative but less diverse prototypes; ProtoDash favors generality, while diversity converges and mean representativeness stays relatively constant as exemplar counts rise.The comparison uses six prototypes per class on Rotated MNIST.Source: §3.2 · p. 6, Figure 2 · p. 6, Figure 3 · p. 7

3.3 Evaluating feature attributions

  • Feature-attribution metrics are evaluated first on a known nonlinear function and then on a higher-dimensional text-classification task.The text task uses news excerpts; the function test includes two non-active variables.Source: §3.3 · p. 6, §3.3.1 · p. 7, §3.3.2 · p. 8

3.3.1 Test Functions

  • On the known function, Random fails all four metrics and dummy-variable detection; Saliency and InpXGrad identify three effective variables, while IntGrad is less monotonic.The authors argue that Saliency and InpXGrad better reflect the function at the tested point because the contribution of x2 is effectively negligible there.Source: §3.3.1 · p. 7

3.3.2 Text Classification

  • For the text sample, IntGrad gives the most general explanation but is non-monotonic; despite perturbing more words, it preserves predictions at a rate similar to other methods.The authors interpret this as identifying an important group of words without accurately estimating each word’s individual importance.Source: §3.3.2 · p. 8

4 Conclusion and Related Work

The paper presents objective, quantifiable interpretability metrics, validates them on benchmark datasets, and frames method choice as a task- and user-dependent trade-off among simplicity, broadness, and fidelity. It also distinguishes its evaluation approach from prior work on feature-attribution metrics.

4 Conclusion and Related Work

  • The proposed metrics were experimentally validated on benchmark datasets and can help practitioners select methods according to task- and user-dependent trade-offs among simplicity, broadness, and fidelity.The appropriate trade-off depends on the task and target user of the explanation.Source: §4 · p. 8
  • The authors present a systematic accounting of functionally grounded interpretability metrics and say they know of no prior programmatic evaluation of feature extractors and example-based methods.Earlier work recognized the feature extractor’s role, but did not use the separation for evaluation.Source: §4 · p. 8
  • Sensitivity-n can characterize algorithmic behavior, but its use for comparing interpretability remains unclear; the authors argue ROAR measures data rather than model interpretability.They argue that retraining after feature removal may not reveal the original model’s decision process.Source: §4 · p. 8
  • Infidelity depends on the distribution used and requires baselines, while Arya et al.’s monotonicity metric [42] measures probability drop rather than uncertainty in probability estimation.The paper’s monotonicity definition can be extended to regression tasks depending on the loss function.Source: §4 · p. 8

Broader Impact

The authors connect interpretability to ethical and safety concerns while arguing that the lack of comparison metrics may hinder the field’s development. They report no direct negative impact from their work.

Broader Impact

  • The authors identify ethical and safety [44] concerns as societal reasons for interpretability.They describe the need for interpretability in relation to these concerns as undeniable.Source: Broader Impact · p. 9
  • They argue that methods remain difficult to compare without qualitative assessments or user studies.The passage places this gap against the rapid growth of interpretable machine learning.Source: Broader Impact · p. 9
  • The authors argue that a lack of metrics may make field development inefficient, and say they see no direct negative impact from this work.Source: Broader Impact · p. 9

A Additional Figures

The supplementary figures show attributions for Park’s test function and perturbation samples for analyzed methods, including a comparison of the number of words perturbed by IntGrad.

A Additional Figures

  • Figure 4 presents attributions computed for the Park test function.Source: Figure 4 · p. 13
  • Figure 5 places the original sample at the top of each method’s perturbation subfigure and shows more perturbed words for IntGrad.Source: Figure 5 · p. 14
  • A supplementary figure example shows perturbations with attributions provided by Saliency.Source: A Additional Figures · p. 14

Related papers

Loading 2007.07584v1…