PaperlayerView exact revision on arXiv

Source-linked AI summary

How to Explain Individual Classification Decisions

David Baehrens, Timon Schroeter, Stefan Harmeling, Motoaki Kawanabe, Katja Hansen, Klaus-Robert Mueller

arXiv:0912.1128v1stat.MLcs.LG

TL;DR

Most machine-learning classifiers predict labels without explaining the features behind an individual decision. This paper proposes local explanation vectors for arbitrary, possibly nonlinear classifiers, and validates them across benchmark and drug-discovery tasks. The explanations recover task-relevant distinctions, agree with existing chemical domain knowledge, and reveal local chemical peculiarities.

  • Problem

    Most classification algorithms do not explain which input features made a nonlinear model assign a label to an individual data point.

  • Method

    The paper defines local explanations as gradients of class probabilities and estimates them for arbitrary classifiers, including those without direct probability outputs.

  • Results

    The method identifies distinguishing properties in Iris and USPS classification and produces drug-discovery explanations that agree with domain knowledge while revealing steroid peculiarities.

  • Takeaways & Limitations

    Local explanation vectors show how features influence individual predictions and how examples can be moved to change their assigned class labels.

  • Takeaways & Limitations

    The approach inherits limitations of models that assume stationarity, and its gradient explanation can become zero where the class probability is locally maximal.

Abstract

from arXiv · show

After building a classifier with modern tools of machine learning we typically have a black box at hand that is able to predict well for unseen data. Thus, we get an answer to the question what is the most likely label of a given unseen data point. However, most methods will provide no answer why the model predicted the particular label for a single instance and what features were most influential for that particular instance. The only method that is currently able to provide such explanations are decision trees. This paper proposes a procedure which (based on a set of assumptions) allows to explain the decisions of any classification method.

1. Introduction

Modern nonlinear classifiers can predict unseen labels effectively, but usually do not explain which features drove an individual decision. The paper introduces local explanation vectors to reveal instance-specific relevance and peculiarities for arbitrary classification methods.

  • Most nonlinear classification algorithms predict unseen data without explaining why a particular label was assigned.
  • Global feature-selection methods summarize input usefulness across training instances rather than explaining individual predictions.
  • The proposed framework produces local explanation vectors applicable to any classification method.
  • Local explanations identify features relevant at a point in data space and expose peculiarities that global views can miss through cancellation effects.
  • The paper defines local explanations as class-probability gradients, estimates them for methods without direct probability outputs, and evaluates them on GPC, k-NN, and SVM classifiers.

2. Definitions of Explanation Vectors

The paper defines an explanation vector as a local gradient describing how feature changes affect class probabilities. It illustrates this definition with Gaussian Process Classification and a simple object-classification example.

  • For Bayes classification, the explanation vector is the derivative of the probability of the alternative class at the point being explained.
  • The vector has one component per feature; large absolute values identify influential features, while the sign indicates how increasing a feature changes the assigned-class probability.
  • The explanation vector forms a local vector field indicating directions away from the corresponding class and thereby characterizing local label changes.
  • The explanation can be zero where the relevant class probability is locally flat or maximal, creating a limitation for deterministic or saturated predictions.
  • For binary probability models, the local explanation is the gradient ηp(x0)=∇p(x)|x=x0, whose direction gives the steepest ascent toward higher positive-class probability.
  • Gaussian Process Classification supplies a probability model whose local gradient can be computed approximately using expectation propagation.
  • In the object-classification illustration, gradient-vector length increases near class transitions, while feature importance can involve both dimensions or single out one dimension.

3. Estimating Explanation Vectors

For classifiers without probability-valued outputs, the paper estimates a probability-like classifier that mimics the original decision rule. Explanation vectors are then computed from this surrogate, using Parzen windows in the presented method.

  • Some classifiers, including SVMs, output decision rules without a direct probability interpretation required by the explanation definition.
  • For SVMs with RBF kernels, the decision-function derivative can point in the wrong direction outside data clusters, motivating comparison with an estimated probability function.
  • The method approximates the classifier with another classifier whose form resembles a Bayes classifier, enabling explanation-vector estimation.
  • Parzen windows estimate class-conditional weighted densities and thereby provide class-probability estimates for the surrogate classifier.
  • The approach can generate arbitrarily much labeled data from the classifier, but estimating the mimicking classifier requires care in high-dimensional spaces.
  • The mimicking classifier is constructed from the original classifier's labels, preserving the orientation of the estimated explanation vectors for interpretation.

4. Explaining Iris Flower Classification by k-Nearest Neighbors

The framework is applied to distinguish Iris versicolor from Iris setosa and Iris virginica using k-nearest neighbors, then explanation vectors reveal which flower features drive individual classifications. Petal area is the main distinguishing feature, with different directions separating the two class-1 species from versicolor.

  • Experimental setup: The experiment classifies Iris versicolor as class 0 and Iris setosa plus Iris virginica as class 1.The 150 flowers are randomly split into 100 training and 50 test examples after normalization using training-set statistics.
  • Experimental setup: A k-nearest neighbor classifier with k = 4 achieves 3% training error and 8% test error.The value k = 4 was selected by leave-one-out cross validation on the training data.
  • Explanation results: Explanation vectors show that petal length and petal width jointly provide the most important feature combination, approximately corresponding to petal area.Sepal dimensions are also relevant, but less distinguishing.
  • Explanation results: Iris setosa differs from versicolor through smaller petal area, whereas Iris virginica differs through larger petal area.Thus, the explanations for the two class-1 species are different even though both are contrasted with versicolor.

5. Explaining USPS Digit Classification by Support Vector Machine

The paper applies its explanation-vector framework to an SVM distinguishing USPS images of twos and eights. The vectors indicate which image parts support each classification and how edits could move examples toward the other class, although test-set transformations are less pronounced.

  • Experimental setup: The USPS digits are represented as 16×16 images reshaped into 256×1 vectors and classified with an RBF-kernel SVM.The SVM uses kernel width σ = 1 and regularization constant C = 10, selected by grid search with training-data cross validation.
  • Experimental setup: The training set contains 47 twos and 53 eights, with training error 0.00.
  • Training-set explanations: For twos, explanation vectors identify missing or extraneous strokes whose addition or removal would support classification as eights.Some examples require both adding black lines and removing existing black spots; one dash-like example is identified as an erroneous dataset digit.
  • Test-set explanations: For eights, vectors similarly suggest removing most dark regions and adding lower-part structure, but transformations do not always reach the twos class.Test-set vectors show the same tendencies as training-set vectors, though they are less pronounced.

6. Explaining Mutagenicity Classification by Gaussian Processes

The paper applies local explanation vectors to Gaussian Process Classification of mutagenicity, using molecular-substructure features to identify globally and locally influential toxicophores. The explanations agree with established toxicophore knowledge while revealing steroid-specific exceptions.

  • Experimental setup: The study predicts mutagenicity for 6512 chemical compounds represented by counts of 142 molecular substructures, using 2000 training and 4512 test examples.The GPC model uses an RBF kernel, with balanced class and steroid distributions in the split.
  • Experimental setup: 84% area under curve confirms the GPC model’s previous mutagenicity-prediction results.Error rates are reported through Figure 6.
  • Local explanations: Local explanation vectors evaluate feature importance for each of the 4512 test compounds through distributions of local gradients.The explanation vectors are calculated together with predictions and then evaluated across selected molecular features.
  • Global feature patterns: Toxicophores generally increase predicted mutagenicity, whereas detoxicophores generally decrease it.These directions correspond to positive local gradients for toxicophores and negative local gradients for detoxicophores.
  • Global feature patterns: 9 out of 10 known toxicophores rank near the top, with mean rank 19, while five detoxicophores have mean rank 138 of 142.The aromatic nitrosamine is the exception at rank 81; the results indicate that established toxicophore knowledge could be discovered from local gradients.
  • Local chemical exceptions: Steroids show near-zero local gradients for epoxide and aliphatic nitrosamine features, unlike non-steroids, exposing steroid-specific immunity to some toxicophores.For non-steroids, the known toxicophores exhibit positive local gradients; this local distinction would be missed by a purely global trend.

7. Related Work

The paper distinguishes its local explanation framework from global feature-selection and omission-based approaches by assigning potentially different explanations to individual data points. It also permits continuous-feature changes in arbitrary directions and captures local feature interactions.

  • Global versus local explanations: Global feature-selection and input-pruning methods provide ensemble-level feature relevance rather than explanations for individual instances.Such methods identify inputs useful for generalization across the training set but do not identify locally influential features.
  • Sensitivity analysis: The framework explains the influence of moving individual data points locally, rather than detecting or removing influential samples.Its explanation vectors extract sensitive features relevant to prediction results.
  • Locality: The framework’s explanations can depend on assigned evidence values, making them local to individual prediction situations.Related Bayesian-network work constructs explanations from variables affecting inference for particular evidence.
  • Methodological differences: Unlike omission-based methods, the approach supports continuous features without requiring feature structure or discretization parameters.It allows changes in any direction, including weighted combinations of variables.
  • Methodological differences: Considering weighted combinations of variables allows the framework to represent feature interactions that single-feature-at-a-time methods cannot measure.This contrasts with approaches that vary one input while fixing all others.

8. Discussion

The framework’s explanations inherit limitations from the models and estimation assumptions they rely on. Zero derivatives can require second-order analysis, while stationarity assumptions and model behavior near sparse-data boundaries constrain interpretation.

  • Limitations: Zero first derivatives produce zero explanation vectors at local probability maxima, so the most influential direction may require the Hessian’s largest-eigenvalue eigenvector.The paper formulates this direction through a local optimization problem when the first-order Taylor term vanishes.
  • Limitations: Analytical gradients follow the predictive model, whereas estimated explanations additionally depend on whether the classifier can be approximated appropriately.The estimation framework uses a probabilistic approximation for models without directly calculable gradients.
  • Limitations: Gaussian Process Classification explanations near training-data boundaries can point away from both the data and positive-prediction regions.The vectors become very small there, so the paper states that this does not prevent identifying influential features through absolute partial-derivative values.
  • Assumptions: Explanation vectors inherit a model’s stationarity limitation, and the proposed estimation method itself assumes stationary data.Non-stationary data therefore require appropriate measures such as separating stationary and non-stationary feature-space components.

9. Conclusion

The paper introduces local-gradient explanations for arbitrary nonlinear classifiers, using a probabilistic approximation when gradients are unavailable. Applications to Iris, USPS digits, and drug discovery produce interpretable distinctions, agree with domain knowledge, and reveal local chemical peculiarities.

  • Conclusion: Local explanation vectors characterize how a data point must move to change an arbitrary nonlinear classifier’s predicted label.For models lacking explicit gradient information, the method uses a probabilistic approximate mimic of the learning machine.
  • Conclusion: The framework identifies distinguishing properties among Iris flowers and features separating selected USPS digit 2 and 8 types.These applications are presented as validation examples showing that the method can draw new conclusions from individual classifications.
  • Conclusion: Drug-discovery results fully agree with existing domain knowledge and discover extraordinary steroid behavior as a local peculiarity in chemical space.The domain knowledge was not available to the method during analysis.
  • Conclusion: The authors suggest future use by practitioners and applications in computational biology and psychophysics decision-making experiments.These are identified as two future directions for the framework.

A.1 Illustration of direct local gradients

The paper illustrates local-gradient explanations across different model parameters and scenarios. It examines kernel choice, outliers, and locally nonlinear class boundaries to show how explanations reflect model behavior.

  • A.1 Illustration of direct local gradients: The illustrations examine local-gradient behavior under different kernels, outliers, and locally nonseparable class boundaries.The purpose is to investigate how explanation acuteness varies with model parameters and instructive scenarios.

A.1.1 Choice of kernel function

The appendix shows that local gradients reflect model specification, outliers, local nonlinearities, and approximation quality. Appropriate smoothing and Parzen-window widths can improve practical explanations, while poor choices introduce distortions or zero and wrongly directed gradients.

  • A.1.1 Choice of kernel function: Linear-kernel model misspecification fails to capture nonlinear class separation, and its explanations reflect that failure.The paper contrasts this with more suitable nonlinear kernels.
  • A.1.1 Choice of kernel function: The rational quadratic kernel models nonlinear separation more accurately, producing results similar to the RBF kernel for other parameter values.A non-optimal degree parameter is used in one illustrative subfigure.
  • A.1.1 Choice of kernel function: Local gradients reveal feature interaction at triangle corners and along the hypotenuse while retaining clear trends toward the positive class.Small model bumps create local perturbations in the explanations.
  • A.1.2 Outliers: A single outlier deforms both the model and the explanation, with the affected region depending on kernel width.Because local gradients are derivatives, their sensitivity to nearby outliers exceeds that of the model prediction itself.
  • A.1.2 Outliers: Near an outlier, a local gradient may not represent the truly important features because the underlying model is already wrong there.The paper proposes averaging gradients within an appropriately sized centered sliding window to reduce this disruption.
  • A.1.3 Local non-linearity: All nonlinear class boundaries are followed by local gradients, including feature interactions around a negative circle enclosed by positive examples.Small valleys introduced along a ridge of isolated positive instances are also reflected in the gradients.
  • Parzen window approximation: Parzen-window explanation quality depends on approximation fit and window width: small widths can yield zero inner-region gradients, useful larger widths point toward borders, and excessive widths can point incorrectly.The small-width setting minimizes validation-label mean absolute error but may be less practically useful than a larger width.
Loading 0912.1128v1…