Source-linked AI summary
Representer Point Selection for Explaining Deep Neural Networks
Chih-Kuan Yeh, Joon Sik Kim, Ian E. H. Yen, Pradeep Ravikumar
TL;DR
Deep neural networks make individual predictions difficult to understand, even for machine learning experts. The paper selects signed representer points by decomposing predictions into weighted training-point activations, and reports richer insight with greater scalability than influence functions.
Problem
Deep neural networks lack clear explanations for why they make particular predictions, creating a need to improve understanding of their behavior.
Method
The method applies a representer theorem to decompose pre-activation predictions into training-point activations weighted by representer values, using positive and negative values to select explanatory points.
Results
Representer points provide richer insight than influential training points, while the method significantly outperforms influence functions computationally and can support real-time explanations.
Takeaways & Limitations
Signed representer points help users understand both examples supporting a prediction and examples contributing to why a model does not make a prediction.
Takeaways & Limitations
The paper identifies extension beyond image datasets and different neural-network types, including NLP datasets with recurrent neural networks, as future work.
Abstract
from arXiv · showhide
We propose to explain the predictions of a deep neural network, by pointing to the set of what we call representer points in the training set, for a given test point prediction. Specifically, we show that we can decompose the pre-activation prediction of a neural network into a linear combination of activations of training points, with the weights corresponding to what we call representer values, which thus capture the importance of that training point on the learned parameters of the network. But it provides a deeper understanding of the network than simply training point influence: with positive representer values corresponding to excitatory training points, and negative values corresponding to inhibitory points, which as we show provides considerably more insight. Our method is also much more scalable, allowing for real-time feedback in a manner not feasible with influence functions.
1 Introduction
The paper addresses the difficulty of understanding why deep neural networks make particular predictions by selecting influential training examples with signed explanatory roles.
- Deep neural networks remain difficult to understand even for machine learning experts, motivating explanations of individual predictions.
- The method decomposes pre-activation predictions into training-point activations weighted by representer values.
- Positive representer values identify excitatory examples, while negative values identify inhibitory examples affecting a test-point prediction.
- Inhibitory examples can reveal why a model avoids a prediction, such as mislabeled animal images contributing to an antelope-as-deer error.
- The paper demonstrates representer-point explanations through theoretical and empirical investigations.
2 Related Work
Prior explanation methods focus either on important input features or on influential training samples, while representer-point explanations build on representer-theorem ideas from machine learning.
- Feature-based methods explain predictions using important input features, including local linear models and locally sufficient conditions.
- Sample-based methods identify training examples related to a model’s prediction, including prototypes, criticisms, and influential samples.
- Representer theorems traditionally express RKHS loss minimizers as linear combinations of kernel evaluations at training points under certain conditions.
3 Representer Point Framework
The framework decomposes a neural network’s pre-activation prediction into training-point contributions, whose signed representer values identify excitatory and inhibitory examples. It applies to regularized models and can also generate explanations for an existing pretrained model, with CIFAR-10 sanity checks showing near-perfect approximation.
- Representer decomposition: The network’s pre-activation prediction is expressed as a linear combination of training-point activations, weighted by representer values.These values measure each training point’s importance to the learned parameters and prediction.
- Signed explanations: Positive representer values identify excitatory examples, whereas negative values identify inhibitory points that suppress a class activation.The contribution is strongest when the representer value and feature similarity are both large in magnitude.
- Signed explanations: The signed points explain both why a network favors a prediction and why it does not, providing richer insight than approaches that only identify influential training samples.The paper specifically highlights inhibitory points as a source of information about competing or suppressed predictions.
- Regularized model: The theorem applies to neural networks with a final linear matrix multiplication before activation and requires a fixed positive L2 regularizer for its decomposition.The framework is stated for stationary points of the regularized optimization problem.
- Pretrained models: For an already trained model, the method adds L2 regularization while constraining the new model’s outputs to remain close to the given model’s outputs.This avoids relying on a retrained model that may converge to a different stationary point and behave differently.
- Sanity check: The CIFAR-10 VGG-16 sanity check took less than a minute, and the Pearson correlation between actual and approximated softmax outputs was almost 1 for both train and test data.The approximation was computed from the representer-point decomposition for multiple training and test points.
4 Experiments
Experiments evaluate representer points for dataset debugging, example retrieval, sensitivity decomposition, computational cost, and numerical stability. Across these settings, the method matches influence functions in recovering mislabeled data while providing clearer examples, interpretable sensitivity contributions, and more scalable computation.
- 4.1 Dataset Debugging: The method recovers the same amount of mislabeled training data as influence functions while achieving higher test accuracy, and both outperform random selection.The experiment flips 40% of CIFAR-10 labels before ranking training points for inspection and correction.
- 4.2 Excitatory (Positive) and Inhibitory (Negative) Examples: On AwA, representer points provide positive examples that support a prediction and negative examples that resemble the test image but have different labels.For grizzly-bear images, both methods return similar positive and negative examples; for a rhino image, influence functions fail to recover useful points while the representer method succeeds.
- 4.3 Understanding Misclassified Examples: Negative representer points reveal a recurring source of antelope-to-deer errors: four mislabeled or visually conflicting antelope-containing training images appear among the most inhibitory examples.All 12 antelopes misclassified as deer shared these four top inhibitory training images.
- 4.4 Sensitivity Map Decomposition: Sensitivity maps decompose into weighted sums of training-point sensitivity maps, with object-focused training images producing more focused contributions than background-heavy images.Zebra maps emphasize facial features, while moose maps emphasize bodily features when those features are more distinctive than the background.
- 4.5 Computational Cost and Numerical Instabilities: Representer values significantly outperform influence functions in computation time and can support real-time explanations, despite a one-time optimization cost.The influence-function Hessian-vector-product computation alone took longer than the representer method’s combined computation.
- 4.5 Computational Cost and Numerical Instabilities: On 1,000 CIFAR-10 test points, over 30% had all-zero influence values, whereas representer values remained non-trivial and more numerically stable across test points.More than 200 influence-function cases fell in the range [10^-40, 10^-28], which may create numerical instabilities.
5 Conclusion and Discussion
The paper introduces an efficient representer-point method for identifying training examples influential to neural-network predictions and demonstrates its advantages on large-scale models and image datasets. It also identifies data poisoning and extension to NLP with recurrent networks as future directions.
- A modified representer theorem decomposes prediction activations into sums of representer values for most deep neural networks.
- The optimization procedure is tractable and efficient, particularly compared with influence functions.
- Experiments on large-scale models and image datasets demonstrate the method’s advantages and provide insights into model behavior.
- Future work includes using representer values for data poisoning and extending the method to NLP datasets with recurrent neural networks.
A Proof of Proposition 3.1
The proposition establishes that the chosen loss is suitable for the corresponding activation by showing that optimization reproduces the given network output under softmax and ReLU settings.
- Softmax: For softmax, first-order optimality yields equality between the predicted softmax outputs under learned and given parameters.
- Softmax: When f_i is nonzero, the proof reduces the softmax condition using the chain rule.
- ReLU: The proof analyzes each output coordinate separately because the j-th row of Θ_1 relates only to Φ_j(x_i, Θ).
- ReLU: For nonpositive target activation, the LReLU minimum occurs at a nonpositive prediction and reproduces the zero ReLU output.
- ReLU: For positive target activation, the LReLU minimum is reached when the prediction matches the given activation, making LReLU suitable for ReLU.
B Relationship with the Influence Function
The paper compares representer values with influence functions, Euclidean distance, and their behavior on toy and CIFAR-10 data. Representer values remain informative where influence values vanish and identify class-relevant helpful or harmful examples.
- Influence functions can approach zero in flat-loss regions, whereas activation-based representer values are comparatively larger in general.
- In the 2-D toy example, influence functions assign exactly zero to all training points, while representer values identify positive and negative examples near the decision boundary.
- Figure 9 compares Euclidean distance, influence function, and representer value for selecting CIFAR-10 training points similar to a test point.
- Euclidean distance does not reflect training-point class, while representer and influence values indicate which class each point belongs to.
- Influence and representer values agree in identifying horse images as harmful and dog images as helpful in the reported CIFAR-10 example.
C More Examples of Positive/Negative Reperesenter Points
Additional AwA examples show that positive representer points match the test image’s class, while negative points can resemble the test image but belong to different classes.
- Positive representer points have the same class label as the test image and high resemblance.
- Negative representer points can resemble the test image despite having different class labels.
- Figure 10 presents test images followed by three positive and three negative representer points for each image.
D Representer Points of LSTM on NLP Data
A preliminary LSTM experiment on IMDB sentiment reviews examines positive and negative representer points for a test review predicted as negative.
- The LSTM was evaluated on IMDB movie reviews for binary positive-versus-negative sentiment classification.The dataset consists of movie reviews, and the task is to identify whether each review expresses positive or negative sentiment.
- 87.5% accuracy was achieved by the pretrained LSTM model.
- For a test review predicted negative, the top-1 positive and negative representer points both contained negative connotations.
- The negative connotation in the top-1 negative representer was attributed to a movie character rather than the reviewer.
- Table 2 presents the top-1 positive and negative representer points, with <> marking words outside the 5,000-word vocabulary.