Source-linked AI summary
Learning to Explain: An Information-Theoretic Perspective on Model Interpretation
Jianbo Chen, Le Song, Martin J. Wainwright, Michael I. Jordan
TL;DR
Interpreting complex machine-learning models requires explanations that identify important features for individual examples. The paper learns an instancewise feature selector by maximizing mutual information with the model response, and reports effectiveness across synthetic and real datasets using quantitative and human evaluations.
Problem
Complex models can improve prediction accuracy while compromising interpretability, creating a need for instance-specific explanations of their predictions.
Method
The method learns a model-agnostic instancewise feature selector that extracts a subset of features maximizing mutual information with the model response.
Results
The method is effective across synthetic and real datasets under both quantitative metrics and human evaluation.
Takeaways & Limitations
Selected features can serve as instance-specific explanations for model behavior, including key words or sentences in text predictions.
Takeaways & Limitations
The framework assumes access to the model’s conditional response distribution, and difficult estimation of Pm(Y | x_S) may require restricting the explainer family to control overfitting.
Abstract
from arXiv · showhide
We introduce instancewise feature selection as a methodology for model interpretation. Our method is based on learning a function to extract a subset of features that are most informative for each given example. This feature selector is trained to maximize the mutual information between selected features and the response variable, where the conditional distribution of the response variable given the input is the model to be explained. We develop an efficient variational approximation to the mutual information, and show the effectiveness of our method on a variety of synthetic and real data sets using both quantitative metrics and human evaluation.
1. Introduction
The paper motivates instancewise feature selection for interpreting complex models and introduces an information-based, learned, model-agnostic approach. It evaluates the method on synthetic and real datasets with quantitative metrics and human evaluation.
- Motivation: Interpretability matters in high-stakes applications, while complex models can improve prediction accuracy at the expense of ease of interpretation.The paper names medicine, financial markets, and criminal justice as example application areas.
- Instancewise feature selection: Instancewise feature selection assigns feature importance locally, allowing the features explaining a model prediction to vary across instances.These importance scores indicate which features are key to a model’s prediction for a specific instance.
- Instancewise feature selection: Unlike global feature selection, instancewise feature selection measures feature importance separately for each instance labeled by the model.Global feature selection instead selects features useful across the entire labeled dataset.
- Approach: The paper approaches instancewise feature selection through mutual information, providing a conceptually different perspective from existing interpretation methods.The cited prior work includes gradient-based saliency maps and locally additive model approximations.
- Contributions and evaluation: The proposed framework learns an efficient, model-agnostic instancewise feature selector and evaluates it on synthetic and real datasets using quantitative metrics and human evaluation.The contributions include an information-based framework and a learning-based method designed for efficiency and model agnosticism.
2. A framework
The framework selects instance-specific feature subsets by maximizing mutual information between selected features and the model response. It gives the selection an information-theoretic interpretation, while noting practical constraints when the model conditional distribution is difficult to estimate.
- Assumptions: The framework assumes access to a model’s conditional response distribution Pm(· | x), with the current discussion restricted to classification models.The model response is represented as Y given an input realization X = x ∈ R^d.
- Mutual information: Mutual information measures dependence between random vectors by quantifying how knowledge of one reduces uncertainty about the other.It is defined through the Kullback–Leibler divergence between the joint distribution and the product of marginals.
- Constructing explanations: An explainer maps each input to a possibly randomized distribution over subsets of k features, whose selected sub-vector is used to explain the model response.The user chooses k, although it may also be tuned as a hyper-parameter.
- Constructing explanations: The optimization seeks a selection rule that maximizes mutual information between the model response and the selected features.The selected feature vector is represented as X_S, obtained from the input according to the explainer.
- Information-theoretic interpretation: A global optimum minimizes the expected encoded-message length for Pm(Y | x) when coding with Pm(Y | x_S), and any global optimum equals the characterized explainer almost surely.This interpretation connects the optimization criterion to the predictive information retained by the selected sub-vector.
- Practical limitations: The theoretical global optimum requires a sufficiently large explanation family, while unknown or expensive Pm(Y | x_S) may require restricting the explainer family to control overfitting.The restriction is presented as a practical response to difficulty estimating the conditional distribution accurately.
3. Proposed method
The method makes instancewise feature selection tractable by variationally approximating mutual information and continuously relaxing subset sampling. A learned explainer assigns input-dependent feature weights, while a neural network approximates the model conditional distribution and enables efficient deterministic explanations.
- Variational formulation: A variational lower bound replaces direct mutual-information optimization, jointly optimizing the explainer and an approximating conditional-distribution family.The approximation is needed because expectations under a generic model’s conditional distribution are impossible to compute directly.
- Variational formulation: A single neural network parametrizes conditional distributions for every k-sized feature subset, with unselected input entries replaced by zeros.For discrete features, their embedding vectors are likewise zeroed when the corresponding feature is unselected.
- Continuous subset sampling: The Gumbel-softmax-based relaxation is used instead of REINFORCE-type methods because it empirically has lower variance and encourages discreteness.The Concrete distribution provides a continuous differentiable approximation to categorical sampling.
- Continuous subset sampling: The explainer maps each input to feature importance scores and uses Concrete random vectors with an elementwise maximum to approximate k-hot subset sampling during training.The continuous relaxation supports efficient optimization while approximating sampling distinct features.
- Optimization and explaining stage: During optimization, stochastic gradients jointly update the variational network and explainer from model-generated class distributions and auxiliary random variables.At explanation time, the learned explainer ranks its input-dependent weights and selects the k largest features.
- Optimization and explaining stage: A single forward pass through the explainer produces each sample’s explanation, unlike LIME or Kernel SHAP, which require thousands of original-model evaluations per sample.This efficiency applies specifically during the explaining stage.
4. Experiments
Experiments evaluate L2X on synthetic and real data using feature-ranking, efficiency, post-hoc accuracy, and human-evaluation measures. L2X performs strongly across nonlinear synthetic settings, is efficient relative to repeated-evaluation methods, and produces useful explanations for text and image models.
- Experimental setup: L2X is compared with Taylor, Saliency, DeepLIFT, SHAP, and LIME for instancewise feature selection on trained neural networks.The experiments use synthetic and real data, with L2X using a neural-network explainer.
- Synthetic data: L2X outperforms all other methods on nonlinear additive and feature-switching data, while DeepLIFT, SHAP, and L2X perform best on XOR.On orange skin, all methods are near optimal; L2X and LIME are the most stable across samples.
- Synthetic data: L2X is the most efficient during explanation because it requires only a forward pass of the subset sampler.SHAP and LIME require multiple model evaluations, while DeepLIFT, Taylor, and Saliency require a backward pass.
- Synthetic data: As the number of samples to explain increases, L2X training becomes a smaller share of total time and its relative efficiency increases.Figure 2 reports clock time on a log scale for explaining 10,000 samples.
- Explaining CNNs with key words: Over 90% of model predictions align when L2X selects only ten words, compared with 84.4% alignment for human judgments from those words.Human alignment using the original review is 83.3%.
- Explaining hierarchical LSTM: With one selected sentence, L2X reaches 84.4% post-hoc accuracy and human judgments align with model predictions for 77.4% of reviews.The hierarchical LSTM experiment uses the same evaluation metrics as the word-based CNN experiment.
- MNIST: Selecting four patches out of 49 yields 95.8% post-hoc alignment on MNIST, with explanations capturing informative patterns distinguishing 3 and 8.Figure 4 visualizes selected patches on randomly chosen validation examples.
5. Conclusion
The paper proposes instancewise feature selection through mutual information and introduces L2X as a variational method with Gumbel-softmax subset sampling. It reports efficiency and feature-selection capacity on synthetic and real data.
- L2X seeks a variational approximation to mutual information for instancewise feature selection.It uses a Gumbel-softmax relaxation of discrete subset sampling during training.
- The framework and L2X method were evaluated on both synthetic and real data sets.
- L2X is presented as the first method to realize real-time interpretation of a black-box model.
A. Proof of Theorem 1
The proof shows that an optimal explanation must select features from the informative subset S∗(X). Any positive-probability deviation would yield strictly lower mutual information, contradicting optimality.
- An explanation is represented as a conditional distribution of feature subsets given the input.The proof compares this distribution with one supported on the informative subset S∗(X).
- When S∗(X) contains multiple features, it is represented by distributions assigning probability only to elements of that set.
- If the optimal explanation assigns nonzero probability outside S∗(X) on a set M with nonzero probability, the proof derives a contradiction.
- The contradiction follows because the deviating explanation has lower mutual information than the explanation supported on S∗.