Source-linked AI summary
Right for the Right Reasons: Training Differentiable Models by Constraining their Explanations
Andrew Slavin Ross, Michael C. Hughes, Finale Doshi-Velez
TL;DR
Models can achieve high accuracy while relying on confounds, making them unsafe or brittle when training and test conditions differ. The paper regularizes differentiable models by penalizing selected input gradients, using expert annotations or iterative discovery of alternative decision boundaries. Across multiple datasets, it reports faithful explanations and improved generalization under changed conditions.
Problem
Training data can contain confounds, while existing explanation methods have limited ability to scale across datasets and correct the problems they reveal.
Method
The method regularizes differentiable models by penalizing input gradients, either using expert annotations or iteratively training models with different decision boundaries.
Results
On multiple diverse datasets, the approach produces faithful explanations and models that generalize better when training and test conditions differ.
Takeaways & Limitations
Input gradients provide scalable, optimizable explanations, while alternate-explanation search can expose qualitatively different classifiers for expert inspection.
Takeaways & Limitations
Input gradients can be difficult to interpret when features lack individual meaning and may be small far from decision boundaries, limiting their salience.
Abstract
from arXiv · showhide
Neural networks are among the most accurate supervised learning methods in use today, but their opacity makes them difficult to trust in critical applications, especially when conditions in training differ from those in test. Recent work on explanations for black-box models has produced tools (e.g. LIME) to show the implicit rules behind predictions, which can help us identify when models are right for the wrong reasons. However, these methods do not scale to explaining entire datasets and cannot correct the problems they reveal. We introduce a method for efficiently explaining and regularizing differentiable models by examining and selectively penalizing their input gradients, which provide a normal to the decision boundary. We apply these penalties both based on expert annotation and in an unsupervised fashion that encourages diverse models with qualitatively different decision boundaries for the same classification problem. On multiple datasets, we show our approach generates faithful explanations and models that generalize much better when conditions differ between training and test.
1 Introduction
High-dimensional datasets can support accurate but qualitatively different decision rules, including rules driven by confounds that fail under changed conditions. This motivates explanations that domain experts can inspect and methods that optimize models toward better reasons, while input gradients offer an efficient differentiable alternative to sample-based explanations.
- Motivation: Training data can contain subtle confounds or differ from test data, allowing accurate classifiers whose learned rules fail to generalize.A pneumonia readmission model incorrectly treated asthma as protective because outcomes also reflected differences in initial care quality.
- Motivation: Interpretability methods let experts inspect implicit decision rules, but prior work had limited ability to optimize models for correct explanations.The paper frames this as seeking models that are right for the right reasons, not merely models with correct predictions.
- Contributions: The paper uses input gradients to match expert-identified explanations and to sequentially discover accurate classifiers with qualitatively different decision boundaries when annotations are unavailable.These contributions close the loop between generating explanations and constraining model behavior.
- Input-gradient explanations: Input-gradient explanations are much faster to compute than LIME-style perturbation-and-fitting explanations, which can be computationally prohibitive across an entire dataset.The paper also notes that input gradients are not perfect for salience, especially far from decision boundaries.
- Input-gradient explanations: Input gradients provide local explanations by describing directions normal to a differentiable model’s decision boundary, with large magnitudes marking influential input elements.They can be visualized over the input and are closely tied to the model’s behavior near each example.
2 Our Approach
The approach regularizes differentiable classifiers by constraining input gradients, either using expert-marked irrelevant features or iteratively finding accurate models with different decision boundaries. These mechanisms preserve the right-answer objective while encouraging decision logic that better matches desired reasons.
- 2.1 Constraining explanations in the loss function: Input-gradient explanations are incorporated into a differentiable loss so domain knowledge can constrain a model’s implicit decision rules.The loss combines familiar cross entropy and parameter regularization with a term penalizing large gradients in marked regions.
- 2.1 Constraining explanations in the loss function: The method shrinks gradients in irrelevant regions rather than enforcing large gradients in relevant regions, because relevant gradients may be small far from decision boundaries.The desired constraint is ∇X ŷ near zero at annotation-marked locations.
- 2.1 Constraining explanations in the loss function: The explanation penalty is differentiable with respect to model parameters, requires annotations for only some inputs, and ranges from no effect to global gradient shrinkage.With A = 0, the explanation term disappears; with an all-ones annotation matrix, it encourages small input gradients and can improve generalization.
- 2.1 Constraining explanations in the loss function: The penalty supports example-specific feature constraints without requiring a single global importance notion or domain-specific perturbation logic.This is useful when nonlinear decision boundaries make feature relevance depend on the particular example.
- 2.2 Find-another-explanation: discovering many possible rules without annotations: Without annotations, the method iteratively trains models, explains the training data, and penalizes gradients that were previously largest to produce a spectrum of accurate classifiers.Experts can inspect the resulting models, which have qualitatively different decision boundaries, and select the model with the best reasons.
3 Empirical Evaluation
Across toy and real-world datasets, input gradients matched or surpassed LIME for faithful explanations and enabled explanation-constrained or explanation-diverse models. These models recovered alternative rules, baseline accuracy under confounds, and more generalizable behavior, while retaining known interpretability and scalability limitations.
- Experimental setup: The evaluation used a two-hidden-layer multilayer perceptron across a toy color dataset and three real-world datasets.The network used hidden layers of sizes 50 and 30, ReLU nonlinearities, a softmax output, and an L2 parameter penalty.
- Toy Color Dataset: Both LIME and input gradients identified the same relevant Toy Color pixels, revealing that the model consistently learned the corner rule.The explanations were evaluated on nine perceptron predictions, with gradients highlighting the top 33% largest-magnitude components and LIME selecting six features.
- Toy Color Dataset: Varying explanation penalties and annotated examples moved the Toy Color model between corner and top-middle rules, including a wide hybrid regime.Switching rules required annotating roughly 10^3 examples, although consistently including annotated examples in Adam minibatches reduced this to 50 examples.
- Toy Color Dataset: The find-another-explanation procedure discovered both Toy Color rules without annotations, while a third iteration produced noisy explanations and low accuracy.Because only two rules achieved high test accuracy, preventing use of both caused performance no better than random guessing.
- Real-world Datasets: On real-world datasets, gradients exposed decoy reliance and outperformed LIME on Iris-Cancer, while LIME produced overly sparse explanations for long documents.On Decoy MNIST, both methods identified the misleading swatch; on Iris-Cancer, LIME identified only one important breast-cancer feature despite the importance of Iris features.
- Interpretation and limitations: Input gradients offer scalability advantages over sample-based perturbation methods but can be difficult to interpret for non-meaningful features and may miss salience far from decision boundaries.Gradient explanations require roughly one model evaluation, whereas perturbation-based methods repeatedly fit local surrogates; gradients can also be uniformly small away from the boundary.
- Real-world Datasets: Penalizing known-incorrect explanations recovered baseline test accuracy, and unsupervised explanation discovery produced more generalizable models without predefined annotations.On Iris-Cancer, later iterations had lower train accuracy but higher test accuracy; on Decoy MNIST, one iteration recovered baseline accuracy.
4 Conclusions and Future Work
The paper concludes that input-gradient penalties support generalizable decision logic and scalable explanation tools, while suggesting extensions to robustness, fairness, and transparency.
- Training with input-gradient penalties can learn generalizable decision logic despite inherent dataset ambiguities.
- The find-another-explanation method presents qualitatively different classifiers when detailed explanation annotations are unavailable.
- Input gradients are consistent with LIME, faster to compute, and sometimes more faithful, especially for continuous inputs.
- Future applications include robustness to adversarial examples and explanation constraints for fairness, accountability, and transparency.
A Cross-Validation
The appendix recommends selecting explanation regularization strength by balancing cross-entropy and right-reasons loss terms rather than relying solely on validation accuracy.
- Increase λ1 until the cross-entropy and right-reasons terms have roughly equal magnitudes when shared confounds make validation accuracy unreliable.
- The highest-accuracy regime occurs when the initial cross-entropy and λ1 loss terms have similar magnitudes.
- Being an order of magnitude off between the terms does not significantly affect accuracy.
B Learning with Less Data
Explanation regularization can reduce the number of labeled examples needed, although penalizing the wrong features can initially reduce accuracy.
- The experiment evaluates whether explanations can reduce data requirements on Toy Color using four variants of A.
- 95% accuracy requires fewer than 100 examples with the Pro-Rule 1 mask, versus almost 10000 examples without explanation regularization.
- Penalizing top-middle pixels or all pixels except the top-middle consistently improves accuracy relative to data.
- Penalizing corner pixels reduces accuracy until a threshold N, possibly because corner patterns have fewer combinations than top-middle patterns.
C Longer 20 Newsgroups Examples
For longer 20 Newsgroups examples, gradients and LIME agree on feature direction, while gradients can expose a fuller set of influential words.
- LIME and input gradients never disagree about whether highlighted words support or oppose the predicted label.
- Input gradients may provide a fuller picture for long documents because LIME is limited in its selected features and samples.