Source-linked AI summary
On the (In)fidelity and Sensitivity for Explanations
Chih-Kuan Yeh, Cheng-Yu Hsieh, Arun Sai Suggala, David I. Inouye, Pradeep Ravikumar
TL;DR
The paper asks how to evaluate and improve saliency explanations for black-box models using objective infidelity and sensitivity measures. It defines perturbation-based infidelity, analyzes optimal explanations, and studies smoothing as a modification. The authors report that varying perturbations yields new explanations and that sensitivity can be reduced while also improving infidelity.
Problem
Objective evaluation of saliency explanations for complex black-box models lacks a unified treatment of infidelity and sensitivity and their optimal explanations.
Method
The paper defines infidelity over meaningful perturbation distributions, analyzes optimal explanations, and proposes perturbation choices and smoothing-based modifications.
Results
Varying perturbations recovers existing explanations and produces new ones, while SmoothGrad improves both sensitivity and infidelity across base explanations and datasets.
Takeaways & Limitations
Sensitivity need not trade off with infidelity: appropriately smoothing an explanation can improve both measures simultaneously.
Takeaways & Limitations
The analysis assumes invertibility conditions for some optimality results and uses the ℓ2 norm unless otherwise specified in experiments.
Abstract
from arXiv · showhide
We consider objective evaluation measures of saliency explanations for complex black-box machine learning models. We propose simple robust variants of two notions that have been considered in recent literature: (in)fidelity, and sensitivity. We analyze optimal explanations with respect to both these measures, and while the optimal explanation for sensitivity is a vacuous constant explanation, the optimal explanation for infidelity is a novel combination of two popular explanation methods. By varying the perturbation distribution that defines infidelity, we obtain novel explanations by optimizing infidelity, which we show to out-perform existing explanations in both quantitative and qualitative measurements. Another salient question given these measures is how to modify any given explanation to have better values with respect to these measures. We propose a simple modification based on lowering sensitivity, and moreover show that when done appropriately, we could simultaneously improve both sensitivity as well as fidelity.
1 Introduction
The paper evaluates saliency explanations for black-box models using objective measures, focusing on a generalized infidelity measure based on perturbations and output changes.
- Motivation: The paper studies how to objectively evaluate feature-attribution explanations for complex black-box machine-learning models.Explanations may assign importance weights to features or identify a salient subset, such as pixels in an image.
- Infidelity: Infidelity measures the discrepancy between an explanation’s predicted output change and the model’s actual output change under input perturbations.The perturbation distribution can represent baselines, subsets, multiple references, or noisy baselines.
2 Objective Measure: Explanation Infidelity
The paper defines infidelity as an expected squared discrepancy under meaningful perturbations, then uses its perturbation distribution to recover existing explanations and construct new ones.
- Defining the infidelity measure: Infidelity replaces correlation-based evaluation with expected mean squared error between explanation-predicted and model-observed output changes.The framework allows random perturbations rather than only setting features to a fixed baseline.
- Defining the infidelity measure: Perturbations can use baselines, feature subsets, noisy baselines, or multiple baselines, allowing infidelity to represent different explanation settings.The perturbation distribution determines which changes in the model are treated as meaningful.
- Explanations with least Infidelity: The infidelity-optimal explanation has a smoothing form related to SmoothGrad applied to Integrated Gradients or another completeness-satisfying explanation.For deterministic perturbations, optimality is equivalent to satisfying the Completeness Axiom.
- Explanations with least Infidelity: Varying perturbations recovers existing explanations, including Shapley values, gradients in the small-perturbation limit, and occlusion explanations.The resulting optimal explanation depends directly on the selected perturbation scheme.
- Some Novel Explanations with New Perturbations: The paper proposes noisy-baseline explanations that robustify completeness and square-removal explanations that use uniformly sampled image patches.Square patches reduce computational complexity and better capture spatial relationships than arbitrary pixel subsets.
- Local and global explanations: Local explanations describe sensitivity to feature changes, whereas global explanations directly estimate function-value changes and incorporate input-minus-baseline scaling.This distinction affects how attribution scores are interpreted and evaluated.
3 Objective Measure: Explanation Sensitivity
The paper defines max-sensitivity as a robust measure of how much an explanation changes under small input perturbations. It also highlights that minimizing sensitivity alone yields a vacuous constant explanation, motivating sensitivity reduction that preserves fidelity.
- Sensitivity measures: Gradient-based sensitivity quantifies how an explanation changes when the input varies infinitesimally.A norm of the sensitivity matrix provides a scalar summary, while a locally uniform bound offers a more robust variant.
- Max-sensitivity: Max-sensitivity measures the maximum change in an explanation within a radius-r neighborhood of the input.It is defined as the largest norm difference between Φ(f, y) and Φ(f, x) for ||y−x||≤r.
- Robustness: Max-sensitivity is robustly estimable by Monte Carlo sampling and remains finite when explanation scores are bounded.This can hold even when local Lipschitz continuity is unbounded for ReLU-based gradient explanations.
- Limitations of sensitivity: The minimum-sensitivity explanation is a constant output across all test inputs, which may be nonsensical and uninformative.This exposes sensitivity as only a partial measure of explanation quality.
- Responsible reduction: The paper proposes reducing sensitivity responsibly by ensuring that infidelity decreases at the same time.This approach addresses the concern that smoothing or stabilization could otherwise damage fidelity to the predictor.
4 Reducing Sensitivity and Infidelity by Smoothing Explanations
The paper smooths explanations by averaging them over nearby inputs, reducing sensitivity through local averaging. Under stated conditions, this smoothing can also reduce infidelity, yielding explanations that are both less sensitive and more faithful.
- Smoothing explanations: A smoothed explanation averages an explanation functional over the input domain using a chosen kernel.With a Gaussian kernel, this construction becomes SmoothGrad.
- Sensitivity reduction: Smoothing bounds max-sensitivity by the kernel-weighted average sensitivity of the original explanation around the input.Sensitivity concentrated in only some directions can therefore be substantially reduced by averaging.
- Infidelity reduction: Under certain assumptions, the infidelity of the smoothed explanation is lower than that of the original explanation.The paper introduces intermediate terms to state the condition relating the two infidelity values.
- Empirical implication: The kernel-weighted average infidelity is usually very close to the original infidelity, while smoothing can improve both sensitivity and faithfulness.The paper reports this joint improvement as validated in experiments.
- Alternative approach: Adversarial training is presented as another route to less sensitive and more faithful gradient explanations.This result is reported in the appendix rather than as the main smoothing procedure.
5 Experiments
Experiments compare sensitivity and infidelity for local and global explanations across image datasets, combining quantitative measures with visual, human, and sanity-check evaluations. The proposed Noisy Baseline, Square, and Smooth-Grad variants show improved behavior under the tested settings.
- Explanation Sensitivity and Infidelity: Experiments evaluate local explanations on MNIST and global explanations on MNIST, CIFAR-10, and ImageNet using sensitivity and infidelity.Infidelity uses noisy-baseline perturbations for local explanations and square removal for global explanations, estimated by Monte Carlo sampling.
- Explanation Sensitivity and Infidelity: Noisy Baseline and Square achieve the lowest infidelity for their corresponding perturbations, as expected from explicitly optimizing those measures.Noisy Baseline is used for local explanations and Square Removal for global explanations.
- Explanation Sensitivity and Infidelity: Smooth-Grad improves both sensitivity and infidelity for all tested base explanations across datasets.The experiments report that modest smoothing improves infidelity rather than increasing it.
- Visualization: Infidelity-optimizing explanations produce cleaner saliency maps that better highlight classified objects, while Noisy Baseline also reveals regions that could increase prediction scores when brightened.Square uniquely highlights the whole bannister in one ImageNet example, and Noisy Baseline clearly shows MNIST digits and score-increasing regions.
- Human Evaluation: The human evaluation tests whether infidelity aligns with judgments when the model’s relevant feature is approximated using images and captions placed in separate blocks.The constructed bird-versus-frog dataset varies image-caption agreement through image noise.
- Sanity Check: The sanity check compares explanations for pretrained and partially randomized ResNet-50 models using visualizations and average rank correlation.Explanations without absolute values pass the check; among explanations with absolute values, Square has the lowest rank correlation and most distinct visualizations.
6 Related Work
Related work spans attribution, perturbation, gradient, game-theoretic, sensitivity-based, and objective evaluation methods. The paper positions its contribution among approaches seeking reliable, quantitatively evaluated explanations.
- Explanation Methods: Attribution-based explanations assign predictions to input features and are broadly applicable and easy to understand.Related approaches include sample-based, concept-based, perturbation-based, gradient-based, and cooperative-game-theoretic explanations.
- Objective Evaluation: Prior work evaluates explanations through continuity, perturbation curves, object localization, input invariance, and retraining after feature removal.These methods place explanation quality on more objective footing using different behavioral or task-based criteria.
- Sensitivity-Based Methods: Sensitivity-focused methods include distractor removal, SmoothGrad’s averaging over Gaussian-noise perturbations, and locally linear surrogate models.These approaches aim to reduce noise or approximate complex model behavior.
- Reliability: Sanity-check studies report that some saliency methods remain insensitive to randomized parameters, while other work shows saliency can change substantially under perceptually indistinguishable image modifications.These findings motivate reliability checks for attribution explanations.
7 Conclusion
The paper evaluates explanation quality through infidelity and sensitivity, analyzes gradient explanations, and shows that smoothing or model regularity can improve both measures.
- The paper proposes infidelity and sensitivity as objective evaluation metrics for machine-learning explanations.
- Existing explanations can be unified as optimizing infidelity under different perturbation distributions, whose optimal solution combines two explanation methods.
- Adversarial training can produce models with smooth gradients, yielding gradient explanations that are less sensitive and more faithful.
- For gradient explanations, lower Hessian upper bounds improve both infidelity and sensitivity.
A.3 Optimizing Explanation Infidelity and Gradient Sensitivity by Adversarial Training
This section explains how adversarial training can improve gradient-based explanations by encouraging models with smaller, smoother gradients and Hessians.
- Adversarial training minimizes expected adversarial risk over perturbations bounded by ϵ using a classification loss such as logistic loss.
- The resulting smooth gradients make gradient-based explanations more robust to input perturbations.
- The adversarial-risk upper bound penalizes both gradient and Hessian norms, encouraging classifiers with smooth gradients.
- The paper uses Projected Gradient Descent to train an adversarially robust network and discusses gradient-norm and weight regularization as alternatives.
- A toy function demonstrates that gradient explanations can switch between (1, 0) and (0, 1) across nearby regions, making them highly sensitive.
B.1 Detailed Experiment Settings
The experiments compare several attribution methods and smoothing variants on image-classification models using specified perturbation, norm, and baseline settings.
- The experiments use a CNN exceeding 99 percent test accuracy on MNIST and a wide-resnet reaching 94 percent on cifar-10.
- They compare Grad, integrated gradients, Guided Back-Propagation, and SmoothGrad applied to the evaluated explanations.
- Sensitivity is estimated with 50 Monte-Carlo samples, using the L8 norm and maximum perturbation r = 0.1 across datasets.
- The baseline image is set to 0 for all explanations.
- For Imagenet, SHAP is not reported because KernelSHAP does not scale well to high-dimensional images and produces random noise even with 2 × 2 superpixels.
B.2 Implementation Tricks
Implementation details address matrix invertibility, perturbation evaluation, and smoothing experiments, while robust training lowers both sensitivity and infidelity.
- When the input contains zero-valued features, the matrix used for the optimal explanation can be non-invertible; the implementation uses alternative forms or regularization to stabilize computation.
- The implementation derives an analytical solution and chooses perturbation variables z so the relevant matrix becomes invertible.
- Under square perturbations, binary perturbations are used to evaluate infidelity for global explanations.
- As SmoothGrad’s radius increases from 0.1 to 2.0 on MNIST, infidelity first decreases and then gradually increases, while sensitivity decreases.
- An adversarially trained MNIST model yields lower sensitivity and infidelity than a regularly trained model for all evaluated explanations.
C A connection between Explanation and Model Sensitivity and Infidelity
The section introduces robust infidelity as a maximum-infidelity measure under small perturbations and connects it theoretically to explanation and model sensitivity. It also shows that robust infidelity preserves the infidelity-optimal explanation while suggesting sensitivity reduction as a route to improving both measures.
- Robust fidelity measures the maximum infidelity produced by perturbations u with norm at most r.The construction evaluates explanation error after shifting the input by a bounded perturbation.
- The explanation minimizing robust infidelity is the same as the explanation minimizing ordinary infidelity.Robust infidelity therefore changes the measurement's robustness without changing the infidelity-optimal explanation.
- Theorem C.1 lower-bounds robust infidelity by A(x) − B(x) − B(x − I), where A measures explanation variation and B measures model variation.The bound is stated as RINFD(x) ≥ A(x) − B(x) − B(x − I).
- When explanation sensitivity exceeds model sensitivity, the lower bound reflects their difference and indicates that reducing explanation sensitivity may also improve explanation infidelity.A(x) can be approximated using SENSMAX, while B(x) represents function sensitivity.
- Special perturbation choices recover familiar explanations, including gradient-based explanations, occlusion-1, and Shapley values.Infinitesimal coordinate perturbations yield ∇_x f(x), coordinate-removal perturbations yield occlusion-1, and binary perturbations connect to Shapley values.
D.7 Proof of Theorem 4.2
The proof of Theorem 4.2 rewrites the relevant sensitivity and infidelity expressions using auxiliary quantities and applies inequalities to derive a bound. The accompanying visual examples compare explanation behavior across methods, models, and evaluation settings.
- Proof of Theorem 4.2: The proof defines T1 as the explanation term and T2 as the model output-difference term before reordering the expressions.These substitutions make the comparison between explanation and model terms explicit in the proof.
- Proof of Theorem 4.2: Cauchy–Schwarz and the preceding inequalities are used to obtain the theorem's bound under the stated C1 condition.The proof then substitutes the intermediate result into the main theorem and notes that the relevant inequality holds for any u.
- Qualitative evaluations: On ImageNet examples, square removal is described as focusing more consistently on dark objects than gradient-based saliency methods.The caption interprets this consistency as greater faithfulness to the model.
- Qualitative evaluations: On MNIST examples, gradient-based local saliency is described as noisier than NB, while NB includes predictive regions such as digit tails and upper regions.The comparison concerns both visual noise and inclusion of regions supporting the prediction.
- Qualitative evaluations: In sanity checks, Square focuses on random objects for randomized networks, whereas gradient-based maps focus more on bright objects.The caption states that Square consequently produces more diversity between original and randomized-model explanations.
- Qualitative evaluations: Human-evaluation examples compare four saliency methods across models whose approximated ground truth is image-based or text-based.The caption reports that correctly identifying the model-reliant ground-truth block is difficult from many saliency maps.