Source-linked AI summary
Interpretable Explanations of Black Boxes by Meaningful Perturbation
Ruth Fong, Andrea Vedaldi
TL;DR
Opaque black-box predictors remain difficult to explain, and many saliency methods are heuristic or tied to model architecture. The paper formalizes explanations as learnable meta-predictors and uses explicit image perturbations to identify regions affecting classifier scores. It presents a model-agnostic, testable saliency framework and applies it to interpretability, minimal deletions, adversarial defense, and neural-network artifacts.
Problem
Existing explanations for opaque black-box predictors are often heuristic, leaving their meaning unclear.
Method
The paper learns explanations as meta-predictors and finds salient image regions by learning deletion masks that suppress a target class score under explicit perturbations.
Results
The framework provides model-agnostic and testable saliency explanations and supports applications to interpretability, minimal deletions, adversarial defense, and network artifacts.
Takeaways & Limitations
Direct image edits give saliency explanations an interpretable and verifiable meaning tied to changes in the classifier’s output score.
Takeaways & Limitations
The single-perturbation approach can be attracted to neural-network artifacts, especially under noise and constant perturbations.
Abstract
from arXiv · showhide
As machine learning algorithms are increasingly applied to high impact yet high risk tasks, such as medical diagnosis or autonomous driving, it is critical that researchers can explain how such algorithms arrived at their predictions. In recent years, a number of image saliency methods have been developed to summarize where highly complex neural networks "look" in an image for evidence for their predictions. However, these techniques are limited by their heuristic nature and architectural constraints. In this paper, we make two main contributions: First, we propose a general framework for learning different kinds of explanations for any black box algorithm. Second, we specialise the framework to find the part of an image most responsible for a classifier decision. Unlike previous works, our method is model-agnostic and testable because it is grounded in explicit and interpretable image perturbations.
1. Introduction
The paper frames explanation as an open problem for opaque black-box predictors because existing visualizations are largely heuristic. It proposes formal, interpretable rules and a general framework for learning explanations, including image saliency based on perturbations.
- Existing explanations for black-box predictors are often intuitive visualizations whose meaning remains unclear because they are primarily heuristic.
- The paper seeks principles and methods for explaining any black-box function by describing its learned input-output properties with interpretable rules.
- Its framework treats explanations as meta-predictors and supports automatically discovering explanatory rules for a classifier.
- For saliency, the paper learns which image regions affect a target score by perturbing the image and measuring the resulting output change.
2. Related work
Prior saliency methods use gradients, activations, or backpropagation heuristics, often requiring neural-network structure or intermediate-layer access. Related perturbation-based approaches instead edit inputs, but differ in convergence speed, resolution, interpretability, and model dependence.
- Gradient-based, activation-based, and relevance-propagation methods produce saliency visualizations through different backpropagation or activation heuristics.
- Most existing techniques are limited to neural networks and may require architectural modifications or access to intermediate layers.
- LIME perturbs images in a neighborhood around an input but uses 5000 iterations and coarse fixed super-pixel heatmaps, compared with the paper’s 300 iterations.
- Feedback networks learn internal ReLU gating masks to maximize class scores, whereas this paper’s masks directly edit images and remain model-agnostic.
3. Explaining black boxes with meta-learning
The paper formulates explanations as interpretable rules that predict a black box’s behavior, learning them with regularized empirical risk minimization. For saliency, it studies output changes under explicit image-region deletions, while highlighting limitations of gradient-based explanations and artifact-prone perturbations.
- Explanations as meta-predictors: A black box is modeled as a map from inputs to outputs, and an explanation is a rule that predicts its response on selected inputs.Examples include class membership rules and relative statements such as rotation invariance.
- Explanations as meta-predictors: Explanation faithfulness is measured as expected prediction error, making explanation learning analogous to supervised prediction.The framework evaluates how accurately a rule captures the black box’s behavior under an input distribution.
- Explanations as meta-predictors: Regularized empirical risk minimization selects explanations that generalize beyond sampled inputs while remaining simple and interpretable.The regularizer balances generalization and simplicity; informativeness can also be encouraged by preferring broader valid transformations such as larger rotation angles.
- Local explanations: Gradient saliency uses ∇f(x0) to identify influential pixels, but for a linear classifier it is independent of the image and therefore cannot express image-specific saliency.For nonlinear black boxes, diffuse responses and strong activations in apparently irrelevant regions can remain.
- Local explanations: Saliency explanations instead vary an image by deleting regions and observing how the black box’s output changes.Deletion can be modeled by multiplying the image pointwise by a mask; the resulting first-order expression is ∇f(x0) ⊙ x0, which depends on both the classifier and the image.
- Local explanations: The meaning of deletion depends on the perturbation, so the paper considers constant replacement, noise injection, and image blurring as explicit proxies.These perturbations are intended to simulate more naturalistic or plausible imaging effects, while artifact attraction motivates additional regularization.
4. Saliency revisited
Saliency is framed as a local deletion game: learn a compact mask whose explicit perturbation causes a target class score to drop. Because unconstrained masks can exploit artifacts, the method uses plausible perturbations and regularization to obtain more interpretable explanations.
- Saliency as input variation: The method identifies image regions used by a black box by observing how its output changes when regions of an image are deleted.This defines saliency as a local explanation around the original image.
- Saliency as input variation: Deletion is modeled with three perturbation proxies: replacing a region with a constant, injecting noise, or blurring the image.These operations aim to approximate naturalistic or plausible imaging effects.
- Deletion and preservation games: The deletion game learns the smallest mask that makes the target class score drop significantly after perturbation.The mask is optimized against the black box’s target score, while the sparsity term encourages deleting only a small subset of the image.
- Deletion and preservation games: The preservation game instead finds the smallest image subset that retains the original target-class score, separating sufficient evidence from evidence whose removal is decisive.The two objectives answer complementary questions about what the classifier relies on.
- Optimization: Gradient descent solves both optimization problems progressively, accumulating information from multiple gradient evaluations of the black box.The approach extracts information iteratively rather than from a single gradient evaluation.
- Dealing with artifacts: Unconstrained masks can trigger neural-network artifacts, so the method applies stochastic jitter and encourages simple, regular masks through total-variation regularization and low-resolution upsampling.These modifications are intended to reduce co-adaptation with artifacts and make perturbations more representative of natural changes.
5. Experiments
Experiments show that learned masks identify compact, suppressive image regions and support interpretable comparisons across perturbations, object parts, and adversarial examples. The method also performs competitively on localization and pointing tasks, though its performance depends on perturbation choice and object scale.
- Interpretability: The deletion game produces minimal masks that prevent the network from recognizing the object without highlighting non-essential evidence.Examples include only the face of a Pekingese dog, the upper half of a truck, or the spoon on a chocolate-sauce plate being minimally sufficient parts.
- Interpretability: A quantitative suppression criterion and direct image edits make saliency explanations verifiable and reveal when the network relies on unexpected associations.For chocolate sauce, perturbing the spoon suppresses the score more than perturbing the jar; a bumper perturbation instead raises the truck score from 0.717 to 0.850.
- Minimality of deletions: Simplified binary masks derived from learned masks quickly suppress GoogLeNet softmax scores across constant, noise, and blur perturbations.The masks are smoothed, thresholded over α ∈ [0 : 0.05 : 0.95], and evaluated on 5,000 ImageNet images.
- Minimality of deletions: 80%, 90%, 95%, and 99% suppression results show that the method yields the smallest average bounding boxes among compared methods.The boxes are formed by thresholding saliency maps, blurring the enclosed image region, and computing normalized target softmax probabilities.
- Object-part analysis: The mean average feet:eyes intensity ratio is 3.82 for tigers and 1.07 for bisons, while eyes are more salient than feet across all 76 analyzed classes.The analysis uses annotated non-occluded eye and foot keypoints from an ImageNet subset.
- Adversarial defense: The method recovers the clean image’s original predicted label 40.64% of the time and the ground-truth label 37.32% of the time from adversarial images.The original predicted label returns among the top five predictions in 100% of the 5,000 mask-plus-adversarial cases.
- Localization and pointing: The method performs competitively across localization and pointing evaluations, but its behavior varies with thresholding, perturbation design, and object size.It outperforms several baselines in pointing, while blur-based occlusion is strong except against contrastive excitation backprop and very small objects can yield weak heatmaps.
6. Conclusions
The paper presents a formal framework for learning explanations as meta-predictors and a saliency method based on image perturbations. It applies these ideas to study neural-network fragility and susceptibility to artifacts.
- The framework learns explanations as meta-predictors for black-box functions.
- The saliency method identifies image regions by measuring how perturbations affect an algorithm’s output score.
- The paper makes explanations interpretable and testable by explicitly editing images.
- Applications reveal insights into neural-network fragility and susceptibility to artifacts.