Source-linked AI summary
RISE: Randomized Input Sampling for Explanation of Black-box Models
Vitali Petsiuk, Abir Das, Kate Saenko
TL;DR
Deep neural networks make image-classification decisions that are difficult to explain, while many existing methods require access to model internals. RISE generates pixel-importance maps by probing black-box models with randomly masked inputs, and it performs competitively or better than existing approaches across automatic causal and human-centric evaluations.
Problem
Image-classification decisions from deep neural networks are difficult to explain, and existing methods often require access to gradients, features, weights, or architecture-specific internals.
Method
RISE estimates pixel saliency for a black-box image network by sampling random masks, recording target-class outputs, and combining the masks using those outputs as weights.
Results
RISE outperforms existing explanation approaches on automatic causal metrics and performs competitively on the human-centric pointing metric.
Takeaways & Limitations
RISE provides a general explanation approach for black-box image models without requiring access to their parameters, features, or gradients.
Takeaways & Limitations
RISE is evaluated as an image-explanation method and does not use information from inside the model.
Abstract
from arXiv · showhide
Deep neural networks are being used increasingly to automate data analysis and decision making, yet their decision-making process is largely unclear and is difficult to explain to the end users. In this paper, we address the problem of Explainable AI for deep neural networks that take images as input and output a class probability. We propose an approach called RISE that generates an importance map indicating how salient each pixel is for the model's prediction. In contrast to white-box approaches that estimate pixel importance using gradients or other internal network state, RISE works on black-box models. It estimates importance empirically by probing the model with randomly masked versions of the input image and obtaining the corresponding outputs. We compare our approach to state-of-the-art importance extraction methods using both an automatic deletion/insertion metric and a pointing metric based on human-annotated object segments. Extensive experiments on several benchmark datasets show that our approach matches or exceeds the performance of other methods, including white-box approaches. Project page: http://cs-people.bu.edu/vpetsiuk/rise/
1 Introduction
RISE addresses the difficulty of explaining image-classification decisions by producing pixel-importance maps for black-box models through randomized input probing. The paper also introduces deletion and insertion metrics intended to evaluate whether explanations reflect the model’s decision process.
- Motivation: Deep neural networks can make consequential decisions whose reasoning, confidence, and trustworthiness remain unclear to end users.The paper motivates transparent explanations especially for medical diagnosis, autonomous driving, and criminal justice.
- Motivation: RISE reveals model-specific errors, including confusing a black sheep with a cow and using a bird as evidence for a person prediction.These examples show how importance maps expose the visual evidence associated with incorrect classifications.
- Problem: Existing saliency methods often require gradients, feature maps, weights, or architecture-specific access to the explained network.This limits their applicability to arbitrary black-box models.
- Method: RISE estimates pixel saliency by masking the input randomly, recording target-class probabilities, and combining masks with those probabilities as weights.The method treats the image network as a complete black box without accessing parameters, features, or gradients.
- Evaluation: Human-annotated regions may not capture the cues a model actually uses, so the paper proposes causal evaluation metrics that do not depend on human judgments.The authors argue that models may rely on non-intuitive background cues, such as grass for detecting cows.
- Evaluation: Deletion removes salient pixels and measures class-probability decline, while insertion adds pixels and measures probability increase; a sharp deletion drop indicates a better explanation.For the goldfish example, RISE has the smallest deletion AUC among the illustrated methods.
2 Related work
Prior image-explanation work includes interpretable architectures, surrogate models, textual or visual justifications, and internal-activation methods. RISE instead explains an existing network through black-box input perturbations.
- Prior approaches: Interpretability research has used rules, decision trees, sparse linear surrogates, attention, and modular architectures to make decisions more understandable.These approaches either represent knowledge directly or approximate complex models with simpler structures.
- Prior approaches: Neural justification methods train additional models from human-annotated explanations or generate evidence from the base model’s internal computations.Third-person approaches depend on labeled explanations, while first-person approaches seek evidence for the base model’s decision.
- RISE: RISE differs from these methods by generating saliency maps from random input masks and model outputs rather than internal features, gradients, or architecture-specific components.The approach is presented as a black-box alternative for arbitrary image networks.
- Prior approaches: CAM computes class-specific importance from feature activations but is restricted to networks with global average pooling before classification.Grad-CAM extends this family by weighting feature activations using gradients.
3 Randomized Input Sampling for Explanation (RISE)
RISE estimates pixel importance for black-box image models by probing randomly masked inputs and combining the resulting confidence scores with the masks. It uses smoothed, shifted masks to reduce adversarial effects and improve sampling efficiency.
- Randomized input sampling: RISE estimates pixel importance by multiplying an image with random [0,1]-valued masks and observing the black-box model’s confidence scores.The model is queried on masked images without requiring access to parameters, features, or gradients.
- Randomized input sampling: Pixel importance is defined as the expected confidence score over masks conditioned on the pixel being preserved.The intuition is that preserved pixels receive higher importance when the masked image produces a high model score.
- Randomized input sampling: The importance map is computed as a weighted sum of random masks, using model confidence scores as weights and correcting for the mask distribution.RISE estimates this sum with Monte Carlo sampling over masked inputs.
- Mask generation: Independent pixel masking can cause adversarial confidence changes and creates a mask space of size 2^H×W, increasing the samples needed for accurate estimation.These issues motivate using smaller masks rather than independently masking every image pixel.
- Mask generation: RISE samples smaller binary masks, upsamples them bilinearly, and randomly shifts them before cropping to image size.Upsampling avoids sharp mask edges and produces a smooth importance map; shifted masks provide more flexible spatial masking.
4 Experiments
RISE is evaluated on benchmark datasets using automatic deletion/insertion metrics, the human-centered pointing game, and an image-captioning extension. The experiments compare RISE with existing explanation methods under black-box and white-box settings.
- Experimental Settings: RISE is evaluated on PASCAL VOC07, MSCOCO2014, and ImageNet object-classification datasets.VOC and MSCOCO explanations target object categories, while ImageNet explanations target each image’s top probable class.
- Evaluation Metrics: Deletion measures how quickly the predicted-class probability falls as salient pixels are removed, while insertion measures how well salient pixels synthesize an image.A sharp deletion drop and a small deletion AUC indicate a better explanation; insertion evaluates the corresponding ability of introduced pixels to recover the prediction.
- Evaluation Metrics: The insertion evaluation uses an initially blurred canvas to reduce spurious evidence from introduced image regions.For deletion, pixels are removed to change the classifier’s decision as quickly as possible.
- Evaluation Metrics: The pointing game counts a hit when the highest-saliency point lies inside a human-annotated object bounding box.Its accuracy is computed as #Hits/(#Hits+#Misses), averaged over target categories.
- Experimental Results: Table 1 compares deletion and insertion scores on ImageNet, with lower deletion and higher insertion scores preferred.Except for Grad-CAM, the compared explanation models are black-box methods.
- RISE for Captioning: RISE extends to image captioning by estimating a separate importance map for each caption word from randomly masked image inputs.The method can explain generated-caption words such as ‘horse’ and ‘carriage’, as well as arbitrary words such as ‘white’.
5 Conclusion
The paper concludes that RISE explains black-box models by estimating input-region importance and performs strongly across causal and human-centric evaluation metrics.
- Conclusion: RISE explains black-box models by estimating the importance of input image regions for model predictions.The approach is described as simple and general.
- Conclusion: RISE outperforms existing explanation approaches on automatic causal metrics and performs competitively on the human-centric pointing metric.The stated future direction is applying its generality to complex networks in video and other domains.
Appendix A Algorithms to compute causal metrics
The appendix specifies deletion and insertion algorithms that iteratively modify an image according to an importance map and accumulate model responses into a score.
- Deletion: The deletion algorithm takes a black-box model, image, importance map, and pixels removed per step, then returns a deletion score.At each iteration, the next most important pixels are set to zero and the model is evaluated again.
- Insertion: The insertion algorithm starts from a blurred image and progressively restores the next most important pixels from the original image.Model responses are recorded after each restoration step to compute the insertion score.
Appendix B More saliency maps and their scores
Additional figures show RISE importance maps alongside deletion and insertion curves, including failure cases where background noise remains in the explanation.
- Additional Saliency Maps: Figures 6–8 pair representative images and RISE importance maps with corresponding deletion and insertion curves.The figures use the same visual layout of image, map, deletion curve, and insertion curve.
- Failure Cases: Some RISE failure cases retain background noise because the Monte Carlo approximation uses only a subset of samples.In those cases, RISE may still identify more important features while failing to remove the background signal.