Source-linked AI summary
XRAI: Better Attributions Through Regions
Andrei Kapishnikov, Tolga Bolukbasi, Fernanda Viégas, Michael Terry
TL;DR
Saliency methods lack straightforward ground-truth evaluation, motivating better attribution methods and validation tools. The paper introduces region-based XRAI, Performance Information Curves, and a perturbation-based sanity check, and reports that XRAI outperforms many other methods while the metrics align with visual observations and standard localization metrics.
Problem
Image-based saliency methods require empirical validation, but the lack of ground truth makes precise numerical comparison difficult.
Method
The paper combines region-based XRAI built on Integrated Gradients with Performance Information Curves and a perturbation-based sanity check for attribution methods.
Results
XRAI is reported to outperform many other saliency methods, while the proposed measurement methods align with visual observations and standard localization metrics.
Takeaways & Limitations
Region-level attribution can produce more robust saliency than pixel-level Integrated Gradients, and the proposed evaluations provide complementary ways to assess saliency quality.
Takeaways & Limitations
XRAI does not explicitly encode segment choices that align with human perception of semantically meaningful regions.
Abstract
from arXiv · showhide
Saliency methods can aid understanding of deep neural networks. Recent years have witnessed many improvements to saliency methods, as well as new ways for evaluating them. In this paper, we 1) present a novel region-based attribution method, XRAI, that builds upon integrated gradients (Sundararajan et al. 2017), 2) introduce evaluation methods for empirically assessing the quality of image-based saliency maps (Performance Information Curves (PICs)), and 3) contribute an axiom-based sanity check for attribution methods. Through empirical experiments and example results, we show that XRAI produces better results than other saliency methods for common models and the ImageNet dataset.
1. Introduction
The paper addresses challenges in identifying and evaluating salient image inputs by proposing XRAI, new performance-based metrics, and a perturbation-based sanity check. Experiments on ImageNet show that XRAI outperforms existing saliency methods, while the proposed metrics align with visual results.
- Saliency methods connect DNN predictions to influential inputs, supporting debugging, spurious-correlation checks, and fairness inspection.
- XRAI over-segments images, tests region importance iteratively, and coalesces smaller regions into larger attribution segments.The method builds on Integrated Gradients and can be used with any DNN model when input features can be clustered into segments.
- The perturbation-based sanity check requires features causing non-zero output changes to receive non-zero attributions.Gradients, Gradients*Input, and Integrated Gradients can sometimes fail this check, whereas region-level sums of pixel attributions are more robust.
- Performance Information Curves compare saliency methods by progressively sharpening important image regions and measuring image information alongside model performance.The two variants are Accuracy Information Curves and Softmax Information Curves, whose rankings align with visual results.
- The paper contributes XRAI, a perturbation-based sanity check, and two attribution-quality metrics, with experiments comparing XRAI against several existing methods on ImageNet models.The evaluation includes GradCAM, Gradient, Gradient*Input, and Integrated Gradients with different baselines, using Inception and ResNet50 models.
2. Related Work
Existing saliency methods use perturbations or back-propagated signals to identify influential inputs, but each approach has evaluation or reliability limitations. XRAI’s region-based strategy builds on Integrated Gradients, while PIC-style evaluation measures saliency quality through progressive information restoration.
- Attribution methods: Perturbation-based methods estimate feature impact directly but require multiple model queries and become difficult to evaluate comprehensively as feature subsets grow.Neural-network nonlinearities make results dependent on the exact subset and modification of features.
- Attribution methods: Back-propagation methods are generally faster because they require a single or constant number of network queries independent of input-feature count.Examples include Integrated Gradients, GradCAM, and Gradient*Input.
- Evaluation and reliability: Saliency reliability is evaluated with sanity checks because some maps can be fragile against adversarial attacks or fail to reflect learned model parameters.Prior evaluation also includes empirical measures such as Sensitivity-n.
- Evaluation and reliability: The paper’s measurement methods rank image regions by progressively restoring information, complementing smallest-sufficient-region evaluations that focus only on achieving the correct prediction.The broader ranking can include less important but potentially useful or problematic regions.
3. Perturbation Sanity Check
The paper proposes Perturbation-ϵ as an attribution sanity check based on whether removing a feature changes the output. A two-feature simulation exposes failures in popular pixel-level methods and motivates using larger regions, whose aggregated attributions are more reliable.
- Axiom and test: Perturbation-ϵ requires a feature’s attribution to be at least ϵ times the output change caused by removing that feature.It is applicable to attribution methods generally and relaxes Sensitivity-1.
- Axiom and test: The sanity-check simulation samples a two-pixel function defined over x1 and x2, using a 20x20 grid and bi-cubic interpolation for continuity.The construction tests attribution behavior around a sharply defined classification peak.
- Axiom and test: At the function’s peak, small changes to either feature sharply reduce the classification score, so both features should receive small non-zero attributions when ϵ is near zero.The prediction is positive only at the peak and negative at other grid points.
- Observed failures: Gradient and Gradient*Input fail the sanity check at the peak because their gradients assign zero attribution to all features.Gradients can also fail in real networks when local directions are arbitrary or activations are saturated.
- Observed failures: Integrated Gradients can also assign zero attribution to one input at the peak, because its result may depend on network behavior far from the classification-relevant region.This motivates seeking locality in attribution methods.
- Regional robustness: Larger regions covering objects of interest produce more reliable attributions than individual pixels for Integrated Gradients.This observation motivates the paper’s region-based modification of Integrated Gradients.
4. XRAI
XRAI builds region-level saliency maps by combining Integrated Gradients with multi-scale image segmentation, black-and-white baselines, and attribution-guided region selection. It addresses unreliable pixel attributions, baseline sensitivity, and gradient-based attribution of irrelevant objects by producing smoother, more robust regions.
- XRAI algorithm: XRAI over-segments an image, computes an attribution map, and iteratively adds regions with the greatest attribution gain per area.The algorithm maintains a saliency mask and trajectory until no segments remain or the mask covers the image.
- Attribution: XRAI uses Integrated Gradients with black and white baselines to give every pixel an equal chance of contributing to attribution.For normalized pixel value x, the distances to the black and white baselines sum to 1.0, producing consistent saliency maps.
- Region selection: XRAI selects regions by their summed attribution, adding areas with more positive values because they are more important to the classifier.This follows IG’s completeness property, in which positive attribution indicates contribution to the predicted class and negative attribution indicates competing regions.
- Region structure: XRAI produces smooth regions that often encapsulate whole objects, but its algorithm does not explicitly encode human-perceived semantic boundaries.The method uses segmentation and merges salient segments, while segment alignment with human perception remains an implicit outcome rather than an explicit objective.
- Class relevance: Gradient-based methods can attribute both objects in a cat-and-dog image, whereas XRAI uses positive and negative region sums to identify the region relevant to the selected class.Gradients may behave like edge detectors because changes to another class can alter the selected class’s softmax output.
- Reliability: Region-level aggregation is more reliable than pixel-level Integrated Gradients, whose individual attributions can be zero or affected by distant model behavior.The paper reports that combining pixels into regions can make their total attribution robust even when individual pixel values are unreliable.
5. Assessing Attribution Quality Through Blurring and Iterative Testing
The paper evaluates saliency by progressively restoring information to blurred images, then measuring model performance as information increases. AIC and SIC provide complementary performance curves while reducing artifacts associated with sharp masking boundaries.
- Masking salient pixels can create unintended effects from sharp boundaries, complicating quantitative evaluation without ground-truth saliency maps.The paper identifies the absence of ground truth as a primary difficulty and motivates less adversarial perturbations.
- Performance Information Curves start from blurred images, restore pixels ranked important by a saliency method, measure image entropy, and evaluate classification.Results are mapped against information level to compare saliency methods.
- Gradually reintroducing content reveals the most important, next-most-important, and later regions for a model’s prediction.Blurred backgrounds also produce more realistic bokeh-style images than direct masking.
- PICs aggregate performance across dataset samples, and their area under the curve measures saliency quality analogously to ROC area.This summarizes performance across information levels rather than selecting only the threshold with highest accuracy.
- AIC plots accuracy by information level, whereas SIC plots the original label’s softmax proportion for the bokeh image relative to the original.Using information level avoids favoring grainy maps that cover many pixels but convey less coherent content.
- Image information is approximated using the compressed-size ratio of a bokeh image to its original under lossless WebP compression.The paper uses this proxy because exact image entropy is infeasible to measure.
6. Experiments
Experiments on ImageNet compare XRAI with established saliency methods using PICs, visual inspection, and localization metrics. XRAI generally achieves the strongest evaluation results and produces tighter, multi-region explanations than competing methods.
- XRAI’s evaluation rankings align with visual observations, while random saliency performs poorly at all information levels.The agreement supports the usefulness of the proposed measurement framework.
- XRAI outperforms other methods on median SIC and AIC curves for Inception and ResNet50, with GradCAM following closely.Figure 9 reports the cross-method comparison for both evaluation measures and models.
- XRAI’s visual regions are tightly bounded around identified objects, whereas IG variants produce grainy regions and edge methods may emphasize background.On a two-dog image, edge attribution works best when one object occupies most of the image.
- GradCAM tends to expand one region, whereas XRAI can focus on multiple areas and tightly cover multiple objects of interest.Examples include parrot, dog, starfish, snails, and birds, where GradCAM can also focus between objects.
- XRAI outperforms IG and GradCAM on all reported localization metrics for the ImageNet segmentation dataset.The localization rankings are similar to those from SIC and PIC measurements.
- The evaluation framework can serve as a proxy for localization tests when ground-truth segments are unavailable.This conclusion follows from the similarity between localization and SIC/PIC rankings.
7. Conclusion
The paper combines a perturbation-based sanity check, region-based integrated gradients algorithm, and performance-based saliency evaluation. Experiments and examples show XRAI is superior to many other methods, while the proposed measurements align with visual and localization assessments.
- The paper proposes a perturbation-based sanity check, XRAI, and a new way to measure saliency quality.These are presented as the paper’s three principal contributions.
- Experiments and example outputs demonstrate that XRAI is superior to many other saliency methods.
- The proposed measurement methods align with visual observations and standard localization metrics.
Supplementary Material
Supplementary examples compare saliency methods on dog images, highlighting differences in which semantically meaningful regions are selected.
- Additional supplementary images provide further comparisons among saliency methods.
- For a Sussex spaniel, XRAI can select both eyes and the snout independently, while GradCAM tends to grow one spot.Ears, eyes, and snout appear as important regions for identifying the breed.