Source-linked AI summary
A Benchmark for Interpretability Methods in Deep Neural Networks
Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, Been Kim
TL;DR
Feature-importance explanations lack a ground-truth evaluation, making it difficult to assess their reliability and choose among many methods. The paper introduces ROAR, which removes estimated-important features from training and test data and retrains models on the modified datasets. Base estimators are no better than random, while VarGrad and SmoothGrad-Squared strongly outperform random guessing; other ensembles can add computation without improving performance.
Problem
Feature-importance explanations lack ground-truth evaluation, while many competing interpretability methods make method selection difficult.
Method
ROAR replaces estimated-important features with a fixed uninformative value in training and test data, then retrains models from random initialization on the modified datasets.
Results
Base estimators are no better than random, whereas VarGrad and SmoothGrad-Squared strongly improve feature-importance quality and far outperform random guessing; Classic SmoothGrad can be worse than a single estimate.
Takeaways & Limitations
The manner of ensembling matters: VarGrad and SmoothGrad-Squared improve estimates, but some more computationally intensive ensembles do not improve upon single estimates.
Takeaways & Limitations
ROAR is not directly suitable for decision stumps with feature redundancy unless feature importance is recomputed after each retraining step.
Abstract
from arXiv · showhide
We propose an empirical measure of the approximate accuracy of feature importance estimates in deep neural networks. Our results across several large-scale image classification datasets show that many popular interpretability methods produce estimates of feature importance that are not better than a random designation of feature importance. Only certain ensemble based approaches---VarGrad and SmoothGrad-Squared---outperform such a random assignment of importance. The manner of ensembling remains critical, we show that some approaches do no better then the underlying method but carry a far higher computational burden.
1 Introduction
The paper introduces ROAR to evaluate feature-importance estimators by removing estimated-important features and retraining models, addressing distribution-shift confounds. Across datasets, base estimators are no better than random, while selected ensemble methods perform substantially better.
- ROAR evaluation: ROAR replaces estimated-important pixels with an uninformative value in training and test images, then retrains models on the modified data.This keeps training and evaluation data aligned after feature removal.
- Experimental scope: ROAR evaluates interpretability methods across ImageNet, Food 101, and Birdsnap.These are described as large-scale, open-source image datasets.
- Robustness to feature removal: 90% random feature removal on ImageNet still yields 63.53 ± 0.13 accuracy versus 76.68% on clean data.The result is averaged across five independent runs.
- Benchmark results: The evaluated base methods are no better or on par with random importance, whereas SmoothGrad-Squared and VarGrad far outperform both their underlying methods and random guessing.This pattern is consistent across datasets and methods.
- Benchmark results: Classic SmoothGrad performs worse than a single estimate despite requiring more computation.The finding shows that ensembling alone does not guarantee improved feature-importance estimates.
2 Related Work
The paper situates ROAR among human-centered, sanity-check, and modification-based evaluations of interpretability. Its distinguishing requirement is retraining from random initialization on modified datasets rather than merely rescoring altered inputs.
- Interpretability evaluation: Interpretability research includes model distillation, constraints, neuron and activation analysis, high-level concepts, and input feature-importance estimators.ROAR belongs to the input feature-importance line of work.
- Existing evaluation approaches: Because correctness lacks a clear measure, prior comparisons often use human studies, manipulation tests, or parameter and label randomization checks.Modification-based methods instead assess degradation after replacing supposedly important inputs.
- ROAR distinction: Unlike prior modification-based measures, ROAR retrains models from random initialization on modified datasets instead of rescoring modified images at inference time.The benchmark is designed to separate replacement artifacts from estimator accuracy.
- ROAR distinction: Figure 1 compares base estimators, ensemble approaches, random modification, and a Sobel edge-filter control under ROAR.Each method replaces its estimated-important pixels with the mean and reports average accuracy from five retrained ResNet-50 models.
3 ROAR: Remove And Retrain
ROAR evaluates feature-importance estimates by removing ranked input features, retraining models on the modified data, and measuring accuracy across degradation levels. The design addresses distribution-shift concerns and reveals how redundancy and retraining affect interpretation of performance changes.
- ROAR ranks features by estimated importance, replaces the top fraction with per-channel means, retrains models from random initialization, and evaluates them on modified test data.
- Retraining is included because modified inputs can shift the data distribution, making accuracy degradation without retraining ambiguous between information removal and introduced artifacts.
- A performance drop after removing inputs suggests the removed dimensions were informative to the original model, whereas no drop can also reflect removal of uninformative or redundant inputs.
- On artificial data, ROAR correctly identifies the inverted ground-truth ranking as poor, showing no degradation until informative features begin to be removed at 75%.
- ROAR may show no accuracy decrease until all completely redundant variables are removed, so the evaluation measures multiple degradation levels to detect possible inflection points.
4 Large scale experiments
The experiments evaluate feature-importance estimators with ROAR across large-scale image datasets, comparing base and ensemble methods against random rankings. Base estimators generally fail to beat random importance, whereas VarGrad and SmoothGrad-Squared consistently improve approximate accuracy, with task-dependent differences in the best underlying estimator.
- Random ranking: 63.53% accuracy remains after retaining only 10% of ImageNet features, compared with 76.68% on unmodified data, showing robust retraining after random feature removal.This robustness supports comparing estimator rankings with random assignments.
- Base estimators: GRAD, IG, and GB perform on par with or worse than random and Sobel controls across datasets and thresholds, with the random gap widest at t = 0.9.At t = 90%, the best-to-worst base-estimator accuracy gaps are 4.22% for Birdsnap, 5.17% for Food101, and 3.62% for ImageNet.
- Ensemble estimators: Classic SmoothGrad is no better than a single estimate, while VarGrad and SmoothGrad-Squared substantially outperform both controls and their underlying estimators.VarGrad and SmoothGrad-Squared improve all base estimators, but the best underlying estimator varies by dataset: GRAD for ImageNet and Food101, and GB for Birdsnap.
- Ensemble estimators: VarGrad and SmoothGrad-Squared behave similarly because both can reduce to closely related statistics when the mean gradient is small relative to the mean squared gradient.Both methods qualitatively appear to remove whole objects.
- Ensemble estimators: Squaring estimates alone slightly improves performance but does not explain the large gains from VarGrad or SmoothGrad-Squared.The ensemble component remains important beyond the squaring transformation.
5 Conclusion and Future Work
The paper proposes ROAR for evaluating feature-importance estimators and finds that performance varies sharply across methods and ensemble strategies. VarGrad and SmoothGrad-Squared strongly outperform random importance assignments, while some computationally intensive ensembles do not improve results.
- ROAR evaluates feature-importance quality by measuring how sharply a retrained model’s accuracy degrades after estimated important features are removed.
- Base estimators such as Gradients, Integrated Gradients, and Guided BackProp perform worse than or on par with random importance assignment.
- VarGrad and SmoothGrad-Squared strongly improve estimator quality and far outperform a random guess.
- Classic SmoothGrad is more computationally intensive but does not improve upon a single estimate and can perform worse.
- The divergence among ensemble estimators remains an important direction for future research, especially for sensitive domains where explanation accuracy matters.
6 Supplementary Charts and Experiments
Supplementary experiments compare ROAR with KAR, test estimator transformations, and document dataset-generation procedures and limitations. ROAR discriminates among estimators more effectively than KAR, while squaring improves base estimates modestly and does not match SmoothGrad-Squared’s gains.
- 6.2 Generation of New Dataset: ROAR experiments generate modified train and test sets across ImageNet, Birdsnap, and Food 101 for multiple modification fractions and estimator variants.The study evaluates 18 estimators and generates 540 modified image datasets across the three original datasets.
- 6.2 Generation of New Dataset: ROAR replaces the most important pixels with a fixed uninformative value, retrains models on the modified data, and ranks estimators by resulting accuracy degradation.The same modification is applied to training and test images so the retrained model sees a comparable distribution.
- 6.2 Generation of New Dataset: KAR preserves estimated important inputs, whereas ROAR removes them; KAR therefore favors accuracy preservation while ROAR favors the largest accuracy degradation.
- 6.2 Generation of New Dataset: KAR is a weaker discriminator because base and ensemble estimators perform in a similar range, unlike the more discriminative ROAR benchmark.The comparison suggests that identifying features to preserve is easier than identifying features whose removal maximally damages performance.
- 6.4 Squaring Alone Slightly Improves the Performance of All Base Variants: Squaring a single estimate makes it slightly better than a random ranking, but its gains remain below those from averaging squared noisy estimates in SmoothGrad-Squared.For GB at t = .9, SQ-GB improves performance relative to GB by 8.43% ± 0.97.
- 6.4 Squaring Alone Slightly Improves the Performance of All Base Variants: Squaring is equivalent to ranking by absolute value, so feature importance depends on magnitude rather than the direction of the estimate.This transformation does not explain the larger gains obtained by averaging squared noisy estimates.
- 6.5 Limitations on the use of ROAR: ROAR is not directly suitable for decision stumps with feature redundancy unless feature importance is recomputed after each retraining step.The paper considers ROAR valid for linear models, multilayer perceptrons, deep neural networks, and models allowing all features to contribute at test time.