What I Cannot Predict, I Do Not Understand: A Human-Centered Evaluation Framework for Explainability Methods

• Faithfulness metrics assess explanation–model relationships without humans, so their ability to predict explanations’ practical usefulness remains unclear.Source: §1 · pp. 1–2
• The paper evaluates explanations by whether users learn general rules that transfer to unseen instances, aligning evaluation with explanations’ role in supporting learning and generalization.Source: §1 · p. 2
• The study covers bias-source detection, strategy discovery for a complex task, and failure understanding using Husky-versus-Wolf, leaf-classification, and ImageNet scenarios.Source: §1 · p. 2
• Across 1,150 participants, SmoothGrad [8] was most useful, faithfulness metrics did not predict practical usefulness, and perceptual scores better predicted attribution-method failures.Source: §1 · p. 2

• Faithfulness evaluations track classification-score drops after removing important pixels; ROAR retrains between removals, but baseline dependence and excluding humans leave practical usefulness uncertain.Source: §2 · p. 2
• Earlier Husky-versus-Wolf work tested whether explanations changed participants’ trust, whereas this study measures whether participants understand the model.Source: §2 · p. 3
• Prior simulatability studies ask whether people learn to copy model predictions on unseen data, but some provide training labels or reuse participants across conditions in a fixed order.Source: §2 · p. 3
• The authors distinguish their framework from prior human evaluations by removing ground-truth labels, using different participants across conditions, and retaining explanation complexity in comparisons.Source: §2 · p. 3
• The framework trains a human Meta-predictor with images, model predictions, and explanations, then scores its relative accuracy improvement over training without explanations.Source: Figure 2 · p. 4

Core concept

• The framework defines understanding as a human user’s ability to infer a model’s rules and correctly predict its outputs on arbitrary inputs.The user is modeled as a Meta-predictor of the black-box classifier.Source: §3 · p. 4

Design trade-off

• Explanation quality depends on both faithfulness to the evidence driving predictions and how much of the highlighted pattern a user understands.Fully faithful information may be too complex to understand, while simplification may mislead if it is not faithful.Source: §3 · p. 4

Usefulness measure

• Utility-K measures a human Meta-predictor’s accuracy after learning with explanations, normalized by accuracy after training without them.Its score uses classifier predictions rather than ground-truth labels and compares explanation-trained accuracy with a no-explanation baseline.Source: §3 · p. 5

Aggregate metric

• The aggregate Utility score is the area under the Utility-K curve across different numbers of observations; higher scores indicate more useful explanations.The curve aggregates usefulness across the selected training-sample counts.Source: §3 · p. 5

Study design

• The design compared explanation training with no-explanation and bottom-up saliency-map control conditions, then tested participants without explanations.Participants completed three training sessions of five samples each, with a brief test on seven new samples after each session.Source: §4 · pp. 5–6

Control condition

• The non-informative saliency-map control decreased participants’ ability to predict the classifier compared with receiving no explanation.The authors interpret this as suggesting that a wrong explanation may be worse than no explanation.Source: §4 · p. 6

Scenarios

• The Husky-versus-Wolf experiment tested bias detection, while the Leaves task used two classes that could not be distinguished by shape to elicit strategy discovery.The ImageNet experiment used Kit Fox and Red Fox, a heavily misclassified class pair, to examine failure cases while balancing correct and incorrect predictions.Source: §4 · pp. 6–7

Scenario results

• On Figure 3, every method improved participants’ model-prediction ability for bias detection, with Grad-CAM, Occlusion, and SmoothGrad particularly useful; on Leaves, Saliency, SmoothGrad, and Integrated Gradients were useful.The figure reports Utility-K after each of three training sessions for Husky versus Wolf and Leaves.Source: Figure 3 · p. 7

Methods and scenarios

• Across three experiments, the study compared six attribution methods: Saliency, Gradient ⊙Input, Integrated Gradients, Occlusion, SmoothGrad, and Grad-CAM.The experiments used different datasets and neural-network architectures for distinct real-world use cases.Source: §4 · pp. 6–7

5 Results

• Grad-CAM, Occlusion, and SmoothGrad significantly outperformed the baseline for bias detection.The overall explanation effect was significant (F(7, 234) = 9.19, p < .001, η2 = 0.089).Source: §5 · p. 7
• For identifying an expert strategy, Saliency, SmoothGrad, and Integrated Gradients were the only methods significantly better than baseline.Saliency was among the worst methods for bias detection but the best in this use case; SmoothGrad was useful in both scenarios where explanations helped.Source: §5 · p. 8
• Attribution methods helped in bias detection and expert-strategy identification, but none of the tested methods exceeded baseline accuracy for ImageNet failure cases.The ImageNet result was not statistically significant (ANOVA: F(7, 233) = 1.26, p > .05).Source: §5 · pp. 7–8

Why do attribution methods fail?

• Faithfulness did not predict practical utility: some of the least useful methods on Bias and Leaves ranked among the leading methods by faithfulness.The study reports a weak, possibly negative relationship; Figure 4 also shows insignificant utility scores for ImageNet methods.Source: Why do attribution methods fail? · p. 8, Figure 4 · p. 8
• Complexity tracked usefulness strongly in one dataset where explanations helped, but results were inconclusive for Leaves and irrelevant for ImageNet.The study approximated Kolmogorov complexity with JPEG compression; complexity differences did not explain ImageNet failures and may not reflect human visual complexity.Source: Why do attribution methods fail? · pp. 8–9
• Attribution maps show where to look but not what feature matters, a proposed reason they may fail for fine-grained, perceptually similar classes.For example, locating an ear does not specify the relevant shape distinction.Source: Why do attribution methods fail? · p. 9
• Greater perceptual similarity between discriminative features across classes was associated with lower attribution usefulness, and all methods had high similarity scores on ImageNet.The authors suggest that beyond a similarity threshold, explanations may not help regardless of faithfulness or complexity.Source: Why do attribution methods fail? · p. 9

6 Discussion

• Perceptual similarity between highlighted features for both classes was strongly correlated with practical usefulness.Figure 6 reports this relationship across the methods and datasets studied.Source: Figure 6 · p. 9
• In two of the three tested scenarios—bias detection and identifying new strategies—explanations meaningfully assisted human end-users, but none helped with failure cases.The failure-case result was consistent with previous work [61].Source: §6 · pp. 9–10
• Faithfulness metrics did not correlate with empirical utility, while complexity showed only a weak relationship too small to explain failures.The authors report that explanations could be similarly faithful and low in complexity whether useful or not.Source: §6 · p. 10
• When diagnostic regions across classes look perceptually similar, attribution methods may fail regardless of explanation faithfulness or complexity.The authors suggest that explanations may need to convey information beyond attribution maps, including what visual features drive decisions.Source: §6 · p. 10

7 Limitation and broader impact

• The usefulness measure does not account for user expertise, so the most useful explanation for a novice may differ from the best one for an expert.Human-participant evaluation also brings challenges compared with automated metrics.Source: §7 · p. 10
• The evaluation approach is intended to help practitioners identify bias and shortcuts that may unfairly discriminate against groups.The paper connects this aim to risks from models relying on shortcuts such as spurious training-set correlations.Source: §7 · p. 10

A.1 Experimental design

• Participants were recruited through AMT, prioritizing Master-qualified or highly rated users, and consent materials stated the study procedure, expected duration, reward, and risks.The stated completion time was approximately 5–8 minutes and the reward was $1.4.Source: §A.1 · p. 15
• Participants saw model predictions with explanations, without explanations, or with non-informative explanations, then predicted outputs for 7 image-only test samples after 5 training samples.Each participant completed 3 training-and-test sessions and experienced only one condition.Source: Figure 7 · p. 15
• Samples were balanced so classifiers were correct and incorrect equally often, making 50% the baseline for guessing true labels rather than learning model rules.Scores above 50% therefore reflect some understanding of the model’s rules.Source: §A.1 · p. 15

A.2 Pruning out uncooperative participants

• A three-stage screen checked participants’ task skills, comprehension of instructions, and attention through catch trials; anyone failing a stage was excluded.The practice required correctly predicting 5 cat images on the first try, and the quiz required every answer correct on the first try.Source: §A.2 · pp. 15–16, Figure 8 · p. 16, Figure 9 · p. 17

A.3 More results

• No trend linked participants’ reaction times with the usefulness of an explanation method across datasets.Table 2 reports average total time by method and dataset.Source: §A.3 · p. 17, Table 2 · p. 17
• Saliency was the worst method on the bias dataset but the best on the leaves dataset, where the model used fine leaf margins or veins rather than a coarse background feature.The authors hypothesize that methods suit different feature granularities.Source: B Why do the best methods for the use… · p. 19
• Grad-CAM and Occlusion performed best on the biased dataset, while their coarser heatmaps were less suited to highlighting fine leaf features.The authors suggest the coarser maps may take participants longer to interpret on the leaves task.Source: B Why do the best methods for the use… · p. 19

C.1 Faithfulness

• Insertion and µFidelity were no better than Deletion at predicting attribution methods’ practical usefulness.Figure 11 presents Utility correlations with Insertion and µFidelity; ImageNet Utility scores are insignificant because none of the methods improves the baseline.Source: §C.1 · p. 19, Figure 11 · p. 19

C.2 Perceptual Similarity

• Perceptual similarity scores were higher on ImageNet, where attribution methods did not help, than on the two datasets where they did help.Table 3 measures similarity of highlighted regions for both classes; higher scores mean greater similarity.Source: §C.2 · p. 19, Table 3 · p. 20
• Figure 12 illustrates Grad-CAM patches extracted from locations its attribution maps considered most important across the three datasets.The figure provides examples of the patches used for perceptual similarity scoring.Source: Figure 12 · p. 20

D.1 Methods

• Saliency uses input gradients to indicate which pixels would most affect a class score, while Gradient ⊙Input multiplies those gradients element-wise by the input to sharpen attribution maps.The paper also notes Gradient ⊙Input’s equivalence to ϵ-LRP and DeepLIFT under specified zero-baseline and zero-bias conditions.Source: §D.1 · p. 21
• Integrated Gradients sums gradients along a path from a baseline to the input; these experiments used a zero baseline and 80 points.The integral was approximated with the Trapezoidal rule.Source: §D.1 · p. 21
• SmoothGrad averages gradients from perturbed inputs to reduce visual noise, using 80 samples and σ = 0.2 in these experiments.The perturbations are drawn from a normal distribution around the input.Source: §D.1 · p. 21
• Grad-CAM uses gradients and last-layer feature maps to produce class localization maps whose size depends on the feature map, then interpolates them to input dimensions.The method is described for convolutional neural networks.Source: §D.1 · p. 21
• Occlusion sweeps an image patch over the input and uses changes in model predictions to identify critical areas; the patch size and stride were each one tenth of the image size.Source: §D.1 · p. 21

D.2 Examples of explanations

• Examples show control and attribution explanations for images from the Husky vs. Wolf, Leaves, and ImageNet experiments.Figures 13–15 present the examples for the respective datasets.Source: §D.2 · p. 22, Figure 13 · p. 22, Figure 14 · p. 23

Checklist

• The checklist states that the paper discussed its limitations, identified no potential negative societal impacts, and made code, data, and instructions available by URL.Source: Checklist · p. 25