Source-linked AI summary
How can I choose an explainer? An Application-grounded Evaluation of Post-hoc Explanations
Sérgio Jesus, Catarina Belém, Vladimir Balayan, João Bento, Pedro Saleiro, Pedro Bizarro, João Gama
TL;DR
Real-task evaluation of XAI methods is limited, despite the need to assess their impact on human decision-making. The authors propose XAI Test and find that data-only decisions are more accurate but slower than decisions with model information; explanations slightly improve accuracy over model scores alone.
Problem
Prior XAI evaluations often use simulated or absent user experiments and lack systematic comparisons of methods on real tasks with end-users.
Source: §1.1 · p. 2, §2.2 · p. 3Method
XAI Test evaluates how progressively adding model scores and explanations affects analysts’ decisions in a real-world fraud-detection task.
Source: §6 · p. 10Results
Data-only decisions were significantly better but slower than variants with model information, while explanations slightly improved accuracy over model scores alone but remained below data-only accuracy.
Source: §5 · p. 6, §6 · p. 10Takeaways & Limitations
When decision speed is a main requirement, explanations can significantly speed human decision-making.
Source: §5 · p. 6Limited participant availability constrained sample size, affecting effect size and statistical-test error rates.
Source: §5.4 · p. 9
Abstract
from arXiv · showhide
There have been several research works proposing new Explainable AI (XAI) methods designed to generate model explanations having specific properties, or desiderata, such as fidelity, robustness, or human-interpretability. However, explanations are seldom evaluated based on their true practical impact on decision-making tasks. Without that assessment, explanations might be chosen that, in fact, hurt the overall performance of the combined system of ML model + end-users. This study aims to bridge this gap by proposing XAI Test, an application-grounded evaluation methodology tailored to isolate the impact of providing the end-user with different levels of information. We conducted an experiment following XAI Test to evaluate three popular post-hoc explanation methods -- LIME, SHAP, and TreeInterpreter -- on a real-world fraud detection task, with real data, a deployed ML model, and fraud analysts. During the experiment, we gradually increased the information provided to the fraud analysts in three stages: Data Only, i.e., just transaction data without access to model score nor explanations, Data + ML Model Score, and Data + ML Model Score + Explanations. Using strong statistical analysis, we show that, in general, these popular explainers have a worse impact than desired. Some of the conclusion highlights include: i) showing Data Only results in the highest decision accuracy and the slowest decision time among all variants tested, ii) all the explainers improve accuracy over the Data + ML Model Score variant but still result in lower accuracy when compared with Data Only; iii) LIME was the least preferred by users, probably due to its substantially lower variability of explanations from case to case.
1 INTRODUCTION
XAI research often lacks systematic evaluation with intended users performing real tasks, despite their differing roles and explainability needs. The paper proposes XAI Test and applies it to fraud analysts, finding a speed–accuracy trade-off as model information is added.
1.1 The evaluation problem in Explainable AI
- Real-task evaluations with intended end-users are scarce, and ad-hoc setups leave explanation methods difficult to compare systematically.Users’ roles, objectives, and domain knowledge shape their explainability requirements, making the gap especially consequential in high-responsibility applications.Source: §1.1 · pp. 1–2
1.2 The impact of showing explanations
- XAI Test evaluates explanations by measuring their impact on end-user task performance in realistic settings.The methodology isolates information levels and calls for real users, real tasks and data, performance metrics, agreement measures, questionnaires, and statistical tests.Source: §1.2 · p. 2
1 INTRODUCTION
- The study applies XAI Test to evaluate post-hoc explanations with fraud analysts performing a real fraud-detection task.It measures human performance and users’ perceptions of three explainers using real data and a deployed model.Source: §1.3 · p. 2, §1.2 · p. 2
1.3 The experiment
- Data Only decisions were more accurate but slower; adding model information sped decisions, while explanations partially improved accuracy over score-only.Figure 1 plots average decision accuracy against average decision time across the tested variants; balanced positive and negative samples make 50% the random-decision accuracy baseline.Source: Figure 1 · p. 1, §1.3 · p. 2
2 RELATED WORK
Prior XAI evaluation often emphasizes explanation desiderata or proxy tasks, while offering limited evidence about effects on end-users performing real-world tasks. This leaves systematic, objective comparison of methods lacking.
2.1 Desiderata
- Fidelity, robustness, and interpretability are common proxies, but do not by themselves establish an explanation’s benefit in a specific real-world application.The related work describes measures of these desiderata, including model-mimicry, output stability, and human comprehensibility.Source: §2.1 · p. 2
- Existing XAI evaluation practices lack consensus, often omit intended end-users, and leave systematic, objective comparison of explanation methods missing.Customized metrics and desiderata make choosing methods for a particular task difficult.Source: §2.1 · p. 3
2.2 Evaluation Practices
- Application-grounded evaluation uses humans performing real-world tasks and assesses effects on whole-system performance.Human-grounded evaluation instead uses simplified proxy tasks and focuses on how people perceive and interact with explanations.Source: §2.2 · p. 3
- Prior experiments often use mocked tasks, limiting generalization to high-responsibility settings, and seldom compare explanation methods themselves.Simulated human behavior may diverge from reality, while comparisons often emphasize presentation rather than explanation content.Source: §2.2 · p. 3
3 EVALUATION METHODOLOGY
XAI Test assesses explanation utility through user performance in realistic tasks rather than relying only on explanation desiderata. It combines staged information exposure with task-specific metrics, user feedback, agreement measures, and statistical comparisons.
3 EVALUATION METHODOLOGY
- XAI Test uses realistic settings, user-performance metrics, and statistical tests to assess and compare explanation utility.Its workflow formulates hypotheses, specifies the setup and statistical tests, conducts staged experiments, and analyzes the measurements.Source: §3 · p. 3
3.2 Experimental Stages
- The methodology compares Data Only, Data + ML Model Score, and Data + ML Model Score + Explanations to isolate the effects of added information.Model scores may be perceived as confidence: scores near 0 or 1 indicate confidence, while scores near 0.5 convey uncertainty.Source: §3 · p. 4, §3.2 · p. 4
3.1 Metrics Choice
- Task-specific performance measures can use known labels and user decisions; accuracy, recall, and FPR support a priori sample-size calculation.Decision time, perceived relevance and usefulness, and agreement between users are also measured.Source: §3.1 · p. 4
3.3 Statistical Tests
- Statistical tests are selected by metric type and comparison goal, with Holm-Bonferroni corrections for pairwise comparisons.The paper uses Chi-squared tests for binary metrics, Kruskal-Wallis H for continuous metrics or multiple-group questionnaire comparisons, Mann-Whitney U for paired continuous data, and corrected Kolmogorov-Smirnov tests for paired questionnaire tests.Source: §3.3 · pp. 4–5
4 EXPERIMENTS
The experiment applies XAI Test to fraud analysts using a deployed model and real transaction data. Analysts receive progressively more information, allowing the study to compare three post-hoc explainers and test hypotheses about performance, perceptions, and agreement.
4 EXPERIMENTS
- The study evaluates a deployed fraud model in a real payment-transaction review task where analysts decide whether uncertain transactions should be accepted or declined.The experiment aims to assess the impact of explanations on real analysts interacting with a real model.Source: §4 · p. 5
4.1 Experimental Hypotheses
- The hypotheses concern model-score effects, explanation effects and explainer differences, user perceptions, and analyst agreement.They include whether scores improve performance over Data Only, whether explanations improve performance, and whether added information increases agreement.Source: §4.1 · p. 5
4.2 Experimental Setup
- Three experienced fraud analysts used a Random Forest variant and evaluated LIME, TreeSHAP, and TreeInterpreter.LIME used 5k perturbed samples; explanations displayed the top six feature-contribution pairs in a standardized, human-readable format, with green for negative and red for positive contributions.Source: §4.2 · p. 5
- The experiment used 1,300 production-period transactions from the review band, stratified to 50% fraud prevalence.The initial stages used 200 transactions each, and the explanation stage used 300, divided equally among analysts and explainers.Source: §4.2 · p. 6, §4.3 · p. 6
4.3 Experiment Outline
- Analysts completed Data Only, Data + ML Model Score, and Data + ML Model Score + Explanations stages in sequence to stabilize their task mental model.An agreement set comprising about 12.5% of transactions at each stage let all analysts review the same instances.Source: §4.3 · p. 6
5 RESULTS AND DISCUSSION
Across the staged fraud-review experiment, Data Only produced better accuracy and recall but slower decisions, while adding model scores or explanations sped decisions. The results compare score-only and three explainer variants, and examine their effects on analysts’ decisions, perceptions, and agreement.
5 RESULTS AND DISCUSSION
- Data Only yielded better accuracy and recall but slower decisions; adding model scores or explanations significantly improved decision time.The comparison covers all experiment variants; Figure 4 reports performance-metric intervals and Figure 5 reports decision-time intervals.Source: §5 · p. 6, Figure 4 · p. 8, Figure 5 · p. 8
5.1 Data + ML Model Score
- Model scores reduced decision time by approximately 25% but were associated with 10% lower accuracy and recall and a roughly 4-percentage-point FPR increase versus Data Only.Analyst agreement also fell: Fleiss’ Kappa changed from 0.41 to −0.02, and average agreement from 76.67% to 63.33%.Source: §5.1.1 · p. 7, §5.1.3 · p. 7, §5.1.4 · p. 7
5.2 Data + ML Model Score + Explanations
- Every explainer significantly reduced decision time versus Data Only, but increased it versus score-only without a statistically significant difference; TreeInterpreter was fastest among explainers.The explainer-versus-Data-Only pairwise p-values ranged from 1 × 10−4 for LIME to 0.09 for TreeInterpreter.Source: §5.2.1 · pp. 7–8, Figure 5 · p. 8
- Explanations did not significantly improve efficacy: all three had lower accuracy and recall than Data Only, but improved on score-only results without statistical significance.The multiple-group Chi-squared test was inconclusive, so no paired tests were conducted between explainer variants.Source: Table 1 · p. 7, §5.2.4 · p. 8, Figure 4 · p. 8
- Users rated explainers differently: TreeInterpreter had the most positive responses, while LIME differed significantly from SHAP and TreeInterpreter on whether explanations helped review faster.Differences were significant for usefulness and review-speed questions, but not for the first questionnaire question.Source: §5.2.3 · p. 8, Figure 6 · p. 9
- LIME alone exceeded Data Only in analyst agreement; all explainers appeared to improve consensus over score-only, with LIME’s Fleiss’ Kappa highest at 0.53.Average agreement was 84.62% for LIME, 69.23% for TreeInterpreter, and 64.10% for SHAP.Source: Table 1 · p. 7, §5.2.5 · pp. 8–9
5.3 Variability in Explanations
- LIME used 34 of 111 features, versus 89 for SHAP and 107 for TreeInterpreter; feature overlap between explainer pairs ranged from 23.5% to 53.0%.The paper relates fewer features used to less explanation variability; LIME’s most common feature appeared in 89.7% of transactions, versus 45.3% for TreeInterpreter.Source: §5.3 · p. 9
5.4 Study Limitations
- The study’s limited participant availability constrained sample size and statistical sensitivity, while uncontrolled external factors could affect results; same-format explainers showed no significant performance differences.The authors identify large randomized controlled trials as the way to mitigate unaccounted factors and propose studying other explanation configurations and visualizations.Source: §5.4 · p. 9
6 CONCLUSION
The paper addresses the gap between proxy-based explanation evaluation and measuring effects on end-user decision performance. Its real-world fraud-detection study finds a trade-off between decision accuracy and speed as analysts receive more information.
6 CONCLUSION
- XAI methods are often assessed with proxy desiderata such as fidelity or robustness, rather than their actual impact on end-user performance.The authors warn that this gap may result in sub-optimal decision processes.Source: §6 · pp. 9–10
- The authors propose XAI Test, an application-grounded methodology that isolates the impact of information levels on end-user performance.They evaluate it with three fraud analysts, a production ML model, and real-world data.Source: §6 · p. 10
- Data-only decisions were significantly better but slower; explanations slightly improved accuracy over model scores yet remained below data-only, while analysts least favored LIME, potentially for low diversity.The study reports that Data Only produced significantly better decisions, and that LIME was least favoured, potentially because of its low explanation diversity.Source: §6 · p. 10
- The results suggest a trade-off between effectiveness and efficiency as analysts receive additional information, cautioning against blindly selecting popular post-hoc explainers.Source: §6 · p. 10