Source-linked AI summary
Challenging common interpretability assumptions in feature attribution explanations
Jonathan Dinu, Jeffrey Bigham, J. Zico Kolter
TL;DR
Research on explainable AI has often asserted interpretability without human evaluation, despite limited behavioral evidence and concerns about inconsistent measures and experimental confounders. The authors ran a 796-person human-subjects experiment using pairwise model-accuracy judgments. They found no evidence that the number of features affected participants’ ability to identify a more accurate model, while noting limits to external validity.
Problem
Limited behavioral inquiry and inconsistent interpretability measures leave unclear whether explanations help people understand algorithmic systems, while experimental confounders can make evaluations misleading.
Source: §1 · pp. 1–2Method
A mixed between- and within-subjects experiment with 796 participants used pairwise model comparisons and a psychometric model to infer participants’ ability to interpret.
Source: §3 · p. 2Results
The estimated effect of the number of features on identifying a more accurate model was 0.02 (95% CI=[-0.03,0.06]), providing no evidence of an effect.
Source: §4 · p. 6Takeaways & Limitations
The authors argue that model simulatability should be examined as a proxy for interpretability’s external validity, rather than treating their results as challenging prior experiments’ internal validity.
Source: §4 · p. 6The authors describe external validity as questionable because universal interpretability is ill-defined and exhaustive evaluation across heterogeneous people, data, methods, tasks, and hyperparameters is intractable.
Source: §5 · p. 6
Abstract
from arXiv · showhide
As machine learning and algorithmic decision making systems are increasingly being leveraged in high-stakes human-in-the-loop settings, there is a pressing need to understand the rationale of their predictions. Researchers have responded to this need with explainable AI (XAI), but often proclaim interpretability axiomatically without evaluation. When these systems are evaluated, they are often tested through offline simulations with proxy metrics of interpretability (such as model complexity). We empirically evaluate the veracity of three common interpretability assumptions through a large scale human-subjects experiment with a simple "placebo explanation" control. We find that feature attribution explanations provide marginal utility in our task for a human decision maker and in certain cases result in worse decisions due to cognitive and contextual confounders. This result challenges the assumed universal benefit of applying these methods and we hope this work will underscore the importance of human evaluation in XAI research. Supplemental materials -- including anonymized data from the experiment, code to replicate the study, an interactive demo of the experiment, and the models used in the analysis -- can be found at: https://doi.pizza/challenging-xai.
1 Introduction
As algorithmic systems enter everyday and high-stakes settings, interest in understanding users’ perceptions and system behavior has grown. Yet little behavioral inquiry has tested what makes an algorithm understandable or whether interpretability is beneficial, while inconsistent measures and experimental confounders complicate evaluation.
1 Introduction
- Growing use of algorithmic systems and societal and regulatory concerns have increased interest in how people perceive and interact with them.The passage cites the GDPR’s “right to explanation” alongside broader public apprehension.Source: §1 · p. 1
- Little behavioral inquiry has established what makes an algorithm understandable or whether interpretability is desirable and beneficial.Source: §1 · p. 1
- Researchers’ differing definitions of interpretability make systematic comparisons difficult without running new human-subjects experiments.Source: §1 · p. 1
- Uncontrolled confounders, including interface colors, can overstate evidence and contribute to unreplicable or misleading XAI evaluations.Source: §1 · p. 2
2 Axiomatic assumptions
The paper identifies three assumptions about interpretability that it tests for post-hoc feature attribution explanations. They concern model simplicity, the generality of model-agnostic explanations, and whether any explanation helps decision-making.
2 Axiomatic assumptions
- The paper tests three assumptions about post-hoc feature attribution explanations, while noting that favorable interpretability claims can go too far.Source: §2 · p. 2
- Simple models such as linear regressions and decision trees are often presumed more interpretable than complex models, although complexity can mislead as a proxy.Source: §2 · p. 2
- Model-agnostic post-hoc methods are treated as data-, task-, and user-agnostic, but external factors can strongly affect human interpretation.Source: §2 · p. 2
- The assumption that any explanation improves decisions is challenged by fragile or adversarially vulnerable methods and by human cognitive factors and biases.Source: §2 · p. 2
3 Replication as retrospective
The study uses a human-subjects experiment to test explanation methods in a model-selection task, with controlled explanation and data conditions. It frames the experiment as a test of generalizability rather than an exact reproduction of earlier studies.
3 Replication as retrospective
- Prior empirical XAI studies used disparate proxies for interpretability, which the paper describes as complicating comparisons and potentially misleading conclusions.Source: §3 · p. 2
- 796 participants completed a mixed between- and within-subjects experiment, with interpretability inferred from pairwise choices using a psychometric model.The authors state that interpretability is not directly measurable and model latent ability from measurable comparisons.Source: §3 · p. 2
- Participants compared explanations for Airbnb price-prediction models and selected which model would perform more accurately in the real world.The task was designed to emulate an authentic setting while minimizing confounding from differences in domain knowledge.Source: §3 · pp. 2–3
- To create a ground truth, models trained on New York City or Los Angeles listings were tested on the same New York City listings, making the Los Angeles model perform worse.Regularization was manipulated so models had comparable validation accuracy within each city, while Los Angeles models had higher validation accuracy across cities.Source: §3 · p. 3
- The experiment compared a post-hoc attribution method, ridge and lasso regression, and random feature importances as a placebo explanation.The data-sparsity condition contrasted 19 numeric features with a version adding eight features, including continuous and one-hot encoded categorical features.Source: §3 · p. 4
- Each subject saw one explainer-sparsity combination; top-n features and explained items varied across ten randomized trials, with all attributions shown as identically styled bar charts.Source: §3 · p. 4
- The study was not an exact reproduction of prior experiments; the authors instead sought similar conclusions if earlier effects were generalizable and replicable.Source: §3 · p. 4
4 Disentangling Interpretability8
The experiment found that placebo explanations sometimes coincided with above-chance decisions, and that explainer performance varied with data sparsity. A psychometric model also revealed substantial variation across people and items, while the study probes the external validity of prior findings rather than recreating them precisely.
4 Disentangling Interpretability8
- 54.8% of responses were correct with random explanations overall, including 49.5% on dense data and 60.2% on sparse data.The authors note that the apparent advantage on sparse data may reflect choosing the model with lower displayed error, but say this mechanism needs testing.Source: §4 · pp. 4–5, Table 2 · p. 5
- Choosing the model with lower displayed error could explain the placebo pattern, yielding correct choices in 5/10 dense and 6/10 sparse examples, but this mechanism remains unconfirmed.The authors call for a proper experiment to confirm or disconfirm this possibility.Source: §4 · pp. 4–5
- A Bayesian multilevel logistic regression modeled responses with person ability and item easiness parameters, alongside experimental-factor interactions and demographic covariates.The authors also describe it as a one-parameter item response model with item and person covariates.Source: §4 · p. 5
- Ridge performed best overall, but sparse data reversed the ranking (ridge worst, SHAP best); feature count showed no evidence of affecting model-accuracy discernment (E=0.02, 95% CI=[-0.03,0.06]).The authors say their results probe the external validity of prior findings, not the internal validity of Poursabzi-Sangdeh et al. [62].Source: §4 · pp. 5–6
- The study tests generalizability in a similar novice Mechanical Turk population and housing-price task, but does not precisely recreate the prior experiment.Its authors frame the question as external validity of model simulatability as a proxy for interpretability.Source: §4 · p. 6
- Item variation was more discriminative than person ability, and the authors report that item-parameter variance dominated group-level variation.Figure 2 shows posterior uncertainty intervals for model parameters, item easiness, and sorted person ability; the authors caution that their single person-ability parameter cannot identify the source of intra-person variation.Source: §4 · p. 6, Figure 2 · p. 6
5 Limitations
The paper’s measured scope is narrow, and its evaluation also has limits in how it characterizes variation. The authors caution that exhaustive evaluation is intractable and that a non-normal item-parameter distribution complicates interpreting its standard deviation.
5 Limitations
- Heterogeneity among people and data, plus the large design space of methods, tasks, and hyperparameters, makes exhaustive evaluation intractable.The authors also note that a universal measure of interpretability is ill-defined.Source: §5 · p. 6
- For the given experiment, the distribution of βitem[i] is very not normal, so its standard deviation cannot be interpreted using the convenient 68–95–99.7 rule.The latent ability parameters are likely correlated with prior knowledge, data-mining experience, and perhaps intrinsic personality traits.Source: §5 · p. 6
- The study measured whether a novice could infer which of two models had better test-set accuracy from ranked feature importances.The authors describe the findings as having questionable external validity and note that a universal interpretability measure is ill-defined.Source: §5 · p. 6
6 Conclusion
The authors argue that XAI research has prioritized complex explanation methods and proxy-measure benchmarks over usable technology. They call for broader evaluation and a critical lens on technical research.
6 Conclusion
- The authors argue that pursuing complex explanation methods and proxy-measure benchmarks has displaced attention from building more usable and humane technology.They call for evaluating more systems and approaching technical research with a critical lens.Source: §6 · p. 7
A Experimental protocol
The appendix provides access to experimental materials and identifies protocol figures, feature tables, and supplementary tables. It distinguishes dense features from additional features in the sparse variant.
A Experimental protocol
- The appendix links to the code, data, and parameters used in the experiment, and identifies figures for subject instructions and the experimental design.It also names tables for black-box model accuracy and the exit survey.Source: A Experimental protocol · p. 11, Figure 3 · p. 11, Figure 4 · p. 11
- The dense-feature table states that review scores are scaled from Airbnb’s 5-star rating to a numeric range of 0-10; listed features include the number of bathrooms, bedrooms, and beds in a listing.Source: A Experimental protocol · p. 12, Table 3
- The sparse variant adds eight features to the dense set, including two continuous and six one-hot encoded categorical features.The appendix identifies the added features in Table 4.Source: Table 4 · p. 13
B.0.1 Experiment
The experiment’s implementation used established tools for data preparation, modeling, explanations, visualization, web delivery, storage, and environment management.
B.0.1 Experiment
- The implementation used Inside Airbnb data, scikit-learn models, SHAP explanations, Altair visualizations, a Flask web app, PostgreSQL storage, and Docker-managed Pipenv.The web app was served with gunicorn.Source: §B.0.1 · p. 15
B.0.2 Analysis
The analysis used an R-based workflow spanning Bayesian modeling, model criticism, visualization, and reproducible typesetting.
B.0.2 Analysis
- Bayesian models were specified with brms and fit using Stan.Source: §B.0.2 · p. 15
- Model criticism used bayesplot and loo, while figures were created with ggplot2 and themed with cowplot.Source: §B.0.2 · p. 15
- The workflow used renv for a reproducible R environment and LaTeX for typesetting.Source: §B.0.2 · p. 15