Source-linked AI summary
Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior?
Peter Hase, Mohit Bansal
TL;DR
The paper asks whether algorithmic explanations improve people’s ability to predict model behavior, addressing limited evidence that isolates explanation effects. It evaluates five explanation methods through controlled forward and counterfactual human simulation tests on text and tabular classification. Clear improvements appear mainly for LIME in tabular classification and Prototype in counterfactual tests, while subjective explanation ratings are not predictive of user correctness.
Problem
Prior simulation studies had not isolated whether algorithmic explanations improve simulatability, the ability to predict model behavior on new inputs.
Method
The study compares five explanation methods in forward and counterfactual human simulation tests across text and tabular classification, using controls against answer leakage and guessing.
Results
Clear simulatability improvements occur only for LIME with tabular data and Prototype in counterfactual tests, while explanation ratings are not predictive of user correctness.
Takeaways & Limitations
Simulatability metrics provide quantitative interpretability measures, but explanation evaluation must be chosen carefully because subjective quality ratings may not predict effectiveness.
Takeaways & Limitations
Text counterfactuals are regularly outside the true data distribution, limiting how directly those results represent real movie reviews.
Abstract
from arXiv · showhide
Algorithmic approaches to interpreting machine learning models have proliferated in recent years. We carry out human subject tests that are the first of their kind to isolate the effect of algorithmic explanations on a key aspect of model interpretability, simulatability, while avoiding important confounding experimental factors. A model is simulatable when a person can predict its behavior on new inputs. Through two kinds of simulation tests involving text and tabular data, we evaluate five explanations methods: (1) LIME, (2) Anchor, (3) Decision Boundary, (4) a Prototype model, and (5) a Composite approach that combines explanations from each method. Clear evidence of method effectiveness is found in very few cases: LIME improves simulatability in tabular classification, and our Prototype method is effective in counterfactual simulation tests. We also collect subjective ratings of explanations, but we do not find that ratings are predictive of how helpful explanations are. Our results provide the first reliable and comprehensive estimates of how explanations influence simulatability across a variety of explanation methods and data domains. We show that (1) we need to be careful about the metrics we use to evaluate explanation methods, and (2) there is significant room for improvement in current methods. All our supporting code, data, and models are publicly available at: https://github.com/peterbhase/InterpretableNLP-ACL2020
1 Introduction
The paper evaluates whether algorithmic explanations help people predict model behavior, using simulation tests designed to isolate explanation effects. Across five methods and text and tabular tasks, clear improvements are limited to specific settings.
- Motivation: Simulatability measures whether a person can predict a model’s behavior on new inputs, including forward and counterfactual simulation tasks.Forward simulation predicts an output for a given input; counterfactual simulation predicts the output for a perturbed input.
- Study design: The study isolates explanation effects by comparing prediction accuracy with explanations against unexplained baselines while separating explained and test instances.The design also balances examples by model correctness and requires predictions on all inputs.
- Evaluation scope: The evaluation includes five explanation methods across text and tabular classification: LIME, Anchor, Decision Boundary, Prototype, and Composite.The Composite method combines complementary explanations from the other approaches.
- Key findings: Clear improvements are limited: LIME helps tabular simulatability, while Prototype helps counterfactual simulatability across text and tabular domains.The introduction reports no definitive improvement across both tests for other methods and domains.
2 Background and Related Work
The paper frames interpretability through simulatability and reviews explanation methods and evaluation procedures. It emphasizes controls that prevent explanations from revealing answers and introduces counterfactual simulation as a scalable human evaluation.
- Conceptual framework: Interpretability is treated as a downstream property assessed through simulatability rather than a separate distinction between interpretability and explainability.Under this framework, an interpretable model is one whose behavior people can simulate.
- Explanation-method taxonomy: Feature importance estimation, case-based reasoning, and latent space traversal organize the explanation methods discussed in the paper.LIME and Anchor are selected as domain-agnostic feature-importance approaches.
- Evaluation design: The evaluation design separates explained from test instances, adds an unexplained baseline, balances model correctness, and forces predictions on every input.These choices address answer leakage, missing controls, label guessing, and bias toward overly specific explanations.
- Evaluation design: Counterfactual simulation asks users to predict model behavior on edited inputs instead of creatively editing inputs to change model outputs.The paper presents this formulation as more scalable than soliciting creative responses.
- Evaluation design: The paper contrasts its human simulation tests with automatic metrics that prescribe expected model behavior on explanation-generated counterfactuals.Automatic tests may mask important features and check whether class scores decrease.
3 Explanation Methods
The paper implements five explanation approaches spanning local feature weights, rules, latent-space paths, prototypes, and a composite combination. Each method exposes different information about model behavior through text and tabular inputs.
- LIME and Anchor: LIME fits a local linear approximation around an input and presents selected features, weights, intercept, summed weights, and predicted output.The study uses five features and class probabilities as model outputs.
- LIME and Anchor: Anchor learns high-confidence rule lists whose applicable rules indicate that an input will likely receive the original prediction.The study uses individual tokens for text features and shared learning parameters across Anchor explanations.
- Prototype model: The Prototype model maps inputs into a latent space, scores classes by similarity to class prototypes, and explains predictions using the most similar prototype.Feature importance scores are prototype-specific and estimated through feature omission.
- Prototype model: Prototype explanations show the predicted class score, most similar prototype, and up to six feature-importance scores when magnitudes exceed a threshold.The model uses 40 prototypes for text classification and 20 for tabular classification.
- Decision Boundary: The Decision Boundary method samples around an input, selects a minimally edited counterfactual crossing the boundary, and presents a path of edits with evidence margins.Paths are built by greedily choosing edits that least change the difference between positive and negative class scores.
- Composite approach: The Composite method combines LIME, Anchor, Decision Boundary, and Prototype explanations as complementary views of model behavior.For combined explanations, only the final decision-boundary step is shown, and prototype features are initialized from the task model.
4 Experimental Design
The study evaluates explanation effectiveness through controlled human simulation tests on text and tabular classification, with matched data and balanced outcomes. It compares user accuracy changes and subjective ratings across explanation conditions.
- Data and Task Models: Experiments use movie-review sentiment data and tabular Adult income classification data.The movie-review dataset contains 10,662 reviews, while Adult contains 15,682 individual records.
- User Pool: Over 2,100 responses were gathered from 32 trained undergraduates assigned across ten dataset-method conditions.Participants had completed at least one computer science or statistics course.
- Measures: Table 1 reports accuracy change relative to Pre by domain, while Table 2 reports change by simulation-test type; Table 3 reports ratings on a 1-to-7 scale.Ratings capture whether explanations show users why the system produced its output.
- Simulation Tests: The forward test measures prediction of model outputs on new inputs before and after users receive explanations.Users first study labeled examples without explanations, then predict model behavior on new inputs after an explanation-learning phase.
- Simulation Tests: The counterfactual test asks users to predict model behavior on perturbations of given inputs, using the same items in Pre and Post rounds.The Post round adds explanations for the original inputs, so performance changes are attributed to the explanations.
- Experimental Controls: True positives, false positives, true negatives, and false negatives are equally represented to prevent success through guessing true labels.Counterfactual perturbations are sampled so each has a 50% chance of receiving the same prediction as the original input.
5 Results
Across 39 user tests and 2166 responses, the study evaluates explanation effectiveness through simulatability, user ratings, and the relationship between ratings and correctness. LIME with tabular data is the only setting showing definitive improvement across both simulation tests, while ratings are not predictive of user correctness.
- Study and measures: 2166 responses from 39 user tests provide the study’s empirical basis across explanation methods and data domains.The study analyzes explanation effectiveness using simulation tests and reports results by data domain and test type.
- Study and measures: Change measures explanation effectiveness as the difference in user accuracy across prediction phases.Hypothesis tests use a block bootstrap over users and unique task items within each condition.
- Do explanations help users?: LIME with tabular data is the only setting showing definitive improvement in both forward and counterfactual simulatability.No other method and data-domain combination shows definitive improvement across tests.
- Do explanations help users?: The Prototype method performs reliably in counterfactual tests across both domains, but not in forward tests.The authors suggest explanations may help when displayed side by side with inputs.
- How do users rate explanations?: Composite and LIME receive the highest explanation ratings in both domains, although rating variance is high.Ratings are generally higher for tabular than text data.
- Can users predict explanation effectiveness?: Explanation ratings are not predictive of user correctness in the counterfactual simulation analysis.The reported confidence intervals for rating changes include both negative and positive changes in expected correctness.
6 Qualitative Analysis
Qualitative examples show that explanations can help users track unchanged predictions, but they can also fail to identify necessary features or relevant similarities for changed predictions. These failures motivate explanations that distinguish sufficient from necessary factors and operate in an appropriate feature space.
- Successful unchanged prediction: For one counterfactual, Prototype and LIME increased correct unchanged-prediction responses from 3 of 6 before explanations to 5 of 6 afterward.Both methods supported the inference that the prediction would remain positive after similar word substitutions.
- Successful unchanged prediction: LIME supported the unchanged prediction by identifying unaffected positive words and a negative word replaced by a similar substitute.The explanation highlighted “funny” and “moment” as positive and assigned “sucks” a weight of −.23.
- Failure on changed prediction: In a changed-prediction example, only 7 of 13 responses were correct after explanations, with no method improving over Pre accuracy.Users needed to predict a change from positive to negative.
- Failure on changed prediction: Anchor’s condition on “bittersweet” did not apply when the counterfactual replaced that word with “teary.”Its probabilistic behavior description was conditioned on the original word being present.
- Failure on changed prediction: LIME assigned small weights to changed words, while Decision Boundary and Prototype did not clearly identify decision-relevant conditions for the prediction change.The Prototype explanation did not select important words or clarify when the output would change.
- Implications: Future explanations should distinguish sufficient from necessary factors and show decision-relevant similarities in the appropriate feature space.The requirement is especially important for models handling complex data.
7 Discussion
The discussion identifies practical constraints in the simulation design, including memory demands in forward tests, distributional concerns for text counterfactuals, and unequal opportunities for method-specific coverage strategies.
- Forward tests: Forward tests require users to retain insights from 16 learning examples without consulting them during prediction.Some users reported difficulty retaining those insights across later prediction rounds.
- Counterfactual generation: Text counterfactuals regularly fall outside the data distribution because their word choices do not resemble real movie reviews.The authors retain them because models still produce behavior on such inputs and the analysis seeks to assess possible behaviors.
- Fair comparison: The comparison controls the number of data points across methods, but not user exposure time or explanation-generation computation time.Coverage-budget approaches for LIME and Anchor were omitted because comparable notions were unavailable for Decision Boundary and Prototype and their usefulness for text was unclear.
8 Conclusion
The paper evaluates five explanation methods with simulation tests on text and tabular data, finding limited improvements in simulatability and no evidence that subjective ratings predict effectiveness. It concludes that evaluation metrics require care and current methods leave substantial room for improvement.
- Conclusion: Simulation tests provide a quantitative measure of interpretability by testing whether explanations improve users’ understanding of model behavior.The paper evaluates five methods across text and tabular data while isolating explanation effects on simulatability.
- Conclusion: Clear simulatability improvements appear only for LIME with tabular data and Prototype in counterfactual tests.Subjective explanation-quality ratings are not predictive of explanation effectiveness in the simulation tests.
- Conclusion: The results indicate that explanation methods should be evaluated with care and that current methods have significant room for improvement.These conclusions follow from the limited observed improvements and the mismatch between ratings and effectiveness.
A.1 Method Implementations
The study implements explanation methods and neural models for text and tabular classification, then evaluates them through forward and counterfactual simulation procedures.
- Explanation methods: LIME and Anchor use existing implementations, while the Prototype and Decision Boundary methods are implemented for both text and tabular data.The study also evaluates a Composite method combining explanations from the individual approaches.
- Task models: The tabular model is a neural network with two hidden layers of width 50, while the text model is a BiLSTM adapted for single-sentence classification.Prototype models are built on the feature-extractor layers of each task model.
- Prototype training: Prototype models initialize feature extractors from pretrained task models, prototype vectors with k-means clustering, and classifier weights according to prototype classes.The initialization procedure operates over latent representations from the training set.
- Importance scores: Prototype feature importance compares model output with a feature present against output when that feature is omitted.Text features are omitted by zeroing their embeddings; tabular importance uses an imputation-based evidence comparison.
- Simulation procedures: Forward tests measure baseline prediction performance before explained examples, whereas counterfactual tests compare accuracy with and without explanations.The forward procedure uses baseline rounds followed by rounds with explained examples.
- Counterfactual selection: Counterfactual inputs are selected from perturbations with the opposite prediction, using a distance function to choose the nearest candidate.The distance combines the count of differing features with squared Euclidean distance between latent representations, with Euclidean distance serving as a tie-breaker.
A.2 Perturbation Distributions
The experiments generate local perturbations for counterfactual selection and decision-boundary explanations, using domain-specific sampling strategies for tabular and text inputs.
- Counterfactual sampling: For both data domains, the study samples 10,000 local perturbations around each input before selecting counterfactual candidates.The candidate prediction type is conditioned on whether the desired output matches or differs from the original prediction.
- User interface: The user-testing interface includes a counterfactual Post test example using LIME for text data.Tests are administered through spreadsheets under experimenter guidance.
- Tabular data: Tabular perturbations randomly make 1 to 3 feature edits, choose edited features uniformly, and assign new values uniformly at random.A variable cannot be assigned its original value.
- Text data: Text perturbations randomly substitute words with nearby GloVe neighbors, with changes adjusted for sentence length and capped at five edited words when possible.Neighbor sampling probability is proportional to embedding similarity.
A.3 Simulation Test Design
The paper presents the experimental design in additional detail through separate depictions of each test and a screenshot of the user-testing interface.
- Test procedures: Figures 3 and 4 provide separate, more detailed representations of the forward and counterfactual simulation tests.They expand on the overall experimental design shown elsewhere in the paper.
- User testing interface: The interface example shows a counterfactual Post test with LIME for text data, administered through spreadsheets with experimenter-guided navigation.Users read test material and enter responses in the spreadsheets.