Source-linked AI summary
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Marco Tulio Ribeiro, Sameer Singh, Carlos Guestrin
TL;DR
Machine-learning users need explanations to assess whether individual predictions and entire models can be trusted. The paper introduces LIME and SP-LIME, finding that explanations support trust-related tasks across text and image models for expert and non-expert users.
Problem
Users need to assess whether individual predictions and deployed models can be trusted, but standard validation metrics may not reflect real-world performance or product goals.
Method
LIME locally approximates any classifier with an interpretable model, while SP-LIME selects representative, non-redundant predictions to provide a global model view.
Results
Explanations were useful across trust-related tasks involving text and image models, including model selection, trust assessment, classifier improvement, and prediction insight.
Takeaways & Limitations
LIME explanations help expert and non-expert users evaluate models and predictions in several trust-related settings.
Takeaways & Limitations
The paper does not specify how to select representative image predictions and identifies this as a limitation for future work.
Abstract
from arXiv · showhide
Despite widespread adoption, machine learning models remain mostly black boxes. Understanding the reasons behind predictions is, however, quite important in assessing trust, which is fundamental if one plans to take action based on a prediction, or when choosing whether to deploy a new model. Such understanding also provides insights into the model, which can be used to transform an untrustworthy model or prediction into a trustworthy one. In this work, we propose LIME, a novel explanation technique that explains the predictions of any classifier in an interpretable and faithful manner, by learning an interpretable model locally around the prediction. We also propose a method to explain models by presenting representative individual predictions and their explanations in a non-redundant way, framing the task as a submodular optimization problem. We demonstrate the flexibility of these methods by explaining different models for text (e.g. random forests) and image classification (e.g. neural networks). We show the utility of explanations via novel experiments, both simulated and with human subjects, on various scenarios that require trust: deciding if one should trust a prediction, choosing between models, improving an untrustworthy classifier, and identifying why a classifier should not be trusted.
1. INTRODUCTION
The paper frames trust in machine learning as requiring both understandable individual predictions and confidence in a model’s real-world behavior before deployment. It introduces LIME and SP-LIME to address these needs and evaluates explanations through simulated and human-subject experiments.
- Motivation: Trust matters because users may not use machine learning models or predictions they do not trust, whether as tools or within deployed products.The introduction distinguishes trusting individual predictions from understanding a model’s behavior rather than treating it as a black box.
- Motivation: Individual predictions require scrutiny in high-stakes decisions, where acting on blind faith can have catastrophic consequences.The paper cites medical diagnosis and terrorism detection as examples.
- Motivation: Whole-model trust requires confidence that performance will generalize to real-world data and that evaluation metrics reflect the product’s goals.Available validation data may differ significantly from real-world data, and its accuracy metrics may not be indicative of product performance.
- Contributions: LIME explains predictions from any classifier or regressor by approximating the model locally with an interpretable model.The method is presented as a faithful local explanation technique.
- Contributions: SP-LIME selects representative instances and their explanations to address trusting the model through submodular optimization.The method addresses model-level trust by selecting multiple predictions and explanations.
- Evaluation: Simulated and human-subject evaluations measure explanations’ effects on trust and related tasks, including model selection and feature engineering.Non-experts using LIME selected the classifier that generalized better in the real world and improved an untrustworthy classifier trained on 20 newsgroups.
2. THE CASE FOR EXPLANATIONS
Faithful, intelligible explanations connect an instance’s components to a model’s prediction, helping people decide whether to trust and use machine learning effectively. They also support broader assessment of model reliability and comparison when validation performance may not reflect real-world behavior.
- Individual prediction explanations: Explanations present textual or visual artifacts that qualitatively show how an instance’s components relate to the model’s prediction.Examples include words in text, patches in images, or symptoms associated with a diagnosis.
- Individual prediction explanations: Faithful and intelligible explanations help humans trust and use machine learning effectively.The paper frames explanation quality as important because people may act on predictions or decide whether to deploy a model.
- Individual prediction explanations: Individual prediction explanations give domain experts evidence for deciding whether to trust a model’s output.In the flu example, symptoms receive relative weights indicating support for or opposition to the prediction, allowing a doctor to use prior knowledge.
- Overall model trust: Validation performance alone may not establish overall model trust because held-out evaluation can diverge from performance “in the wild”.Practitioners may overestimate model accuracy, so automated validation cannot be the sole basis for trust.
- Comparing models: Individual explanations can reveal that a higher-accuracy validation model is worse than an alternative, supporting model selection alongside accuracy.Human prior knowledge makes this difference easier to detect from explanations than without them.
Desired Characteristics for Explainers
The paper identifies interpretability, local fidelity, and model agnosticism as essential characteristics of explanation methods. It also emphasizes representative explanations that provide a global perspective for assessing trust in a model.
- Interpretability: Explanations must be interpretable, providing qualitative understanding of how input variables relate to the response while accounting for users’ limitations.A linear, gradient-based, or additive explanation may not be interpretable for every user or context.
- Interpretability: Explanations should remain understandable when many features contribute, potentially using inputs different from the model’s original features.Hundreds or thousands of significant features can make individual weights impractical for users to comprehend.
- Local Fidelity: Explanations must be locally faithful, corresponding to the model’s behavior near the instance being predicted.Local fidelity does not imply global fidelity because globally important features may be unimportant in a particular local context.
- Global Perspective: A global perspective can support trust by presenting a few representative explanations built from individual prediction explanations.The goal is to select explanations that are representative of the model rather than evaluating it solely through accuracy.
3. LOCAL INTERPRETABLE MODEL-AGNOSTIC EXPLANATIONS
LIME explains individual predictions by learning an interpretable model that is locally faithful to any classifier, using human-understandable representations and locality-weighted perturbations. Sparse linear explanations highlight influential words or image regions, while their limitations depend on the representation and explanation family.
- Core objective: LIME identifies an interpretable model that is locally faithful to the classifier around the prediction being explained.Its goal is local rather than global fidelity.
- Interpretable representations: Interpretable representations are human-understandable inputs that may differ from the features used by the underlying model.For text, a binary word-presence representation can explain a classifier using more complex features.
- Optimization: LIME balances local fidelity against explanation complexity and can use different explanation families, fidelity functions, and complexity measures.The paper focuses on sparse linear models learned through perturbations.
- Model-agnostic learning: To remain model-agnostic, LIME samples perturbed interpretable inputs, obtains black-box predictions, and weights them by proximity to the instance.The learned explanation is locally, but not globally, faithful.
- Applications: For text, LIME uses a bag-of-words representation with a limit K on the number of words; for images, it highlights influential superpixels.The image example sets K = 10 and explains the top three predicted classes.
- Trust and limitations: LIME can expose untrustworthy reasoning despite 94% held-out accuracy and can clarify reasonable image predictions, but representations or sparse linear models may fail to capture some behaviors.The text example reveals reliance on arbitrary header words and poster names, while the image example links an acoustic-guitar prediction to its fretboard.
5. SIMULATED USER EXPERIMENTS
Simulated experiments evaluate whether explanations are faithful, help assess trust in individual predictions, and support model-level evaluation. Across these tasks, LIME achieves strong results, including superior trust assessment and improved classifier selection when paired with submodular pick.
- Experimental setup: The experiments test explanation faithfulness, individual-prediction trust assessment, and whole-model evaluation across sentiment classifiers and datasets.The study uses books and DVDs sentiment datasets with multiple classifier families and compares LIME with Parzen, greedy, and random procedures.
- Faithfulness: Faithfulness is measured by recall of gold-important features recovered from explanations for sparse logistic regression and decision-tree classifiers.Both interpretable classifiers use at most 10 features per instance, enabling direct comparison against known important-feature sets.
- Trust assessment: LIME dominates competing explainers in trustworthy-prediction F1 on both datasets and all black-box models, with all results significant at p = 0.01.Competing methods either mistrust too many predictions or trust too many, whereas LIME maintains high precision and high recall.
- Model selection: For model selection, LIME consistently outperforms greedy explanations, while submodular pick combined with LIME performs best, especially when users see few examples.The experiment averages classifier-selection accuracy over 800 runs and excludes Parzen combinations because their explanations were only slightly better than random.
6. EVALUATION WITH HUMAN SUBJECTS
Human-subject experiments show that LIME and SP-LIME explanations help users choose better-generalizing classifiers, improve models through feature engineering, and detect why a classifier should not be trusted. Submodular instance selection is especially useful for classifier selection and feature engineering.
- 6.2 Choosing Between Classifiers: Explanations helped users identify the better-generalizing classifier, whereas relying on test-set accuracy would select the wrong model.Submodular pick greatly improved selection over random pick, and LIME outperformed greedy for both selection strategies.
- 6.3 Feature Engineering: SP-LIME enabled crowd workers to improve the model by removing features they judged unimportant, outperforming RP-LIME in feature engineering.The experiment used explanations to guide iterative word deletion and measured generalization on a separate religion dataset.
- 6.3 Feature Engineering: 3.6 minutes per round was the average cleaning time, producing a better-generalizing classifier in just under 11 minutes.Each path removed an average of 200 words with SP and 157 with RP.
- 6.4 Identifying Classifier Irregularities: After examining explanations, almost all subjects identified that snow drove the husky-versus-wolf classifier’s predictions, while trust in the classifier dropped substantially.Before explanations, more than a third trusted the classifier and fewer than half identified the snow pattern.
7. RELATED WORK
Prior work addresses model-trust problems through instance-navigation tools, failure prediction, interpretable models, architecture-specific explanations, and model-agnostic surrogates. LIME differs by providing general explanations that approximate any classifier locally, prioritizing local fidelity and interpretability.
- Trust and failure analysis: Validation accuracy alone can mislead practitioners through overconfidence, feedback loops, and data leakage; Gestalt and Modeltracker help navigate individual instances but do not address the broader problem.These tools are complementary to LIME in explaining models.
- Trust and failure analysis: Failure-prediction methods can increase trust by warning about likely errors, but they require vision-specific annotations or engineering, lack reasons for distrust, and assume reliable evaluation metrics.They may therefore miss problems such as data leakage.
- Interpretable and architecture-specific explanations: Interpretable models support trust assessment but can sacrifice flexibility, accuracy, or efficiency, while existing text and vision explanations are restricted to particular models or architectures.Examples include a supersparse linear model with 5 −10 features for medical applications and vision systems based on object detection or attention.
- Model-agnostic explanations: Model-agnostic surrogates and gradient explanations capture locality but struggle with global approximation, near-zero gradients, or local fidelity; LIME instead learns an interpretable model locally around each prediction.Prior perturbation methods focused on specific contribution models, whereas LIME provides a general framework applicable to classifiers or regressors.
8. CONCLUSION AND FUTURE WORK
The paper argues that trust is essential for effective human interaction with machine learning systems and presents LIME and SP-LIME to support interpretable individual and global explanations. Future work includes exploring additional explanation families, studying them with real users, and addressing image-specific picking.
- Conclusion: LIME is a modular, extensible approach for faithfully explaining any model’s predictions in an interpretable manner.The paper identifies explaining individual predictions as important for assessing trust.
- Conclusion: SP-LIME selects representative, non-redundant predictions to provide users with a global view of the model.
- Future Work: Future work will explore explanation families beyond sparse linear models, including decision trees, and compare them with real users.The framework supports exploring a variety of explanation families.
- Future Work: The authors identify image-specific prediction picking as an unaddressed limitation to resolve in future work.