Source-linked AI summary
Quantifying Interpretability and Trust in Machine Learning Systems
Philipp Schmidt, Felix Biessmann
TL;DR
Evaluating interpretability and trust in ML systems lacks unified quantitative measures. This paper introduces information-transfer metrics and finds that COVAR explanations are more interpretable than LIME while explanations more than double annotation productivity and a trust score detects excessive reliance on ML predictions.
Problem
Interpretability quality and trust in ML predictions are difficult to measure, limiting quantitative evaluation and comparison of interpretability methods.
Source: Conclusion · p. 7Method
The paper measures interpretability through the information transfer rate of humans replicating model predictions and derives a trust coefficient comparing reliance on model predictions with true labels.
Source: §4 · pp. 3–4Results
COVAR yields more interpretable explanations than LIME, while explanations more than double annotation productivity and the trust measure identifies decisions overly biased toward ML predictions.
Source: Conclusion · p. 7Takeaways & Limitations
The proposed metrics provide quantitative bases for comparing interpretability methods, assessing human–AI productivity, and detecting when ML predictions dominate human judgement.
Source: Conclusion · p. 7The LIME evaluation assumes perturbations usually produce uncorrelated features, allowing its linear approximation parameters to be interpreted directly.
Source: §4 · p. 5
Abstract
from arXiv · showhide
Decisions by Machine Learning (ML) models have become ubiquitous. Trusting these decisions requires understanding how algorithms take them. Hence interpretability methods for ML are an active focus of research. A central problem in this context is that both the quality of interpretability methods as well as trust in ML predictions are difficult to measure. Yet evaluations, comparisons and improvements of trust and interpretability require quantifiable measures. Here we propose a quantitative measure for the quality of interpretability methods. Based on that we derive a quantitative measure of trust in ML decisions. Building on previous work we propose to measure intuitive understanding of algorithmic decisions using the information transfer rate at which humans replicate ML model predictions. We provide empirical evidence from crowdsourcing experiments that the proposed metric robustly differentiates interpretability methods. The proposed metric also demonstrates the value of interpretability for ML assisted human decision making: in our experiments providing explanations more than doubled productivity in annotation tasks. However unbiased human judgement is critical for doctors, judges, policy makers and others. Here we derive a trust metric that identifies when human decisions are overly biased towards ML predictions. Our results complement existing qualitative work on trust and interpretability by quantifiable measures that can serve as objectives for further improving methods in this field of research.
1. A quality score for interpretability methods
Empirical evidence shows that interpretability can more than double productivity in audit tasks.
1. A quality score for interpretability methods
- Interpretability more than doubled audit productivity in empirical evaluation.The passage reports empirical evidence for this productivity gain.Source: §1 · p. 1
- The reported productivity improvement concerns auditing tasks.Source: §1 · p. 1
- The finding provides empirical support for interpretability’s practical value.Source: §1 · p. 1
3. Comparisons of popular interpretability approaches · 4. A quantifiable trust score for ML models
The paper compares interpretability methods using information transfer rate (ITR), measuring how accurately and quickly humans replicate ML predictions, and derives a trust coefficient contrasting reliance on model predictions with reliance on true labels. Experiments show that explanations improve annotation accuracy and speed, while trust scores reveal when humans follow erroneous model predictions.
3. Comparisons of popular interpretability approaches
- The evaluation contrasts qualitative or task-specific interpretability work with a quantitative, human-centered approach that targets intuitive understanding rather than computational efficiency or perturbation robustness.The study uses crowdsourcing to scale evaluations with lay annotators, while recognizing the tension between faithful explanations and explanations humans can understand (Herman 2017; Lipton 2016).Source: §4 · pp. 2–3
Comparing Interpretability Approaches
- ITR measures intuitive understanding through mutual information between human annotations and model predictions, normalized by annotation time, enabling comparisons across models, tasks, and interpretability approaches.Explanations are evaluated by comparing ITR with and without explanations; randomly highlighted words provide a sanity-check condition.Source: §4 · p. 3, Results on IMDb sentiment · p. 6
4. A quantifiable trust score for ML models
- The trust coefficient compares ITR based on human–model agreement with ITR based on human–true-label agreement, incorporating both label agreement and decision time.Values above one indicate stronger bias toward model predictions; values below one indicate greater reliance on human judgment.Source: §4 · pp. 3–4
Experiments
- The experiments use book categorization and IMDb sentiment datasets with L2-regularized multinomial logistic regression, comparing model-agnostic LIME (Ribeiro, Singh, and Guestrin 2016) with covariance-based COVAR.The methods were selected for general applicability, popularity, simplicity, and speed, although the experiments focus on text classification.Source: §4 · pp. 4–5
Results on Book Categories
- Explanations increased book-category annotation accuracy by more than 4% on average, with COVAR slightly outperforming LIME and response times decreasing in all but one class.These results compare both explanation methods against the no-explanation control condition.Source: Results on Book Categories · p. 6
Results on IMDb sentiment
- Accuracy rose from 78.01% without explanations to 81.72% with LIME and 84.52% with COVAR on IMDb sentiment, while explanations also reduced response times.The improvement was statistically significant for accuracy and annotation times (p < 0.01).Source: Results on IMDb sentiment · p. 6
Comparing Interpretability Approaches
- COVAR explanations were 53 times faster to compute than LIME explanations, taking 0.014±0.008 seconds versus 0.750±0.111 seconds per test instance.LIME used 2500 perturbations, whereas COVAR computed top pattern activations.Source: Results on IMDb sentiment · p. 7
Quantifying Trust in ML Predictions
- In book categorization, the trust coefficient below one indicates that humans retained their own judgment, whereas in IMDb sentiment, a coefficient above one indicates bias toward ML predictions even when they were wrong.Without explanations or random highlights, humans generally did not replicate model mistakes, but the IMDb condition showed excessive reliance on model predictions.Source: Explanations improve audit efficiency · p. 7
Conclusion
The paper proposes quantitative metrics for comparing interpretability methods through human replication of ML predictions and for detecting excessive human bias toward ML outputs. Experiments show that explanations can more than double annotation productivity, while the trust metric identifies decisions that diverge from independent judgment.
Conclusion
- The proposed information-transfer-rate metric quantifies interpretability quality in bit/s, enabling comparisons across tasks, models, and interpretability approaches.It addresses the absence of a gold-standard strategy for evaluating interpretability methods.Source: Conclusion · p. 7
- Explanations more than doubled auditors’ productivity in an annotation task, supporting interpretability as a basis for improving AI-assisted technology.The finding provides empirical evidence that interpretability improves interaction between humans and AI systems.Source: Conclusion · p. 7
- The proposed trust metric identifies cases where human decisions deviate from personal judgment and instead reflect what an ML system would have predicted.The measure is intended to help assess ethical and legal boundaries of algorithmic decision making.Source: Conclusion · p. 7