Source-linked AI summary
How do Humans Understand Explanations from Machine Learning Systems? An Evaluation of the Human-Interpretability of Explanation
Menaka Narayanan, Emily Chen, Jeffrey He, Been Kim, Sam Gershman, Finale Doshi-Velez
TL;DR
Human interpretability remains poorly characterized, especially regarding which explanation properties matter for verification. The paper conducts controlled user studies with decision-set explanations across two parallel domains. It finds that complexity generally slows verification and lowers satisfaction, with lines and new concepts having stronger effects than repeated variables.
Problem
Research has proposed many explanation forms but provides limited evidence about which properties humans can use effectively in specific contexts.
Method
The paper conducts controlled human-subject experiments testing verification of decision-set explanations across explanation size, cognitive chunks, repeated terms, and two matched domains.
Results
Increased explanation complexity generally raises response time and lowers satisfaction, with new concepts and line count affecting response time more than variable repetition.
Takeaways & Limitations
The findings provide initial evidence about which explanation factors to optimize when designing machine learning rationales for human verification.
Takeaways & Limitations
The experiments cannot distinguish whether complexity slows responses because it induces more careful thinking or because it increases errors, and the findings require testing on other tasks.
Abstract
from arXiv · showhide
Recent years have seen a boom in interest in machine learning systems that can provide a human-understandable rationale for their predictions or decisions. However, exactly what kinds of explanation are truly human-interpretable remains poorly understood. This work advances our understanding of what makes explanations interpretable in the specific context of verification. Suppose we have a machine learning system that predicts X, and we provide rationale for this prediction X. Given an input, an explanation, and an output, is the output consistent with the input and the supposed rationale? Via a series of user-studies, we identify what kinds of increases in complexity have the greatest effect on the time it takes for humans to verify the rationale, and which seem relatively insensitive.
1 Introduction
The paper asks which explanation properties humans can use most easily for verification, focusing on whether an output is consistent with an input and rationale. It empirically examines decision-set explanations and argues that controlled human studies can guide interpretability design.
- Motivation: Interpretability research has proposed many explanation forms, but offers limited evidence about when they are appropriate or what human reasoning limits apply.The paper contrasts face-validity judgments with the need for evidence about explanation properties and contexts.
- Research question: The main question is which decision-set properties most affect humans’ ability to utilize explanations for verification.The study varies explanation properties while presenting explanations that could have been machine-generated.
- Research question: The study focuses on verification: given an input, explanation, and output, humans determine whether the output is consistent with the other two.Verification is treated as the simplest intrinsic explanation-use task.
- Explanation form: Decision sets contain clauses in disjunctive normal form that map input conditions to outputs, allowing users to scan for an applicable rule.The paper selects decision sets because they are presented as relatively easy to parse for a specific instance.
- Contribution: Almost all increases in explanation complexity lengthen response times, but line count and newly introduced concepts matter more than variable repetition.The paper also reports an unexpected preference for one complex line over multiple simpler lines defining intermediate concepts.
2 Related Work
Prior work evaluates explanation formats and application-specific effects, but rarely identifies which explanation properties matter most in particular contexts. The paper positions controlled human-subject experiments as a way to obtain more generalizable evidence about human interpretability.
- Interpretability methods: Interpretable machine learning includes simple regressions, rule-based methods, local black-box explanations, and visualizations, while interpretability measures often remain informal.Researchers commonly appeal to whether results make sense to human readers rather than using standardized human-factor measures.
- Psychological foundations: Psychology research suggests that people prefer simple, probable explanations and that human explanations often use causal structure and counterfactuals.Working-memory limits are also cited as relevant to explanation design.
- Prior evaluations: Existing A-B studies compare explanation representations, finding performance differences between examples and non-examples, decision sets and rule lists, and alternative classifier formats.These studies provide quantitative evaluations of human interpretability.
- Research gap: Prior work rarely identifies which properties are essential in which contexts, limiting insight into whether findings generalize.The paper addresses this gap through controlled comparisons of explanation properties.
- Application contexts: Application-focused studies examine effects such as trust, decision quality, and mental-model accuracy, including the roles of soundness and completeness.Some studies report that larger models can sometimes be more interpretable.
3 Methods
The study uses controlled human verification experiments with decision-set explanations, varying explanation size, cognitive chunks, repeated terms, and domain while holding corresponding task structures parallel. Participants judge whether recommendations are consistent with inputs and explanations.
- Design: The experiments vary three explanation factors: size, newly introduced cognitive chunks, and repeated terms.These factors were selected following pilot studies.
- Domain variation: Two parallel domains compare familiar recipe recommendations with unfamiliar clinical decision support while matching inputs, outputs, categories, and explanation forms.The design tests domain effects while controlling other task structure.
- V1: Explanation Size: Explanation size varies decision-set length from 2, 6, or 10 lines and output-clause length from 2 or 5 terms.Each combination was tested twice, producing 12 questions per experiment.
- V2: Cognitive Chunks: Cognitive-chunk experiments vary 1 to 5 introduced concepts and whether each is explicit or embedded within the explanation.All chunks were necessary for verification.
- V3: Repeated Terms: Repeated-term experiments vary input-condition appearances from 1 to 5 while holding line count and clause length constant.Each combination was tested twice, producing 10 questions per experiment.
- Task control: Recommendations were consistent with the explanation and input 50% of the time, preventing participants from learning a constant response pattern.The interface also supplied dictionaries for relevant food or medication concepts.
- Participants and procedure: Six experiments recruited 100 subjects each through Amazon Mechanical Turk, with tutorials, practice questions, randomized core questions, and accuracy prioritized over speed.Each participant completed only one experiment, and experiments were kept short to reduce fatigue.
4 Results
Across six experiments, explanation complexity reliably increased response times and reduced satisfaction, while accuracy varied little. These patterns were broadly consistent across recipe and clinical domains, despite uncertainty about whether slower responses reflected greater care or greater error risk.
- Participants: 600 subjects were recruited across six experiments, with 100 subjects per experiment.Participants were primarily from the US or Canada, generally under 50, and mostly had Bachelor’s degrees.
- Participants: Participants who failed practice-question criteria were excluded, which may have artificially increased overall accuracy rates.The filtering retained participants who could already perform the task to a reasonable extent.
- Complexity effects: Greater complexity increased response times and reduced satisfaction across explanation variations.Adding lines produced the largest response-time increases, while variable repetition had a more subtle effect; explicit new concepts also consistently increased response time.
- Complexity effects: Participants took longer and reported lower satisfaction when new cognitive chunks were explicit rather than implicitly embedded.The authors suggest this may reflect the presence of multiple relevant lines, but the mechanism remains open for future experiments.
- Accuracy: Different explanation variations had little effect on accuracy.The authors suggest that instructions prioritizing accuracy and participant filtering may have shifted difficulty effects toward response time rather than errors.
- Domain comparison: General metric trends were consistent across recipe and clinical domains, although some effects were weaker or unclear and clinical explanations received lower satisfaction.No effect was clearly reversed between domains.
5 Discussion and Conclusion
The paper identifies which explanation complexities most affect human verification, showing that explanation size and new cognitive chunks matter more than repeated variables. It also frames these findings as an empirical basis for deciding what explanation systems should optimize, while leaving broader task and context generalization open.
- Implications and scope: These results provide empirical guidance about which explanation factors to optimize for human ability to process explanations and support future study of faithfulness.The paper notes that such knowledge may help expand toward faithfulness with minimal sacrifices in human processing ability.
- Empirical findings: Explanation size, new cognitive chunks, and repeated terms were evaluated for their effects on verifying output consistency with an input and explanation across two controlled domains.The study varied explanation size, newly introduced cognitive chunks, and repeated terms while controlling everything except domain.
- Empirical findings: Increases in explanation complexity increase response time and decrease subjective satisfaction.
- Empirical findings: New concepts and the number of lines increase response time more than variable repetition or longer lines.
- Empirical findings: Embedding a new concept was faster to process and more appealing than creating a new definition.
- Implications and scope: The findings require further verification on other tasks and contexts, including forward simulation, counterfactual reasoning, regression, and high-dimensional inputs.
Description of Pilot Studies
Pilot studies shaped both the response procedure and the explanation factors used in the main experiments. Fast-response instructions reduced accuracy, while preliminary regressions identified several factors that affected response time.
- Response procedure: Quick-response or time-limited instructions produced much lower accuracies because subjects preferred answering near the deadline rather than risking omission.
- Interface design: Interface choices involving block placement, text coloring, rule-versus-narrative presentation, and rule ordering affected response time and accuracy.
- Interface design: The researchers selected the interface configuration with the highest accuracy and user satisfaction.
- Factor selection: Preliminary regressions found no significant effect difference between conjunctions and disjunctions.
- Factor selection: Number of lines, global variables, and output-clause length affected response time regardless of whether clause length came from conjunctions or disjunctions.