Source-linked AI summary
An Evaluation of the Human-Interpretability of Explanation
Isaac Lage, Emily Chen, Jeffrey He, Menaka Narayanan, Been Kim, Sam Gershman, Finale Doshi-Velez
TL;DR
The paper addresses limited evidence about which explanation properties make machine-learning systems human-interpretable and studies this question through controlled human-subject experiments. Across simulation, verification, and counterfactual reasoning tasks, cognitive chunks affect performance more than variable repetitions, suggesting common design principles while leaving some effects for further investigation.
Problem
Existing work provides limited guidance about which explanation complexities most influence humans’ ability to use explanations.
Method
The study uses controlled human-subject experiments to vary explanation size, cognitive chunks, and variable repetitions across three tasks and two domains.
Results
Cognitive chunks affect performance more than variable repetitions, with consistent effects across tasks, metrics, and domains.
Takeaways & Limitations
For decision sets, explanation design should attend more to new cognitive chunks and explanation length than to variable repetition.
Takeaways & Limitations
The study leaves the unexpectedly faster response times for implicitly rather than explicitly introduced cognitive chunks for further investigation.
Abstract
from arXiv · showhide
Recent years have seen a boom in interest in machine learning systems that can provide a human-understandable rationale for their predictions or decisions. However, exactly what kinds of explanation are truly human-interpretable remains poorly understood. This work advances our understanding of what makes explanations interpretable under three specific tasks that users may perform with machine learning systems: simulation of the response, verification of a suggested response, and determining whether the correctness of a suggested response changes under a change to the inputs. Through carefully controlled human-subject experiments, we identify regularizers that can be used to optimize for the interpretability of machine learning systems. Our results show that the type of complexity matters: cognitive chunks (newly defined concepts) affect performance more than variable repetitions, and these trends are consistent across tasks and domains. This suggests that there may exist some common design principles for explanation systems.
1 Introduction
Interpretable machine learning offers explanations alongside predictions, but the human factors determining whether explanations are usable remain insufficiently understood. This work uses controlled human-subject experiments to study explanation complexity across tasks and domains, finding that cognitive chunks matter more than variable repetitions.
- Interpretable machine learning systems provide explanations alongside their predictions or decisions.These explanations can help users assess whether recommendations are reasonable.
- Existing explanation methods have face validity, but offer limited guidance about when each form is appropriate or which complexity factors affect human reasoning.The introduction contrasts large and small decision trees without resolving which regularizer matters most.
- The study evaluates explanation size, cognitive chunks, and variable repetitions across three tasks, two domains, and three performance metrics.Its tasks involve simulation, verification, and determining whether a suggested response changes under changed inputs.
- Cognitive chunks affect performance more than variable repetitions, with consistent trends across tasks and domains.The result supports the possibility of common design principles for explanation systems.
2 Related Work
Prior interpretable-ML research spans many explanation forms and human-factors studies, but interpretability is often assessed through face validity or application-specific evaluations. These strands provide useful insights while leaving generalizable properties difficult to identify.
- Interpretable-ML methods include simple regressions, logic-based approaches, and representations built around human-relevant concepts.The literature covers a broad range of model and explanation forms.
- Interpretability measures are often less sophisticated than the algorithms, relying on the face validity of results.Researchers may judge an explanation by whether it appears sensible to a human reader.
- Psychology research suggests that people prefer simple, probable explanations and often use causal structure and counterfactuals.These findings offer general considerations for designing interpretable AI systems.
- A/B studies compare explanation forms on intrinsic tasks, while application-specific studies evaluate desired properties within particular domains.Examples include comparing example-based representations and studying explanations in recommendation systems.
- Generalizable explanation properties remain difficult to identify because existing studies often depend on specific explanation types and application contexts.Related work includes studies of soundness, completeness, model size, and human-recognizable intermediate predicates.
3 Methods
The study examines how properties of decision-set explanations affect human performance across controlled tasks and domains. It varies explanation size, cognitive chunks, and repeated terms while measuring task performance and controlling participants’ prior knowledge.
- Study design: Decision sets were used because people can scan for the applicable rule and choose its accompanying output.Each decision-set line contains an input clause that maps to an output when the clause is true.
- Study design: The experiments tested decision-set explanations across two parallel domains, three cognitive tasks, and three performance metrics.The domains were low-risk recipes and high-risk clinical recommendations; the metrics were accuracy, response time, and subjective satisfaction.
- Tasks and domains: Participants completed simulation, verification, and counterfactual tasks using synthetic recipe and medicine domains designed to limit outside knowledge.The domains used alien preferences and dictionaries defining relevant concepts so participants shared the same information.
- Explanation variation: The study manipulated explanation size, cognitive chunks, and repeated terms to assess which explanation properties affect usability.Size varied the number of lines and output terms; cognitive chunks varied in number and explicitness; repeated terms varied in frequency.
- Participant criteria: The study excluded participants who failed the required practice questions and six participants who took more than five minutes on one question.The authors note that practice-based exclusion may artificially increase overall accuracy while filtering participants who did not engage with the task.
- Participant criteria: Demographic proportions did not differ significantly between the experimental populations.The comparison used chi-squared tests of independence with a Yates continuity correction.
4 Results
Across six experiments, greater explanation complexity generally increased response times, with effects varying by complexity type. These patterns were broadly consistent across domains and tasks, while accuracy effects were less clear.
- Greater complexity generally increased response times, but the effect varied by complexity type.Cognitive chunks significantly affected response time in both domains; explanation size did so only in the recipe domain, while repeated variables were not significant in either domain.
- Cognitive chunks produced response-time increases of about 20 seconds, compared with about 10 seconds for explanation length and only a few seconds for variable repetition.
- Explicit cognitive chunks made participants slower than implicitly embedded concepts, contrary to the expectation that smaller reasoning units would be easier to process.
- Across domains, effect magnitudes differed but trends were not clearly reversed, supporting the possibility of general explanation-design principles.Among 21 cross-domain significant effects, 19 had confidence intervals with the same sign in both domains.
- Response-time trends were consistent across tasks, although simulation was fastest, counterfactual reasoning was slowest, and verification was generally intermediate.
- Subjective satisfaction largely followed response-time findings, but accuracy showed no statistically significant effects from explanation variation.Counterfactual tasks nevertheless had significantly lower accuracy than simulation tasks.
5 Discussion
The study identifies consistent patterns in how explanation complexity and task type affect human performance, while highlighting open questions about cognitive chunks, interfaces, and transfer to real-world use.
- New cognitive chunks had the greatest effect, followed by explanation length, while variable repetition had relatively little effect across metrics, tasks, and domains.The authors suggest focusing design efforts on explanation length and how new concepts are introduced rather than primarily minimizing repeated variables.
- Simulation was fastest, verification followed, and counterfactual reasoning was slowest and least accurate across explanation forms.The authors suggest participants may simulate through the explanation before performing verification or counterfactual reasoning.
- Participants responded faster when new cognitive chunks were implicit rather than explicit, an unexpected result requiring further investigation.Possible explanations include resolving the answer in one long line instead of two, or differences in concept familiarity.
- Future experiments should examine subjective satisfaction, alternative interfaces, and how display choices reflect downstream task needs.The study’s interface was fixed after pilot studies, but the authors note possibilities such as highlighting relevant lines.
- Broader work is needed to determine whether explanation principles generalize across explanation forms, tasks, regression settings, and input dimensionalities.The authors also call for connecting performance on basic tasks to error finding, trust decisions, and other real-world tasks.
6 Conclusion
The paper studies how explanation size, cognitive chunks, and repeated terms affect human performance on three explanation-use tasks. Consistent effects across tasks, metrics, and domains suggest common design principles may exist.
- The study varies explanation size, new cognitive chunks, and repeated terms while measuring human performance on simulation, verification, and counterfactual tasks.
- Consistent effects across tasks, metrics, and domains suggest that explanation systems may share common design principles.
Interface
The interfaces present participants with observations, preferences, and explanations for completing simulation or counterfactual reasoning in the Recipe domain.
- The simulation interface asks participants to provide a valid recommendation satisfying the alien given observations and preferences.
- The counterfactual interface asks participants whether the alien’s satisfaction changes after specified observations change.
Description of Pilot Studies
Pilot studies shaped the experiment’s timing instructions, interface configuration, and selected complexity factors.
- Time limits lowered accuracy because subjects preferred answering as time expired rather than leaving questions unanswered.The authors suggest coaching participants into fast or careful thinking modes as a possible future adjustment.
- Pilot comparisons of layout, text coloring, rule-versus-narrative presentation, and rule ordering showed effects on response time and accuracy.The researchers selected the configuration with the highest accuracy and user satisfaction.
- Preliminary regressions found no significant difference between conjunctions and disjunctions, while number of lines, global variables, and output-clause length affected results.