Source-linked AI summary
Interpretable & Explorable Approximations of Black Box Models
Himabindu Lakkaraju, Ece Kamar, Rich Caruana, Jure Leskovec
TL;DR
Complex predictive models are difficult for decision makers to understand and trust, motivating faithful and interpretable global explanations. BETA learns compact, non-overlapping decision sets through joint optimization and supports user-guided exploration. Experiments and user studies report compact, high-fidelity approximations and faster, more accurate human reasoning than baseline approximations.
Problem
Increasingly complex predictive models are difficult to explain faithfully and interpretably, limiting decision makers' ability to understand and trust their behavior.
Method
BETA jointly optimizes fidelity, unambiguity, and interpretability by learning two-level decision sets for defined feature-space regions, with user input supported for exploring selected subspaces.
Results
BETA generated compact, high-fidelity approximations, while users reasoned about complex predictive models accurately and about 1.5 and 2.3 times faster than with IDS and BDL, respectively.
Takeaways & Limitations
The framework provides global, interpretable approximations whose regions are explicitly defined and can be customized to users' feature interests.
Takeaways & Limitations
The objective weights can be set by cross-validation or an end user, while ε1, ε2, and ε3 are application-dependent and must be set by an end user.
Abstract
from arXiv · showhide
We propose Black Box Explanations through Transparent Approximations (BETA), a novel model agnostic framework for explaining the behavior of any black-box classifier by simultaneously optimizing for fidelity to the original model and interpretability of the explanation. To this end, we develop a novel objective function which allows us to learn (with optimality guarantees), a small number of compact decision sets each of which explains the behavior of the black box model in unambiguous, well-defined regions of feature space. Furthermore, our framework also is capable of accepting user input when generating these approximations, thus allowing users to interactively explore how the black-box model behaves in different subspaces that are of interest to the user. To the best of our knowledge, this is the first approach which can produce global explanations of the behavior of any given black box model through joint optimization of unambiguity, fidelity, and interpretability, while also allowing users to explore model behavior based on their preferences. Experimental evaluation with real-world datasets and user studies demonstrates that our approach can generate highly compact, easy-to-understand, yet accurate approximations of various kinds of predictive models compared to state-of-the-art baselines.
1 INTRODUCTION
BETA addresses the difficulty of explaining complex black-box classifiers by constructing compact, faithful global explanations and supporting interactive exploration. The framework combines interpretable decision sets with optimization guarantees and shows promising results in experiments and user studies.
- Unlike prior local explanations, BETA specifies which explanatory rules apply to different parts of the feature space.Earlier approaches selected representative instances and local models without clearly assigning models to regions.
- BETA constructs compact global explanations of black-box classifiers using decision sets that capture behavior in specific feature-space regions.The approach targets whole-model behavior rather than only individual predictions.
- Its explanations aim to preserve fidelity, avoid ambiguity, and remain interpretable by minimizing rule and predicate complexity.Non-overlapping decision-set regions provide an unambiguous rationale for assigned labels.
- On a depression-diagnosis dataset and in user studies, BETA produced lower-complexity, high-fidelity approximations that enabled accurate and fast reasoning about complex models.The evaluation compares BETA with state-of-the-art baselines and measures human reasoning performance.
2 OUR FRAMEWORK
BETA constructs global black-box explanations as compact, two-level decision sets that jointly target fidelity, unambiguity, interpretability, and user-directed exploration. Its objective formalizes these goals and admits provably near-optimal optimization under matroid constraints.
- Design goals: The framework targets fidelity, unambiguity, and interpretability, requiring accurate behavior, one rationale per instance, and human-understandable complexity.Interpretability includes limiting rules, features, predicates, and rule width rather than relying only on an intuitive representation.
- Design goals: Users can explore black-box behavior through customized approximations focused on subspaces defined by feature values of interest.This addresses cases where a generic explanation may omit the features users want to examine.
- Representation: BETA uses two-level decision sets: outer neighborhood descriptors identify feature-space regions, while inner rules specify black-box decision logic within each region.The representation consists of nested if-then structures with unordered inner decision rules.
- Objective: Default and tie-breaking functions handle uncovered or multiply covered instances, although the optimization is designed to minimize their use.The default assigns the majority black-box label among uncovered training instances, while ties use higher agreement with the black box.
- Objective: The optimization quantifies fidelity with disagreement, unambiguity with rule overlap and coverage, and interpretability with size, width, predicate count, and feature overlap.These measures are defined relative to a black-box model, dataset, and candidate neighborhood and decision-logic conjunctions.
- Optimization: The objective is non-normal, non-negative, non-monotone, and submodular, while its constraints are matroids; the resulting optimization problem is NP-hard.The framework therefore uses an optimization structure with theoretical properties suitable for approximation analysis.
3 EXPERIMENTAL EVALUATION
BETA is evaluated on depression-diagnosis data against LIME, IDS, BDL, and related variants using fidelity, interpretability, and user-study measures. The approximations achieve strong agreement with the black-box model using fewer neighborhoods or simpler rules, while users answer questions more accurately and quickly.
- The evaluation uses about 33K individuals with medical-history, symptom, and demographic features labeled depressed or healthy.
- BETA is benchmarked against LIME, IDS, BDL, LIME-DS, and BETA-LM.BETA-LM groups instances using BETA's neighborhood descriptors and fits a separate linear model for each neighborhood.
- Fidelity–interpretability trade-offs: At an average width of 10 predicates per rule, BETA reaches about 85% agreement, while other approaches require at least 20 predicates.Agreement rate is the fraction of instances whose approximation label matches the black-box prediction.
- Structural properties: BETA and IDS produce ruleoverlap between 1 and 2% and cover between 95 and 98%.Decision lists achieve zero ruleoverlap and N cover by design.
- User studies: In a 33-participant user study, BETA yielded higher answer accuracy than IDS and BDL, with users about 1.5 and 2.3 times faster, respectively.A second study with 11 participants customized approximations to the features in each question to measure interactive exploration.