Source-linked AI summary
Interpretability via Model Extraction
Osbert Bastani, Carolyn Kim, Hamsa Bastani
TL;DR
Machine learning informs consequential decisions, yet deployed models can exhibit defects, motivating ways to inspect their reasoning. This paper approximates complex black-box models with interpretable decision trees and applies the approach to several model types and tasks. The authors report improvement over CART and show how extracted trees can support debugging.
Problem
Machine learning informs consequential decisions, but deployed algorithms can exhibit defects such as unfairness and lack of user trust.
Source: §1 · p. 1Method
Model extraction approximates a black-box model with an interpretable decision tree, using only the model's input-output behavior rather than its internal implementation.
Source: §1 · p. 1Results
The active-learning approach substantially improves over CART and is demonstrated for interpreting and debugging random forests, neural networks, and learned control policies.
Source: §1 · p. 2Takeaways & Limitations
Extracted trees can help assess dependence on prejudiced features, investigate poor model performance, and understand learned control policies.
Source: §1 · p. 2Model extraction does not guarantee fairness; it can be useful for evaluating a model's fairness.
Source: §3.2 · p. 3
Abstract
from arXiv · showhide
The ability to interpret machine learning models has become increasingly important now that machine learning is used to inform consequential decisions. We propose an approach called model extraction for interpreting complex, blackbox models. Our approach approximates the complex model using a much more interpretable model; as long as the approximation quality is good, then statistical properties of the complex model are reflected in the interpretable model. We show how model extraction can be used to understand and debug random forests and neural nets trained on several datasets from the UCI Machine Learning Repository, as well as control policies learned for several classical reinforcement learning problems.
1 INTRODUCTION
Model extraction uses an interpretable decision tree to approximate a complex model, allowing its reasoning to be examined while the original model remains available for deployment. The paper applies this black-box approach to random forests, neural networks, and learned control policies, and reports improvement over CART.
1 INTRODUCTION
- Interpretability can help users diagnose defects and verify model correctness; testing dependence on a prejudiced feature may be more informative than omitting it.A model may reconstruct an omitted feature from other features, motivating examination of how predictions change when the feature changes.Source: §1 · p. 1
- Model extraction approximates a complex model with an interpretable decision tree so users can inspect and debug its reasoning while retaining the original model for deployment.The paper argues that issues in the complex model should be reflected in the tree when the approximation is sufficiently good.Source: §1 · p. 1
- Unlike prior extraction approaches tied to specific model families, this black-box approach needs only input-output access and is implementation-independent.The paper contrasts it with approaches that leverage internal model structure and with methods focused on specific inputs rather than the model as a whole.Source: §1 · p. 1
- The algorithm uses active learning to construct the tree by sampling many training points from the model.Decision trees can overfit and perform poorly, while complex models such as random forests and deep neural networks are better regularized.Source: §1 · pp. 1–2
- With sufficient generated data, the extracted tree converges to the exact greedy decision tree under mild assumptions.The authors attribute this convergence to sampling error going to zero, avoiding overfitting to the initial training points.Source: §1 · p. 2
- The active-learning approach substantially improves over CART and is used to debug random forests, neural networks, and learned control policies.The extracted trees help assess dependence on prejudiced features, investigate why models perform worse, and understand control-policy structure.Source: §1 · p. 2
2 MODEL EXTRACTION
Model extraction approximates a black-box function with an interpretable decision tree, aiming to make the function’s behavior inspectable. The algorithm greedily estimates tree gains from samples, and the theoretical guarantee concerns agreement with the exact greedy tree under stated distribution assumptions.
2.1 Problem Formulation
- The task is to learn an interpretable axis-aligned decision tree that approximates a black-box classifier, evaluated by held-out accuracy relative to that function.The formulation focuses on classification.Source: §2.1 · p. 2
2.2 Algorithm
- The algorithm greedily approximates a CART-style tree whose branch gains and leaf labels are computed exactly under the fitted input distribution.The fitted distribution is a mixture of axis-aligned Gaussians; component probabilities are normalized for sampling.Source: §2.2 · p. 2
- The algorithm estimates each node’s gain from n i.i.d. samples conditioned on that node’s axis-aligned constraints.It fits a mixture of axis-aligned Gaussians to the training data, then samples from the distribution conditioned on node constraints.Source: §2.2 · p. 2
2.3 Teoretical Guarantees
- With probability at least 1−δ over training samples, some sample size n makes the probability of T(x)=T*(x) at most ε, assuming a well-defined T* and a bounded, continuous input density with bounded support.This is the probabilistic condition stated in Theorem 2.1.Source: §2.3 · p. 3
3 EVALUATION
The evaluation tests model extraction on UCI supervised models and an OpenAI Gym policy, first benchmarking against CART and then using extracted trees to examine model behavior.
3.1 Comparison to CART
- The extracted tree outperformed CART on every problem instance under the reported absolute and relative evaluations.Table 1 compares each tree’s ground-truth performance and performance relative to the original model.Source: §3.1 · p. 3, Table 1 · p. 4
3.2 Examples of Use Cases
- The invalid time-to-recurrence feature appeared in every extracted tree and was the top branch in 6 of 10 trees.The forest predicted recurrence within 24 months while improperly using time to recurrence as a feature.Source: §3.2 · p. 3
- Gender appeared below other class grades or prior failures in 7 of 10 trees, and trees included it when models showed relatively large gender dependence.The top features were consistently grades in other classes or prior failing grades.Source: §3.2 · p. 3
- Subgroup effects ranged from 0.44 to 0.77 grade points, while the estimated affected-student share ranged from 18.3% to 39.1%.The two largest-effect subgroups accounted for 67.3% and 65.6% of the total gender effect, respectively; in 5 of 7 trees, affected students had low grades.Source: §3.2 · pp. 3–4
- On the wine-origin dataset, random forests all reached F1 scores of at least 0.961, while neural-net performance was bimodal: five scored at least 0.955 and the rest at most 0.741.Source: §3.2 · p. 4
- Chlorides nearly tracked poor neural-net performance, while alcohol appeared near the top of 19 of 20 extracted trees.High-performing models tended to split on alcohol at thresholds of 749.8 to 999.6, versus 574.4 to 837.3 for poorly performing models; feature-influence metrics alone were insufficient to identify this pattern.Source: §3.2 · p. 4
- A 7-node cartpole tree captured a significant fraction of policy behavior but had estimated reward 152.3, below that of larger trees.It moved right only for pole velocities in [−2.0, 2.0] and pole angles in [−0.5, 0.5]; the resulting asymmetry reflected the initially left-moving cart and relatively unexplored right-moving states.Source: §3.2 · p. 4
4 CONCLUSIONS
The paper proposes model extraction to interpret black-box models and demonstrates its use across different kinds of models.
4 CONCLUSIONS
- The paper proposes model extraction as an approach for interpreting black-box models.Source: §4 · p. 4
- The approach is demonstrated on a variety of different kinds of models.Source: §4 · p. 4
- Future work includes extraction algorithms for more expressive input distributions and new ways to gain insight from extracted decision trees.Source: §4 · p. 4