Source-linked AI summary
Human-in-the-Loop Interpretability Prior
Isaac Lage, Andrew Slavin Ross, Been Kim, Samuel J. Gershman, Finale Doshi-Velez
TL;DR
Interpretability is subjective and depends on the user and downstream task, while proxy-based optimization assumes it can be quantified through model properties. This paper uses human feedback and model-based optimization to search among predictive models with few user studies. Across datasets, the model users found most expedient varied in which interpretability proxy it resembled.
Problem
Interpretability depends on users and downstream applications, yet proxy-based optimization assumes it can be represented by a computational property of the model.
Source: §1 · p. 1Method
Results
Takeaways & Limitations
Human-selected models often corresponded to a known interpretability proxy, but the proxy varied across datasets, making proxies a starting point rather than a complete account of human interpretability.
Source: §7 · p. 8Pilot between-subjects studies on Amazon Mechanical Turk had too much subject variance for cost-effective optimization, unlike the smaller, longer within-subjects studies.
Source: §7 · p. 8
Abstract
from arXiv · showhide
We often desire our models to be interpretable as well as accurate. Prior work on optimizing models for interpretability has relied on easy-to-quantify proxies for interpretability, such as sparsity or the number of operations required. In this work, we optimize for interpretability by directly including humans in the optimization loop. We develop an algorithm that minimizes the number of user studies to find models that are both predictive and interpretable and demonstrate our approach on several data sets. Our human subjects results show trends towards different proxy notions of interpretability on different datasets, which suggests that different proxies are preferred on different tasks.
1 Introduction
Interpretability is subjective and task-dependent, making it difficult to encode directly in computational learning methods. The paper estimates an interpretability prior from human feedback and uses few user-study queries to search for models that are both predictive and interpretable.
1 Introduction
- Interpretability depends on users’ subjective experience and the downstream application, complicating its incorporation into computational learning methods.The authors motivate interpretability partly by its potential to help people discover confounders, dangerous associations, and scientific insights in learned models.Source: §1 · p. 1
- Prior approaches either use model classes considered inherently interpretable or optimize computational proxies such as sparsity, monotonicity, or explainability by decision trees.Examples of model classes include decision trees, generalized additive models, and decision sets; proxy-based methods include sparse linear models.Source: §1 · p. 1
- The method estimates an interpretability prior from human-subject feedback, aiming to find models better suited to a task than methods that optimize an imperfect proxy.The authors develop a cost-effective search that first identifies high-likelihood models, then uses model-based optimization to choose which models receive costly prior evaluations.Source: §1 · p. 2
- Optimizing an interpretability property can be framed as MAP estimation with a model prior, but this requires specifying a functional form for subjective interpretability.The paper notes that priors can encode inductive biases, including sparsity-inducing penalties.Source: §1 · pp. 1–2
2 Related Work
Related work includes proxy-based model optimization, studies that evaluate interpretability with people, and methods for finding optima through expensive evaluations. The paper combines human feedback with model-based optimization to search for interpretable models.
Learning interpretable models with proxies
- Proxy-based approaches optimize computational model properties assumed to capture interpretability, but examples of doctors preferring longer trees suggest proxies may not fit every context.Examples include tree depth, coefficient counts, rule overlap, and neural-network sparsity.Source: §2 · p. 2
Learning from human feedback
- Prior user studies test explanation factors or compare model classes, while this work uses studies to optimize interpretability rather than only compare models.The paper relates its approach to human computation methods for learning preferences that are difficult to express computationally.Source: §2 · p. 2
Model-based optimization
- Model-based optimization seeks optima with few evaluations; here, the expensive evaluation is the human-derived prior rather than model likelihood.The authors adapt techniques used for expensive posterior evaluation, contrasting their user-study cost with the cost of training a neural network in related work.Source: §2 · p. 2
3 Framework and Modeling Considerations
The framework searches for models with both high data likelihood and a human-informed interpretability prior. It uses a performance threshold to screen candidates, response time to measure interpretability, and local explanations to extend the approach to complex models.
3 Framework and Modeling Considerations
- The pipeline first finds diverse high-likelihood models, then uses model-based optimization to select which candidates receive costly user-study evaluations of the interpretability prior.The framework treats model computation as cheaper than evaluating the prior, which requires a user study.Source: §3 · pp. 2–3, Figure 1 · p. 3
3.1 Likelihood
- The likelihood uses a soft-insensitive-loss formulation so models above a validation-accuracy threshold qualify as high-likelihood candidates.The threshold is set to the task’s validation-performance requirement, and the authors note that other domain-specific quality measures can replace accuracy.Source: §3.1 · p. 3
3.2 A Prior for Interpretable Models
- For interpretable model classes, the human-interpretability score uses response time to simulate a model’s label, with a cap excluding outliers.The paper notes that measuring time to predict a label follows a proposed simulation proxy when no downstream task is specified; domain-specific tasks and metrics could also be substituted.Source: §3.2 · p. 3
3.3 A Prior for Arbitrary Models
- For complex models, the method samples perturbations around each point and trains a simple local model to mimic the original model’s predictions.The prior is modified to evaluate human interpretability through this local proxy rather than through the entire model.Source: §3.3 · p. 4
4 Inference
Inference approximates a MAP solution by assembling predictive models, estimating their interpretability priors, and selecting user-study queries to find a strong combined candidate.
4 Inference
- The pipeline returns the candidate model with the largest product of predictive likelihood p(X|M) and interpretability prior p(M) as its approximate MAP solution.It first identifies models with high likelihood, then uses model-based optimization to choose which priors to evaluate through user studies.Source: §4 · p. 4
4.1 Identifying models with high likelihood p(X|M)
- For simpler models, multiple hyperparameter restarts produce accurate candidates; for neural networks, differing input-gradient patterns help create a diverse candidate set.Candidates failing the accuracy threshold are rejected.Source: §4.1 · p. 4
4.2 Computing the prior p(M)
- The prior p(M) for inherently interpretable models is approximated by sampling points, since evaluating human interpretability for every input would require a user study.The empirical input distribution is used as p(x).Source: §4.2 · p. 4
- For arbitrary model classes, p(M) integrates human interpretability of local approximations, which are fitted by perturbing inputs and matching the original model's outputs.Only points near the decision boundary need human evaluation; other integral terms can be approximated without user studies.Source: §4.2 · pp. 4–5
4.3 Model-based Optimization of the MAP Objective
- A Gaussian Process estimates unlabeled candidates' interpretability, and a lower-confidence-bound acquisition rule selects the next query to minimize response time.The search targets high p(M) among candidates already labeled as likely under p(X|M); the paper reports κ = 1 works well in practice.Source: §4.3 · p. 5
5 Experimental Setup
The setup evaluates human interpretability and four automatic proxies across four datasets, using model classes selected for dataset complexity.
Datasets and Training Details
- The experiments use synthetic, mushroom, census-income, and covertype datasets, with train-validation splits and preprocessing specified for each.The first three datasets are class-balanced; no test set is included because held-out accuracy is not reported.Source: §5 · pp. 5–6
- Decision trees are trained for synthetic, mushroom, and census data, while neural networks are used for the more complex covertype dataset.Covertype also requires local explanations for the neural network.Source: §5 · p. 6
Proxies for Interpretability
- Four proxies are compared: mean path length, distinct features per path, node count, and nonzero features.The first two are local measures and the latter two global; proxy comparisons use the same 1,000 validation points across models.Source: §5 · p. 6
Human Experiments
- Human interpretability is measured by how long users take to simulate a model's label, using the model itself or a local explanation as the explanation shown.Users see feature values and a graphical depiction, then identify the prediction.Source: §5 · p. 6
6 Experimental Results
Human-feedback optimization reduces response times while converging on different proxy notions across datasets; experiments also show that proxy mismatch can mis-rank models.
6 Experimental Results
- Choosing the wrong proxy can seriously mis-rank the model that is best under the human interpretability measure, and proxy preferences differ across datasets.The synthetic dataset's mis-ranking distribution also differed from those of the real datasets.Source: §6 · p. 7
- Using the correct interpretability measure on a small sample of points performed better than using the wrong proxy across datasets and models.The small sample reflects limits on human cognitive capacity and produces noisy measurements.Source: §6 · p. 7
- Across three datasets and four proxies, model-based optimization outperformed random sampling when evaluating the same number of models.The pipeline was run for 10 iterations and compared with random draws of 10 models.Source: §6 · p. 7
- Mean response-time distributions decreased as more models were evaluated on census and mushroom, with reported practice effects small relative to the differences.Figure 4a presents the human-subject response-time results.Source: §6 · p. 8, Figure 4 · p. 8
- Human-feedback optimization converged to the fewest nodes and shortest paths on mushroom, but the fewest features on census, as response times and proxy scores fell.The approach did not use proxy scores during optimization.Source: §6 · p. 8
7 Discussion and Conclusion
The human-in-the-loop pipeline efficiently finds models people prefer for forward simulation, while preferred interpretability proxies vary by dataset. Human evaluation can be costly and variable, motivating studies with intended users and improved query allocation.
7 Discussion and Conclusion
- The pipeline efficiently identified models humans found most expedient for forward simulation, and different interpretability proxies selected different models across datasets.Human-selected models often matched a known proxy, but the matching proxy varied by dataset.Source: §7 · p. 8
- Human-selected models often matched a known interpretability proxy, but the proxy that matched varied across datasets.The authors conclude that proxies may be a useful starting point but do not capture the full story of human interpretability.Source: §7 · p. 8
- Pilot Mechanical Turk studies had too much between-subject variance for cost-effective optimization, whereas smaller, longer within-subject studies had lower variance.The authors suggest conducting studies with intended model users, who may be more familiar with the task and more patient.Source: §7 · p. 8
- Future work includes reducing variance through smarter input selection, making sequential studies more engaging, and refining kernels for nuanced interpretability across model classes.User studies remain necessary, but intelligent optimization can reduce their number and cost.Source: §7 · pp. 8–9
A Similarity Kernel for Models and GP parameters
The model similarity kernel uses feature importances, with different importance representations for decision trees and neural networks. The section also specifies Gaussian-process settings and describes the synthetic dataset’s feature groups.
A Similarity Kernel for Models and GP parameters
- The RBF kernel compares decision trees by feature importance and neural networks by average magnitudes of normalized input gradients for each class logit.Model-based optimization requires a notion of model similarity.Source: A Similarity Kernel for Models and GP… · p. 11
- The Gaussian process normalizes y, restarts optimization 10 times, and adds α = 10^-7 to the kernel diagonal; other hyperparameters use defaults.The covariance function uses an RBF kernel over the model features described above.Source: A Similarity Kernel for Models and GP… · p. 11
- The synthetic dataset combines two noise dimensions with two lower-accuracy interpretable dimensions and two higher-accuracy less-interpretable dimensions.Features were generated in pairs, then points sharing labels across dimensions were randomly concatenated.Source: Figure 5 · p. 11
B Experimental Details: Identifying a Collection of Predictive Models
The experiments construct predictive model collections using dataset-specific accuracy thresholds and randomized model settings. Decision trees are used for three datasets, while neural networks are trained for covertype.
B Experimental Details: Identifying a Collection of Predictive Models
- Decision-tree searches trained 500 models per dataset at accuracy thresholds of 0.9 for synthetic, 0.95 for mushroom, and 0.8 for census.The synthetic threshold exceeds achievable accuracy on the interpretable dimensions; the mushroom and census thresholds sit slightly below their achievable validation accuracies.Source: B Experimental Details: Identifying a… · p. 11
- Decision-tree searches varied depth, minimum leaf samples, features per split, and splitting strategy, then iteratively removed leaves when validation accuracy did not decrease.Depth and leaf-size settings encourage simpler trees, while split-feature and strategy settings increase tree diversity.Source: B Experimental Details: Identifying a… · p. 11
- For covertype, the neural-network search trained 250 models at an accuracy threshold of 0.75, varying regularization, activation, architecture, and simultaneous model count.Models were trained for 50 epochs with batch size 512 using Adam.Source: B Experimental Details: Identifying a… · p. 11
C Experimental Details: Parameters and Sensitivity to Local Region Choices
The neural-network sensitivity analysis examines local explanations under alternative region settings and sample sizes. Proxy-selected model ranks generally agree across region settings, and the settings require similar numbers of input samples for robust rank estimates.
C Experimental Details: Parameters and Sensitivity to Local Region Choices
- For each neural-network prediction, local explanations use sampled nearby points, balance labels matching the model’s prediction, and exclude regions with class imbalance above 0.75.The local dataset is sampled around the point’s 20 nearest neighbors; points judged not to be on the boundary are excluded.Source: C Experimental Details: Parameters and… · pp. 11–12
- Local trees must reach 90% validation fidelity, are fit up to depth 10 with at least five samples per leaf, and are post-processed to remove accuracy-preserving leaves.The authors assume deeper local trees would not be interpretable and would not improve the search for the most interpretable model.Source: C Experimental Details: Parameters and… · p. 12
- Across local-region settings, proxy-selected models generally retained low ranks under alternatives; all highest node-count mismatches used the excluded variance scaling factor 0.1.Rank 0 means the same model remains best under both settings; higher ranks indicate its position under the alternative setting.Source: C Experimental Details: Parameters and… · p. 12, Figure 6 · p. 12
- Different local-region settings required similar numbers of input samples to robustly approximate the proxy integral for p(M).The comparison uses the smallest, middle, and largest local-region settings, without cross-pairs of parameters.Source: C Experimental Details: Parameters and… · p. 12, Figure 6 · p. 12
D Experimental Details: Human Subject Experiments
The experiments used human quizzes to assess how participants simulated model predictions, with global studies reusing data points and local studies sampling near each model’s boundary. In a pilot, response-time intervals for two models overlapped, while collecting enough additional responses was costly.
Human Experiments
- For global models, users evaluated the same data points across models; local studies sampled points near each model’s boundary, and quizzes used 8 or 16 questions plus practice questions.Participants who missed the practice threshold were excluded from the Amazon Mechanical Turk experiments.Source: D Experimental Details: Human Subject… · pp. 12–13
Experiments with Machine Learning Graduate Students and Postdocs
- The full pipeline used 10 eight-question iterations, with 7 census subjects and 9 mushroom subjects per model; the smaller subject pool left differences statistically nonsignificant.Using the same subjects across experiments substantially reduced response variance.Source: D Experimental Details: Human Subject… · p. 13
Experiments with Amazon Mechanical Turk
- Amazon Mechanical Turk required a between-subjects design because participants would be difficult to recontact for later quizzes as the acquisition function selected models.Pilot quizzes had 16 questions, with 33 and 24 responses for the two models after excluding specified practice and timing failures.Source: D Experimental Details: Human Subject… · p. 13
- The pilot’s mean response times were 31.62s for the medium-mean-path-length model and 30.94s for the high-mean-path-length model, with overlapping intervals.The authors note that reducing variance with more samples would be costly; an end-to-end pipeline experiment at these sample sizes would cost around $1,000.Source: D Experimental Details: Human Subject… · p. 13