PaperlayerView exact revision on arXiv

Source-linked AI summary

From Human Explanation to Model Interpretability: A Framework Based on Weight of Evidence

David Alvarez-Melis, Harmanpreet Kaur, Hal Daumé III, Hanna Wallach, Jennifer Wortman Vaughan

arXiv:2104.13299v2cs.AIcs.LG

TL;DR

Many interpretability methods explain predictions through feature importance or intrinsic metrics, despite explanations being intended for human use. This paper derives human-centered design principles and applies weight of evidence to generate explanations; it reports finite-sample estimation and robustness results, alongside a practitioner study in which most participants used the tool despite difficulty with prior class probabilities.

  • Problem

    Many local explanation methods focus on feature contributions and intrinsic evaluation, while comparatively less attention is given to how people use explanations in practice.

    Source: Introduction · pp. 1–2

  • Method

    The paper derives human-centered explanation principles from prior research and uses weight of evidence to build a method for sequential, contrastive explanations.

    Source: Introduction · p. 2, Human-Centered Design Principles · p. 2, Explaining with the Weight of Evidence · p. 3

  • Results

    The authors report that WoE can be estimated from finite samples and is robust to small input perturbations, and that most of ten ML practitioners used the tool to answer questions despite difficulty with prior class probabilities.

    Source: Introduction · p. 2

  • Takeaways & Limitations

    The practitioner study highlights the importance of well-designed tutorials for interpretability tools, including for experienced ML practitioners.

    Source: Introduction · p. 2

    WoE requires conditional likelihoods that are accessible or can be accurately estimated, and estimating densities for complex data such as images remains an active research area.

    Source: Limitations · p. 10

Abstract

from arXiv · show

We take inspiration from the study of human explanation to inform the design and evaluation of interpretability methods in machine learning. First, we survey the literature on human explanation in philosophy, cognitive science, and the social sciences, and propose a list of design principles for machine-generated explanations that are meaningful to humans. Using the concept of weight of evidence from information theory, we develop a method for generating explanations that adhere to these principles. We show that this method can be adapted to handle high-dimensional, multi-class settings, yielding a flexible framework for generating explanations. We demonstrate that these explanations can be estimated accurately from finite samples and are robust to small perturbations of the inputs. We also evaluate our method through a qualitative user study with machine learning practitioners, where we observe that the resulting explanations are usable despite some participants struggling with background concepts like prior class probabilities. Finally, we conclude by surfacing~design~implications for interpretability tools in general.

Introduction

The paper argues that interpretability should be designed and evaluated around human needs, while many local methods focus on feature-importance scores and intrinsic properties. It proposes human-centered explanation principles, a weight-of-evidence method, and a practitioner study that examines its use.

Introduction

  • The paper situates interpretability’s importance in applications where explanations may support debugging, knowledge discovery, trust, fairness, accountability, and communication.These potential uses are discussed in the context of complex models’ expanding reach into high-stakes and decision-critical applications.Source: Introduction · p. 1
  • Complex black-box models remain widely used, motivating post-hoc local explanations alongside transparent glass-box models.The passage lists decision trees, point systems, and generalized additive models as examples of transparent models.Source: Introduction · p. 1
  • Many local explanation methods operationalize why a prediction was made as feature-importance scores, often evaluating mathematical properties rather than whether people find the explanations useful.This leaves a gap between explanations’ intended human uses and their evaluation in generic settings.Source: Introduction · pp. 1–2
  • The authors argue that interpretability is human-centered and draw on research about explanation and people’s use of interpretability tools.Prior work found that practitioners can misunderstand, over-trust, and misuse such tools.Source: Introduction · pp. 1–2
  • The paper surveys human-explanation research and proposes design principles, then uses weight of evidence to generate explanations for high-dimensional, multi-class settings.The authors report that WoE can be estimated from finite samples and is robust to small input perturbations.Source: Introduction · p. 2
  • In an artifact-based interview study with ten ML practitioners, most participants used the tool to answer questions, though some struggled with prior class probabilities.Participants often used multiple tool modalities and requested the ability to switch between them interactively.Source: Introduction · p. 2

Human-Centered Design Principles

Drawing on research about human explanation, the paper proposes principles that contrast with common one-shot feature-importance explanations. The principles favor explanations that distinguish alternatives, build from understandable components, and include only relevant facts.

Human-Centered Design Principles

  • In local ML explanations, the phenomenon to explain is why a model predicted output y for input x; the explanatory facts may concern inputs, parameters, training data, or training.Source: Human-Centered Design Principles · p. 2
  • Human explanations often contrast a target with alternatives and can combine factual and counterfactual reasoning, unlike methods that explain only why the input supports one prediction.The paper proposes that explanations be contrastive and account for why every alternative was not predicted.Source: Human-Centered Design Principles · p. 2
  • Explanations should be modular and compositional, using simple components that build toward a prediction.The paper contrasts this approach with one-shot explanations that provide importance scores for all features simultaneously.Source: Human-Centered Design Principles · p. 2
  • Explanations should use easily understandable quantities and be parsimonious, presenting only the most relevant facts.The paper identifies these as its fourth and fifth design principles.Source: Human-Centered Design Principles · p. 3
  • The authors present these principles as a starting point, not an exhaustive set, and frame human explanation as a process as well as a product.They argue that the principles shift interpretability methods toward that process framing.Source: Human-Centered Design Principles · p. 3

Explaining with the Weight of Evidence

The paper operationalizes human-centered explanation principles with weight of evidence (WoE), which measures evidence for one hypothesis relative to alternatives. It extends WoE to multi-class and high-dimensional inputs and describes a customizable procedure for producing explanations.

Explaining with the Weight of Evidence

  • WoE frames feature or feature-group contributions as evidence for a prediction relative to alternatives, aligning the explanation with contrastive and modular design principles.The paper characterizes evidence as a familiar concept and says the method can be formalized using simple quantities.Source: Explaining with the Weight of Evidence · p. 3

Weight of Evidence: Foundations

  • WoE compares the odds of a hypothesis with evidence against its marginal odds, with positive or negative values indicating evidence for or against the hypothesis.The paper also presents an evidence-likelihood interpretation and notes that WoE separates prior class probabilities from input likelihoods.Source: Weight of Evidence: Foundations · pp. 3–4

Composite Hypotheses and Evidence

  • For multi-class problems, a nested sequence of class subsets makes explanations sequential, ruling out classes in stages rather than contrasting one class with every other class at once.The motivating challenge is that a single contrast against all remaining classes may be uninformative when the number of classes is large.Source: Composite Hypotheses and Evidence · p. 4

A Meta-Algorithm for WoE Explanations

WoE Estimation for Black-box Models

  • For black-box models without conditional likelihoods, the method estimates them using model predictions; a Naive Bayes approximation is an option when only marginal feature likelihoods are available.The paper states that fitting a conditional-likelihood model is done once and may be performed offline, unlike LIME’s per-prediction model fitting.Source: A Meta-Algorithm for WoE Explanations · p. 5, WoE Estimation for Black-box Models · p. 6

Relation to LIME and SHAP

  • WoE focuses on conditional input likelihoods, whereas LIME and SHAP explain predictive posteriors; the approaches can be equivalent for some simple classifiers but not generally.Even when they highlight similar features, their interpretations and units differ.Source: Relation to LIME and SHAP · p. 6

Quantitative Experiments

The quantitative experiments assess finite-sample estimation of weight of evidence (WoE) and its robustness to input perturbations. They test estimation under a controlled Naive Bayes setup and compare robustness with LIME and SHAP on benchmark datasets.

Quantitative Experiments

Quality of Finite-Sample WoE Estimation

  • The finite-sample experiment used a smoothed Gaussian Naive Bayes classifier for both the target model and likelihood estimator, isolating estimation quality from model misspecification.The estimator was fit from a separate sample and used to compute WoE at explanation time; exact WoE from the classifier served for evaluation only.Source: Quality of Finite-Sample WoE Estimation · p. 6
  • WoE estimation improved with more fitting samples and degraded gracefully as input dimension increased, suggesting accurate finite-sample estimates even in relatively high dimensions.The authors caution that other models may behave differently and that the experiment does not measure misspecification error from the Naive Bayes approximation.Source: Quality of Finite-Sample WoE Estimation · p. 7

Robustness of WoE

  • Across all but one benchmark dataset, WoE explanations were on average as close or closer to ideal Lipschitz robustness than LIME and SHAP.Figure 3 reports ten repetitions per dataset and method; values farther from L = 1 are undesirable.Source: Robustness of WoE · p. 7, Figure 3 · p. 7

User Study

The user study examined how ten machine-learning practitioners understood and used a WoE-based interpretability tool in a controlled task. Participants found several features useful, while the study also surfaced conceptual difficulties and requests for greater flexibility and transparency.

Study Design

  • The artifact-based think-aloud study used a tutorial and a controlled task with a pre-trained model; the ten participants were ML practitioners.The task used a six-class random-forest model on the Online News popularity dataset, and the study was designed to examine use of feature groups and sequential explanations.Source: Study Design · pp. 7–8, Main Study · p. 8

Results

  • Seven of ten participants struggled with prior class probabilities, often ignoring or misusing them; nevertheless, participants answered main-study questions where those probabilities mattered.The authors suggest participants may have used the information indirectly through displayed class probabilities.Source: Understanding of Relevant Concepts · pp. 8–9
  • Participants had no overwhelming preference between feature-group and individual-feature explanations, and switched between them for complementary overview and detail.More experienced ML practitioners tended to rely on feature-level plots, while customer-facing participants more often gave high-level answers from feature groups; the authors caution that this pattern is inconclusive given the small sample.Source: Tool Usability and Participants’… · p. 9
  • Most participants accurately answered questions requiring sequential explanations: seven of ten used them successfully to reason about how classes were ruled out.Participants said those questions could not be answered precisely with one-shot explanations.Source: Tool Usability and Participants’… · p. 9
  • Participants valued the tutorial, and those who spent longer on it tended to give more accurate and thoughtful answers, including participants with less ML experience.Participants also reused the tutorial example when answering study questions.Source: General Needs for Interpretability Tools · p. 9
  • Interactivity was the most commonly requested functionality, particularly easier switching between explanation options without rerunning code.Source: General Needs for Interpretability Tools · p. 9

Limitations

The paper identifies limits in both the scope of WoE explanations and the evidence from its user study. Its design principles are not universal, and its study was an initial evaluation rather than a representative field assessment.

Limitations

  • Interpretability has no ground truth, even for proxy concepts such as feature importance, so methods operationalize different notions and no method is universally ideal.Explanations may also be partial, locally scoped, or based on strong data assumptions, and may reveal different aspects of model behavior.Source: Limitations · pp. 9–10
  • The human-centered design principles are neither exhaustive nor universal; some may not be necessary in every setting and remain open to refinement.Source: Limitations · p. 10
  • WoE requires accessible or accurately estimable conditional likelihoods, and its current applicability to complex data such as images is limited.The authors identify generative, log-linear, and multiclass models as settings with a clear case for WoE; choices for class and feature-group partitioning may also produce different explanations.Source: Limitations · p. 10
  • The user study included only ML practitioners, was not guaranteed to reflect day-to-day use, and its small sample precluded statistical analyses.The authors describe it as an initial evaluation and note that a longitudinal field study with multiple participant types would be preferable.Source: Limitations · p. 10

Discussion

The discussion presents WoE as a human-centered approach that can be adapted to high-dimensional and multiclass settings, with estimates that are accurate from finite samples, robust to small input changes, and usable by practitioners. It also draws broader design lessons from the study and emphasizes explaining a method’s scope.

Discussion

  • The paper combines human-explanation literature and design principles with WoE to produce explanations intended to be meaningful to people.Source: Discussion · p. 10
  • Future work could adapt density-estimation methods for complex data and study design choices with participants’ own models and day-to-day questions.The authors specifically identify class-partitioning choices for sequential explanations as a topic for follow-up study.Source: Discussion · p. 10
  • The authors identify user-friendly, engaging tutorials and flexibility as broader interpretability-tool design implications from their study.They argue that design and evaluation should put human needs at the center.Source: Discussion · p. 10
  • The authors recommend that interpretability papers state both suitable and out-of-scope settings and explicitly describe the notion of explanation they operationalize.Source: Discussion · p. 10

Design Principles in Further Detail

The proposed design principles translate recurring features of human explanation into guidance for machine-generated explanations. They call for explanations that distinguish alternatives, cover alternatives, break evidence into understandable components, and remain selective.

D1. Contrastive

D2. Exhaustive

  • Explanations should justify why every alternative was not predicted, making them counterfactual and complete in Hempel’s terminology.A cough alone does not exhaustively explain a pneumonia diagnosis because it is also a symptom of other diseases.Source: Design Principles in Further Detail · p. 14

D3. Modular and compositional

  • Explanations should use multiple simple, accumulative statements, each addressing a few aspects of the evidence.This modularity trades off the number of components against their complexity and shifts explanations from products toward processes.Source: Design Principles in Further Detail · p. 14

D4. Easily understandable quantities

D5. Parsimonious

Axiomatic Derivation of the WoE

The paper uses weight of evidence (WoE) to realize human-centered explanation principles. Its axiomatic basis characterizes WoE as a function of evidence and hypothesis with a role in posterior probabilities and additive decomposition.

Axiomatic derivation

Well-Known Model Families

The paper examines WoE expressions for familiar classifier families, including logistic regression, naïve Bayes, and discriminant analysis. Their forms depend on model assumptions, and logistic regression requires additional assumptions for per-feature WoE.

Well-Known Model Families

  • The section examines WoE expressions for familiar classifier families, beginning with binary classification over n input features.Source: Well-Known Model Families · p. 14

Logistic Regression

  • For logistic regression, WoE removes the base-rate log-odds from the linear attribution, but per-feature values require assumptions because the model does not specify needed conditional probabilities.One proposed identification assumes coefficients do not model feature interactions; without penalization and asymptotically, LIME’s surrogate coincides with the true log-linear model and its WoE values.Source: Logistic Regression · p. 15

Na¨ıve Bayes

  • Naïve Bayes models conditional probabilities under conditional independence, and Gaussian naïve Bayes assigns positive WoE when a feature is closer by variance-normalized distance to the class-1 mean.Source: Na¨ıve Bayes · p. 15

LDA and QDA

  • LDA and QDA allow dependent features; conditional per-feature WoE has more complex expressions, while LDA’s total WoE uses its shared-covariance assumption.Source: LDA and QDA · p. 16

Use Cases and Limitations

WoE explanations are not inherently better than other methods; their usefulness depends on the model, data, user, and intended use. They are especially suited to generative, log-linear, or multiclass settings, subject to conditions on likelihood access and stakeholder understanding.

Use Cases and Limitations

No Explanation is Inherently “Better”

  • No explanation method is inherently better: post-hoc explanations are imperfect, and their usefulness depends on the model, data, user, and intended use.Different methods may rely on different assumptions and approximations or reveal different aspects of model behavior.Source: No Explanation is Inherently “Better” · p. 16

When Are WoE Explanations Useful?

  • WoE explanations are especially useful for generative, log-linear, and multiclass models, where likelihoods, simple log-odds, or sequential class contrasts fit the setting.In multiclass settings, WoE can contrast sets of classes; sequential explanations helped study participants understand how classes were ruled out.Source: When Are WoE Explanations Useful? · p. 16, Figure 4 · p. 17
  • For other models, WoE requires accessible or accurately estimated conditional likelihoods and log-odds ratios that stakeholders can understand.The authors recommend experimentally verifying stakeholder understanding in the intended context.Source: When Are WoE Explanations Useful? · pp. 16–17

Sequential Explanations

For many-class predictions, sequential explanations progressively rule out subsets of classes while retaining the predicted class, until only that class remains.

Sequential Explanations

  • At each step, the procedure rules out a subset of classes while retaining the predicted class, until the retained set contains only that class.This sequential process ensures every alternative class is eventually ruled out.Source: Sequential Explanations · p. 17

Jupyter Notebooks

The notebooks teach practitioners to interpret Weight of Evidence explanations for a multiclass news-classification model, combining or grouping feature contributions and examining example predictions. They illustrate sequential explanations for correctly classified examples, showing how prior log-odds and total WoE contribute to posterior log-odds.

User Study on Interpretability - Tutorial

  • WoE’s sign indicates whether evidence favors, opposes, or is neutral toward a hypothesis, while its magnitude indicates evidence strength.The tutorial gives fever at WoE 3 as strong evidence for flu and headache at 0.7 as some, but not much, evidence.Source: §1 · p. 18
  • Under feature independence, individual WoE scores can be added to decompose a classifier’s prediction into feature contributions.The tutorial illustrates the sum with cough and fever, then describes the resulting breakdown as individual feature contributions to a prediction.Source: §1 · p. 18
  • When many features have meaningful groupings, the notebook presents aggregated group WoE scores as a convenient alternative to individual feature scores.Its running example groups symptoms into respiratory, overall body feeling, and temperature categories.Source: §1 · p. 19

User Study on Interpretability - Main Sections

  • The dataset contains 33,510 Mashable news articles, each described by 53 features, and the task is to predict its publication channel among six named categories; similar input features can be grouped.The examples use an 80%/20% training/testing split.Source: §1 · p. 20
  • For the shown article, the model predicts ‘world’ at p=0.99, with prior log-odds -1.09 plus total WoE 5.69 yielding posterior log-odds 4.60.The notebook identifies ‘world’ as the primary hypothesis and the other five channels as the alternative hypothesis.Source: §1 · p. 21

Jupyter Notebooks

  • For an entertainment example, the model predicted the true class with p=0.59 and total WoE of 1.68, added to prior log-odds of -1.32.The posterior log-odds were 0.36.Source: §1 · p. 21
  • In another example, the model predicted the true business class with p=0.75; its explanation favored business and socmed against world, with total WoE 3.74.The decomposition was 3.76 posterior log-odds = 0.02 prior log-odds + 3.74 total WoE.Source: §1 · p. 22
  • The notebook’s second study part uses the same dataset and model for different questions, with another correctly predicted business example at probability 0.75.Its sequential explanation first groups business, socmed, and world against entertainment, lifestyle, and tech, then compares business against socmed.Source: §1 · p. 22
  • For that example, step 1 assigns total WoE 1.96 to the three-class comparison, while step 3 assigns 0.92 to business against socmed.The corresponding decompositions are 1.99 = 0.03 + 1.96 and 1.92 = 1.00 + 0.92, respectively.Source: §1 · p. 22

Related papers

Loading 2104.13299v2…