PaperlayerView exact revision on arXiv

Source-linked AI summary

The Disagreement Problem in Explainable Machine Learning: A Practitioner's Perspective

Satyapriya Krishna, Tessa Han, Alex Gu, Steven Wu, Shahin Jabbari, Himabindu Lakkaraju

arXiv:2202.01602v6cs.LGcs.AI

TL;DR

Post hoc explanation methods can disagree, yet prior research has not adequately characterized those disagreements or how practitioners resolve them. This paper formalizes disagreement from practitioner input, evaluates it across datasets, modalities, methods, and models, and studies practitioner responses; disagreements are common and resolution often relies on ad hoc heuristics.

  • Problem

    Prior research has not extensively investigated disagreement among explanation methods or how practitioners resolve conflicting explanations in high-stakes applications.

  • Method

    The paper interviews data scientists, formalizes practitioner-informed disagreement criteria, evaluates explanations empirically, and studies how practitioners resolve conflicts.

  • Results

    State-of-the-art explanation methods consistently disagree across datasets and data modalities, while 84% of interviewed practitioners regularly face such disagreements.

  • Takeaways & Limitations

    The findings support principled and systematic evaluation and selection of explanation methods instead of practitioners’ current ad hoc resolution strategies.

  • Takeaways & Limitations

    The study prioritizes measuring disagreement prevalence rather than investigating its underlying causes.

Abstract

from arXiv · show

As various post hoc explanation methods are increasingly being leveraged to explain complex models in high-stakes settings, it becomes critical to develop a deeper understanding of whether and when the explanations output by these methods disagree with each other, and how such disagreements are resolved in practice. However, there is little to no research that provides answers to these critical questions. In this work, we formalize and study the disagreement problem in explainable machine learning. More specifically, we define the notion of disagreement between explanations, analyze how often such disagreements occur in practice, and how practitioners resolve these disagreements. We first conduct interviews with data scientists to understand what constitutes disagreement between explanations generated by different methods for the same model prediction, and introduce a novel quantitative framework to formalize this understanding. We then leverage this framework to carry out a rigorous empirical analysis with four real-world datasets, six state-of-the-art post hoc explanation methods, and six different predictive models, to measure the extent of disagreement between the explanations generated by various popular explanation methods. In addition, we carry out an online user study with data scientists to understand how they resolve the aforementioned disagreements. Our results indicate that (1) state-of-the-art explanation methods often disagree in terms of the explanations they output, and (2) machine learning practitioners often employ ad hoc heuristics when resolving such disagreements. These findings suggest that practitioners may be relying on misleading explanations when making consequential decisions. They also underscore the importance of developing principled frameworks for effectively evaluating and comparing explanations output by various explanation techniques.

1 Introduction

This paper formalizes disagreement among post hoc explanations, measures how often it occurs, and studies how practitioners resolve it. Interviews, empirical analysis, and a user study show that disagreements are widespread and often handled with ad hoc strategies.

  • Motivation: Prior work has studied explanation quality, but disagreement among different methods and how practitioners resolve it remain insufficiently investigated.Ground truth may be unavailable in real-world applications, and retraining-based evaluation can be impractical.
  • Contributions: The paper formalizes explanation disagreement and introduces a quantitative framework for comparing explanations of the same model prediction.The framework is based on practitioner input from semi-structured interviews.
  • Contributions: The empirical analysis measures disagreement across four real-world datasets, three data modalities, six explanation methods, and multiple predictive models.The predictive models include logistic regression, tree-based models, and neural networks.
  • Contributions: The online user study presents conflicting explanation pairs to data scientists and examines which explanations they would rely on and how they resolve disagreements.The study includes another set of 25 data scientists.
  • Findings: 84% of interviewees reported encountering explanation disagreements, while 86% of online-study participants used ad hoc heuristics or were uncertain about resolving them.The findings indicate widespread discrepancies across tabular, text, and image data modalities.

2 Related Work

Related work covers interpretable models, post hoc explanations, explanation-quality metrics, and human factors. The paper focuses specifically on disagreement among local post hoc explanation methods and extends prior disagreement analysis with practitioner-informed formalization and resolution studies.

  • Interpretable and post hoc models: Complex models such as deep neural networks often outperform simpler inherently interpretable models, motivating interest in post hoc explanations.Post hoc methods explain predictions after a complex model has been trained.
  • Interpretable and post hoc models: Global explanation methods summarize whole-model behavior, whereas this paper analyzes disagreements among local post hoc explanations.The distinction is between explanations of individual predictions and summaries of model behavior as a whole.
  • Explanation evaluation: Prior research evaluates explanation properties including faithfulness, stability, consistency, and sparsity using proposed metrics.These metrics have been used to analyze popular post hoc explanation methods theoretically and empirically.
  • Explanation evaluation: Neely et al. showed that several post hoc methods disagree according to Kendall’s τ, but did not formalize disagreement using practitioner input or study resolution strategies.The present work addresses both of those gaps.
  • Human factors: Human-factors studies report that data scientists can struggle to understand and effectively use state-of-the-art explanation techniques.Related studies also identify stakeholders and goals across the model life cycle.

3 Understanding and Measuring Disagreement Between Model Explanations

The authors use practitioner interviews to characterize explanation disagreement and derive metrics for comparing explanations. Practitioners focus on differences in selected features, their ordering, and contribution signs rather than directly comparing method-specific importance values.

  • Understanding disagreement: Interviews with 25 data scientists characterize disagreement in practice and inform a framework for quantitatively comparing two explanations of the same prediction.Participants used explainability techniques to understand model behavior and communicate it to customers and managers.
  • Understanding disagreement: 88% of participants almost always use multiple explanation methods for the same prediction, and 84% often encounter disagreements between them.The interviews also elicited the practical criteria participants use to identify disagreement.
  • Practitioner criteria: 84% identified differing top features as critical disagreement, while 72% also considered differences in the ordering of top features.Participants typically inspect the top 5 to 10 features.
  • Practitioner criteria: 76% considered mismatched contribution signs a disagreement, including whether a feature contributes positively or negatively to a prediction.The interviews include a loan-denial example where bankruptcy receives opposite contribution directions.
  • Practitioner criteria: 64% considered contradictory relative ordering between features of interest a disagreement.One example compares whether salary or credit score is more important for loan approval.
  • Practitioner criteria: 96% said feature-importance values from different methods are not directly comparable, so disagreement is judged by consistent insights rather than equal numerical values.LIME coefficients and SHAP Shapley values have different meanings and scales.
  • Disagreement metrics: Feature agreement measures the fraction of shared top-k features, while rank agreement additionally requires shared positions and is therefore stricter.A weighted rank variant could give partial credit when feature ranks are close rather than identical.
  • Disagreement metrics: Sign agreement requires common top-k features to have matching contribution signs, while signed rank agreement additionally requires matching ranks.Signed rank agreement is the strictest of the listed metrics because it combines feature identity, sign, and ordering.

4 Empirical Analysis of Explanation Disagreement

Across four real-world datasets, three modalities, six explanation methods, and varied predictive models, explanation methods frequently disagreed, with disagreement patterns depending on top-k size, method pair, model complexity, and data modality.

  • Experimental Setup: Four datasets spanning three modalities were analyzed with six post hoc explanation methods and predictive models including logistic regression, tree-based models, and neural networks.The datasets included tabular, text, and image data.
  • Tabular Data: As k increased, rank agreement and signed rank agreement decreased for COMPAS explanations.Larger top-k sets were less likely to contain shared features with the same rank.
  • Method-Pair Patterns: Vanilla Gradient–SmoothGrad and Gradient times Input–Integrated Gradients agreed strongly, whereas several other gradient-based pairs consistently disagreed more.The stronger-disagreement pairs included Vanilla Gradient–Integrated Gradients, Vanilla Gradient–Gradient times Input, SmoothGrad–Gradient times Input, and SmoothGrad–Integrated Gradients.
  • Tabular Data: 14 out of 15 COMPAS explanation-method pairs exhibited negative rank correlation when explaining multiple data points.The rank-correlation distribution varied widely across method pairs.
  • Text Data: Text explanations showed severe disagreement: rank and signed rank agreement were below 0.1 in most cases, while Gradient times Input–Vanilla Gradient feature agreement was 0.61.LIME had average rank correlation 0.273 with other methods, compared with 0.113 for KernelSHAP.
  • Image Data: Image-method agreement varied with representation granularity: KernelSHAP and LIME showed high agreement, but Integrated Gradients–SmoothGrad pixel-level rank correlation was 0.001.For KernelSHAP and LIME, the six reported metrics ranged from 0.8193 to 0.9535.

5 Resolving the Disagreement Problem in Practice: A Qualitative Study

A qualitative user study examined how practitioners perceive and resolve conflicting explanations. Participants reported substantial disagreement, showed no consistent resolution strategy, and often relied on method-specific preferences or heuristics.

  • User Study Design: The user study included 25 participants: 13 from academia and 12 from industry, with 20 reporting prior use of explanation methods.The study presented participants with COMPAS prediction explanations and asked how they judged and resolved disagreements.
  • User Study Design: Prompts displayed two explanations for the same COMPAS prediction, with feature importance values for all seven features and color-coded positive or negative contributions.Participants rated agreement and, when they perceived disagreement, chose one explanation, the other, or “it depends.”
  • Observed Disagreement: Only 4% of responses indicated complete agreement, while 50% indicated mostly disagreement and 18% complete disagreement.Another 28% of responses indicated mostly agreement.
  • Resolution Choices: Resolution choices varied substantially across prompts, although participants often favored the same method when they selected one rather than abstaining.KernelSHAP was chosen 66.7% of the time during disagreements, whereas Gradient times Input was selected less often.
  • Decision Themes: Participants’ selection rationales included method theory, publication recency, implementation ease, interpretability, stability, and package documentation.These themes summarized how participants decided between explanations when faced with disagreement.
  • Day-to-Day Workflow: Among 19 practitioners answering workflow questions, 74% had encountered explanation disagreement; 50% used ad hoc heuristics, 36% expressed uncertainty, and 14% used other metrics such as fidelity.The heuristics varied across participants and included personal preferences and rules of thumb based on published results.

6 Discussion and Conclusion

The paper finds that explanation methods frequently disagree across datasets and modalities, while practitioners often resolve conflicts with ad hoc heuristics. It formalizes disagreement through metrics grounded in practitioner-identified differences in features, ordering, signs, and relative rankings.

  • Discussion and Conclusion: State-of-the-art explanation methods consistently disagree across datasets and data modalities, with empirical disagreement estimates aligning closely with human assessments.The study combines empirical analyses with semi-structured interviews and an online user study.
  • Discussion and Conclusion: 84% of interviewed practitioners regularly face explanation disagreements, while 86% of online-study respondents use ad hoc heuristics or remain uncertain when resolving them.These findings indicate a gap between available evaluation frameworks and practitioners’ resolution practices.
  • Metrics: The framework measures feature, rank, sign, signed rank, rank correlation, and pairwise rank agreement, with lower values indicating stronger disagreement.The first four metrics target top-k features; rank correlation and pairwise rank agreement target user-selected features.
  • Practitioner-identified disagreement: Practitioners identify disagreement through mismatched top-k features, ordering, attribution signs, and relative ordering of features of interest.These operational notions motivate the framework’s agreement metrics.
  • Top-k metrics: Feature agreement compares shared top-k features, rank agreement compares shared positions, and sign agreement additionally requires matching attribution directions.Signed rank agreement combines feature overlap, sign agreement, and rank position, making it stricter than the other top-k metrics.
  • User-selected features: Rank correlation and pairwise rank agreement evaluate whether explanations preserve rankings or pairwise orderings over features selected by an end user.Pairwise rank agreement computes the fraction of feature pairs with matching relative order.

B.1 Black Box Models: Training and Performance

The experiments use tabular, text, and image models with task-specific architectures and train-test settings. Explanation methods are applied across these models, with gradient-based methods restricted to differentiable logistic-regression and neural-network settings for tabular data.

  • Tabular models: Tabular experiments train logistic regression, gradient-boosted trees, random forests, and densely connected neural networks.The neural network has three ReLU hidden layers with 50, 100, and 50 neurons.
  • Tabular performance: COMPAS test accuracies are 0.75, 0.79, 0.79, and 0.73 for logistic regression, gradient-boosted trees, random forests, and neural networks, respectively.All four models use an 80%-20% train-test split.
  • Tabular performance: German Credit test accuracies are 0.65, 0.70, 0.73, and 0.64 for the same four model types, respectively.The German Credit models also use an 80%-20% train-test split.
  • Text model: The text experiment uses an LSTM classifier with a 300-dimensional embedding, 256 hidden units, and a four-dimensional output layer.The classifier is trained on 120,000 samples and evaluated on 7,600 test samples, achieving 90.67% accuracy.
  • Image model: The image experiment uses pretrained ResNet-18 for 1,000-class image classification, achieving 69.758% Accuracy@1 and 89.078% Accuracy@5.The model is evaluated on image predictions using the stated top-1 and top-5 accuracy metrics.
  • Explanation methods: LIME and KernelSHAP explain all four tabular models, while Vanilla Gradients, Integrated Gradients, Gradient*Input, and SmoothGRAD explain logistic regression and neural networks.The gradient-based methods are not applied to tree-based models because gradients are unavailable there.

C.1 Tabular Data: COMPAS and German Credit Datasets

Across COMPAS and German Credit, explanation disagreement generally strengthens under stricter metrics and larger top-k sets. More complex models and the higher-dimensional German Credit dataset show similar or stronger disagreement, while gradient-based methods agree more with each other than with perturbation-based methods.

  • Metric trends: Stricter metrics, such as signed rank agreement, show stronger disagreement than less strict metrics, such as feature and sign agreement.Feature agreement equals one when k includes the total number of features.
  • Metric trends: Increasing the number of top-k features generally produces stronger disagreement between explanations.The feature-agreement exception follows directly from its definition when all features are included.
  • Model trends: Explanations for neural networks tend to show similar or stronger disagreement than explanations for logistic regression models.The paper relates this pattern to the greater difficulty of explaining more complex black-box models.
  • Dataset trends: German Credit explanations show similar or stronger disagreement than COMPAS explanations, possibly because German Credit has more features and therefore more ranking and sign combinations.The stated explanation is presented as one possible reason rather than a demonstrated cause.
  • Method trends: Agreement among gradient-based methods is significantly greater than agreement involving perturbation-based methods, especially for larger k values.This comparison uses feature agreement for k = {3, 7, 11}.

C.3 Image Data: ImageNet Dataset

The image analyses compare explanation disagreement across attribution resolutions and ranking metrics. Agreement can be high at the super-pixel level but little to none at the pixel level, while L2X shows particularly low agreement with other methods on COMPAS.

  • COMPAS comparisons: L2X shows poor agreement with other explanation methods for COMPAS neural-network and logistic-regression models.The comparison is presented in Figures 10 and 11.
  • Weighted rank agreement: Weighted rank agreement is a softer rank metric that accounts for rank differences, assigning 1 to exact rank matches, 0 to no top-k overlap, and intermediate values otherwise.It uses the top-k feature sets and ranks from both explanations.
  • Weighted rank agreement: Weighted rank agreement reports higher agreement than the original rank agreement on COMPAS because the original metric treats non-exact rank matches as zero.The comparison is shown in Figure 12 against the corresponding plots in Figures 10 and 11.
  • Top-k pairwise rank agreement: Top-k pairwise rank agreement applies pairwise ordering comparisons to the union of the two explanations’ top-k features.The relative ordering function indicates whether one feature is more important than another according to an explanation.
  • Top-k comparisons: L2X exhibits the lowest top-k rank correlation with other methods for both COMPAS logistic-regression and neural-network models.Figures 13 and 14 show this pattern for top-k pairwise rank agreement and top-k rank correlation.
  • Top-k rank correlation: Top-k rank correlation computes Spearman correlation over the union of the two explanations’ top-k features.The rank vector contains the rank of every feature in that union according to each explanation.
  • Feature-count analysis: Figure 15 summarizes rank correlation and pairwise rank agreement across increasing feature counts, with lighter heatmap cells indicating stronger disagreement.Each cell averages the metric over test-set data points for a pair of explanation methods.

D.3 Effects of Dataset Complexity

The analysis examined whether increasing tabular dataset complexity changes disagreement among explanation methods, while the accompanying study materials introduced the COMPAS setting and participant tasks.

  • D.3 Effects of Dataset Complexity: Four tabular datasets ranged from 7 to 128 features for analyzing disagreement among six explanation methods on neural-network models.The datasets were COMPAS, German Credit, Forest Cover, and Gas Concentration.
  • D.3 Effects of Dataset Complexity: Increasing tabular features from 7 to 128 did not produce a drastic overall change in disagreement.
  • D.3 Effects of Dataset Complexity: LIME and KernelSHAP showed substantially lower rank correlation with other methods as tabular feature counts increased.
  • D.3 Effects of Dataset Complexity: The study introduction familiarized participants with the COMPAS prediction setting, six explanation methods, and the explanation plots used in prompts.
  • D.3 Effects of Dataset Complexity: Participants evaluated agreement, explained their judgments, selected a preferred explanation or “It depends,” and reported their methods, modalities, and disagreement-resolution practices.

E.4 Further analysis of overall agreement levels

Participant responses showed substantial disagreement between explanation pairs, with disagreement levels varying across prompts and algorithm combinations.

  • E.4 Further analysis of overall agreement levels: 68% of responses indicated that the paired explanations mostly or completely disagreed.
  • E.4 Further analysis of overall agreement levels: Only 32% of responses indicated that the paired explanations mostly or completely agreed.
  • E.4 Further analysis of overall agreement levels: Disagreement levels varied across prompts and algorithm pairs.All participants judged Gradient versus SmoothGrad to agree to some extent, whereas all participants judged Gradient versus Integrated Gradients to disagree to some extent.
  • E.4 Further analysis of overall agreement levels: Figure 19 aggregated the four response categories across prompts and also displayed response frequencies separately by prompt.

E.6 Analysis of reasons participants chose neither algorithm

The analysis examined why participants selected “It depends” when choosing between disagreeing explanations, using responses from the online study and representative preference quotes.

  • E.6 Analysis of reasons participants chose neither algorithm: “It depends” was selected in 38% of cases when participants were asked to choose between disagreeing explanations.
  • E.6 Analysis of reasons participants chose neither algorithm: The analysis examined 19 responses from participants who answered follow-up questions about their explanation-method use and preferences.One of 20 positive responses declined the remaining questions.
  • E.6 Analysis of reasons participants chose neither algorithm: Participants used explanations for understanding and debugging models, communicating with clients, and conducting research.
  • E.6 Analysis of reasons participants chose neither algorithm: The follow-up analysis included participants’ day-to-day methods, data modalities, and explanations for resolving disagreements.
  • E.6 Analysis of reasons participants chose neither algorithm: Participants preferred methods because they were broadly applicable, understandable, theoretically justified, easy to implement, or suited to local behavior.

E.8 Breakdown of the Results

Academic and industry participants used different preferences and rationales when resolving explanation disagreements, although theoretical foundations, intuitive explanations, and data suitability were recurring themes.

  • E.8 Breakdown of the Results: Across both groups, participants relied on theoretical foundations, intuitive explanations, and suitability for tabular data.
  • E.8 Breakdown of the Results: One reported resolution strategy was making arbitrary decisions, accounting for 50% of responses.
  • E.8 Breakdown of the Results: Academic and industry participants differed in preferred explanations for specific method pairs when disagreements occurred.Academics preferred neither for LIME versus Integrated Gradients, while half of industry participants preferred LIME; industry participants always preferred SHAP over Gradients.
  • E.8 Breakdown of the Results: SHAP was preferred by 80% of industry participants and 60% of academic participants when explanations disagreed.
  • E.8 Breakdown of the Results: Industry participants emphasized theoretical rigor more often than academics, at 42% versus 25%.
  • E.8 Breakdown of the Results: Academic participants prioritized explanations matching their intuition more often than industry participants, at 40% versus 26%.
Loading 2202.01602v6…