Source-linked AI summary
Explainable Machine Learning in Deployment
Umang Bhatt, Alice Xiang, Shubham Sharma, Adrian Weller, Ankur Taly, Yunhan Jia, Joydeep Ghosh, Ruchir Puri, José M. F. Moura, Peter Eckersley
TL;DR
Organizations increasingly use explainability to inspect and debug ML models, but there is limited understanding of these deployments in practice. Through interviews and synthesis of organizational use cases, the paper finds that explanations primarily serve ML engineers rather than affected end users and identifies limitations that constrain direct user-facing use.
Problem
There has been little empirical understanding of how organizations deploy explainability techniques for stakeholder consumption, despite their proposed role in transparency.
Method
The study synthesizes interviews with data scientists and organizational practitioners, examining deployed techniques, audiences, contexts, evaluation, benefits, and shortcomings.
Results
Explanations are used mainly by ML engineers and data scientists as development-time sanity checks and model audits rather than directly informing end users.
Takeaways & Limitations
The paper develops a framework for organizations to establish clear explainability goals and identifies directions for making explanations more suitable for end users.
Takeaways & Limitations
Current techniques often lack causal explanations, while organizations report difficulties scaling local heatmap analysis across very large video datasets.
Abstract
from arXiv · showhide
Explainable machine learning offers the potential to provide stakeholders with insights into model behavior by using various methods such as feature importance scores, counterfactual explanations, or influential training data. Yet there is little understanding of how organizations use these methods in practice. This study explores how organizations view and use explainability for stakeholder consumption. We find that, currently, the majority of deployments are not for end users affected by the model but rather for machine learning engineers, who use explainability to debug the model itself. There is thus a gap between explainability in practice and the goal of transparency, since explanations primarily serve internal stakeholders rather than external ones. Our study synthesizes the limitations of current explainability techniques that hamper their use for end users. To facilitate end user interaction, we develop a framework for establishing clear goals for explainability. We end by discussing concerns raised regarding explainability.
1 INTRODUCTION
Explainability aims to help stakeholders understand why ML models behave as they do, supporting transparency targeted to human users. This study examines how organizations deploy these techniques and proposes goals for their use.
- Explainability helps users or developers understand why an ML model behaves the way it does.
- Transparency efforts target stakeholders, particularly end users, with information about model operation and decision-making.
- Organizations’ real-world deployment of explanations has received little empirical characterization despite growing research interest.
- The study explores organizational deployments of local explainability, reports practical shortcomings, and recommends future research directions.
- Interviews with roughly fifty people from approximately thirty organizations examine which techniques are used and how.
- The paper proposes a framework for organizations to clarify their goals when deploying explainability.
2 METHODOLOGY
The study uses two interview groups to examine explainability needs and deployments across organizations. Interviews cover motivations, audiences, contexts, benefits, and shortcomings of deployed methods.
- Group 1 comprised around twenty interviews with data scientists whose organizations did not currently use explainability tools.
- Group 1 interviews examined ML use cases, development workflows, deployment pain points, and whether explainability could address them.
- Group 2 included roughly thirty people across approximately twenty for-profit and nonprofit organizations.
- Group 2 interviews investigated organizational explainability use, motivations, benefits, and shortcomings through thirty-minute to two-hour semi-structured sessions.
- The study gathered perspectives from executives, research scientists, engineers, professors, academics, civil societies, think tanks, corporations, and startups.
3 SUMMARY OF FINDINGS
Organizations use explainability mainly for internal model auditing and debugging rather than direct end-user communication. Current methods face expert-evaluation, causal, technical, privacy, and goal-setting limitations.
- Explainability Needs: Data scientists seek explainability for model debugging, monitoring distribution drift, and communicating predictions to external audiences.
- Explainability Usage: Feature importance was the most common technique, while Shapley values were the most common feature-importance explanation.
- Stakeholders: ML engineers use explanations to debug learned behavior by checking whether important features and similar or opposite-class examples make sense.
- Stakeholders: Local explainability is mostly consumed by ML engineers and data scientists to audit models before deployment, not to explain decisions to end users.
- Key Takeaways: Organizations often rely on domain experts to evaluate explanations because current methods can reflect spurious, noncausal correlations.
- Key Takeaways: Real-time end-user explanations are hindered by computation costs, implausible counterfactuals, and privacy risks from model inversion.
4 DEPLOYING LOCAL EXPLAINABILITY
The paper reviews how local explainability techniques are used across organizations, beginning with technical notation and definitions before presenting use cases and technique-specific takeaways.
- The section examines local explainability deployments across organizations and reports use cases and takeaways for each technique.
4.1 Preliminaries
The paper focuses on local explainability: explaining why a model produced a prediction for one fixed input. It distinguishes this from global explainability, which seeks to characterize the model as a whole.
- A black-box model f maps an input x to an output f(x), with training based on a loss function over input-output pairs.
- Local explainability explains why a model predicted f(x) for a fixed input x.
- Local methods ask which feature, training datapoint, or minimal input change most influenced a prediction.
- The paper deliberately emphasizes popularly deployed local techniques rather than methods that explain the model globally.
4.2 Feature Importance
Feature importance is the most widely used explainability technique in the organizations studied, supporting model checks across financial, healthcare, content-moderation, and transportation-related applications. Its practical value is strongest for internal review, while scaling heatmap explanations and exposing them to end users remain difficult.
- Feature importance was by far the most popular technique across the study and appeared in finance, healthcare, facial recognition, and content moderation.
- Feature importance functions take a model and input point and return importance scores for all features.
- Shapley explanations let risk analysts check whether models use expected features and can support customer-facing accounts of loan or cash-distribution decisions.Organizations represented contributions with signed percentages, such as age contributing 55% positively to a decision.
- Organization A found that LIME produced unexpected explanations misaligned with human intuition, while end users were not yet shown feature-importance explanations.
- Integrated Gradients helped content moderators see which word was most important when flagging malicious reviews.
- Organization B used gradient-based visualizations for qualitative debugging, including false-positive analysis and detection of spurious artifacts.
- Aggregating heatmaps across 100M frames was difficult because pixel attributions were not meaningful in aggregate and proposed clustering still required manual labeling.
- Spurious correlations could be detected through explanations, while simpler models over engineered descriptors made important descriptors more obvious for downstream predictions.
4.3 Counterfactual Explanations
Counterfactual explanations describe nearby inputs that would change a model’s decision, making them flexible and accessible to end users. Organizations nevertheless face unresolved questions about feasibility, optimization, efficiency, and guarantees for black-box models.
- Counterfactual explanations provide recourse by identifying nearby inputs for which the classifier’s decision changes.For example, increasing income by $5000 could change a loan rejection to an approval.
- Counterfactual generation can be formulated as an optimization problem over a classifier, input, and distance metric, with constraints on the resulting prediction.
- The method can restrict which features may change, but its meaning differs from counterfactuals in the causality literature.
- Organization C used a faster counterfactual formulation for Medicare applications and used a normalized average distance as a model robustness score.
- The approach supports black-box models, any input data type, and multiple explanations in one algorithm run.
- Clients wanted explainability, fairness, and robustness measures alongside counterfactuals, prompting Organization C to develop an explainability score for comparing models.
- Organizations value counterfactual solutions because they are flexible and easy for end users to understand.
- It remains unclear what objective to optimize and how to generate counterfactuals efficiently, although approximate solutions may suffice in practice.
4.4 Adversarial Training
Organizations use adversarial training and perturbation-based analyses to evaluate robustness, while feature importance helps interpret and guide minimal adversarial changes. More robust models can produce more convincing gradient-based explanations.
- Adversarial Training: Adversarial training can improve model robustness and produce more human-interpretable features.The paper links robustness improvements with higher-quality feature importance, specifically saliency maps.
- Image Content Moderation: Organization D measures image-model robustness by the minimum perturbation required to evade detection.Its QA team averages the ℓ2 perturbation distance over randomly sampled test images; larger averages indicate greater robustness.
- Image Content Moderation: Organization D finds that more robust image models produce gradient explanations focused on relevant image regions.These explanations are used as evidence that the model is attending to appropriate portions of images.
- Text Content Moderation: Organization E ranks words by gradient-based importance and perturbs the most important words to find minimal changes that flip sentiment predictions.The importance score is based on the partial derivative of the predicted class confidence with respect to each input word.
- Adversarial Training: Feature importance helps identify minimal adversarial perturbations for language models in practice.The paper presents this as a practical connection between explainability and adversarial robustness.
4.5 Influential Samples
Influence functions identify training examples that affect a model’s output, but their practical usefulness is limited by computational cost and sensitivity to outliers. One organization intended to use them to explain insurance pricing, yet influential samples often reflected anomalous drivers rather than similar customers.
- Influential Samples: Traditional influence measures such as Cook’s distance require exhaustive searches and do not scale well to large datasets.The paper identifies tractability as a barrier to deploying influence-based explanations in practice.
- Influential Samples: Influence functions estimate which training data points most affect a model’s output for a particular test point.The method measures the effect of upweighting a training example on the test point’s loss.
- Organizational Use: Organization F hoped to use influential training data to explain how drivers’ histories affect insurance premiums.The intended explanation would show end users which past drivers most influenced pricing for a driver like them.
- Limitations: Organization F found that influential samples were often outliers rather than drivers similar to the observed customer.These anomalous points were far out of distribution and influential across many drivers, potentially making the method more useful for outlier detection.
- Limitations: Influence functions are sensitive to outliers, and substantial effort is needed to make them easier to deploy.The paper treats both computational intractability and outlier sensitivity as practical limitations.
5 RECOMMENDATIONS
The paper recommends designing explainability around clearly identified stakeholders, their needs, and the intended purpose of explanations, while recognizing unresolved concerns including limited causal explanations, privacy risks, and possible misuse.
- Establish Clear Desiderata: Organizations should identify who will consume explanations, especially stakeholders affected by or shown model outputs.
- Establish Clear Desiderata: Stakeholders should be engaged to determine what they need explanations to clarify, trust, or contest about model predictions.
- Establish Clear Desiderata: Organizations should clarify whether explanations support static consumption, dynamic model updates, or another stakeholder task.
- Establish Clear Desiderata: After desiderata are clarified, stakeholders should be consulted again to assess whether the explanation serves its intended purpose.
- Concerns of Explainability: Causal explanations remain largely unavailable, although organizations expressed interest in using them and non-causal explanations can still be useful.
- Concerns of Explainability: Explanations can expose information about models or training data, creating privacy concerns that motivate methods such as differentially private models.
- Concerns of Explainability: Explainability tools can improve model performance while potentially enabling malicious users to increase capabilities of undesirable systems.
- Concerns of Explainability: Practitioners’ continued use of classical models creates demand for model-specific explanations, while model-agnostic methods may be computationally expensive and poorly approximated for kernel-based models.
6 CONCLUSION
The study examines how organizations use explainability in practice and finds that engineers increasingly use it as a development sanity check, while limitations still restrict direct use with end users. It recommends clearer organizational goals and further research toward trustworthy explainability.
- The study finds that ML engineers increasingly use explainability techniques as sanity checks during model development.
- Current techniques have limitations that prevent them from directly informing end users, including domain-expert evaluation needs, spurious correlations, limited causal intuition, and real-time latency.
- The paper calls for organizations to establish clear desiderata and recognize concerns associated with explainability.
- The authors present the analysis as a step toward trustworthy explainability solutions and identify these limitations as directions for future research.