Source-linked AI summary
Human Factors in Model Interpretability: Industry Practices, Challenges, and Needs
Sungsoo Ray Hong, Jessica Hullman, Enrico Bertini
TL;DR
As ML interpretability research expands, little is known about how practitioners understand and use interpretability in organizational workflows. The paper analyzes 22 semi-structured interviews with industry practitioners and finds interpretability to be cooperative, socially negotiated work aimed at building trust among people and between people and models.
Problem
Little is known about how practitioners perceive and provide interpretability within existing workflows, limiting understanding of practical needs.
Method
The study conducts semi-structured interviews with 22 industry ML practitioners and qualitatively analyzes their interpretability roles, processes, goals, and strategies.
Results
Interpretability is context-dependent and socially negotiated across stakeholder roles and development stages, involving comparisons of human mental models and trust-building.
Takeaways & Limitations
Interpretability research and technology should address communication-oriented challenges and bridge gaps between research framings and practitioners’ needs.
Takeaways & Limitations
The study focuses on human-consumed predictions and reflects the perspectives of many technically trained data scientists, excluding fully automated settings and some domains.
Abstract
from arXiv · showhide
As the use of machine learning (ML) models in product development and data-driven decision-making processes became pervasive in many domains, people's focus on building a well-performing model has increasingly shifted to understanding how their model works. While scholarly interest in model interpretability has grown rapidly in research communities like HCI, ML, and beyond, little is known about how practitioners perceive and aim to provide interpretability in the context of their existing workflows. This lack of understanding of interpretability as practiced may prevent interpretability research from addressing important needs, or lead to unrealistic solutions. To bridge this gap, we conducted 22 semi-structured interviews with industry practitioners to understand how they conceive of and design for interpretability while they plan, build, and use their models. Based on a qualitative analysis of our results, we differentiate interpretability roles, processes, goals and strategies as they exist within organizations making heavy use of ML models. The characterization of interpretability work that emerges from our analysis suggests that model interpretability frequently involves cooperation and mental model comparison between people in different roles, often aimed at building trust not only between people and models but also between people within the organization. We present implications for design that discuss gaps between the interpretability challenges that practitioners face in their practice and approaches proposed in the literature, highlighting possible research directions that can better address real-world needs.
1 INTRODUCTION
As ML adoption and model complexity increased, interpretability became important for understanding and verifying models in sensitive applications. This study examines how industry practitioners practice interpretability and finds it is socially negotiated across organizational roles and stages.
- Motivation: Growing ML adoption and complexity make it harder for people to understand model behavior and predict future failures.The concern is especially acute in sensitive applications where mistakes can have catastrophic consequences.
- Interpretability research: Interpretability research generally seeks alignment between a user’s mental model and a model’s actual behavior.Such alignment can improve human-model task performance and help users debug models.
- Study focus: 22 practitioners from 20 companies were interviewed about interpretability roles, processes, goals, and challenges in industry.Participants built models used by millions of daily users or supporting critical decisions.
- Findings: Interpretability work involves model builders, model breakers, and model consumers across conceptualization, building, and deployment.Their tasks, strategies, and roadblocks vary by stakeholder role and development stage.
- Findings: Interpretability is socially negotiated through comparisons among stakeholders’ mental models rather than being solely a property of a model.Practitioners connect this process with trust in models and organizational trust among people.
- Implications: The study motivates design opportunities for communication-oriented interpretability challenges in organizations.It aims to bridge differences between interpretability as understood in research and practice.
2 RELATED WORK
Related work frames interpretability through model structure, explanation techniques, human mental-model alignment, and concerns about misuse and ambiguous definitions. The paper identifies a need for empirical study of interpretability in real organizational contexts.
- Background: White-box models expose decision logic, whereas black-box models have complex input-output relationships that are less intelligible.Even transparent models can become difficult to understand when they contain many variables, nodes, or conditions.
- Interpretability approaches: Global techniques explain model structure, while local techniques explain individual predictions using weights, rules, or counterfactual changes.Model extraction and surrogate methods can approximate black-box behavior globally.
- Interpretability approaches: Interactive visualization systems help data scientists build, interpret, debug, and compare models.Examples include multiple confusion matrices and single-view comparisons of model performance.
- Critiques: Interpretability tools can contribute to over-reliance or over-trust, including reliance on model recommendations over professional expertise.Research also examines how explanation complexity affects accuracy, efficiency, and user satisfaction.
- Critiques: Interpretability is criticized as ill-defined because related terms describe differing concepts and trust has multiple possible meanings.This ambiguity makes the scope of interpretability and its evaluation metrics difficult to establish.
- Research gap: Few studies empirically explain how ML professionals perform interpretability-related tasks and communicate model behavior in organizations.The paper therefore seeks to ground interpretability research in practitioners’ real-world needs and unsupported gaps.
3 RESEARCH METHOD
The researchers used open-ended semi-structured interviews and iterative qualitative analysis to characterize interpretability practices among industry ML professionals across domains.
- Research design: The study used open-ended conversational interviews to characterize how ML experts conceive of interpretability and identify technology gaps.Interviews were chosen to reach practitioners across multiple industry domains.
- Recruitment: Convenience and snowball sampling recruited practitioners with experience addressing interpretability issues or communicating model results.Participants were building or using state-of-the-art ML solutions in their fields.
- Participants: 22 individuals from 20 companies represented domains including banking, healthcare, software, transportation, consulting, and manufacturing.Most participants identified as data scientists or machine learning engineers, and teams communicated with product managers, customers, and other stakeholders.
- Data collection: Semi-structured interviews covered participants’ roles, projects, interpretability experiences, organizational challenges, and desired future developments.The researchers collected 19 hours and 10 minutes of audio across 22 sessions averaging 52 minutes.
- Analysis: Professional transcription and multiple-author analysis produced an interpretive framework representing participants’ perspectives.The first and third authors analyzed the data together, while the second author independently corroborated and extended the analysis.
- Analysis: Iterative qualitative coding alternated among coding text segments, writing analytic memos, and diagramming themes and categories.The process began with open coding and pre-coding on subsets of interviews.
4 RESULTS
The results are organized around interpretability roles, stages, and goals. These anchors support an account of interpretability as a socially situated activity spanning organizational model work.
- Interpretability Roles: Interpretability Roles describes who is involved in interpretability work.
- Stages: Stages describe what activities occur and when during planning, building, deploying, and managing ML models.
- Goals: Goals describe why practitioners perceive interpretability as worth pursuing, while intersecting themes extend and problematize prior characterizations.
4.1 Interpretability Roles, Stages, Goals
Interpretability work spans stakeholder roles and model-building stages, combining technical analysis with collaboration to validate models, communicate their behavior, and build trust.
- Roles: Stakeholders contribute as Model Builders, Model Breakers, or Model Consumers, with distinct roles in interpretability work.
- Conceptualization Stage: Interpretability begins during conceptualization, when builders consider feature choices and involve domain experts or auditors to ensure features make sense and avoid legal risks.
- Building and Validation Stage: During building and validation, practitioners use instance-, feature-, and model-focused lenses, including test cases, feature importance, model comparison, and attribution methods.
- Building and Validation Stage: LIME and SHAP help practitioners develop intuitions about individual decisions, but participants expressed uneasiness about instability, limited transparency, and marginal claims.
- Goals: Interpretability supports builders’ confidence in model reliability and validity while helping them obtain trust from stakeholders, especially in high-stakes settings.
- Communication: Communication with model breakers and other stakeholders is iterative, yet practitioners lack effective methods and tools for conveying model behavior, trustworthiness, and failure conditions.
- Deployment, Maintenance and Use Stage: After deployment, interpretability concerns extend from individual models to the surrounding infrastructure, including diagnosing whether failures arise from the model or data engineering.
4.2 Themes: Characterizing Interpretability Work
Participants characterized interpretability as collaborative work that coordinates stakeholder knowledge, builds organizational trust, persists across the model lifecycle, and compares multiple mental models. Interpretability also supported dialogue, learning, sense-making, and context-sensitive decisions about model behavior.
- Interpretability is Cooperative: Interpretability frequently involved collaboration that coordinated values and knowledge across stakeholder roles, including domain experts, during ideation, validation, and deployment.Participants linked this collaboration to stronger business reasoning and clearer modeling implications.
- Interpretability is Cooperative: Interpretability helped build trust among data scientists, clients, auditors, and other organizational stakeholders, sometimes enabling greater flexibility in model choice and monitoring.Teams used tools, plots, and information to help others communicate model behavior up the organizational chain.
- Interpretability is Process: Interpretability was treated as an ongoing organizational target rather than a property definitively achieved at one point in the model lifecycle.Practices and concerns recurred from early feature selection through post-deployment monitoring and stakeholder meetings.
- Interpretability is Dialogue: Participants valued interpretability as dialogue in which users perturb models, ask questions, and use counterfactuals to learn about modeled phenomena.This dialogue supported continuing information transfer, human learning, and sense-making.
- Interpretability is Comparative: Interpretability work compared mental models held by models, ML experts, domain experts, and end-users to refine goals and judge mismatches.Teams used stakeholder expectations and contextual knowledge to decide whether behavior reflected a model problem or an inaccurate human assumption.
- Interpretability is Contextual: Interpretability solutions were shaped by user-group needs, including which information domain experts considered actionable or relevant to treatment decisions.Participants emphasized that generic feature rankings might not help particular users without domain-specific curation.
5 DESIGN OPPORTUNITIES FOR INTERPRETABILITY CHALLENGES
The paper identifies design opportunities for interpretability tools that better support human expectations, communication, workflow integration, model comparison, lifecycle stages, and evaluation of practical value. These opportunities respond to challenges involving collaboration, accessibility, scalability, and post-deployment monitoring.
- 5.1 Identifying, Representing, and Integrating Human Expectations: Identifying important edge cases systematically could improve interpretability productivity for both individual and group comparisons with models.The paper frames edge cases as anchors for interpretability work.
- 5.1 Identifying, Representing, and Integrating Human Expectations: Shape constraints can represent human expectations and regulate model behavior, but mismatches may reflect inaccurate human assumptions rather than model errors.Participants described such assumptions as “superstitions” when they were folk wisdom rather than factual expectations.
- 5.1 Identifying, Representing, and Integrating Human Expectations: Interpretability tools should help users articulate expectations, detect gaps between expectations and model behavior, gather evidence, and debug or change models.The paper points to belief elicitation and mental-model discovery as relevant design resources.
- 5.2 Communicating and Summarizing Model Behavior: Interpretability communication should provide direct, customized information about why model behavior disagrees with a human mental model.Participants specifically connected this need to debugging non-monotonic behavior against expected shape constraints.
- 5.2 Communicating and Summarizing Model Behavior: More accessible model-building and visualization tools could serve non-specialists and communicate model uncertainty through intuitive abstractions.The proposed direction includes hiding technical complexity behind model construction and improving public-facing visualizations.
- 5.3 Integrating Interpretability into Workflows: Industry adoption is constrained when interpretability methods lack robust software, cannot adapt to existing platforms, or fail to scale to large datasets.Participants described difficulty integrating academic tools into complex workflows and organizational infrastructure.
- 5.3 Integrating Interpretability into Workflows: Interpretability support should cover model comparison, pre-training data debugging and feature design, post-deployment monitoring, and analysis of whether explanations improve business decisions.Participants sought tools for comparing models, testing feature hypotheses, continuously evaluating deployed systems, and tracing root causes.
6 LIMITATIONS
The study’s scope excludes several domains and settings and reflects primarily data scientists’ technically informed perspectives. The authors therefore identify observational work and broader stakeholder participation as future ways to corroborate the findings.
- Scope: The analysis omits domains such as governance and agriculture, covers cases where humans consume predictions, and excludes fully automated or semi-automated applications.Examples include predictive policing, crop monitoring, autonomous driving, and warehouse robots.
- Participants: Many participants were technically trained data scientists, and the study did not directly hear from every stakeholder role identified as relevant to interpretability.The authors describe the results as reflecting this specific perspective through qualitative analysis.
- Future Work: Future ethnographic and observational studies could corroborate the interview-based findings.This is proposed alongside broader coverage of stakeholder roles and application domains.
7 CONCLUSION
The paper empirically characterizes how industry practitioners organize interpretability through roles, processes, goals, and strategies. Its findings contrast with treating interpretability solely as alignment between one person’s mental model and an ML model.
- Conclusion: The study describes interpretability practices, challenges, and needs among ML practitioners working across multiple industry domains.Its characterization covers roles, processes, goals, and strategies.
- Conclusion: The findings challenge the predominant framing of interpretability as a model property defined solely by an individual’s alignment with an ML model.The conclusion presents this as an important contrast between industry practice and existing research.