Source-linked AI summary
"The Human Body is a Black Box": Supporting Clinical Decision-Making with Deep Learning
Mark Sendak, Madeleine Elish, Michael Gao, Joseph Futoma, William Ratliff, Marshall Nichols, Armando Bedoya, Suresh Balu, Cara O'Brien
TL;DR
Healthcare machine learning research has focused more on state-of-the-art models than on ethical and professional challenges arising during situated implementation. Through an empirical case study of Sepsis Watch, the paper examines machine learning as a socio-technical system and argues that trust, accountability, and transparency require more than model interpretability.
Problem
Healthcare machine learning lacks sufficient attention to real-world implementation challenges involving fairness, transparency, accountability, bias, and disruptions to professional expertise and work cultures.
Method
The paper presents an empirical case study of Sepsis Watch spanning its development, implementation, and evaluation, informed by the authors’ practitioner and research experiences.
Results
The case study shows that treating Sepsis Watch as a socio-technical system, rather than merely a deployed model, has implications for trust, accountability, and transparency in practice.
Takeaways & Limitations
Clinical machine learning should rigorously define problems in context, build stakeholder relationships, respect professional discretion, and create ongoing feedback loops.
Takeaways & Limitations
Model interpretability alone is limited as a way to ensure transparency, accuracy, and accountability in healthcare practice.
Abstract
from arXiv · showhide
Machine learning technologies are increasingly developed for use in healthcare. While research communities have focused on creating state-of-the-art models, there has been less focus on real world implementation and the associated challenges to accuracy, fairness, accountability, and transparency that come from actual, situated use. Serious questions remain under examined regarding how to ethically build models, interpret and explain model output, recognize and account for biases, and minimize disruptions to professional expertise and work cultures. We address this gap in the literature and provide a detailed case study covering the development, implementation, and evaluation of Sepsis Watch, a machine learning-driven tool that assists hospital clinicians in the early diagnosis and treatment of sepsis. We, the team that developed and evaluated the tool, discuss our conceptualization of the tool not as a model deployed in the world but instead as a socio-technical system requiring integration into existing social and professional contexts. Rather than focusing on model interpretability to ensure a fair and accountable machine learning, we point toward four key values and practices that should be considered when developing machine learning to support clinical decision-making: rigorously define the problem in context, build relationships with stakeholders, respect professional discretion, and create ongoing feedback loops with stakeholders. Our work has significant implications for future research regarding mechanisms of institutional accountability and considerations for designing machine learning systems. Our work underscores the limits of model interpretability as a solution to ensure transparency, accuracy, and accountability in practice. Instead, our work demonstrates other means and goals to achieve FATML values in design and in practice.
1 INTRODUCTION
The paper examines Sepsis Watch as a socio-technical system for addressing the ethical and practical challenges of implementing machine learning in clinical care. It argues that trust, accountability, and transparency require contextual practices beyond model interpretability.
- Healthcare machine learning research has emphasized state-of-the-art models more than situated implementation and its fairness, transparency, and accountability challenges.The authors also identify unresolved questions about ethical model construction, explaining outputs, addressing bias, and protecting professional expertise and work cultures.
- The paper presents an empirical case study of Sepsis Watch, covering its development, implementation, and evaluation as a tool supporting clinicians’ early diagnosis and treatment of sepsis.The case study draws on the authors’ experiences as members of the development and evaluation team.
- Because sepsis lacks a universally accepted definition and is difficult to predict, the team prioritized documentation and institution-specific validation over model interpretability for building clinician trust.The model had to articulate a ground truth despite the absence of an accepted one.
- Sepsis Watch was developed as a socio-technical system integrating technical and institutional infrastructures with professionals making contextual clinical decisions.This framing treats the tool as more than a model that generates risk scores.
- The paper identifies four practices for clinical machine learning: define problems rigorously in context, build stakeholder relationships, respect professional discretion, and create ongoing feedback loops.These practices are presented as conclusions drawn from the team’s experiences and observations.
- The case study argues that model interpretability alone does not ensure fairness or accountability, pointing instead to alternative mechanisms for achieving FATML values in design and practice.Its contribution is socially situated evidence about a deeply integrated clinical decision-support tool and the institutional contexts required for responsible use.
2 MACHINE LEARNING FOR HEALTHCARE
This section questions whether explainability can by itself address accountability, transparency, trust, and adoption in healthcare machine learning. It emphasizes that clinical technologies must be understood within professional, organizational, and institutional contexts.
- Healthcare machine learning adoption remains underexamined as a process affecting organizational functioning, professional relationships, fairness, transparency, and accountability.The literature contains limited analysis of how models challenge healthcare organizations and clinicians’ relationships to their work.
- Explainable machine learning is often proposed as a way to protect patient safety and ensure professional accountability, but this focus rests on several assumptions.One assumption is that clinicians possess the technical and quantitative expertise needed to interrogate and improve such systems.
- Clinicians may not want greater orientation toward technology rather than patients, and clinical decisions can use information without full knowledge of how it was generated.The section notes that specialized pathologists generate laboratory results used by clinicians who understand operating characteristics without necessarily understanding the underlying mechanisms.
- Sepsis illustrates the limits of explainability-centered approaches because its definition and underlying pathophysiology remain incompletely understood.The section asks what an explainable algorithm should mean when the clinical condition itself is difficult to define and explain.
- There is no consensus about what constitutes interpretable or explainable artificial intelligence, and explanations must serve specific normative ends within broader social contexts.This lack of consensus further complicates explainability as a general solution for healthcare accountability and transparency.
- Machine learning adoption discussions often treat the mechanisms by which professionals trust and use technologies as an afterthought, with clinicians rarely involved in development or explainability discussions.Innovation frameworks additionally emphasize perceived benefit, ease of use, alignment with users’ values, and socio-technical organizational change.
- Healthcare machine learning requires new trust and accountability mechanisms involving hospital leaders and front-line staff, while regulatory frameworks remain nascent.The section connects responsible adoption to incorporating machine-generated predictions into clinical practice.
3 CASE STUDY: SEPSIS WATCH
The case study examines Sepsis Watch, a machine learning platform integrated into clinical practice to support sepsis detection, treatment, and guideline compliance. It combines practitioner reflection with interviews and observations to analyze the tool’s development, implementation, and effects within a hospital context.
- System and setting: Sepsis Watch is a platform combining a deep learning model, application, data pipeline, database, and real-time web services.It processes Emergency Department EHR data and presents high-risk patients to monitoring nurses through an iPad application.
- Study approach: The case study draws on practitioner, clinician, and researcher experiences over 2.5 years, supplemented by interviews, observations, and grounded-theory analysis.The research included 27 on-site interviews and more than 30 hours of observation.
- Clinical problem: Sepsis is a major inpatient health concern, but professional consensus on its diagnosis remains limited.Different definitions identify different patient groups, and experts have struggled to produce one all-encompassing definition.
- Clinical problem: Hospitals face close regulatory monitoring of sepsis treatment through CMS reporting and SEP-1 compliance requirements.Although SEP-1 guidelines have improved outcomes in some settings, treatment adherence and patient outcomes remain difficult for many hospital systems.
3.2 PROJECT BACKGROUND
The project began with a locally defined need to improve sepsis detection and treatment, informed by an earlier intervention that produced alarm fatigue without improving outcomes. The team organized its work around local data, workflow design, model development, validation, and clinical evaluation.
- Project initiation: In 2016, front-line physicians proposed a project to improve local sepsis detection and treatment, and health-system leaders approved implementation support.The proposal was evaluated primarily by the importance of the problem and opportunity for improvement, rather than technological novelty.
- Project initiation: A prior sepsis intervention caused significant alarm fatigue among front-line staff and failed to improve patient outcomes.The new project explicitly incorporated lessons from that implementation.
- Problem definition: The team defined the problem as extending beyond detection because clinicians viewed treatment follow-up failures as a major source of poor performance.Sepsis treatment compliance requires completing dozens of steps during the six hours after detection.
- Project phases: The project spent its first year assembling stakeholders and characterizing the problem, its second year developing and validating Sepsis Watch, and its final six months preparing for launch.The development work covered the model, application, data pipeline, and integration platform collectively called Sepsis Watch.
- Evaluation: Model development and clinical evaluation received institutional review board approval, followed TRIPOD reporting guidance, and used a preregistered clinical trial.The trial was registered as ClinicalTrials.gov NCT03655626.
3.3 DESIGN AND DEVELOPMENT
Sepsis Watch was designed around locally curated EHR data, rapidly updating predictions, and a workflow intended to avoid the alarm fatigue associated with an earlier clinical decision-support system. Model explainability was not prioritized because sepsis definitions and mechanisms are contested.
- Data: The team curated data from the local hospital because clinicians expected local data to outperform external models and healthcare models often generalize poorly.The dataset was assembled from a local quaternary academic hospital with more than 1,000 beds.
- Model: The model explored broad medical-history, vital-sign, and laboratory features to predict a rapidly evolving condition across multiple data modalities.It was designed to generate accurate predictions from the beginning of a hospital encounter and update as new information arrived.
- Model: Model explainability was not prioritized because sepsis treatment was standardized despite disagreement about diagnosis, disease definitions, and underlying causes.The paper notes that human experts and major organizations use distinct sepsis definitions.
- Model: The MGP-RNN model generated hourly risk scores for every adult patient to detect sepsis.It coupled a recurrent neural network with multi-task Gaussian processes, without a deliberate effort to explain its output to end users.
- Workflow: The workflow assigned primary monitoring to rapid response team nurses and was designed in response to a prior system whose alerts produced substantial cancellation and alarm fatigue.The earlier system generated more than 100 daily EHR pop-ups for some high-risk patients, with 86% of notifications canceled.
- Workflow: Patients with risk scores above 60% were displayed in red, while patients meeting sepsis criteria were displayed in black and prioritized on the Triage page.At the 60% threshold, the model’s positive predictive value was 20%.
3.4 IMPLEMENTATION
Implementation treated Sepsis Watch as a socio-technical system whose trustworthiness and accountability depended on stakeholder relationships, professional discretion, communication, technical integration, monitoring, and governance. Clinicians helped shape the system while retaining responsibility for clinical judgment.
- Trust and accountability: The team pursued trust and accountability through strategies other than model interpretability, engaging stakeholders across design, development, integration, and clinical care.The paper focuses accountability on those who designed and managed implementation rather than solely on individual clinicians.
- Stakeholder relationships: The implementation team embedded clinical, technical, and implementation expertise within existing professional networks and local organizational relationships.Members’ local experience and credibility were described as crucial to the project’s success.
- Communication: Different stakeholders received different forms of evidence, including model performance measures, patient stories, and direct experience with the tool.This communication strategy recognized that stakeholders valued different evidence and communication modes.
- Communication: A “Model Facts” sheet was designed with clinical professionals to communicate relevant model information transparently to front-line staff.Aggregate measures and individual patient cases were also used to show model performance and early detection relative to clinical diagnosis.
- Accountability: The team preregistered a clinical trial, set improvement targets, discussed turning off the system if targets were missed, and established external safety oversight.An external monitoring board included clinical researchers and the chief nursing officer.
- Enterprise integration: Sepsis Watch required sustained collaboration with the EHR vendor and information technology team to become an enterprise solution.The teams held weekly calls for six months to identify a technical integration path.
- Monitoring: The innovation team completed enterprise security and load testing and continued monitoring the model and application after the usual pilot-transition period.A 55-inch monitor displayed the live application and aggregate model risk scores for ongoing oversight.
- Professional discretion: RRT nurses and ED physicians were treated as expert professionals whose expertise shaped Sepsis Watch’s impact in practice.The tool was intended to ensure that clinically relevant follow-up tasks did not slip through the cracks.
3.5 INTEGRATION: UNEXPECTED INTERACTIONS THAT FACILITATED USE
Sepsis Watch was integrated as a socio-technical system in which clinical work practices shaped how information was interpreted and used. The case also questions whether model interpretability alone can secure trust and accountability in clinical practice.
- Socio-technical integration: Sepsis Watch was treated as a socio-technical system whose use could not be separated from the people, institutions, contexts, and power hierarchies shaping it.The project therefore centered on understanding and improving interactions and work practices around the tool.
- Unexpected interactions: Clinical professionals re-calibrated their reliance on Sepsis Watch over time, while RRT nurses developed practices that contextualized its information within patient care.RRT nurses reviewed patients’ EHR charts before contacting physicians, an ad hoc step that supported effective integration.
- Professional discretion: RRT nurses and ED physicians retained professional discretion to diagnose sepsis, and respecting professional boundaries enabled new expertise to emerge.The nurses’ practices enhanced use of the machine learning-driven tool rather than replacing clinical judgment.
- Beyond interpretability: The paper argues that clinical transparency and accountability should be pursued through additional means rather than relying exclusively on model interpretability.The authors identify problem definition, stakeholder relationships, professional discretion, and feedback loops as key practices.
4.1 PROBLEM DEFINITION: FROM END USER TO DATA SCIENTIST
Sepsis Watch’s problem definition was grounded in a specific hospital’s clinical needs and existing care processes. Clinicians helped shape the model’s population, inputs, evaluation, and intended use.
- Contextual problem definition: Sepsis Watch addressed a specific problem in a specific hospital, defined with frontline clinicians rather than treated as value-neutral or self-evident.The project aligned clinical problems with innovation priorities and drew on team members’ prior local implementation experience.
- Clinical involvement: Clinicians participated in selecting the training cohort, model inputs, and evaluation design using local patient data.
- Institutional processes: Existing care and treatment processes needed to inform model development because ignoring them could produce inaccurate or harmful interventions.Sepsis Watch incorporated relevant aspects of care delivery as model features from the start.
- End-user guidance: A clinician-developed Model Fact sheet articulated the model’s indicated and contraindicated uses, including its strengths and weaknesses.
4.2 RELATIONSHIP BUILDING: ENGAGE EARLY AND OFTEN
The project treated trust as rooted in relationships and engaged relevant stakeholders throughout development and implementation. Integration also required substantial educational resources and dedicated institutional support.
- Engage early and often: Trust was treated as rooted in relationships, so each development phase included substantive engagement with relevant stakeholders.Lead clinicians regularly reviewed and validated the model with the development team during early model-building and tuning.
- Allocate human resources: Integrating the tool into clinical practice required resource-intensive professional education and a formally devoted full-time role.
4.3 RESPECT PROFESSIONAL DISCRETION: AUGMENT, DON’T REPLACE
Sepsis Watch was designed to augment rather than replace professional diagnosis, while implementation created new expertise around digital clinical data. The study notes that machine learning systems require complementary approaches to improve human decision-making.
- Augment, don’t replace: Sepsis Watch was designed as a software diagnostic aide and “algorithm in the loop,” retaining human expertise for final diagnostic decisions.
- Governance and learning: Monthly governance reviews of sepsis diagnosis and treatment failures supported learning among emergency department physicians.These case discussions remained central to distilling and disseminating lessons.
- Open questions: The study leaves open questions about how reliance on technology affects professional skill, training, and certification.
- Elevate new expertise: RRT nurses developed expertise in remotely evaluating sepsis by contextualizing and synthesizing digital patient data.Their expanded expertise bridged digital data gathering with clinical interpretation.
4.4 STAKEHOLDER FEEDBACK LOOPS: BETWEEN DESIGNERS AND USERS
Sepsis Watch treated stakeholder communication as an ongoing, multidirectional process that both informed users and surfaced issues during implementation. The project also recognized patients as stakeholders whose interests require dedicated accountability and trust-building.
- Between Designers and Users: Meetings and information sessions served both to communicate with stakeholders and to receive issue reports and discussion.This feedback process supported changes during the pilot, including a revised user interface launched after two months.
- Between Designers and Users: A second version of the user interface was launched two months into the pilot based on stakeholder feedback.
- Between Designers and Users: Developers maintained ongoing monitoring after deployment by taking responsibility for the tool’s behavior in use.A monitor displaying model behavior kept Sepsis Watch visible in the office after active development ended.
- Between Designers and Patient Proxies: Patients were treated as stakeholders whose interests needed representation even though the paper primarily focused on clinicians.The authors note that patient-facing applications require additional ways to cultivate trust and accountability with patients.
5 CONCLUSION
The conclusion frames Sepsis Watch as a socio-technical implementation that requires institutional, professional, and stakeholder practices beyond model interpretability. It emphasizes contextual problem definition, relationships, professional discretion, and ongoing feedback as central to accountable clinical decision support.
- Socio-technical framing: Machine learning implementation must be understood as a socio-technical system combining technical and social dimensions within a specific institutional context.
- Project goals: Sepsis Watch aimed to improve patient care while enhancing professional expertise and clinical decision-making.
- Trust and accountability: Because sepsis lacks a consensus clinical definition, the team used mechanisms beyond model interpretability to build trust and accountability.
- Clinical decision-making: Clinical decisions require explanations and justifications even when model outputs themselves do not need to be explainable.The authors report significant labor in mapping model outputs to clinical decisions.
- Design practices: The paper identifies four practices: define problems in context, build stakeholder relationships, respect professional discretion, and create ongoing feedback loops.
- Institutional accountability: Scaling machine learning in healthcare requires substantial resources to integrate technologies and build human capacity and expertise.