Source-linked AI summary
Designing AI for Trust and Collaboration in Time-Constrained Medical Decisions: A Sociotechnical Lens
Maia Jacobs, Jeffrey He, Melanie F. Pradier, Barbara Lam, Andrew C. Ahn, Thomas H. McCoy, Roy H. Perlis, Finale Doshi-Velez, Krzysztof Z. Gajos
TL;DR
Selecting antidepressants for MDD is difficult because clinical guidance offers limited support amid variable patient responses and constrained primary-care visits. The paper uses iterative co-design with primary care clinicians to examine how DSTs can fit healthcare’s sociotechnical system. It finds that DSTs should support patient-provider collaboration, integrate with clinical processes, and provide on-demand comparisons between AI recommendations and standards of care.
Problem
Antidepressant selection for MDD lacks sufficient guidance despite variable patient needs, while primary-care settings face brief appointments and uneven clinician training.
Method
The authors used iterative co-design with primary care clinicians, including interviews, focus groups, and prototype-based discussions of DST expectations.
Results
Clinicians wanted DSTs that engage patients, connect predictions to healthcare processes, reduce repeated trust judgments, and compare outputs with standards of care.
Takeaways & Limitations
Clinical DSTs should be designed as multi-user systems supporting patient-provider-AI collaboration with on-demand explanations suited to time-constrained care.
Takeaways & Limitations
The study focused solely on clinicians and identifies engagement with patients and other stakeholders as a next step.
Abstract
from arXiv · showhide
Major depressive disorder is a debilitating disease affecting 264 million people worldwide. While many antidepressant medications are available, few clinical guidelines support choosing among them. Decision support tools (DSTs) embodying machine learning models may help improve the treatment selection process, but often fail in clinical practice due to poor system integration. We use an iterative, co-design process to investigate clinicians' perceptions of using DSTs in antidepressant treatment decisions. We identify ways in which DSTs need to engage with the healthcare sociotechnical system, including clinical processes, patient preferences, resource constraints, and domain knowledge. Our results suggest that clinical DSTs should be designed as multi-user systems that support patient-provider collaboration and offer on-demand explanations that address discrepancies between predictions and current standards of care. Through this work, we demonstrate how current trends in explainable AI may be inappropriate for clinical environments and consider paths towards designing these tools for real-world medical systems.
1 INTRODUCTION
This paper examines how machine-learning decision support tools could assist antidepressant selection for major depressive disorder while fitting real clinical practice. Using iterative design and clinician feedback, it argues that these tools must account for healthcare’s sociotechnical context and support collaborative, time-constrained decisions.
- Motivation: Antidepressant selection for MDD is difficult because guidelines offer limited help, patient responses vary, and ineffective trials are common.An estimated one-third of patients do not reach remission after four antidepressant trials.
- Motivation: Existing ML tools for antidepressant selection are rarely integrated into practice because they have low user acceptance and do not adequately address user expectations.The paper frames these failures through a sociotechnical approach focused on the social, technical, and organizational conditions of clinical use.
- Findings: Iterative design and two qualitative studies with primary care providers identified requirements for DSTs in short clinical encounters.Clinicians wanted tools that engage patients, connect with healthcare processes, avoid repeated trust judgments, and compare predictions with standards of care.
- Design implications: The paper recommends multi-user DSTs that facilitate patient-provider-AI collaboration and provide on-demand explanations contrasting recommendations with current standards of care.This approach challenges explanation designs that require users to determine trust for every prediction.
- Contribution: The authors use an iterative design process to create an MDD DST combining patient-level prognostic predictions with treatment-selection support.The prototype was informed by primary care providers’ feedback and presented as part of the paper’s contribution.
2 RELATED WORK
Prior work shows that clinical decision support tools often fail not because of model performance alone, but because they do not fit healthcare’s complex sociotechnical systems. For MDD, difficult treatment selection and limited clinical time motivate tools designed around clinician expectations, patient needs, and system integration.
- Sociotechnical integration: Research has increasingly used co-design and HCI methods to incorporate end-user voices and examine how people understand and interact with AI tools.This work extends those efforts toward complex clinical systems.
- Decision support tools: Existing DSTs can provide treatment recommendations, prognosis predictions, and diagnoses, but implementing MDD models requires understanding clinician expectations and system integration.The paper therefore seeks to clarify PCP support needs for future primary-care DSTs.
- Sociotechnical integration: Many deployed healthcare DSTs fail because they overlook workflow integration, context awareness, and clinicians’ expectations rather than simply performing poorly.The related work positions sociotechnical fit as a central implementation concern.
- MDD treatment context: MDD treatment selection is difficult because primary care provides much mental healthcare, appointments are brief, and PCP training varies considerably.U.S. primary care appointments average 20 minutes and can be only a few minutes internationally.
- MDD treatment context: Guidelines organize 25 antidepressants into treatment lines, but symptom and tolerability differences make identifying an effective medication for an individual patient difficult.Later-line options may involve more severe side effects or drug interactions.
3.1 Methods
The study used clinician-centered co-design to explore current antidepressant decision processes and expectations for future DSTs. Researchers combined qualitative sessions, low-fidelity prototypes, a realistic patient scenario, and grounded-theory analysis.
- Participants and recruitment: Researchers recruited primary care clinicians who prescribe antidepressants from a large academic medical center.Participants included physicians, nurse practitioners, and residents.
- Study design: Semi-structured interviews and small focus groups lasted 30 minutes, dividing discussion between current decision-making and future-state ideas.Sessions were conducted remotely using Zoom because the study occurred in March 2020.
- Study design: Low-fidelity prototypes demonstrated potential DST features derived from electronic medical record-based ML research for clinician co-design activities.Participants discussed which features were helpful, unhelpful, or worth changing.
- Prototype: The prototype included stability and dropout predictions, feature-importance explanations, personalized treatment recommendations, and drug-interaction rules.Stability represented continued medication use for at least 3 months, while dropout represented early discontinuation within the same health system.
- Prototype: Treatment recommendations used drug-specific interaction rules curated by two psychopharmacologists with a mean of 12 years of U.S. clinical practice.Example rules linked patient characteristics such as anxiety or poor concentration to medication preferences.
- Prototype: A fabricated patient scenario vetted by psychopharmacology experts supported discussion of realistic antidepressant treatment selection.The scenario included essential patient information such as age, gender, comorbidities, and a prior ineffective SSRI trial.
- Analysis: Researchers audio-recorded and transcribed sessions, then used grounded-theory coding, team review, and recoding to identify themes.The analysis compared current decision processes with prototype feedback.
3.2 Findings: Clinical Expectations for ML Decision Support
Clinicians expected ML decision support tools to fit time-constrained workflows, support collaborative treatment decisions, and connect predictions to actionable clinical processes. They also wanted trustworthy systems whose validation information is available without disrupting patient care.
- Include patient preferences.: Clinicians viewed MDD treatment decisions as collaborative and wanted interactive tools that incorporate patient preferences into recommendations.They wanted clinicians and patients to modify input variables and discuss medication effects together.
- Recommend appropriate clinical processes.: High dropout-risk predictions could support earlier follow-up, slower titration, behavioral therapy, and involvement of additional care-team members.These responses used resources and procedures already established within clinics.
- Recommend appropriate clinical processes.: Stability scores were viewed less favorably because they did not reliably indicate interventions and could imply that follow-up should be lengthened inappropriately.Clinicians worried that seemingly stable patients might still suffer from medication side effects without timely contact.
- Recommend appropriate clinical processes.: Clinicians wanted DST predictions paired with explicit next steps that connect model outputs to existing healthcare processes and resources.Displaying predictions alone was insufficient because providers differed in their ability to identify appropriate follow-up actions.
- Understand healthcare system resource constraints.: Limited appointment time made prediction-level trust assessments impractical, while clinicians preferred accessible validation evidence over forced feature-importance explanations.Participants expected trust to be established through broader validation, professional adoption, and colleagues’ experience rather than at every decision point.
4 PROTOTYPE REDESIGN
The redesigned prototype operationalized clinicians’ expectations by incorporating patient preferences, connecting predictions to clinical processes, supporting treatment comparison, and fitting time-constrained care.
- Patient preferences: Treatment recommendations became interactive, allowing clinicians to edit which aspects of a patient’s medical history drive the recommendation.This redesign addressed clinicians’ view of treatment selection as mutual and collaborative.
- Clinical processes: The prototype made dropout predictions more prominent and added a patient-distribution graph plus suggested next steps for high-risk cases.Suggested actions included slower medication titration, earlier follow-up, and additional behavioral therapy.
- Treatment comparison: A matrix interface displayed antidepressants side by side so clinicians could compare many treatment options in a glanceable format.The design supported consideration of all possible options rather than requiring clinicians to search for one treatment.
- Resource constraints: Feature-importance explanations were replaced with a link describing model validation procedures, including future clinical trial protocols, results, and publications.The validation page was designed to make evidence available on demand during time-constrained encounters.
5 USER STUDY 2: PROTOTYPE FEEDBACK
The second user study evaluated a redesigned MDD decision-support prototype with PCPs through remote, 30-minute interactive sessions. Participants generally valued workflow integration, personalization, and validation information, but unexpected predictions often caused confusion or abandonment.
- Study design: The study used remote 30-minute sessions in which PCPs freely interacted with the redesigned prototype and discussed treatment decisions.Participants were recruited from the same academic medical center and included both new and returning clinicians.
- Prototype redesign: The prototype combined patient information, dropout-risk information with validation links, and interactive personalized treatment recommendations.These features were intended to make model outputs more actionable and interpretable within treatment selection.
- Clinical-process integration: Participants viewed recommended steps associated with dropout-risk predictions as actionable and aligned with their usual clinical processes.Clinicians described using elevated dropout risk to identify patients needing additional discussion or closer follow-up.
- Healthcare-system integration: Clinicians saw opportunities to connect recommendations with prescription systems and generate patient-friendly educational handouts.They also suggested adding conditions such as pregnancy and suicidality because these influence treatment decisions.
- Patient preferences: Editing patient conditions helped clinicians incorporate patient concerns and observe how different conditions changed medication recommendations.Participants specifically described toggling conditions such as poor concentration to reassess the medication regimen.
- Trust: Links to model validation prompted interest in the evidence clinicians would need to establish trust in the tool.Participants requested outcome-oriented validation, including treatment response, adherence, and adverse-effect outcomes.
- Contrasting outputs: When recommendations diverged from clinical knowledge or guidelines, clinicians became confused and often abandoned them.Surprising dropout predictions prompted requests for contributing factors, while unexpected medication recommendations led clinicians to reconsider the patient’s case.
6 DISCUSSION
The discussion frames MDD decision-support tools as components of a broader sociotechnical healthcare system rather than isolated prediction interfaces. It argues for collaborative, workflow-aware designs with time-sensitive explanations and careful handling of predictions that conflict with clinical knowledge.
- Sociotechnical design: Designing MDD decision-support tools requires attention to patient preferences, clinical processes, system constraints, and domain knowledge.These aspects emerged from clinicians’ expectations for AI support in treatment decisions.
- Scope and future work: The study’s implications are most directly situated in MDD and healthcare settings where patients participate in decisions and providers have short, infrequent appointments.The authors suggest transfer to similar primary-care decisions while identifying specialists, nonparticipating patients, and direct patient perspectives as future areas of study.
- Multi-user collaboration: Clinicians’ feedback challenged single-user AI systems and supported tools that foster collaboration among patients, providers, and AI.The prototype’s interactive tailoring of recommendations was described as an initial step, with future co-design involving patients directly.
- Clinical processes: Decision-support outputs should connect model predictions to actionable next steps that may involve multiple healthcare providers.Treatment selection is presented as affecting broader aspects of care rather than functioning as a siloed task.
- Resource constraints: Because clinicians preferred determining trust once rather than at every decision point, the authors recommend displaying evidence-based validation methods such as randomized controlled trial results.This shifts explanation design toward establishing confidence in the tool rather than repeatedly explaining each output.
- Resource constraints: Time constraints make explanations for every prediction potentially unusable in primary care, increasing the importance of upfront decisions about when to show the tool.The paper notes that short appointments impose time and mental burdens on clinicians reviewing detailed explanations.
- Contrasting information: When outputs conflict with clinical knowledge or guidelines, clinicians often abandon recommendations, while more detailed causal information is favored for surprising predictions.The authors propose on-demand explanations that contrast AI recommendations with existing clinical guidelines and support actionable next steps.
7 LIMITATIONS
The study treats sociotechnical integration as a central implementation challenge but acknowledges important scope and visualization limitations. It examines clinicians’ perspectives as an initial step rather than a complete account of healthcare stakeholders or ML uncertainty.
- Scope: The study focused solely on clinicians’ perspectives, leaving patients, nurses, pharmacists, and therapists for future stakeholder studies.The authors describe clinician-only involvement as an important first step.
- Visualization: The authors do not claim that the proposed visualization is optimal for representing ML prediction distributions and model uncertainty.They state that substantial visualization work remains.
- Boundary: The study addresses one real-world implementation challenge—considering the broader healthcare system—within healthcare’s practical, ethical, and legal complexity.The paper explicitly frames this as one challenge among numerous issues raised by ML in healthcare.
8 CONCLUSION
The paper identifies four healthcare sociotechnical factors that should guide real-world ML decision support design: patient preferences, multi-provider clinical processes, system constraints, and domain knowledge. It presents these factors as an initial step toward systems that support collaborative, time-critical medical work while recognizing that the framework is not exhaustive.
- Conclusion: Four sociotechnical factors shape DST design: patients’ preferences, multi-provider clinical processes, healthcare-system constraints, and existing domain knowledge.These factors were identified through co-design studies with primary care providers.
- Conclusion: Centering these factors may help DSTs support healthcare’s collaborative nature, identify adverse events from ML predictions, work in time-critical environments, and recognize conflicting information.The paper frames these as potential capabilities rather than demonstrated outcomes.
- Conclusion: The proposed sociotechnical factors are an initial contribution to a broader research agenda on accounting for the complexity of medical work.The authors explicitly state that the factors do not represent the full set of relevant considerations.