PaperlayerView exact revision on arXiv

Source-linked AI summary

"It Feels Weird to Just Rely on AI": How Experienced Healthcare Professionals Work With Decision Support Systems in Clinical Practice

Simon W. S. Fischer, Hanna Schraffenberger, Miranda L. van Hooff, Casper Reijnen, Serge Thill, Pim Haselager

arXiv:2610.40224v1cs.HC

TL;DR

Prior human-AI research often examines simplified tasks, lay participants, or prototype interactions, leaving clinicians’ decision-making with DSS in practice insufficiently understood. The authors combine surveys and interviews with clinicians in two medical cohorts. They report that DSS use changes with experience, predictions matter more in complex cases, and workflow and other socio-technical factors shape decision-making.

  • Problem

    Human-AI studies often use simplified tasks, lay participants, and prototype interactions, while clinicians’ decision-making processes and reliance on DSS in daily practice remain poorly understood.

    Source: §1 · p. 2

  • Method

    The authors triangulated an online survey and interviews with clinicians from two medical cohorts using DSS in clinical practice.

    Source: §1 · p. 2, §4 · p. 7

  • Results

    Clinicians’ DSS use changes with experience; predictions become more important in complex cases, and workflow integration affects decision-making.

    Source: §5 · p. 10, §8 · p. 23

  • Takeaways & Limitations

    DSS design and research should consider interacting human, technical, contextual, and organisational factors and support decision-making individually and collectively.

    Source: §8 · p. 23

    The 12-person qualitative sample provides context-specific insights and does not aim to support general conclusions about clinical human-AI decision-making.

    Source: §7 · p. 22

Abstract

from arXiv · show

Decision support systems (DSS) assist with clinical decision-making. Little is known, however, about the decision-making process with DSS in clinical practice and the extent to which clinicians consider or rely on the predictions. Studies on human-AI decision-making usually focus on laypeople, simplified decision tasks, or one-shot interactions with prototypes that are not integrated into existing workflows. To gain insights from real-world practice, we conducted an online survey (n=12) and semi-structured interviews (n=10) with clinicians from two different medical cohorts each working with a DSS. Findings show, amongst other things, that the interaction with DSS changes as experience increases, the predictions become more important in complex cases, that decisions are discussed with colleagues, and that workflow integration affects decision-making. Our findings on human, technical, contextual, and organisational factors contribute to a socio-technical perspective on DSS reliance behaviour. We conclude by discussing the implications of our findings for the design of DSS.

1 Introduction

Clinicians need to evaluate DSS advice without either accepting incorrect advice or rejecting correct advice, but prior research offers limited insight into how this unfolds in clinical practice. The paper investigates clinicians’ decision-making with DSS across two medical contexts and identifies factors shaping their reliance.

1 Introduction

  • Appropriate reliance means neither accepting incorrect DSS advice nor rejecting correct advice, and may support patient well-being.The paper notes that human oversight requirements include appropriate reliance.Source: §1 · p. 2
  • Prior human-AI studies often use simplified tasks, lay participants, or prototype interactions, leaving clinicians’ decision-making processes in daily practice poorly understood.Outcome-focused measures can also miss how clinicians reflect on advice and weigh it against other information in open-ended decisions.Source: §1 · p. 2
  • The study asks how clinicians use DSS in practice, including how they handle differing judgments and what information they need, across two Dutch clinical contexts.The cohorts use DSS for chronic low back pain treatment decisions and endometrial cancer lymph-node metastasis assessment.Source: §1 · p. 2
  • Survey and interview findings identify human, technical, contextual, and organisational factors shaping clinical DSS decision-making and inform design pathways for appropriate reliance.The authors frame this as a socio-technical perspective on reliance behaviour.Source: §1 · p. 2

2 Background and Related Work

Prior reliance research offers useful design insights, but its tasks, participants, and settings may not represent complex clinical decisions. This paper foregrounds the decision-making process and the interplay between clinicians, DSS, and their work context.

2 Background and Related Work

  • The background situates the paper alongside work on appropriate reliance and interaction methods that address decision processes rather than only recommendations.This contrasts with a recommendation-centric approach.Source: §2 · p. 3

2.1 Appropriate Reliance in Human-AI Decision-Making

  • Reliance research often uses simplified tasks, lay participants, outcome metrics, or prototype interactions, limiting its fit to uncertain, complex decisions and established workflows.The authors argue that process-sensitive, real-world evidence is needed to complement these studies.Source: §2.1 · p. 3
  • The paper treats appropriate reliance as evaluating and reflecting on DSS advice, within an interaction among human, technical, and environmental factors.This extends outcome-only definitions based on accepting or rejecting correct or incorrect advice.Source: §2.1 · pp. 3–4

2.2 Supporting the Decision-Making Process

  • Decision-process support can help clinicians consider alternatives and hypotheses, while surveys of information needs can inform secondary assistance beyond recommendations.Examples include counterfactual reasoning, what-if analysis, reflective questions, and feedback.Source: §2.2 · p. 4

3 Clinical Use Cases

The paper examines two clinical DSS use cases and compares their prediction outputs, data, entry processes, and explanations. The systems support decisions in contexts where clinicians must weigh treatment options or risks.

3 Clinical Use Cases

  • The two use cases concern DSS-assisted treatment decisions for chronic low back pain and lymph-node metastasis in endometrial cancer.The first is a spine-department use case; the second involves clinicians in gynaecology departments.Source: §3 · p. 4

3.1 Cohort 1: Treatment Prognosis for Chronic Low Back Pain

  • For chronic low back pain, the DSS presents six prognoses: success and failure predictions for each of three treatment options.Because treatment outcomes are uncertain and there is no clear ground truth, clinicians need to assess predictions critically.Source: §3.1 · p. 5, Figure 1 · p. 5

3.2 Cohort 2: Risk Estimation of Lymph Node Metastasis in Endometrial Cancer

  • For endometrial cancer, clinicians use a lymph-node metastasis risk estimate to inform whether lymph nodes should also be removed.The DSS combines tabular clinical data and displays feature contributions explaining its prediction.Source: §3.2 · pp. 5–6, Figure 2 · p. 6

3.3 Comparison of Both Decision Support Systems

  • The systems differ in prediction and recommendation, data source, data entry, and the explanations they provide.Cohort 1 displays six prognoses from subjective questionnaire data without explanations; cohort 2 displays a risk estimate from objective data entered manually, with feature-contribution explanations.Source: §3.3 · p. 6

4 Methods

The study combines an online survey and semi-structured interviews with clinicians working with DSS in two medical cohorts. Descriptive survey data and iterative thematic analysis are used to examine their decision-making processes.

4 Methods

  • The study combines an online survey and individual semi-structured interviews, analysing cohort 1 interview data alongside new survey and cohort 2 data.The cohort 1 interviews originated in a larger study that also evaluated a prototype intervention.Source: §4 · p. 7

4.1 Participants

  • Twelve clinicians participated: eight of 11 spine specialists in cohort 1 and four clinicians from about 50 DSS users in cohort 2.Participants were experienced, and most completed both the survey and interview.Source: §4.1 · p. 7

4.2 Online Survey

  • The survey assessed DSS attitudes, confidence, interaction behaviour, information considered and needed, and open-ended reflections.It used selected USE-scale items, Likert-scale questions, and items adapted from a human-AI decision-making taxonomy.Source: §4.2 · pp. 7–9

4.3 Expert Interviews

  • Semi-structured interviews explored DSS use, perceived usefulness, differing judgments, explanations, and decision support needs, with later interviews adjusted iteratively.Interview formats and duration varied across cohorts and participants.Source: §4.3 · p. 9

4.4 Data Analysis

  • The researchers iteratively coded interviews, connected qualitative themes with survey findings, and used descriptive statistics rather than statistical tests.The sample size was considered too small for meaningful statistical insights or cohort comparisons.Source: §4.4 · pp. 9–10

5 Results

The results cover how clinician and DSS familiarity, experience, and clinical context shape reliance behaviour. Clinicians’ use of the DSS changes with experience, though the direction can differ.

5 Results

  • The results identify eleven themes spanning familiarity, experience, decision-making, colleagues, and workflow integration.Source: §5 · p. 10

5.1 Both Familiarity With the Task and With the DSS Shape Reliance Behaviour

  • Participants were experienced with both treatment selection and DSS use, understood the system through experience or data entry, and did not view overreliance as a risk for themselves.Source: §5.1 · p. 10

5.2 DSS Use Changes With Experience

  • With experience, some clinicians used the DSS less except in complex cases, while one reported greater reliance after learning the model’s workings and pitfalls.Overall, experience changed use and reliance behaviour, but not in one direction.Source: §5.2 · p. 12

5.3 The Same DSS is Used for Different Purposes

The same DSS serves different clinicians and decision stages, with distinct information needs. Participants found the predictions useful partly because numerical risk made decisions more concrete, though perceived security could invite overreliance.

5.3 The Same DSS is Used for Different Purposes

  • Clinicians used the DSS for different purposes and at different stages, including triage, staging, treatment selection, and patient consultation.Their specialties and roles shaped their information needs.Source: §5.3 · p. 12

5.4 The DSS Predictions Make Decision-Making More Concrete

  • Both cohorts rated the DSS’s usefulness at a mean of 5 out of 7, and participants said its numerical predictions made risk more concrete and easier to explain to patients.Source: §5.4 · pp. 13–14
  • All participants considered the predictions, usually after forming an initial judgement; predictions generally confirmed their assessment and rarely changed their decision.Participants described the numbers as reassuring.Source: §5.4 · p. 14
  • One participant warned that the sense of security provided by the DSS could lead to overreliance.Source: §5.4 · p. 14

5.5 A Critical Attitude Encourages Clinicians to Follow Their Own Judgement

Clinicians described a critical approach to DSS predictions, often following their own judgement when it differed. Predictions could prompt reflection, while clinicians also wanted explanations and ways to explore alternative outcomes.

5.5 A Critical Attitude Encourages Clinicians to Follow Their Own Judgement

  • When predictions contradicted their assessment, most participants said they followed their own judgement and critically examined why the predictions differed.Participants also described evaluating data quality and decision consequences as part of careful, context-shaped decision-making.Source: §5.5 · p. 15

5.6 The DSS Predictions Help Clinicians Reflect on Their Judgement

  • Predictions helped clinicians compare and reconsider their initial judgements, sometimes affecting decisions and requiring them to justify their reasoning.Source: §5.6 · p. 15
  • Cohort 1 participants wanted explanations such as feature contributions, while cohort 2 participants, whose DSS already provided them, did not express that need.Participants in cohort 2 said the explanations helped their decision-making.Source: §5.6 · p. 15
  • Participants in both cohorts wanted to examine alternative outcomes and how changing input data affected predictions.They described this as a way to further reflect on assessments and find appropriate treatments.Source: §5.6 · p. 17

5.7 The DSS Predictions Become More Important When Clinicians Are in Doubt

Clinicians give DSS predictions more weight when cases are uncertain or complex, but treat them as one input among several. Experience, patient context, and colleague discussions shape how predictions inform decisions.

5.7 The DSS Predictions Become More Important When Clinicians Are in Doubt

  • In complex or doubtful cases, clinicians give DSS predictions more weight; in clear cases, they may not consult them.Predictions can act as a second opinion, surface overlooked factors, and increase confidence.Source: §5.7 · p. 17

5.8 Inexperienced Clinicians May Be More Influenced by DSS Predictions

  • Less experienced clinicians may rely more on DSS scores when uncertain, despite training on how to use the system.Participants linked this susceptibility to limited experience and the security the DSS can convey; most survey respondents supported residents practicing with the DSS.Source: §5.8 · p. 17

5.9 The DSS Predictions Are Just One Aspect of a Multi-Faceted Decision-Making Process

  • Clinicians weigh DSS predictions against patient history, diagnostic tests, examination findings, and patient preferences.Surgeons noted the DSS does not incorporate MRI data, leaving them to combine separate sources; participants also described a trade-off between completeness and usability.Source: §5.9 · p. 18

5.10 Decisions are Discussed with Colleagues

  • Colleague discussions provide second opinions on difficult cases, support shared learning, and surface the need for agreed model thresholds.Cohort 1 also discusses surgical cases a week before the procedure, sometimes leading surgeons to cancel surgery.Source: §5.10 · pp. 18–19

5.11 Workflow Integration of the DSS Affects Decision-Making

Workflow integration affects access to DSS predictions and the effort required to use them. In one cohort, system changes disrupted access; in the other, manual entry meant clinicians formed an initial judgement before seeing predictions.

5.11 Workflow Integration of the DSS Affects Decision-Making

  • Changes to the electronic patient-file system disrupted DSS access, requiring an additional app and leaving only a few clinicians with direct access.Those clinicians had to record scores in patient notes, adding work; participants said unreliable operation could reduce DSS use.Source: §5.11 · pp. 19–20
  • In cohort 2, clinicians manually enter patient data into a DSS that is not integrated into the electronic patient file.Participants rated time-saving and productivity among the lowest USE Scale items.Source: §5.11 · p. 20
  • Manual data entry and engagement with patient data mean cohort 2 clinicians have formed an initial judgement before seeing DSS predictions.Cohort 1 clinicians also reported forming an initial judgement, although predictions were available when they opened the patient file.Source: §5.11 · p. 20

6 Discussion

The discussion frames clinical DSS reliance as shaped by interacting human, technical, contextual, and organisational factors. It highlights how expertise, uncertainty, patient context, teamwork, and workflow affect clinicians’ use of predictions.

6 Discussion

  • The discussion identifies four interacting dimensions shaping clinical human-AI decision-making: human, technical, contextual, and organisational factors.The authors add contextual factors to the three aspects identified by Eckhardt et al..Source: §6 · p. 20

6.1 Human Factors

  • Experience and familiarity with the task and DSS can support appropriate reliance, while clinicians’ disagreement with a prediction can prompt reflection.The authors argue that long-term DSS use deserves more attention because familiarity changes over clinicians’ careers.Source: §6.1 · pp. 20–21

6.2 Technical Factors

  • DSS predictions usually confirm clinicians’ judgements and rarely change decisions, but increased confidence in complex cases may raise overreliance risk.The value of explanations remains unclear; the authors note that clinicians use the same DSS for different purposes and at different stages of care.Source: §6.2 · p. 21

6.3 Contextual Factors

  • Clinical decisions combine DSS predictions with other diagnostics and patient preferences, and the two use cases suggest reflective decision-making occurs in practice.The authors caution that overreliance remains an issue, particularly among less-trained people, and call for evaluation in realistic use cases.Source: §6.3 · pp. 21–22

6.4 Organisational Factors

  • Group meetings, collegial support, and DSS training can help clinicians learn from one another and develop decision-making skills.Because decisions span multiple stages and clinicians, the authors argue DSS interactions should support team discussion and shared reasoning.Source: §6.4 · p. 22
  • Workflow infrastructure can disrupt DSS use, while manual entry may require more effort but support cognitive engagement before clinicians see predictions.The authors note that the impact of DSS on patient outcomes remains uncertain without evaluations in existing workflows.Source: §6.4 · p. 22

7 Limitations and Future Work

The study offers context-specific insights rather than general conclusions, while identifying limitations in self-reported reliance and simulated decision tasks. Future work could broaden the evidence with reliable measures and participants with different expertise.

7 Limitations and Future Work

  • With 12 participants, the qualitative findings are context-specific and are not intended as general conclusions about clinical human-AI decision-making.Recurring themes across two clinical contexts nevertheless suggest commonalities and transferability.Source: §7 · p. 22
  • Clinicians’ accounts cannot establish how much DSS predictions influenced particular decisions, and low attention or time pressure may still permit automation bias.The authors also note that reproducing real decision tasks, such as patient consultations, has limitations.Source: §7 · p. 23
  • Future studies could broaden understanding with reliable measures of overreliance in multifaceted decision-making and participants with varied expertise.The authors note that inexperienced decision-makers may incorporate advice differently.Source: §7 · p. 23

8 Conclusion

The authors triangulated survey and interview data from clinicians working with DSS in two medical cohorts. They report that experience, case complexity, and clinical context shape how clinicians engage with predictions, and call for DSS design and research grounded in socio-technical factors and realistic use.

8 Conclusion

  • The study combined survey statistics and thematic analysis of semi-structured interviews with 12 clinicians from two medical cohorts using different DSS.It identified eleven themes.Source: §8 · p. 23
  • Experienced participants described predictions as a second opinion, especially useful in complex cases, while their reliance changed with experience and hospital infrastructure.They described weighing predictions against other medical information, discussing decisions with colleagues, and potentially using what-if predictions.Source: §8 · p. 23
  • The findings connect human, technical, contextual, and organisational factors to DSS reliance and suggest designs that support individual and team decision-making.The authors recommend realistic use cases and attention to long-term use, accumulated experience, and diverse needs within a context.Source: §8 · p. 23

Generative AI Usage Statement

The authors report using Grammarly with AI features turned off, DeepL Translator, and noScribe for interview transcription, and state that no other generative AI tool was used.

Generative AI Usage Statement

Related papers

Loading 2610.40224v1…