Source-linked AI summary
Limits of trust in medical AI
Joshua Hatherley
TL;DR
The paper asks whether medical AI can occupy the role of a trusted participant in clinical decision-making as it increasingly challenges clinicians' authority. It argues, through philosophical accounts distinguishing trust from reliance, that AI can be reliable but cannot be trusted or trustworthy. Consequently, shifting medical decisions toward AI may weaken the rich trusting relationships patients can have with human clinicians.
Problem
Medical AI's growing role in clinical decisions raises a concern about whether reliance on AI can preserve trust between patients and clinicians.
Method
The paper evaluates medical AI using philosophical accounts that distinguish interpersonal trust from mere reliance, emphasizing motivation, goodwill, agency, and normative obligation.
Results
AI systems may be reliable and relied upon, but they are not appropriate objects of trust or trustworthiness because they lack agency, goodwill, and responsibility-bearing capacity.
Takeaways & Limitations
The pursuit of trustworthy AI should be reframed as the pursuit of reliable AI while reserving trust for reciprocal relations between beings with agency.
Takeaways & Limitations
The paper notes that trust may be misguided or unwelcome when someone lacks the expertise or competence to perform the entrusted task.
Abstract
from arXiv · showhide
Artificial intelligence (AI) is expected to revolutionize the practice of medicine. Recent advancements in the field of deep learning have demonstrated success in a variety of clinical tasks: detecting diabetic retinopathy from images, predicting hospital readmissions, aiding in the discovery of new drugs, etc. AI's progress in medicine, however, has led to concerns regarding the potential effects of this technology upon relationships of trust in clinical practice. In this paper, I will argue that there is merit to these concerns, since AI systems can be relied upon, and are capable of reliability, but cannot be trusted, and are not capable of trustworthiness. Insofar as patients are required to rely upon AI systems for their medical decision-making, there is potential for this to produce a deficit of trust in relationships in clinical practice.
1 Introduction
AI has shown success across several clinical tasks, prompting expectations of major improvements in medicine. The paper argues that these advances may threaten trust in doctor-patient relationships.
- Deep learning has demonstrated success in detecting diabetic retinopathy, predicting hospital readmissions, and aiding drug discovery.
- The paper argues that AI's medical progress raises concerns about its effects on trust between doctors and patients.
- If patients rely on AI for medical decisions, clinical relationships may develop a deficit of trust.
2 Trust in medicine
Trust in medicine has intrinsic value because it gives the vulnerable doctor-patient relationship its distinctive importance. It also has instrumental value by encouraging patients' engagement with medical care.
- Trust gives the doctor-patient relationship inherent value because patients are vulnerable and depend on physicians for competent, caring treatment.
- Trust makes patients more likely to seek care, disclose sensitive information, accept treatment, participate in research, and adhere to treatment regimens.
3 AI in medicine
The effect of AI on medical trust depends on whether it remains a clinician's tool or gains epistemic authority in clinical decisions. If AI outperforms clinicians, human authority and patient trust may shift toward machines.
- If AI remains a tool whose outputs clinicians interpret, its effect on trust would likely be minimal.
- AI's adoption raises the prospect that human clinicians will lose epistemic authority in clinical decision-making.
- Substitutionists expect AI eventually to make doctors obsolete, whereas extensionists expect it to enhance clinicians without replacing them.
- Neural-network AI has limited capacity to identify causes of illness because it learns from correlations alone.
- Only 14 of 31,587 articles identified in one review compared deep-learning systems and human clinicians on the same test dataset.
- If AI surpasses clinicians on key tasks, doctors may have an epistemic obligation to defer to or align their judgments with AI.
- Displacing clinicians' epistemic authority would imply a shift of patient trust from human clinicians to AI systems.
4 Trust in AI
The paper distinguishes trust from mere reliance by requiring suitable motivations and normative obligations. Because AI lacks goodwill, agency, and moral responsibility, reliance on AI should not be described as interpersonal trust.
- Accounts of interpersonal trust distinguish it from reliance by requiring more than confidence that someone will behave predictably.
- Trust involves believing that the trusted person's motivations take one's interests into account or reflect goodwill.
- AI systems lack goodwill or motivation to act in patients' interests, unlike human clinicians.
- Trust also generates an obligation for the trusted person to genuinely attempt the entrusted action.
- An AI system cannot bear the moral responsibility associated with an entrusted clinical decision; responsibility instead falls on people or institutions around it.
- Because AI lacks agency and cannot bear normative obligations, relations with AI may be shallow or deficient compared with human trusting relationships.
5 Conclusion
The paper argues that AI should be described as reliable rather than trustworthy because trust requires capacities AI lacks. Reliance on AI for medical decisions may displace human clinicians and reduce opportunities for interpersonal trust.
- AI systems can be reliable and relied upon, but they are not appropriate objects of trust or trustworthiness.The paper argues that familiar philosophical accounts of trust require characteristics associated with agency.
- The pursuit of trustworthy AI should instead be reframed around reliable AI, reserving trust for reciprocal relations between beings with agency.
- When patients rely on AI for important assessments and decisions, they may sacrifice opportunities for the rich interpersonal trust available with human clinicians.The paper calls for deploying AI in ways that retain trust’s valuable role in medicine.