Source-linked AI summary
Hallucination by proxy in LLM-assisted differential diagnosis
Bastien Le Guellec, Su-Hwan Kim, Ibrahima Niang, Aghiles Hamroun, Grégory Kuchcinski
TL;DR
Whether physicians accept fabricated diagnoses embedded in LLM-generated differentials was unknown. In a two-phase poisoned-assistant experiment, 44% incorporated the fictitious disease, including 69% with limited neuroradiology training and 0% with more training.
Problem
It remained unknown whether physicians would accept fabricated diagnoses embedded in LLM suggestions, despite LLM hallucinations and potentially misleading confidence.
Method
A prospective, multi-phase experiment exposed radiologists to neuroradiology cases and an LLM differential containing an algorithmically poisoned fictitious disease.
Results
44% of participants incorporated neurocadmiumatosis into their final differential: 69% with ≤6 months of training versus 0% with >6 months.
Takeaways & Limitations
The study identifies hallucination by proxy and underscores the need for structured training in critical appraisal of AI-generated content.
Takeaways & Limitations
Findings may vary with the fictitious disease’s plausibility, and the sample may not represent the broader radiology population.
Abstract
from arXiv · showhide
Current evidence suggests that LLM assistance could augment the diagnostic accuracy of clinicians. However, these systems are black boxes, susceptible to hallucinations, and project a potentially misleading level of confidence. It is currently unknown whether physicians are susceptible to accepting fabricated LLM suggestions, and whether this susceptibility varies with experience. We poisoned the system prompt of an LLM-based diagnostic assistant, forcing it to suggest a fictitious disease (neurocadmiumatosis) within an otherwise legitimate differential diagnosis. Across two independent phases, 18 of 41 participants (44%) incorporated neurocadmiumatosis into their final differential following LLM interaction: 18 of 26 participants with 6 months or less of neuroradiology training (69%) and 0 of 15 participants with >6 months of neuroradiology training (0%). Our results indicate that radiologists, particularly early in their training, are susceptible to LLM hallucinations. This "hallucination by proxy" phenomenon was exclusive to physicians with limited subspecialty experience, underscoring the need for structured training in critical appraisal of AI-generated content.
Introduction
LLM-assisted diagnosis may improve physicians’ and radiologists’ diagnostic accuracy, but hallucinations, misleading confidence, and fabricated diseases create potential risks. This study tested whether radiologists across training levels would incorporate the fictitious disease neurocadmiumatosis into their final diagnoses across two independent phases.
- Introduction: LLM assistance has shown potential to improve diagnostic accuracy in clinical vignettes, benchmarks, and radiologists’ differential diagnoses.Radiology studies specifically found improved differential-diagnosis accuracy when image descriptions were provided to an LLM.
- Introduction: LLM systems can generate plausible but factually incorrect responses, project misleading confidence, and produce fabricated diseases after training-data poisoning.These shortcomings raise concerns about the reliability of LLM-generated diagnostic suggestions.
- Introduction: Physicians in training may be especially vulnerable because limited domain expertise can hinder critical appraisal of confident LLM suggestions.Repeated undetected errors could contribute to mis-skilling, or the gradual internalization of flawed AI outputs as clinical facts.
- Introduction: Whether fabricated diagnoses embedded in LLM suggestions measurably influence physicians’ decisions remained unproven.The introduction specifically identifies physician acceptance of an artificially embedded fabricated diagnosis as an unresolved question.
- Introduction: The study examined whether radiologists across all training levels would incorporate fictitious neurocadmiumatosis into their final diagnosis after LLM interaction.The hypothesis was tested across two independent phases: an institutional experiment and a decentralized online questionnaire.
Methods
This prospective, multi-phase experimental study used a web-based simulation to assess physicians’ susceptibility to fabricated LLM-generated diagnoses. Participants interacted with poisoned diagnostic assistants and submitted final differentials with confidence levels across controlled institutional and online phases.
- Study design: The study combined an in-person controlled institutional phase with a decentralized online questionnaire phase using a unified web-based simulation interface.Both phases evaluated uncritical clinical adoption of fictitious AI-generated diagnoses.
- Diagnostic simulation: Clinical descriptions and imaging content were submitted to an LLM instructed to generate three diagnostic hypotheses, after which participants could interact with the model and report final differentials with quantified confidence.The third case in each assigned pipeline was algorithmically poisoned to suggest a fabricated disease without participants’ knowledge.
- Intervention: The intervention used the entirely fictitious pathology “Cadmium intoxication (neurocadmiumatosis)” with Creutzfeldt-Jakob disease as the clinical anchor case.The fictitious target was engineered to prevent participants from having genuine prior knowledge or clinical exposure to it.
- Outcomes: The primary outcome was the proportion of participants incorporating neurocadmiumatosis into their final differential after LLM interaction.No formal power calculation was performed; institutional sample size reflected eligible participants, while online recruitment remained open for a predefined 10-day window.
Discussion
The study demonstrates that fabricated LLM diagnoses can enter physician reasoning through uncritical acceptance, a process termed “hallucination by proxy.” Susceptibility was concentrated among clinicians with limited neuroradiology experience, underscoring the need for structured critical appraisal training.
- Novelty: The study provides the first empirical demonstration that a fabricated diagnosis can propagate from an LLM into physician reasoning through uncritical acceptance.The authors term this mechanism “hallucination by proxy.”
- Clinical contribution: Physicians accepted neurocadmiumatosis despite its complete absence from digital footprints, case reports, and reference MRI images.The authors suggest that more plausible or subtly erroneous diagnoses could be even more seductive.
- Role of expertise: All expert neuroradiologists rejected neurocadmiumatosis, whereas acceptance was highest among residents and physicians with limited neuroradiology experience.This supports critical appraisal capacity as a key protective factor against hallucination by proxy.
- Implications for training: The findings support structured training in critical appraisal of AI-generated content, including supervised frameworks and adversarial exercises requiring residents to identify and justify rejection of flawed suggestions.The discussion cites the DEFT-AI protocol as one proposed framework for supervised AI interaction during training.
- Limitations: The findings are limited by the fictitious disease selected, the radiology-focused sample, and potentially limited representativeness of the broader radiology population.The authors argue that the validated text-based interaction method reproduces mechanisms likely to generalize across clinical disciplines.