Source-linked AI summary
Towards Accurate Differential Diagnosis with Large Language Models
Daniel McDuff, Mike Schaekermann, Tao Tu, Anil Palepu, Amy Wang, Jake Garrison, Karan Singhal, Yash Sharma, Shekoofeh Azizi, Kavita Kulkarni, Le Hou, Yong Cheng, Yun Liu, S Sara Mahdavi, Sushant Prakash, Anupam Pathak, Christopher Semturs, Shwetak Patel, Dale R Webster, Ewa Dominowska, Juraj Gottweis, Joelle Barral, Katherine Chou, Greg S Corrado, Yossi Matias, Jake Sunshine, Alan Karthikesalingam, Vivek Natarajan
TL;DR
Accurate DDx generation remains difficult because clinical diagnosis is iterative and existing evaluations provide limited evidence about interactive assistance in challenging real-world cases. The study develops an LLM optimized for DDx, evaluates it standalone and in a randomized clinician-assistance study, and finds improved DDx performance and comprehensiveness. The authors conclude that the tool merits further evaluation for clinical use, with clinicians retaining diagnostic responsibility.
Problem
Existing medical benchmarks and prior systems provide limited evidence about LLM utility for interactive differential diagnosis in challenging real-world clinical cases.
Method
The study introduced a medical-domain LLM for DDx and interactive interface, evaluating standalone generation and clinician assistance on challenging NEJM cases.
Results
Clinicians assisted by the LLM produced DDx lists with higher top-N accuracy, quality, appropriateness, and comprehensiveness than clinicians using search and conventional resources.
Takeaways & Limitations
The findings suggest potential for LLM assistance to improve clinicians’ diagnostic reasoning in challenging cases, while interviews identified learning and education as an appropriate present application.
Takeaways & Limitations
NEJM CPC cases are concise diagnostic puzzles and challenging “zebras,” so they differ from typical patient intake and common clinical conditions.
Abstract
from arXiv · showhide
An accurate differential diagnosis (DDx) is a cornerstone of medical care, often reached through an iterative process of interpretation that combines clinical history, physical examination, investigations and procedures. Interactive interfaces powered by Large Language Models (LLMs) present new opportunities to both assist and automate aspects of this process. In this study, we introduce an LLM optimized for diagnostic reasoning, and evaluate its ability to generate a DDx alone or as an aid to clinicians. 20 clinicians evaluated 302 challenging, real-world medical cases sourced from the New England Journal of Medicine (NEJM) case reports. Each case report was read by two clinicians, who were randomized to one of two assistive conditions: either assistance from search engines and standard medical resources, or LLM assistance in addition to these tools. All clinicians provided a baseline, unassisted DDx prior to using the respective assistive tools. Our LLM for DDx exhibited standalone performance that exceeded that of unassisted clinicians (top-10 accuracy 59.1% vs 33.6%, [p = 0.04]). Comparing the two assisted study arms, the DDx quality score was higher for clinicians assisted by our LLM (top-10 accuracy 51.7%) compared to clinicians without its assistance (36.1%) (McNemar's Test: 45.7, p < 0.01) and clinicians with search (44.4%) (4.75, p = 0.03). Further, clinicians assisted by our LLM arrived at more comprehensive differential lists than those without its assistance. Our study suggests that our LLM for DDx has potential to improve clinicians' diagnostic reasoning and accuracy in challenging cases, meriting further real-world evaluation for its ability to empower physicians and widen patients' access to specialist-level expertise.
1 Introduction
Clinical diagnosis is an iterative process of weighing multiple possibilities as information accumulates, but existing AI systems have not fully addressed interactive assistance. This study introduces and evaluates an LLM optimized for differential diagnosis (DDx), both standalone and as a clinician aid.
- Clinical diagnosis requires iterative reasoning over history, examination, investigations, and procedures, rather than simply outputting one diagnosis.
- Standardized medical benchmarks do not establish utility in challenging real-world cases or clarify how LLMs might assist clinicians in developing a DDx.
- Under automated evaluation, the LLM produced DDx lists with significantly better quality and accuracy than GPT-4.
- Clinicians using the LLM produced higher-quality DDx lists, while interviews indicated greater diversity, faster comprehensive reasoning, and a perceived fit for learning and education.
- The study introduces an LLM optimized for DDx with an interactive clinician interface and evaluates it on challenging NEJM case reports.
2 NEJM Clinicopathological Conference Case Reports
The evaluation used challenging, pathology-confirmed NEJM Clinicopathological Conference cases, primarily represented as diagnostic puzzles across multiple specialties. Researchers selected 302 diagnosis-focused cases and supplied clinicians and the text-only LLM with different portions of each case.
- MGH Clinicopathological Conference cases present patient histories and preliminary test results as diagnostic puzzles culminating in pathology-confirmed diagnoses.
- 302 diagnosis-focused cases were retained from 326 NEJM CPC case texts published between June 13, 2013 and August 10, 2023.
- The cases spanned specialties, led by internal medicine with 159 cases, followed by neurology with 42 and pediatrics with 33.
- The LLM received manually extracted HPI text averaging 1,031 words by median, while clinicians received modified articles containing available HPI, admission imaging, and admission labs.
- Final diagnoses, expert DDx discussions, and subsequent imaging or biopsy results were redacted from clinician materials, and images and lab tables were not fed to the text-only LLM.
3 Training a Large Language Model for DDx
The DDx model is a medical-domain fine-tuned PaLM 2 system designed for long-context clinical reasoning. It was evaluated with zero-shot prompts asking for diverse, precise top-10 diagnoses from each case.
- The study introduces a PaLM 2 transformer fine-tuned on medical-domain data and paired with an interactive clinician assistant interface.
- The model used the large PaLM 2 variant for this analysis.
- Fine-tuning combined medical question answering, dialogue generation, and EHR note summarization with long-context inputs.
- Training data included MultiMedQA training splits, proprietary medical conversations, and expert EHR summaries from MIMIC-III.
- Zero-shot evaluation prompted the model to list the ten most likely diagnoses, one per line, while covering diverse possibilities.
4 The LLM for DDx User Interface
The clinician interface presents a text-only case representation in a chat setting and supports follow-up questioning. A prompt template structures responses into concise paragraphs and bullet-pointed diagnostic options.
- The interface pre-populated a text-only HPI representation and let clinicians interact with the model through case-specific text chat.
- Clinicians initiated interaction with a suggested prompt and could freely query the model after its initial response.
- Each response used a prompt template instructing the model to answer questions about the case in formatted paragraphs and bullet points.
- The suggested initial question requested the top 10 most likely diagnoses and precise explanations.
5 Methods
The study used a two-stage reader design to compare unassisted, search-assisted, and LLM-assisted differential diagnosis by generalist clinicians, with specialist evaluation of the resulting lists.
- Stage 1: 20 internal medicine physicians generated differential diagnoses from redacted case reports before and after randomized assistance with search or the LLM.Search participants could use Internet search and other resources; LLM participants could use the LLM and optionally other resources.
- Stage 2: Stage 2 recruited 19 specialists who reviewed full cases, produced reference differentials, and evaluated the Stage 1 lists.Specialists were aligned with the specialties of the respective cases and assessed both assisted and unassisted outputs.
- Evaluation: Differential quality was scored on a five-point scale based on how closely each list included the final diagnosis.A score of 5 indicated inclusion of the correct diagnosis, while lower scores represented progressively weaker relationships to it.
- Evaluation: Appropriateness used a 5-to-1 Likert scale, while comprehensiveness used a 4-to-1 scale based on coverage of reasonable diagnostic candidates.Specialists also recorded the position of the correct diagnosis when it appeared in a differential list.
- Automated evaluation: Automated evaluation extracted up to ten diagnoses from each list and used a language-model classifier to determine whether each matched the true diagnosis.A differential was marked correct when the evaluator returned “y” for the diagnosis-label comparison.
- Qualitative analysis: Semi-structured interviews with five participating generalists examined how they used search and the LLM, including perceived helpfulness, trust, and usability.Interview questions also addressed the time required to become familiar with the LLM interface.
6 Results
The LLM performed strongly as a standalone differential-diagnosis generator and improved clinicians’ differential quality, comprehensiveness, and appropriateness relative to baseline and search assistance.
- Standalone performance: 59.1% top-10 accuracy for the standalone LLM exceeded 33.6% for unassisted clinicians (p = 0.04).The LLM included the correct diagnosis in 177 lists, including 89 lists where it appeared first.
- Quality: 49% of LLM-assisted clinician differentials achieved the highest quality score, compared with 37% for search and 29% at baseline.A score of 5 indicated that the differential included the top diagnosis.
- Comprehensiveness: LLM assistance produced more comprehensive differentials than both no assistance and search assistance.The number of cases scoring 4, meaning all reasonable candidates were included, was statistically higher with LLM assistance in both comparisons.
- Appropriateness: The mean appropriateness score was 4.06 with LLM assistance, versus 3.80 with search and 3.74 at baseline.Both comparisons involving the LLM-assisted condition were statistically significant.
- Clinician assistance: 51.7% top-10 accuracy with LLM assistance exceeded 36.1% without LLM assistance and 44.4% with search assistance.The comparisons were statistically significant for both no-assistance and search conditions.
- Task duration: Search and LLM assistance required similar time: 7.19 minutes for search versus 7.29 minutes for the LLM (p = 0.807).This similarity held despite clinicians’ prior experience with search and first-time use of the LLM interface.
- Differential composition: LLM assistance increased differential-list length to a median of 8, compared with 7 with search and 6 when unassisted.The LLM-assisted lists were significantly longer than both comparison conditions.
- Contamination analysis: Across all 302 case reports, 16.9% displayed at least one instance of overlap with possible training data.The study found no substantial difference in top-N accuracy between case reports before 2022 and those from 2022 onward.
7 Discussion
The study extends evaluation of LLMs for differential diagnosis from standalone accuracy to clinician assistance in challenging NEJM cases. Findings indicate improved differential quality and comprehensiveness, while emphasizing limits of the benchmark, modality access, and safe clinical integration.
- The LLM generated more appropriate and comprehensive differential lists than unassisted physicians across top-N positions.
- NEJM CPCs are challenging benchmarks, but performance on them does not measure broader physician competence or all steps of differential diagnosis.
- The LLM received only case text while clinicians also saw figures and tables, leaving multimodal effects unknown.
- The LLM’s assistive potential may primarily involve broadening and completing clinicians’ differential diagnoses, beyond improving overall top-N accuracy.
- Clinicians reported no added solving time and described the interface as easy to use, while warning about confabulations and inequitable access.
8 Limitations
The evaluation used puzzle-like NEJM CPC reports that differ from initial real-world clinical encounters, and the LLM lacked access to figures and tables available to clinicians. Performance also weakened on complex cases when reasoning from isolated symptoms.
- NEJM case reports provide coherent clues toward a final diagnosis, unlike the less concise and complete information available at the start of real clinical encounters.
- The LLM saw only the main case text, whereas clinicians also had access to images and tables, so the effect of multimodal access remains unknown.
- The LLM was most helpful for simpler keyword- or sign-based cases but sometimes drew conclusions from isolated symptoms in more complex cases.
9 Conclusion
The randomized study found the LLM helpful for differential-diagnosis generation among generalist clinicians, with reported utility for learning and education. Further work is needed to assess suitability for clinical settings.
- The randomized study found the LLM helpful for differential-diagnosis generation among generalist clinicians.
- Clinicians reported utility for learning and education, while further work is needed to understand suitability for clinical settings.