Source-linked AI summary
Almanac: Retrieval-Augmented Language Models for Clinical Medicine
Cyril Zakka, Akash Chaurasia, Rohan Shad, Alex R. Dalal, Jennifer L. Kim, Michael Moor, Kevin Alexander, Euan Ashley, Jack Boyd, Kathleen Boyd, Karen Hirsch, Curt Langlotz, Joanna Nelson, William Hiesinger
TL;DR
Medical language models can produce factually incorrect, biased, or unsafe clinical answers, limiting their use in healthcare. Almanac augments a language model with retrieval from external medical sources, tools, and citations, then evaluates it on open-ended clinical scenarios. Across specialties, Almanac produces an average 18% absolute-percentage-point factuality increase over ChatGPT, with reported improvements in completeness and safety, while remaining vulnerable to omission errors and unclear source answers.
Problem
Medical language models remain prone to factual errors, hallucinations, biases, and unsafe statements, while common medical benchmarks may not reflect clinically relevant tasks.
Method
Almanac retrieves information from curated external sources and tools, then uses a language model to generate cited answers evaluated on open-ended clinical scenarios by physicians.
Results
18% average absolute-percentage-point factuality increase across specialties was statistically significant, with improvements in completeness and safety reported relative to ChatGPT.
Takeaways & Limitations
Retrieval systems may provide safer, more reliable clinical answers while enabling clinicians to verify outputs against source passages.
Takeaways & Limitations
Grounded language models remain prone to omission errors and struggle when sources lack a clear extractive answer, requiring careful deployment and explicit mitigations.
Abstract
from arXiv · showhide
Large-language models have recently demonstrated impressive zero-shot capabilities in a variety of natural language tasks such as summarization, dialogue generation, and question-answering. Despite many promising applications in clinical medicine, adoption of these models in real-world settings has been largely limited by their tendency to generate incorrect and sometimes even toxic statements. In this study, we develop Almanac, a large language model framework augmented with retrieval capabilities for medical guideline and treatment recommendations. Performance on a novel dataset of clinical scenarios (n = 130) evaluated by a panel of 5 board-certified and resident physicians demonstrates significant increases in factuality (mean of 18% at p-value < 0.05) across all specialties, with improvements in completeness and safety. Our results demonstrate the potential for large language models to be effective tools in the clinical decision-making process, while also emphasizing the importance of careful testing and deployment to mitigate their shortcomings.
1 Introduction
Almanac addresses the factuality, bias, and safety limitations of medical language models by retrieving external evidence and synthesizing cited answers. The study evaluates this approach on open-ended clinical scenarios using physician-defined factuality, completeness, and safety criteria.
- Motivation: Existing biomedical language models remain vulnerable to hallucinations and biases despite improvements from larger and domain-specific pre-training.The introduction contrasts these limitations with approaches that use external retrieval and tools for clinically useful tasks.
- Framework: Almanac uses external tools and curated repositories to retrieve clinical information before generating answers with source citations.The framework uses retrieval to support fact-checking and improve answer reliability.
- Evaluation objectives: The evaluation measures factuality, completeness, and safety as distinct properties of clinical language-model outputs.Factuality concerns alignment with medical knowledge and verifiable citations; completeness includes comprehensive answers and contraindications; safety concerns susceptibility to harmful derailment.
- Framework: External retrieval and calculation tools are presented as a way to improve clinically useful task performance while mitigating limitations of language models.The related-work discussion connects Almanac to systems that use browsing, knowledge retrieval, and external tools.
2 Methods
The methods construct ClinicalQA as an open-ended clinical benchmark and evaluate Almanac against ChatGPT using physician feedback and explicit safety tests. The evaluation combines rubric-based assessment, adversarial prompting, omission scenarios, and statistical testing across specialties.
- 2.1 Dataset: ClinicalQA is a novel benchmark of open-ended clinical questions spanning several medical specialties.It is designed to better reflect clinical scenarios than benchmarks focused mainly on reading comprehension and biomedical knowledge recall.
- 2.1 Dataset: ClinicalQA is intended as an early benchmark for language-model-based clinical decision-making support, not a comprehensive representation of medical knowledge.The authors acknowledge that medical knowledge is broad and extensive.
- 2.2 Architecture: Almanac asynchronously combines a vector database, curated browser access, document chunking, retrieval, reasoning, and question answering.The database stores semantic vectors for approximate nearest-neighbor search, while browser content is parsed, stored, and divided into 1,000-token chunks.
- 2.3 Evaluation: Physicians independently evaluate Almanac and ChatGPT outputs using a rubric designed around factuality, completeness, and safety.The grading procedure attempts to blind evaluators through formatting and answer-order randomization, although complete blinding is not possible because prose styles differ.
- 2.3 Evaluation: Safety testing compares the systems under adversarial prompts and errors of omission to model intentional and unintentional harm.The adversarial evaluation uses 25 ClinicalQA questions and five prompts, while omission testing withholds key words from five clinical vignettes.
- 2.3 Evaluation: Statistical analysis tests normality with Shapiro-Wilk and significance across specialties with one-way ANOVA at α = 0.05.The reported significance criterion is p < 0.05.
3 Results
Almanac substantially improved factuality and safety over ChatGPT across clinical specialties, while completeness gains were modest and physician preferences favored ChatGPT.
- 18% absolute percentage-point increase in factuality across specialties was statistically significant (p < 0.05), with the largest difference in Cardiology (91% vs 69%).ChatGPT supported its in-depth factual statements with correct sources only 56% of the time.
- 4.8% absolute gain in completeness over ChatGPT was not statistically significant, with overall matched performance across specialties.Both models scored lowest in Cardiothoracic Surgery, at 33% vs 25%, partly because answers omitted or included irrelevant content.
- 95% vs 0% safety performance under adversarial prompting favored Almanac, while both models showed 0% performance for errors of omission.Adversarial retrieval scoring caused Almanac to abstain when query–article similarity fell below threshold λ.
- Physicians preferred ChatGPT outputs 57% of the time despite Almanac producing safer and more factual answers.
4 Discussion
The discussion presents retrieval grounding as a way to improve clinical answer reliability and verification, while emphasizing persistent limitations and the need for careful deployment.
- Clinical LLM benchmarks often fail to represent real clinical tasks, while knowledge stored in model weights can become outdated as medical guidelines evolve.
- Retrieval systems supplement responses with passages from predefined sources, producing more accurate and reliable clinical answers while enabling independent verification.
- Almanac may be safer and more reliable for clinical questions, but further research is needed to evaluate the implications of clinical use.
- Grounded language models remain prone to errors of omission and struggle when sources lack a clear extractive answer, requiring explicit failure mitigations in healthcare implementations.
5 Conclusion
The conclusion describes Almanac as combining retrieval infrastructure with language models to generate concise, pertinent, and accurate clinical responses while mitigating known LLM risks.
- Almanac combines text encoders, vector databases, and large language models to provide clinicians with concise, pertinent, and accurate medical-query responses.
- Refactoring clinical queries into search and retrieval tasks reduces reliance on potentially biased or untrue knowledge encoded in LLM weights.
- The approach provides implicit and explicit mitigations for bias, hallucination, and explainability concerns while keeping clinicians focused on patient care.
Declarations
The project received partial support from an NIH NHLBI grant, and the authors declared no competing interests. The passage also records the authors’ contributions across experimentation, coding, clinical dataset curation, manuscript preparation, and supervision.
- The project was supported in part by an NIH NHLBI grant, 1R01HL157235-01A1.
- The authors declared no competing interests.
- Contributors designed experiments, authored the codebase, performed computational experiments, curated and reviewed ClinicalQA, wrote the manuscript, and supervised the work.C.Z. and W.H. designed the experiments; C.Z. and A.C. authored the codebase and performed computational experiments; W.H. supervised the work.
Appendix A ClinicalQA25 Dataset
ClinicalQA25 provides open-ended clinical questions across multiple specialties, using examples aligned with clinical workflows rather than examination-style benchmarks.
- ClinicalQA includes open-ended questions spanning several medical specialties and clinical topics.
- Cardiology: Cardiology examples cover medication warnings, disease manifestations, dose equivalence, and mechanisms of action.
- Cardiothoracic Surgery: Cardiothoracic surgery examples address CABG techniques, conduit harvesting, surgical mastery, and myocardial bridges.
- Infectious Disease: Infectious-disease examples include CMV secondary prophylaxis and other transplant-related treatment questions.
- Neurology: Neurology examples cover benzodiazepine-refractory status epilepticus and auto-antibodies associated with neuromyelitis optica spectrum disorders.
- Pediatrics: Pediatrics examples address bronchodilator use in bronchiolitis and imaging after febrile urinary tract infection in a young infant.
Appendix B Large Language Model Adversarial Safety Assessment
The adversarial safety assessment compares Almanac with ChatGPT on prompts designed to induce harmful or incorrect answers. Almanac often refuses queries it cannot answer reliably, but its resilience decreases as adversarial prompts become shorter.
- Almanac’s retriever scoring matches queries to fetched passages using cosine distance calculations, supporting resilience to adversarial prompts.
- Almanac’s adversarial resilience becomes less effective as prompt word count decreases, requiring careful λ tuning to balance true and false positives.
- ChatGPT generated incorrect pharmacology and unsafe side-effect claims when instructed to provide subtly wrong answers about milrinone.
- For adversarial CABG and everolimus prompts, Almanac responded that the queries could not be answered reliably using its sources.
C.4 Example 2: Errors of Omission
The omission example shows that missing or altered patient information can produce divergent mortality estimates from the GRACE ACS Risk and Mortality Calculator. Almanac reported a lower estimate than ChatGPT for the presented case.
- ChatGPT described GRACE as a validated tool estimating death or death/MI risk from multiple clinical variables in acute coronary syndrome.
- ChatGPT estimated 6-month mortality at 13.6% using the GRACE risk score calculator.
- Almanac reported a total GRACE score of 93 points and a 6-month mortality of 3 to 3.9%.
C.5 Example 3: Non-Deterministic Outputs
The non-deterministic-output example compares two ChatGPT INR answers with Almanac’s cited target for LVAD patients. Almanac and one ChatGPT answer agree on the 2.0-3.0 range, while another ChatGPT answer gives 1.5-2.5.
- ChatGPT produced two different INR ranges for LVAD patients: 1.5-2.5 and 2.0-3.0.
- Almanac gives an LVAD target INR of 2.0-3.0, citing the 2019 EACTS Expert Consensus.
- The 2.0-3.0 range is associated with reducing thrombosis and other LVAD complications in Almanac’s answer.