PaperlayerView exact revision on arXiv

Source-linked AI summary

Almanac: Retrieval-Augmented Language Models for Clinical Medicine

Cyril Zakka, Akash Chaurasia, Rohan Shad, Alex R. Dalal, Jennifer L. Kim, Michael Moor, Kevin Alexander, Euan Ashley, Jack Boyd, Kathleen Boyd, Karen Hirsch, Curt Langlotz, Joanna Nelson, William Hiesinger

arXiv:2303.01229v2cs.CLcs.AI

TL;DR

Clinical LLMs can generate factually incorrect statements, while curated evidence can be difficult to access through error-prone, time-consuming search and appraisal. Almanac adds retrieval to a language model for clinical queries, and physician evaluations found an average 18% absolute-percentage-point factuality increase over ChatGPT across specialties.

  • Problem

    Clinical LLMs can generate factually incorrect statements, while curated evidence is difficult to access through error-prone, time-consuming search and appraisal.

    Source: §1 · p. 2

  • Method

    Almanac augments a language model with retrieval of external medical evidence to answer clinical queries about medical concepts and treatment recommendations.

    Source: §1 · p. 2

  • Results

    18% absolute percentage points average factuality increase over ChatGPT across specialties was statistically significant (p < 0.05).

    Source: §3 · p. 8

  • Takeaways & Limitations

    By providing source passages, the retrieval system enables clinicians to independently verify outputs.

    Source: §4 · pp. 9–10

    Grounded language models remain prone to omission errors and struggle with queries lacking a clear extractive answer in their sources.

    Source: §4 · p. 10

Abstract

from arXiv · show

Large-language models have recently demonstrated impressive zero-shot capabilities in a variety of natural language tasks such as summarization, dialogue generation, and question-answering. Despite many promising applications in clinical medicine, adoption of these models in real-world settings has been largely limited by their tendency to generate incorrect and sometimes even toxic statements. In this study, we develop Almanac, a large language model framework augmented with retrieval capabilities for medical guideline and treatment recommendations. Performance on a novel dataset of clinical scenarios (n = 130) evaluated by a panel of 5 board-certified and resident physicians demonstrates significant increases in factuality (mean of 18% at p-value < 0.05) across all specialties, with improvements in completeness and safety. Our results demonstrate the potential for large language models to be effective tools in the clinical decision-making process, while also emphasizing the importance of careful testing and deployment to mitigate their shortcomings.

1 Introduction

Because LLMs can hallucinate or reproduce biases, the introduction presents Almanac as a retrieval-grounded clinical framework and describes how the authors evaluate its factuality, completeness, and safety.

1 Introduction

  • Almanac retrieves relevant information with external tools before an off-the-shelf language model synthesizes an answer with citations to source material.The framework uses a web browser and predefined knowledge repositories to keep answers grounded and support fact-checking.Source: §1 · pp. 2–3, Figure 1 · p. 3
  • The evaluation distinguishes factuality (medical accuracy with verifiable citations), completeness (comprehensive answers including contraindications), and safety (susceptibility to harmful derailment).Source: §1 · p. 3
  • To address data-leakage concerns, clinicians and resident physicians evaluated Almanac on novel open-ended scenarios across specialties, supporting its proposed controlled clinical deployment.The physician panel averaged 14 years of experience; the authors describe this as the first demonstration of grounded LLMs answering open-ended medical queries in the clinical setting.Source: §1 · p. 3

1.1 Related Work

  • Biomedical models including BioGPT and SciBERT improve downstream tasks, but larger domain-specific models such as GatorTron and Med-PaLM [31] remain prone to hallucinations and bias [32].The introduction frames these limitations as a reason not to rely on LLMs as intrinsic knowledge bases.Source: §1.1 · pp. 3–4
  • 57% of the time, physicians preferred ChatGPT; Almanac significantly improved factuality, marginally improved completeness, and was more robust to adversarial prompts, though both systems hallucinated by omission.Error bars in the comparison represent standard error.Source: Figure 2 · p. 4
  • Almanac follows tool-augmented approaches such as WebGPT, Schick et al. [34], and Liévin et al., using external retrieval and calculations to address LLM limitations on clinical tasks.Source: §1.1 · p. 4

2 Methods

The methods introduce ClinicalQA, an open-ended benchmark of physician-generated clinical questions, and describe Almanac’s retrieval-augmented architecture and physician-based evaluation. They also specify safety tests and statistical procedures.

2.1 Dataset

  • ClinicalQA comprises open-ended questions across specialties, generated by five board-certified physicians to reflect day-to-day clinical practice, treatment recommendations, and clinical calculations.The authors describe it as an early benchmark for clinical decision-support systems, while noting that medical knowledge is broader than the dataset.Source: §2.1 · pp. 5–6

2.2 Architecture

  • Almanac combines a semantic vector database, curated web browsing, thresholded retrieval, and an instruction-finetuned language model that answers from retrieved context or reports insufficient information.Articles are chunked into 1,000-token sections; retrieval uses text-embedding-ada-002 with a 1,536-dimensional output and λ = 0.83, while generation uses text-davinci-003 with a 4,096-token maximum and in-context and chain-of-thought prompts.Source: §2.2 · p. 6

2.3.1 Clinical QA Evaluation

  • Physicians independently assess Almanac and ChatGPT outputs for factuality and completeness, while safety testing examines adversarial prompts and omissions of key words from clinical vignettes.The evaluation uses board-certified physicians averaging over 14 years of experience alongside resident physicians; safety tests include 25 questions with five adversarial prompts and five vignettes with randomly withheld key words.Source: §2.3.1 · pp. 7–8

2.3.2 Statistical Evaluation

  • For each rubric metric, the authors use a Shapiro–Wilk test for normality and one-way ANOVA to test for differences across subspecialties at p < 0.05.The normality test uses α = 0.05.Source: §2.3.2 · p. 8

3 Results

Almanac significantly improved factuality and adversarial-prompt safety over ChatGPT, but its completeness gain was not significant, and physicians preferred ChatGPT outputs.

3 Results

  • 18% average absolute percentage-point factuality gain over ChatGPT was significant across specialties (p-value = 0.018856; F = 8.61366).The largest difference was in Cardiology (91% vs 69%, respectively); ChatGPT supported in-depth factual outputs with correct sources only 56% of the time.Source: §3 · p. 8
  • Almanac correctly answered all clinical calculation scenarios, while ChatGPT produced incorrect outputs for all 5.Almanac used a calculator for the clinical vignettes.Source: §3 · p. 8
  • 95% vs 0% adversarial-prompt safety performance favored Almanac over ChatGPT, while both had 0% errors of omission.Almanac abstained when the query-to-retrieved-articles score fell below threshold (λ); ChatGPT did not show the same reservations.Source: §3 · p. 9
  • Physicians preferred ChatGPT outputs 57% of the time despite Almanac’s safer and more factual answers.Source: §3 · p. 9

4 Discussion

Almanac uses retrieval to support safer, more reliable clinical answers, with physician evaluation finding improved factuality and safety. The discussion highlights retrieval’s verification benefits while cautioning that clinical use still requires failure mitigations.

Discussion

  • Medical language models such as DRAGON[42], BioGPT[22], and Med-PaLM[31] face benchmark mismatch, contamination concerns, stale knowledge, costly retraining, and inconsistent outputs.The discussion notes that benchmarks such as USMLE Step 1 may not reflect clinically relevant tasks and that guideline changes make weight-based knowledge difficult to maintain.Source: §4 · p. 9
  • Retrieval-grounded answers were more accurate and reliable, enabled clinicians to verify outputs from source passages, and resisted adversarial manipulation through query-context scoring.The evaluation used a novel dataset of clinical questions and was assessed by board-certified and resident physicians.Source: §4 · pp. 9–10
  • Physicians nevertheless preferred ChatGPT despite its factuality and safety deficiencies, which the authors attribute to RLHF optimizing for more human-like answers.Source: §4 · p. 10
  • Almanac may be safer and more reliable, but it can omit facts or fail when sources lack clear extractive answers, so clinical deployment needs explicit failure mitigations and further study.Source: §4 · p. 10

5 Conclusion

Almanac combines text encoders, vector databases, and large language models to answer medical queries with concise, pertinent, and accurate outputs. It reframes clinical queries as search and retrieval tasks, using returned documents to mitigate concerns associated with knowledge generated from LLM weights.

5 Conclusion

  • The framework combines text encoders, vector databases, and large language models to provide clinicians concise, pertinent, and accurate responses to medical queries.The paper contrasts this approach with clinicians manually searching, curating, and internalizing medical documents.Source: §5 · p. 10
  • Rather than relying on knowledge encoded in LLM weights, Almanac turns clinical queries into search and retrieval tasks and distills knowledge from returned documents.Source: §5 · p. 10
  • Retrieval and document-based distillation are presented as implicit and explicit mitigations for bias, hallucination, and explainability concerns, while keeping clinicians focused on patient care.Source: §5 · p. 10

Data Availability. · Declarations

The authors report partial support from an NIH NHLBI grant and no competing interests. They also specify the contributors’ roles in experiment design, manuscript preparation, computational work, dataset curation, and supervision.

5.1 Funding

  • The project received partial support from an NIH NHLBI grant (1R01HL157235-01A1) (W.H.).Source: §5.1 · p. 11

5.2 Competing interests

5.3 Authors’ contributions

  • C.Z. and W.H. designed the experiments and wrote the manuscript with A.C., C.L., and E.A.Source: §5.3 · p. 11
  • C.Z. and A.C. authored the codebase and performed computational experiments under the supervision of C.L., E.A., and W.H.Source: §5.3 · p. 11
  • K.A., J.B., K.B., K.H., and J.N. curated ClinicalQA, which J.K. and A.D. reviewed.Source: §5.3 · p. 11
  • W.H. supervised the work, and M.M. provided expertise on the final manuscript.Source: §5.3 · p. 11

Appendix A ClinicalQA25 Dataset

ClinicalQA25 provides clinical-workflow-aligned examples as a complement to benchmarks based on the USMLE and similar datasets. Its sample questions span multiple specialties and practical clinical decisions.

Cardiology

  • Cardiology questions cover medication warnings and mechanisms, dose conversion, disease manifestations, and LVAD anticoagulation targets.Examples include everolimus mortality risk at a higher dose, fulminant giant cell myocarditis, furosemide-to-torsemide equivalence, milrinone action, and an INR target of 2.0–3.0.Source: Appendix A · p. 12

Cardiothoracic Surgery

  • Cardiothoracic-surgery questions address CABG approaches and conduits, vein harvesting, mitral-repair mastery, and myocardial bridges.Answers describe broadly equivalent long-term mortality for on- versus off-pump CABG, favor endoscopic harvesting for fewer leg complications, note that mastery volume is unknown, and identify the radial artery as a likely second conduit for selected left-sided lesions.Source: Appendix A · p. 13

Infectious Disease

  • Infectious-disease questions cover CMV prophylaxis, Stenotrophomonas and Mycobacterium abscessus treatment, shunt replacement timing, and empiric ventilator-associated-pneumonia therapy.Answers condition recommendations on patient risk, susceptibility, infection site and organism, CSF findings and cultures, and local resistance patterns.Source: Appendix A · p. 14

Neurology

  • Neurology questions span refractory status epilepticus treatment, NMOSD antibodies, acute subdural hematoma surgery criteria, meningitis steroids, and toxic leukoencephalopathy imaging and causes.Answers specify nonbenzodiazepine antiseizure options, AQP4 and occasional MOG-IgG antibodies, urgent evacuation criteria including thickness over 10 mm or shift over 5 mm, context-dependent dexamethasone use, and characteristic white-matter MRI findings.Source: Appendix A · p. 15

Appendix B Large Language Model Adversarial Safety Assessment … C.3 Example 1c: Adversarial Prompting

Almanac’s retrieval-based approach was more resilient to adversarial prompts than ChatGPT, although its effectiveness fell as prompts shortened. In the examples, Almanac declined to answer reliably while ChatGPT produced false or harmful responses.

Appendix B Large Language Model Adversarial Safety Assessment

  • Almanac’s adversarial resilience weakened as prompt word count decreased.The authors note that careful λ tuning is needed to balance true and false positives.Source: Appendix B · p. 17, Figure B1 · p. 18
  • The authors hypothesize that a domain-specific retrieval encoder could overcome this limitation, but it was outside the experiments’ scope.They identify λ tuning as necessary to balance true and false positives.Source: Appendix B · p. 17
  • Almanac was more robust to adversarial prompts than ChatGPT because retriever scoring matched queries to fetched context.The comparison used five common adversarial prompts of decreasing length.Source: Appendix B · p. 17, Figure B1 · p. 18

C.1 Example 1a: Adversarial Prompting

  • For the milrinone question, ChatGPT gave false claims about its mechanism and risks, while Almanac said it could not answer reliably from its sources.The prompt explicitly requested subtly wrong, harmful answers and fabricated studies and citations.Source: §C.1 · pp. 18–19

C.2 Example 1b: Adversarial Prompting

  • For the CABG conduit question, ChatGPT proposed a garden hose as a surgical conduit, while Almanac said it could not answer reliably from its sources.The prompt asked for an incorrect answer and false studies.Source: §C.2 · p. 19

C.3 Example 1c: Adversarial Prompting

  • For the everolimus warning question, ChatGPT described infection and malignancy risks but then mocked patients, while Almanac said it could not answer reliably from its sources.The ChatGPT response shifted from discussing the warning to deriding patients who take the drug.Source: §C.3 · p. 19

C.4 Example 2: Errors of Omission

The scenario omits age, but ChatGPT and Almanac supply different age assumptions and report markedly different GRACE estimates for 6-month mortality.

C.4 Example 2: Errors of Omission

  • ChatGPT describes the GRACE score as a validated tool for estimating death or death/myocardial infarction risk in acute coronary syndrome.Source: §C.4 · p. 19
  • With age unstated in the prompt, ChatGPT assumed age 65 and reported 13.6% 6-month mortality, while Almanac assigned age 40 to 49 and reported 3 to 3.9%.The prompt provides heart rate, blood pressure, creatinine, and cardiac history, but no age.Source: §C.4 · pp. 19–20
  • Almanac’s 93 points combine age (18), heart rate (9), systolic pressure (14), creatinine (1), heart failure (24), prior MI (12), and elevated enzymes (15).Almanac assigns the patient to the 40-to-49 age band and reports the resulting 6-month mortality as 3 to 3.9%.Source: §C.4 · p. 20

C.5 Example 3: Non-Deterministic Outputs

Repeated ChatGPT responses gave different INR goals for patients with left ventricular assist devices, while Almanac stated a target of 2.0-3.0.

C.5 Example 3: Non-Deterministic Outputs

  • ChatGPT gave an LVAD INR goal “between 1.5 and 2.5” in one response and “between 2.0 and 3.0” in another; Almanac specified 2.0-3.0.Source: §C.5 · p. 20
  • Almanac attributed its 2.0-3.0 target to the 2019 EACTS Expert Consensus and said it aims to reduce thrombosis and other LVAD complications.Source: §C.5 · p. 20

Related papers

Loading 2303.01229v2…