Source-linked AI summary
MedAlpaca -- An Open-Source Collection of Medical Conversational AI Models and Training Data
Tianyu Han, Lisa C. Adams, Jens-Michalis Papaioannou, Paul Grundmann, Tom Oberhauser, Alexei Figueroa, Alexander Löser, Daniel Truhn, Keno K. Bressem
TL;DR
Medical applications require open-source models that can be deployed locally to protect sensitive patient data. The paper presents medical instruction-tuning data and fine-tunes open-source LLMs, finding that fine-tuned models outperform pre-trained-only counterparts while parameter-efficient approximation methods reduce accuracy.
Problem
Sensitive medical data and unclear data-management practices create a need for open-source models that support on-site deployment and privacy safeguards.
Method
The study constructs medical instruction-following data, fine-tunes several open-source LLM variants using parameter-efficient methods, and evaluates them on USMLE Steps 1, 2, and 3.
Results
Fine-tuned LLMs consistently surpassed pre-trained-only counterparts, while 8-bit precision and LoRA produced less optimal results than standard fine-tuning.
Takeaways & Limitations
The dataset and parameter-efficient fine-tuning approach demonstrate a viable foundation for developing open-source medical LLMs under limited computing resources.
Takeaways & Limitations
LLMs can confabulate plausible but factually incorrect information, creating serious patient-care and safety concerns that require rigorous evaluation and continuous monitoring.
Abstract
from arXiv · showhide
As large language models (LLMs) like OpenAI's GPT series continue to make strides, we witness the emergence of artificial intelligence applications in an ever-expanding range of fields. In medicine, these LLMs hold considerable promise for improving medical workflows, diagnostics, patient care, and education. Yet, there is an urgent need for open-source models that can be deployed on-premises to safeguard patient privacy. In our work, we present an innovative dataset consisting of over 160,000 entries, specifically crafted to fine-tune LLMs for effective medical applications. We investigate the impact of fine-tuning these datasets on publicly accessible pre-trained LLMs, and subsequently, we juxtapose the performance of pre-trained-only models against the fine-tuned models concerning the examinations that future medical doctors must pass to achieve certification.
1 Introduction
Medical LLMs could support workflows, information access, and education, but privacy concerns create a need for openly deployable models. The paper introduces fine-tuned biomedical models, evaluates them on the USMLE, and publicly releases the models and datasets.
- Medical LLMs may assist with note-taking, discharge letters, information retrieval, summarization, structured extraction, consultations, and medical education.These applications depend on sufficient medical knowledge, coherence, and accuracy.
- Sensitive patient data makes opaque models with unclear data practices unsuitable for some medical applications.The paper identifies local deployment of open-source models as a way to mitigate privacy concerns and unauthorized data transfers.
- The paper presents language models fine-tuned for biomedical tasks using new and established open-source datasets converted into instruction-following format.This format supports supervised fine-tuning.
- Model effectiveness is assessed on the United States Medical Licensing Examination, a certification examination for medical students in the United States.The evaluation is intended to provide insight into model competencies and prospective medical applications.
- The authors make all models and datasets publicly available for medical and AI researchers and practitioners.
2 Materials and Methods
The study constructs Medical Meadow from diverse medical and biomedical resources, then fine-tunes LLaMA models with full and parameter-efficient procedures. Evaluation uses zero-shot USMLE Steps 1–3 self-assessment datasets after excluding image-containing questions.
- 2.1 Datasets: Medical Meadow combines reformatted established medical NLP tasks with crawled internet resources covering different aspects of medical knowledge and practice.The collection is intended to support both fine-tuning and evaluation.
- 2.1 Datasets: The dataset incorporates medical curricula, biomedical Stack Exchange question-answer pairs, WikiDoc content, and open NLP benchmarks and conversational datasets.The Stack Exchange component contains 52,475 question-answer pairs from five biomedical-science forums, while selected responses received at least five up-votes.
- 2.2 Model Training: The models use LLaMA foundation models, with 7-billion- and 13-billion-parameter variants fine-tuned in this study.
- 2.2 Model Training: Full fine-tuning trains all parameters, whereas LoRA freezes pretrained weights and adds trainable rank-decomposition matrices to reduce trainable parameters, GPU memory, and training time.
- 2.2 Model Training: Combining LoRA with 8-bit matrix multiplication and an 8-bit optimizer further reduces training memory demands.LoRA models were trained for three epochs at a learning rate of 2e-5.
3 Results
Fine-tuned LLMs outperformed their pre-trained-only counterparts on the USMLE test set, although LoRA and 8-bit fine-tuning accelerated training at the cost of reduced accuracy.
- Fine-tuned LLMs consistently surpassed pre-trained-only counterparts on the USMLE test set.
- LoRA and 8-bit fine-tuning expedited training but resulted in reduced accuracy.
4 Discussion and conclusion
The study contributes medical training data and evaluates fine-tuned open-source LLMs, while identifying computational and safety constraints for medical deployment. Performance increased with model size and fine-tuning, but approximation methods and limited optimization reduced or may limit results.
- The study introduced a high-quality medical text collection designed to train instruction-following medical LLMs.
- Several open-source LLM variants were fine-tuned with parameter-efficient methods because full fine-tuning is often infeasible for academic institutions.
- Performance improved with larger pre-trained models, whereas 8-bit precision and LoRa produced less optimal fine-tuning results.
- Potential applications include structured information extraction, medical-student question answering, patient health communication, and assistance with clinical documentation.
- Medical deployment requires local handling of sensitive data and safeguards against bias, inaccuracies, and factually incorrect confabulations.
6 Appendix
The appendix presents representative questions from several medical and biomedical information sources, alongside unrelated illustrative questions and answers. These examples span medical flashcards, Stack Exchange, WikiDoc, pharmacology, death determination, exercise, and light wavelengths.
- Table 3 presents a representative question from the medical flashcards dataset.
- Table 4 presents a representative question from the Stack Exchange dataset.
- The appendix also includes discussions of treatment trade-offs involving severe side effects, including leprosy and multiple myeloma.
- Additional examples address how death is determined, home exercises for the lower back, and the wavelengths of light used by organisms.
- Table 5 presents a representative question from the WikiDoc Living Textbook and Patient Information.