PaperlayerView exact revision on arXiv

Source-linked AI summary

PMC-LLaMA: Towards Building Open-source Language Models for Medicine

Chaoyi Wu, Weixiong Lin, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, Weidi Xie

arXiv:2304.14454v3cs.CL

TL;DR

Medical LLMs can struggle with precision, and the paper identifies a need for systematic open-source adaptation to the domain. It builds PMC-LLaMA through medical knowledge injection and instruction tuning, and reports higher average medical-QA accuracy than competing models, including ChatGPT.

  • Problem

    General LLMs can produce seemingly accurate but incorrect medical conclusions, and systematic open-source medical adaptation remains insufficiently examined.

    Source: Introduction · p. 1

  • Method

    The authors adapt LLaMA through medical knowledge injection using papers and textbooks, followed by medical-specific instruction tuning.

    Source: Introduction · p. 2, Dataset Construction · p. 3

  • Results

    PMC-LLaMA improves average accuracy from 54.97% to 64.43% on medical QA benchmarks and surpasses ChatGPT and LLaMA-2 with fewer parameters.

    Source: Results · p. 6, Conclusion · p. 6

  • Takeaways & Limitations

    The authors release PMC-LLaMA as an open-source, lightweight medical-specific language model.

    Source: Introduction · p. 2

    The comparison with commercial models is not claimed to be completely fair because their training data and architecture are undisclosed.

    Source: Training Details · p. 5

Abstract

from arXiv · show

Recently, Large Language Models (LLMs) have showcased remarkable capabilities in natural language understanding. While demonstrating proficiency in everyday conversations and question-answering situations, these models frequently struggle in domains that require precision, such as medical applications, due to their lack of domain-specific knowledge. In this paper, we describe the procedure for building a powerful, open-source language model specifically designed for medicine applications, termed as PMC-LLaMA. Our contributions are threefold: (i) we systematically investigate the process of adapting a general-purpose foundation language model towards medical domain, this involves data-centric knowledge injection through the integration of 4.8M biomedical academic papers and 30K medical textbooks, as well as comprehensive fine-tuning for alignment with domain-specific instructions; (ii) we contribute a large-scale, comprehensive dataset for instruction tuning. This dataset encompasses medical question-answering (QA), rationale for reasoning, and conversational dialogues, comprising a total of 202M tokens; (iii) we conduct thorough ablation studies to demonstrate the effectiveness of each proposed component. While evaluating on various public medical question-answering benchmarks, our lightweight PMCLLaMA, which consists of only 13 billion parameters, exhibits superior performance, even surpassing ChatGPT. All models, codes, datasets can be found in https://github.com/chaoyi-wu/PMC-LLaMA.

Introduction

General-purpose LLMs can give seemingly accurate but incorrect medical conclusions; the paper attributes this in part to gaps in comprehensive medical knowledge. It introduces a systematic open-source adaptation procedure and reports that PMC-LLaMA surpasses ChatGPT and LLaMA-2 on medical QA benchmarks.

Introduction

  • Existing medical adaptations had not resolved the need for systematic open-source investigation, while Med-PaLM-2's training data and architecture remained unclear.The paper identifies Med-PaLM-2 as the only cited work then reported to outperform ChatGPT.Source: Introduction · p. 1
  • The authors propose adapting LLaMA through medical knowledge injection, reasoning augmentation, and instruction alignment for diverse zero-shot tasks.They describe knowledge injection as building medical terminology representations and reasoning augmentation as linking knowledge with case information.Source: Introduction · p. 1
  • PMC-LLaMA surpasses ChatGPT and LLaMA-2 on various medical QA benchmarks after medical knowledge injection and instruction tuning.The paper describes the model as lightweight and open source.Source: Introduction · p. 2
  • MedC-K combines 4.8M biomedical papers and 30K textbooks, while MedC-I provides medical QA, rationale, and conversational instruction data totaling 202M tokens.The paper reports evaluation on various medical QA benchmarks.Source: Introduction · p. 2

Related Work

Prior work includes general-purpose open models, medical instruction-tuned systems, and medical foundation models. The paper positions its approach as adapting a general LLM to address limitations in medical knowledge integration and downstream utility.

Large Language Model

  • Many biomedical models were trained only on small medical corpora, limiting comprehensive medical knowledge integration.The related-work discussion also notes that GPT-series training methods and weights were undisclosed.Source: Related Work · p. 2

Instruction Tuning

  • Instruction tuning supports zero- and few-shot generalization, and medical systems such as Chat-Doctor and Med-Alpaca target QA and dialogue.The related-work section notes that Med-PaLM's code and data were not publicly accessible.Source: Related Work · p. 2

Medical Foundational Language Model

  • Medical foundation models trained exclusively on medical corpora may lack exposure to diverse knowledge domains, while BERT-based scale can restrict downstream zero-shot utility.The paper proposes adapting a general LLM with knowledge injection followed by medical-specific instruction tuning to address these limitations.Source: Related Work · p. 2

Problem Formulation

The paper formulates medical adaptation as two training stages: autoregressive knowledge injection followed by instruction tuning. Their loss differs in whether it is computed over the whole token sequence or only the response.

Problem Formulation

  • The training objective minimizes autoregressive loss, with loss computed either over the full token sequence or only a response subsequence.The input is treated as a sequence of tokens, and the distinction depends on the training stage.Source: Problem Formulation · p. 2
  • Knowledge injection minimizes the default autoregressive loss over free-form medical text to accumulate medical knowledge contexts.The model is denoted by Φ, and u<i refers to tokens before index i.Source: Problem Formulation · p. 2
  • Instruction tuning splits tokens into instruction I and response R, ignores instruction tokens in the loss, and trains the model to answer user queries.At inference, the user's question serves as the instruction and the model output is the answer.Source: Problem Formulation · p. 2

Dataset Construction

Dataset construction pairs data-centric medical knowledge injection with instruction tuning for medical use. It builds a medical corpus from academic papers and textbooks, then assembles instruction data spanning conversations, rationale QA, and knowledge-graph tasks.

Dataset-I: Fundamental Medical Knowledge

  • MedC-K combines about 4.8M biomedical papers totaling over 75B tokens with 30K textbooks containing approximately 4B tokens.The papers are selected from S2ORC using corresponding PubMed Central IDs; textbook text is cleaned before use.Source: Dataset Construction · p. 3

Combination

  • Knowledge-injection batches use a 15:4:1 ratio of book, paper, and general RedPajama tokens, emphasizing books while retaining general text to avert catastrophic forgetting.Books provide more fundamental medical knowledge, while papers predominantly capture cutting-edge insights.Source: Dataset Construction · p. 3

Knowledge Injection Training

  • Autoregressive training on MedC-K injects medical knowledge into LLaMA, producing the intermediate model PMC-LLaMAK.The largest share of training tokens comes from PubMed Central academic papers.Source: Dataset Construction · p. 3

Medical Conversation

  • Conversation data from Med-Alpaca and ChatDoctor is expanded into synonymous instructions, with random instruction selection intended to simulate user inputs and avoid template overfitting.The expansion query uses GPT-4, while training randomly selects from the instruction base.Source: Dataset Construction · p. 4

Medical Rationale QA

  • Medical rationale QA supplements plain-choice USMLE, PubMedQA, and MedMCQA examples with ChatGPT-generated causality-analysis explanations.The rationale data addresses the absence of detailed reasoning guidance in most source datasets.Source: Dataset Construction · p. 4

Medical Knowledge Graph Prompting

  • Knowledge-graph prompts convert UMLS knowledge into QA pairs that ask for entity descriptions or relationships between entities.The prompts align medical terminology with descriptions or corresponding relationships.Source: Dataset Construction · p. 4

Medical-Specific Instruction Tuning

  • MedC-I combines medical conversation, rationale QA, and knowledge-graph prompting into 202M tokens, then tunes PMC-LLaMAK to produce PMC-LLaMA.Knowledge-graph prompts target entity descriptions and relationships.Source: Dataset Construction · p. 4

Experiment

The experiment evaluates medical QA performance using three benchmarks and both task-specific fine-tuning and zero-shot instruction settings. The authors caution that comparisons with commercial models are not fully fair because details such as training data and architecture are undisclosed.

Training

  • Knowledge injection ran for 5 epochs on 32 A100 GPUs; instruction tuning ran for 3 epochs on 8 A100 GPUs.The knowledge-injection epoch is defined by seeing all book tokens, while an instruction-tuning epoch loops through all sequences.Source: Training Details · p. 4

Benchmarks

  • Evaluation used PubMedQA (Jin et al. 2019), MedMCQA (Pal, Umapathi et al. 2022), and USMLE (Jin, Pan et al. 2021), following accuracy as the conventional main metric.PubMedQA uses PQA-A for training and PQA-L for testing; the USMLE evaluation uses English questions.Source: Training Details · pp. 4–5
  • PubMedQA used PQA-A/PQA-L, MedMCQA's test split had 4,183 questions, and English USMLE followed official 10,178/1,273/1,273 train/validation/test splits.MedMCQA's training split contains 182,822 questions.Source: Training Details · p. 5

Baseline Models

  • The comparison baselines span LLaMA, LLaMA-2, ChatGPT, Med-Alpaca, and Chat-Doctor.The paper treats comparisons as references rather than completely fair because commercial-model training details, including data and architecture, remain undisclosed.Source: Training Details · p. 5

Evaluation Settings

  • Task-specific evaluation fine-tunes on three QA training sets, whereas zero-shot evaluation tests an instruction directly without task-specific fine-tuning.LLaMA and PMC-LLaMAK use the former by default; LLaMA-2, MedAlpaca, Chat-Doctor, ChatGPT, and PMC-LLaMA use the latter.Source: Training Details · p. 5

Results

Across QA benchmarks, PMC-LLaMA outperformed most compared models and exceeded ChatGPT on average accuracy; ablations report gains associated with scaling, medical corpora, and instruction-tuning choices.

Model scale

  • Increasing model size from 7B to 13B improved performance on all benchmarks for both baseline LLaMA and medically pretrained PMC-LLaMAK.The same scaling pattern held for both model variants.Source: Results · p. 5, Table 1 · p. 6

Data-centric knowledge injection

  • Biomedical papers raised accuracy from 44.54% to 44.70% on MedQA and from 48.51% to 50.54% on MedMCQA; adding books yielded gains of 1.02%, 2.94%, and 1.2% on MedQA, MedMCQA, and PubMedQA.The ablation compares medical knowledge-injection choices against the 7B LLaMA baseline.Source: Results · p. 5, Table 1 · p. 6

Comparison with Baselines

  • Average accuracy rose from 54.97% to 64.43%, with PMC-LLaMA outperforming most compared models and surpassing ChatGPT despite having significantly fewer parameters.The average is across three QA datasets.Source: Table 2 · p. 6, Results · p. 6

Medical-specific instruction tuning

  • On MedQA, task-specific fine-tuning on plain-choice data improved performance by 1.17%; adding rationale conversations for instruction tuning raised performance from 49.32% to 54.43%, while a knowledge graph added 1.93%.Source: Results · p. 6, Table 1 · p. 6

Zero-shot Case Study

  • In case studies, PMC-LLaMA gave specific recurrent-UTI advice and fuller microbiology rationales; both models handled Rifampin, while the UTI query postdated data collection.For the UTI case, PMC-LLaMA suggested a sensitivity test, whereas ChatGPT gave a general suggestion to investigate underlying causes.Source: Results · p. 6

Supplementary

The supplementary material details prompts for generating medical rationales and presents broader zero-shot QA and conversation examples. These examples show thorough option analyses alongside limitations in free-form conversation, particularly beyond medical topics.

Supplementary Materials

Prompt ChatGPT for Rationale QA

  • Rationale-QA prompts ask ChatGPT to generate explanations from question-answer pairs, using general-wise or option-wise response formats.The option-wise format requests separate true-or-false analyses for each answer choice, with examples intended to guide generation.Source: Supplementary · p. 9, Optional-wise Rationale Prompts · p. 9

QA Rationales

  • Across diverse medical branches, PMC-LLaMA and ChatGPT show commensurate QA abilities, while PMC-LLaMA tends to explain each option more thoroughly.The supplementary examples include zero-shot QA rationales, with correct options marked in bold.Source: Optional-wise Rationale Prompts · p. 9, Figure 5 · p. 10

Conversations

  • PMC-LLaMA is less capable in free-form conversations, especially when the topic leaves the medical domain.Examples include potentially overestimating a minor symptom or recommending medication where daily practice would be preferable; ChatGPT gives wiser advice in another case.Source: Optional-wise Rationale Prompts · p. 9

Zero-shot Samples

Loading 2304.14454v3…