PaperlayerView exact revision on arXiv

Source-linked AI summary

AlpaCare:Instruction-tuned Large Language Models for Medical Application

Xinlu Zhang, Chenxin Tian, Xianjun Yang, Lichang Chen, Zekun Li, Linda Ruth Petzold

arXiv:2310.14558v6cs.CLcs.AI

TL;DR

Medical LLM datasets often cover limited tasks or topics, leaving diverse medical instruction-following needs underexplored. The paper builds MedInstruct-52k from clinician-curated seeds using GPT-4 and ChatGPT, then fine-tunes LLaMA models. AlpaCare reports gains in medical and general evaluations, though teacher-generated medical knowledge may hallucinate.

  • Problem

    Medical instruction-tuning datasets often rely on benchmarks or narrow topics, limiting their coverage of diverse user inquiries and model generalizability.

    Source: §1 · pp. 1–2

  • Method

    The authors use GPT-4 and ChatGPT with clinician-curated seed tasks to create MedInstruct-52k, then fine-tune LLaMA models to develop AlpaCare.

    Source: §1 · p. 2

  • Results

    AlpaCare gains 38.1% over the best baselines in medical free-form instruction evaluations and 6.7% averaged over multiple general domain benchmarks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Experiments support stronger medical and general instruction-following capabilities for AlpaCare trained on the diverse dataset.

    Source: §8 · p. 9

    Teacher LLMs used to generate medical instruction-response pairs may hallucinate medical knowledge.

    Source: LIMITATIONS · p. 14

Abstract

from arXiv · show

Instruction-finetuning (IFT) has become crucial in aligning Large Language Models (LLMs) with diverse human needs and has shown great potential in medical applications. However, previous studies mainly fine-tune LLMs on biomedical datasets with limited diversity, which often rely on benchmarks or narrow task scopes, and hence significantly limit the effectiveness on their medical instruction-following ability and generalizability. To bridge this gap, we propose creating a diverse, machine-generated medical IFT dataset, MedInstruct-52k, using GPT-4 and ChatGPT with a high-quality expert-curated seed set. We then fine-tune LLaMA-series models on the dataset to develop AlpaCare. Despite using a smaller domain-specific dataset than previous medical LLMs, AlpaCare not only demonstrates superior performance on medical applications, with up to 38.1% absolute gain over best baselines in medical free-form instruction evaluations, but also achieves 6.7% absolute gains averaged over multiple general domain benchmarks. Human evaluation further shows that AlpaCare consistently outperforms best baselines in terms of both correctness and helpfulness. We offer public access to our data, model, and codebase in https://github.com/XZhang97666/AlpaCare.

1 INTRODUCTION

Medical instruction-tuning datasets often cover narrow tasks or topics, leaving diverse user inquiries underexplored. The paper builds MedInstruct-52k from clinician-curated seeds and reports gains in medical and general evaluations.

1 INTRODUCTION

  • Existing medical LLM datasets often focus on benchmarks or narrow topics, limiting diversity for medical instruction-following and generalizability.The paper identifies a need for diverse medical instruction data that addresses varied domain-specific inquiries while balancing cost.Source: §1 · pp. 1–2
  • The authors create MedInstruct-52k using GPT-4 and ChatGPT, guided by 167 clinician-curated tasks spanning medical topics, viewpoints, task types, and difficulty levels.GPT-4 generates new instructions from clinician-curated demonstrations, and ChatGPT supplies responses for valid tasks.Source: §1 · p. 2
  • The authors introduce MedInstruct-test, a clinician-crafted free-form evaluation set covering medical tasks at different difficulty levels.The set contains 216 clinician-crafted medical tasks.Source: §1 · p. 2
  • 38.1% absolute gain over the best baselines in medical free-form instruction evaluations, alongside a 6.7% absolute gain averaged over multiple general domain benchmarks.Human evaluation also found gains of +12% in correctness and +49% in helpfulness over the best baselines.Source: §1 · p. 2
  • The paper reports that tuning on diverse medical instruction data improves medical applications and generalization simultaneously, and releases MedInstruct-52K and MedInstruct-test.The released test set contains 216 clinician-crafted tasks.Source: §1 · p. 2

2 RELATED WORKS

Prior instruction-tuning work explores benchmark-derived, real-world, and teacher-generated data, while medical LLMs commonly use biomedical datasets with narrow task or topic coverage. The paper’s pipeline illustrates its approach to generating diverse medical instruction data.

2 RELATED WORKS

  • Instruction-tuning approaches range from benchmark-derived instructions to real-world examples and data distilled from stronger teacher models.The related work contrasts simpler benchmark-derived instructions with diverse real-world instruction-response data and machine-generated datasets.Source: §2 · pp. 2–3
  • The AlpaCare pipeline uses GPT-4 to generate task instructions from clinician-curated seeds, filters similar instructions, uses ChatGPT for responses, and tunes LLaMA models.The resulting instruction-response pairs form MedInstruct-52K.Source: Figure 2 · p. 3
  • Medical LLM datasets have often focused on benchmarks or specific topics, leaving diverse medical user intentions underrepresented.Examples include doctor-patient dialogues, medical question-answer pairs, and biomedical papers or textbooks used for continued training.Source: §2 · p. 3

3 METHOD

AlpaCare’s method starts with clinician-curated medical tasks, expands them into machine-generated instruction-response pairs, and uses those pairs for supervised tuning of LLaMA models.

3 METHOD

  • The pipeline begins with 167 clinician-curated seed tasks; similar generated instructions are removed, leaving 52k instances for response generation and model tuning.The resulting MedInstruct-52k pairs are used to tune LLaMA models.Source: §3 · pp. 3–4

CLINICIAN-CURATED SEED DATASET

  • Seed tasks span medical topic, viewpoint, task type, and difficulty, with difficulty levels ranging from 1 to 5.Task formats include summarization, rewriting, and single-hop or multi-hop reasoning; viewpoints include researchers, medical students, and patients.Source: CLINICIAN-CURATED SEED DATASET · p. 4

MEDICAL IFT DATASET GENERATION AND LLM TUNING

  • GPT-4 generates 12 tasks per run from three randomly selected seed tasks; instructions above 0.7 Rouge-L similarity are discarded before ChatGPT generates responses.A clinician graded 49 of 50 randomly selected generated responses as correct.Source: MEDICAL IFT DATASET GENERATION AND LLM… · p. 4
  • LLaMA models are fine-tuned using instructions and available inputs to generate corresponding responses with standard supervised fine-tuning and cross-entropy loss.The training prompt and hyperparameter setup follow Taori et al. (2023).Source: MEDICAL IFT DATASET GENERATION AND LLM… · p. 4

4 EXPERIMENTAL SETUP

The experiments evaluate medical instruction following, medical benchmark performance, and general-domain capabilities. They compare AlpaCare with general and medical LLM baselines using distinct datasets and evaluation procedures.

FREE-FORM INSTRUCTION EVALUATION

  • Free-form medical instruction evaluations use iCliniq patient-doctor conversations and MedInstruct-test, which contains 216 clinician-crafted tasks across difficulty levels.GPT-3.5-turbo pairwise judges compare model outputs with references from four APIs; comparisons are repeated with output order alternated, and win rates are reported.Source: FREE-FORM INSTRUCTION EVALUATION · p. 5

BENCHMARK EVALUATION

  • Medical benchmark evaluation covers MedQA, HeadQA, PubmedQA, MedMCQA, and MeQSum, reporting accuracy for multiple-choice tasks and ROUGE-L for summarization.Source: BENCHMARK EVALUATION · p. 5

BASELINES

  • Baselines include Alpaca, ChatDoctor, MedAlpaca, PMC-LLaMA, and Baize-Healthcare, while general-domain evaluation covers AlpacaFarm, MMLU, BBH, and TruthfulQA.The general-domain table caption identifies AlpacaFarm as free-form instruction evaluation, MMLU and BBH as knowledge benchmarks, and TruthfulQA as a truthfulness task.Source: BASELINES · p. 5, Table 3 · p. 6

5 EXPERIMENT RESULTS

AlpaCare shows strong medical free-form instruction performance and generalizability, with further tests finding advantages across model sizes, backbones, and judges.

MAIN RESULTS

  • 130% on iCliniq and 90% on MedInstruct are AlpaCare’s relative gains over the best baselines, averaged across reference models.AlpaCare also outperforms Alpaca and medical models trained on considerably larger datasets across reference LLMs.Source: MAIN RESULTS · p. 6

GENERALIZABILITY EVALUATION

  • 57.8% relative improvement on AlpacaFarm over the best baseline accompanies higher general-knowledge scores and comparable truthfulness scores.Medical LLMs often perform worse than or comparably to Alpaca on generalizability.Source: GENERALIZABILITY EVALUATION · p. 6

ABLATION STUDY

  • AlpaCare-13B consistently outperforms other 13B models in both tasks; it was fine-tuned on LLaMA-13B and compared against other 13B LLMs.Source: ABLATION STUDY · p. 7
  • AlpaCare-LLaMA2/3 consistently and significantly outperforms Alpaca-LLaMA2/3 on both datasets, supporting performance across backbone families.The comparisons include LLaMA2-7B and LLaMA3-8B.Source: ABLATION STUDY · p. 7
  • Claude-2 evaluation finds that AlpaCare consistently outperforms its IFT baselines by a large margin, aligning with GPT-3.5-turbo judgments.Claude-2 was introduced as an alternative judge to address potential biases from relying on one judge family.Source: ABLATION STUDY · p. 7

6 HUMAN STUDY

In a head-to-head clinician study, AlpaCare-13B was preferred to PMC-13B for both correctness and helpfulness.

HUMAN STUDY

  • 54% of answers were preferred for correctness and 69% for helpfulness for AlpaCare-13B over PMC-13B.Annotators used majority vote; comparisons without agreement by at least two annotators were treated as ties.Source: Figure 3 · p. 7, §6 · p. 8

7 ANALYSIS & CASE STUDY

MedInstruct-52k exhibits diverse medical intents and high linguistic entropy; case studies illustrate specific medical guidance and instruction-following strengths.

IFT DATASET DIVERSITY ANALYSIS

  • The dataset analysis displays the top 20 topics, views, and types, difficulty levels, and common verb–object pairs representing 22% of instructions.The authors report diverse medical intents and textual formats in MedInstruct-52k.Source: IFT DATASET DIVERSITY ANALYSIS · p. 8
  • MedInstruct-52k has the highest linguistic entropy among the compared medical instruction datasets, with higher entropy indicating greater diversity.The analysis uses the top 20 root verbs and their four primary direct noun objects for each dataset.Source: IFT DATASET DIVERSITY ANALYSIS · p. 8, Table 7 · p. 8
  • In a high-difficulty correctness case, AlpaCare gives more specific conditions and actionable guidance than PMC’s general overview.The example identifies hypothyroidism and iron deficiency anemia and recommends medical attention for severe symptoms.Source: IFT DATASET DIVERSITY ANALYSIS · p. 9, Figure 5 · p. 9
  • In a Metoprolol helpfulness case, AlpaCare follows the requested bullet-point format while LLAMA and PMC do not.Both the base and medical models accurately describe Metoprolol, but their disregard of the requested format reduces clarity.Source: IFT DATASET DIVERSITY ANALYSIS · p. 9, Figure 5 · p. 9

8 CONCLUSION

The paper presents a GPT-4 and ChatGPT pipeline for creating MedInstruct-52k and reports stronger medical and general instruction-following than medical LLM baselines.

CONCLUSION

  • The proposed semi-automated GPT-4 and ChatGPT pipeline creates MedInstruct-52k, which supports AlpaCare’s stronger medical and general instruction-following performance than medical LLM baselines.The conclusion attributes the results to training with the diverse dataset.Source: §8 · p. 9

LIMITATIONS

The approach uses teacher LLMs to generate medical instruction-response data, treating them as medical knowledge bases. The authors identify hallucinated medical knowledge as a risk and propose internet and knowledge-graph integration as future work.

  • Teacher LLMs such as GPT-4 and ChatGPT generate medical instruction-response datasets and serve as medical knowledge bases.Source: LIMITATIONS · p. 14
  • The authors warn that this generation approach could produce hallucinations in medical knowledge.Source: LIMITATIONS · p. 14
  • Future work aims to improve generation reliability by integrating LLMs with the internet and knowledge graphs.Source: LIMITATIONS · p. 14

ETHICS STATEMENT

The paper cautions that AlpaCare can produce incorrect answers and should complement, not replace, medical professionals. Its appendices describe the released resources, difficulty scoring, and prompts used to generate tasks and responses.

ETHICS STATEMENT

  • AlpaCare has comparable truthfulness scores to baselines, but its correct-answer rate remains low; licensed practitioners must validate outputs before practical use.The authors frame it as an assistant tool that complements rather than replaces professional expertise and judgment.Source: ETHICS STATEMENT · p. 14
  • Practical medical use requires further research on safety, privacy, and reliability, including quality assessment in clinical environments and protection of patient information.Source: ETHICS STATEMENT · p. 14

A DATA AND CODE

  • The authors provide the codebase, MedInstruct-52k, MedInstruct-test, and AlpaCare models.Source: A DATA AND CODE · p. 14

B MEDICAL TASK DIFFICULTY LEVEL SCORING SYSTEM

C PROMPT DETAILS FOR MedInstruct-52k GENERATION

D LLM APIS COST ANALYSIS AND SELECTION REASONS

The appendix reports API-generation costs, training and test-set materials, and evaluation setups. Its analysis states that AlpaCare’s stronger performance extends beyond medical tasks to broader general-domain tasks.

D LLM APIS COST ANALYSIS AND SELECTION REASONS

  • $900 was spent on GPT-4 task generation and $500 on GPT-3.5-Turbo response generation; a clinician judged 49 of 50 sampled responses correct.The authors selected GPT-3.5-Turbo for responses to balance cost and effectiveness; the passage states GPT-4 API cost was 30 times higher at the time.Source: D LLM APIS COST ANALYSIS AND SELECTION… · p. 17

E TRAINING HYPERPARAMETER DETAILS

F MedInstruct-test STATISTICS

GENERAL DOMAIN FREE-FORM INSTRUCTION EVALUATION

  • Table 13 compares AlpaCare and baselines on AlpacaFarm using four reference models, with AVG as the mean score across them.Source: Table 13 · p. 18

MORE ANALYSIS IN GENERAL DOMAIN PERFORMANCE

  • AlpaCare’s enhanced performance extends beyond the medical domain to broader general-domain tasks, supporting the authors’ claim of superior generalizability.The accompanying analysis discusses Alpaca comparisons and additional StrategyQA and DROP evaluations.Source: MORE ANALYSIS IN GENERAL DOMAIN… · p. 18, Table 14 · p. 18

Related papers

Loading 2310.14558v6…