PaperlayerView exact revision on arXiv

Source-linked AI summary

Med42-v2: A Suite of Clinical LLMs

Clément Christophe, Praveen K Kanithi, Tathagata Raha, Shadab Khan, Marco AF Pimentel

arXiv:2408.06142v1cs.CLcs.AI

TL;DR

Generic models have limitations in healthcare, including hallucinations, biases, and knowledge gaps. Med42-v2 combines specialized clinical fine-tuning with multi-stage preference alignment, and the paper reports that it outperforms original Llama3 models and GPT-4 across medical benchmarks.

  • Problem

    Generic models have healthcare limitations, including hallucinations, biases, knowledge gaps, and privacy and ethics risks.

    Source: §1 · p. 1

  • Method

    Med42-v2 is built on Llama3, fine-tuned with specialized clinical data, and trained through multi-stage preference alignment using open-access AI-feedback preference datasets.

    Source: §1 · pp. 1–2, §2.2 · p. 3

  • Results

    Med42-v2 outperforms original Llama3 models in both 8B and 70B configurations and GPT-4 across various medical benchmarks.

    Source: §4 · p. 4

  • Takeaways & Limitations

    The suite is designed to handle clinical queries and support clinical understanding and reasoning tasks.

    Source: §1 · p. 1

    Med42-v2 may retain hallucinations, biases, and ethical concerns, and training-data gaps or biases could affect its effectiveness.

    Source: §4 · pp. 4–5

Abstract

from arXiv · show

Med42-v2 introduces a suite of clinical large language models (LLMs) designed to address the limitations of generic models in healthcare settings. These models are built on Llama3 architecture and fine-tuned using specialized clinical data. They underwent multi-stage preference alignment to effectively respond to natural prompts. While generic models are often preference-aligned to avoid answering clinical queries as a precaution, Med42-v2 is specifically trained to overcome this limitation, enabling its use in clinical settings. Med42-v2 models demonstrate superior performance compared to the original Llama3 models in both 8B and 70B parameter configurations and GPT-4 across various medical benchmarks. These LLMs are developed to understand clinical queries, perform reasoning tasks, and provide valuable assistance in clinical environments. The models are now publicly available at \href{https://huggingface.co/m42-health}{https://huggingface.co/m42-health}.

1 INTRODUCTION

Generic models face healthcare-specific limitations, while Med42-v2 is designed to engage with clinical queries through specialized training and preference alignment. The paper reports better performance than original Llama3 models across medical benchmarks.

1 INTRODUCTION

  • Generic models face hallucination, bias, knowledge-gap, privacy, and ethics concerns that limit their effectiveness across healthcare tasks.The passage names diagnostic support, medical-literature interpretation, patient education, and clinical guidelines and decision support.Source: §1 · p. 1
  • Med42-v2 combines the Llama3 architecture and specialized clinical data with multi-stage preference alignment to respond to natural prompts.Unlike generic models often aligned to avoid clinical queries, Med42-v2 is specifically trained to engage with them.Source: §1 · pp. 1–2
  • Med42-v2 outperforms original Llama3 models in both 8B and 70B configurations across various medical benchmarks.The supplied Table 1 passage identifies an overview of Med42 and Med42-v2 models, not benchmark results.Source: §1 · pp. 1–2, Table 1 · p. 2

2 METHOD

Med42-v2 is developed through clinical instruction fine-tuning followed by preference alignment. Its training data combine clinical and general-domain material, while alignment uses AI-feedback preference datasets and iterative optimization.

2 METHOD

  • The two-stage process builds on Llama3 and Llama3.1, first fine-tuning on curated clinical datasets and then aligning outputs with human preferences.The alignment stage is intended to support instruction-following while safeguarding against unethical or biased behavior.Source: §2 · p. 2

2.1 CLINICAL FINE-TUNING STAGE

  • Clinical fine-tuning uses medical and biomedical resources, chain-of-thought and chat examples, and general-domain data comprising 26.5% of the final dataset.The general-domain subset is included to support generalizability and linguistic understanding.Source: §2.1 · p. 2

2.2 PREFERENCE-ALIGNMENT STAGE

  • Preference alignment uses UltraFeedback and Snorkel-DPO datasets, with the latter built through iterative generation and reranking of responses to selected prompts.Five Mistral-7B-Instruct-v0.23 responses per prompt are reranked with PairRM across three iterations of 20,000 prompts each.Source: §2.2 · p. 3
  • Direct Preference Optimization aligns clinically fine-tuned checkpoints, and successive iterations use the prior iteration’s model as a reward model.The authors chose DPO over more complex reinforcement-learning algorithms for stability and scalability.Source: §2.2 · p. 3

3 BENCHMARKS

The evaluation measures zero-shot performance on English medical benchmarks, with image-containing questions excluded. Med42 models show improvements over earlier versions and strong comparisons with other models, but results are bounded by the zero-shot setup.

3 BENCHMARKS

  • Evaluation uses zero-shot performance on medical benchmarks with multiple-choice answers; all datasets are English and image-containing questions are excluded.The harness evaluates log-likelihood over the entire response sequence.Source: §3 · p. 4
  • Med42 improves over its previous iteration across multiple benchmarks, and 70B models outperform leading general-purpose and domain-specific models, including GPT-4.0 on all datasets.The supplied passage reports this comparison in Table 4.Source: §3 · p. 4
  • The reported benchmark results are zero-shot; prompting techniques or search integration can yield higher accuracy, with Med-Gemini reported at 91.2% on MedQA.The authors identify detailed evaluation of conversational capabilities in clinical settings as future work.Source: §3 · p. 4

4 CONCLUSIONS AND LIMITATIONS

Med42-v2 combines Llama3-based clinical models fine-tuned on specialized clinical data with multi-stage preference alignment, and outperforms original Llama3 models and GPT-4 across medical benchmarks. The authors note unresolved clinical risks and data-quality constraints, and plan real-world evaluation of clinical utility, safety, and reasoning.

4 CONCLUSIONS AND LIMITATIONS

  • Med42-v2 outperforms original Llama3 models in both 8B and 70B configurations and GPT-4 across various medical benchmarks.Table 4 covers zero-shot evaluations on closed-ended medical benchmarks.Source: §4 · p. 4, Table 4 · p. 5
  • Med42-v2 may still have hallucinations, biases, and ethical concerns, which are particularly critical in medicine.Source: §4 · pp. 4–5
  • Gaps or biases in high-quality, domain-specific training data could affect model effectiveness; future work proposes real-world evaluation of clinical data understanding, safety, and reasoning.The planned framework is intended to assess clinical utility in practical, high-stakes settings.Source: §4 · p. 5

A APPENDIX

The appendix describes fine-tuning data composition, including its medical-domain share and selection criteria for some included subsets.

A APPENDIX

  • The included categories for one subset are academia, bioinformatics, biology, cognitive science, fitness, and health.Source: A APPENDIX · p. 10
  • Only English samples were used, and the listed subject subset spans medical, biological, and psychology topics.The listed subjects include anatomy, clinical knowledge, college medicine, nutrition, medical genetics, professional medicine, college biology, high-school biology, virology, high-school psychology, human sexuality, and human aging.Source: A APPENDIX · p. 10
  • The appendix includes samples from 47 of more than 1,000 tasks related to science, healthcare, and medicine.Source: A APPENDIX · p. 10
  • Medical-domain data account for 73.5% of the fine-tuning dataset.Source: Table 5 · p. 10

Related papers

Loading 2408.06142v1…