PaperlayerView exact revision on arXiv

Source-linked AI summary

BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domains

Yanis Labrak, Adrien Bazoge, Emmanuel Morin, Pierre-Antoine Gourraud, Mickael Rouvier, Richard Dufour

arXiv:2402.10373v3cs.CLcs.AIcs.LG

TL;DR

Open-source medical LLM adoption is limited by privacy concerns and a lack of lightweight models with performance comparable to larger or proprietary systems. BioMistral addresses this gap by further pre-training Mistral 7B Instruct on PubMed Central and evaluating it across multilingual medical QA, quantization, and model-merging settings. It demonstrates state-of-the-art performance among open-source 7B models, while retaining weaknesses on PubMedQA and unfamiliar or non-English medical content.

  • Problem

    Open-source medical LLM adoption is limited by privacy concerns and a lack of lightweight models with performance comparable to larger or proprietary systems, while biomedical evaluation beyond English remains relatively understudied.

  • Method

    BioMistral further pre-trains Mistral 7B Instruct on PubMed Central, evaluates it on 10 English medical QA tasks translated into 7 languages, and studies supervised fine-tuning, quantization, and model merging.

  • Results

    BioMistral 7B shows state-of-the-art performance among open-source 7B models, with gains over prior biomedical models on MMLU and MedMCQA but lower PubMedQA accuracy than other models.

  • Takeaways & Limitations

    The released models, multilingual benchmarks, datasets, and scripts provide open resources for evaluating and adapting biomedical LLMs, including lightweight quantized and merged variants.

  • Takeaways & Limitations

    BioMistral 7B may struggle with unfamiliar or rare medical procedures and terminology, and its English-language training causes degraded non-English performance with occasional erroneous predictions.

Abstract

from arXiv · show

Large Language Models (LLMs) have demonstrated remarkable versatility in recent years, offering potential applications across specialized domains such as healthcare and medicine. Despite the availability of various open-source LLMs tailored for health contexts, adapting general-purpose LLMs to the medical domain presents significant challenges. In this paper, we introduce BioMistral, an open-source LLM tailored for the biomedical domain, utilizing Mistral as its foundation model and further pre-trained on PubMed Central. We conduct a comprehensive evaluation of BioMistral on a benchmark comprising 10 established medical question-answering (QA) tasks in English. We also explore lightweight models obtained through quantization and model merging approaches. Our results demonstrate BioMistral's superior performance compared to existing open-source medical models and its competitive edge against proprietary counterparts. Finally, to address the limited availability of data beyond English and to assess the multilingual generalization of medical LLMs, we automatically translated and evaluated this benchmark into 7 other languages. This marks the first large-scale multilingual evaluation of LLMs in the medical domain. Datasets, multilingual evaluation benchmarks, scripts, and all the models obtained during our experiments are freely released.

1 Introduction

BioMistral addresses limited adoption of open-source medical LLMs by adapting Mistral 7B to biomedical text and evaluating lightweight, multilingual, and merged variants. The work introduces a broad evaluation and releases its resulting resources.

  • Open-source medical LLM adoption is limited by the lack of lightweight, commercially usable models with performance comparable to larger or proprietary systems.
  • BioMistral 7B is derived from Mistral 7B Instruct v0.1 and further pre-trained on PubMed Central for biomedical use.
  • The benchmark contains 10 English medical QA tasks and automatic translations into 7 other languages for multilingual assessment.
  • The study evaluates model truthfulness, calibration, quantization, and model merging alongside the core biomedical model.
  • Datasets, multilingual benchmarks, preprocessing scripts, and models are released under an Apache 2.0 license.

2 Related Work

Related work describes the adaptation of general-purpose language models to medicine, including open-source and proprietary medical models. These efforts motivate BioMistral as a Mistral-based biomedical model.

  • The broader open-source movement has made powerful language-model capabilities available to a wider community through models including Llama 2, Vicuna, Falcon, and Mistral.
  • Domain adaptation encodes medical knowledge into general-purpose models, but earlier encoder and encoder-decoder approaches faced challenges on QA tasks.
  • Open-source medical decoder-only models include MedAlpaca, Chat-Doctor, BioMedGPT-LM-7B, and MediTron-7B.
  • Proprietary medical models such as GPT-4 MedPrompt and Med-PALM 2 have usability issues similar to general-purpose proprietary models.

3 BioMistral

BioMistral adapts Mistral 7B Instruct using a commercial-use subset of PubMed Central, efficiency-oriented batching, model merging, and quantization. The section also motivates merging general and biomedical models to broaden capabilities.

  • 3 BioMistral: BioMistral’s construction uses training, adaptation, model-merging, and quantization modules.
  • 3 BioMistral: The training corpus is the commercially usable subset of the freely accessible PMC Open Access Subset.
  • 3 BioMistral: The pre-training objective uses 3 billion tokens from PubMed Central and targets 1.5 epochs within a 20-hour Jean Zay HPC limit.
  • 3.2 Model Adaptation: Post-tokenization grouping fills 2,048-token sequences without padding and reduces the sequence count by 87.88%.
  • 3 BioMistral: Model merging combines parameters without additional training, while SLERP, TIES, and DARE provide alternative merging strategies.
  • 3 BioMistral: AWQ and BitsandBytes reduce memory requirements to support LLM execution on smaller devices.

4 Evaluation Protocol

The evaluation covers 10 English medical QA tasks, multilingual translations, controlled instruction prompting, and supervised fine-tuning. Models answer multiple-choice tasks through restricted answer-token probabilities.

  • 4 Evaluation Protocol: The benchmark contains 10 English QA tasks from 4 medical corpora spanning genetics, anatomy, clinical cases, and PubMed comprehension.
  • 4 Evaluation Protocol: The benchmark includes MMLU medical subjects, MedQA, MedMCQA, and PubMedQA, with PubMedQA uniquely providing context in the prompt.
  • 4.2 Multilingual Evaluation: The multilingual benchmark translates questions, options, and context into Spanish, German, Portuguese, Russian, French, Arabic, and Chinese.
  • 4.3 Instruction Prompting: Inference filters the vocabulary to answer-choice letters, preventing generation of irrelevant tokens or hallucinations.
  • 4.4 Supervised Fine-Tuning (SFT): QLoRA and 8-bit quantization are used as cost-effective alternatives for supervised fine-tuning of BioMistral and baseline models.

5 Results and Discussions

BioMistral 7B is evaluated across few-shot, fine-tuning, merging, multilingual, quantization, calibration, and truthfulness settings. It generally outperforms open-source biomedical baselines, while results reveal task-, language-, efficiency-, and calibration-dependent trade-offs.

  • 5.1 Few-shot Learning: BioMistral 7B outperforms Mistral 7B Instruct on 8 of 10 few-shot tasks and surpasses all other open-source biomedical baselines on every task.
  • 5.2 Supervised Fine-Tuning (SFT): Supervised fine-tuning improves performance across almost all datasets, with BioMistral 7B outperforming Mistral 7B Instruct on 7 of 10 tasks.It also surpasses every other open-source biomedical baseline and improves substantially on PubMedQA.
  • 5.3 Model Merging: SLERP yields a 5.11% average accuracy gain over BioMistral 7B in few-shot evaluation, while DARE and TIES yield 4.35% and 0.82%.No merging strategy dominates universally; each performs best on specific tasks.
  • 5.4 Multilingual Generalization: Across seven languages, performance decreases relative to English, while merging variants consistently outperform the original model and existing open-source medical counterparts.For BioMistral 7B on MedQA, accuracy ranges from 26.3% in Arabic to 33.7% in Spanish, a 7.4% delta.
  • 5.5 Quantization Techniques: Quantization produces task-dependent accuracy changes and efficiency trade-offs: AWQ increases PubMedQA accuracy by 24.1%, while MedMCQA drops 4.05% on average.AWQ + GEMM processes the MMLU professional medicine test set in 57.96 seconds, 86.23% faster than AWQ + GEMV, with a slight performance loss.
  • 5.6 Calibration: Additional pre-training improves calibration in all languages overall, including a 3.3% ECE gain in English and French, but Chinese shows a 1.6% loss.BioMistral 7B and its base model nevertheless exhibit worse calibration and confidence scores than other models in the reported comparison.

6 Conclusion

BioMistral 7B is a collection of biomedical LLMs created by further pre-training Mistral 7B Instruct on PubMed Central, with quantized and merged variants. It achieves state-of-the-art performance among evaluated open-source 7B models on a multilingual medical benchmark.

  • BioMistral 7B results from further pre-training Mistral 7B Instruct on high-quality PubMed Central resources.
  • The collection includes quantized and merged model variants.
  • BioMistral 7B demonstrates state-of-the-art performance on the multilingual medical evaluation benchmark compared with other open-source 7B models.
  • Future work will assess generation quality through human evaluation and improve multilingual and chat capabilities using supervised fine-tuning and direct preference optimization.
  • Future work also targets improved calibration and reliability through techniques such as Jeffrey’s divergence or Platt scaling during further pre-training.

Limitations

The study’s reproducibility is constrained by substantial computational and inference costs, while replication may become impractical if the GPT-3.5 Turbo version used is no longer maintained. BioMistral may also underperform on unfamiliar medical terminology, non-English contexts, and benchmarks that do not reflect real-world usage.

  • 5,000 hours of A100 80GB GPU computation and additional GPT-3.5 Turbo inference costs make reproduction challenging with limited resources.The reported environmental cost was 1,295,000 Wh or 73.8 kg CO2eq, and GPT-3.5 Turbo translation and evaluation cost $355.47.
  • Future replication may become impractical if the GPT-3.5 Turbo version used in the experiments is no longer maintained.
  • BioMistral may encounter difficulties with unfamiliar or rare medical procedures or terminology and may show degraded performance in non-English contexts.
  • The benchmark may not accurately reflect end users’ actual usage patterns or priorities because it uses selected tasks and metrics for academic assessment.

Ethics Statement

The ethics statement assigns responsibility for generated content to users and notes that harmful, biased, and toxic content disclosure mechanisms are absent. The paper also limits its evaluation scope and identifies unresolved risks and biases.

  • Users are solely responsible for content generated with BioMistral 7B, which has no mechanisms for harmful, biased, or toxic content disclosure.
  • BioMistral 7B was evaluated only on MCQA datasets, which may not reflect effectiveness outside that scope.
  • The model may possess inherent risks and biases that have not yet been thoroughly assessed.
  • Further evaluation across domains is required to assess toxic, rude, or hateful content and understand how bias may perpetuate stereotypical social biases.

C Grouping method algorithm

The grouping method converts unequal-length token sequences into sequences of 2048 tokens by inserting an end-of-sequence separator and flattening the tokens. The supplied passages do not describe the quantization table’s specific values or comparison.

  • Grouping method: The algorithm takes an input list of unequal-length token sequences and returns a list of 2048-token-long sequences.
  • Grouping method: It inserts </s> separators, flattens the token sequences, measures the resulting length, and proceeds when the length is at least 2048.
  • Quantization table: The supplied quantization-table caption identifies memory footprint and inference time on MMLU professional medicine using different approaches on an RTX 3090 GPU.

E Training Loss

BioMistral’s further pre-training on PubMed Central was planned around a 1.5-epoch milestone and evaluated through training loss. The evaluation materials used a shared prompt template across zero-shot, few-shot, and SFT settings, including specialized medical and truthfulness prompts.

  • Training strategy: 1.5 epochs was targeted to balance model performance and training time within the 20-hour Jean-Zay resource limit.The team estimated processing 3 billion tokens per epoch to meet this constraint.
  • Training loss: Figure 1’s training-loss behavior supported the estimated processing plan and resembled Zephyr’s behavior.The loss was measured during further pre-training of Mistral 7B Instruct v0.1 on PubMed Central.
  • Evaluation prompts: PubMedQA was formatted as a multiple-choice task with context and options corresponding to yes, no, and maybe.The answer mapping was A=yes, B=no, and C=maybe.
  • Evaluation prompts: The same instruction template was used across zero-shot, 3-shot, and SFT scenarios, with few-shot examples and context included when applicable.The template was applied across datasets, while optional components depended on the dataset.
  • Evaluation prompts: TruthfulQA medical evaluation used zero-shot accuracy with both a question-answering prompt and a truthful-answer prompt.The two prompts were taken from Chen et al. (2023).

H Multilingual Results

The multilingual-results section reports few-shot model performance on evaluation tasks translated into multiple languages. Scores are presented as accuracy values, but the supplied passage does not include the individual results.

  • Multilingual results: Table 9 reports few-shot model results for evaluation tasks translated into multiple languages.Scores are expressed as accuracy.
Loading 2402.10373v3…