PaperlayerView exact revision on arXiv

Source-linked AI summary

Med42 -- Evaluating Fine-Tuning Strategies for Medical LLMs: Full-Parameter vs. Parameter-Efficient Approaches

Clément Christophe, Praveen K Kanithi, Prateek Munjal, Tathagata Raha, Nasir Hayat, Ronnie Rajan, Ahmed Al-Mahrooqi, Avani Gupta, Muhammad Umar Salman, Gurpreet Gosal, Bhargav Kanakiya, Charles Chen, Natalia Vassilieva, Boulbaba Ben Amor, Marco AF Pimentel, Shadab Khan

arXiv:2404.14779v1cs.CL

TL;DR

The study addresses how full-parameter and parameter-efficient fine-tuning compare for medical LLMs, where domain-specific adaptation must be balanced against computational demands. The authors compare full-parameter fine-tuning with LoRA for medical LLMs using a diverse instruction-tuning dataset combining multiple open datasets focused primarily on medical question-answering. Full-parameter fine-tuning generally outperformed LoRA, while LoRA produced results remarkably close to full-parameter tuning across medical tasks.

  • Problem

    The study addresses how full-parameter and parameter-efficient fine-tuning compare for medical LLMs, where domain-specific adaptation must be balanced against computational demands.

    Source: Introduction · p. 1

  • Method

    The authors compare full-parameter fine-tuning with LoRA for medical LLMs using a diverse instruction-tuning dataset combining multiple open datasets focused primarily on medical question-answering.

    Source: Introduction · p. 1, Training Dataset · p. 2, Modelling · p. 2

  • Results

    Full-parameter fine-tuning generally outperformed LoRA, while LoRA produced results remarkably close to full-parameter tuning across medical tasks.

    Source: Discussion · p. 4

  • Takeaways & Limitations

    Parameter-efficient tuning can be a viable alternative when computational resources are limited, despite lower overall performance than full-parameter tuning.

    Source: Discussion · p. 4

    The authors used minimal experimentation with complex prompt engineering and identify further opportunities to improve response quality and accuracy.

    Source: Discussion · p. 5

Abstract

from arXiv · show

This study presents a comprehensive analysis and comparison of two predominant fine-tuning methodologies - full-parameter fine-tuning and parameter-efficient tuning - within the context of medical Large Language Models (LLMs). We developed and refined a series of LLMs, based on the Llama-2 architecture, specifically designed to enhance medical knowledge retrieval, reasoning, and question-answering capabilities. Our experiments systematically evaluate the effectiveness of these tuning strategies across various well-known medical benchmarks. Notably, our medical LLM Med42 showed an accuracy level of 72% on the US Medical Licensing Examination (USMLE) datasets, setting a new standard in performance for openly available medical LLMs. Through this comparative analysis, we aim to identify the most effective and efficient method for fine-tuning LLMs in the medical domain, thereby contributing significantly to the advancement of AI-driven healthcare applications.

Introduction

The study develops a medical LLM by comparing full-parameter and parameter-efficient fine-tuning, motivated by the need for domain-specific medical understanding while managing computational demands.

Introduction

  • In-domain training data helps LLMs interpret medical terminology and generate text suited to specialized healthcare contexts.The paper notes potential applications in diagnosis support, literature interpretation, patient education, and clinical decision support, alongside concerns about hallucinations, bias, privacy, and ethics.Source: Introduction · p. 1
  • The study compares full-parameter fine-tuning with parameter-efficient methods, especially LoRA, for developing medical LLMs.Full-parameter tuning updates all model parameters, whereas parameter-efficient methods modify a smaller subset to reduce resource requirements.Source: Introduction · p. 1
  • The paper releases Med42 as its most performant model on HuggingFace.Source: Introduction · p. 2

Methods

The methods combine a diverse medical instruction dataset with Llama-2 models, LoRA and full-parameter tuning, response-only loss optimization, and zero-shot evaluation across medical benchmarks.

Training Dataset

  • The training dataset combines open medical question-answering resources, Stack Exchange discussions, and selected medical material from general-domain datasets.This composite design covers diverse medical subfields and contexts.Source: Training Dataset · p. 2

Model evaluation

  • The study fine-tuned 7B and 70B Llama-2 models and evaluated zero-shot accuracy across MedQA, HeadQA, MedMCQA, PubMedQA, MMLU clinical topics, and USMLE materials.Evaluation used the EleutherAI harness; image-containing questions were excluded.Source: Modelling · p. 2, Model evaluation · pp. 2–3

Modelling

  • LoRA inserts trainable low-rank matrices while freezing pretrained weights, reducing trainable parameters and computational costs.Source: Modelling · p. 2
  • Applying LoRA to every linear layer reduced trainable parameters from 7B and 70B to 20M and 104M, respectively.The paper reports this configuration as its best-performing LoRA setting.Source: Modelling · p. 2
  • Training concatenated system, user, and response samples while backpropagating loss only through response tokens.Source: Modelling · p. 2

Model evaluation

  • A decontamination pipeline flagged evaluation examples resembling training samples and enabled re-evaluation after their removal.Source: Model evaluation · p. 3

Results

Across medical benchmarks, fine-tuned models outperformed their base counterparts, with full-parameter tuning generally ahead of LoRA; the best 70B models were also compared with published systems.

Results

  • Fine-tuned models outperformed corresponding base models across all medical benchmark datasets, while full-parameter tuning beat LoRA on most datasets.Figure 1 reports zero-shot accuracy for both 7B and 70B models.Source: Results · p. 3
  • The best fine-tuned 70B models were compared with published medical models across multiple benchmark datasets in Table 1.The comparison includes PE-FT and FP-FT models alongside Clinical Camel, MediTron, GPT-3.5, GPT-4, and MedPaLM-2; some comparator results are few-shot rather than zero-shot.Source: Results · p. 3, Table 1 · p. 4
  • Accuracy changes after removing contaminated examples were reported in Figure 2 alongside the full results.Source: Results · p. 4

Discussion

The study finds that comprehensive fine-tuning improves medical benchmark performance, while LoRA remains a close alternative when computational resources are limited. Decontamination produced only very small accuracy changes, supporting the robustness of the findings.

Discussion

  • The fine-tuned medical LLMs demonstrated high-level medical reasoning and improved performance on domain-specific benchmarks, including complex medical tasks.Source: Discussion · p. 4
  • Full-parameter fine-tuning outperformed LoRA across most medical benchmark datasets, although LoRA results remained remarkably close.The study identifies parameter-efficient tuning as a viable alternative when computational resources are limited.Source: Discussion · p. 4
  • Very small accuracy changes followed removal of contaminated examples from the two larger fine-tuned models’ benchmark evaluations.The authors report that only a small number of samples were contaminated and that decontamination caused no substantial deviation from the original findings.Source: Discussion · p. 4, Figure 2 · p. 4
  • The study compared its fine-tuned models with commercial and other LLMs to position them within the medical-LLM landscape.Source: Discussion · p. 4
  • Med42 showed superior performance to ClinicalCamel, MediTron, and GatorTronGPT, underscoring the importance of a large, well-structured instruction-tuning dataset.Source: Discussion · p. 4
  • Minimal experimentation with complex prompt engineering leaves opportunities to improve response quality and accuracy.Source: Discussion · p. 5
  • Further research is needed to establish the utility of fine-tuned medical LLMs in diverse practical use cases.Source: Discussion · p. 5

Ethical Considerations and Reproducibility

The study emphasizes ethical development and reproducibility through open licensing, freely available evaluation resources, and explicit acknowledgment of clinical-use limitations.

Ethical Considerations and Reproducibility

  • The model is openly licensed, while datasets and evaluation frameworks are freely available to support transparency, collaboration, reproducibility, and independent verification.Source: Ethical Considerations and… · p. 5
  • The authors state that the model should not yet be used as a clinical decision-making tool because it may generate harmful information and perpetuate biases.Source: Ethical Considerations and… · p. 5
  • Ongoing human evaluation focuses on biases, fairness, and safety through red-teaming for clinical settings.Source: Ethical Considerations and… · p. 5

A.1 Related work

Prior medical LLM research includes general-purpose models, biomedical pre-training, and instruction-tuned systems, but training details and comparative evidence remain limited. This study extends that work through a broader comparison of medical fine-tuning methods across multiple benchmarks.

A.1 Related work

  • Decoder-only transformer LLMs have been adapted from general-purpose training to specialized healthcare domains.Source: §A.1 · p. 8
  • General-purpose models such as GPT-3.5, GPT-4, and Med-PaLM have performed strongly on medical tasks, but key training methodologies and weight parameters remain undisclosed for some systems.Source: §A.1 · p. 8
  • Biomedical pre-training systems such as GatorTron, MedGatorTron, and PMC-LLaMA use medical corpora to develop domain-specific representations and capabilities.Source: §A.1 · p. 8
  • Instruction-tuned systems such as ChatDoctor and MedAlpaca adapt LLaMA using medical dialogue and task datasets, with MedAlpaca’s LoRA variant underperforming its full-parameter counterpart on USMLE self-assessment.Source: §A.1 · p. 8
  • The study extends prior work by comparing fine-tuning strategies, including LoRA, across multiple medical benchmarks.Source: §A.1 · p. 8

A.2 Computational Setup

The computational setup used separate infrastructures for parameter-efficient and full-parameter fine-tuning, matching each approach to its computational demands.

A.2 Computational Setup

  • Parameter-efficient fine-tuning and evaluation used eight nodes with sixteen NVIDIA V100 GPUs each.Source: §A.2 · p. 8
  • Full-parameter fine-tuning ran on Cerebras’s Condor Galaxy 1 supercomputer to provide the computational power required for its extensive demands.Source: §A.2 · p. 8

A.3 Training dataset

The fine-tuning dataset combines sampled medical and biomedical sources with general-domain data to balance specialized medical coverage and broader linguistic generalizability.

  • The dataset used sampled subsets of medical and biomedical datasets to keep the training corpus comprehensive yet manageable.Source: §A.3 · p. 8
  • OpenOrca T0, Flan, and CoT contributed 295,649 samples, representing 41.83% of the mixture.Source: §A.3 · p. 9
  • 60% of the final dataset was medical-domain data, while 40% was general-domain data.Source: §A.3 · pp. 8–9

A.4 Decontamination analysis

The decontamination analysis identified potentially overlapping instruction-tuning and evaluation samples using similarity-based screening, then tested whether excluding them changed reported performance.

  • The pipeline used cosine similarity between SentenceTransformers embeddings and marked samples contaminated when similarity exceeded 0.8.Source: §A.4 · p. 9
  • Less than 2% of evaluation samples were contaminated, mostly through differently phrased questions about similar diseases or clinical concepts.Source: §A.4 · p. 9
  • Longer use-case questions could be flagged because of high wording overlap even when their final questions differed.Source: §A.4 · p. 9
  • Removing potentially contaminated samples produced no substantial deviation from the original evaluation results.Source: §A.4 · pp. 9–10
Loading 2404.14779v1…